As mentioned in a previous post, with LibreNMS you can easily manage the monitoring of a network. One of the features of LibreNMS is to send alerts to your email when something fails.
The logical next step is to open a ticket with the Internet Service Providers (ISPs) to check their connection (if you did NOT buy an expensive service that provides monitoring from ISPs).
But, why do this manually ? (or have a NOC 1st line team do it)
Here is where automation comes into place.
Automation Flow
So lets you already configured LibreNMS to send alerts to email and that you receive an alert in an email called: noc@example.com
If that is the case, then this could be a possible automation Flow:
- LibreNMS generates Alert ->
- Alert arrives to noc@example.com ->
- Script parses the alert ->
- Script sends automatically an email to the ISPs (with users in CC) ->
- Email is deleted (so that when the script runs again, we won’t spam the ISPs)
Script
import requests
import ipaddress
import exchangelib as E
import sys
import email
import requests
import json
import re
import time
librenms_api = "http://IP_ADDRESS/api/v0"
TOKEN = "TOKEN"
exchange_username = "domain\\username"
exchange_password = "username_password"
exchange_server = "webmail.example.com"
exchange_smtp_address = "username@example.com"
exchange_cc_recipients= ['noc@example.com','helpdesk@example.com']
email_ISP1 = ['email1@isp1.com','email2@isp1.com']
email_ISP2 = ['email1@isp2.com','email2@isp2.com']
email_ISP3 = ['email1@isp3.com','email2@isp3.com']
headers = {'User-Agent': 'LibreNMS',
'X-Auth-Token': TOKEN
}We are going to run over the script now and what everything means so that you can easily use/change it.
First we import all our needed libraries.
We then define the following:
librenms_api = the IP address of where we installed LibreNMS, this is needed so once we get an alert, we can get its public address! (as a device will have also private IP addresses and if we send that to ISP they won’t know what to do with it)
TOKEN = the token from LibreNMS
Exchange credentials = username, pass, server, smtp server for your exchange server
exchange_cc_recipients = who would you like to add to CC in the email sent to the ISPs?
email_ISPx = emails of the ISP. I’ve created the script so it can take up to 3 different ISPs
headers = needed to make the API request towards LibreNMS
def is_ISP1(ipaddress):
ISP1 = {
#Example Usage: "IP" : ( "ServiceID", "Location" )
"1.1.1.1" : ("SERVICEID" , "Manhattan, New York"),
"1.1.1.2" : ("SERVICEID" , "Manhattan, New York"),
}
if ipaddress in ISP1.keys():
service_id = ISP1[ipaddress][0]
location = ISP1[ipaddress][1]
return service_id,location
else: return False
def is_ISP2(ipaddress):
ISP2 = {
#Example Usage: "IP" : ( "ServiceID", "Location" )
"2.1.1.1" : ("SERVICEID" , "Manhattan, New York"),
"2.1.1.2" : ("SERVICEID" , "Manhattan, New York"),
}
if ipaddress in ISP2.keys():
service_id = ISP2[ipaddress][0]
location = ISP2[ipaddress][1]
return service_id,location
else: return False
def is_ISP3(ipaddress):
ISP3 = {
#Example Usage: "IP" : ( "ServiceID", "Location" )
"3.1.1.2" : ("SERVICEID" , "Manhattan, New York"),
"3.1.1.2" : ("SERVICEID" , "Manhattan, New York"),
}
if ipaddress in ISP3.keys():
service_id = ISP3[ipaddress][0]
location = ISP3[ipaddress][1]
return service_id,location
else: return FalseWe define 3 functions (is_ISP1, 2 ,3)
Here you define all your external IP addresses of your routers, the Service ID and Location.
You need this so that you can send email to the proper ISP, with the proper Location and ServiceID
def query_ip(query_ip):
query_url = librenms_api + "/devices?type=ipv4&query=" + \
str(query_ip)
r = requests.get(query_url, headers=headers)
json = r.json()
device_id = json["devices"][0]["device_id"]
print(device_id)
return device_id
def get_all_ips(device_id):
ip_url = librenms_api + "/devices/" + str(device_id) + "/ip"
r = requests.get(ip_url, headers=headers)
json = r.json()
total_ips = len(json["addresses"])
for i in range(total_ips):
# print (json["addresses"][i]["ipv4_address"])
ip_address = json["addresses"][i]["ipv4_address"]
PUBLIC = ipaddress.ip_address(ip_address).is_private
if PUBLIC is False:
return ip_address
else:
return None
We then have 2 functions:
- query_ip = it searches LibreNMS API for the IP address is saw in the email
- get_all_ips = based on the IP from the email, it will search LibreNMS API for all IP addresses your device has (it will grab and use only the public one).
def send_alert_email(public_ip, service_id, location, email_isp):
email_body = "Hello, \n Our monitoring system shows that a device is down. The IP is: " + public_ip + "\n ServiceID: " + service_id + "\n Locaation: " + location + "\n Can you please help us in checking the issue ? \n P.S.: Please do not forget to add our colleagues from CC when replying.\n Have a good day, \n NOC Team \n"
creds = E.Credentials(username=exchange_username, password=exchange_password)
config = E.Configuration(server=exchange_server, credentials=creds)
a = E.Account(primary_smtp_address=exchange_smtp_address, config=config, autodiscover=False, access_type=E.DELEGATE)
m = E.Message(
account=a,
subject='Router Down',
body=email_body,
to_recipients=email_isp,
cc_recipients=exchange_cc_recipients, # Simple strings work, too
)
m.send_and_save()This sends the alert to the ISP. You can change the body and subject without any issues.
def check_email():
creds = E.Credentials(username=exchange_username, password=exchange_password)
config = E.Configuration(server=exchange_server, credentials=creds)
account = E.Account(primary_smtp_address=exchange_smtp_address, config=config, autodiscover=False, access_type=E.DELEGATE) # logging onto Exchange
email_filtered = account.inbox.filter(subject__icontains="Devices up/down")
re_ip = re.compile("\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}")
if email_filtered.exists(): # if there are any e-mails for this filter
for i in email_filtered: # for each e-mail
#print(i.body)
ip = re.findall(re_ip,i.body)
for j in ip:
if not None:
email_filtered.delete()
return j
This in the functions that checks the email for alerts from LibreNMS. Please be aware that all alerts for router down should have the subject
” Devices up/down“ (the default of LibreNMS).
If you changed that, change it here as well.
get_alert_ip = check_email()
device_id = query_ip(get_alert_ip)
public_ip = get_all_ips(device_id)
print(public_ip)
if is_ISP1(public_ip):
email_isp = email_ISP1
service_id,location = is_ISP1(public_ip)
send_alert_email(public_ip, service_id, location, email_isp)
print("Sending Email to ISP1")
if is_ISP2(public_ip):
email_isp = email_ISP2
service_id,location = is_ISP2(public_ip)
send_alert_email(public_ip, service_id, location, email_isp)
print("Sending Email to ISP2")
if is_ISP3(public_ip):
email_isp = email_ISP3
service_id,location = is_ISP3(public_ip)
send_alert_email(public_ip, service_id, location, email_isp)
print("Sending Email to ISP3")This code just starts executing all the code above and sends email based on ISPs.
How to use all of this ?
Well just copy paste all the config to a file and set it up into a crontab and set it to check every minute.
All source code can be found also here: https://github.com/skillfulist-com/netdevops/blob/master/email-alert-automation.py
Hope it help you guys!
If you have any questions, remarks please let me know.
Some considerations of the script:
- There is a better method then just adding IPs/ServiceIDs/Locations to a script, maybe pulling that info from Netbox. Will create it soon.
- If you get an alert from a router and it is NOT in the list for any ISP, the script will give an error and exit