Experiencing a breach?

Automatically alert ISPs if Link/Router is down

I wanted to show an easy way to automate alerting ISPs in case your link/connection/router is down.

As mentioned in a previous post, with LibreNMS you can easily manage the monitoring of a network. One of the features of LibreNMS is to send alerts to your email when something fails.

The logical next step is to open a ticket with the Internet Service Providers (ISPs) to check their connection (if you did NOT buy an expensive service that provides monitoring from ISPs).

But, why do this manually ? (or have a NOC 1st line team do it)

Here is where automation comes into place.

Automation Flow

So lets you already configured LibreNMS to send alerts to email and that you receive an alert in an email called: noc@example.com

If that is the case, then this could be a possible automation Flow:

  1. LibreNMS generates Alert ->
  2. Alert arrives to noc@example.com ->
  3. Script parses the alert ->
  4. Script sends automatically an email to the ISPs (with users in CC) ->
  5. Email is deleted (so that when the script runs again, we won’t spam the ISPs)

Script

import requests
import ipaddress
import exchangelib as E
import sys
import email
import requests
import json
import re
import time

librenms_api = "http://IP_ADDRESS/api/v0"
TOKEN = "TOKEN"
exchange_username = "domain\\username"
exchange_password = "username_password"
exchange_server = "webmail.example.com"
exchange_smtp_address = "username@example.com"
exchange_cc_recipients= ['noc@example.com','helpdesk@example.com']

email_ISP1 = ['email1@isp1.com','email2@isp1.com']
email_ISP2 = ['email1@isp2.com','email2@isp2.com']
email_ISP3 = ['email1@isp3.com','email2@isp3.com']


headers = {'User-Agent': 'LibreNMS',
           'X-Auth-Token': TOKEN
           }

We are going to run over the script now and what everything means so that you can easily use/change it.

First we import all our needed libraries.

We then define the following:

librenms_api = the IP address of where we installed LibreNMS, this is needed so once we get an alert, we can get its public address! (as a device will have also private IP addresses and if we send that to ISP they won’t know what to do with it)

TOKEN = the token from LibreNMS

Exchange credentials = username, pass, server, smtp server for your exchange server

exchange_cc_recipients = who would you like to add to CC in the email sent to the ISPs?

email_ISPx = emails of the ISP. I’ve created the script so it can take up to 3 different ISPs

headers = needed to make the API request towards LibreNMS


def is_ISP1(ipaddress):
    ISP1 = {
        #Example Usage: "IP" : ( "ServiceID", "Location" )
    "1.1.1.1" : ("SERVICEID" , "Manhattan, New York"),
    "1.1.1.2" : ("SERVICEID" , "Manhattan, New York"),
    }

    if  ipaddress in ISP1.keys():
        service_id = ISP1[ipaddress][0]
        location = ISP1[ipaddress][1]
        return service_id,location
    else: return False

def is_ISP2(ipaddress):
    ISP2 = {
        #Example Usage: "IP" : ( "ServiceID", "Location" )
    "2.1.1.1" : ("SERVICEID" , "Manhattan, New York"),
    "2.1.1.2" : ("SERVICEID" , "Manhattan, New York"),
    }
    if  ipaddress in ISP2.keys():
        service_id = ISP2[ipaddress][0]
        location = ISP2[ipaddress][1]
        return service_id,location
    else: return False

def is_ISP3(ipaddress):
    ISP3 = { 
        #Example Usage: "IP" : ( "ServiceID", "Location" )
    "3.1.1.2" : ("SERVICEID" , "Manhattan, New York"),
    "3.1.1.2" : ("SERVICEID" , "Manhattan, New York"),

    }
    if  ipaddress in ISP3.keys():
        service_id = ISP3[ipaddress][0]
        location = ISP3[ipaddress][1]
        return service_id,location
    else: return False

We define 3 functions (is_ISP1, 2 ,3)

Here you define all your external IP addresses of your routers, the Service ID and Location.

You need this so that you can send email to the proper ISP, with the proper Location and ServiceID

def query_ip(query_ip):
    query_url = librenms_api + "/devices?type=ipv4&query=" + \
        str(query_ip)
    r = requests.get(query_url, headers=headers)
    json = r.json()
    device_id = json["devices"][0]["device_id"]
    print(device_id)
    return device_id


def get_all_ips(device_id):
    ip_url = librenms_api + "/devices/" + str(device_id) + "/ip"
    r = requests.get(ip_url, headers=headers)
    json = r.json()
    total_ips = len(json["addresses"])
    for i in range(total_ips):
        # print (json["addresses"][i]["ipv4_address"])
        ip_address = json["addresses"][i]["ipv4_address"]
        PUBLIC = ipaddress.ip_address(ip_address).is_private
        if PUBLIC is False:
            return ip_address
        else:
            return None

We then have 2 functions:

  1. query_ip = it searches LibreNMS API for the IP address is saw in the email
  2. get_all_ips = based on the IP from the email, it will search LibreNMS API for all IP addresses your device has (it will grab and use only the public one).
def send_alert_email(public_ip, service_id, location, email_isp):
    email_body = "Hello, \n Our monitoring system shows that a device is down. The IP is: " + public_ip + "\n ServiceID: " + service_id + "\n Locaation: " + location + "\n Can you please help us in checking the issue ? \n P.S.: Please do not forget to add our colleagues from CC when replying.\n Have a good day, \n NOC Team \n"
    creds = E.Credentials(username=exchange_username, password=exchange_password)
    config = E.Configuration(server=exchange_server, credentials=creds)
    a = E.Account(primary_smtp_address=exchange_smtp_address, config=config, autodiscover=False, access_type=E.DELEGATE)
    m = E.Message(
        account=a,
        subject='Router Down',
        body=email_body,
        to_recipients=email_isp,
        cc_recipients=exchange_cc_recipients,  # Simple strings work, too
    )
    m.send_and_save()

This sends the alert to the ISP. You can change the body and subject without any issues.

def check_email():
    creds = E.Credentials(username=exchange_username, password=exchange_password)
    config = E.Configuration(server=exchange_server, credentials=creds)
    account = E.Account(primary_smtp_address=exchange_smtp_address, config=config, autodiscover=False, access_type=E.DELEGATE)  # logging onto Exchange
    email_filtered = account.inbox.filter(subject__icontains="Devices up/down")
    re_ip = re.compile("\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}")				
    if email_filtered.exists():  # if there are any e-mails for this filter
        for i in email_filtered:			# for each e-mail
            #print(i.body)
            ip = re.findall(re_ip,i.body)
            for j in ip:
                if not None:
                    email_filtered.delete()
                    return j

This in the functions that checks the email for alerts from LibreNMS. Please be aware that all alerts for router down should have the subject

” Devices up/down“ (the default of LibreNMS).

If you changed that, change it here as well.

get_alert_ip = check_email()
device_id = query_ip(get_alert_ip)
public_ip = get_all_ips(device_id)
print(public_ip)

if is_ISP1(public_ip):
    email_isp = email_ISP1
    service_id,location = is_ISP1(public_ip)
    send_alert_email(public_ip, service_id, location, email_isp)  
    print("Sending Email to ISP1")
if is_ISP2(public_ip):
    email_isp = email_ISP2
    service_id,location = is_ISP2(public_ip)
    send_alert_email(public_ip, service_id, location, email_isp)  
    print("Sending Email to ISP2")
if is_ISP3(public_ip):
    email_isp = email_ISP3
    service_id,location = is_ISP3(public_ip)
    send_alert_email(public_ip, service_id, location, email_isp)    
    print("Sending Email to ISP3")

This code just starts executing all the code above and sends email based on ISPs.

How to use all of this ?

Well just copy paste all the config to a file and set it up into a crontab and set it to check every minute.

All source code can be found also here: https://github.com/skillfulist-com/netdevops/blob/master/email-alert-automation.py

 

Hope it help you guys!

If you have any questions, remarks please let me know.

 

Some considerations of the script:

  1. There is a better method then just adding IPs/ServiceIDs/Locations to a script, maybe pulling that info from Netbox. Will create it soon.
  2. If you get an alert from a router  and it is NOT in the list for any ISP, the script will give an error and exit