Create your own DownNotifier with OpenFaaS

Johnny Mkhael -

I went to a three weeks holiday during august, and I was happy to see that my blog (hosted on an old Raspberry Pi 1) kept working the whole time!

Call it good or bad luck, but on the day I returned my blog just stopped working.

I discovered it by chance, while I was playing with ansible and inadvertently triggered an uptime command on the Raspberry Pi hosting the blog. It just hang endlessly.

All I had setup back then, was a free alert on https://www.downnotifier.com/
I knew that this isn't enough and I had to do something.

DownNotifier sends a free alert when your website is down

The kind of alerts you get are mail and SMS. You need to buy credits for SMS to work. Free means that all you get is an email when DownNotifier see that your site is down.

I need a better way to monitor my blog - JM

What we will build

Using Docker, OpenFaaS and a bit of typing, this is what I ended up with in a matter of an hour:

In the top section you can find logs of an OpenFaaS function, the brain of all this, in the bottom section I invoke this function on an inexistant site, and on the right you can see an incoming message to Slack with the site down information!

Problem and solution

The problem I wanted to solve, can be simply formulated as:

I need to be notified in under 15 seconds if my blog goes down.
The notification medium is Slack. I want to see the observed error from the client side.

I wanted needed to write something in a matter of an hour, so I feel a bit safer, then make it better later on.

The idea I had in mind is very simple:

  1. Schedule a regular ping to the blog
  2. If everything is ok, say nothing on Slack
  3. If it fails, send a message to a Slack channel dedicated to this (#blog)

But of course I didn't want to write a single script to do that! I wanted the solution to remain composable, or to quote Docker’s CTO - Solomon Hykes:

"batteries included, but replaceable"

So here comes my solution using serverless functions to allow a fast, scalable and modular design. We’re going to use the OpenFaaS serverless framework for Docker and Kubernetes.

Why is OpenFaaS right for Serverless?

With OpenFaaS you can go serverless, without any vendor lock-in.

Some of the nice features of OpenFaaS include:

  • It runs on existing hardware or public/private cloud - Kubernetes or Docker Swarm
  • It’s easy to use and has a built-in UI and one-click deployment
  • The CLI takes you step-by-step, from scaffolding new functions to deploying them on your cluster
  • It comes with bells and whistles such as auto-scaling with respect to demand increase and alerting/monitoring...

but the reason #1 remains the same:

But, How?

In order to implement the above steps, I though of creating 3 functions: one that pings, one that send to slack and the last one, the brain, coordinating all of this.

  • url_ping_node: this function is a rewrite of url_ping but in node. It basically pings a site and returns a json with error details in case it fails. It also uses RxJS, reactive model for composing asynchronous and event-based programs by using observable sequences. I will need this to retry/timeout later on.
  • slack_it: this function sends a message to a slack channel
    It uses Botkit. It needs a token to access the Slack team. By default it will post to a #blog channel.
  • down_notifier: this is the top level function or the brain. It calls other functions: first url_ping_node and then decides to pipe a message to slack_it if the ping fails or not.

and basically, all what remains is to schedule a recurring call to the down_notifier. (This part is still hackish)

Note

You can find all the code in the accompanying GitHub repository here: https://github.com/jmkhael/blog-pinger, but be sure to read along, as I'll highlight the most relevant sections.

Implementation

Before starting to write our functions, let's install faas-cli which is a very nice CLI to list, create new, build, deploy, invoke, push and remove functions to list a few.

It is the Official CLI for the OpenFaaS Serverless framework - GitHub repo

Pre-reqs

Get the faas-cli, if you haven't yet. You really have to if you want to be productive!
The easiest way to install the faas-cli is through curl or brew:

$ curl -sSL https://cli.openfaas.com | sudo sh

or:

$ brew install faas-cli

Start faas-cli and be amazed by it's ascii art and help message:

Scaffold

Time to create our functions. Let's use the faas-cli to do so.

faas-cli new --name url_ping_node --lang node
faas-cli new --name down_notifier --lang node
faas-cli new --name slack_it --lang node
rm master.zip
ls

Have a look at what faas-cli just scaffolded for us:

$ ls
down_notifier  down_notifier.yml  
slack_it  slack_it.yml  template  
url_ping_node  url_ping_node.yml

Each yml file is basically a specification we can use to configure/build/deploy individual functions.

You can merge them out into a single 'stack'.

cat *.yml > samples.yml

Check out my samples.yml file for an example.

Writing the functions logic

faas-cli prepared pretty much everything!

faas-cli, is like Rick, building dedicated robots in no time!

At the time of writing this post, faas-cli don't know, yet, what's in our mind. So we have to write the functions logic ourselves...

Below I am highlighting the most important aspects of these functions, but you can always refer to the github repo for details.

down_notifier is kind of more evolved as it chain several functions together. Make sure to check it's code.

url_ping_node

I won't detail much of the code here as it doesn't do a lot besides what will be covered below.

This function will connect to a URL over HTTP and return the status code, or return the error encountered.

$ echo '{"url":"http://jmkhael.io/"}' \
| faas-cli invoke --name url_ping_node

{"url":"http://jmkhael.io/","status":"ok","status_code":200}

$ echo '{"url":"http://jmkhael.no/"}' \
| faas-cli invoke --name url_ping_node 
{
   "url":"http://jmkhael.no/",
   "status":"nok",
   "error":{
         "code":"ENOTFOUND",
         "errno":"ENOTFOUND",
         "syscall":"getaddrinfo",
         "hostname":"jmkhael.no",
         "host":"jmkhael.no",
         "port":80
   }
}

Note that you can do the same with curl if you like. But using faas-cli is more elegant as we saw above:

$ curl -d '{"url":"http://jmkhael.io/"}' \
http://localhost:8080/function/url_ping_node

$ curl -d '{"url":"http://jmkhael.no/"}' \
http://localhost:8080/function/url_ping_node

slack_it

This function mission in life is to send a message to a slack channel.

This uses Botkit. By default it will post to a #blog channel.

You need to specify in the environment section a token enabling the function to join your Slack team.

  slack_it:
    lang: node
    handler: ./slack_it
    image: jmkhael/faas-slack
    environment:
      token: "xoxb-insert-your-slack-token-here-but-do-not-add-it-to-github"

The function will require an input in the form of:

$ curl -d '{"message" : "Anything really"}' \
 http://localhost:8080/function/slack_it

The message can be any json object. It will be forwarded to the Slack channel.

$ echo '{"message": {"happy": "yes, always!"} }' \
| faas-cli invoke --name slack_it

down_notifier

This function is "the most complicated part" as it orchestrates the whole.
It calls url_ping_node and decides to send a message to slack_it if the ping fails. You can find below the full code (~50 lines)!

The interesting bits, are the calls to post('http://gateway:8080/function/url_ping_node') where we call the first function, and then pipe the result to slack_it if the status is "nok".

'use strict'

const RxHttpRequest = require('rx-http-request').RxHttpRequest;
var Rx = require('rxjs/Rx');

function down_notifier(url) {
  return RxHttpRequest.post('http://gateway:8080/function/url_ping_node', {
      body : {
        url: url
      },
      json: true
    })
    .map(r => r.body)
    .map(r => {
      //console.log("parsed json from url_ping: " + r.status);
      if(r.status !== "ok") {
        return RxHttpRequest.post('http://gateway:8080/function/slack_it', {
            body: r,
            json: true
        })
        .map(r => r.body);
        // "ping failed - told slack"
      } else {
        return Rx.Observable.of("all ok - slack not informed");
      }
    });
}

module.exports = (content, callback) => {
    //console.log("Slackin: " + content);
    let req = JSON.parse(content);

    down_notifier(req.url).subscribe(
        (x) => {
          x.subscribe(r => {
            let res = {
              "DownNotifier" : "v1",
              "request" : req,
              "response": r
            };

            callback(null, JSON.stringify(res));
          });
        },
        (err) => {
            console.log('Error: ' + err);
        },
        () => {
            //console.log('Completed');
        });
};

Build and deploy

Building our functions is as easy as doing:

$ faas-cli -action build -f ./samples.yml

faas-cli is getting better everyday thanks to the great contributions of the community!
e.g. do faas-cli build --parallel 4 for instance to make things quicker and faas-cli build --filter=name_of_single_function to build just that one.

In order to deploy, you just issue the command:

$ faas-cli -action deploy -f ./samples.yml

To stay within Dockerland, I am currently using docker-flow-cron to ping the down_notifer function at regular intervals. I'd like to replace this part but something more native to OpenFaaS.

My deploy.sh script basically deploys the faas functions we just wrote, and docker-flow-cron stuff. It can be simplified later on.

What can be made better?

Let's start by saying that I did this in a matter of an hour or two tops. It took more time to write this blog than to code this!

I wanted something functioning. I didn't have time to nitpick.

Given time, I would like to:

  • Replace the cron logic with a native OpenFaaS one: wait for or contribute back a faas-cron or a faas-scheduler.
  • Make the deployment better: now I am dealing with docker stack deploy, docker service create and faas-cli deploy. I need an easier way when my stack is mixed.
  • Check a list of pages, not just the home page
  • Setup prometheus as well explained by Alex, here and there, and send alerts to slack or wherever
  • Use prometheus alertmanager, which is already part of OpenFaaS stack
  • Refactor and move some of these function to faas-and-furious organization

Where to go next?

Things you can do next:

  • Play with this/propose pull requests
  • Star my repository if you want to show support
  • Checkout OpenFaaS and deploy your first function
  • Check OpenFaaS samples on faas-and-furious

You can even contribute to OpenFaaS as an active community that welcomes contributions, if you’d like to get involved head over to:

  • OpenFaaS or/and faas-cli
  • Samples on faas-and-furious. a function like faas-cron would be nice :-)

Show your support by giving us a Star on the project's GitHub.

If you'd like to join the amazing OpenFaaS community Slack channel to chat with contributors or get some help - then send a Tweet to @openfaas, @alexellisuk or email alex@openfaas.com.

You might also like: