Why?

Why would anyone sane code using such an old language?

Why not? - JM

After my quick dive into FaaS, I was thinking what would be the oldest language I've written with and how much time would it take me to wrap it up in a Docker image and drop in a FaaS function?

Less than an hour

You can check out the code here:
https://github.com/jmkhael/faas-qbasic-morse/

Building and deploying

Build the Docker image.

docker build . -t jmkhael/faas-qbasic-morse

Ship it to FaaS

Either via faas-cli

Build the FaaS cli first:

git clone https://github.com/alexellis/faas-cli
cd faas-cli/
go get -d -v
go build

Then use the cli to ship the newly built image:

./faas-cli -action=deploy \
   -image=jmkhael/faas-qbasic-morse \
   -name=morse \
   -handler="python handler.py" \
   -lang=python

Alternatively, ship with curl

You can always use curl to deploy the function to FaaS:

curl localhost:8080/system/functions -d '{
   "service": "morse",
   "image": "jmkhael/faas-qbasic-morse",
   "envProcess": "python handler.py",
   "network": "func_functions"
  }'

or if you are a UI portal kind of person, navigate to: http://localhost:8080

Test the Morse Code Serverless Function

Time to test our function, let's run it:

curl -X POST http://localhost:8080/function/morse -d "FaaS-inating!".

and see it answer in Morse:

  ..-. .- .- ...  .. -. .- - .. -. --.

Now you can talk Morse via QBasic/Faas!

(I've based most of my code shamelessly on the repo and adapted it to be able to drop in a QBasic FaaS function here - it was about going fast anyway)

This post will introduce Serverless, then showcase Functions as a Service - a serverless framework for Docker, winner in the Cool Hacks contest for Dockercon 2017, by Docker Captain @alexellisuk.

In this episode, we will:

0. briefly touch on what Serverless is/means (to me)
1. test drive FaaS for Docker
2. test one of the already available functions (demo)
3. create and deploy a new function

What is "Serverless"?

Serverless architectures are internet based systems where the application development does not use the usual server process. Instead they rely solely on a combination of third-party services, client-side logic, and service hosted remote procedure calls (FaaS). - You can read more on Martin Fowler's bliki

Internet based does not mean you cannot host this on your machine!

An easier definition

In laymen terms, you write the code, usually event driven, and someone else provision the servers, the running environment, and execute your code!
It promises to handle spikes in traffic, scheduling your code as necessary to serve requests. Requests are usually billed per execution rather than per execution time.

Why use it?

The real incentive to use a Serverless architecture are cost, (if you have non constant or bursty traffic) and the architectural style/programming model (one step closer to a functional world).

AWS Lambda, Google Cloud Functions, IBM OpenWhisk or Azure Functions are examples of implementation of today's cloud providers.

There is no free lunch though

Like most things, Serverless comes with it's set of concerns. but we won't talk about those for now. (I'll name just one: testing is harder)

A misnomer, really...

But wait, after all, there are servers! Then why do we call it Serverless?

should it be Server(less-ops)?

It is just that:

• we do not have to provision, or operate those servers
• we do not have to do the capacity planning for the peak demand
• we do not have to handle the autoscaling
• we approach a pay as you go model

What can I use Serverless for, today?

Do you want to build some RESTful APIs to respond to an action, respond with data, or just serve static content, like images or gif?

Maybe you want to create a webhook for when your Github repo gets a star and you want to light up a bulb in your house?

Maybe even spin off your own IFTTT or Zapier?

Well, these kind of use cases hit Serverless sweet spot.

Where it doesn't really fit (yet)

• HPC (due to loss of pricing benefits and/or latencies associated with the model)

How to get started today?

You might say to yourself, ok this seems like a thing to learn, but I don't want to spend hours paying while/learning/locked in/.. about consoles of AWS, Azure, GCP or any other.

This is one of the main drives behind FaaS - to make it easy to build and test Serverless architecture on your own hardware. (For the whole background story leading to FaaS, you can read this post by Alex Ellis)

Let's go already.

Hand's on FaaS

To test drive FaaS, Alex was nice enough to get you going in a one liner!

All you need to do first is install Docker:

curl -sSL get.docker.com | sh

(And if you are lazy, head over to http://play-with-docker.com/ and click "Add new Instance". - but this won't be on your server - unless you deployed play with docker on your machine as per my previous post)

docker swarm init && \
  git clone https://github.com/alexellis/faas && \
  cd faas && \
  ./deploy_stack.sh && \
  docker service ls

Once all services listed with docker service ls are up and running (all services have at least one replica) :

NAME                MODE                REPLICAS            IMAGE
func_echoit         replicated          1/1                 functions/alpine:health
func_base64         replicated          1/1                 functions/alpine:health
func_markdown       replicated          1/1                 alexellis2/faas-markdownrender:latest
func_nodeinfo       replicated          1/1                 alexellis2/faas-nodeinfo:latest
func_decodebase64   replicated          1/1                 functions/alpine:health
func_wordcount      replicated          1/1                 functions/alpine:health
func_gateway        replicated          1/1                 functions/gateway:0.5.3
func_webhookstash   replicated          1/1                 functions/webhookstash:latest
func_prometheus     replicated          1/1                 functions/prometheus:1.5.2
func_hubstats       replicated          1/1                 alexellis2/faas-dockerhubstats:latest
func_alertmanager   replicated          1/1                 quay.io/prometheus/alertmanager:latest

First encounter with FaaS

At this point you can start interacting with FaaS.

Navigate to: http://localhost:8080/ in order to see the UI portal of FaaS.

This UI reminds me a bit of Postman.

Here you will see a list of functions prepackaged with FaaS, already deployed by the above command, waiting for you to test!

There is more than meet the eyes in this simple deployment. For instance, here we already have a portable, easy to use, auto-scaling on demand with Prometheus metrics/alerting! and a Gateway... and the ability to add new functions...

Testing a prepackaged function

Let's test two prepackaged functions, one through the UI and the other through the REST interface.

Interacting through the UI

Let's test the wordcount function through the UI. This function expects a text in and will respond with the number of words/letters in that text.

For that, just click on the left func_wordcount:

Type anything you like (including the text, anything you like ;)) in the Request body then click INVOKE.
You should see the Response body adapt with the number of words/letters and the function's invocation count increment!

Sweet.

Interacting via curl

Let's now test the markdown function through the command line. This function expects a markdown text, and will convert it to an html equivalent:

curl -X POST http://localhost:8080/function/func_markdown -d "> This is FaaS in action! - JM"

and get the html back:

<blockquote>
<p>This is FaaS in action! - JM</p>
</blockquote>

Packaging my own function

Ok that's really nice. But what about adding my own function?

Again, we can do this either through the UI or via the REST API.

Let's create a function to reverse text given as input...

To do that, I will use an Alpine base image and the rev command.

Click CREATE NEW FUNCTION link on the left

and fill the form as per the below:

The equivalent to this from the command line:

curl localhost:8080/system/functions -d '{"service": "esrever", "image": "functions/alpine", "envProcess": "rev", "network": "func_functions"}'

Or you can also use the new and shiny faas-cli to scaffold, build and deploy to FaaS. (comming in next post)

Now, check the UI to see the newly added function:

Test it by giving it the service name we used: esrever, so it become reverse.

You can also do that from the command line of course:

curl -X POST http://localhost:8080/function/esrever -d ".emosewA si SaaF###"

FaaS is Awesome.

Coming up soon

In a future post we will explore how FaaS will behave on my hybrid Swarm of Raspberry Pi, composed of one Zero, few Rpi2 and few Rpi3.

We will also be writing some code in Python, and deploying it to FaaS ... all this using the new and shiny faas-cli

We may wire an IFTTT webhook or do an image recognition using TensorFlow :)

Stay tuned.

If you don't know what play-with-docker is, GO CHECK IT OUT NOW!
It is one of the Moby cool hacks winners! (The other one is FaaS by @alexellisuk - check my post about it here)

play-with-docker - PWD

So, what is Play With Docker?

Hacked by two Docker captains, Marcos Nils(@marcosnils) and Jonathan Leibiusky(@xetorthio), basically, Play with Docker is a Docker playground which you can run directly in your browser. With this, you have no excuse anymore to, well, not play with docker...

In the below screenshot I've deployed a local PWD, and in it you see a swarm of 5, (1 manager, 4 workers) running locally on my machine, with a stack deployed.

But I wanted mine too

If for whatever reason you want to deploy this on your hardware or VMs, you can (Give access to people via the browser on premises without AWS cost or whatnot), just follow along this quick and dirty post.

What I wanted to do is some Docker-inception. Have a PWD running in my local Swarm and in it a Swarm running LinuxKit :)

git clone https://github.com/franela/play-with-docker
cd play-with-docker/
docker pull franela/dind

sudo apt install golang-go

export GOPATH=$HOME/.go
go get -v -d -t ./...
docker-compose up

that's it!

Head over to http://localhost and you can start clicking "Add New Instance"

Let's try to deploy a stack onto it now:

Create an instance, you'll get a shell, then on it run:

docker swarm init --advertise-addr eth0

You'll get something like the below output:

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-31ovfkdk8wrna2y3igjzdfqhdm0e72ghksb7m441j1hxr1wtwh-81smoz5hu4mrdqzblrydjh86i \
    10.0.0.7:2377

You can join other workers in if you'd like. To do so, click "Add New Instance" again and paste the command in the shell to join your workers.

Then, let's deploy our monitoring stack, which basically has portainer.io and manomarks/visualizer:

curl -o docker-compose-stack-monitoring.yml \
https://raw.githubusercontent.com/jmkhael/compose-samples/master/docker-compose-stack-monitoring.yml

docker stack deploy -c docker-compose-stack-monitoring.yml monitoring
docker stack ps monitoring

and here you'll see that the services are up and running!

ID            NAME                    IMAGE                        NODE   DESIRED STATE  CURRENT STATE           ERROR  PORTS
qjqciab49qqa  monitoring_viz.1        manomarks/visualizer:latest  node5  Running        Running 30 minutes ago
ske7lcocg8v6  monitoring_portainer.1  portainer/portainer:latest   node5  Running        Running 32 minutes ago
[node5] (local) root@10.0.0.7 ~

Sweet! It took me much more time to write this blog than to have play-for-docker deployed locally!

Bonus

Basically, I want to build the redis os from the keynote in PWD from the Moby cool hack!

Beyond this point, you are left alone in the Docker inception world...

Prepare LinuxKit

apk update
apk upgrade
apk add --no-cache go
export GOPATH=~/.go
go get -v -d -t ./...  

git clone https://github.com/linuxkit/linuxkit
cd linuxkit
make

Snipped output below:

mkdir -p bin
tar cf - vendor src/initrd src/pad4 -C src/cmd/moby . | docker run --rm --net=none --log-driver=none -i  linuxkit/go-compile:4513068d9a7e919e4ec42e2d7ee879ff5b95b7f5@sha256:bdfadbe3e4ec699ca45b67453662321ec270f2d1a1dbdbf09625776d3ebd68c5 --package github.com/linuxkit/linuxkit --ldflags "-X main.GitCommit=22514db912b9bec73e2a3c3ffd1611633c25b9db -X main.Version="0.0" " -o bin/moby > tmp_moby_bin.tar
Unable to find image 'linuxkit/go-compile:4513068d9a7e919e4ec42e2d7ee879ff5b95b7f5@sha256:bdfadbe3e4ec699ca45b67453662321ec270f2d1a1dbdbf09625776d3ebd68c5' locally
sha256:bdfadbe3e4ec699ca45b67453662321ec270f2d1a1dbdbf09625776d3ebd68c5: Pulling from linuxkit/go-compile
627beaf3eaaf: Pulling fs layer
...
49418cbb2668: Pull complete
Digest: sha256:bdfadbe3e4ec699ca45b67453662321ec270f2d1a1dbdbf09625776d3ebd68c5
Status: Downloaded newer image for linuxkit/go-compile@sha256:bdfadbe3e4ec699ca45b67453662321ec270f2d1a1dbdbf09625776d3ebd68c5
gofmt...
govet...
golint...
ineffassign...
go build...
...
gofmt...
govet...
golint...
ineffassign...
go build...
tar xf tmp_linuxkit_bin.tar > bin/linuxkit
rm tmp_linuxkit_bin.tar
touch bin/linuxkit

If all goes well, you'll have two binaries created under bin directory: moby and linuxkit. As per this pr:

• moby: just does the simple build case,
• linuxkit: is useful to push and run the constructed image

jmkhael@jmkhael-lx:~/workspace/linuxkit
$ ls bin/
linuxkit  moby

I've had to patch the Makefile so it works in PWD (It worked without modifications on my local machine directly. as per the previous post)
I've cheated a bit. I've added -v /root/.go:/go to the Docker run command for GO_COMPILE to have the dependencies.

Building the Redis OS sample project

So, let's make our OS:

make
./bin/moby build projects/demo/redis-os/redis-os

output of the command:

Extract kernel image: linuxkit/kernel:4.9.x
Add init containers:
Process init image: linuxkit/init:63eed9ca7a09d2ce4c0c5e7238ac005fa44f564b
Process init image: linuxkit/runc:b0fb122e10dbb7e4e45115177a61a3f8d68c19a9
Process init image: linuxkit/containerd:18eaf72f3f4f9a9f29ca1951f66df701f873060b
Add onboot containers:
Add service containers:
  Create OCI config for linuxkit/dhcpcd:0d4012269cb142972fed8542fbdc3ff5a7b695cd
  Create OCI config for redis:3.0.7-alpine
Add files:
Create outputs:
  redis-os-bzImage redis-os-initrd.img redis-os-cmdline

Testing the Redis OS

For that, just execute linuxkit run with the image. This will leverage qemu (on Linux) to run your OS in a VM! On Mac, all this given by moby/HyperKit. - note that as of the time of writing, Hyperkit only supports Mac.

"HyperKit is a toolkit for embedding hypervisor capabilities in your application. Supported backends are:
gcp
hyperkit [macOS]
qemu [linux]
vmware
packet"

Note that if you want to access the machine from outside, make sure to set the ip for the VM:

bin/linuxkit run redis-os

After a very short wait, you'll see a shell!

Let's abuse netcat to test redis-server from the image itself:

(printf "PING\r\n"; sleep 1) | nc localhost 6379
+PONG
(echo PING; sleep 1) | nc localhost 6379
+PONG

If you want to connect from the host machine,(where you built the os-redis image), you need to pass additional parameters for linuxkit/qemu runner (introduced by this pull request)

linuxkit run qemu -publish 6379:6379/tcp redis-os

You can then run on the host:

(printf "PING\r\n"; sleep 1) | nc localhost 6379
+PONG
(echo PING; sleep 1) | nc localhost 6379
+PONG

Enjoy your redis-os built by LinuxKit in PWD in Docker!

Diagnostics

If you need to diagnose why stuff are not working, netshoot docker image can save your day!

docker run -it --net host nicolaka/netshoot

and in it, run for instance tcpdump:

tcpdump -s 65535 tcp port 6379 -i any

and see the magic happens:

13:34:20.880434 IP localhost.38050 > localhost.6379: Flags [S], seq 3409053076, win 43690, options [mss 65495,sackOK,TS val 239977870 ecr 0,nop,wscale 7], length 0
13:34:20.880447 IP localhost.6379 > localhost.38050: Flags [S.], seq 195769136, ack 3409053077, win 43690, options [mss 65495,sackOK,TS val 239977870 ecr 239977870,nop,wscale 7], length 0
13:34:20.880458 IP localhost.38050 > localhost.6379: Flags [.], ack 1, win 342, options [nop,nop,TS val 239977870 ecr 239977870], length 0
13:34:20.880487 IP localhost.38050 > localhost.6379: Flags [P.], seq 1:6, ack 1, win 342, options [nop,nop,TS val 239977870 ecr 239977870], length 5: RESP "PING"
13:34:20.880493 IP localhost.6379 > localhost.38050: Flags [.], ack 6, win 342, options [nop,nop,TS val 239977870 ecr 239977870], length 0
13:34:20.881104 IP localhost.6379 > localhost.38050: Flags [P.], seq 1:8, ack 6, win 342, options [nop,nop,TS val 239977870 ecr 239977870], length 7: RESP "PONG"
13:34:20.881112 IP localhost.38050 > localhost.6379: Flags [.], ack 8, win 342, options [nop,nop,TS val 239977870 ecr 239977870], length 0
13:34:21.880774 IP localhost.38050 > localhost.6379: Flags [F.], seq 6, ack 8, win 342, options [nop,nop,TS val 239978120 ecr 239977870], length 0
13:34:21.881299 IP localhost.6379 > localhost.38050: Flags [F.], seq 8, ack 7, win 342, options [nop,nop,TS val 239978121 ecr 239978120], length 0
13:34:21.881319 IP localhost.38050 > localhost.6379: Flags [.], ack 9, win 342, options [nop,nop,TS val 239978121 ecr 239978121], length 0

Cleaning up

When ready, and you want to stop this Redis OS, type halt in the VM shell. Notice that it takes more time to shutdown than to boot!

You can also kill linuxkit ;)

You should also remove the PWD running containers...

Next

Now, how can I put SwarmKit to handle a hierarchical Swarms built in PWD built on Docker? My head hurt now... and I have a flight to catch!

Au revoir Austin! Keep Austin weird!

You can also have a go at the interactive tutorial I built here: LinuxKit 101: building the redis-os from DockerCon2017

You can now build your first #LinuxKit OS interactively in the web through PWD https://t.co/T0z3b8MYjU thx to @jmkhael. @moby @docker

— Marcos Nils (@marcosnils) May 14, 2017

Moby and LinuxKit?

So, DockerCon2017 is amazing!

As you should know by now, Docker introduced the Moby project!
If you missed it, go read the announcement here by Solomon Hykes. https://blog.docker.com/2017/04/introducing-the-moby-project/

At it's core, LinuxKit.

LinuxKit, a toolkit for building custom minimal, immutable Linux distributions.

Build Moby

I like to get my hands dirty. So let's build one of the examples supplied to understand more.

Prerequisites

But first, install qemu if you haven't done so yet. You'll probably need build-essentials too.

sudo apt install qemu

Then just clone and build the Moby tools:

git clone https://github.com/linuxkit/linuxkit
cd linuxkit/
make 
sudo make install

If all goes well, you'll have two binaries created under bin directory: moby and linuxkit. As per this pr:

• moby: just does the simple build case,
• linuxkit: is useful to push and run the constructed image

Let's make sure it works:

moby

and see the usage output:

Please specify a command.

USAGE: moby [options] COMMAND

Commands:
  build       Build a Moby image from a YAML file
  version     Print version information
  help        Print this message

Run 'moby COMMAND --help' for more information on the command

Options:
  -q	Quiet execution
  -v	Verbose execution

Build one of the examples

Cool. Let's build the sshd example, which should produce a kernel:4.9, with some services, among them, linuxkit/sshd

cd examples/
moby build sshd.yml

After about a minute or so:

linuxkit run sshd

... after a while...

Starting containerd

Welcome to LinuxKit

                            ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/


  / # INFO[0000] starting containerd boot...                   module=containerd
  INFO[0000] starting debug API...                         debug="/run/containerd/debug.sock" module=containerd
  INFO[0000] loading monitor plugin "cgroups"...           module=containerd
  INFO[0000] loading runtime plugin "linux"...             module=containerd
  INFO[0000] loading snapshot plugin "snapshot-overlay"...  module=containerd
  INFO[0000] loading grpc service plugin "content-grpc"...  module=containerd
  INFO[0000] loading grpc service plugin "runtime-grpc"...  module=containerd
  INFO[0000] loading grpc service plugin "images-grpc"...  module=containerd
  INFO[0000] loading grpc service plugin "rootfs-grpc"...  module=containerd
  INFO[0000] loading grpc service plugin "healthcheck-grpc"...  module=containerd
  INFO[0000] loading grpc service plugin "metrics-grpc"...  module=containerd
  INFO[0000] starting GRPC API server...                   module=containerd
  INFO[0000] containerd successfully booted in 0.107088s   module=containerd
   - 000-sysctl
   - dhcpcd
   - rngd
   - sshd

Once the machine booted up, check it's kernel is 4.9 built by moby and that sshd is running:

/ # uname -a
Linux moby-525400123456 4.9.22-moby #1 SMP Fri Apr 14 12:32:33 UTC 2017 x86_64 Linux

/ # pgrep -l sshd
404 /usr/sbin/sshd

Nice! Cannot get easier than that. (Well, with Docker, it will... you just wait)

Next

Where to go next? Maybe we will build something tailored to a particular stack, or tailored for HPC/GPUs maybe?

As you should know if you scroll to the bottom of my pages, my blog is

"Proudly published with Docker, Ghost and ABC Theme on my Raspberry Pi"

And as you should also know by now, I am pretty much lazy.

If I won the award for laziness, I would send somebody to pick it up for me.

Problem

My Raspberry Pi is hanging on the floor, the blog being served from an SD card (ouch!) and the whole thing is within reach of my iRobot Roomba Vacuum, which managed to bring down my blog several times already :)

This seems like a recipe for failure!

But, instead of mounting an external disk and serving my blog from it, or worse moving my Raspberry Pi away from the Roomba, in this blog post we will setup our Continuous deployment of the blog and favor high availability and resilience use cases (and laziness)!

What we will do

To do that, we will:

1. Build a docker image for the blog without it's data
2. Prepare the data and load it as a Docker volume
3. Push the volume over to an S3 bucket, a Dropbox folder or a google drive one
4. Setup dyndns client for Namecheap (this is where I bought my domain name)

Once we have these three elements, upgrading, deploying, or replicating our blog is a matter of a simple Jenkins job or a shell script.

Ok. Let's go!

Build a Docker image for the blog (without the data)

Note that for the sake of using my rather old Raspberry Pi to serve my blog, I am basing my Docker image on an already package image done by @alexellisuk for an ARM V6. If you have a different architecture for your Pi, (or x86), you can find other images. Check out his Git repo here: https://github.com/alexellis/ghost-on-docker

I am customizing a bit to account for my domain name, and replacing bcryptjs by bcrypt, as the later is much more quicker. (Especially on Rpi)

FROM alexellis2/ghost-armv6:0.11.2

RUN sed -i s/my-ghost-blog.com/jmkhael.io/g config.js
RUN npm install bcrypt
RUN sed -i "s/require('bcryptjs')/require('bcrypt')/" core/server/models/user.js

You can also just clone and adapt my repo to suit your needs:

git clone https://github.com/jmkhael/blog
pushd blog/jmkhael.io

Time to build the image:

docker build -t blog-0-11-2 .

This should have prepared a docker image, which we will be using below.

Prepare the data and load it as a Docker volume

Let's prepare a folder where the data will be stored on disk:

mkdir -p /home/pi/google-drive/ghost-content/

It is time to run our Ghost blog in a Docker container:

docker run --name blog-0-11-2 \
   -d -p 80:2368 \
   -v /home/pi/google-drive/ghost-content/:/var/www/ghost/content/ \
   blog-0-11-2

Of course, you know by now that I prefer to launch everything as a Service in a Docker Swarm ;)

Wait a (rather big) moment for it to startup, create the schemas and all that.
You can now login to the blog at http://localhost.

(substitute localhost with the Raspberry Pi or the machine name/or ip where you started the container)

Push the volume over to google drive

We will be using drive tool from https://github.com/odeke-em/drive.

installation on a Raspberry Pi

Make sure to chose from the releases page https://github.com/odeke-em/drive/releases, the latest version which works for your architecture:

Given that I am running on an ARM v6 (to get yours, execute a uname -a):

cd /tmp
wget wget https://github.com/odeke-em/drive/releases/download/v0.3.8/drive_armv6
chmod +x drive-armv6
sudo cp drive-armv6 /usr/local/bin/gdrive
cd ~
gdrive version

Init gdrive

Navigate to the folder in which you want to store google drive root, and call gdrive init:

cd /home/pi/google-drive/ghost-content/
gdrive init

This will ask you to visit a url to grant the app an OAuth token. Follow that; then paste the token on the command line and press Enter.

Afterwards you can push the folder using:

gdrive push -quiet /home/pi/google-drive/ghost-content/

You can also pull it when you need to clone it or replicate on another machine:

gdrive pull -quiet /home/pi/google-drive/ghost-content/

That great. But not enough! We can go further.
Let's write a small shell script to backup the drive, and put it in a cron:

/home/pi/blog/backup.sh

#!/bin/bash
echo Starting backup...
/usr/local/bin/gdrive \
    push -no-prompt -exclude-ops \
    delete /home/pi/google-drive/ghost-content
echo Done.

Now give execution rights and add it to a cron:

chmod a+x /home/pi/blog/backup.sh

crontab -e

Then add the below line to the end:

30 * * * * bash /home/pi/blog/backup.sh > /tmp/backup.txt

To do the same into an S3 bucket, or a Dropbox folder you can follow my other guides.

Setup dyndns client for namecheap

For the below section, you will need to buy a Namecheap domain (for as little as few dollars). Once done, you need to login and retrieve dyndns password to be used below.

Check the below screenshots to see where to do that:

We will also be using ddclient to update the DNS entry for the domain.

sudo nano /etc/ddclient.conf

The login is your domain name and the password is retrieved from the second picture from the field Dynamic DNS Password

# Configuration file for ddclient generated by debconf
#
# /etc/ddclient.conf
use=web, web=dynamicdns.park-your-domain.com/getip
daemon=3600
protocol=namecheap
server=dynamicdns.park-your-domain.com
login=jmkhael.io
password='<my namecheap password goes here>'
@, www, api, blog

Force a ddclient update then make sure to restart the service:

sudo ddclient --force
sudo service ddclient restart

Put it in a crontab, so we update namecheap each and every while:

sudo crontab -e

then add the below line to the end and save:

45 04 * * * /usr/sbin/ddclient --force

wait a while, and navigate to your domain name and see your blog!

Some useful links for dealing with ddclient and Namecheap

http://workshop.botter.ventures/2014/02/02/how-to-use-ddclient-with-namecheap/
https://help.ubuntu.com/community/DynamicDNS

Setup reverse proxy and SSL (Optional)

• Check the following blog posts to setup reverse proxy as HAProxy or Traefik for your Swarm.

What is coming next

In next blog posts, I might document how to:

• Setup ssl on reverse proxy
• Healthcheck/monitor the blog externally (uptime monitoring or something)

On of the new additions of Docker 1.13 is the ability to use compose files to deploy swarm mode services!

This means that we can use docker-compose files to provision Swarm services.

That also means that we do not need anymore to type all the long docker service create commands.
A long list of docker service create commands would be simply replaced by a command like the below:

docker stack deploy --compose-file=docker-compose.yml my_stack

In this blog post, we will see that in action and discuss what is covered and what still lacks.

Life before Docker stack deploy

Or precisely, before docker stack deploy --compose-file <Path to a Compose file> we had to type a lot and maintain a shell script to start our services accross the Swarm.

What made things worse, is that before the addition of this command, as developers, we loved the docker-compose command to deploy a stack on our machines. But as you know, docker-compose couldn't benefit from all the nodes of the swarm cluster. It gave us a warning like the below:

docker-compose up -d

WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use the bundle feature of the Docker experimental build.

More info:
https://docs.docker.com/compose/bundles

So even if you had a compose-file for local testing, you would have to prepare a set of commands to create the service on the Swarm and align those! Talk about technical debt...

Let's take a simple example where we want to deploy two service to monitor the swarm.

These two service I tend to always create first thing on my Swarms are the Visualizer and Portainer. Here is how I used to do so:

docker service create \
  --publish 9000:9000 \
  --name portainer \
  --constraint=node.role==manager \
  --mount=type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
  portainer/portainer 

docker service create \
  --publish=9090:8080 \
  --limit-cpu 0.5 \
  --name=viz \
  --constraint=node.role==manager \
  --mount=type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
  dockersamples/visualizer

Life after Docker stack deploy

Let's now deploy our monitoring stack again, but this time with the new shiny way:

curl -o docker-compose-stack-monitoring.yml \
https://raw.githubusercontent.com/jmkhael/compose-samples/master/docker-compose-stack-monitoring.yml

docker stack deploy -c docker-compose-stack-monitoring.yml monitoring

The first command downloaded the Compose file docker-compose-stack-monitoring.yml from the jmkhael/compose-samples repository.
The second command creates the services which form what I called the monitoring stack.

Note from the below output that it does incrementally handle new service:

Creating service monitoring_viz
Updating service monitoring_portainer (id: jbt006a7yoi8t0ok941b358k2)

If you check the created services, you'll see portainer and visualizer:

docker service ls
ID            NAME                  MODE        REPLICAS  IMAGE
jbt006a7yoi8  monitoring_portainer  replicated  1/1       portainer/portainer:latest
wqfb3odzmnfh  monitoring_viz        replicated  1/1       dockersamples/visualizer:latest

that's neat!

The docker-compose-stack-monitoring.yml file look like this:

version: '3'

services:

  portainer:
    image: portainer/portainer
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - 9000:9000
    deploy:
      placement:
        constraints: [node.role == manager]

  viz:
    image: dockersamples/visualizer
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      placement:
        constraints: [node.role == manager]
    ports:
      - 9090:8080

Note that the docker-compose file uses the version 3, introduced with Docker 1.13. (Check the schema here)

You can use some nice configurations under the deploy section such as deployment mode (replicated or global), replicas count (replicas: 3) or labels...

deploy:
  mode: replicated
  replicas: 2
  labels: [APP=CONQUER_UNIVERSE]

Is everything covered?

TL;DR:

The short answer is of course, No.

The longer answer is around these lines:

Networks

No support for attachable networks yet in the docker-compose v3.

Extends not supported

You cannot use the extends keyword. But there is a quick workaround for that. You'd need to resolve the docker-compose keyword by issuing:

docker-compose config > docker-compose-resolved.yml 

Unsupported docker-compose v2 keywords

The below keywords are still not supported.

• "build",
• "cap_add",
• "cap_drop",
• "cgroup_parent",
• "devices",
• "dns",
• "dns_search",
• "domainname",
• "external_links",
• "ipc",
• "links",
• "mac_address",
• "network_mode",
• "privileged",
• "read_only",
• "restart",
• "security_opt",
• "shm_size",
• "stop_signal",
• "tmpfs"

That's it for this post. Now go deploy some stacks!

As you start setting up swarms, you quickly get tired of doing the same commands all over the place, on each node that is.

In this blog post, we are going to document a quick (and dirty) way to start using ansible and that to run some shell commands on the whole swarm, be it to set it up or do some housekeeping.

Ansible. Deploy apps. Manage systems. Crush complexity.
Ansible helps you build a strong foundation for DevOps.

Let's go.

Preparation

first off, let's install ansible and sshpass:

This is as easy as doing:

apt-get install ansible -y
apt-get install sshpass -y

Ansible needs to be able to access the machines, and we don't want to type each time a password. So we basically need to configure ssh password based or passwor-dless based access to the machines.

This post will show both:

Configure password based logins

Make sure to backup the ansible configuration before we mess everything:
sudo cp /etc/ansible/hosts /etc/ansible/hosts.orig

Let's edit it, sudo vi /etc/ansible/hosts:

[all:vars]
ansible_port=22
ansible_connection=ssh
ansible_ssh_user=uburoot
ansible_ssh_pass=uburoot

[workers]
mx7264vm
mx7265vm

Time to run some ansible commands!

Let's try:

ansible all -m shell -a "docker ps"

mx7264vm | SUCCESS | rc=0 >>
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

mx7265vm | SUCCESS | rc=0 >>
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Ok, nothing is running on the cluster.

To see how the Docker daemons are getting started:

ansible all -m shell -a "pgrep -a docker"

To claim some disk space back:

ansible all -m shell -a "docker system prune --all --force"

I hope you get the idea :)

Configure password-less based logins

Let's first generate an ssh key on the first machine:

ssh-keygen -t rsa

Always from the first machine, login to the second machine and create the .ssh folder:

ssh uburoot@mx7265vm mkdir -p .ssh

Now, we should add the public key to the authorized_keys on the target machine:

cat /root/.ssh/id_rsa.pub | ssh uburoot@mx7265vm 'cat >> .ssh/authorized_keys'

If all went according to plan, you should be able to ssh into the second machine without entering a password! Sweet.

ssh uburoot@mx7264vm

Welcome to Ubuntu 16.04 LTS (GNU/Linux 4.4.0-53-generic x86_64)
root@mx7264vm:~#

Do that for all your machines, or use an ansible password based script to set this up and then switch to a password-less login.

In a later post we will write a playbook most probably to configure the Docker daemon flags (connect to an internal registry or to connect to a host socket...)

Stay tuned!

If you don't know what JHipster is, stop now and go check it out!

Now that you are back, in this blog post, we will build a Netflix Stack using Jhipster on the Raspberry Pi and we will deploy it on a Raspberry Pi Docker Swarm!

But for that to happen, we need to build a Docker Raspberry Pi image for JHipster (and make it more hip if that is possible)

As a plus, we will do a demo, which would go like that:

1. pull jmkhael/rpi-jhipster (to be created below)
2. do some jhipster magic to create a Netflix microservice stack
3. build and deploy the Docker stack on the Raspberry Pi swarm!

Let's Go.

Dockerize the Raspberry Pi

As a pre-requisites, you'd need to have Docker up and running on the Raspberry Pi.

Docker installations on Rasperry Pi

• install docker: curl -sSL https://get.docker.com | sh
• install docker-compose for Arm by following the below commands:

bash:

sudo apt-get update
sudo apt-get install -y apt-transport-https
echo "deb https://packagecloud.io/Hypriot/Schatzkiste/debian/ jessie main" | sudo tee /etc/apt/sources.list.d/hypriot.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 37BBEE3F7AD95B3F

sudo apt-get update
sudo apt-get install docker-compose

Confirm that everything is fine by running:

docker info
docker-compose --version

Now is the time to build the JHipster Arm image.

Jhipster on Raspberry Pi

We need to adapt the project's Dockerfile so it generates an image which we could run on the Raspberry Pi.

If you try to run the jhipster/jhipster docker image, it would fail with the dreaded panic exec format. or equivalent, depending on your luck :)

Create Dockerfile.rpi

I won't repeat here the Dockerfile, you can see it on my Github.

https://github.com/jmkhael/generator-jhipster/blob/master/Dockerfile.rpi

(I still need to do a pull request on the original project to have this natively.)

Build the image:

docker build -t jmkhael/rpi-jhipster -f Dockerfile.rpi .

Run JHipster the Docker way!

In order to be able to play with (or git commit) the generated files by JHipster, we will create some directories and mount them as volumes in the container.

Create working directory:

mkdir -p jhipster/app
mkdir -p jhipster/app/.m2
mkdir -p jhipster/app/.gradle

Give ownership of this directory to jhipster user (id 1000)

chown -R 1000 jhipster/app

In order to be able to execute Docker commands from within the container, we will need to pass the docker group as the group-add flag, so JHipster container can launch docker commands. (discussed here)

Now we can run Jhipster's Docker container on the raspberry Pi and allow it to execute Docker commands on the host!

docker run --name jhipster \
    --group-add $(cat /etc/group | grep docker | awk -F":" '{print $3}') \
    -v $(which docker):/usr/bin/docker \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $(pwd)/jhipster/app/.m2:/home/jhipster/.m2 \
    -v $(pwd)/jhipster/app/.gradle:/home/jhipster/.gradle \
    -v $(pwd)/jhipster/app:/home/jhipster/app \
    -p 8080:8080 -p 9000:9000 -p 3001:3001 -d -t jmkhael/rpi-jhipster

At this point, Docker should give you the container id in response. If you see anything else, feel free to share in the comments.

Make sure everything works well by executing a shell in the image, and maybe executing a yo jhipster and a docker info commands in there if you feel adventurous:

docker exec -it jhipster bash

jhipster@9df16edc8e52:~/app$ 

Note that you can stop and start this container if needed using:

docker stop jhipster
docker start jhipster

Sweet! Time to build our application.

Build our application stack

Let's prepare a directory and call JHipster to generate it:

mkdir blog
cd blog
yo jhipster
yo jhipster:docker-compose

Answer this in order to have a monolithic application, Angular 1.x, named blog, MySQL, No caching, Gradle build, Social Login, Websockets, English and French support, Gatling and Cucumber!

./gradlew bootRepackage -Pprod buildDocker

If you need to force the regeneration of the application:

rm -rf src/main
yo jhipster --force --with-entities

(For fun, try running:

bower install bootstrap-material-design#0.3.0 --save

then rebuild the application.)

Once ready, go ahead and deploy the application using the generated docker-compose file:

docker-compose up -d

WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All    containers will be scheduled on the current node.

To deploy your application across the swarm, use the bundle feature of the Docker experimental build.

More info:
https://docs.docker.com/compose/bundles

Recreating rpiblog_blog-app_1
blog-mysql is up-to-date

Navigate to http://pi1:8080 and see your application:

You can find the resulting application on https://github.com/jmkhael/rpi-blog.

Changes to the (original) Dockerfile

Below are the changes done on the Dockerfile in order to make it compatible with the Arm processor:

1. Modify the base image to rpi-node
2. I needed to install oracle jdk8 instead of openjdk, as it suffers from the below stack.
3. Skip node, npm installation
4. Skip changing ownership of /usr/lib/node_modules

Stack:

Exception in thread "main" javax.net.ssl.SSLHandshakeException: 
   sun.security.validator.ValidatorException: PKIX path validation failed: 
   java.security.cert.CertPathValidatorException: signature check failed

This is also discussed here

TODO

I still have todo the below when time permits.

rpi-Jhipster

[] pull request for rpi image

[] override Docker image names for rpi in ./node_modules/generator-jhipster/generators/generator-constants.js:const DOCKER_JAVA_JRE = 'openjdk:8-jre-alpine';

Why use an aggregation pipeline?

In this post we will give an example of an aggregation pipeline step by step. It would be as a light tutorial to touch on the subject. We will start with the given context and expose the problem afterwards, then write an aggregation pipeline and dissect it!

The aggregate pipeline is a framework for data aggregation modeled on the concept of data processing pipelines. Documents enter a multi-stage pipeline that transforms the documents into aggregated results.
https://docs.mongodb.com/manual/core/aggregation-pipeline/

READY? Let's go.

Given

Given a bunch of documents in a mongoDB collection like the below.
Let's imagine for a moment that those are events generated by a game, concerning win or loss (only, for simplification). Each kind of event has some specific data to it. So we would have a bunch of documents like so:

{
  "playerId": "c5f348a003840f8",
  "level": 0,
  "event": {
	"score": 54,
	"damagedBy": "ASTEROID",
	"levelKey": {
	  "id": 0
	},
	"type": "lost"
  }
}

{
  "playerId": "5f348a0038cb8d",
  "level": 2,
  "event": {
	"flawless": true,
	"collector": false,
	"levelKey": {
	  "id": 2
	},
	"score": 315,
	"retries": 3,
	"type": "won"
  }
}
 ... imagine many events of those, let's say from a game  ...

The problem

Lets say we need to construct for a given player (identified by it's id), an aggregate for each level. This aggregate would tell how many times the level was played, how many times it was won, if it was flawless, if it has the collector property and the score (as coins).

To give you a better idea, the document should look like so:

{
  "played": 1,
  "won": 0,
  "flawless": false,
  "collector": false,
  "coins": 38,
  "level": 5
}

• played = count of events
• won =count of event with type=won
• coins = sum(score),
• flawless and collector are true when at least one event for the level has them true
• level = the level id.

Solution

The below aggregation pipeline does that. Have a look at it and then we will break it down bit by bit.

[{
	"$match": {
		"playerId": "c5f348a003840f8"
	}
}, {
	"$group": {
		"_id": "$level",
		"played": {
			"$sum": 1
		},
		"won": {
			"$sum": {
				"$cond": [{
					"$eq": [
						"$event.type",
						"won"
					]
				},1,0]
			}
		},
		"flawless": {
			"$sum": {
				"$cond": [{
					"$eq": [
						"$event.flawless",
						true
					]
				},1,0]
			}
		},
        "collector": {
			"$sum": {
				"$cond": [{
					"$eq": [
						"$event.collector",
						true
					]
				},1,0]
			}
		},
		"coins": {
			"$sum": "$event.score"
		}
	}
}, {
	"$project": {
	    "level": '$_id',
		"coins": 1,
		"played": 1,
		"won": 1,
		"flawless":  { $gt: ['$flawless', 0] } , 
        "collector": { $gt: ['$collector', 0] } 
	}
	}]

Let's break it down

A bit of definitions first

The aggregate pipeline is

a framework for data aggregation modeled on the concept of data processing pipelines. Documents enter a multi-stage pipeline that transforms the documents into aggregated results.
https://docs.mongodb.com/manual/core/aggregation-pipeline/

In layman's terms, a pipeline consists of several stages. Each stage transforms the data as it passes through it, and pass it along the next stage. Easy, right?

Ok, so next logical question you ask is what kind of stages are there?
I am glad you asked! At the time of writing there is around 2Oish operator. (too many to list or explain - check here if you want to see them all)

For the concern of this blog post, we will use 3 stages to perform our data transformation, namely the match, group and project.

$match stage

Filters the document stream to allow only matching documents to pass unmodified into the next pipeline stage. $match uses standard MongoDB queries. For each input document, outputs either one document (a match) or zero documents (no match).

Remember that in the context of our problem we are interested in filtering the events for a given player. We do not want to mix results from different players. That wouldn't make any sense for the problem we want to solve!

To match the documents for a given player, let's say id=c5f348a003840f8, our $match stage is as easy as this:

{ "$match": { "playerId": "c5f348a003840f8" } }

Great. Now we need to start munching some data, grouping and transforming it. Shall we?

$group stage

Groups input documents by a specified identifier expression and applies the accumulator expression(s), if specified, to each group. Consumes all input documents and outputs one document per each distinct group. The output documents only contain the identifier field and, if specified, accumulated fields.

If all you've understood resembles, blah blah blah blah, don't be discouraged. Let me try to explain with our example.

Let's start small. (Most things start like that.)
If we want to see how many times the player played a given level, we would have to group by the level id, and count the number of documents related to that level. (remember we only generate won and lost event in our "complex" game)

{ "$group": {
      "_id": "$level",
      "played": { "$sum": 1 }
  }
}

• _id: this is the key you pass to SQL's GROUP BY, or RxJava groupBy key. Set this to the things you want to group on
• played: here we use an accumulator, the sum. Practically we would add 1 each time we see a document. Thus counting them.

Sweet! Let's make it a bit harder.

In order to see how many time the player won his level, we'd have to count (accumulate the sum) the documents having the event.type == "won"

"won": {
	"$sum": {
		"$cond": [{
			"$eq": [
				"$event.type",
				"won"
			]
		}, 1, 0]
	}
}

A bit harder, but not by much! We basically used an equality conditional ($cond and $eq), which operates like a ternary operator, and returns 1 if true or 0 otherwise. and then we sum those. Basically adding 1 each time we see a won.

After adding almost the same for the other fields, our documents look like so:

{
   "_id": 7,
   "played": 4,
   "won": 2,
   "flawless": 2,
   "collector": 2,
   "coins": 1044
}

Pretty close, but not there yet.

inner voice: I wish there were a way to reshape somehow the document.

$project stage

Reshapes each document in the stream, such as by adding new fields or removing existing fields. For each input document, outputs one document.

Great! Santa DO EXIST! :'(

Finally, we will use a projection in order to filter out the _id field, and rename or reshape the others:

 "$project": {
       "level": '$_id',
       "_id": 0,
       "coins": 1,
       "played": 1,
       "won": 1,
       "flawless":  { $gt: ['$flawless', 0] } ,
       "collector": { $gt: ['$collector', 0] }
 }

Here is what we are saying:

• level: we set it to be equal
• _id: we restrict it from showing into the final document by specifying it as 0
• coins played and won are passed as they are - so we set them to 1
• collector and flawless: we convert it to boolean by checking if we had at least 1 as the sum

and that's it! It was easy :)

I'm pretty sure we could have written that in another way. Maybe you want to tell me about it in the comments?

In this post we will setup Traefik as a Dynamic reverse proxy for a Docker Swarm. We will be able to run the same on a Raspberry Pi or on x86.

If you want to see how to do the same with HAProxy, check this post!

Why a Dynamic reverse proxy and load balancer?

Classic loadblancers and proxies (nginx, haproxy, apache) were built with a static backend in mind and are usually enhanced with other projects such as Victor Farcic' HA docker-flow-proxy or jwilder/nginx. Traefik is built for a dynamic world from the start.

Why Traefik?

Træfɪk is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease. It supports several backends (Docker, Swarm, Kubernetes, Marathon, Mesos, Consul, Etcd, Zookeeper, BoltDB, Eureka, Rest API, file...) to manage its configuration automatically and dynamically.

To be honest, I wanted to try it as I really like the logo :)
(of course in order to learn a new reverse proxy too)

Ok. Let's go now.

Deploy Traefik on the Swarm

Create an overlay network

docker network create --driver=overlay traefik-net

Deploy Traefik

docker service create \
--name traefik \
--constraint 'node.role==manager' \
--publish 80:80 \
--publish 8080:8080 \
--mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
--network traefik-net \
traefik:camembert \
--docker \
--docker.swarmmode \
--docker.domain=jmkhael.io \
--docker.watch \
--logLevel=DEBUG \
--web

We will next deploy some services to mimic the below screenshot. We will configure several frontends serving different hosts while themselves are being served by different backends.

Deploy api service

docker service create \
--name api \
--label 'traefik.port=5000' \
--network traefik-net \
jmkhael/myservice:0.0.1

Deploy backoffice service

docker service create \
--name backoffice \
--label 'traefik.port=5000' \
--network traefik-net \
jmkhael/myservice:0.0.2

Scale the backoffice service to 3 instances to match the screenshot:

docker service scale backoffice=3

Deploy web service

docker service create \
--name web \
--label 'traefik.port=5000' \
--label traefik.frontend.rule="Host:jmkhael.io; Path: /web/" \
--network traefik-net \
jmkhael/myservice:0.0.2

Test this

Issue the commands:

curl -H Host:api.jmkhael.io http://0.0.0.0
curl -H Host:backoffice.jmkhael.io http://0.0.0.0
curl -H Host:jmkhael.io http://0.0.0.0/web/

and we should get valid output, like the below:

I'm 24ab5c2a9402
I'm eec66086b687
I'm aaf5402a2387

Checking the Visualizer we can see all the services running:

Checking http://stratus-clay:8080/ we can reach the Web dashboard of Traefik and see our frontends and backends well defined:

All this happening like...

Running the same thing on the Raspberry Pi

Deploy Traefik on Rpi

docker service create \
--name traefik \
--constraint 'node.role==manager' \
--publish 81:80 \
--publish 8081:8080 \
--mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
--network traefik-net \
hypriot/rpi-traefik \
--docker \
--docker.swarmmode \
--docker.domain=jmkhael.io \
--docker.watch \
--logLevel=DEBUG \
--web

Deploy two API Endpoints Service

One to respond on api.jmkhael.io

docker service create \
--name api \
--label 'traefik.port=8000' \
--network traefik-net \
hypriot/rpi-whoami

Another to respond on api.jmkhael.io/products/

docker service create \
--name products \
--label 'traefik.port=8000' \
--label traefik.frontend.rule="Host:api.jmkhael.io; Path: /products/" \
--network traefik-net \
hypriot/rpi-whoami

Deploy Blog service on blog.jmkhael.io

Constraint it to node blog as it has the data there... until we handle the statefulness better (nfs?)

docker service create \
--name blog \
--label 'traefik.port=2368' \
--label traefik.frontend.rule="Host:api.jmkhael.io, blog.jmkhael.io;" 
--constraint 'node.hostname==blog' \
--network traefik-net \
alexellis2/ghost-on-docker:armv6

Notice that we've written frontend rules for api and blog, as Ghost needs both to be redirected to it...
(that means that we cannot host api endpoints, at least not without thinking of a potential clash, as our /products/ above still works)

Test api

curl -H Host:api.jmkhael.io http://0.0.0.0:81
curl -H Host:api.jmkhael.io http://0.0.0.0:81/products/

If you check http://blog.jmkhael.io:81, you'll see the blog running :)
Sweet!

Let's also make sure that our products api endpoint is working too:

curl http://api.jmkhael.io:81/products/
I'm eec66086b687

Scale app

docker service scale api=3

curl -H Host:api.jmkhael.io http://0.0.0.0
curl -H Host:api.jmkhael.io http://0.0.0.0
curl -H Host:api.jmkhael.io http://0.0.0.0

This might seem to be "not working" (as expected), because the sessions are kept alive for a while - this issue will "correct" it.