Unleash the Artist within you with Docker, Tensorflow and OpenFaaS

“What I cannot create, I do not understand.” — Richard Feynman

Context

Since my early age, I always felt incapable of drawing. My talent was miserable at best, leading me to think that drawing is more of an innate talent than a craft I can learn (in short time and without dedication of course!).

In this post, the second of the "Machine Learning on Serverless" blog post series, we will unleash the artist within, and prove that we can use code to generate some compelling art!

(In the first post of the serie we developed a serverless function to identify Doppelgängers.
You can read about it here: Kittens vs Tarsiers - an introduction to serverless machine learning)

We will be using Google Brain project Magenta, OpenFaaS and Docker to build a Serverless function which will enable us to do multi-style transfer.

Show me what it does

Let's start with a demo of what this is capable of before running through the explanation of what this is and how it works.

The image on the left is the original one.
The one on the right was stylized with one of the 40+ styles of the function we are going to discuss below.

Gordon (Evil mastermind behind Docker)

Original picture of Gordon eating a strawberry in real life

9 different monet styles are available

(open in new tab to see it bigger or just check the album for more samples https://photos.app.goo.gl/MCNnnrmlCSBwR9xH3)

31 varied styles are also available

What is OpenFaaS?

OpenFaaS (Functions as a Service) is a framework for building serverless functions with Docker and Kubernetes which has first class support for metrics. Any process can be packaged as a function enabling you to consume a range of web events without repetitive boiler-plate coding.

Based on it's active community adoption and it's 8k+ stars, it is the best developer experience to build serverless function for any langage you fancy!

What is Magenta?

Magenta is a Google Brain project to ask and answer the questions, “Can we use machine learning to create compelling art and music? If so, how? If not, why not?”
Check it out here: https://magenta.tensorflow.org/

Magenta already has published models to generate drums tracks, image stylization, neural audio synthesis, polyphonic music generation with reinforcement learning and sketches generation!

Sketch interpolation between a cat and a bus!

In this blog post, we will focus on multi-style transfer, future blog post may focus on music or sketches generation.

What is style transfer and multi-style image transfer?

As the proverb goes, an image is worth a thousand words. Here are a couple to show you the power of style transfer and multi-style transfer.

This first one, is a high level view of pastiche or style transfer.

You take a content, you take a style, you transfer the style on the content, you get a pastiche.

The second one, is a high level view of style transfer.

On the X axis lies different styles by William Glackens, Paul Signac, Georges Rouault, Edvard Munch and Vincent van Gogh.
On the Y axis we have known photographs, a.k.a. the content to be styled.

The crossing between the X and Y axis is the manifestation of what style transfer can do.

What about Multi-style?

if one could combine many different styles, exploring unique mixtures of well known artists to create an entirely unique pastiche?

That's what multi-style transfer is all about!

Writing the Artist serverless function

All the code lives in my Github repository here: https://github.com/jmkhael/faas-artist, but stick around as I explain the code below.

Implementation

Before starting to write our function, let's install faas-cli which is a very nice CLI to list, create new, build, deploy, invoke, push and remove functions to list a few - the CLI verbs are growing every day!

It is the Official CLI for the OpenFaaS Serverless framework - GitHub repo

Pre-reqs

Get the faas-cli, if you haven't yet. You really have to if you want to be productive!
The easiest way to install the faas-cli is through curl or brew:

$ curl -sSL https://cli.openfaas.com | sudo sh

or:

$ brew install faas-cli

Start faas-cli and be amazed by it's ascii art and help message:

Scaffold

Time to create our function. Let's use the faas-cli to do so.

faas-cli new artist --lang python

Build and deploy

faas-cli build -f artist.yml
faas-cli deploy -f artist.yml

Test the scaffolded function

echo "Hello, Magenta and OpenFaaS" | faas-cli invoke artist

Implementation

Now we will fill in the function to do the actual work. You can find the full code on the github repository.
We highlight the important aspects below.

As shown above, magenta delivered two models:

monet: offers 9 styles
varied: offers 31 styles

In order to chose a given style and style index, we will pass some information as a headers to the function so we take the appropriate decision in there.
We can also apply a multi-style blending several styles of a given model together, based on weights. e.g.: {1:0.3, 10:0.5, 24:0.2} to tell the artist to use style 1, style 10 and style 24 in the proportions 30% - 50% - 20%.

In the function, we need to get these headers back.

OpenFaaS comprehensively documents this: Reference documentation for Function Watchdog

Headers and other request information are injected into environmental variables in the following format: The X-Style-Name header becomes available as Http_X_Style_Name.

style_name = os.environ.get('Http_X_Style_Name', 'varied')
style_index = os.environ.get('Http_X_Style_Index', '1')
which_styles = os.environ.get('Http_X_Which_Styles', "{%s:1}" % style_index)

Then it is a matter of calling the correct model with the correct parameters in the function. Most of the code comes from Magenta's repository, so make sure to check that out.

if style_name == 'monet':
    FLAGS.num_styles = 10
    FLAGS.checkpoint = "/magenta-models/multistyle-pastiche-generator-monet.ckpt"
    FLAGS.input_image = input_image_name
    FLAGS.which_styles = which_styles
    FLAGS.output_dir = "out_content.jpg"
    FLAGS.output_basename = "monet_styles"
  elif style_name == 'varied':
    FLAGS.num_styles = 32
    FLAGS.checkpoint = "/magenta-models/multistyle-pastiche-generator-varied.ckpt"
    FLAGS.input_image = input_image_name
    FLAGS.which_styles = which_styles
    FLAGS.output_dir = "out_content.jpg"
    FLAGS.output_basename = "varied_styles"
  else:
    raise ValueError('Style %s is not supported. Accepted values are "monet" or "varied"' % style_name)

Style transfer example

If you want to try my repository locally:

git clone https://github.com/jmkhael/faas-artist
cd faas-artist
faas-cli build -f artist.yml
faas-cli deploy -f artist.yml

curl -X POST -H X-style-name:varied -H X-style-index:24 \
  --data-binary @artist/input/vespa-faas.jpg \
  "http://localhost:8080/function/artist" > styled/vespa-faas-varied-24.jpg

or use the helper script:

usage:

./pastiche.sh path_to_image_file style style_index

style can be monet or varied. style_index from 0 to 9 for monet, and from 0 to 30 for varied.

try them all, and see what you like!

./pastiche.sh artist/input/tree.jpg varied 24
./pastiche.sh artist/input/tree.jpg monet 2

Multi-style transfer example

We can also apply a multi-style, blending several styles of a given model together, based on weights. e.g.: {1:0.3, 10:0.5, 24:0.2} to tell the artist to use style 1, style 10 and style 24 in the proportions 30% - 50% - 20%.

curl -X POST -H X-style-name:varied -H X-which-styles:'{1:0.2,10:0.3,24:0.5}' \
  --data-binary @artist/input/faas-community.jpg \
  "http://localhost:8080/function/artist" > styled/faas-community-varied-1-10-24.jpg

Original

style-transfer (varied, 1)

multi-style-transfer (varied, {1:0.2,10:0.3,24:0.5})

multi-style-transfer (varied, {1:0.2,10:0.7,24:0.1})

You can visit the repository for more instructions of how to run multi-style transfer.

What to do next?

You can experiment with all this, and drop me a message if you create something nice!

If you like what we've built in this post, star my repository to show support: https://github.com/jmkhael/faas-artist

Go star the OpenFaaS project - https://github.com/openfaas/faas and be part of it's vibrant community!

If you'd like to join OpenFaaS community Slack channel to chat with me or other contributors or get some help - then send a Tweet to @alexellisuk or email alex@openfaas.com.

Join and add your photo in there!

Also, make sure to visit Magenta's website and repository.

References

I've used the below references to be able to generate most of this blog or code.
Go visit each if you need more detail.