Making application configuration files dynamic with confd and Azure Redis

Service discovery and hot reconfiguration is a common problem we face in cloud development nowadays. In some cases we can rely on an orchestration engine like Kubernetes to do all the work for us. In other cases we can leverage a configuration management system and do the orchestration ourselves. However, there are still some cases where either of these solutions are impractical or just too complex for the immediate problem… and you don’t have a Consul cluster at hand either :(.

confd to the rescue

Confd is a Golang written binary that allows us to make configuration files dynamic by providing a templating engine driven by backend data stores like etcd, Consul, DynamoDb, Redis, Vault, Zookeeper. It is commonly used to allow classic load balancers like Nginx and HAProxy to automatically reconfigure themselves when new healthy upstream services come online under different IP addresses.

NOTE: For the sake of simplicity I will use a very simple example to demonstrate how to use confd to remotely reconfigure an Nginx route by listening to changes performed against an Azure Redis Cache backend. However, this idea can be extrapolated to solve service discovery problems whereby application instances continuously report their health and location to a Service Registry (in our case Azure Redis) that is monitored by the Load Balancer service in order to reconfigure itself if necessary.

https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture

Just as a side note, confd was created by Kelsey Hightower (now Staff Developer Advocate, Google Cloud Platform) in the early Docker and CoreOS days. If you haven’t heard of Kelsey I totally recommend you YouTube around for him to watch any of his talks.

Prerequisites

Azure Redis Cache

Redis, our Service Discovery data store will be listening on XXXX-XXXX-XXXX.redis.cache.windows.net:6380 (whereXXXX-XXXX-XXXX is your DNS prefix). confd will monitor changes on the /myapp/suggestions/drink cache key and then update Nginx configuration accordingly.

Container images

confd + nginx container image
confd’s support for Redis backend using a password is still not available under the stable or alpha release as of August 2017. I explain how to easily compile the binary and include it in an Nginx container in a previous post.

TLDR: docker pull xynova/nginx-confd

socat container image
confd is currently unable to connect to Redis through TLS (required by Azure Redis Cache). To overcome this limitation we will use a protocol translation tool called socat which I also talk about in a previous post.

TLDR: docker pull xynova/socat

Preparing confd templates

Driving Nginx configuration with Azure Redis

We first start a xynova/nginx-confd container and mount our prepared confd configurations as a volume under the /etc/confd path. We are also binding port 80 to 8080 on localhost so that we can access Nginx by browsing to http://localhost:8080.


The interactive session logs show us that confd fails to connect to Redis on 127.0.0.1:6379 because there is no Redis service inside the container.

To fix this we bring xynova/socat to create a tunnel that confd can use to talk to Azure Redis Cache in the cloud. We open a new terminal and type the following (note: replace XXXX-XXXX-XXXX with your own Azure Redis prefix).

Notice that by specifying --net container:nginx option, I am instructing the xynova/socat container to join the xynova/nginx-confd container network namespace. This is the way we get containers to share their own private localhost sandbox.

Now looking back at our interactive logs we can see that confd is now talking to Azure Redis but it cannot find the/myapp/suggestions/drink cache key.

Lets just set a value for that key:

confd is now happily synchronized with Azure Redis and the Nginx service is up and running.

We now browse to http://localhost:8080 and check test our container composition:

Covfefe… should we fix that?
We just set the /myapp/suggestions/drink key to coffee.

Watch how confd notices the change and proceeds to update the target config files.

Now if we refresh our browser we see:

Happy hacking.

 

Build from source and package into a minimal image with the new Docker Multi-Stage Build feature

Confd is a Golang written binary that can help us make configuration files dynamic. It achieves this by providing a templating engine that is driven by backend data stores like etcd, consul, dynamodb, redis, vault, zookeeper.

https://github.com/kelseyhightower/confd

A few days ago I started putting together a BYO load-balancing PoC where I wanted to use confd and Nginx. I realised however that some features that I needed from confd were not yet released. Not a problem; I was able to compile the master branch and package the resulting binary into an Nginx container all in one go, and without even having Golang installed on my machine. Here is how:

confd + Nginx with Docker Multi-Stage builds

First I will create my container startup script  docker-confd/nginx-confd.sh.
This script launches nginx and confd in the container but tracks both processes so that I can exit the container if either of them fail.

Normally you want to have only once process per container. In my particular case I have inter-process signaling between confd and Nginx and therefore it is easier for me to keep both processes together.

Now I create my Multi-Stage build Dockerfile:  docker-confd/Dockerfile
I denote a build stage by using the AS <STAGE-NAME> keyword: FROM golang:1.8.3-alpine3.6 AS confd-build-stage. I can reference the stage by name further down when I am copying the resulting binary into the Nginx container.

Now I build my image by executing docker build -t confd-nginx-local docker-confd-nginx.

DONE!, just about 15MB extra to the Nginx base alpine image.

Read more about the Multi-Stage Build feature on the Docker website.

Happy hacking.

Microsoft Azure Cross Platform Command Line Step by Step

Microsoft Azure is not just about Windows, Microsoft Azure also supports Linux workloads. Spinning up Linux VMs in Microsoft’s fabric offers alternative options for open-source technologies with Microsoft Azure services.

Microsoft also provides Azure Cross-Platform Command-Line Interface (X-Plat CLI) which is a set of Open-Source, Cross-Platform commands for managing Microsoft Azure platform. X-Plat CLI has few top-level commands which correspond to different set of Microsoft Azure features. Typing “azure” will list each of the sub commands.

X-Plat CLI command line tool is implemented in JavaScript (powered by Node.js).

Deploying Linux VM from Microsoft Azure Management Portal
This blog provides Step by Step instructions via Linux VM. It is quite straight forward deploying secure Linux VM by providing a PEM Certificate associated with Private Key. With this Certificate it is possible to create “Password-less” Linux VM when using Azure Management Portal “From Gallery” functionality or Command Line Tool.

Below instructions will provide Step-by-Step guide how to generate key pair with Git. OpenSSL parameter is used on this occasion.

  • Install and launch Git Bash for Windows computer or launch Terminal for OSX
  • Use command below to generate a key pair:

    nix1

  • Fill in some information for the key pair

    nix2

Once it is done, save the .pem file – this is the public key to be “attached” to new Windows Azure VM. Store the “.key” file safely. For Linux system or Mac OSX command chmod 0600 *.key can be leveraged to secure it from unwanted access.

Create new Azure Linux VM by clicking New button on the bottom left Azure Management Portal and select Compute > Virtual Machine > From Gallery > Pick the Linux OS (Recommended to create the Affinity Group, Storage Account and Virtual Network) > Ensure not to check the “Provide a Password” checkbox, instead upload the Certificate. This is the .pem certificate above. Simply follow the rest of deployment wizard to complete the deployment.

nix3

Connecting to Linux VM
SSH Client is needed to connect to Linux VM:

  • Linux: Use SSH
  • Mac OSX: Use SSH built-in the Terminal App
  • Windows: Use SSH built-in Git Bash shell or Download and use Putty

Below is sample SSH command, simply by passing various parameters:

$ssh –i ./kloudadmin.key kloudadmin@kloudlinux1.cloudapp.net –p22

  • ssh = command we use to connect to Linux VM
  • -i ./kloudadmin.key = pointing to private .key file associated .pem used for Linux VM
  • kloudadmin@kloudlinux1.cloudapp.net = Linux VM user name @ VM DNS name
  • -p 22 = The port to connect to, 22 is the default endpoint (the endpoint can be specified)

nix4

Installing Cross Platform Command Line (X-Plat CLI)
There are few ways to install the X-Plat CLI; using installer packages for Windows and OS X or combination of Node.js and NPM for Linux.

Node.js and npm via nave
Nave is a tool for handling node.js installations. Nave is to node.js just like RVM is to Ruby. It pulls directly from nodejs.org

Follow below instructions:

Note: # = explanation; $ command = execute on Linux VM

$ sudo su –#install node.js through nave
$ wget https://raw.github.com/isaacs/nave/master/nave.sh
$ chmod +x nave.sh
$ ./nave.sh install 0.10.15
$ ./nave.sh use 0.10.15
$ node –v

nix5

#install npm

$ curl -s https://npmjs.org/install.sh > npm-install-$$.sh
$ sh npm-install-*.sh

 nix6

Microsoft Azure X-Plat CLI
use npm command to install Azure X-Plat CLI

#install X-Plat CLI
$ npm install azure-cli -g

Using Microsoft Azure X-Plat CLI

Type $azure to test and show sub-commands

nix7

Microsoft Azure Publish Settings File
MIcrosoft Azure Publish Settings File needs to be downloaded and imported in order to create resources on related subscription.

$azure account download
$azure account import “path to the publishing file”

Note: You need a browser to download the publish settings file or You can download the file from local machine and upload it to Azure Linux VM
Example Azure Account Import: $ azure account import My.publishsettings

nix8

Click here for more details info how to use Microsoft Azure X-Plat CLI