Supercharge your CloudFormation templates with Jinja2 Templating Engine

If you are working in an AWS public cloud environment chances are that you have authored a number of CloudFormation templates over the years to define your infrastructure as code. As powerful as this tool is, it has a glaring shortcoming: the templates are fairly static having no inline template expansion feature (think GCP Cloud Deployment Manager.) Due to this limitation, many teams end up copy-pasting similar templates to cater for minor differences like environment (dev, test, prod etc.) and resource names (S3 bucket names etc.)
Enter Jinja2. A modern and powerful templating language for Python. In this blog post I will demonstrate a way to use Jinja2 to enable dynamic expressions and perform variable substitution in your CloudFormation templates.
First lets get the prerequisites out of the way. To use Jinja2, we need to install Python, pip and of course Jinja2.
Install Python
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
sudo yum install python
[/sourcecode]
Install pip
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
sudo python get-pip.py
[/sourcecode]
Install Jinja2
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
pip install Jinja2
[/sourcecode]
To invoke Jinja2, we will use a simple python wrapper script.
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
vi j2.py
[/sourcecode]
Copy the following contents to the file j2.py
[sourcecode language=”python” wraplines=”false” collapse=”false”]
import os
import sys
import jinja2
sys.stdout.write(jinja2.Template(sys.stdin.read()).render(env=os.environ))
[/sourcecode]
Save and exit the editor
Now let’s create a simple CloudFormation template and transform it through Jinja2:
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
vi template1.yaml
[/sourcecode]
Copy the following contents to the file template1.yaml
[sourcecode language=”cpp” wraplines=”false” collapse=”false”]
—
AWSTemplateFormatVersion: ‘2010-09-09’
Description: Simple S3 bucket for {{ env[‘ENVIRONMENT_NAME’] }}
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: InstallFiles-{{ env[‘AWS_ACCOUNT_NUMBER’] }}
[/sourcecode]
As you can see it’s the most basic CloudFormation template with one exception, we are using Jinja2 variable for substituting the environment variable. Now lets run this template through Jinja2:
Lets first export the environment variables
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
export ENVIRONMENT_NAME=Development
export AWS_ACCOUNT_NUMBER=1234567890
[/sourcecode]
Run the following command:
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
cat template1.yaml | python j2.py
[/sourcecode]
The result of this command will be as follows:
[sourcecode language=”cpp” wraplines=”false” collapse=”false”]
—
AWSTemplateFormatVersion: ‘2010-09-09’
Description: Simple S3 bucket for Development
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: InstallFiles-1234567890
[/sourcecode]
As you can see Jinja2 has expanded the variable names in the template. This provides us with a powerful mechanism to insert environment variables into our CloudFormation templates.
Lets take another example, what if we wanted to create multiple S3 buckets in an automated manner. Generally in such a case we would have to copy paste the S3 resource block. With Jinja2, this becomes a matter of adding a simple “for” loop:
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
vi template2.yaml
[/sourcecode]
Copy the following contents to the file template2.yaml
[sourcecode language=”cpp” wraplines=”false” collapse=”false”]
—
AWSTemplateFormatVersion: ‘2010-09-09’
Description: Simple S3 bucket for {{ env[‘ENVIRONMENT_NAME’] }}
Resources:
{% for i in range(1,3) %}
S3Bucket{{ i }}:
Type: AWS::S3::Bucket
Properties:
BucketName: InstallFiles-{{ env[‘AWS_ACCOUNT_NUMBER’] }}-{{ i }}
{% endfor %}
[/sourcecode]
Run the following command:
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
cat template2.yaml | python j2.py
[/sourcecode]
The result of this command will be as follows:
[sourcecode language=”cpp” wraplines=”false” collapse=”false”]
—
AWSTemplateFormatVersion: ‘2010-09-09’
Description: Simple S3 bucket for Development
Resources:
S3Bucket1:
Type: AWS::S3::Bucket
Properties:
BucketName: InstallFiles-1234567890-1
S3Bucket2:
Type: AWS::S3::Bucket
Properties:
BucketName: InstallFiles-1234567890-2
[/sourcecode]
As you can see the resulting template has two S3 Resource blocks. The output of the command can be redirected to another template file to be later used in stack creation.
I am sure you will appreciate the possibilities Jinja2 brings to enhance your CloudFormation templates. Do note that I have barely scratched the surface of this topic, and I highly recommend you to have a look at the Template Designer Documentation found at http://jinja.pocoo.org/docs/2.10/templates/ to explore more possibilities. If you are using Ansible, do note that Ansible uses Jinja2 templating to enable dynamic expressions and access to variables. In this case you can get rid of the Python wrapper script mentioned in this article and use Ansible directly for template expansion.