Using Ansible to create an inventory of your AWS resources

First published on Nivlesh’s personal blog at https://nivleshc.wordpress.com

Background

I was recently at a customer site, to perform an environment review of their AWS real-estate. As part of this engagement, I was going to do an inventory of all their AWS resources. Superficially, this sounds like an easy task, however when you consider the various regions that resources can be provisioned into, the amount of work required for a simple inventory can easily escalate.

Not being a big fan of manual work, I started to look at ways to automate this task. I quickly settled on Ansible as the tool of choice and not long after, I had two Ansible playbooks ready (the main and the worker playbook) to perform the inventory.

In this blog, I will introduce the two Ansible playbooks that I wrote. The first playbook is the main actor. This is where the variables are defined. This playbook iterates over the specified AWS regions, calling the worker playbook each time, to check if any resources have been provisioned in these regions. The output is written to comma separated value (.CSV) files (I am using semi-colons instead of commas), which can be easily imported into Microsoft Excel (or any spreadsheet program of your choice) for analysis.

Introducing the Ansible playbooks

The playbooks have been configured to check the following AWS resources

  • Virtual Private Cloud (VPC)
  • Subnets within the VPCs
  • Internet Gateways
  • Route Tables
  • Security Groups
  • Network Access Control Lists
  • Customer Gateways
  • Virtual Private Gateways
  • Elastic IP Addresses
  • Elastic Compute Cloud Instances
  • Amazon Machine Images that were created
  • Elastic Block Store Volumes
  • Elastic Block Store Snapshots
  • Classic Load Balancers
  • Application Load Balancers
  • Relational Database Service Instances
  • Relational Database Service Snapshots
  • Simple Storage Service (S3) Buckets

The table below provides details for the two Ansible playbooks.

FilenamePurpose
ansible-aws-inventory-main.ymlThis is the controller playbook. It iterates over each of the specified regions, calling the worker playbook to check for any resources that are provisioned in these regions.
ansible-aws-inventory-worker.ymlThis playbook does all the heavy lifting. It checks for any provisioned resources in the region that is provided to it by the controller playbook

Let’s go through each of the sections in the main Ansible playbook (ansible-aws-inventory-main.yml), to get a better understanding of what it does.

First off, the variables that will be used are defined

aws_regions – this defines all the AWS regions which will be checked for provisioned resources

verbose – to display the results both on screen and to write it to file, set this to true. Setting this to false just writes the results to file.

owner_id – this is the account id for the AWS account that is being inventoried. It is used to retrieve all the Amazon Machine Images (AMI) that are owned by this account

Next, the column headers for each of the .CSV files is defined.

After this, the output filenames are defined. Do note that the file names use timestamps (for when the playbook is run) as prefixes. This ensures that they don’t overwrite any output files from previous runs.

When I was generating the inventory list, at times I found that I needed only a subset of resource types inventoried, instead of all (for instance when I was looking for only EC2 instances). For this reason, I found it beneficial to have boolean variables to either enable or disable inventory checks for specific resource types.

The next section lists boolean variables that control if a particular resource type should be checked or not. Set this to true if it is to be checked and false if it is to be skipped. You can set this to your own preference.

After all the variables have been defined, the tasks that will be carried out are configured.

The first task initialises the output .CSV files with the column headers.

Once the initialisation has been completed, the inventory process is started by looping through each of the specified AWS regions and calling the worker Ansible playbook to check for provisioned resources.

The last task displays the path for the output files.

The main Ansible playbook (ansible-aws-inventory-main.yml) can be downloaded from https://gist.github.com/nivleshc/64ea7201fb0ba8cb6f87d06adc6152de.

The worker playbook (ansible-aws-inventory-worker.yml) has the following format.

  • go through each of the defined resource types and confirm that it is to be checked (checks for a particular resource type are enabled using the boolean variable that is defined in the main playbook)
  • If checks are enabled for that particular resource type, find all provisioned resources of that type in the region provided by the main Ansible playbook
  • write the results to the respective output file
  • if verbose is enabled, write the results to screen

The worker file (ansible-aws-inventory-worker.yml) can be downloaded from https://gist.github.com/nivleshc/bedd2c440c816ebc86dbaeddef50d500.

Running the Ansible playbooks

Use the following steps to run the above mentioned Ansible playbooks to perform an inventory of your AWS account.

1. On a computer that has Ansible installed, create a folder and name it appropriately (for instance inventory)

2. Download ansible-aws-inventory-main.yml from https://gist.github.com/nivleshc/64ea7201fb0ba8cb6f87d06adc6152de and put it in the folder that was created in step 1 above

3. Download ansible-aws-inventory-worker.yml from https://gist.github.com/nivleshc/bedd2c440c816ebc86dbaeddef50d500 and put it in the folder that was created in step 1 above

4. Download the Ansible inventory file from https://gist.github.com/nivleshc/bc2e300fe1d2779ecc15c0876fc4db62, rename it to hosts and put it in the folder that was created in step 1 above

5. Customise ansible-aws-inventory-main.yml by adding your account id as the owner_id and change the output folder by updating the output_root_folder variable. If you need to disable inventory for certain resource types, you can set the respective boolean variable to false.

6. Create a user account with access keys enabled within your AWS account. For checking all the resources defined in the playbook, at a minimum, the account must have the following permissions assigned.

AmazonVPCReadOnlyAccess
AmazonEC2ReadOnlyAccess
ElasticLoadBalancingReadOnly
AmazonRDSReadOnlyAccess
AmazonS3ReadOnlyAccess

7. Open a command line and then run the following to configure environment variables with credentials of the account that was created in step 6 above (the following commands are specific to MacOS).

export AWS_ACCESS_KEY_ID="xxxxx"
export AWS_SECRET_ACCESS_KEY="xxxxxxx"

8. There is a possibility that you might encounter an error with boto complaining that it is unable to access region us-west-3. To fix this, define the following environment variable as well.

export BOTO_USE_ENDPOINT_HEURISTICS=True

9. Run the Ansible playbook using the following command line.

ansible-playbook -i hosts ansible-aws-inventory-main.yml

Depending on how many resources are being inventoried, the playbook can take anywhere from five to ten minutes to complete. So, sit back and relax, while the playbook runs.

I found a bug with the Ansible “AWS S3 bucket facts” module. It ignores the region parameter and instead of returning S3 buckets in a specific region, it returns buckets in all regions. Due to this, the s3 buckets .CSV file will have the same buckets repeated in all the regions.

Hope you enjoy the above Ansible playbooks and they make your life much easier when trying to find all resources that are deployed within your AWS account.

Till the next time, enjoy!

Weekly AWS Update: Friday 5th April 2019

Well, it’s been a little quiet this past week in the world of AWS updates and releases. But there are still a couple of interesting announcements to cover off in this weeks article. We have a new feature available to Fargate workloads and a price Drop for those using DynamoDB global tables. A new Search expression for CloudWatch dashboards, and new policy abilities when working with CloudWatch alarms.

As always, this list is not meant to be an exhaustive list of all the updates and changes to the AWS eco-system, but simply a summary of changes that might have an impact on the business and trends we at Kloud are seeing within the industry. As always, if you would like to talk to somebody about how you might be able to leverage some of these new technologies and services, please feel free to reach out using the contact link at the top of the page.

The key takeaways from this week are:

  • AWS Fargate PV1.3 adds secrets and enhanced container dependency management
  • Amazon CloudWatch Launches Search Expressions
  • You can now use resource level policies for Amazon CloudWatch Alarms
  • Amazon DynamoDB drops the price of global tables by eliminating associated charges for DynamoDB Streams

AWS Fargate PV1.3 adds secrets and enhanced container dependency management

First off in this week’s rundown is the announcement coming out of the Fargate team. On Wednesday it was announced that AWS Secrets Manager is available when using AWS Fargate Platform Version 1.3. This means you can store sensitive information in either AWS Secrets manager or AWS Systems Manager Parameter Store. Previously, you had to directly reference this sensitive information in the task definition (more information about that release can be referenced here) however now supports task definition conventions for accessing sensitive information stored in either AWS Secrets Manager and AWS Systems Manager Parameter Store. This new option for handling secrets within container-based workloads will help teams manage containers in a simpler manner.

In addition to new ways of managing secrets within container-based workloads is the ability to add task definition parameters to define dependencies for container start-up and shutdown. Previously there was no way to ensure that your container workloads started or stopped in a particular order, making the management on micro-service based workloads a difficult platform to manage. With this new update allows teams to define things such as extended stop timeouts for workloads requiring long-running clean-up activities or ensuring that backend services (such as telematics or database workloads) are started prior to front-end workloads.

More information on both of these features is available here and details on configuring dependency management are available in the ECS developer guide here.

Amazon CloudWatch Launches Search Expressions

Coming out of the CloudWatch team on Tuesday was the announcement that “You can now use search expressions to define Amazon CloudWatch dashboards. This enables you to create dashboards that update automatically as new resources have created that match the search query, providing up-to-the-minute visibility with reduced operational overhead.”

This is going to be a really handy feature and will make CloudWatch Dashboards simpler for a lot of customers. I haven’t had a chance to play with this new feature as of yet. However, will take it for a test drive in the coming days and release a blog next week. In the meantime, anybody who’s interested in investigating it for themselves can do so by visiting the link here.

You can now use resource level policies for Amazon CloudWatch Alarms

While we are on the topic of monitoring comes another announcement on Monday stating that “You can now create tags in CloudWatch alarms that let you define policy controls for your AWS resources. This enables you to specify fine-grained permissions, improving security for monitoring resources and cost allocation.”. Where this is going to get really powerful is when it’s partnered with IAM policies. You could define an IAM policy based on the resource group, and control which users (if any) can delete/alter the associated resource tags (such as an environment tag for defining between PROD and DEV). This would ensure that tags used for reporting/alarming and budgeting are now accidentally altered by users ensuring their accuracy. The official announcement is available here and details on creating groups of resources here.

Amazon DynamoDB drops the price of global tables by eliminating associated charges for DynamoDB Streams

And lastly, for our NoSQL people, was the announcement that “Amazon DynamoDB has reduced the price of global tables by eliminating associated charges for DynamoDB Streams”. This announcement didn’t get as much public press as the announcement around the new S3 storage announcement (with the announcement of the new Glacier Deep Archive storage tier can be found here) but has the potential to save large amounts of money for those using Global Tables. Previous, Cross-Region replication of Global tables incurred charges for the associated DynamoDB stream. This billing model could add up for those running Global tables with a large number of writes. Under this price change, these streams are no longer billed for. However, it is important to note that you will continue to be billed for other DynamoDB stream workloads. Full details of the announcement are available here

And that’s it for the AWS update for Friday the 5th of April 2019. Please keep an eye out for our weekly updates on the happenings within the AWS eco-system. We post updates every Friday as well as detailed tutorials and deep dives on products throughout the week. If there something you’d like to see on the Kloud Blog, please feel free to drop a comment below.

 

Weekly AWS Update: Friday 29th March 2019

Have you heard about containers?

Today is Friday, March 29th, are we are 25% of the way through the year and things are coming quick and fast in the world of Amazon Web Services. When I was going through the recent announcements in preparation for this article it was a pleasure to see that the world of containers has gotten a number of handy feature updates this week. From new pre-configured container images to testing tools and even new deployment options, it’s all happening this week. As always, this list is not meant to be an exhaustive list of all the updates and changes to the AWS ecosystem, but simply a summary of changes that might have an impact on the business and trends we at Kloud are seeing within the industry. If you would like to talk to somebody about how you might be able to leverage some of these new technologies and services, please feel free to reach out using the contact link at the top of the page.

The key takeaways from this week are:

  • AWS Fargate and Amazon ECS Support external Deployment Controllers for ECS Services
  • Amazon EKS Opens Public Preview of Windows Container Support
  • New Local Testing Tools Now Available for Amazon ECS
  • New AWS Deep Learning Containers
  • Service control policies in AWS Organizations enable fine-grained permission controls

AWS Fargate and Amazon ECS Support external Deployment Controllers for ECS Services

First off in this week’s rundown is the announcement coming out of the Containers team that both AWS Fargate and Amazon Elastic Container Services (ECS) now support deployment controllers for ECS services with the launch of task set management API’s. What does that mean for the masses? Well, Task Sets are a new primitive to allow controlled management of application revisions within a single ECS Service.

Taken Directly from the announcement “You typically deploy new versions of your services in a staged deployment sequence to ensure everything is functioning smoothly before fully transitioning your current production workload from one version to another.” While this process does work, it limited a team’s available to granularly control how the updates and transfer of traffic would occur. If you wanted to fully manage the release of a new version, you would typically create a new ECS service and transition users across.

With this announcement “Now, you may leverage task sets to configure multiple revisions of your application all scoped within a single service. When you want to deploy a new version of your application you may create a new task set within an existing service that has its own task definition. Individual task sets also provide a scale parameter which will ensure that it is running a specific percentage of the overall desired tasks of your service.” This means that you can release a new revision of your application and control how it is eased into your production environment, reducing the size of the change and in turn lowering the likelihood of unexpected issues arising. This is shown to save a lot of time for teams currently working on Fargate and ECS workloads and we will be releasing an article in the coming weeks with a deep dive on the new functionality. If you would like to investigate the new feature for yourself in the meantime the can get started by revising the updated documentation available here.

Amazon EKS Opens Public Preview of Windows Container Support

Here at Kloud, we are no stranger to running Windows-based container workloads for our customers on Amazon ECS. Well, on Wednesday it was announced that AWS has just launched a developer preview of Windows nodes for Amazon Elastic Container Service for Kubernetes (EKS). Starting with Kubernetes version 1.11, users can begin to test and validate migrating windows workloads to containers managed by Kubernetes with Amazon EKS.

I’m sure there are a lot of people itching to get started playing with Windows nodes, and you can start right now by heading over to the GitHub project, which has all of the information you’ll need to get up and running. It’s important to remember, however, that this is a developer preview and as such there are a number of Important considerations to be aware of before starting:

  • EKS Windows nodes are only supported by Kubernetes version 1.11 (1.10 is not supported).
  • Windows EC2 instance types C3, C4, D2, I2, M4 (excluding m4.16xlarge), and R3 instances are not supported.
  • Microsoft doesn’t support hostnetworking mode in Windows yet. Hence an EKS Windows cluster will be a mixed mode cluster (1 Linux node and 3+ Windows nodes).
  • The VPC resource controller and coredns will be running in a Linux node.
  • Kubelet and kube-proxy event logs are redirected to Windows Event log (Log: EKS) and is set to 200 MB limit.
  • There is no support for secondary CIDR blocks with Windows nodes.
  • Workloads must have valid node selectors:

As of the time of writing, there are 159 open issues on the GitHub project (quite a number are feature requests rather than actual issues/bugs) with another 60 already closed so it’s clearly a very active project.

New Local Testing Tools Now Available for Amazon ECS

Continuing on the Container theme, the ECS team also announced on Wednesday the availability of a new set of open sources tools to test an application locally before deploying to ECS.

“Previously, you had to deploy your applications to ECS to ensure that updates were technically working properly for the credentials and task metadata service. If something was misconfigured, it required subsequent deployments to production to resolve the issue.

 

Now, you can use a new open source repository to locally test containers that use AWS credentials or the task metadata service end-points. This speeds up the local development iteration loop to allow you to get your applications running faster on ECS with confidence”

I’m a little torn when it comes to local development tools. On the one hand, if you have a solid CI/CD toolchain, there is an argument to be had that decoupling some of your development tasks may reduce the traceability of your development and deployment workflows. On the other hand, I’ve lost count of the number of times I’ve deployed broken applications and had to redeploy the previous version to fix the issue, so being able to valid earlier in the development cycle will save time and effort (Not to mention my personal sanity level). I’ll defiantly be giving this a test of my next ECS project so if nobody else beats me to it, I’ll document my experience and post it here. As with most things AWS, they’ve released the code under the awslabs GitHub organization and it’s available here if you’d like to give it a spin for yourself.

New AWS Deep Learning Containers

And lastly, for our Container updates, is the release of a range of newly created AWS Deep Learning Containers. “AWS Deep Learning Containers (AWS DL Containers) are Docker images pre-installed with deep learning frameworks to make it easy to deploy custom machine learning environments quickly. AWS DL Containers support TensorFlow and Apache MXNet, with PyTorch coming soon.” While I’m not the right person to be putting these new containers through their paces (We have much smarter people than I when it comes to Deep Learning), I’m excited to see AWS continue to make these services available and lower the barrier of entry to Machine Learning. It’s important to note that AWS Deep Learning Containers can be deployed on Amazon EKS, Amazon ECS, EC2 and Self-managed Kubernetes environments for specific information available at the product page here.

Service control policies in AWS Organizations enable fine-grained permission controls

Breaking away from the world of containers is an announcement that I’ve been waiting for ever since AWS Organizations was first announced. On Monday it was announced that “you can use Service Control Policies (SCPs) to set permission guardrails with the fine-grained controls used in AWS Identity and Access Management (IAM) policies” This is huge, with the update, you can now specify conditions, Resources and NotActions to deny access across accounts in you AWS organization. These new features greatly open up the options customers have when designing and managing their Multi-Account environments and I am already in the process of writing an article showcasing how they can be used to simplify your security footprint… Keep an eye out for it in the next day or two. If you’re like me and want to dive right in…. take a look at the updated Service Control Policies Documentation here.

And that’s it for the AWS update for Friday the 29th of March 2019. Please keep an eye out for our weekly updates on the happenings within the AWS eco-system. We post updates every Friday as well as detailed tutorials and deep dives on products throughout the week. If there something you’d like to see on the Kloud Blog, please feel free to drop a comment below.

 

Deploy VPN tunnel between Azure cloud and AWS cloud environment

Recently, I was working with a client who utilizes both AWS and Azure in their business and needed to establish a VPN connection across these cloud providers for certain services. This gave me the opportunity to work on establishing a VPN tunnel between the two public cloud offerings.

I would like to share the steps I used to establish the VPN (Site-to-Site) tunnel between Azure and AWS.

In the ideal world, the Azure VPN Gateway and AWS Gateway offering should have been enough to establish the VPN connection. But I soon discovered that:

  • The Azure VPN Gateway will act both as initiator or responder for VPN access, but the AWS Gateway can only act as a responder. Due to this limitation, the AWS environment requires an appliance or a supported VPN device.

I chose to build out my solution by implementing a Windows Server with Routing and Remote Access service (RRAS) in AWS as my VPN device. There are other appliances that can do this job as well and for more information around the supported Azure Site-to-Site VPN devices please refer to this link: https://docs.microsoft.com/en-us/Azure/vpn-gateway/vpn-gateway-about-vpn-devices?WT.mc_ID=ITOpstalk-Blog-pierrer

Pre-requisites:

  • You need to have valid subscriptions in both the Azure and AWS environment.
  • When configuring a Site-to-Site connection, public-facing IPv4 IP addresses are required for your VPN device
  • In my lab environment, I don’t have any custom DNS Servers. If you do have DNS servers, enter these details in your Virtual network settings > DNS servers
  • Windows and Linux VMs across Azure and AWS to establish and test our Site-to-Site (S2S) VPN connectivity.

Preparing your Azure Environment

We will complete the following tasks in our Azure portal:

  • Create a Virtual network (VNet)
  • Create a Gateway Subnet
  • Create a VPN Gateway
  • Create a Local Network gateway (your remote site i.e. AWS)
  • Deploy a Windows server 2016

Create a Virtual network (VNet):

  • Click + create a resource button, Select, Networking > Virtual Network
  • Enter the details to create a single virtual network with one subnet

 

 

Create a Gateway Subnet:

– Once your Virtual network is created, open the blade and click on subnets
– Click the + Gateway subnet button at the top
– The Name for subnet is automatically filled in with the value ‘GatewaySubnet’. It is worth nothing this name cannot be changed to a custom name.
– Enter the IP Address range as below. Microsoft recommends keeping the gateway subnet CIDR range to a /27 or /28.
– Leave all the other settings on the defaults and click OK.

Create a VPN Gateway:

On the left side of the portal page, click + create a resource and type Virtual Network Gateway in search. In the results, click Virtual network gateway and click Create

On the Create virtual network gateway blade enter the following details:

  • Name: Enter Name of gateway
  • Gateway type: Select VPN
  • VPN type: Route-based (This the new name for dynamic routing)
  • SKU: For our lab, select VpnGw1. Refer here for more about gateway SKUs

NOTE: Only select Enable active-active mode if you are creating an active-active gateway configuration. Otherwise, leave this setting unselected.

Location: Scroll down to the bottom of the blade and select the region where the new VNet is located otherwise you will not see the desired virtual network.

  • Virtual network: Click ‘Choose a virtual network’ page and select the VNet created in the previous step. Again, if you don’t see your VNet, make sure the Location field is pointing to the region in which your virtual network is located.
  • Public IP address: Use an existing public IP address name if you have one otherwise, create new one by typing in a name in the text box.
  • Verify the details entered and click Create to begin creating the VPN gateway. Once the settings are validated, and you’ll see the “Deploying Virtual network gateway” tile on the dashboard.
  • NOTE: Creating a Virtual Network gateway can take up to 20 minutes

Create the local network gateway:

Pre-requisite: Before creating the local network gateway, follow the steps to create your VPC in the AWS section and then proceed.

The local network gateway represents the VPN device in your local network which in this instance is our AWS environment and is used to setup your site-to-site VPN connection. The local network gateway is treated as the remote location. This should not be confused with the virtual network gateway.

  • In the Azure portal, click + Create a resource. In the search box, type Local network gateway, then press Enter to search. Click Local network gateway, then click Create.
  • On the Create local network gateway page, specify the values for AWS environment in local network gateway.
    • Name: Choose a name
    • IP address: This is the public IP address of the VPN device (Windows RRAS Server in AWS) that you want Azure to connect to. In this exercise, will provide 52.62.62.149. Refer to Create Elastic IP Address in the AWS Environment section.
    • Address Space: refers to the address ranges for the network in AWS Environment. In this exercise, we will provide 172.0.0.0/16. Refer to Create VPC in the AWS Environment section.
    • (Optional) Configure BGP settings: Use only when configuring BGP. Otherwise, don’t select this.
    • Subscription: Select your subscription
    • Resource Group: Select the same resource group as your virtual network
    • Location: Select the location.
    • Click the Create button at the bottom of the page to create the local network gateway.
    • In the meantime, create a windows server in virtual network. This will be used to test the connection.

 

Establish the VPN connection:

Navigate to the local network gateway and select it.

  • Click on Overview -> Connected devices -> Azure-LNGW
  • On the page for Azure-LNGW, click Connections. At the top of the Connections page, click +Add to open the Add connection page.
  • On the Add connection page, configure the values for your connection.
    • Name: Name your connection
    • Connection type: Site-to-site (IPSec). This is selected by default.
    • Virtual network gateway: Click Choose a local network gateway
    • Local network gateway: The value is fixed because you are connecting from this gateway.
    • Shared Key: This is used for encryption for tunnel.
    • NOTE: The value you specify here must be the same value that you specify when configuring your VPN device.
    • The remaining values for Subscription, Resource Group, and Location are fixed.
    • Click OK to create your connection.
    • After its created, the status will be
    • NOTE: Wait till all the steps are done for to turn this to connected state.

Deploy Virtual Machines:

Deploy your Windows VM in Azure environments as per table below:

Azure Environment
·       Windows server 2016

I assume you are familiar with deploying virtual machines in Azure and will not show these steps below. Please refer to the links below for assistance with this.
https://docs.microsoft.com/en-us/azure/virtual-machines/windows/quick-create-portal

Preparing your AWS Environment

We will complete the following tasks in our AWS subscription:

  • Create a Virtual Private Cloud (VPC)
  • Create an Elastic IP
  • Create the VPN device
  • Add routes to the route table in the VPC

Create a Virtual Private Cloud (VPC):

Log into your AWS console and type in VPC in the find services box.  Click VPC Dashboard, otherwise, click Launch VPC Wizard.

  • Select VPC with a Single Public Subnet, and then click Select.
  • Fill in the following information in this page
    • IP CIDR block: 172.0.0.0/16
    • VPC name: AWS-VPC01
    • Public subnet: 172.0.0.0/24
    • Availability Zone: No Preference
    • Subnet name: public_subnet_aws01
    • Leave other settings on the defaults and click on Create VPC.
  • Once the VPC is created, select it and identify your VPC id.
  • On left-side of the AWS VPC dashboard
  • Select Internet gateways and verify that it is attached the VPC you created. This is required to allow the resources in your newly deployed VPC to communicate with the Internet.
  • NOTE: The Internet Gateway does not have name by default, you can give it a name if you wish.

Deploy Virtual Machines:

Deploy your Windows and Linux VMs in AWS environments as per the table below:

AWS Environment
·       Redhat Server 7.5 – Generic server

·       Windows server 2016 with RRAS

Create an Elastic IP Address:

An Elastic IP address is a public IPv4 address, which is reachable from the internet. This required for our windows server (RRAS), which will act as the VPN device in AWS.

    • From the VPC Dashboard pane, choose Elastic IPs.
    • Choose Allocate new address
    • Now leaving the settings on defaults, and click Allocate.

NOTE: Take down this IP address, as we will use it when we setup the Azure local network gateway.

    • While configuring the Windows server, please ensure you select auto assign public IP address to this instance. The server network interface cards will not have a name, you can give it a label if you wish.
    • We will use Elastic IP Address to mask the instance public ip address. This will be useful during failure.
    • Once our virtual machines are up and running, associate the Elastic IP address to the Windows server NIC.
    • In the AWS navigation pane, choose Elastic IPs.
    • Select the Elastic IP address, which we created earlier in this exercise and choose Actions > Associate address.
    • Select instance and then choose Associate.
    • Click Close

Each EC2 instance performs source/destination checks by default. This means that the instance must be the source or destination of any traffic it sends or receives. However, a NAT instance must be able to send and receive traffic when the source or destination is not itself.

In our case, we need to disable source/destination on our Windows server.

  • In the AWS console, choose Instances.
  • Select the Windows instance, choose Actions > Networking > Change Source/Dest. Check.
  • choose Yes, Disable.
  • Verify the instances settings once again, before we move to next steps.

Configure the VPN Device:

We will now configure the windows server to be our VPN device in our AWS tenant

  • RDP into AWS Windows server and install Remote Routing Access (RRAS) service role
  • Using an elevated PowerShell command (Legacy mode), run: Install-WindowsFeature RSAT-RemoteAccess-Mgmt
  • Run msc to launch the RRAS console.
  • Right-click on the server name and choose “Configure and Enable Routing and Remote Access”.
  • Click Next
  • Click Custom configuration and click Next.
  • Check the box for VPN access and click Next.
  • Click Finish to complete the initial configuration.
  • You will get a popup indicating a policy was created.
  • Click “Start this service” to get RRAS running.

Configure the S2S VPN via PowerShell:

    • Run the PowerShell script below replacing the values highlighted in yellow. The script has been taken from the Azure-vpn-config-samples GitHub repository.
    • I have taken only the relevant sections for our task and have performed the remaining steps manually.
    • The PowerShell script as:

#Install Remote Access binaries

Install-WindowsFeature DirectAccess-VPN -IncludeManagementTools
#Install S2S VPN

Import-Module RemoteAccessif ((Get-RemoteAccess).VpnS2SStatus -ne “Installed”)

{

Install-RemoteAccess -VpnType VpnS2S

}

# Add and configure S2S VPN interface

Add-VpnS2SInterface -Protocol IKEv2 `

-AuthenticationMethod PSKOnly `

-NumberOfTries 3 `

-ResponderAuthenticationMethod PSKOnly `

-Name ‘Azure VPN Gateway name‘ `

-Destination IP Address of Azure VPN Gateway `

-IPv4Subnet @(“CIDR IP Address of Azure:100″) `

-SharedSecret “Pre-shared Key from Azure “

 

Set-VpnServerIPsecConfiguration -EncryptionType MaximumEncryption

Set-VpnS2Sinterface -Name ‘Azure VPN Gateway name

Set the S2S VPN connection to be persistent:

  • From the RRAS management console, select your local server and click on Network Interfaces.
  • Select the Azure VPN Gateway name.
  • Set S2S VPN connection type to Persistent connection
  • Change Dialling policy to Zero and click Ok.

Bring up the VPN tunnel:

  • From a PowerShell console, run the commands below:

# Restart the RRAS service

Restart-Service RemoteAccess

 # Dial-in to Azure gateway

Connect-VpnS2SInterface -Name Azure VPN Gateway name

  • Verify the Azure VPN is connected from the RRAS console.
  • In Azure portal, verify Virtual Network Gateway STATUS under Connections. This should now say connected.

Add routes to the Route table in the VPC:

  • In the AWS console, switch to the VPC view and select Route tables.
  • Select the route table associated with your VPC.
  • Under Routes, click Edit routes and add routes for the following:
  • DestinationTarget
    0.0.0.0/0Igw-xxxxxx (your internet gateway)
    10.0.0.0/16Windows RRAS server instance
  • This covers the network address for the Azure Virtual network so that traffic is routed through network interface of instance ID of Windows RRAS Server.

Test Site-to-Site Connectivity:

To verify the Site-to-Site VPN is connected, we will log into Azure windows server and try to ping AWS linux server.

I hope this article has been helpful to you to understand how to create a VPN tunnel between Azure and AWS.

Using Ansible to deploy an AWS environment

First published at https://nivleshc.wordpress.com

Background

Over the past few weeks, I have been looking at various automation tools for AWS. One tool that seems to get a lot of limelight is Ansible, an open source automation tool from Red Hat. I decided to give it a go, and to my amazement, I was surprised at how easy it was to learn Ansible, and how powerful it can be.

All that one must do is to write up a list of tasks using YAML notation in a file (called a playbook) and get Ansible to execute it. Ansible reads the playbook and executes the tasks in the order that they are written. Here is the biggest advantage, there are no agents to be installed on the managed computers! Ansible connects to each of the managed computers using ssh or winrm.

Another nice feature of Ansible is that it supports third party modules. This allows Ansible to be extended to support many of the services that it natively does not understand.

In this blog, we will be focusing on one of the third-party modules, the AWS module. Using this, we will use Ansible to deploy an environment within AWS.

Scenario

For this blog, we will use Ansible to provision an AWS Virtual Private Cloud (VPC) in the North Virginia (us-east-1) region. Within this VPC, we will create a public and a private subnet. We will then deploy a jumphost in the public subnet and a server within the private subnet.

Below is a diagram depicting what will be done.

Figure 1: Environment that will be deployed within AWS using Ansible Playbook

Preparation

The computer that is used to run Ansible to manage all other computers is referred to as the control machine. Currently, Ansible can be run from any machine with Python 2 (version 2.7) or Python 3 (version 3.5 or higher) installed. The Ansible control machine can run the following operating systems

  • Red Hat
  • Debian
  • CentOS
  • macOS
  • any of the BSD variants

Note: Currently windows operating system is not supported for running the control machine.

For this blog, I am using a MacBook to act as the control machine.

Before we run Ansible, we need to get a few things done. Let’s go through them now.

  1. We will use pip (Python package manager) to install Ansible. If you do not already have pip installed, run the following command to install it
    sudo easy_install pip
  2. With pip installed, use the following command to install Ansible
    sudo pip install ansible

    For those that are not using macOS for their control machine, you can get the relevant installation commands from https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html.

  3. Next, we must install the AWS Command Line Interface (CLI) tools. Use the following command for this.
    sudo pip install awscli

    More information about the AWS CLI tools is available at https://aws.amazon.com/cli/

  4. To provision items within AWS, we need to provide Ansible with a user account that has the necessary permissions. Using the AWS console, create a user account ensuring it is assigned an access key and a secret access key. At a minimum, this account must have the following policies assigned to it.
    AmazonEC2FullAccess
    AmazonVPCFullAccess

    Note: As this is a privileged user account, please ensure that the access key and secret access key is kept in a safe place.

  5. To provision AWS Elastic Compute Cloud (EC2) instances, we require key pairs created in the region that the EC2 instances will be deployed in. Ensure that you already have key pairs for the North Virginia (us-east-1) region. If not, please create them.

Instructions

Create an Ansible Playbook

Use the following steps to create an Ansible playbook to provision an AWS environment.

Open your favourite YAML editor and paste the following code

The above code instructs Ansible that it should connect to the local computer, to run all the defined tasks. This means that Ansible modules will use the local computer to connect to AWS APIs in order to carry out the tasks.

Another thing to note is that we are declaring two variables. These will be used later in the playbook.

  • vpc_region – this is the AWS region where the AWS environment will be provisioned (currently set to us-east-1)
  • my_useast1_key – provide the name of your key pair for the us-east-1 region that will be used to provision EC2 instances

Next, we will define the tasks that Ansible must carry out. The format of the tasks is as follows

  • name – this gives a descriptive name for the task
  • module name – this is the module that Ansible will use to carry out the task
  • module Parameters – these are parameters passed to the module, to carry out the specific task
  • register – this is an optional keyword and is used to record the output that is returned from the module, after the task has been carried out.

Copy the following lines of code into your YAMl file.

The above code contains two tasks.

  • the first task creates an AWS Virtual Private Cloud (VPC) using the ec2_vpc_netmodule. The output of this module is recorded in the variable ansibleVPCusing the registercommand
  • the second task outputs the contents of the variable ansibleVPC using the debugcommand (this displays the output of the previous task)

Side Note

  • Name of the VPC has been set to ansibleVPC
  • The CIDR block for the VPC has been set to 172.32.0.0/16
  • The state keyword controls what must be done to the VPC. In our case, we want it created and to exist, as such, the value for state has been set to present.
  • The region is being set by referencing the variable that was defined earlier. Variables are referenced with the notation “{{ variable name }}”

Copy the following code to create an AWS internet gateway and associate it with the newly created VPC. The second task in the below code displays the result of the internet gateway creation.

The next step is to create the public and private subnets. However, instead of hardcoding the availability zones into which these subnets will be deployed, we will pick the first availability zone in the region for our public and the second availability zone in the region for our private subnet. Copy the following code into your YAML file to show all the availability zones that are present in the region, and which ones will be used for the public and private subnets.

Copy the following code to create the public subnet in the first availability zone in us-east-1 region. Do note that we are provisioning our public subnet with CIDR range 172.32.1.0/24

Copy the following code to deploy the private subnet in the second availability zone in us-east-1 region. It will use the CIDR range 172.32.2.0/24

Hold on! To make a public subnet, it is not enough to just create a subnet. We need to create routes from that subnet to the internet gateway! The below code will address this. The private subnet does not need any such routes, it will use the default route table.

As planned, we will be deploying jumphosts within the public subnet. By default, you won’t be able to externally connect to the EC2 instances deployed within the public subnet because the default security group does not allow this.

To remediate this, we will create a new security group that will allow RDP access and assign it to the jumphost server. For simplicity, the security group will allow RDP access from anywhere, however please ensure that for your environment, you have locked it down to a few external IP addresses.

Phew! Finally, we are ready to deploy our jumphost! Copy the following code for this

I would like to point out a few things

  • The jumphost is running on a t2.micro instance. This instance type is usually sufficient for a jumphost in a lab environment, however if you need more performance, this can be changed (changing the instance type from t2.micro can take you over the AWS free tier limits and subsequently add to your monthly costs)
  • The image parameter refers to the AMI ID of the Windows 2016 base image that is currently available within the AWS console. AWS, from time to time, changes the images that are available. Please check within the AWS console to ensure that the AMI ID is valid before running the playbook
  • Instance tags are tags that are attached to the instance. In this case, the instance tags have been used to name the jumphost win2016jh.

Important Information

The following parameters are extremely important, if you do not intend on deploying a new EC2 instance for the same server every time you re-run this Ansible playbook.

exact_count– this parameter specifies the number of EC2 instances of a server that should be running whenever the Ansible playbook is run. If the current number of instances doesn’t match this number, Ansible either creates new EC2 instances for this server or terminates the extra EC2 instances. The servers are identified using the count_tag

count_tag– this is the instance tag that is used to identify a server. Multiple instances of the same server will have the same tag applied to them. This allows Ansible to easily count how many instances of a server are currently running.

Next, we will deploy the servers within the private subnet. Wait a minute! By default, the servers within the private subnet will be assigned the default security group. The default security group allows unrestricted access to all EC2 instances that have been attached to the default security group. However, since the jumphost is not part of this security group, it will not be able to connect to the servers in the private subnet!

Let’s remediate this issue by creating a new security group that will allow RDP access from the public subnet to the servers within the private subnet (in a real environment, this should be restricted further, so that the incoming connections are from particular servers within the public subnet, and not from the whole subnet itself). This new security group will be associated with the servers within the private subnet.

Copy the following code into your YAML file.

We are now at the end of the YAML file. Copy the code below to provision the windows 2016 server within the private subnet (the server will be tagged with name=win2016svr)

Save the playbook with a meaningful name. I named my playbook Ansible-create-AWS-environment.yml

The full Ansible playbook can be downloaded from https://gist.github.com/nivleshc/344dca91e3d0349c8a359b03853886be

Running the Ansible Playbook

Before we run the playbook, we need to tell Ansible about all the computers that are within the management scope. This is done using an inventory file, which contains a group name within square brackets eg [webservers] and below that, all the computers that will be in that group. Then in the playbook, we just target the group, which in turn targets all the computers in that group.

However, in our scenario, we are directly targeting the local computer (refer to the second line in the YAML file that shows hosts: localhost). In this regard, we can get away with not providing an inventory file. However, do note that doing so will mean that we can’t use anything other than localhost to reference a computer within our playbook.

Let’s create an inventory file called hostsin the same folder as where the playbook is saved. The contents of the file will be as listed below.

[local]
localhost

We are ready to run the playbook now.

Open a terminal session and change to the folder where the playbook was saved.

We need to create some environment variables to store the user details that Ansible will use to connect to AWS. This is where the access key and secret access key that we created initially will be used. Run the following command

export AWS_ACCESS_KEY_ID={access key id}
export AWS_SECRET_ACCESS_KEY={secret access key}

Now run the playbook using the following command (as previously mentioned, we could get away with not specifying the inventory file, however this means that we only can use localhost within the playbook)

ansible-playbook -i hosts ansible-create-aws-environment.yml

You should now see each of the tasks being executed, with the output being shown (remember that after each task, we have a follow-up task that shows the output using the debug keyword? )

Once the playbook execution has completed, check your AWS console to confirm that the following items have been created within the us-east-1 (North Virginia) region

  • A VPC called ansibleVPC with the CIDR 172.32.0.0/16
  • An internet gateway called ansibleVPC_igw
  • A public subnet in the first availability zone with CIDR 172.32.1.0/24
  • A private subnet in the second availability zone with CIDR 172.32.2.0/24
  • A route table called rt_ansibleVPC_PublicSubnet
  • A security group for jumphosts called sg_ansibleVPC_publicsubnet_jumphost
  • A security group for the servers in private subnet called sg_ansibleVPC_privatesubnet_servers
  • An EC2 instance in the public subnet representing a jumphost named win2016jh
  • An EC2 instance in the private subnet representing a server named win2016svr

Once the provisioning is complete, to test, connect to the jumphost and then from there connect to the server within the private subnet.

Don’t forget to turn off the EC2 instances if you don’t intend on using them

Closing Remarks

Ansible is a great automation tool and can be used to both provision and manage infrastructure within AWS.

Having said that, I couldn’t find an easy way to do post provisioning tasks (eg assigning roles, installing additional packages etc) after the server has been provisioned, without getting Ansible to connect directly to the provisioned server. This can be a challenge if the Ansible control machine is external to AWS and the provisioned server is within an AWS private subnet. With AWS CloudFormation, this is easily done. If anyone has any advice on this, I would appreciate it if you can leave it in the comments below.

I will surely be using Ansible for most of my automations from now on.

Till the next time, enjoy!

Weekly AWS Update: Friday 1st March 2019

Well, it’s Friday, March 1st, 2 months into 2019 which means Its time again for my weekly update on all things AWS. The last couple of weeks have been a little quiet when it comes to Amazon Web Services product announcements, but there are still a few interesting things to cover off this week. Numerous announcements have been made in the Database space as well as an update to Amazon Worklink and Amazon FXs for windows. It’s not meant to be an exhaustive list of all the updates and changes to the AWS Eco-system, but simply a summary of changes that might have an impact on the business and trends we at Kloud are seeing within the industry. As always, if you would like to talk to somebody about how you might be able to leverage some of these new technologies and services, please feel free to reach out using the contact link at the top of the page.

The key takeaways from this week are:

  • Amazon RDS for Oracle Now Supports Amazon S3 Integration
  • Amazon Aurora Serverless Publishes Logs to Amazon CloudWatch
  • Amazon Athena Now Supports Resource Tagging
  • Amazon WorkLink now works with Android phones
  • Amazon FSx for Windows File Server Now Supports On-Premises Access to File Systems and Supports Access Across AWS VPCs, Accounts, and Regions

Amazon RDS for Oracle Now Supports Amazon S3 Integration

First off in this week’s rundown is the announcement coming out of the Amazon RDS team the Amazon RDS for Oracle now supports Amazon S3 Integration. This is going to be a really handy feature for those people running Oracle workloads on AWS as it allows for the ingress and egress of data between S3 and your RDS instance. Quoting from the official feature announcement (available here) “With Amazon S3 Integration, you can now use the RDS Oracle DB instance to transfer files to or from Amazon S3 using new RDS Oracle procedures.” This new capability allows for organizations to rethink their current backup and restoration strategies. RMAN or Oracle Data Pump exports can now be stored in other regions in either S3 or S3 glacier. The AWS user guide on configuring the new S3 integration feature can be found here

Amazon Aurora Serverless Publishes Logs to Amazon CloudWatch

And while we’re on the topic of new features coming to database workloads, the Aurora Serverless team announced on Monday that Amazon Aurora Servers can now publish logs to Amazon CloudWatch. Anybody who’s ever spent any time with me will know that I’m a huge advocate for Serverless architecture. So, it should come as no surprise that Aurora Serverless is my go-to relational database solution.

For those who are not aware, Aurora Serverless is a Relational Database solution that allows you to pay for database storage, plus the database capacity and I/O your database consumes only while it is active. Capacity is measured in Aurora Capacity Unit’s (ACU’s) which equates to approximately 2GB of memory with corresponding CPU and networking. At the time of writing, an Aurora Capacity unit is $0.10 per ACU hour when running in the Sydney region.

With this announcement, users gain the ability to configure Amazon Aurora Serverless to publish general logs, slow query logs, audit logs and error logs to Amazon CloudWatch Logs. This means that we can now configure event-based triggers for notification and operational activities. Watch out for a future blog article where I take a look at configuring events in CloudWatch to alert on unwanted changes. For those who want to get starting with publishing their own logs to CloudWatch, you can take a look at the product announcement here or the Aurora Serverless Documentation here.

Amazon Athena Now Supports Resource Tagging

Moving from Databases to Analytics, there was another announcement on Monday (Monday was a good day for new features) that Amazon Athena now supports resource tagging. Taken from the official announcement (available here) “Athena Workgroup is a new resource type that can be used to separate query execution and query history between Users, Teams, or Applications running under the same AWS account. With the latest release, you can assign metadata to Athena Workgroups in the form of tags”. A solid tagging taxonomy is one of the key cornerstones in accurately and efficiently monitoring, reporting on and managing resources within your AWS environment. I recommend taking a look at AWS’s tagging strategies answers page here for those wanting to review or improve their tagging policies.

Amazon WorkLink now works with Android phones

A few weeks ago, AWS released a new product call Amazon Worklink. It’s a fully managed service that lets you provide your employees with secure, easy access to your internal corporate websites and web apps using their mobile phones. As a part of the initial release, they provided support for Apple iOS devices with an app available on the iOS App Store. This week marks the release and support for the Android mobile platform with the application now available on the Google Play Store. Now both leading mobile platforms are supported, Worklink is a useful solution for providing users secure access to internal resources.

Amazon FSx for Windows File Server Now Supports On-Premises Access to File Systems and Supports Access Across AWS VPCs, Accounts, and Regions

And finally, for this week’s update is that Amazon FSx for Windows File Server now supports On-Premises access to File Systems and supports access across AWS VPCs, Accounts, and Regions. This addition to the Amazon FSx for Windows File Server product opens up the possible use cases. Quoting from the official announcement (available here) “With on-premises access, you can easily migrate your on-premises data sets to Amazon FSx. In addition, you can use it for hosting user shares accessible by on-premises end-users and use Amazon FSx for backup and disaster recovery solutions. With inter-VPC, inter-account, and inter-Region access, you can share your file data sets across multiple applications, internal organizations, or environments spanning multiple VPCs, accounts, or Regions.” For a more detailed look at Amazon FSx for Windows File Server, please take a look at a recent blog article by our own Jason Wood available here.

And that’s it for the AWS update for Friday the 1st of March 2019. Please keep an eye out for our weekly updates on the happenings within the AWS eco-system. We post updates every Friday as well as detailed tutorials and deep dives on products throughout the week. If there something you’d like to see on the Kloud Blog, please feel free to drop a comment below.

Enabling Billing Visability for IAM Users in AWS

Today I’ve going to cover off an issue that I’ve seen in a couple of customer environments recently and that’s around allowing users the correct level of access to their AWS billing, Budgets, and account settings. This is an important topic as cost optimization and operational excellence are two of the 5 pillars of the AWS Well Architected Framework and if you can’t monitor your spend, it’s hard to ensure your delivering business value.

AWS Cost Management Tools

Firstly, when we talk about billing and cost management within AWS, there are typically 4 main tools/dashboards that you might want to use:

  • Billing & Cost Management Dashboard. This is the main dashboards that you can use to see the current months spend across the AWS services, your forecast spends for the month and how your tracking compared to last month. Below is an example of the dashboard.

 

  • Billing Details. The bills page is a breakdown of all AWS charges for a particular month by service and (if you have AWS Organizations or consolidated billing enabled) account as shown in the below screenshot.

  • AWS Cost Explorer. AWS Cost Explorer is a reporting and graphing tool that can be used to show and report on costs, usage, and trends across your AWS accounts. This tool can be useful for things such as trend analysis, breakdown by cost allocation tags and tracking budgets. Below is an example of the Cost & usage report.

  • BudgetsThe Budgets page does pretty much what you’d expect, it allows you to set budgets across AWS services and allows for alerts to be raised based on actual or forecast spend. An example of a basic EC2 spend budget is shown below and the alert is triggered.

Enabling IAM Access to Cost Management Tools

By default, an AWS account is configured to only allow the Root user to access the 4 key areas to view and report on costs. Given that we’re following AWS best practice and not using our AWS Root account (and if you are, please stop), how do we go about enabling our IAM users to access them? Firstly, we need to enable IAM user/role access to billing information. To do this is a four-step process:

  • Scroll down to “IAM User and Role Access to Billing Information” and click edit.

  • Tick “Active IAM Access” and click “Update

Configuring IAM Policies

What this has done is allowed access to billing and cost management features from IAM policies, allowing you to assign these features to your users. This means the next step is to configure an IAM policy and assign it to the users that require billing visibility and control. The exact configuration of IAM policies will change form organization to organization but for the purposes of this article and keeping with the notation of least privilege, we’re going to focus on two main use cases.

Firstly, for users that simply require the ability to view the billing information we can set a simple IAM policy that just allows access to “aws-portal:ViewBillig” which will provide them with view access to:

  • Billing Dashboard
  • Bills
  • Cost Explorer
  • Budgets
  • Payment History
  • Consolidated Billing
  • Preferences
  • Credits

The policy itself is rather simple:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Action": "aws-portal:ViewBilling",

            "Resource": "*"

        }

    ]

}

If we also want to allow them access to our reports, we can simply add an additional Action to our policy:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Action": [

                "aws-portal:ViewUsage",

                "aws-portal:ViewBilling"

            ],

            "Resource": "*"

        }

    ]

}

For users who in addition to needing visibility over your AWS also need to be able to set budgets and alerts, we need to expand our policy a little bit and give them modify rights as well as permission to setup CloudWatch and SNS resources (used for setting alerts) as shown below:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Action": [

                "aws-portal:ViewBilling",

                "aws-portal:ModifyBilling",

                "budgets:ViewBudget",

                "budgets:ModifyBudget"

            ],

            "Resource": [

                "*"

            ]

        },

        {

            "Effect": "Allow",

            "Action": [

                "cloudwatch:*"

            ],

            "Resource": [

                "*"

            ]

        },

        {

            "Effect": "Allow",

            "Action": [

                "sns:*"

            ],

            "Resource": [

                "*"

            ]

        }

    ]

}

You could further secure the policy by defining appropriate regions for resource usage.

Summary

And that’s it, once you assign the new policy to IAM users they will now have access to your billing and cost reporting information. In future articles, we will be taking a more in-depth look at the AWS Cost explorer as well as providing some useful template for alerts and reports. If you have questions or queries about controlling your AWS spend or anything in this article, please feel free to reach out to us using the contact us feature located at the top of the page.

Weekly AWS update: Friday 15th February 2019

Well, it’s Friday again and that can only mean one thing…. It’s time again for my weekly update on all things AWS. Last week was a big week for developers and while this week has also seen a number of new features for our developer friends, Amazon Web Services has also brought us new instance types, storage options and functionality to what’s becoming a favourite of mine, Amplify. This article continues our weekly series on the happenings in the world of Amazon Web Services. It’s not meant to be an exhaustive list of all the updates and changes to the AWS Eco-system, but simply a summary of changes that might have an impact on the business and trends we at Kloud are seeing within the industry. As always, if you would like to talk to somebody about how you might be able to leverage some of these new technologies and services, please feel free to reach out using the contact link at the top of the page.

The key take away’s from this week are:

  • Amazon Corretto 11 is Now in Preview
  • Amplify Framework Adds new features
  • Five New Amazon EC2 Bare Metal Instances
  • Amazon EFS Introduces Lower Cost Storage Class
  • Amazon GuardDuty Adds Three New Threat Detection’s

Amazon Corretto 11 is Now in Preview

As I mentioned last week, AWS Corretto has recently reached General Availability for Corretto version 8 and that AWS are planning to release version 11 before April of this year. Well, on Wednesday AWS announced that Corretto version 11 has now reached preview and is available for download from the Corretto product page here. Amazon Corretto is a no-cost, multi-platform, production-ready distribution of the Open Java Development Kit (OpenJDK). For an introduction to Amazon Corretto, you can visit the announcement page here. We will be keeping an eye on the progression of version 11 and will update you on its progress.

Amplify Framework Adds new features

And while we’re on the topic of new features that will delight the AWS developers among us, this week also saw an update to AWS Amplify. The Amplify CLI, part of the Amplify Framework, now supports multiple environments and teams by offering a Git style workflow for creating and switching between environments for your Amplify project. When you work on a project within a team, you can create isolated back-ends per developer or alternatively share back-ends across developers, including to those outside your organisation. This week’s announcement sees the introduction of several new features for the Amplify framework including:

  • support for IAM roles and MFA (Multi-Factor Authentication)
  • Custom resolvers for AWS AppSync
  • Increase in support for 150 Graphql transformer models, up from 15
  • Support for multiple environments

These new announcements really open up some of the possibilities that Amplify can solve within enterprise and large team environments. The addition of support for IAM roles and MFA means it now supports standard best practise deployments (everybody should have MFA enabled on their IAM accounts and if you don’t, do it now… I’ll wait), while the added support for multiple environments is going to greatly simplify the workflows within larger teams keen to leverage Amplify’s capabilities. The addition of custom resolver support (such as Amazon DynamoDB tables, Amazon Elasticsearch Service domains, or HTTP endpoints that were provisioned independently of the Amplify GraphQL Transformer) from within your Amplify project and the increase in the number of supported transformer models already has my mind racing with possibilities, so don’t be surprised if you see more Amplify focused articles from me in the future.

Five New Amazon EC2 Bare Metal Instances

Did somebody say new instance? I think they did. Another announcement on Wednesday saw the Bare Metal team release 5 new instances in a range of regions throughout the world, and yes… Sydney is in the list (or at least is in the list for some of them). The release sees the addition of five (5) new instances within the Bare Metal family, including:

  • Metal, a 48 physical/96 logical core instance with 384 GB of RAM, 25Gbps of available network bandwidth and 14,000 Mbps of EBS Bandwidth.
  • Metal, the same as it’s M5 counterpart only with the addition of 4 x 900GB NVMe SSD local drives.
  • Metal, a 48 physical/96 logical core instance with 796 GB of RAM, 25Gbps of available network bandwidth and 14,000 Mbps of EBS Bandwidth.
  • Metal, the same as it’s R5 counterpart only with the addition of 4 x 900GB NVMe SSD local drives.
  • Metal, a 24 physical/48 logical core instance with 384 GB of RAM, 25Gbps of available network bandwidth, 14,000 Mbps of EBS Bandwidth and 4 x 900GB NVMe SSDs.

For a full listing on where these new instances are available, you can visit the announcement here, but both the M5.Metal and M5D.metal are available in Sydney and are ready for deployment.

Amazon EFS Introduces Lower Cost Storage Class

Next cab off the rank is yet another announcement on Wednesday (Wednesday was a busy day in Seattle), this time from our storage friends with the release of a lower cost storage class for Elastic File Service. This one is exciting as I always like announcements that can save me money. In case you’ve not heard (and you might not have if you live in the windows world), EFS provides a simple, scalable, elastic file system for Linux-based workloads for use with AWS Cloud services and on-premises resources. With this new feature, you can create a new EFS file system and configure it as Infrequently used storage (S3 users should see where we are going here) and apply Life-cycle management policies to automatically move infrequently accessed files to the new storage tier. Much like the S3 equivalent, Infrequently Accessed storage comes at a much cheaper price (at the time of writing, Standard EFS is $0.36 per GB for standard and $0.054 per GB for IA in the Sydney region) however you must also pay a charge for access requests (currently $0.12 per GB transferred) when that IA data is transferred off the storage.

Why am I so excited about this, well if we configure a new File System and a Life-cycle Management policy to automatically migrate any data that hasn’t been accessed in 30 days to it, we instantly start saving money without having to change anything at the server end (no changes to workflow or application settings). Guess I know what I’ll be doing over the weekend.

Amazon GuardDuty Adds Three New Threat Detection’s

And finally, for this week’s roundup, we have three (3) new features from our friends over in GuardDuty. GuardDuty has very quickly become an “on by default” service for us here at Kloud as the benefits you gain from its insights are invaluable and these three new additions only make it more attractive for anybody running workloads in AWS. As stated in the product documentation “Once enabled, Amazon GuardDuty continuously monitors for malicious or unauthorised behaviour to help protect your AWS resources, including your AWS accounts and access keys. GuardDuty identifies unusual or unauthorised activity, like cryptocurrency mining or infrastructure deployments in a region that has never been used. When a threat is detected, you are alerted with a GuardDuty security finding that provides detail of what was observed, and the resources involved. Powered by threat intelligence and machine learning, GuardDuty is continuously evolving to help you protect your AWS environment.

These three new features add the ability to alert when access requests are identified as coming from penetration testing focused operating systems (such as Parrot and Pentoo Linux. Kali has been identified for a while) as they are unlikely to be legitimate traffic. The third new feature is a new policy violation detection policy that alerts you to any request in which AWS account root credentials are used. This one makes monitoring of your root account a tick box on the audit checklist as nobody should EVER be using their Root account to perform tasks (and if you are, please call us and we’ll help you fix it), so any requests originating from the root account should be treated as suspicious.

And that’s it for the AWS update for Friday the 15th of February 2019. Please keep an eye out for our weekly updates on the happenings within the AWS Eco-system and for the continuation of my upcoming blogs on new AWS products and features.

AWS Site-to-Site VPN and Transit Gateway

I recently implemented an AWS site-to-site VPN for a customer to connect their on-premise network to their newly deployed AWS account.

The requirement was network level connectivity from their on-premise network to their management VPC. Support of production VPC resources would be carried out from bastion hosts in the management VPC.

The setup of this was simple from an AWS perspective. With Cloud Formation we deployed a Customer Gateway (CGW) using the IP address of their on-premise firewall, created a Virtual Private Gateway (VPG) and then the VPN Gateway (VPN).


The on-premise configuration took a bit of research to get right but once configured correctly it worked as expected.

However, after deployment it was determined that an on-premise server needed to connect to a Production VPC resource.

We already have a VPC peer connection between the management and production VPCs, but a VPN will only route traffic to the VPC it is connected to and VPC peer connections are not ‘transitive’,


For a more detailed explanation of this see; https://docs.aws.amazon.com/vpc/latest/peering/invalid-peering-configurations.html

The solutions considered to allow access from on-premise to the production VPC resource were;

  • Create another VPN connection from the on-premise datacenter to the production VPC
  • Deploy an application proxy in the management VPC
  • Deploy the newly announced AWS Transit Gateway service

The customer wasn’t keen on adding VPN connections, as it would add configuration and complexity to the on-premise firewall, and we weren’t confident that an application proxy would work, so we decided on the new Transit Gateway service.

AWS Transit Gateway

AWS Transit Gateway was release at the end of 2018. It will allow our customer to connect their on-premise network to both of their AWS VPCs, and any future VPCs, without having to configure and support multiple VPN endpoints on their on-premise firewall and support multiple VPN gateways in AWS.


The steps to implement this were fairly simple, however Cloud Formation doesn’t cover all of the steps;

Deploy the ‘Transit Gateway’ and ‘Transit Gateway Attachment’ for the VPCs

This Cloud Formation template assumes that two VPCs already exist, and each have one subnet.
The VPC and subnet Ids need to be entered into the parameters section. The on-premise VPN endpoint is setup after the AWS VPN setup, but the IP address is added to the VPNAddress parameter. This also assumes a non BGP on-premise endpoint.

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Transit Gateway",
    "Parameters": {
        "VPCIdMgmt": {
            "Type": "String",
            "Description": "Management VPC Id",
            "Default": "vpc-12345678901234567"
        },
        "VPCIdProd": {
            "Type": "String",
            "Description": "Production VPC Id",
            "Default": "vpc-9876543210987654"
        },
        "MgmtPrivateAzASubnetId": {
            "Type": "String",
            "Description": "Az A Subnet in Mgmt VPC",
            "Default": "subnet-23456789012345678"
        },
        "ProdPrivateAzASubnetId": {
            "Type": "String",
            "Description": "Az A Subnet in Prod VPC",
            "Default": "subnet-34567890123456789"
        },
        "VPNAddress": {
            "Type": "String",
            "Description": "On-premise VPN endpoint",
            "Default": "201.65.1.1"
        }
    },
    "Resources": {
        "CustomerGateway": {
            "Type": "AWS::EC2::CustomerGateway",
            "Properties": {
                "Type": "ipsec.1",
                "BgpAsn": "65000",
                "IpAddress": {
                    "Ref": "VPNAddress"
                }
            }
        },
        "TransitGateway": {
            "Type": "AWS::EC2::TransitGateway",
            "Properties": {
                "AmazonSideAsn": 65001,
                "DefaultRouteTableAssociation": "enable",
                "DefaultRouteTablePropagation": "enable",
                "Description": "Transit Gateway",
                "DnsSupport": "enable",
                "VpnEcmpSupport": "enable"
            }
        },
        "TransitGatewayMgmtAttachment": {
            "Type": "AWS::EC2::TransitGatewayAttachment",
            "Properties": {
                "SubnetIds": [{
                    "Ref": "MgmtPrivateAzASubnetId"
                }],
                "TransitGatewayId": {
                    "Ref": "TransitGateway"
                },
                "VpcId": {
                    "Ref": "VPCIdMgmt"
                }
            }
        },
        "TransitGatewayProdAttachment": {
            "Type": "AWS::EC2::TransitGatewayAttachment",
            "Properties": {
                "SubnetIds": [{
                    "Ref": "ProdPrivateAzASubnetId"
                }],
                "TransitGatewayId": {
                    "Ref": "TransitGateway"
                },
                "VpcId": {
                    "Ref": "VPCIdProd"
                }
            }
        }
    },
    "Outputs": {
        "CustomerGateway": {
            "Description": "CustomerGateway Id",
            "Value": {
                "Ref": "CustomerGateway"
            },
            "Export": {
                "Name": "TransitGateway-CustomerGatewayId"
            }
        },
        "TransitGateway": {
            "Description": "TransitGateway Id",
            "Value": {
                "Ref": "TransitGateway"
            },
            "Export": {
                "Name": "TransitGateway-TransitGatewayId"
            }
        }
    }
}

After the Cloud Formation stack is deployed, the ‘Outputs’ section will list the CustomerGatewayId and TransitGatewayId, these are needed in the next steps

Create the site to site VPN

This step is completed in the AWS CLI as Cloud Formation doesn’t support it yet. Change customer-gateway-id and transit-gateway-id to the values in the output section of the Cloud Formation stack, or look it up in the AWS console. 

aws ec2 create-vpn-connection --customer-gateway-id cgw-045678901234567890 
--transit-gateway-id tgw-56789012345678901  --type ipsec.1 --options "{\"StaticRoutesOnly\":true}"

Create the VPN Transit Gateway route

The attached VPCs have had routes added by default, but as we are using non-BGP on-premise endpoint, the VPN needs routes specifically added.

The route we are adding here is the CIDR of the on-premise network e.g. 172.31.0.0/16

Get the Id of the ‘Transit Gateway Route Table’ and VPN’s ‘Transit Gateway Attachment Id’ from the AWS console under ‘Transit Gateway Route Tables’ and ‘Transit Gateway Attachments’

aws ec2 create-transit-gateway-route --destination-cidr-block 172.31.0.0/16 
--transit-gateway-route-table-id tgw-rtb-67890123456789012 --transit-gateway-attachment-id tgw-attach-7890123456789012

Configure VPC subnet routing

The routing that you will see configured on the Transit Gateway is only used within the Transit Gateway itself. So we now need to manually add routes to VPC subnets that you want to use the VPN.

In our case we are leaving VPC-VPC traffic to use the VPC peer, and only adding an on-premise network to the subnet routes.

Get the Transit Gateway Id from the Cloud Formation template output, and get the route VPC subnet’s route table Id;

aws ec2 create-route --route-table-id rtb-89012345678901234 --destination-cidr-block 172.31.0.0/16 --transit-gateway-id tgw-56789012345678901

Summary

Using a Transit Gateway can make site-to-site VPNs simpler and less messy by allowing a single VPN connection to AWS that can reach more than one VPC.

One important limitation is that Transit Gateway doesn’t yet support security groups.

If you use security groups over VPC peer connections, and switch from VPC peer connections to Transit Gateway, you will see your security groups become listed as ‘stale’, you would need to re-add them as IP based rules.

The Transit Gateway FAQs states the following;

  •  Q: Which Amazon VPC features are not supported in the first release?
  • A: Security Group Referencing on Amazon VPC is not supported at launch. Spoke Amazon VPCs cannot reference security groups in other spokes connected to the same AWS Transit Gateway. 

This implies that it will be supported in the future, but for now resource access between VPCs using security groups must remain over VPC Peer connections.

More information on Transit Gateway
https://docs.aws.amazon.com/vpc/latest/tgw/working-with-transit-gateways.html

Weekly AWS update: Friday 8th February 2019

DEVELOPERS, DEVELOPERS, DEVELOPERS… oh wait, wrong cloud. Regardless of who said those words, this week has been a busy one for our friends over at Amazon Web Services with a host of new products and features that are sure to delight the developers among us. This article continues the weekly series we are doing this year to help customers with a brief overview of the happenings within the AWS world over the last week. This is to try and help surface some of the more important announcements. This is not meant to be an exhaustive list of all the updates and changes to the AWS eco-system. It’s simply a summary of changes that might have an impact on the business and trends we at Kloud are seeing within the industry. As always, if you would like to talk to somebody about how you might be able to leverage some of these new technologies and services, please feel free to reach out using the contact link at the top of the page.

The key take away’s from this week are:

  • AWS X-Ray SDK for .NET Core is Now Generally Available
  • Amazon SNS Message Filtering Adds Support for Multiple String Values in Blacklist Matching
  • Develop and Test AWS Step Functions Workflows Locally
  • Amazon DynamoDB Local Adds Support for Transactional APIs, On-Demand Capacity Mode, and 20 GSIs
  • Amazon Corretto is Now Generally Available

AWS X-Ray SDK for .NET Core is Now Generally Available

For those who aren’t aware or who haven’t had the chance to leverage it yet, AWS X-Ray is an analysis and debugging tool that helps developers analyse and debug production, distributed applications, such as those built using a microservices architecture. At its core, it helps developers and support staff maintain a level of visibility of individual requests as they traverse multiple interconnected micro services resulting in quicker analysis and faster issue resolution. This week release brings X-Ray integration to .NET Core functions and services. As stated in the announcement article (available here) ” You can use AWS X-Ray to view a map of your applications and its services in development and in production. Your applications can be simple three-tier applications to complex micro services consisting of thousands of services such as built using AWS Lambda.

You can get the AWS X-Ray SDK for .NET Core from the X-Ray GitHub repository

Amazon SNS Message Filtering Adds Support for Multiple String Values in Blacklist Matching

Next on the list is the addition of support for multiple string values in blacklist matching within Amazon Simple Notification Service. Amazon SNS message filtering allows you to leverage AWS SNS to perform messaging filtering across your pub/sub solution without having to handle the logic within your applications infrastructure, reducing operational complexity and cost. To date, you’ve been able to match messages based on string white and black listing as well as string prefix and numerical matching. This new addition adds the ability to use multiple string values within your blacklisting operators further increasing the flexibility of the SNS message filtering service. This new addition should allow more customers to transition their existing EC2 or Lambda hosted filtering logic into AWS SNS, further reducing their operational footprint. Further information on the new feature release can be found here and detailed instruction on getting started with messaging filtering is available within the SNS developer guide.

Develop and Test AWS Step Functions Workflows Locally

One of the issues with migrating logic and application features into AWS native services is that you are then required to conduct all of your development and testing activities while connected to the cloud. Well, with the announcement made earlier this week AWS Step functions can now be developed and tested on your local development machine through the use of the new AWS Step Functions local. Quoting from the official feature announcement (available here) “AWS Step Functions Local is a downloadable version of Step Functions that lets you develop and test applications using a version of Step Functions running in your own development environment. Using the service locally rather than over the Internet can be faster in some situations, save on Step Functions state transitions, and allow you to easily enforce sandbox restrictions.” This means that I can finally work on my step functions when I’m flying around the country. It’s available now for anybody to get started with in both JAR and Docker versions ready for download. I hope to have an article out shortly with my first impressions of the tool set.

Amazon DynamoDB Local Adds Support for Transactional APIs, On-Demand Capacity Mode, and 20 GSIs

But that’s not all for development of cloud resources on a local machine…. Not at all. AWS have also announced and update to DynamoDB Local with the addition of several new features including:

  • Transactional API’s
  • On-Demand Capacity Mode
  • As many as 20 Global Secondary Indexes per table.

DynamoDB Local has been around for a while now and has been a part of my development toolkit for almost as long. With the addition of these new features, I can now test and validate additional aspects of my DynamoDB tables without having to interrupt my workflow and test it in a cloud environment. This includes simulating On-Demand behaviours and GSI focused operations. You can be sure I’m going to be giving this a good run through in the coming weeks and will post an update with my experience. For those who want to get started for themselves, the link is available on the Setting up DynamoDB Local page.

Amazon Corretto is Now Generally Available

And finally, for this week’s roundup, is yet another developer update with the announcement that AWS Corretto has reached General Availability after being in preview since its original announcement back in November of last year. In case you didn’t hear the original announcement, Amazon Corretto is a no-cost, multiplatform, production-ready distribution of the Open Java Development Kit (OpenJDK). As stated in the official announcement (available here), there are far to many updates to be listed in a single article sufficed to say that it’s been updated to OpenJDK version 8u202 and that a more comprehensive list of platforms is now supported, including Amazon Linux 2 and an official Docker image. AWS also note that they are currently working on Corretto 11 corresponding to OpenJDK 11 and will release it in time for testing before April of this year.

And that’s it for the AWS update for Friday the 8th of February 2019. Please keep an eye out for our weekly updates on the happenings within the AWS eco-system and for the continuation of my upcoming blog. Further articles will include my expereinces testing out the new updates to DynamoDB Local and Step Functions Local.

Follow Us!

Kloud Solutions Blog - Follow Us!