Experiences with the new AWS Application Load Balancer

Originally posted on Andrew’s blog @ cloudconsultancy.info

Summary

Recently I had an opportunity to test drive AWS Application load balancer as my client had a requirement for making their websocket application fault tolerant. The implementation was complete windows stack and utilised ADFS 2.0 for SAML authentication however this should not affect other people’s implementation.

The AWS Application load balancer is a fairly new feature which provides layer 7 load balancing and support for HTTP/2 as well as websockets. In this blog post I will include examples of the configuration that I used to implement as well is some of the troubleshooting steps I needed to resolve.

The application load balancer is an independent AWS resource from classic ELB and is defined as aws elbv2 with a number of different properties.

Benefits of Application Load Balancer include:

  • Content based routing, ie route /store to a different set of instances from /apiv2
  • Support for websocket
  • Support for HTTP/2 over HTTPS only (much larger throughput as it’s a single stream multiplexed meaning it’s great for mobile and other high latency apps)
  • Cheaper cost than classic, roughly 10% cheaper than traditional.
  • Cross-zone load balancing is always enabled for ALB.

Some changes that I’ve noticed:

  • Load balancing algorithm used for application load balancer is currently round robin.
  • Cross-zone load balancing is always enabled for an Application Load Balancer and is disabled by default for a Classic Load Balancer.
  • With an Application Load Balancer, the idle timeout value applies only to front-end connections and not the LB-> server connection and this prevents the LB cycling the connection.
  • Application Load balancer is exactly that and performs at Layer 7, so if you want to perform SSL bridge use Classic load balancer with TCP and configure SSL certs on your server endpoint.
  • cookie-expiration-period value of 0 is not supported to defer session timeout to the application. I ended up having to configure the stickiness.lb_cookie.duration_seconds value. I’d suggest making this 1 minute longer than application session timeout, in my example a value of 1860.
  • The X-Forwarded-For parameter is still supported and should be utilised if you need to track client IP addresses, in particular useful if going through a proxy server.

For more detailed information from AWS see http://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html.

Importing SSL Certificate into AWS – Windows

(http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_server-certs.html )

  1. Convert the existing pkcs key into .pem format for AWS

You’ll need openssl for this, the pfx and the password for the SSL certificate.

I like to use chocolatey as my Windows package manager, similar to yum or apt-get for Windows, which is a saviour for downloading package and managing dependencies in order to support automation, but enough of that, check it out @ https://chocolatey.org/

Once choco is installed I simply execute the following from an elevated command prompt.

“choco install openssl.light”

Thereafter I run the following two commands which breaks out the private and public keys (during which you’ll be prompted for the password):

openssl pkcs12 -in keyStore.pfx -out SomePrivateKey.key –nodes –nocerts

openssl pkcs12 -in keyStore.pfx -out SomePublic.cert –nodes –nocerts

NB: I’ve found that sometimes copy and paste doesn’t work when trying to convert keys.

  1. Next you’ll need to also break out the trust chain into one contiguous file, like the following.
-----BEGIN CERTIFICATE-----

Intermediate certificate 2

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

Intermediate certificate 1

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

Optional: Root certificate

-----END CERTIFICATE-----

Save the file for future use,

thawte_trust_chain.txt

Example attached above is for a Thawte trust chain with the following properties

“thawte Primary Root CA” Thumbprint ‎91 c6 d6 ee 3e 8a c8 63 84 e5 48 c2 99 29 5c 75 6c 81 7b 81

With intermediate

“thawte SSL CA - G2” Thumbprint ‎2e a7 1c 36 7d 17 8c 84 3f d2 1d b4 fd b6 30 ba 54 a2 0d c5

Ordinarily you’ll only have a root and intermediate CA, although sometimes there will be second intermediary CA.

Ensure that your certificates are base 64 encoded when you export them.

  1. Finally execute the following after authenticating to the AWS CLI (v1.11.14+ to support aws elbv2 function) then run “aws configure” applying your access and secret keys, configuring region and format type. Please note that this includes some of the above elements including trust chain and public and private keys.

If you get the error as below

A client error (MalformedCertificate) occurred when calling the UploadServerCertificate operation: Unable to validate certificate chain. The certificate chain must start with the immediate signing certificate, followed by any intermediaries in order. The index within the chain of the invalid certificate is: 2”

Please check the contents of the original root and intermediate keys as they probably still have the headers and maybe some intermediate,

ie

Bag Attributes

localKeyID: 01 00 00 00

friendlyName: serviceSSL

subject=/C=AU/ST=New South Wales/L=Sydney/O=Some Company/OU=IT/CN=service.example.com

issuer=/C=US/O=thawte, Inc./CN=thawte SSL CA - G2

Bag Attributes

friendlyName: thawte

subject=/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA

issuer=/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA

AWS Application LB Configuration

Follow this gist with comments embedded. Comments provided based on some gotchas during configuration.

You should be now good to go, the load balancer takes a little while to warm up, however will be available within multiple availability zones.

If you have issues connecting to the ALB, validate connectivity direct to the server using curl.  Again chocolatey comes in handy “choco install curl”

Also double check your security group registered against the ALB and confirm NACLS.

WebServer configuration

You’ll need to import the SSL certificate into the local computer certificate store. Some of the third party issuing (Ensign, Thawte, etc) CAs may not have the intermediate CA within the computed trusted Root CAs store, especially if built in a network isolated from the internet, so make sure after installing the SSL certificate on the server that the trust chain is correct.

You won’t need to update local hosts file on the servers to point to the load balanced address.

Implementation using CNAMEs

In large enterprises where I’ve worked there have been long lead times associated with fairly simple DNS changes, which defeats some of the agility provided by cloud computing. A pattern I’ve often seen adopted is to use multiple CNAMEs to work around such lead times. Generally you’ll have a subdomain domain somewhere where the Ops team have more control over or shorter lead times. Within the target domain (Example.com) create a CNAME pointing to an address within the ops managed domain (aws.corp.internal) and have a CNAME created within that zone to point to the ALB address, ie

Service.example.com -> service.aws.corp.internal -> elbarn.region.elb.amazonaws.com

With this approach I can update service.aws.corp.internal to reflect a new service which I’ve built via a new ELB and avoid the enterprise change lead times associated with a change in .example.com.

Site Delivery with AWS CloudFront CDN

Nowadays, most companies are using some sort of a Content Delivery Network (CDN) to improve the performance and high availability of their sites, those include Azure CDN, CloudFlare, CloudFront, Varnish, and so on.

In this blog however, I will demonstrate how you can deliver your entire website through AWS’s CloudFront. This blog will not go through other CDN services. This blog also assumes you have knowledge of AWS services, DNS, and CDN.

What is CloudFront?

Amazon CloudFront is a global content delivery network (CDN) service that accelerates delivery of your websites, APIs, video content or other web assets. It integrates with other Amazon Web Services products to give developers and businesses an easy way to accelerate content to end users with no minimum usage commitments.

CloudFront delivers the contents of your websites through global datacentres known as “Edge Locations”.

Assuming the webserver is located in New York, and you’re accessing the website from Melbourne, then the latency will be greater than someone trying to access the website from London.

CloudFront’s Edge Locations will serve the content of a website depending on location. That is, if you’re trying to access a New York based website from Melbourne, you will be directed to the closest Edge Location available for users from Australia. There are two Edge Locations in Australia, one in Melbourne and one in Sydney.

How CloudFront delivers content?

Please note that contents aren’t delivered from the first request (whether you use CloudFront or any other CDN solution). That is, the first user who accesses a page from Melbourne (first request), the contents of the page hadn’t been cached (yet) and it will be fetched from the webserver. The second user who accesses the website (second request), will get the contents from the Edge Location.

Here’s how:

drawing1

The main features of CloudFront are:

  • Edge Location: This is the location where content will be cached. This is separate to an AWS Region or Availability Zone
  • Origin: This is the origin of all the files that the CDN will distribute. This can be either an S3 bucket, an EC2 instance, and ELB or Route53.
  • Distribution: The name given the CDN which consists of collection of Edge Location
  • Web Distribution: Typically used for Websites.
  • RTMP: Typically used for Media Streaming (Not covered in this blog).

In this blog, we will be covering “Web Distribution”.

There are multiple ways to “define” your origin. You could either upload your contents to an S3 bucket, or let CloudFront cache objects from your webserver.

I advise you to keep the same naming conventions you have previously used for your website.

There’s no real difference between choosing an S3 bucket, or your webserver to deliver contents, or even an Elastic Load Balancer.

What matters however, should you choose for CloudFront to cache objects from your origin, you may need to change your hostname. Alternatively, if your DNS registrar allows it, you can make an APEX DNS change.

Before we dive deep into setting up and configuring CloudFront, know that it is fairly a very simple process, and we will be using CloudFront GUI to achieve this.

Alternatively you can use other third party tools like S3 Browser, Cloudberry Explorer, or even CloudFormation if you have several websites you’d like to enable CloudFront for. These tools are excluded from this blog.

Setup CloudFront with S3

Although I do not recommend this approach because S3 is designed as a storage service and not a delivery (content) service, under load it will not provide you with optimum performance.

  1. Create your bucket
  2. Upload your files
  3. Make your content public (this is achieved through Permissions. Simply choose grantee “everyone”.

Configuring CloudFront with S3

As aforementioned, configuring CloudFront is very straightforward. Here are the steps for doing so.

I will explain the different settings at the end of each image.

Choose your origin (domain name, S3 bucket, ELB…). If you have an S3 bucket or an ELB already configured, they will show in the drop down menu.

You could simply follow the selected options in the image for optimal performance and configuration of your CloudFront distribution.

5-cdn

  • Origin path: This is optional and usually not needed to be specified. This is basically a directory in your bucket in which you’re telling CloudFront to request the content from.
  • Origin ID: This is automatically populated, but you can change it. Its only function is for you to distinguish origins if you have multiple origins in the same distributions.
  • Restrict Bucket Access: This is for users to access your CloudFront URL e.g. 123456.cloudfront.net rather than the S3 URL.
  • Origin Access Identity: This is required if you want your users to always access your Amazon S3 content using CloudFront URLs. You can use the same Access Identity for all your distributions. In fact, it is recommended you do so to make life simpler.
  • Grant Read Permissions on Bucket: This applies on the “Origin Access Identity” so CloudFront can access objects in your Amazon S3 bucket. This is automatically applied.
  • Viewer Protocol Policy: This is to specify how users should access your origin domain name. If you have a website that accepts both HTTP and HTTPS, then choose that. CloudFront will fetch the contents based on this viewer policy. That is, if a user typed in http://iknowtech.com.au then CloudFront will fetch content over HTTP. If HTTPS is used, then CloudFront will fetch contents over HTTPS. If your website only accepts HTTPS, then choose that option.
  •  Allowed HTTP Methods: This basically is used for commerce websites or websites with login forms which requires data from end users for better performance. You can keep it default on “Get, Head”. Nevertheless, make sure to configure your webserver to handle “Delete” appropriately, otherwise users might be able to delete contents.
  • Cached HTTP Methods: You will have an additional “Options”, if you choose the specified HTTP Methods shown above. This is to specify the methods in which you want CloudFront to do caching.

In the second part of the configuration:

6-cdn

  • Price Class: This is to specify which regions of available Edge Locations you want to “serve” your website from.
  • AWS WAF Web ACL: Web Application Firewall (WAF) is a set of ACL rules which you create to protect your website from attacks, e.g. SQL Injections etc. I highlighted that on purpose as there will be another blog for that alone.
  • CNAME: If you don’t want to use CloudFront’s URL e.g.123456.cloudfront.net, and instead you want to use your own domain, then specifying a CNAME is a good idea, and you can specify up to 100 CNAMEs. Nevertheless, there may be a catch. Most DNS hosting services may not allow you to edit the APEX zone of your records. And if you create a CNAME for http://www.domain.com to point to 123456.cloudfront.net, then any requests coming from htttp://domain.com will not be going through CloudFront. And if you have a redirection set up in your webserver, for any request coming from http://www.domain.com to go http://domain.com then there’s no point configuring CloudFront.
  • SSL Certificates: You could either use CloudFront’s certificate, and it is a wildcard certificate of *.cloudfront.net, or you can request to use your own domain’s certificate.
  • Supported HTTP Versions: What you need to know is that CloudFront always forwards requests to the origin using HTTP/1.1. This also is based on the viewer policy, most modern websites support all HTTP methods shown above. HTTP/2 is usually faster. Read more here for more info on HTTP/2 support. In theory this sounds ideal, technically however, nothing much is happening in the backend of CloudFront.
  • Logging: Choose to have logging on. Logs are saved in an S3 bucket.
  • Bucket for Logs: Specify the bucket you want to save the logs onto.
  • Log Prefix: Choose a prefix for your logs. I like to include the domain name for each log of each domain.
  • Cookie logging: Not quite important to have it turned on.
  • Enable IPv6: You can have IPv6 enabled, however as of this writing this is still being deployed.
  • Distribution State: You can choose to deploy/create your CloudFront distribution with either in an enabled or disabled state.

Once you’ve completed the steps above, click on “Create Distribution”. It might take anywhere from 10 to 30 minutes for CloudFront to be deployed. Average waiting time is 15 minutes.

Setup and Configure your DNS Records

Once the Distribution is started, head over to “Distributions” in CloudFront, then click on the Distribution ID, take note of the domain name: d2hpa1xghjdn8b.cloudfront.net.

Head over to your DNS records and add the CNAME (or CNAMEs) you have specified in earlier steps to point to d2hpa1xghjdn8b.cloudfront.net.

Do not wait until the Distribution is complete, add the DNS records while the Distribution is being deployed. This will at least give time for your DNS to propagate, since CloudFront takes anywhere between 10 to 30 minutes to be deployed.

17-dns

18-dns

If you’re delivering the website through an ELB, then you can use the ELB’s CNAME to point your site to it.

Here’s what will appear eventually once the CloudFront Distribution is complete. Notice the URLs: http://d80u8wk4w5p58.cloudfront.net/nasa_blue_marble.jpg (I may remove this link in the future).

8-cdn

You can also access it via: http://cdn.iknowtech.com.au/nasa_blue_marble.jpg (I may also remove this link in the future).

10-cdn

Configuring CloudFront with Custom Origin

Creating a CloudFront Distribution based on Custom Origin, that is to allow CloudFront to cache directly from your domain, is pretty much the same as above, with some differences, as shown below. Every other setting is the same as above.

9-cdn

The changes, as you can see relate to SSL Protocols, HTTP and HTTPS ports.

  • Original SSL Protocols: This is to specify which SSL Protocols CloudFront will use when establishing a connection to your origin. If you don’t have SSLv3, keep it disabled. If you do, and your origin does not support v1, v1.1. and v1.2, then choose SSLv3.
  • Origin Protocol Policy: This is the same as Viewer Protocol Policy discussed above. If you choose “Match Viewer” then it will work with both HTTP and HTTPS. Obviously, it also depends on how your website is set up.
  • HTTP and HTTPS ports: Leave default ports.

Configure CloudFront with WordPress

If you have a WordPress page, it is most probably the easiest way to configure CloudFront for WordPress. Through the use of plugins, you can change the hostname.

  1. Install the W3 Total Cache plugin in your WordPress page.
  2. Enable CDN and choose CloudFront. This is found in the “General Settings” tab of the W3 plugin.11-cdn
  3. While scrolling down to CDN, you may enable other forms of “caching” found in settings.
  4. After saving your changes, click on “CDN” tab of the W3 plugin.
  5. Key in a the required information. I suggest you create an IAM user with permission to CloudFront to be used here.

Note that I used cdn2.iknowtech.com.au because I had already used cdn.iknowtech.com.au. CloudFront will detect this setting and give you an error if you try and use the same CNAME.

12-cdn

Once your settings are saved, here’s how it’ll look.

Note the URLs: http://d2hpa1xghjdn8b.cloudfront.net (I may remove this link in the future).13-cdn

You can also access it via: http://cdn2.iknowtech.com.au (I may also remove this link the future).

14-cdn

 

 

To make sure your CDN is working, you can perform some test, using any of the followings: gtmetrix.com, pingdom.com or webpagetest.org .

Here are the results from gtmetrix, tested for iknowtech.com.au

15-cdn

The same result for cdn2.iknowtech.com.au.

Notice the page load time after the second request.

16-cdn

And that’s it. All you need to know about to create a CloudFront Distribution.

Conclusion

CloudFront is definitely a product to use if you’re looking for CDN. It is true you have many other CDN products out there, but CloudFront is one of the easiest, highly-available CDNs in the market.

Before you actually utilise CloudFront or any other CDN solutions, just be mindful of your hostnames. You need your primary domain or record to be cached.

I hope you found this blog informative. Please feel free to post your comments below.

Thanks for reading.

 

Ubuntu security hardening for the cloud.

Hardening Ubuntu Server Security For Use in the Cloud

The following describes a few simple means of improving Ubuntu Server security for use in the cloud. Many of the optimizations discussed below apply equally to other Linux based distribution although the commands and settings will vary somewhat.

Azure cloud specific recommendations

  1. Use private key and certificate based SSH authentication exclusively and never use passwords.
  2. Never employ common usernames such as root , admin or administrator.
  3. Change the default public SSH port away from 22.

AWS cloud specific recommendations

AWS makes available a small list of recommendation for securing Linux in their cloud security whitepaper.

Ubuntu / Linux specific recommendations

1. Disable the use of all insecure protocols (FTP, Telnet, RSH and HTTP) and replace them with their encrypted counterparts such as sFTP, SSH, SCP and HTTPS

yum erase inetd xinetd ypserv tftp-server telnet-server rsh-server

2. Uninstall all unnecessary packages

dpkg --get-selections | grep -v deinstall
dpkg --get-selections | grep postgres
yum remove packageName

For more information: http://askubuntu.com/questions/17823/how-to-list-all-installed-packages

3. Run the most recent kernel version available for your distribution

For more information: https://wiki.ubuntu.com/Kernel/LTSEnablementStack

4. Disable root SSH shell access

Open the following file…

sudo vim /etc/ssh/sshd_config

… then change the following value to no.

PermitRootLogin yes

For more information: http://askubuntu.com/questions/27559/how-do-i-disable-remote-ssh-login-as-root-from-a-server

5. Grant shell access to as few users as possible and limit their permissions

Limiting shell access is an important means of securing a system. Shell access is inherently dangerous because of the risk of unlawfully privilege escalations as with any operating systems, however stolen credentials are a concern too.

Open the following file…

sudo vim /etc/ssh/sshd_config

… then add an entry for each user to be allowed.

AllowUsers jim,tom,sally

For more information: http://www.cyberciti.biz/faq/howto-limit-what-users-can-log-onto-system-via-ssh/

6. Limit or change the IP addresses SSH listens on

Open the following file…

sudo vim /etc/ssh/sshd_config

… then add the following.

ListenAddress <IP ADDRESS>

For more information:

http://askubuntu.com/questions/82280/how-do-i-get-ssh-to-listen-on-a-new-ip-without-restarting-the-machine

7. Restrict all forms of access to the host by individual IPs or address ranges

TCP wrapper based access lists can be included in the following files.

/etc/hosts.allow
/etc/hosts.deny

Note: Any changes to your hosts.allow and hosts.deny files take immediate effect, no restarts are needed.

Patterns

ALL : 123.12.

Would match all hosts in the 123.12.0.0 network.

ALL : 192.168.0.1/255.255.255.0

An IP address and subnet mask can be used in a rule.

sshd : /etc/sshd.deny

If the client list begins with a slash (/), it is treated as a filename. In the above rule, TCP wrappers looks up the file sshd.deny for all SSH connections.

sshd : ALL EXCEPT 192.168.0.15

This will allow SSH connections from only the machine with IP address 192.168.0.15 and block all other connection attemps. You can use the options allow or deny to allow or restrict access on a per client basis in either of the files.

in.telnetd : 192.168.5.5 : deny
in.telnetd : 192.168.5.6 : allow

Warning: While restricting system shell access by IP address be very careful not to loose access to the system by locking the administrative user out!

For more information: https://debian-administration.org/article/87/Keeping_SSH_access_secure

8. Check listening network ports

Check listening ports and uninstall or disable all unessential or insecure protocols and deamons.

netstat -tulpn

9. Install Fail2ban

Fail2ban is a means of dealing with unwanted system access attempts over any protocol against a Linux host. It uses rule sets to automate variable length IP banning sources of configurable activity patterns such as SPAM, (D)DOS or brute force attacks.

“Fail2Ban is an intrusion prevention software framework that protects computer servers from brute-force attacks. Written in the Python programming language, it is able to run on POSIX systems that have an interface to a packet-control system or firewall installed locally, for example, iptables or TCP Wrapper.” – Wikipedia

For more information: https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-14-04

10. Improve the robustness of TCP/IP

Add the following to harden your networking configuration…

10-network-security.conf

… such as

sudo vim /etc/sysctl.d/10-network-security.conf
Ignore ICMP broadcast requests
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Disable source packet routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0 
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0

# Ignore send redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Block SYN attacks
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 5

# Log Martians
net.ipv4.conf.all.log_martians = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Ignore ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0 
net.ipv6.conf.default.accept_redirects = 0

# Ignore Directed pings
net.ipv4.icmp_echo_ignore_all = 1

And load the new rules as follows.

service procps start

For more information: https://blog.mattbrock.co.uk/hardening-the-security-on-ubuntu-server-14-04/

11. If you are serving web traffic install mod-security

Web application firewalls can be helpful in warning of and fending off a range of attack vectors including SQL injection, (D)DOS, cross-site scripting (XSS) and many others.

“ModSecurity is an open source, cross-platform web application firewall (WAF) module. Known as the “Swiss Army Knife” of WAFs, it enables web application defenders to gain visibility into HTTP(S) traffic and provides a power rules language and API to implement advanced protections.”

For more information: https://modsecurity.org/

12. Install a firewall such as IPtables

IPtables is a highlight configurable and very powerful Linux forewall which has a great deal to offer in terms of bolstering hosts based security.

iptables is a user-space application program that allows a system administrator to configure the tables provided by the Linux kernel firewall (implemented as different Netfilter modules) and the chains and rules it stores.” – Wikipedia.

For more information: https://help.ubuntu.com/community/IptablesHowTo

13. Keep all packages up to date at all times and install security updates as soon as possible

 sudo apt-get update        # Fetches the list of available updates
 sudo apt-get upgrade       # Strictly upgrades the current packages
 sudo apt-get dist-upgrade  # Installs updates (new ones)

14. Install multifactor authentication for shell access

Nowadays it’s possible to use multi-factor authentication for shell access thanks to Google Authenticator.

For more information: https://www.digitalocean.com/community/tutorials/how-to-set-up-multi-factor-authentication-for-ssh-on-ubuntu-14-04

15. Add a second level of authentication behind every web based login page

Stolen passwords are a common problem whether as a result of a vulnerable web application, an SQL injection, a compromised end user computer or something else altogether adding a second layer of protection using .htaccess authentication with credentials stored on the filesystem not in a database is great added security.

For more information: http://stackoverflow.com/questions/6441578/how-secure-is-htaccess-password-protection

Viewing AWS CloudFormation and bootstrap logs in CloudWatch

Mature cloud platforms such as AWS and Azure have simplified infrastructure provisioning with toolsets such as CloudFormation and Azure Resource Manager (ARM) to provide an easy way to create and manage a collection of related infrastructure resources. Both tool sets allow developers and system administrators to use JavaScript Object Notation (JSON) to specify resources to provision, as well as provide the means to bootstrap systems, effectively allowing for single click fully configured environment deployments.

While these toolsets are an excellent means to prevent RSI from performing repetitive monotonous tasks, the initial writing and testing of templates and scripts can be incredibly time consuming. Troubleshooting and debugging bootstrap scripts usually involves logging into hosts and checking log files. These hosts are often behind firewalls, resulting in the need to use jump hosts which may be MFA integrated, all resulting in a reduced appetite for infrastructure as code.

One of my favourite things about Azure is the ability to watch the ARM provisioning and host bootstrapping process through the console. Unless there’s a need to rerun a script on a host and watch it in real time, troubleshooting the deployment failure can be performed by viewing the ARM deployment history or viewing the relevant Virtual Machine Extension. Examples can be seen below:

This screenshot shows the ARM resources have been deployed successfully.

This screenshot shows the ARM resources have been deployed successfully.

This screenshot shows the DSC extension status, with more deployment details on the right pane.

This screenshot shows the DSC extension status, with more deployment details on the right pane.

While this seems simple enough in Azure, I found it a little less straight forward in AWS. Like Azure, bootstrap logs for the instance reside on the host itself, however the logs aren’t shown in the console by default. Although there’s a blog post on AWS to view CloudFormation logs in CloudWatch, it was tailored to Linux instances. Keen for a similar experience to Azure, I decided to put together the following instructional to have bootstrap logs appear in CloudWatch.

To enable CloudWatch for instances dynamically, the first step is to create an IAM role that can be attached to EC2 instances when they’re launched, providing them with access to CloudWatch. The following JSON code shows a sample policy I’ve used to define my IAM role.

The next task is to create a script that can be used at the start of the bootstrap process to dynamically enable the CloudWatch plugin on the EC2 instance. The plugin is disabled by default and when enabled, requires a restart of the EC2Config service. I used the following script:

It’s worth noting that the EC2Config service is set to recover by default and therefore starts by itself after the process is killed.

Now that we’ve got a script to enable the CloudWatch plugin, we need to change the default CloudWatch config file on the host prior to enabling the CloudWatch plugin. The default CloudWatch config file is AWS.EC2.Windows.CloudWatch.json and contains details of all the logs that should be monitored as well as defining CloudWatch log groups and log streams. Because there’s a considerable number of changes made to the default file to achieve the desired result, I prefer to create and store a customised version of the file in S3. As part of the bootstrap process, I download it to the host and place it in the default location. My customised CloudWatch config file looks like the following:

Let’s take a closer look at what’s happening here. The first three components are windows event logs I’m choosing to monitor:

You’ll notice I’ve included the Desired State Configuration (DSC) event logs, as DSC is my preferred configuration management tool of choice when it comes to Windows. When defining a windows event log, a level needs to be specified, indicating the verbosity of the output. The values are as follows:

1 – Only error messages uploaded.
2 – Only warning messages uploaded.
4 – Only information messages uploaded.

You can add values together to include more than one type of message. For example, 3 means that error messages (1) and warning messages (2) get uploaded. A value of 7 means that error messages (1), warning messages (2), and information messages (4) get uploaded. For those familiar with Linux permissions, this probably looks very familiar! 🙂

To monitor other windows event logs, you can create additional components in the JSON template. The value of “LogName” can be found by viewing the properties of the event log file, as shown below:

img_57c431da7dfb4

The next two components monitor the two logs that are relevant to the bootstrap process:

Once again, a lot of this is self explanatory. The “LogDirectoryPath” specifies the absolute directory path to the relevant log file, and the filter specifies the log filename to be monitored. The tricky thing here was getting the “TimeStampFormat” parameter correct. I used this article on MSDN plus trial and error to work this out. Additionally, it’s worth noting that cfn-init.log’s timestamp is the local time of the EC2 instance, while EC2ConfigLog.txt takes on UTC time. Getting this right ensures you have the correct timestamps in CloudWatch.

Next, we need to define the log groups in CloudWatch that will hold the log streams. I’ve got three separate Log Groups defined:

You’ll also notice that the Log Streams are named after the instance ID. Each instance that is launched will create a separate log stream in each log group that can be identified by its instance ID.

Finally, the flows are defined:

This section specifies which logs are assigned to which Log Group. I’ve put all the WindowsEventLogs in a single Log Group, as it’s easy to search based on the event log name. Not as easy to differentiate between cfn-init.log and EC2ConfigLog.txt entries, so I’ve split them out.

So how do we get this customised CloudWatch config file into place? My preferred method is to upload the file with the set-cloudwatch.ps1 script to a bucket in S3, then pull them down and run the PowerShell script as part of the bootstrap process. I’ve included a subset of my standard cloudformation template below, showing the monitoring config key that’s part of the ConfigSet.

What does this look like in the end? Here we can see the log groups specified have been created:

img_57c4d1ddcd273

If we drill further down into the cfninit-Log-Group, we can see the instance ID of the recently provisioned host. Finally, if we click on the Instance ID, we can see the cfn-init.log file in all its glory. Yippie!

img_57c4d5c4c3b89

Hummm, looks like my provisioning failed because a file is non-existent. Bootstrap monitoring has clearly served its purpose! Now all that’s left to do is to teardown the infrastructure, remediate the issue, and reprovision!

The next step to reducing the amount of repetitive tasks in the infrastructure as code development process is a custom pipeline to orchestrate the provisioning and teardown workflow… More on that in another blog!

Migrating resources from AWS to Microsoft Azure

Kloud receives a lot of communications in relation to the work we do and the content we publish on our blog. My colleague Hugh Badini recently published a blog about Azure deployment models from which we received the following legitimate follow up question…

So, Murali, thanks for letting us know you’d like to know more about this… consider this blog a starting point :).

Firstly though…

this topic (inter-cloud migrations), as you might guess, isn’t easily captured in a single blog post, nor, realistically in a series, so what I’m going to do here is provide some basics to consider. I may not answer your specific scenario but hopefully provide some guidance on approach.

Every cloud has a silver lining

The good news is that if you’re already operating in a cloud environment then you have likely had to deal with many of the fundamental differences between traditional application hosting and architecture and that of cloud platforms.

You will have dealt with how you ensure availability of your application(s) across outages; dealing with spikes in traffic via use of elastic compute resources; and will have come to recognise that is many ways, Infrastructure-as-a-Service (IaaS) in the cloud has many similarities to the way you’ve always done things on-prem (such as backups).

Clearly you have less of a challenge in approaching a move to another cloud provider.

Where to start

When we talk about moving from AWS to Azure we need to consider a range of things – let’s take a look at some key ones.

Understand what’s the same and what’s different

Both platforms have very similar offerings, and Microsoft provides many great resources to help those utilising AWS to build an understanding of which services in AWS map to which services in Azure. As you can see the majority of AWS’ services have an equivalent in Azure.

Microsoft’s Channel 9 is also a good place to start to learn about the similarities, with there being no better place than the Microsoft Azure for Amazon AWS Professional video series.

So, at a platform level, we are pretty well covered, but…

the one item to be wary of in planning any move of an existing application is how it has been developed. If we are moving components from, say, an EC2 VM environment to an Azure VM environment then we will probably have less work to do as we can build our Azure VM as we like (yes, as we know, even Linux!) and install whatever languages, frameworks or runtimes we need.

If, however, we are considering moving an application from a more Platform-as-a-Service capability such AWS Lambda we need to look at the programming model required to move its equivalent in Azure – Azure Functions. While AWS Lambda and Azure Functions are functionally the same (no pun intended) we cannot simply take our Lambda code and drop it into an Azure Function and have it work. It may not even make sense to utilise Azure Functions depending on what you are shifting.

It’s also important to consider the differences in the availability models in use today in AWS and Azure. AWS uses Availability Zones to help you manage the uptime of your application and it’s components. In Azure we manage availability at two levels – locally via Availability Sets and then geographically through use of Regions. As these models differ it’s an important area to consider for any migration.

Tools are good, but are no magic wand

Microsoft provides a way to migrate AWS EC2 instances to Azure using Azure Site Recovery (ASR) and while there are many tools for on-prem to cloud migrations and for multi-cloud management, they mostly steer away from actual migration between cloud providers.

Kloud specialises in assessing application readiness for cloud migrations (and then helping with the migration), and we’ve found inter-cloud migration is no different – understanding the integration points an application has and the SLAs it must meet are a big part of planning what your target cloud architecture will look like. Taking into consideration underlying platform services in use is also key as we can see from the previous section.

If you’re re-platforming an application you’ve built or maintain in-house, make sure to review your existing deployment processes to leverage features available to you for modern Continuous Deployment (CD) scenarios which are certainly a strength of Azure.

Data has a gravitational pull

The modern application world is entirely a data-driven one. One advantage to cloud platforms is the logically bottomless pit of storage you have at your disposal. This presents a challenge, though, when moving providers where you may have spent years building data stores containing Terabytes or Petabytes of data. How do you handle this when moving? There are a few strategies to consider:

  • Leave it where it is: you may decide that you don’t need all the data you have to be immediately available. Clearly this option requires you to continue to manage multiple clouds but may make economic sense.
  • Migrate via physical shipping: AWS provides Snowball as a way to extract data out of AWS without needing to pull it over a network connection. If your solution allows it you could ship your data out of AWS to a physical location, extract that data, and then prepare it for import into Azure, either over a network connection using ExpressRoute or through the Azure Import/Export service.
  • Migrate via logical transfer: you may have access to a service such as Equinix’s Cloud Exchange that allows you to provision inter-connects between cloud and other network providers. If so, you may consider using this as your migration enabler. Ensure you consider how much data you will transfer and what, if any, impact the data transfer might have on existing network services.

Outside of the above strategies on transferring of data, perhaps you can consider a staged migration where you only bring across chunks of data as required and potentially let older data expire over time. The type and use of data obviously impacts on which approach to take.

Clear as…

Hopefully this post has provided a bit more clarity around what you need to consider when migrating resources from AWS to Azure. What’s been your experience? Feel free to leave comments if you have feedback or recommendations based on the paths you’ve followed.

Happy dragon slaying!

AWS CloudFormation AWS::RDS::OptionGroup Unknown option: Mirroring

Amazon recently announced Multi-AZ support for SQL Server in Sydney, which provides high availability for SQL RDS instances using SQL Server mirroring technology. In an effort to make life simpler for myself, I figured I’d write a CloudFormation template for future provisioning requests, however it wasn’t as straight forward as I’d expected.

I began by trying to guess my way through the JSON resources, based on what I’d already knew for MySQL deployments. I figured the MultiAZ property was still relevant, so I hacked together a template and attempted to provision the stack, which failed, indicating the following error:

CREATE_FAILED        |  Invalid Parameter Combination: MultiAZ property cannot be used with SQL Server DB instances, use the Mirroring option in an option group associated with the DB instance instead.

The CloudFormation output clearly provides some guidance on the correct parameters required to enable mirroring in SQL Server. I had a bit of trouble tracking down documentation for the mirroring option, but after crawling the web for sample templates, I managed to put together the correct CloudFormation template, which can be seen below.

Excited and thinking I had finalised my template, I attempted to create the stack in ap-southeast-2 (Sydney, Australia), only for it to fail with a different error this time…

CREATE_FAILED        | Unknown option: Mirroring

Finding this output strange, I attempted to run the CloudFormation template in eu-west-1, which completed successfully.

With no options left, I decided to contact AWS Support who indicated that this is a known issue in the ap-southeast-2 region, which is also evident when attempting to create an option group in the GUI and the dropdown box is greyed out, as shown below.

What it should look like:

What it currently looks like:

The suggested workaround is to manually create the SQL RDS instance in the GUI which provides the mirroring option in the deployment wizard. Although the limitation is being treated with priority, there’s no ETA for resolution at the moment.

Hopefully this assists someone out there banging their head against the wall trying to get this to work!

Creating a simple nodejs API on AWS (including nginx)

On a recent project I was part of a team developing an AngularJS website with a C# ASP.NET backend API hosted in Azure.  It was a great project as I got to work with a bunch of new tools, but it got me wondering on how simple it could be to use a Javascript API instead.  That way the entire development stack would be written in Javascript.

And so a personal project was born.  To create a simple JS API and get it running in the cloud.

Getting started

So here goes, a quick guide on setting up a nodejs API using the express framework.

I’ll start by getting the environment running locally on my mac in 6 simple steps:

# 1. Create a directory for your application
$ mkdir [your_api_name]

# 2. Install Express (the -g will install it globally)
$ npm install express -g

# 3. Use the express generator as it makes life easier!
$ npm install express-generator -g

# 4. Create your project
$ express [your_project_name_here]

# 5. Install any missing dependencies
$ npm install

# 6. Start your API
$ npm start

That’s it. You now have a nodejs API running locally on your development environment!

To test it, and prove to yourself it’s working fine, run the following curl command:

$ curl http://localhost:3000/users

If everything worked as planned, you should see “respond with a resource” printed in your terminal window. Now this is clearly as basic as it gets, but you can easily make it [a bit] more interesting by adding a new file to your project called myquickapi.js in your [app name]/routes/ folder, and add the following content:

var express = require('express');
var router = express.Router();

// get method route
router.get('/', function (req, res) {
res.send('You sent a get request');
});

// post method route
router.post('/', function(req, res) {
res.send('You sent me ' + req.param('data'));
});

module.exports = router;

A quick change to the app.js file to update our routes, by adding the following 2 lines:

var myquickapi = require(‘./routes/myquickapi');
app.use('/myquickapi', myquickapi);

Re-start your node service, and run:


$ curl -X POST http://localhost:3000/myquickapi?data=boo

And you will see the API handle the request parameter and echo it back to the caller.

Spin up an AWS EC2 instance

Log into the AWS portal and create a new EC2 instance.  For my project, as it is only a dev environment, I have gone for a General Purpose t2.Micro Ubuntu Server.  Plus it’s free which happens to be okay too!

Once the instance is up and running you will want to configure the security group to allow all inbound traffic using port 443, and 80 – after all it is a web api and I guess you want to access it!  I also enabled SSH for my local network:

Security_group

Using your pem file ssh into your new instance, and once connected, run the following commands:


# 1. update any out of date packages
sudo apt-get update

# 2. install nodejs
sudo apt-get install nodejs

# 3. install node package manager
sudo apt-get install npm

Now you can run node using the nodejs command. This is great, but not for the JS packages we’ll be using later on.  They reference the node command instead.  A simple fix is to create a symlink to the nodejs command:


$ sudo ln -s /usr/bin/nodejs /usr/bin/node

Set up nginx on you EC2 instance

We’ll use nginx on our server to proxy network traffic to the running nodejs instance.  Install nginx using the following commands:


# 1. install nginx

$ sudo apt-get install nginx

# 2. make a directory for your sites
$ sudo mkdir -p /var/www/express-api/public

# 3. set the permission of the folder so it is accessible publicly
$ sudo chmod 755 /var/www

# 4. remove the default nginx block
$ sudo rm /etc/nginx/sites-enabled/default

# 5. create the virtual hosts file
$ sudo nano /etc/nginx/sites-available/[your_api_name]

Now copy the following content into your virtual hosts file and save it:

upstream app_nodejs {
server 127.0.0.1:3000;
keepalive 8;
}

server {
listen 80;
listen [::]:80 default_server ipv6only=on;
listen 443 default ssl;

root /var/www/[your_site_folder]/public/[your_site_name];
index index.html index.htm;

# Make site accessible from http://localhost/
server_name [your_server_domain_name];

location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;

proxy_pass http://localhost:3000/;
proxy_redirect off;
}
}

This basically tells your server to listen on ports 80 and 443 and redirect any incoming traffic to your locally running nodes server on port 3000. A simple approach for now, but all that is needed to get our API up and running.

Activate your newly created hosts file by running the following command:

$ sudo ln -s /etc/nginx/sites-available/[your_api_name] /etc/nginx/sites-enabled/[your_api_name]

Now restart nginx to make your settings take place:

$ sudo service nginx restart

As a sanity test you can run the following command to confirm everything is setup correctly. If you are in good shape you will see a confirmation that the test has run successfully.

$ sudo nginx -c /etc/nginx/nginx.conf -t

Final steps

The last step in the process, which you could argue is the most important, is to copy your api code onto your new web server.  If you are creating this for a production system then I would encourage a deployment tool at this stage (and to be frank, probably a different approach altogether), but for now a simple secure copy is probably all that’s required:


$ scp -r [your_api_directory] your_username@aws_ec2_api:/var/www/[your_site_folder]/public/

And that’s it.  Fire up a browser and try running the curl commands against your EC2 instance rather than your local machine.  If all has gone well then you should get the same response as you did with your local environment (and it’ll be lightning fast).

… Just for fun

If you disconnect the ssh connection to your server it will stop the application from running.  A fairly big problem for a web api, but a simple fix to resolve.

A quick solution is to use the Forever tool.

Install it, and run your app (you’ll be glad you added the symlink to nodejs earlier):


$ sudo npm install -g forever

$ sudo forever start /var/www/[your_site_folder]/public/[your_site_name]/bin/www

 

Hopefully this will have provided a good insight into setting up a nodejs API on AWS. At the moment it is fairly basic, but time permitting, I would like to build on the API and add additional features to make it more useable – I’d particularly like to add a Javascript OAuth 2.0 endpoint!

Watch this space for future updates, as I add new features I will be sure to blog about the learnings I find along the way.

As Always; any questions, then just reach out to me, or post them below

AWS Direct Connect in Australia via Equinix Cloud Exchange

I discussed Azure ExpressRoute via Equinix Cloud Exchange (ECX) in my previous blog. In this post I am going to focus on AWS Direct Connect which ECX also provides. This means you can share the same physical link (1GBps or 10GBps) between Azure and AWS!

ECX also provides connectivity service to AWS for connection speed less than 1GBps. AWS Direct Connect provides dedicated, private connectivity between your WAN or datacenter and AWS services such as AWS Virtual Private Cloud (VPC) and AWS Elastic Compute Cloud (EC2).

AWS Direct Connect via Equinix Cloud Exchange is Exchange (IXP) provider based allowing us to extend our infrastructure that is:

  • Private: The connection is dedicated bypassing the public Internet which means better performance, increases security, consistent throughput and enables hybrid cloud use cases (Even hybrid with Azure when both connectivity using Equinix Cloud Exchange)
  • Redundant: If we configure a second AWS Direct Connect connection, traffic will failover to the second link automatically. Enabling Bidirectional Forwarding Detection (BFD) is recommended when configuring your connections to ensure fast detection and failover. AWS does not offer any SLA at the time of writing
  • High Speed and Flexible: ECX provides a flexible range of speeds: 50, 100, 200, 300, 400 and 500MBps.

The only tagging mechanism supported by AWS Direct Connect is 802.1Q (Dot1Q). AWS always uses 802.1Q (Dot1Q) on the Z-side of ECX.

ECX pre-requisites for AWS Direct Connect

The pre-requisites for connecting to AWS via ECX:

  • Physical ports on ECX. Two physical ports on two separate ECX chassis is required if redundancy is required.
  • Virtual Circuit on ECX. Two virtual circuits are also required for redundancy

Buy-side (A-side) Dot1Q and AWS Direct Connect

The following diagram illustrates the network setup required for AWS Direct Connect using Dot1Q ports on ECX:

AWSDot1Q

The Dot1Q VLAN tag on the A-side is assigned by the buyer (A-side). The Dot1Q VLAN tag on the seller side (Z-side) is assigned by AWS.

There are a few steps needing to be noted when configuring AWS Direct Connect via ECX:

  • We need our AWS Account ID to request ECX Virtual Circuits (VC)s
  • Create separate Virtual Interfaces (VI)s for Public and Private Peering on AWS Management Console. We need two ECX VCs and two AWS VIs for redundancy Private or Public Peering.
  • We can accept the Virtual Connection either from ECX Portal after requesting the VCs or  on AWS Management Console.
  • Configure our on-premises edge routers for BGP sessions. We can download the router configuration which we can use to configure our BGP sessions from AWS Management Console
    awsdot1q_a
  • Attach the AWS Virtual Gateway (VGW) to the Route Table associated with our VPC
  • Verify the connectivity.

Please refer to the AWS Direct Connect User Guide on how to configure edge routers for BGP sessions. Once we have configured the above we will need to make sure any firewall rules are modified so that traffic can be routed correctly.

I hope you’ve found this post useful – please leave any comments or questions below!

Read more from me on the Kloud Blog or on my own blog at www.wasita.net.

Automate your Cloud Operations Part 2: AWS CloudFormation

Stacking the AWS CloudFormation

Automate your Cloud Operations blog post Part 1 have given us the basic understanding on how to automate the AWS stack using CloudFormation.

This post will help the reader on how to layer the stack on top of the existing AWS CloudFormation stack using AWS CloudFormation instead of modifying the base template. AWS resources can be added into existing VPC using the outputs detailing the resources from the main VPC stack instead of having to modify the main template.

This will allow us to compartmentalize, separate out any components of AWS infrastructure and again versioning all different AWS infrastructure code for every components.

Note: The template I will use for this post for educational purposes only and may not be suitable for production workloads :).

Diagram below will help to illustrate the concept:

CloudFormation3

Bastion Stack

Previously (Part 1), we created the initial stack which provide us the base VPC. Next, we will  provision bastion stack which will create a bastion host on top of our base VPC. Below are the components of the bastion stack:

  • Create IAM user that can find out information about the stack and has permissions to create KeyPairs and actions related.
  • Create bastion host instance with the AWS Security Group enable SSH access via port 22
  • Use CloudFormation Init to install packages, create files and run commands on the bastion host instance also take the creds created for the IAM user and setup to be used by the scripts
  • Use the EC2 UserData to run the cfn-init command that actually does the above via a bash script
  • The condition handle: the completion of the instance is dependent on the scripts running properly, if the scripts fail, the CloudFormation stack will error out and fail

Below is the CloudFormation template to build the bastion stack:

Following are the high level steps to layer the bastion stack on top of the initial stack:

I put together the following video on how to use the template:

NAT Stack

It is important to design the VPC with security in mind. I recommend to design your Security Zone and network segregation, I have written a blog post regarding how to Secure Azure Network. This approach also can be impelemented on AWS environment using VPC, subnet and security groups. At the very minimum we will segregate the Private subnet and Public subnet on our VPC.

NAT instance will be added to our Initial VPC “public” subnets so that the future private instances can use the NAT instance for communication outside the Initial VPC. We will use exact same method like what we did on Bastion stack.

Diagram below will help to illustrate the concept:

CloudFormation4

The components of the NAT stack:

  • An Elastic IP address (EIP) for the NAT instance
  • A Security Group for the NAT instance: Allowing ingress traffic tcp, udp from port 0-65535 from internal subnet ; Allowing egress traffic tcp port 22, 80, 443, 9418 to any and egress traffic udp port 123 to Internet and egress traffic port 0-65535 to internal subnet
  • The NAT instance
  • A private route table
  • A private route using the NAT instance as the default route for all traffic

Following is the CloudFormation template to build the stack:

Similar like the previous steps on how to layer the stack:

Hopefully after reading the Part 1 and the Part 2 of my blog posts, the readers will gain some basic understanding on how to automate the AWS cloud operations using AWS CloudFormation.

Please contact Kloud Solutions if the readers need help with automating AWS production environment

http://www.wasita.net

Amazon Web Services (AWS) networking: public IP address and subnet list

Originally posted on Lucian’s blog over at clouduccino.com

Amazon Web Services (AWS) has many data centre’s in many continents and countries all over the world. AWS has two key grouping methods of these data centres: regions and availability zones.

It can be very handy to either reference the IP address or subnet of a particular service in say a proxy server to streamline connectivity. This is a good practice to avoid unnecessary latency via proxy authentication requests. Below is an output of Amazon Web Services IP address and subnet details split into the key categories as listed by AWS via thier publishing of information through the IP address JSON file available here.

Sidebar: Click here to read up more on regions and availability zones or click here or click here. Included in these references is also information about the DNS endpoints for services that are therefore IP address agnostic. Also, If you’d like more details about the JSON file click here.

Read More