Remove/Modify Specific AWS Tags from the Environment- PowerShell

Why use TAGs

To help you manage your instances, images, and other Amazon EC2 resources, you can optionally assign your own metadata to each resource in the form of tags. This topic describes tags and shows you how to create them.

(Ref: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html)

Problem :

Sometimes tags are applied in environments prior to developing a tagging strategy. The problem in exponentially increased with the size of the environment and the number of users creating resources.

Currently we are looking for a solution to remove specific unwanted tags from EC2 instances or modify the tag values which are incorrect.

For this purpose , the below mentioned script was developed that solves the problem for AWS.

Solution :

The below mentioned script performs the following tasks

  • Get the list of all the EC2 instances in the tenant
  • Loop through all the EC2 instances
  • Get values of all the tags in the environment
  • Check each Tag Key and Tag Value.
  • Modify of remove the tag value ( based on requirement )

Code:

#Set up the AWS profile using the Access Key and Secret Key

Set-AWSCredential -AccessKey AccessKey -SecretKey SecretKEy -StoreAs ProfileName

#Getting the list of all the instances in the Tenant

$instances = (Get-EC2Instance -ProfileName ProfileName -Region RegionName).Instances

$tagkeytoremove = 'TAG1' # Declaring the TAG Key to remove / modify

$tagvaluetoremove = 'ChangePLease' # Declaring the Tag Value to Remove / Modify

$NewTagValue = "NewTagValue" # Declaring the new tag value.

Foreach ( $instance in $instances ) # Looping through all the instances
{
    $OldTagList = $instance.tags
    foreach ($tag in $OldTagList) # Looping through all the Tags
    {
        if($tag.key -ceq $tagkeytoremove -and $tag.Value -ceq $tagvaluetoremove ) # Comparing the TAG Key and Values
        {
            Remove-EC2Tag -Resource $instances.instanceid -Tag $tag -Force # Removing the Old Tag Key Value Pair
            New-EC2Tag -Resource $instances.instanceid -Tag @{ Key=$tag.key;Value=$NewTagValue} -Force #Adding the New Tag Key Value pair.

        }
    }
} # Loop Ends

 

Highly Available deployment and availability pattern for heritage Windows applications on AWS

Quite often I am presented with the challenge of deploying Windows COTS applications onto the AWS platform with a need to requirement for taking advantage of cloud native patterns like auto-scaling and auto-healing. In this blog post I’m going to describe how I’ve used Auto Scaling Groups, Load Balancers, Cloudwatch Alarms and Route 53 to provide a self healing implementation for a heritage COTS Windows application. This pattern was also extended to use lifecycle hooks to support a Blue/Green deployment with zero downtime.
This pattern works quite nicely for heritage applications which suggest an Active/Passive configuration and the web tier is stateless, ie the Passive node does not have full application capability or is write only. When the primary node is unavailable during failure or blue/green upgrade, clients are transparently redirected to the passive node. I like to use the term “heritage” as it seems to have a softer ring than “legacy”. During actual failure, outage is less than 2 minutes in order for automatic failover to complete.
The diagram below summarises a number of the key components used in the design. In essence we have two autoscaling groups each within minimum, maximum of 1 within two availability zones. We have a private Route 53 hosted zone (int.aws) to host custom CNAME records which typically point to load balancers. A cross zone load balancer, in the example below I’m using a classic load balancer as I’m not doing SSL offload, however it could just as easily be an application load balancer. Route 53 and custom Cloudwatch alarms have been utilised to reduce the time required to fail over between nodes and support separate configuration of primary and secondary nodes.

A number of other assumptions:

  • Cloudwatch Alarm is set to detect where number of healthy nodes in AutoScaleGroup (ASG) ELB is less than 1. Current minimum polling interval is 60 seconds.
  • Independent server components – can support different configurations, ie primary/secondary config
  • Route 53 component (TTL 30 seconds) with a CNAME created with internal DNS (app.corp.com) to point to Route 53 CNAME (dns.master.int.aws). I use
  • ASG health checks on TCP port 443 configured (5 seconds interval, Healthy and Unhealthy threshold of 2). No point in setting any more granular as dependent on Cloudwatch alarm interval.
  • Single ASG deployed within each availability zone
  • Web tier is stateless
  • ELB still deployed over two availability zones.
  • TCP port monitors configured without SSL offload
  • No session stickiness configured as there is only a single server behind each ASG/ELB. In failover scenario clients will need to re-authenticate.
  • Use pre-baked AMIs to support shortest possible healing times.

Normal behaviour, client traffic is directed to Active node in AZ A.

 
Instance fails, and within 60 seconds Cloudwatch Alarm is triggered.

 
Route 53 health check is updated and Route 53 updates DNS record to the passive node. Clients now access to secondary/passive server. Clients may need to re-authenticate if application requires a stateful session.

 
Auto-healing rebuilds the failed server within AZ A.
 

Client now passes Route 53 health check and so Route 53 updates DNS record back to the primary node. Clients may need to re-authenticate if application requires a stateful session.

Secondary Node Failure

If secondary instance fails, there is no service disruption to service as traffic is never actively sent to this node, except during primary node failure.

Availability Zone Failure

These behave in a similar manner to instance failure and are dependent upon Cloudwatch alarm being sent.

Blue Green deployments

Blue Green deployments can be achieved using similar behaviour as experienced before.
On the left we see the existing release/build of the application stack, whilst on the right is the environment to be built. These are all within the same account and same availability zones, just different cloudformation stacks. There will be two stages described, a deploy stage where the new environment is being deployed and a release stage, where DNS is cutover. No additional build activities are conducted during the release stage.

DEPLOY Stage

1.Servers are built as independent components and then baked as AMIs.

2.Scales down server 2 component from previous build.

3. Server 2 is scaled up as part of deploy stage. Team can now test and validate this release prior to release via ELB for second instance. I like to include custom host headers with the servername and specific build number in order to easily identify which server I am hitting, which can be identified through Chrome debugger or fiddler.

RELEASE STAGE

4.Route 53 DNS is automatically updated to point to server 2 ELB. No service outage

5.Terminates the previous primary instance of the build and the primary server is now built within the new stack.
 

6. Server 1 bootstrap is initiated within the new cloudformation stack.

7 Route 53 DNS is updated to the CNAME of the ELB in front of the primary node and normal service resumes in the newly deployed/released environment.

 
Originally published at http://cloudconsultancy.info

Implementing a Break Glass Process with AWS Systems Manager

Modern day organisations rely on systems to perform critical, sometimes lifesaving tasks. As a result, a common requirement for many organisations is a break-glass process, providing the ability to bypass normal access control procedures when existing authentication mechanisms fail. The implementation of a break glass system often involves considerable effort to ensure the process is not open to malicious use and is auditable, yet simple and efficient. The good news is AWS Systems Manager (SSM) with AWS Key Management Service (KMS) can be leveraged to allow administrative users the ability to recover access to systems on-demand, without having to bake in privileged users with predefined passwords on systems.

How the AWS Systems Manager Break Glass solution works

Before we get into the configuration details, let’s walk through how this all works.

  1. The break-glass process is initiated when an administrative user invokes SSM Run Command against a target system using a custom SSM document for Windows or Linux.
  2. The commands in the SSM document are invoked and the root/admin password is set to a random string of characters. The string is then encrypted using KMS and stored in the SSM Parameter store.
  3. CloudWatch events detects that SSM Run Command has completed successfully and initiates a Lambda function to clean up the reset password.
  4. The Lambda function waits for 60 seconds, then removes the password from the parameter store.

As you can see, there’s minimal password management required for this solution without having to compromise security. Now that we have an understanding of how the solution hangs together, let’s take a look at how to set it up.

Creating the Customer Master Key

To begin, we need to create a key that will be used to encrypt passwords written to SSM parameter store. You can use the IAM section of the AWS Management Console to create a Customer Master Key by performing the following:

  1. Open the Encryption Keys section of the Identity and Access Management (IAM) console.
  2. For Region, choose the appropriate AWS region.
  3. Choose Create key.
  4. Type an alias for the CMK. Choose Next Step.
  5. Select which IAM users and roles can administer the CMK. Choose Next Step.
  6. Select which IAM users can use the CMK to encrypt and decrypt data. These users will be able to perform the break glass process. Choose Next Step.
  7. Choose Finish to create the CMK.

Creating the EC2 Policy

Great, so we’ve got a key set up. We now need to provide our instances access to encrypt the password and store it in the SSM parameter store. To do this, we need to create a custom IAM policy by performing the following:

  1. Open the IAM console.
  2. In the navigation column on the left, choose Policies.
  3. At the top of the page, choose Create Policy.
  4. On the Create Policy page choose Select on Create Your Own Policy.
  5. For Policy Name, type a unique name.
  6. The policy document you’ll want to use is defined below. Note that the key ARN defined here is the CMK created in the previous step.
  7. When you are done, choose Create Policy to save your completed policy.

Creating the EC2 Role

We now need to assign the policy to our EC2 instances. Additionally, we need to allow our instances access to communicate with the SSM endpoint. To do this, we’ll need to create an appropriate EC2 role:

  1. Open the IAM console.
  2. In the navigation pane, choose Roles, Create new role.
  3. On the Select role type page, choose Select next to Amazon EC2.
  4. On the Attach Policy page, select AmazonEC2RoleforSSM and the policy you created in the previous step.
  5. On the Set role name and review page, type a name for the role and choose Create role.

Attaching the Role to the EC2 Instance

After creating the EC2 role, we then need to attach it to the target instance(s).

  1. Navigate to the EC2 console.
  2. Choose Instances in the navigation pane.
  3. Select the target instance you intend to test the break-glass process on.
  4. Choose Actions, choose Instance Settings and then Attach/Replace IAM role from the drop-down list.
  5. On the Attach/Replace IAM role page, choose the role created in the previous step from the drop-down list.
  6. After choosing the IAM role, proceed to the next step by choosing Apply.

Creating the Password Reset SSM Document

An AWS Systems Manager Document defines the actions that are performed on the target instance(s). We need to create a multi-step cross-platform document that can reset Linux or Windows passwords based on the target platform. To do this, perform the following:

  1. Open the Amazon EC2 console.
  2. In the navigation pane, choose Documents.
  3. Choose Create Document.
  4. Type a descriptive name for the document.
  5. In the Document Type list, choose Command.
  6. Delete the brackets in the Content field, and then paste the document below containing scripts for Windows and Linux. Remember to replace the CMKs and region in the both scripts.
  7. Choose Create Document to save the document.

Congratulations! So far, you’ve set up the password reset functionality. Technically, you could stop here and you’d have a working break-glass capability, however we’re going to go one step further and add a clean-up process to remove the password from the parameter store for added security, as described below.

Creating the Lambda Function Policy

Our password clean-up process will use a Lambda function to delete the password from the parameter store. We’ll need to create an IAM policy to allow the Lamda function to do this.

  1. Open the IAM console.
  2. In the navigation column on the left, choose Policies.
  3. At the top of the page, choose Create Policy.
  4. On the Create Policy page choose Select on Create Your Own Policy.
  5. For Policy Name, type a unique name.
  6. The policy document to use is defined below.
  7. When you are done, choose Create Policy to save your completed policy.

Creating the Lambda Function Role

We now need to attach the policy to a role that will be used by our lambda function.

  1. Open the IAM console.
  2. In the navigation pane, choose RolesCreate new role.
  3. On the Select role type page, choose Select next to AWS Lambda.
  4. On the Attach Policy page, select CloudWatchLogsFullAccess (for logging purposes) and the policy you created in the previous step.
  5. On the Set role name and review page, type a name for the role and choose Create role.

Creating the Lambda Function

We now need to create the Lambda Function that will delete the password, and attach the role created in the previous step.

  1. Open the AWS Lambda console.
  2. Choose Create Function.
  3. Choose Author from scratch.
  4. On the triggers page, click Next.
  5. Under Basic Information enter a name for your function and select the Python 2.7 runtime.
  6. Under Lambda function code, enter the code below.
  7. Under Lambda Function handler and role, choose the role you created in the previous step.
  8. Expand advanced settings and extend the timeout to 90 seconds.
  9. Choose Next and review the summary page then click Create Function.

Creating the CloudWatch Event

Almost there! The last step is to capture a successful execution of SSM Run Command, then trigger the previously created Lambda function. We can capture this using CloudWatch events:

  1. Open the CloudWatch console.
  2. In the navigation pane, choose Events.
  3. Choose Create rule.
  4. For Event Source, Choose Event Pattern, then choose Build custom event pattern, from the dropdown box.
  5. Enter the following into the text box, replacing the document name with the SSM document that was created earlier.
  6. For Targets, choose Add target and then choose Lambda function.
  7. For Function, select the Lambda function that was created in the previous step.
  8. Choose Configure details.
  9. For Rule definition, type an appropriate name.
  10. Choose Create rule.

That’s it! All that’s left is taking the process for a test drive. Let’s give it a shot.

Testing the Process

Assuming you’ve logged into the console with a user that has decrypt access for the CMK used, the following process can be used to access the password:

  1. Open the Amazon EC2 console.
  2. In the navigation pane under Systems Manager Services, choose Run Command.
  3. Choose Run a command.
  4. For Command document, choose the SSM Document created earlier.
  5. For Target instances, choose an instance that has the previously created EC2 role attached. If you do not see the instance in this list, it might not have the correct role attached, or may not be able to access the SSM endpoint.
  6. Choose Run, and then choose View results.
  7. In the commands list, choose the command you just executed. If the command is still in progress, click the refresh icon in the top right corner of the console.
  8. When the Status column shows Success, click the Output tab.
  9. The output tab will display successful execution of both plugins described in our SSM document. If we click View Output on both we’ll notice that one didn’t execute due to not meeting the platform precondition we set. The other plugin should show that the password reset executed successfully.

  1. In the navigation pane under Systems Manager Shared Resources, choose Parameter Store.
  2. Assuming 60 seconds hasn’t elapsed (because our clean-up function will kick in and delete the password) there should be a parameter called pwd-<instance-ID>. After selecting the parameter, the Description tab below will show a SecureString.
  3. Click on Show to view the password.


You can now use this password to access the administrator/root account of your instance. Assuming the password clean-up script is configured correctly, the password should disappear from the parameter store within 60 seconds of the Run Command completing.

Conclusion

The above process provides a simple and secure method for emergency access to both Windows and Linux systems, without the complex process and inherent risk of a traditional break-glass system. Additionally, this method has no running systems, providing a break-glass capability at nearly no cost.
 

Supercharge your CloudFormation templates with Jinja2 Templating Engine

If you are working in an AWS public cloud environment chances are that you have authored a number of CloudFormation templates over the years to define your infrastructure as code. As powerful as this tool is, it has a glaring shortcoming: the templates are fairly static having no inline template expansion feature (think GCP Cloud Deployment Manager.) Due to this limitation, many teams end up copy-pasting similar templates to cater for minor differences like environment (dev, test, prod etc.) and resource names (S3 bucket names etc.)
Enter Jinja2. A modern and powerful templating language for Python. In this blog post I will demonstrate a way to use Jinja2 to enable dynamic expressions and perform variable substitution in your CloudFormation templates.
First lets get the prerequisites out of the way. To use Jinja2, we need to install Python, pip and of course Jinja2.
Install Python
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
sudo yum install python
[/sourcecode]
Install pip
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
sudo python get-pip.py
[/sourcecode]
Install Jinja2
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
pip install Jinja2
[/sourcecode]
To invoke Jinja2, we will use a simple python wrapper script.
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
vi j2.py
[/sourcecode]
Copy the following contents to the file j2.py
[sourcecode language=”python” wraplines=”false” collapse=”false”]
import os
import sys
import jinja2
sys.stdout.write(jinja2.Template(sys.stdin.read()).render(env=os.environ))
[/sourcecode]
Save and exit the editor
Now let’s create a simple CloudFormation template and transform it through Jinja2:
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
vi template1.yaml
[/sourcecode]
Copy the following contents to the file template1.yaml
[sourcecode language=”cpp” wraplines=”false” collapse=”false”]

AWSTemplateFormatVersion: ‘2010-09-09’
Description: Simple S3 bucket for {{ env[‘ENVIRONMENT_NAME’] }}
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: InstallFiles-{{ env[‘AWS_ACCOUNT_NUMBER’] }}
[/sourcecode]
As you can see it’s the most basic CloudFormation template with one exception, we are using Jinja2 variable for substituting the environment variable. Now lets run this template through Jinja2:
Lets first export the environment variables
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
export ENVIRONMENT_NAME=Development
export AWS_ACCOUNT_NUMBER=1234567890
[/sourcecode]
Run the following command:
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
cat template1.yaml | python j2.py
[/sourcecode]
The result of this command will be as follows:
[sourcecode language=”cpp” wraplines=”false” collapse=”false”]

AWSTemplateFormatVersion: ‘2010-09-09’
Description: Simple S3 bucket for Development
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: InstallFiles-1234567890
[/sourcecode]
As you can see Jinja2 has expanded the variable names in the template. This provides us with a powerful mechanism to insert environment variables into our CloudFormation templates.
Lets take another example, what if we wanted to create multiple S3 buckets in an automated manner. Generally in such a case we would have to copy paste the S3 resource block. With Jinja2, this becomes a matter of adding a simple “for” loop:
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
vi template2.yaml
[/sourcecode]
Copy the following contents to the file template2.yaml
[sourcecode language=”cpp” wraplines=”false” collapse=”false”]

AWSTemplateFormatVersion: ‘2010-09-09’
Description: Simple S3 bucket for {{ env[‘ENVIRONMENT_NAME’] }}
Resources:
{% for i in range(1,3) %}
S3Bucket{{ i }}:
Type: AWS::S3::Bucket
Properties:
BucketName: InstallFiles-{{ env[‘AWS_ACCOUNT_NUMBER’] }}-{{ i }}
{% endfor %}
[/sourcecode]
Run the following command:
[sourcecode language=”bash” wraplines=”false” collapse=”false”]
cat template2.yaml | python j2.py
[/sourcecode]
The result of this command will be as follows:
[sourcecode language=”cpp” wraplines=”false” collapse=”false”]

AWSTemplateFormatVersion: ‘2010-09-09’
Description: Simple S3 bucket for Development
Resources:
S3Bucket1:
Type: AWS::S3::Bucket
Properties:
BucketName: InstallFiles-1234567890-1
S3Bucket2:
Type: AWS::S3::Bucket
Properties:
BucketName: InstallFiles-1234567890-2
[/sourcecode]
As you can see the resulting template has two S3 Resource blocks. The output of the command can be redirected to another template file to be later used in stack creation.
I am sure you will appreciate the possibilities Jinja2 brings to enhance your CloudFormation templates. Do note that I have barely scratched the surface of this topic, and I highly recommend you to have a look at the Template Designer Documentation found at http://jinja.pocoo.org/docs/2.10/templates/ to explore more possibilities. If you are using Ansible, do note that Ansible uses Jinja2 templating to enable dynamic expressions and access to variables. In this case you can get rid of the Python wrapper script mentioned in this article and use Ansible directly for template expansion.

Replacing the service desk with bots using Amazon Lex and Amazon Connect (Part 2)

Welcome back! Hopefully you had the chance to follow along in part 1 where we started creating our Lex chatbot. In part 2, we attempt to make the conversation more human-like and begin integrating data validation on our slots to ensure we’re getting the correct input.

Creating the Lambda initialisation and validation function

As data validation requires compute, we’ll need to start by creating an AWS Lambda function. Head over to the AWS console, then navigate to the AWS Lambda page. Once you’re there, select Create Function and choose to Author from Scratch then specify the following:
Name: ResetPWCheck
Runtime: Python 2.7 (it’s really a matter of preference)
Role: I use an existing Out of the Box role, “Lambda_basic_execution”, as I only need access to CloudWatch logs for debugging.

Once you’ve populated all the fields, go ahead and select Create Function. The script we’ll be using is provided (further down) in this blog, however before we go through the script in detail, there are two items worth mentioning.

Input Events and Response Formats

It’s well worth familiarising yourself with the page on Lambda Function Input Event and Response Formats in the Lex Developer guide. Every time input is provided to Lex, it invokes the Lambda initalisation and validation function. For example, when I tell my chatbot “I need to reset my password”, the lambda function is invoked and the following event is passed:

Amazon Lex expects a response from the Lambda function in JSON format that provides it with the next dialog action.

Persisting Variables with Session Attributes

There are many ways to determine within your Lambda function where you’re up to in your chat dialog, however my preferred method is to pass state information within the SessionAttributes object of the input event and response as a key/value pair. The SessionAttributes can persist between invocations of the Lambda function (every time input is provided to the chatbot), however you must remember to collect and pass the attributes between input and responses to ensure it persists.

Input Validation Code

With that out of the way, let’s begin looking at the code. The below script is what I’ve used which you can simply copy and paste, assuming you’re using the same slot and intent names in your Lex bot that were used in Part 1.

Let’s break it down.
When the lambda function is first invoked, we check to see if any state is set in the sessionAttributes. If not, we can assume this is the first time the lambda function is invoked and as a result, provide a welcoming response while requesting the User’s ID. To ensure the user isn’t welcomed again, we set a session state so the Lambda function knows to move to User ID validation when next invoked. This is done by setting the “Completed” : “None” key/value pair in the response SessionAttributes.

Next, we check the User ID. You’ll notice the chkUserId function checks for two things; That the slot is populated, and if it is, the length of the field. Because the slot type is AMAZON.Number, any non-numeric characters that are entered will be rejected by the slot. If this occurs, the slot will be left empty, hence this is something we’re looking out for. We also want to ensure the User ID is 6 digits, otherwise it is considered invalid. If the input is correct, we set the session state key/value pair to indicate User ID validation is complete then allow the dialog to continue, otherwise we request the user to re-enter their User ID.

Next, we check the Date of Birth. Because the slot type is strict regarding input, we don’t do much validation here. An utterance for this slot type generally maps to a complete date: YYYY-MM-DD. For validation purpose, we’re just looking for an empty slot. Like the User ID check, we set the session state and allow the dialog to continue if all looks good.

Finally, we check the last slot which is the Month Started. Assuming the input for the month started is correct, we then confirm the intent by reading all the slot values back to the user and asking if it’s correct. You’ll notice here that there’s a bit of logic to determine if the user is using voice or text to interact with Lex. If voice is used, we use Speech Synthesis Markup Language (SSML) to ensure the UserID value is read as digits, rather than as the full number.
If the user is happy with the slot values, the validation completes and Lex then moves to the next Lambda function to fulfil the intent (next blog). If the user isn’t happy with the slot values, the lambda function exits telling the user to call back and try again.

Okay, now that our Lambda function is finished, we need to enable it as a code hook for initialisation and validation. Head over to your Lex bot, select the “ResetPW” intent, then tick the box under Lambda initialisation and validation and select your Lambda function. A prompt will be given to provide permissions to allow your Lex bot to invoke the lambda function. Select OK.

Let’s hit Build on the chatbot, and test it out.

So, we’ve managed to make the conversation a bit more human like and we can now detect invalid input. If you use the microphone to chat with your bot, you’ll notice the User ID value is read as digits. That’s it for this blog. Next blog, we’ll integrate Active Directory and actually get a password reset and sent via SNS to a mobile phone.

Replacing the service desk with bots using Amazon Lex and Amazon Connect (Part 1)

“What! Is this guy for real? Does he really think he can replace the front line of IT with pre-canned conversations?” I must admit, it’s a bold statement. The IT Service Desk has been around for years and has been the foot in the door for most of us in the IT industry. It’s the face of IT operations and plays an important role in ensuring an organisation’s staff can perform to the best of their abilities. But what if we could take some of the repetitive tasks the service desk performs and automate them? Not only would we be saving on head count costs, we would be able to ensure correct policies and procedures are followed to uphold security and compliance. The aim of this blog is to provide a working example of the automation of one service desk scenario to show how easy and accessible the technology is, and how it can be applied to various use cases.
To make it easier to follow along, I’ve broken this blog up into a number of parts. Part 1 will focus on the high-level architecture for the scenario and begin creating the Lex chatbot.

Scenario

Arguably, the most common service desk request is the password reset. While this is a pretty simple issue for the service desk to resolve, many service desk staff seem to skip over, or not realise the importance of user verification. Both the simple resolution and the strict verification requirement make this a prime scenario to automate.

Architecture

So what does the architecture look like? The diagram below dictates the expected process flow. Let’s step through each item in the diagram.

 

Amazon Connect

The process begins when the user calls the service desk and requests to have their password reset. In our architecture, the service desk uses Amazon Connect which is a cloud based customer contact centre service, allowing you to create contact flows, manage agents, and track performance metrics. We’re also able to plug in an Amazon Lex chatbot to handle user requests and offload the call to a human if the chatbot is unable to understand the user’s intent.

Amazon Lex

After the user has stated their request to change their password, we need to begin the user verification process. Their intent is recognised by our Amazon Lex chatbot, which initiates the dialog for the user verification process to ensure they are who they really say they are.

AWS Lambda

After the user has provided verification information, AWS Lambda, which is a compute on demand service, is used to validate the user’s input and verify it against internal records. To do this, Lambda interrogates Active Directory to validate the user.

Amazon SNS

Once the user has been validated, their password is reset to a random string in Active Directory and the password is messaged to the user’s phone using Amazon’s Simple Notification Service. This completes the interaction.

Building our Chatbot

Before we get into the details, it’s worth mentioning that the aim of this blog is to convey the technology capability. There’s many ways of enhancing the solution or improving validation of user input that I’ve skipped over, so while this isn’t a finished production ready product, it’s certainly a good foundation to begin building an automated call centre.
To begin, let’s start with building our Chatbot in Amazon Lex. In the Amazon Console, navigate to Amazon Lex. You’ll notice it’s only available in Ireland and US East. As Amazon Connect and my Active Directory environment is also in US East, that’s the region I’ve chosen.
Go ahead and select Create Bot, then choose to create your own Custom Bot. I’ve named mine “UserAdministration”. Choose an Output voice and set the session timeout to 5 minutes. An IAM Role will automatically be created on your behalf to allow your bot to use Amazon Polly for speech. For COPPA, select No, then select Create.

Once the bot has been created, we need to identify the user action expected to be performed, which is known as an intent. A bot can have multiple intents, but for our scenario, we’re only creating one, which is the password reset intent. Go ahead and select Create Intent, then in the Add Intent window, select Create new intent. My intent name is “ResetPW”. Select Add, which should add the intent to your bot. We now need to specify some expected sample utterances, which are phrases the user can use to trigger the Reset Password intent. There’s quite a few that could be listed here, but I’m going to limit mine to the following:

  • I forgot my password
  • I need to reset my password
  • Can you please reset my password


The next section is the configuration for the Lambda validation function. Let’s skip past this for the time being and move onto the slots. Slots are used to collect information from the user. In our case, we need to collect verification information to ensure the user is who they say they are. The verification information collected is going to vary between environments. I’m looking to collect the following to verify against Active Directory:

  • User ID – In my case, this is a 6-digit employee number that is also the sAMAccountName in Active Directory
  • User’s birthday – This is a custom attribute in my Active Directory
  • Month started – This is a custom attribute in my Active Directory

In addition to this, it’s also worth collecting and verifying the user’s mobile number. This can be done by passing the caller ID information from Amazon Connect, however we’ll skip this, as the bulk of our testing will be text chat and we need to ensure we have a consistent experience.
To define a slot, we need to specify three items:

  • Name of the slot – Think of this as the variable name.
  • Slot type – The data type expected. This is used to train the machine learning model to recognise the value for the slot.
  • Prompt – How the user is prompted to provide the value sought.

Many slot types are provided by Amazon, two of which has been used in this scenario. For “MonthStarted”, I’ve decided to create my own custom slot type, as the in-built “AMAZON.Month” slot type wasn’t strictly enforcing recognisable months. To create your own slot type, press the plus symbol on the left-hand side of the page next to Slot Types, then provide a name and description for your slot type. Select to Restrict to Slot values and Synonyms, then enter each month and its abbreviation. Once completed, click Add slot to intent.

Once the custom slot type has been configured, it’s time to set up the slots for the intent. The screenshot below shows the slots that have been configured and the expected order to prompt the user.

Last step (for this blog post), is to have the bot verify the information collected is correct. Tick the Confirmation Prompt box and in the Confirm text box provided, enter the following:
Just to confirm, your user ID is {UserID}, your Date of Birth is {DOB} and the month you started is {MonthStarted}. Is this correct?
For the Cancel text box, enter the following:
Sorry about that. Please call back and try again.

Be sure to leave the fulfillment to Return parameters to client and hit Save Intent.
Great! We’ve built the bare basics of our bot. It doesn’t really do much yet, but let’s take it for a spin anyway and get a feel for what to expect. In the top right-hand corner, there’s a build button. Go ahead and click the button. This takes some time, as building a bot triggers machine learning and creates the models for your bot. Once completed, the bot should be available to text or voice chat on the right side of the page. As you move through the prompts, you can see at the bottom the slots getting populated with the expected format. i.e. 14th April 1983 is converted to 1983-04-14.

So at the moment, our bot doesn’t do much but collect the information we need. Admittedly, the conversation is a bit robotic as well. In the next few blogs, we’ll give the bot a bit more of a personality, we’ll do some input validation, and we’ll begin to integrate with Active Directory. Once we’ve got our bot working as expected, we’ll bolt on Amazon Connect to allow users to dial in and converse with our new bot.

Re-execute the UserData script in an AWS Windows Instance

First published at https://nivleshc.wordpress.com
Bootstrapping is an awesome way of customising your instances in AWS (similar capability exists in Azure).
To enable bootstrapping, while configuring the launch instance, in Step 3: Configure Instance Details scroll down to the bottom and then expand Advanced Details.
You will notice a User data text box. This is where you can provide your bootstrap script. The script will be run when your instance is first launched.
AWS_BootstrapScript
I went ahead and entered my script in the text box and proceeded to complete my instance configuration. Once my instance was running, I initiated a Remote Desktop connection to it, to confirm that my script had run. Unfortunately, I couldn’t see any customisations (which meant my script didn’t run)
Thinking that the instance had not been able to access the user data, I opened up Internet Explorer and then browsed to the following url (this is an internal url that can be used to access the user-data)
http://169.254.169.254/latest/user-data/
I was able to successfully access the user-data, which meant that there were no issues with that.  However when checking the content, I noticed a typo! Aha, that was the reason why my customisations didn’t happen.
Unfortunately, according to AWS, user-data is only executed during launch (for those that would like to read, here is the official AWS documentation). To get the fixed bootstrap script to run, I would have to terminate my instance and launch a new one with the corrected script (I tried re-booting my windows instance after correcting my typo, however it didn’t run).
I wasn’t very happy on terminating my current instance and then launching a new one, since for those that might not be aware, AWS EC2 compute charges are rounded up to the next hour. Which means that if I terminated my current instance and launched a new one, I would be charged for 2 x 1hour sessions instead of just 1 x 1 hour!
So I set about trying to find another solution. And guess what, I did find it 🙂
Reading through the volumes of documentation on AWS, I found that when Windows Instances are provisioned, the service that does the customisations using user-data is called EC2Config. This service runs the initial startup tasks when the instance is first started and then disables them. HOWEVER, there is a way to re-enable the startup tasks later on 🙂 Here is the document that gives more information on EC2Config.
The Amazon Windows AMIs include a utility called EC2ConfigService Settings. This allows you to configure EC2Config to execute the user-data on next service startup. The utility can be found under All Programs (or you can search for it).
AWS_EC2ConfigSettings_AllApps
AWS_EC2ConfigSettings_Search
Once Open, under General you will see the following option
Enable UserData execution for next service start (automatically enabled at Sysprep) eg. or <powershell></powershell>
AWS_EC2ConfigSettings
Tick this option and then press OK. Then restart your Windows Instance.
After your Windows Instance restarts, EC2Config will execute the userData (bootstrap script) and then it will automatically remove the tick from the above option so that the userData is not executed on subsequent restarts (or service starts)
There you go. A simple way to re-run your bootstrap scripts on an AWS Windows Instance without having to terminate the current instance and launching a new one.
There are other options available in the EC2ConfigService Settings that you can explore as well 🙂

A Closer Look at Amazon Chime

In news this quarter AWS have released a web conferencing cloud service to their existing ‘Business Productivity‘ services which already includes Amazon WorkDocs and Amazon WorkMail. So my thought was to help you gauge where this sits in relation to Skype for Business. I don’t want to put this into a Microsoft versus Amazon review but I do want you to understand the product names that ‘somewhat’ align with each other.

Exchange = WorkMail

SharePoint/OneDrive for Business  =  WorkDocs

Skype for Business  = Chime

The Microsoft products are reasonably well known in the world so I’ll give you a quick one liner about the Amazons products:

WorkMail “Hosted Email”

WorkDocs “Hosted files accessible via web, PC, mobile devices with editing and sharing capability”

So what is Chime?

Chime isn’t exactly a one-to-one feature set for Skype for Business so be wary of articles conveying this sentiment as they haven’t really done their homework. Chime can be called either a web meeting, online meeting, or online video conferencing platform. Unlike Skype for Business, Chime is not a PBX replacement. So what does it offer?

  • Presence
  • 1-1 and group IM
  • Persistent chat / chat rooms
  • Transfer files
  • Group HD video calling
  • Schedule meetings using Outlook or Google calendar
  • Join meeting from desktop or browser
  • Join meeting anonymously
  • Meeting controls for presenter
  • Desktop sharing
  • Remote desktop control
  • Record audio and video of meetings
  • Allow participants to dial-in with no Chime client (PSTN conferencing bridge)
  • Enable legacy H.323 endpoints to join meetings (H.323 bridge)
  • Customisable / personalised meeting space URL

The cloud hosted based products that I see are similar to Chime are WebEx, Google Hangouts, GoToMeeting and ReadyTalk to name just a few. As here at Kloud we are Microsoft Gold Partners and have a Unified Communication team that deliver Skype for Business solutions and not the other products I have previously mentioned, I will tell you a few things that differentiate SFB from Chime.

  • Direct Inward Dial (DID) user assignment
  • Call forwarding and voicemail
  • Automated Call Distribution (ACD)
  • Outbound PSTN dialling and route based on policy assignment
  • Integrated Voice Recording (IVR)
  • Hunt Groups / Call Pickup Groups
  • Shared Line Apperance
  • SIP IP-Phone compatibility
  • Attendant / Reception call pickup
  • On-premises, hybrid and hosted deployment models

Basically all things that replace a PBX Solution Skype for Business will do. Now this isn’t a cheap shot at Amazon, cause that isn’t where they are positioning their product. What I’ve hoped to have done is clarify any misconception about where the product sits in the market and how it relates to features in a well known product like Microsoft Skype for Business.
For the price that Amazon are offering Chime for in the online meeting market it is very competitive against some other hosted products. Their ‘rolls-royce’ plan is simply purchased for $15 USD per user per month. If you’re not invested in the Office 365 E3/E5 license ecosystem and you need a online meeting platform at a low cost, then Chime might be right for you. Amazon offer a 30 day free trial for free that is a great way to test it out.
https://chime.aws
 

Dynamically rename an AWS Windows host deployed via a syspreped AMI

One of my customers presented me with a unique problem last week. They needed to rename a Windows Server 2016 host deployed using a custom AMI without rebooting during the bootstrap process. This lack of a reboot rules out the simple option of using the PowerShell Rename-Computer Cmdlet. While there are a number of methods to do this, one option we came up with is dynamically updating the sysprep unattended answer file using a PowerShell script prior to the unattended install running during first boot of a sysprepped instance.
To begin, we looked at the Windows Server 2016 sysprep process on an EC2 instance to get an understanding of the process required. It’s important to note that this process is slightly different to Server 2012 on AWS, as EC2Launch has replaced EC2Config. The high level process we need to perform is the following:

  1. Deploy a Windows Server 2016 AMI
  2. Connect to the Windows instance and customise it as required
  3. Create a startup PowerShell script to set the hostname that will be invoked on boot before the unattended installation begins
  4. Run InitializeInstance.ps1 -Schedule to register a scheduled task that initializes the instance on next boot
  5. Modify SysprepInstance.ps1 to replace the cmdline registry key after sysprep is complete and prior to shutdown
  6. Run SysprepInstance.ps1 to sysprep the machine.

Steps 1 and 2 are fairly self-explanatory, so let’s start by taking a look at step 3.
After a host is sysprepped, the value for the cmdline registry key located in HKLM:\System\Setup is populated with the windeploy.exe path, indicating that it should be invoked after reboot to initiate the unattended installation. Our aim is to ensure that the answer file (unattend.xml) is modified to include the computer name we want the host to be named prior to windeploy.exe executing. To do this, I’ve created the following PowerShell script named startup.ps1 and placed it in C:\Scripts to ensure my hostname is based on the internal IP address of my instance.

Once this script is in place, we can move to step 4, where we schedule the InitializeInstance.ps1 script to run on first boot. This can be done by running InitializeInstance.ps1 -Schedule located in C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts
Step 5 requires a modification of the SysprepInstance.ps1 file to ensure the shutdown after Sysprep is delayed to allow modification of the cmdline registry entry mentioned in step 3. The aim here is to replace the windeploy.exe path with our startup script, then shutdown the host. The modifications to the PowerShell script to accommodate this are made after the “#Finally, perform Sysprep” comment.

Finally, run the SysprepInstance.ps1 script.
Once the host changes to the stopped state, you can now create an AMI and use this as your image.

Experiences with the new AWS Application Load Balancer

Originally posted on Andrew’s blog @ cloudconsultancy.info

Summary

Recently I had an opportunity to test drive AWS Application load balancer as my client had a requirement for making their websocket application fault tolerant. The implementation was complete windows stack and utilised ADFS 2.0 for SAML authentication however this should not affect other people’s implementation.
The AWS Application load balancer is a fairly new feature which provides layer 7 load balancing and support for HTTP/2 as well as websockets. In this blog post I will include examples of the configuration that I used to implement as well is some of the troubleshooting steps I needed to resolve.
The application load balancer is an independent AWS resource from classic ELB and is defined as aws elbv2 with a number of different properties.
Benefits of Application Load Balancer include:

  • Content based routing, ie route /store to a different set of instances from /apiv2
  • Support for websocket
  • Support for HTTP/2 over HTTPS only (much larger throughput as it’s a single stream multiplexed meaning it’s great for mobile and other high latency apps)
  • Cheaper cost than classic, roughly 10% cheaper than traditional.
  • Cross-zone load balancing is always enabled for ALB.

Some changes that I’ve noticed:

  • Load balancing algorithm used for application load balancer is currently round robin.
  • Cross-zone load balancing is always enabled for an Application Load Balancer and is disabled by default for a Classic Load Balancer.
  • With an Application Load Balancer, the idle timeout value applies only to front-end connections and not the LB-> server connection and this prevents the LB cycling the connection.
  • Application Load balancer is exactly that and performs at Layer 7, so if you want to perform SSL bridge use Classic load balancer with TCP and configure SSL certs on your server endpoint.
  • cookie-expiration-period value of 0 is not supported to defer session timeout to the application. I ended up having to configure the stickiness.lb_cookie.duration_seconds value. I’d suggest making this 1 minute longer than application session timeout, in my example a value of 1860.
  • The X-Forwarded-For parameter is still supported and should be utilised if you need to track client IP addresses, in particular useful if going through a proxy server.

For more detailed information from AWS see http://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html.

Importing SSL Certificate into AWS – Windows

(http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_server-certs.html )

  1. Convert the existing pkcs key into .pem format for AWS

You’ll need openssl for this, the pfx and the password for the SSL certificate.
I like to use chocolatey as my Windows package manager, similar to yum or apt-get for Windows, which is a saviour for downloading package and managing dependencies in order to support automation, but enough of that, check it out @ https://chocolatey.org/
Once choco is installed I simply execute the following from an elevated command prompt.
“choco install openssl.light”
Thereafter I run the following two commands which breaks out the private and public keys (during which you’ll be prompted for the password):
openssl pkcs12 -in keyStore.pfx -out SomePrivateKey.key –nodes –nocerts
openssl pkcs12 -in keyStore.pfx -out SomePublic.cert –nodes –nokeys
NB: I’ve found that sometimes copy and paste doesn’t work when trying to convert keys.

  1. Next you’ll need to also break out the trust chain into one contiguous file, like the following.
-----BEGIN CERTIFICATE-----
Intermediate certificate 2
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
Intermediate certificate 1
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
Optional: Root certificate
-----END CERTIFICATE-----

Save the file for future use,
thawte_trust_chain.txt
Example attached above is for a Thawte trust chain with the following properties

“thawte Primary Root CA” Thumbprint ‎91 c6 d6 ee 3e 8a c8 63 84 e5 48 c2 99 29 5c 75 6c 81 7b 81

With intermediate

“thawte SSL CA - G2” Thumbprint ‎2e a7 1c 36 7d 17 8c 84 3f d2 1d b4 fd b6 30 ba 54 a2 0d c5

Ordinarily you’ll only have a root and intermediate CA, although sometimes there will be second intermediary CA.
Ensure that your certificates are base 64 encoded when you export them.

  1. Finally execute the following after authenticating to the AWS CLI (v1.11.14+ to support aws elbv2 function) then run “aws configure” applying your access and secret keys, configuring region and format type. Please note that this includes some of the above elements including trust chain and public and private keys.

If you get the error as below
A client error (MalformedCertificate) occurred when calling the UploadServerCertificate operation: Unable to validate certificate chain. The certificate chain must start with the immediate signing certificate, followed by any intermediaries in order. The index within the chain of the invalid certificate is: 2”
Please check the contents of the original root and intermediate keys as they probably still have the headers and maybe some intermediate,
ie

Bag Attributes
localKeyID: 01 00 00 00
friendlyName: serviceSSL
subject=/C=AU/ST=New South Wales/L=Sydney/O=Some Company/OU=IT/CN=service.example.com
issuer=/C=US/O=thawte, Inc./CN=thawte SSL CA - G2
Bag Attributes
friendlyName: thawte
subject=/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
issuer=/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA

AWS Application LB Configuration

Follow this gist with comments embedded. Comments provided based on some gotchas during configuration.

You should be now good to go, the load balancer takes a little while to warm up, however will be available within multiple availability zones.
If you have issues connecting to the ALB, validate connectivity direct to the server using curl.  Again chocolatey comes in handy “choco install curl”
Also double check your security group registered against the ALB and confirm NACLS.

WebServer configuration

You’ll need to import the SSL certificate into the local computer certificate store. Some of the third party issuing (Ensign, Thawte, etc) CAs may not have the intermediate CA within the computed trusted Root CAs store, especially if built in a network isolated from the internet, so make sure after installing the SSL certificate on the server that the trust chain is correct.
You won’t need to update local hosts file on the servers to point to the load balanced address.

Implementation using CNAMEs

In large enterprises where I’ve worked there have been long lead times associated with fairly simple DNS changes, which defeats some of the agility provided by cloud computing. A pattern I’ve often seen adopted is to use multiple CNAMEs to work around such lead times. Generally you’ll have a subdomain domain somewhere where the Ops team have more control over or shorter lead times. Within the target domain (Example.com) create a CNAME pointing to an address within the ops managed domain (aws.corp.internal) and have a CNAME created within that zone to point to the ALB address, ie
Service.example.com -> service.aws.corp.internal -> elbarn.region.elb.amazonaws.com
With this approach I can update service.aws.corp.internal to reflect a new service which I’ve built via a new ELB and avoid the enterprise change lead times associated with a change in .example.com.

Follow ...+

Kloud Blog - Follow