Modern day organisations rely on systems to perform critical, sometimes lifesaving tasks. As a result, a common requirement for many organisations is a break-glass process, providing the ability to bypass normal access control procedures when existing authentication mechanisms fail. The implementation of a break glass system often involves considerable effort to ensure the process is not open to malicious use and is auditable, yet simple and efficient. The good news is AWS Systems Manager (SSM) with AWS Key Management Service (KMS) can be leveraged to allow administrative users the ability to recover access to systems on-demand, without having to bake in privileged users with predefined passwords on systems.

How the AWS Systems Manager Break Glass solution works

Before we get into the configuration details, let’s walk through how this all works.

1. The break-glass process is initiated when an administrative user invokes SSM Run Command against a target system using a custom SSM document for Windows or Linux.
2. The commands in the SSM document are invoked and the root/admin password is set to a random string of characters. The string is then encrypted using KMS and stored in the SSM Parameter store.
3. CloudWatch events detects that SSM Run Command has completed successfully and initiates a Lambda function to clean up the reset password.
4. The Lambda function waits for 60 seconds, then removes the password from the parameter store.

As you can see, there’s minimal password management required for this solution without having to compromise security. Now that we have an understanding of how the solution hangs together, let’s take a look at how to set it up.

Creating the Customer Master Key

To begin, we need to create a key that will be used to encrypt passwords written to SSM parameter store. You can use the IAM section of the AWS Management Console to create a Customer Master Key by performing the following:

1. Open the Encryption Keys section of the Identity and Access Management (IAM) console.
2. For Region, choose the appropriate AWS region.
3. Choose Create key.
4. Type an alias for the CMK. Choose Next Step.
5. Select which IAM users and roles can administer the CMK. Choose Next Step.
6. Select which IAM users can use the CMK to encrypt and decrypt data. These users will be able to perform the break glass process. Choose Next Step.
7. Choose Finish to create the CMK.

Creating the EC2 Policy

Great, so we’ve got a key set up. We now need to provide our instances access to encrypt the password and store it in the SSM parameter store. To do this, we need to create a custom IAM policy by performing the following:

1. Open the IAM console.
2. In the navigation column on the left, choose Policies.
3. At the top of the page, choose Create Policy.
4. On the Create Policy page choose Select on Create Your Own Policy.
5. For Policy Name, type a unique name.
6. The policy document you’ll want to use is defined below. Note that the key ARN defined here is the CMK created in the previous step.
7. When you are done, choose Create Policy to save your completed policy.

Creating the EC2 Role

We now need to assign the policy to our EC2 instances. Additionally, we need to allow our instances access to communicate with the SSM endpoint. To do this, we’ll need to create an appropriate EC2 role:

1. Open the IAM console.
2. In the navigation pane, choose Roles, Create new role.
3. On the Select role type page, choose Select next to Amazon EC2.
4. On the Attach Policy page, select AmazonEC2RoleforSSM and the policy you created in the previous step.
5. On the Set role name and review page, type a name for the role and choose Create role.

Attaching the Role to the EC2 Instance

After creating the EC2 role, we then need to attach it to the target instance(s).

1. Navigate to the EC2 console.
2. Choose Instances in the navigation pane.
3. Select the target instance you intend to test the break-glass process on.
4. Choose Actions, choose Instance Settings and then Attach/Replace IAM role from the drop-down list.
5. On the Attach/Replace IAM role page, choose the role created in the previous step from the drop-down list.
6. After choosing the IAM role, proceed to the next step by choosing Apply.

Creating the Password Reset SSM Document

An AWS Systems Manager Document defines the actions that are performed on the target instance(s). We need to create a multi-step cross-platform document that can reset Linux or Windows passwords based on the target platform. To do this, perform the following:

1. Open the Amazon EC2 console.
2. In the navigation pane, choose Documents.
3. Choose Create Document.
4. Type a descriptive name for the document.
5. In the Document Type list, choose Command.
6. Delete the brackets in the Content field, and then paste the document below containing scripts for Windows and Linux. Remember to replace the CMKs and region in the both scripts.
7. Choose Create Document to save the document.

Congratulations! So far, you’ve set up the password reset functionality. Technically, you could stop here and you’d have a working break-glass capability, however we’re going to go one step further and add a clean-up process to remove the password from the parameter store for added security, as described below.

Creating the Lambda Function Policy

Our password clean-up process will use a Lambda function to delete the password from the parameter store. We’ll need to create an IAM policy to allow the Lamda function to do this.

1. Open the IAM console.
2. In the navigation column on the left, choose Policies.
3. At the top of the page, choose Create Policy.
4. On the Create Policy page choose Select on Create Your Own Policy.
5. For Policy Name, type a unique name.
6. The policy document to use is defined below.
7. When you are done, choose Create Policy to save your completed policy.

Creating the Lambda Function Role

We now need to attach the policy to a role that will be used by our lambda function.

1. Open the IAM console.
2. In the navigation pane, choose Roles, Create new role.
3. On the Select role type page, choose Select next to AWS Lambda.
4. On the Attach Policy page, select CloudWatchLogsFullAccess (for logging purposes) and the policy you created in the previous step.
5. On the Set role name and review page, type a name for the role and choose Create role.

Creating the Lambda Function

We now need to create the Lambda Function that will delete the password, and attach the role created in the previous step.

1. Open the AWS Lambda console.
2. Choose Create Function.
3. Choose Author from scratch.
4. On the triggers page, click Next.
5. Under Basic Information enter a name for your function and select the Python 2.7 runtime.
6. Under Lambda function code, enter the code below.
7. Under Lambda Function handler and role, choose the role you created in the previous step.
8. Expand advanced settings and extend the timeout to 90 seconds.
9. Choose Next and review the summary page then click Create Function.

Creating the CloudWatch Event

Almost there! The last step is to capture a successful execution of SSM Run Command, then trigger the previously created Lambda function. We can capture this using CloudWatch events:

1. Open the CloudWatch console.
2. In the navigation pane, choose Events.
3. Choose Create rule.
4. For Event Source, Choose Event Pattern, then choose Build custom event pattern, from the dropdown box.
5. Enter the following into the text box, replacing the document name with the SSM document that was created earlier.
6. For Targets, choose Add target and then choose Lambda function.
7. For Function, select the Lambda function that was created in the previous step.
8. Choose Configure details.
9. For Rule definition, type an appropriate name.
10. Choose Create rule.

That’s it! All that’s left is taking the process for a test drive. Let’s give it a shot.

Testing the Process

Assuming you’ve logged into the console with a user that has decrypt access for the CMK used, the following process can be used to access the password:

1. Open the Amazon EC2 console.
2. In the navigation pane under Systems Manager Services, choose Run Command.
3. Choose Run a command.
4. For Command document, choose the SSM Document created earlier.
5. For Target instances, choose an instance that has the previously created EC2 role attached. If you do not see the instance in this list, it might not have the correct role attached, or may not be able to access the SSM endpoint.
6. Choose Run, and then choose View results.
7. In the commands list, choose the command you just executed. If the command is still in progress, click the refresh icon in the top right corner of the console.
8. When the Status column shows Success, click the Output tab.
9. The output tab will display successful execution of both plugins described in our SSM document. If we click View Output on both we’ll notice that one didn’t execute due to not meeting the platform precondition we set. The other plugin should show that the password reset executed successfully.

1. In the navigation pane under Systems Manager Shared Resources, choose Parameter Store.
2. Assuming 60 seconds hasn’t elapsed (because our clean-up function will kick in and delete the password) there should be a parameter called pwd-<instance-ID>. After selecting the parameter, the Description tab below will show a SecureString.
3. Click on Show to view the password.

You can now use this password to access the administrator/root account of your instance. Assuming the password clean-up script is configured correctly, the password should disappear from the parameter store within 60 seconds of the Run Command completing.

Conclusion

The above process provides a simple and secure method for emergency access to both Windows and Linux systems, without the complex process and inherent risk of a traditional break-glass system. Additionally, this method has no running systems, providing a break-glass capability at nearly no cost.