Remove/Modify Specific AWS Tags from the Environment- PowerShell

Why use TAGs

To help you manage your instances, images, and other Amazon EC2 resources, you can optionally assign your own metadata to each resource in the form of tags. This topic describes tags and shows you how to create them.

(Ref: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html)

Problem :

Sometimes tags are applied in environments prior to developing a tagging strategy. The problem in exponentially increased with the size of the environment and the number of users creating resources.

Currently we are looking for a solution to remove specific unwanted tags from EC2 instances or modify the tag values which are incorrect.

For this purpose , the below mentioned script was developed that solves the problem for AWS.

Solution :

The below mentioned script performs the following tasks

  • Get the list of all the EC2 instances in the tenant
  • Loop through all the EC2 instances
  • Get values of all the tags in the environment
  • Check each Tag Key and Tag Value.
  • Modify of remove the tag value ( based on requirement )

Code:

#Set up the AWS profile using the Access Key and Secret Key

Set-AWSCredential -AccessKey AccessKey -SecretKey SecretKEy -StoreAs ProfileName

#Getting the list of all the instances in the Tenant

$instances = (Get-EC2Instance -ProfileName ProfileName -Region RegionName).Instances

$tagkeytoremove = 'TAG1' # Declaring the TAG Key to remove / modify

$tagvaluetoremove = 'ChangePLease' # Declaring the Tag Value to Remove / Modify

$NewTagValue = "NewTagValue" # Declaring the new tag value.

Foreach ( $instance in $instances ) # Looping through all the instances
{
    $OldTagList = $instance.tags
    foreach ($tag in $OldTagList) # Looping through all the Tags
    {
        if($tag.key -ceq $tagkeytoremove -and $tag.Value -ceq $tagvaluetoremove ) # Comparing the TAG Key and Values
        {
            Remove-EC2Tag -Resource $instances.instanceid -Tag $tag -Force # Removing the Old Tag Key Value Pair
            New-EC2Tag -Resource $instances.instanceid -Tag @{ Key=$tag.key;Value=$NewTagValue} -Force #Adding the New Tag Key Value pair.

        }
    }
} # Loop Ends

 

Disk Space Reporting through Lamba Functions- Windows servers

Solution Objective:

The solution provides detailed report related to hard disk space for all the Windows Ec2 instances in the AWS environment.

Requirements:

Mentioned below are the requirements the solution should be able to fulfil.

  • Gather information related to all mount points in all the Windows EC2 instances in the environment.
  • Able to generate cumulative report based on all instances in the environment.

3. Assumptions:

The following assumptions are considered

  • All the EC2 instances have SSM agent installed.
  • The personnel responsible for the configuration have some understanding of IAM Roles, S3 buckets and lambda functions

4. Solutions Description:

The following services provided by Amazon will be utilized to generate the report

  • PowerShell Scripts
  • AWS S3
  • AWS Lambda
  • AWS IAM Roles
  • Maintenances Windows

4.1      Linux Shell Script.

PowerShell Script will be utilized to generate information about the instance and the mount points space utilization.
Mentioned below script needs to be executed on all Windows Ec2 instances to generate the mount point information.

$instanceId = Invoke-WebRequest -Uri http://169.254.169.254/latest/meta-data/instance-id -UseBasicParsing
$instanceId.content
Get-WmiObject Win32_logicaldisk | select DeviceID,Size,Used,FreeSpace,PlaceHolder,VolumeName | ft -Autosize

4.1      AWS S3

The result of the shell script will be posted to an S3 bucket for further use.
The EC2 instances will need write access to the nominated S3 bucket for certificate Maintenance.
S3 Bucket Name: eomreport ( sample name )

4.2      AWS Lambda Functions

Lambda Functions will be used to perform the following activities.

  • Acquire the result of the Shell script from the S3 bucket
  • Generate a Report
  • Email the report to the relevant recipient

The Lambda Functions would need read access to the S3 bucket and access to AWS SES to send emails to recipients.
Mentioned below is the Lambda Functions that performs the mentioned above tasks.

import boto3
import codecs
import pprint
from datetime import datetime, date, time
def lambda_handler(event,Context):
    s3 = boto3.resource('s3')
    mybucket = s3.Bucket('diskspacewindows')
    resulthtml = ["<h1>Report : Hard disk Space Client Name </h1>"] # Adds heading to the email body
    resulthtml.append('<html><body><table border="1">') # Creates a table
    resulthtml.append('<tr><td><b>InstanceID</b></td><td><b>Drive Letter</b></td><td><b> FreeSpace</b></td><td><b>Total Space </b></td></b></tr>')
    for file_key in mybucket.objects.all():
        complete_string = str(file_key)
        search = "stdout"
        check = complete_string.find(search)
        if check > 0 :
            body = file_key.get()['Body'].read().decode('utf-8')
            complete=body.splitlines() #splits data into lines.
            id="".join(complete[0])
            details=complete[4:]
            resulthtml.append(("<td>'{}'</td><td></td><td></td><td></td></tr>").format(id)) # for the HTML email to be sent.
            for line in details:
                    output_word=line.split()
                    dstr="".join(line)
                    #print(output_word)
                    #print(len(output_word))
                    if len(output_word) > 0:
                      resulthtml.append(("<td></td><td>'{}'</td><td>'{}'</td><td>'{}'</td></tr>").format(output_word[0],output_word[1],output_word[2])) # for the HTML email to be sent.
    resulthtml.append('</table></body></html>')
    final=str("".join(resulthtml))
    final=final.replace("'","")
    print(final)
    sender = "syed.naqvi@kloud.com.au"
    recipient = "syed.naqvi@kloud.com.au"
    awsregion = "us-east-1"
    subject = "Client Hard Disk Space - Windows "
    charset = "UTF-8"
    mylist="mylist update"
    client = boto3.client('ses',region_name=awsregion)
    try:
        response = client.send_email(
           Destination={
               'ToAddresses': [
                   recipient,
                ],
            },
         Message={
                  'Body': {
                      'Html': {
                        'Charset': charset,
                        'Data': final,
                             },
                    'Text': {
                     'Charset': charset,
                     'Data': mylist,
                    },
                },
                'Subject': {
                    'Charset': charset,
                    'Data': subject,
                },
            },
            Source=sender,
        )
    # Display an error if something goes wrong.
    except Exception as e:
        print( "Error: ", e)
    else:
       print("Email sent!")

 
4.1 AWS IAM Roles
Roles will be used to grant

  • AWS S3 write access to all the EC2 instances as they will submit the output of the  the S3 bucket
  • AWS SES access to Lambda Functions to send emails to relevant recipients.

4.2 AWS SES

Amazon Simple Email Service (Amazon SES) evolved from the email platform that Amazon.com created to communicate with its own customers. In order to serve its ever-growing global customer base, Amazon.com needed to build an email platform that was flexible, scalable, reliable, and cost-effective. Amazon SES is the result of years of Amazon’s own research, development, and iteration in the areas of sending and receiving email.( Ref. From https://aws.amazon.com/ses/).
We would be utilizing AWS SES to generate emails using AWS lambda.
The configuration of the Lambda functions can be modified to send emails to a distribution group to provide Certificate reporting, or it can be used to send emails to ticketing system in order to provide alerting and ticket creation in case a certificate expiration date crosses a configured threshold.

5Solution Configuration

5.1 Configure IAM Roles

The following Roles should be configured

  • IAM role for Lambda Function.
  • IAM for EC2 instances for S3 bucket Access

5.1.1 Role for Lambda Function

Lambda function need the following access

  • Read data from the S3 bucket
  • Send Emails using Amazon S3

To accomplish the above the following policy should be created and attached to the IAM Role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1501474857000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::S3BucketName/*"
            ]
        },
        {
            "Sid": "Stmt1501474895000",
            "Effect": "Allow",
            "Action": [
                "ses:SendEmail"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

6.1.2  Role for EC2 instance

All EC2 instances should have access to store the Shell output in the S3 bucket.
To accomplish the above , the following policy should be assigned to the EC2 roles

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1501475224000",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::eomreport"
            ]
        }
    ]
}

6.2 Configure Maintenance Window.

The following tasks need to be performed for the maintenance window

  • Register a Run Command with Run-Shell Script using the script in section 4.1
  • Register targets based on the requirements
  • Select the schedule based on your requirement

Maintenance Window Ref : 
http://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html

6.3 Configure Lambda Function:

The following tasks need to be performed for the Lambda Function

  • Create a blank lambda function with the S3 put event as the trigger\lambda function
  • Click on Next
  • Enter the Name and Description
  • Select run time Python 3.6
  • Copy and paste the lambda function mentioned in section 4.3

    6.4 Configuring AWS SES

The following tasks need to be completed before the execution of the Run-commands.

  • Email Addresses should be added to the AWS SES section of the tenant.
  • The email addresses should be verified.

 7. Result:

Based on the above configuration, whenever the run command is executed, the following report is generated and sent to the nominated email account.

Removing Specific Azure Tags – PowerShell

Azure Tags

You apply tags to your Azure resources to logically organize them by categories. Each tag consists of a name and a value. For example, you can apply the name “Environment” and the value “Production” to all the resources in production.
After you apply tags, you can retrieve all the resources in your subscription with that tag name and value. Tags enable you to retrieve related resources from different resource groups. This approach is helpful when you need to organize resources for billing or management.
 

Problem:

Sometimes tags are applied in environments prior to developing a tagging strategy. The problem in exponentially increased with the size of the environment and the number of users creating resources.
Currently we are looking for a solution to remove specific unwanted tags from Virtual Machines.
For this purpose , the below mentioned script was developed that solves the problem.

Solution :

The below mentioned script performs the following tasks

  • Get the list of all the VMs based on the Scope
  • Get all the Tag Values for each VM in a $VMTags variable.
  • Copy all the values from $VMtags to $newtags except the $tagtoremove value.
  • Configure the resources with the $newtags values using Set-AzureRmResource command.

 

Code:

#Getting the list of VMs based on the resource group. THe Scope can be changed to include more resources.
$VMS = Get-AzureRmVM -ResourceGroupName ResourceGroupName
#Details of the tag to remove are stored in the $TagtoRemove variable.
$TagtoRemove = @{Key="TestVmName";Value="abcd"}
foreach ($VM in $VMs)
{
 $VMtags = $VM.tags # Getting the list of all the tags for the VM.
 $newtag = @{} # Creating a new Hashtable variable to store the Tag Values.
 foreach ( $KVP in $VMtags.GetEnumerator() )
 {
 Write-Host "`n`n`n"
 If($KVP.Key -eq $TagtoRemove.Key)
 {
 write-host $TagtoRemove.Key "exists in the "$VM.Name " will be removed `n"}
 Else
 {
 $newtag.add($KVP.Key,$KVP.Value) # Adding all the tags in the $newtag Variable except the $TagtoRemove.key values
 }
}
 #Updating the Virtual machine with the updated tag values $newtag.
 Set-AzureRmResource -ResourceGroupName $VM.ResourceGroupName -ResourceName $VM.Name -Tag $newtag -Force -ResourceType Microsoft.Compute/VirtualMachines
}

 
 

Key Vault Secrets and ARM Templates

What is Azure Key Vault

Azure Key Vault helps safeguard cryptographic keys and secrets used by cloud applications and services. By using Key Vault, you can encrypt keys and secrets (such as authentication keys, storage account keys, data encryption keys, .PFX files, and passwords) using keys protected by hardware security modules (HSMs).
Key Vault streamlines the key management process and enables you to maintain control of keys that access and encrypt your data. Developers can create keys for development and testing in minutes, and then seamlessly migrate them to production keys. Security administrators can grant (and revoke) permission to keys, as needed.
Anybody with an Azure subscription can create and use key vaults. Although Key Vault benefits developers and security administrators, it could be implemented and managed by an organization’s administrator who manages other Azure services for an organization. For example, this administrator would sign in with an Azure subscription, create a vault for the organization in which to store keys, and then be responsible for operational tasks, such as:

  • Create or import a key or secret
  • Revoke or delete a key or secret
  • Authorize users or applications to access the key vault, so they can then manage or use its keys and secrets
  • Configure key usage (for example, sign or encrypt)
  • Monitor key usage

This administrator would then provide developers with URIs to call from their applications, and provide their security administrator with key usage logging information.
( Ref: https://docs.microsoft.com/en-us/azure/key-vault/key-vault-whatis ).
 

Current Scenario of the Key Vault.

In the current scenario , we are utilizing Key Vault for the provisioning of arm resources. Instead of any username and passwords , the template only contains references to these values stored in Azure Key Vault.
Theses secrets are extracted while the resources are being deployed using the template and the parameter file together.

Utilizing the Key Vault.

The following tasks are involved in utilizing the Key Vault

  • Creating the key Vault.
  • Adding Keys and Secrets in the Vault
  • Securing the Key Vault
  • Referencing Keys

Creating a Key Vault

Step 1 : Login to Azure Portal and click on All Services  and select Key Vault
1.png
Step 2 : Click on Add and enter the following details and click on Create

  • Key Vault Name
  • Subscription
  • Resource Group
  • Pricing Tier

2.PNG
Step 3 : Select the Key Vault Name
Step 4 : Select Secrets
Step 5 : Click on Generate/Import
3
Step 6: Select Manual in upload options
Step 7: Enter the following information.

  • Name of the Secret ( Like MyPassword )
  • Value of the Secret ( example P@ssword1)
  • Set Activation Date ( If required )
  • Set Expiration Date ( If required)

Step 8 : Select Yes on the Enabled option
Step 9: Click on Create
4

Securing the Key Vault

Step 1: Select the Newly created Key Vault
Step 2:  Select Access Policies
Step 3 : Select click to show advanced access policies
5
Step 4: Select the checkboxes as shown in the snapshot below.
Step 5: Click on Add New
6
Step 6: Select Secret management in the configure from template option.
Step 7:  Select the Principal ( name of the resource which needs access to the secret.)
Step 8: Select the permissions required from the Secret Permissions
Step 9: Select OK.
8
 

Referencing the Secrets.

Currently we are referencing the secrets stored in the Key vault using the arm templates.
A parameter type “securestring” is created in the parameter section of the arm template armtemplate.json file
kvsecret
We add the parameter in the parameters file of the template armtemplate.parameters.json with the following details

  • ID of the key vault ( resource ID of the KeyVault under the properties section )
  • name of the secret to extract (Mypassword)

kvsecretparamters.PNG
 
 

Summary:

Based on the above example , we achieved the following

  • Secure information ( usernames and passwords ) are not mentioned anywhere in the template or the parameter files
  • The secure values will be acquired when the template is running
  • The values are only accessible to those who have access to the Key vault.

 

Azure Policy for Auditing VM Tags

Azure Policy

Azure Policy is a service in Azure that you use to create, assign and, manage policy definitions. Policy definitions enforce different rules and actions over your resources, so those resources stay compliant with your corporate standards and service level agreements. Azure Policy runs an evaluation of your resources, scanning for those not compliant with the policy definitions you have.
Ref: https://docs.microsoft.com/en-us/azure/azure-policy/azure-policy-introduction
Mentioned below are the steps required for configuring Azure policies to monitor tags
Step 1 : Login on to the Azure Portal and search for Policy.
1
Step 2: Click on Definitions in the next screen.
2
Step 3 : Click on +Policy Definition.
3
Step 4: Enter the following details in the next screen.

  • Definition Location
  • Policy Name
  • Category

4
 
Step 5: Paste the below mentioned code in the Policy rules and Parameters section
( The code checks for all VMs with tags , and selects the VMS which do not have the tags mentioned in the script configured ).
Step 6 : Click on Save

{
 "policyRule": {
    "if": {
        "allof": [
            {
            "field": "type",
            "equals": "Microsoft.Compute/VirtualMachines"
             },
            {
            "anyOf": [
            {
             "field": "tags.Tag1Name",
             "exists": "false"
             },
            {
            "field": "tags.Tag2Name",
            "exists": "false"
             },
            {
             "field": "tags.Tag3Name",
             "exists": "false"
            },
            {
            "field": "tags.Tag4Name",
             "exists": "false"
             },
             {
             "field": "tags.Tag5Name",
             "exists": "false"
             }
         ]
     }
    ]
 },
 "then": {
    "effect": "audit"
    }
 }
}

Step 7 : Click on Assignments
5
Step 8 : Click on Assign Policy
Step 9 : Complete the form with the following values.

  • Scope ( Subscription / Resource Group/ Management Group )
  • Exclusions ( Exclusions within the selected Scope )
  • Select the policy definition just created.
  • Enter an Assignment name
  • Description of the Assignment.
  • Name of the User ( by default it takes the name of the user account used for the assignment process.

6
Step 10 : Click on Assign

OMS Patching – Active Directory Groups

OMS : Update Management Solution in Azure 
The Update Management solution in Azure automation allows you to manage operating system updates for your Windows and Linux computers deployed in Azure, on-premises environments, or other cloud providers. You can quickly assess the status of available updates on all agent computers and manage the process of installing required updates for servers.
Ref: https://docs.microsoft.com/en-us/azure/automation/automation-update-management
This blog details the steps required for configuring patching through Azure OMS service.
Assumptions: 

  • The OMs client is installed on all the VMs.
  • OMS is configured to import Active Directory Groups memberships from client

AD Groups Memberships
 
Step 1 : Login to Azure Portal , Search for Log Analytics and click on it.
1
Step 2: Select the workspace you want to utilize for OMS patching configuration.
2
Step 3: Click on OMS Portal
3
Info : The following dashboard will be displayed.
4.PNG
Step 4: Click on System Update Assessment from the Dashboard
5
Step 5 : You will be navigated to different dashboard as shown in snapshot below. Click on “ Manage Update Deployments” option from the dashboard screen.
6
Info  : Displayed dashboard have different schedules. You can browse through Scheduled, Running and Completed tab from left menu to see activities.
7
Step 6 : Click on the Add Option
7
Step 7 : Add the details of the Name of the group and click on Computers
Step 8 : Select the group with type Active Directory based on the configuration
Important :

  • The Active directory group will only be visible if a machine with the OMS agent is part of it.
  • It does take some time to reflect the group memberships in the OMS portal.

8.PNG
Step 9 : Configure the schedule with the following details.

  • Time zone
  • Recurrence
  • Start Time
  • Duration.

Step 10 : Click on Save
9

Patching LINUX EC2 through SSM

This blog deals with configuring patches for Linux EC2 instances through AWS Systems Manager (SSM).
Mentioned below is the link for patching Windows-based EC2 instances using SSM
https://blog.kloud.com.au/2017/05/08/patching-ec2-through-ssm/
The configuration has three major sections

  • EC2 instance configuration for patching
  • Default Patching Baseline Configuration
  • Maintenance Window configuration.

1 Instance Configuration

We will start with the First section which is configuring the Instances to be patched. This requires the following tasks.

  1. Create Amazon EC2 Role for patching with two policies attached
    • AmazonEC2RoleForSSM
    • AmazonSSMFullAccess
  2. Assign Roles to the EC2 Instances
  3. Configure Tags to ensure patching in groups.

Mentioned below are the detailed steps for the creation of an IAM role for Instances to be Patched using Patch Manager.
Step 1: Select IAM —–> Roles and Click on Create New Role
1
Step 2: Select Role Type —-> Amazon EC2
2.PNG
Step 3: Under Attach Policy Select the following and Click Next

  • AmazonEC2RoleForSSM
  • AmazonSSMFullAccess

3.PNG
Step 4: Enter the Role Name and Select Create Role (At the bottom of the page)
4.PNG
Now you have gone through the first step in your patch management journey.
Instances should be configured to use the above created role to ensure proper patch management. (or any roles which has AmazonEC2RoleforSSM and AmazonSSMFullAccess policies attached to it.)
5.PNG
We need to group our AWS hosted servers in groups cause no one with the right frame of mind wants to patch all the servers in one go.
To accomplish that we need to use Patch Groups (explained later).
Patch groups can be created by simply creating a tag for EC2 instances with a  tag key of “Patch Group” which is case sensitive.
We can use Group01 and Group02 as the value for the “Patch Group” tag.
To utilize patch groups, all EC2 instances should be tagged to support cumulative patch management based on Patch Groups.
Congratulations, you have completed the first section of the configuration. Keep following just two to go.

2 Default Patch Baseline Configuration

Patch baseline configuration controls what patches are to be installed on the instances based on the following classification of Patches

  • Product Type  : OS Version
  • Classification: CriticalUpdates, SecurityUpdates, ServicePacks, UpdateRollUps
  • Severity : Critical , Important , etc.

Mentioned below are the steps for creating patch baseline for AmazonLinux EC2 instances.
Note: The process is quite similar for Ubuntu, Red Hat Enterprise Linux and Windows.
Step 01: Select EC2 —> Select Patch Baselines (under the Systems Manager Services Section) and Click on Create Patch Baseline
pb01
Step 03: Fill in the details

  • Name : MyAmazonLinuxPatchBaseline
  • Description: MyAmazonLinuxPatchBaseline
  • Operating System : AmazonLinux ( Choose operating system based on the workload )

pb02
Step 04 : Configure Approval Rules based on the patching policy.
pb06
Step 05: Complete Patch Exceptions and click on Create Patch Baseline
pb031.png
Step 06: Select the created Patch baseline
Step 07: Go to Actions and click Set Default Patch Baseline
pb04
Step 08: Click on Set Default Patch Baseline
pb05
At this point, the instances to be patched are configured and we have also configured the patch policies. The next section we provide AWS the when (Date and Time) and what (task) of the patching cycle.

3 Maintenance Windows Configuration

As the name specifies, Maintenance Windows give us the option to Run Tasks on EC2 Instances on a specified schedule.
What we wish to accomplish with Maintenance Windows is to Run a Command (Apply-AWSPatchBaseline), but on a given schedule and on a subset of our servers. This is where all the above configurations gel together to make patching work.
Configuring Maintenance windows consist of the following tasks.

  • IAM role for Maintenance Windows
  • Creating the Maintenance Window itself
  • Registering Targets (Selecting servers for the activity)
  • Registering Tasks (Selecting tasks to be executed)

Mentioned below are the detailed steps for configuring all the above.
Step 01: Create a Role with the following policy attached

  • AmazonSSMMaintenanceWindowRole

9.PNG
Step 02: Enter the Role Name and Role Description
10.PNG
Step 03: Click on Role and copy the Role ARN
Step 04: Click on Edit Trust Relationships
11.PNG
Step 05: Add the following values under the Principal section of the JSON file as shown below

“Service”: “ssm.amazonaws.com”

Step 06: Click on Update Trust Relationships (on the bottom of the page)
12.PNG
At this point the IAM role for the maintenance window has been configured. The next section details the configuration of the maintenance window.
Step 01: Click on EC2 and select Maintenance Windows (under the Systems Manager Shared Resources section)
13.PNG
Step 02: Enter the details of the maintenance Windows and click on Create Maintenance Windows
14.PNG
At this point the Maintenance Window has been created. The next task is to Register Targets and Register Tasks for this maintenance window.
Step 01: Select the Maintenance Window created and click on Actions
Step 02: Select Register Targets
15.PNG
Step 03: Enter Target Name, Description, Owner Information and select the Tag Name and Tag Value
Step 04: Select Register Targets
registertargets
At this point the targets for the maintenance window have been configured. This leaves us with the last activity in the configuration which is to register the tasks to be executed in the maintenance window.
Step 01: Select the Maintenance Window and Click on Actions
Step 02: Select Register run command Task
registerruncmd02
Step 03: Select AWS-RunPatchBaseline from the Document section
registerruncmd03.PNG
Step 04: Click on Registered targets and select the instances based on the Patch Group Tag
Step 05: Select Operation SCAN or Install based on the desired function (Keep in mind that an Install will result in a server restart).
Step 06: Select the MaintenanceWindowsRole
Step 07: Click on Register Tasks
19.PNG
After completing the configuration, the Registered Task will run on the Registered Targets based on the schedule specified in the Maintenance Window.
The status of the Maintenance Window can be seen in the History section (as Shown below)
20.PNG
Hope this guide does get you through the initial patching configuration for your EC2 instances in Amazon.
Thanks for Reading.

Update FSTAB on multiple EC2 instances using Run Commands

Scenario:

  • Customer Running multiple Linux Ec2 instance in AWS.
  • Customer reports that Instances are loosing mount points after a reboot.

Solution :

The resolution requires to update the fstab file on all the instances.
fstab is a system configuration file on Linux and other Unix-like operating systems that contains information about major filesystems on the system. It takes its name from file systems table, and it is located in the /etc directory ( ref : http://www.linfo.org/etc_fstab.html)
In order to update files on multiple servers we will utilize the following

  • ECHO command with append parameter (>>) to update the text file through shell
  • SSM Run Command to execute the command on multiple machines.

Note : All the concerned EC2 instances should have SSM manager configured.
Step 1 : Login to the AWS Console and click  EC2
click on ec2
 Step 2: Click on Run Command on the Systems Manager Services section
click on Run command
Step 3: Click on Run Command in the main panel
click-on-run-command-2.png
Step 4: Select Run-Shell Script
select run-shell script
Step 5: Select Targets 
Note : Targets can be selected manually or we can use Tags to perform the same activity on multiple instances with the matching tag.
select targets and stuff
Step 6:
Enter the following information :

  • Execute on : Specifies the number of target the commands can be executed concurrently. Concurrently running commands save time in execution.
  • Stop After : 1 errors
  • Timeout ( seconds) : leave the default 600 seconds

Step 7: Select the commands section and paste the command

echo '10.x.x.x:/ /share2 nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev 0 0' >> /etc/fstab
echo '10.x.x.x:/ /share1 nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev 0 0' >> /etc/fstab
echo '192.x.x.x:/ /backup1 nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev 0 0' >> /etc/fstab
echo '172.x.x.x:/ /backup2 nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev 0 0' >> /etc/fstab           

 
Step 8 : Click on Run click on run
Step 9: Click on command id to get update regarding the execution success of failure
click on command id to check the status of the coomand

Disk Space Reporting through Lamba Functions- Linux servers

Solution Objective:

The solution provides detailed report related to hard disk space for all the Linux Ec2 instances in the AWS environment.

Requirements:

 
Mentioned below are the requirements the solution should be able to fulfil.

  • Gather information related to all mount points in all the Linux EC2 instances in the environment.
  • Able to generate cumulative report based on all instances in the environment.

3. Assumptions:

The following assumptions are considered

  • All the EC2 instances have SSM agent installed.
  • The personnel responsible for the configuration have some understanding of IAM Roles, S3 buckets and lambda functions

4. Solutions Description:

The following services provided by Amazon will be utilized to generate the report

  • Linux shell Scripts
  • AWS S3
  • AWS Lambda
  • AWS IAM Roles
  • Maintenances Windows

4.1      Linux Shell Script.

Linux Shell Script will be utilized to generate information about the instance and the mount points space utilization.
Mentioned below script needs to be executed on all Linux Ec2 instances to generate the mount point information.

curl http://169.254.169.254/latest/meta-data/instance-id # Prints the Instance ID
printf "\n" # Adds line
df # provides details of the mount point

4.1      AWS S3

The result of the shell script will be posted to an S3 bucket for further use.
The EC2 instances will need write access to the nominated S3 bucket for certificate Maintenance.
S3 Bucket Name: eomreport ( sample name )

4.2      AWS Lambda Functions

Lambda Functions will be used to perform the following activities.

  • Acquire the result of the Shell script from the S3 bucket
  • Generate a Report
  • Email the report to the relevant recipient

The Lambda Functions would need read access to the S3 bucket and access to AWS SES to send emails to recipients.
Mentioned below is the Lambda Functions that performs the mentioned above tasks.

import boto3
import codecs
import pprint
from datetime import datetime, date, time
def lambda_handler(event,Context):
    s3 = boto3.resource('s3')
    mybucket = s3.Bucket('eomreport')
    resulthtml = ["<h1>Report : Hard disk Space </h1>"] # Adds heading to the email body
    resulthtml.append('<html><body><table border="1">') # Creates a table
    resulthtml.append('<tr><td><b>InstanceID</b></td><td><b>Available Space</b></td><td><b>Used Space</b></td><td><b>Use %</b></td></td><td><b>Mounted on</b></td></b></tr>')
    for file_key in mybucket.objects.all():
        complete_string = str(file_key)
        search = "stdout"
        check = complete_string.find(search)
        if check > 0 :
            body = file_key.get()['Body'].read().decode('utf-8')
            complete=body.splitlines() #splits data into lines.
            id="".join(complete[0])
            hr=complete[1]
            hr2=hr.split()
            hr2.append("InstanceID")
            hstr=",".join(hr2)
            details=complete[2:]
            for line in details:
                    output_word=line.split()
                    dstr="".join(line)
                    resulthtml.append(("<td>'{}'</td><td>'{}'</td><td>'{}'</td><td>'{}'</td><td>'{}'</td></tr>").format(id,output_word[3],output_word[2],output_word[4],output_word[5])) # for the HTML email to be sent.
    resulthtml.append('</table></body></html>')
    final=str("".join(resulthtml))
    final=final.replace("'","")
    print(final)
    sender = "email@email.com"
    recipient = "email@email.com"
    awsregion = "us-east-1"
    subject = "Certificate Update list"
    charset = "UTF-8"
    mylist="mylist update"
    client = boto3.client('ses',region_name=awsregion)
    try:
        response = client.send_email(
           Destination={
               'ToAddresses': [
                   recipient,
                ],
            },
         Message={
                  'Body': {
                      'Html': {
                        'Charset': charset,
                        'Data': final,
                             },
                    'Text': {
                     'Charset': charset,
                     'Data': mylist,
                    },
                },
                'Subject': {
                    'Charset': charset,
                    'Data': subject,
                },
            },
            Source=sender,
        )
    # Display an error if something goes wrong.
    except Exception as e:
        print( "Error: ", e)
    else:
       print("Email sent!")

 
4.1 AWS IAM Roles
Roles will be used to grant

  • AWS S3 write access to all the EC2 instances as they will submit the output of the  the S3 bucket
  • AWS SES access to Lambda Functions to send emails to relevant recipients.

4.2 AWS SES

Amazon Simple Email Service (Amazon SES) evolved from the email platform that Amazon.com created to communicate with its own customers. In order to serve its ever-growing global customer base, Amazon.com needed to build an email platform that was flexible, scalable, reliable, and cost-effective. Amazon SES is the result of years of Amazon’s own research, development, and iteration in the areas of sending and receiving email.( Ref. From https://aws.amazon.com/ses/).
We would be utilizing AWS SES to generate emails using AWS lambda.
The configuration of the Lambda functions can be modified to send emails to a distribution group to provide Certificate reporting, or it can be used to send emails to ticketing system in order to provide alerting and ticket creation in case a certificate expiration date crosses a configured threshold.

5Solution Configuration

5.1 Configure IAM Roles

The following Roles should be configured

  • IAM role for Lambda Function.
  • IAM for EC2 instances for S3 bucket Access

5.1.1 Role for Lambda Function

Lambda function need the following access

  • Read data from the S3 bucket
  • Send Emails using Amazon S3

To accomplish the above the following policy should be created and attached to the IAM Role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1501474857000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::S3BucketName/*"
            ]
        },
        {
            "Sid": "Stmt1501474895000",
            "Effect": "Allow",
            "Action": [
                "ses:SendEmail"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

6.1.2  Role for EC2 instance

All EC2 instances should have access to store the Shell output in the S3 bucket.
To accomplish the above , the following policy should be assigned to the EC2 roles

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1501475224000",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::eomreport"
            ]
        }
    ]
}

6.2 Configure Maintenance Window.

The following tasks need to be performed for the maintenance window

  • Register a Run Command with Run-Shell Script using the script in section 4.1
  • Register targets based on the requirements
  • Select the schedule based on your requirement

Maintenance Window Ref : 
http://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html

6.3 Configure Lambda Function:

The following tasks need to be performed for the Lambda Function

  • Create a blank lambda function with the S3 put event as the trigger\lambda function
  • Click on Next
  • Enter the Name and Description
  • Select run time Python 3.6
  • Copy and paste the lambda function mentioned in section 4.3

    6.4 Configuring AWS SES

The following tasks need to be completed before the execution of the Run-commands.

  • Email Addresses should be added to the AWS SES section of the tenant.
  • The email addresses should be verified.

 7. Result:

Based on the above configuration, whenever the run command is executed, the following report is generated and sent to the nominated email account.

InstanceIDAvailable SpaceUsed SpaceUse %Mounted on
i-sampleID112398420818326040.02/
i-sampleID1772098000/dev
i-sampleID1774628800/dev/shm
i-sampleID17721456248320.01/run
i-sampleID1774628800/sys/fs/cgroup
i-sampleID212222057235962400.03/
i-sampleID2772062800/dev
i-sampleID2774628080.01/dev/shm
i-sampleID275328722134160.03/run
i-sampleID2774628800/sys/fs/cgroup
i-sampleID281554964162834040.17/sit
i-sampleID283340832144975360.15/uat
i-sampleID2154926000/run/user/1000
i-sampleID312398366418331480.02/
i-sampleID3772098000/dev
i-sampleID3774628800/dev/shm
i-sampleID37721448248400.01/run
i-sampleID3774628800/sys/fs/cgroup

 

Certificate Management using PowerShell and Lambda Functions

Certificate Management

1. Why Certificate Management is required.

Certificates installed on client machines are one of the critical resources in the client’s infrastructure. Monitoring certificates is critical to any company willing to successfully provide Certificate Management service. The process of manually reporting certificate details is tedious is time consuming, so it better to automate it.
The following document will explain the steps to configure AWS services to provide certificate management for customers with AWS hosted infrastructure.

2. Solution Requirements

Mentioned below are the requirements that should be fulfilled to provide proper certificate management services.

  • The Solution should scan all instances for installed Certificates
  • The solution should be able to generate Reports which include the following information
    • Instance ID of the machine the certificate is installed on
    • Subject of the Certificate
    • Days till expiration
  • The solution should be able to generate an alert if the “days to expire” falls below 30 Days.
  • The alert should be able to create a ticket which can later be worked on to rectify the situation

Note: The requirements for certificate management are prone to change based on the requirements for the clients.

3. Assumptions:

The following assumptions are considered while configuring certificate management:

  • All the EC2 instances are running proper windows PowerShell.
  • All the EC2 instances have SSM agent installed.
  • The personnel responsible for the configuration of certificate management have some understanding of IAM Roles, S3 buckets and lambda functions

4. Solutions Description:

The following services provided by Amazon will be utilized to provide certificate management

  • Windows PowerShell
  • AWS S3
  • AWS Lambda
  • AWS IAM Roles
  • Maintenances Windows

4.1      Windows PowerShell

PowerShell will be utilized to generate information about the certificate and instance id for the AMI.
Mentioned below script needs to be executed on all AMI’s to generate the certificate information.

Set-Location Cert:\LocalMachine\my #Sets the location
$CertificateDetails = get-childitem # Get all the machine certificates
$instanceId = Invoke-WebRequest -usebasicparsing -Uri http://169.254.169.254/latest/meta-data/instance-id # Gets the instance ID
foreach ($i in $CertificateDetails) # writes the output with certificate Thumbprint , Subject, Instance ID and Expiration Date
{ 
Write-host "Thumbprint=" $i.Thumbprint " Expiration Date="$i.NotAfter " InstanceID ="$instanceID.Content" Subject="$i.Subject 
}

The following output is generated as a result of the PowerShell script.
 

Thumbprint= ****************** Expiration Date= 1/1/2040 10:59:59 AM InstanceID = i-******* Subject= CN=SubjectName Blah Blah Blah
Thumbprint= ****CYXZ********** Expiration Date= 1/1/2040 10:59:59 AM InstanceID = i-******* Subject= CN=SubjectName Blah Blah Blah
Thumbprint= ****************** Expiration Date= 1/1/2040 10:59:59 AM InstanceID = i-******* Subject= CN=SubjectName Blah Blah Blah
Thumbprint= ****************** Expiration Date= 1/1/2040 10:59:59 AM InstanceID = i-******* Subject= CN=SubjectName Blah Blah Blah
Thumbprint= ****************** Expiration Date= 1/1/2040 10:59:59 AM InstanceID = i-******* Subject= CN=SubjectName Blah Blah Blah

Where * represents different values for different certificates.

4.1      AWS S3

The result of the PowerShell script will be posted to an S3 bucket for further use.
The EC2 instances will need write access to the nominated S3 bucket for certificate Maintenance.
S3 Bucket Name: CertificateManagement

4.2      AWS Lambda Functions

Lambda Functions will be used to perform the following activities.

  • Acquire the result of the PowerShell script from the S3 bucket
  • Process the data to determine the Days till expiration of all the certificates
  • Generate a Report
  • Email the report to the relevant recipient

The Lambda Functions would need read access to the S3 bucket and access to AWS SES to send emails to recipients.
Mentioned below is the Lambda Functions that performs the mentioned above tasks.

import boto3
import codecs
import pprint
from datetime import datetime, date, time
def lambda_handler(event,Context):
    s3 = boto3.resource('s3')
    mybucket = s3.Bucket('CertificateManagement')
    result=["this is the result","\n"]
    for file_key in mybucket.objects.all():
        print("\n")
        body = file_key.get()['Body'].read().decode('utf-8')
        output_lines=body.splitlines() #splits data into lines.
        resulthtml = ["<h1>Client Certificate Report</h1>"] # Adds heading to the email body
        resulthtml.append('<html><body><table border="1">') # Creates a table
        resulthtml.append('<tr><b><td>InstanceID</td><td>Thumbprint</td><td>Subject</td><td>Days to Expiration</td></b></tr>')
        try:
            for line in output_lines:
                x=1
                output_word=line.split() # splits every line into words
                cnlist=output_word[10:] # gets the complete subject name and adds into an array
                cnstr=" ".join(cnlist) # joins the array to create a string.
                dt = datetime.strptime(str(output_word[4]), "%m/%d/%Y") # Converts the Date values string into a date format.
                ct=datetime.now()
                difference=dt-ct # Gets the difference between the date value and current date.
                Values = str(difference) # converts the difference into string
                Valuelist=Values.split() # converts the string into arrays.
                Days = Valuelist[0] # selcts the first value in the array which is Number of days.
                result.append(("Certificate with '{}' will expire in '{}'").format(cnstr,difference)) # to display output when testing the script in the lambda section.
                resulthtml.append(("<td>'{}'</td><td>'{}'</td><td>'{}'</td><td>'{}'</td></tr>").format(output_word[9],output_word[1],cnstr,Days)) # for the HTML email to be sent.
            resulthtml.append('</table></body></html>')
        except:
            print("some errors encountered")
        for i in result: # prints the result in the testing window in the Lambda console
            print(i)
    myhtmllist = str(''.join(resulthtml)) # converts all the resulthtml into string to be used in Email.
    myhtmllist = myhtmllist.replace("'","") #removes the "'" from the data.
    textbody="this is a text body default"
    sender = "syed.naqvi@kloud.com.au"
    recipient = "syed.naqvi@kloud.com.au"
    awsregion = "us-east-1"
    subject = "Certificate Update list"
    charset = "UTF-8"
    client = boto3.client('ses',region_name=awsregion)
    try:
        response = client.send_email(
           Destination={
               'ToAddresses': [
                   recipient,
                ],
            },
         Message={
                  'Body': {
                      'Html': {
                        'Charset': charset,
                        'Data': myhtmllist,
                             },
                    'Text': {
                     'Charset': charset,
                     'Data': mylist,
                    },
                },
                'Subject': {
                    'Charset': charset,
                    'Data': subject,
                },
            },
            Source=sender,
        )
    # Display an error if something goes wrong.
    except Exception as e:
        print( "Error: ", e)
    else:
       print("Email sent!")

 4.1 AWS IAM Roles
Roles will be used to grant

  • AWS S3 write access to all the EC2 instances as they will submit the output of the PowerShell script in the S3 bucket
  • AWS SES access to Lambda Functions to send emails to relevant recipients.

4.2 AWS SES

Amazon Simple Email Service (Amazon SES) evolved from the email platform that Amazon.com created to communicate with its own customers. In order to serve its ever-growing global customer base, Amazon.com needed to build an email platform that was flexible, scalable, reliable, and cost-effective. Amazon SES is the result of years of Amazon’s own research, development, and iteration in the areas of sending and receiving email.( Ref. From https://aws.amazon.com/ses/).
We would be utilizing AWS SES to generate emails using AWS lambda.
The configuration of the Lambda functions can be modified to send emails to a distribution group to provide Certificate reporting, or it can be used to send emails to ticketing rsystem in order to provide alerting and ticket creation in case a certificate expiration date crosses a configured threshold.

5. Solution Work Flow

Mentioned below is the workflow for the tasks.
workflow

6Solution Configuration

6.1 Configure IAM Roles

The following Roles should be configured

  • IAM role for Lambda Function.
  • IAM for EC2 instances for S3 bucket Access

6.1.1 Role for Lambda Function

Lambda function need the following access

  • Read data from the S3 bucket
  • Send Emails using Amazon S3

To accomplish the above the following policy should be created and attached to the IAM Role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1501474857000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::S3BucketName/*"
            ]
        },
        {
            "Sid": "Stmt1501474895000",
            "Effect": "Allow",
            "Action": [
                "ses:SendEmail"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

6.1.2  Role for EC2 instance

All EC2 instances should have access to store the PowerShell output in the S3 bucket.
To accomplish the above , the following policy should be assigned to the EC2 roles

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1501475224000",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::CertificateMaintenance"
            ]
        }
    ]
}

6.2 Configure Maintenance Window.

The following tasks need to be performed for the maintenance window

  • Register a Run Command with Run-PowerShell Script using the script in section 4.1
  • Register targets based on the requirements
  • Select the schedule based on your requirement

Maintenance Window Ref : 
http://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html

6.3 Configure Lambda Function:

The following tasks need to be performed for the Lambda Function

  • Create a blank lambda function with the S3 put event as the triggerlambda function
  • Click on Next
  • Enter the Name and Description
  • Select run time Python 3.6
  • Copy and paste the lambda function mentioned in section 4.3

6.4 Configuring AWS SES

The following tasks need to be completed before the execution of the Run-commands.

  • Email Addresses should be added to the AWS SES section of the tenant.
  • The email addresses should be verified.

 7. Result:

Based on the above configuration, whenever the run command is executed, the following report is generated and sent to the nominated email account.
report

8. Next Steps:

We can modify the script to only report on Certificate expiring in 45 days or less.
We can also use SNS instead of SES to send out SMS to relevant teams as well.
That will be catered in another blog

Follow Us!

Kloud Solutions Blog - Follow Us!