First published on Nivlesh’s personal blog at https://nivleshc.wordpress.com
Background
I was recently at a customer site, to perform an environment review of their AWS real-estate. As part of this engagement, I was going to do an inventory of all their AWS resources. Superficially, this sounds like an easy task, however when you consider the various regions that resources can be provisioned into, the amount of work required for a simple inventory can easily escalate.
Not being a big fan of manual work, I started to look at ways to automate this task. I quickly settled on Ansible as the tool of choice and not long after, I had two Ansible playbooks ready (the main and the worker playbook) to perform the inventory.
In this blog, I will introduce the two Ansible playbooks that I wrote. The first playbook is the main actor. This is where the variables are defined. This playbook iterates over the specified AWS regions, calling the worker playbook each time, to check if any resources have been provisioned in these regions. The output is written to comma separated value (.CSV) files (I am using semi-colons instead of commas), which can be easily imported into Microsoft Excel (or any spreadsheet program of your choice) for analysis.
Introducing the Ansible playbooks
The playbooks have been configured to check the following AWS resources
- Virtual Private Cloud (VPC)
- Subnets within the VPCs
- Internet Gateways
- Route Tables
- Security Groups
- Network Access Control Lists
- Customer Gateways
- Virtual Private Gateways
- Elastic IP Addresses
- Elastic Compute Cloud Instances
- Amazon Machine Images that were created
- Elastic Block Store Volumes
- Elastic Block Store Snapshots
- Classic Load Balancers
- Application Load Balancers
- Relational Database Service Instances
- Relational Database Service Snapshots
- Simple Storage Service (S3) Buckets
The table below provides details for the two Ansible playbooks.
| Filename | Purpose | |
| ansible-aws-inventory-main.yml | This is the controller playbook. It iterates over each of the specified regions, calling the worker playbook to check for any resources that are provisioned in these regions. | |
| ansible-aws-inventory-worker.yml | This playbook does all the heavy lifting. It checks for any provisioned resources in the region that is provided to it by the controller playbook | |
Let’s go through each of the sections in the main Ansible playbook (ansible-aws-inventory-main.yml), to get a better understanding of what it does.
First off, the variables that will be used are defined
aws_regions – this defines all the AWS regions which will be checked for provisioned resources
verbose – to display the results both on screen and to write it to file, set this to true. Setting this to false just writes the results to file.
owner_id – this is the account id for the AWS account that is being inventoried. It is used to retrieve all the Amazon Machine Images (AMI) that are owned by this account
Next, the column headers for each of the .CSV files is defined.
After this, the output filenames are defined. Do note that the file names use timestamps (for when the playbook is run) as prefixes. This ensures that they don’t overwrite any output files from previous runs.
When I was generating the inventory list, at times I found that I needed only a subset of resource types inventoried, instead of all (for instance when I was looking for only EC2 instances). For this reason, I found it beneficial to have boolean variables to either enable or disable inventory checks for specific resource types.
The next section lists boolean variables that control if a particular resource type should be checked or not. Set this to true if it is to be checked and false if it is to be skipped. You can set this to your own preference.
After all the variables have been defined, the tasks that will be carried out are configured.
The first task initialises the output .CSV files with the column headers.
Once the initialisation has been completed, the inventory process is started by looping through each of the specified AWS regions and calling the worker Ansible playbook to check for provisioned resources.
The last task displays the path for the output files.
The main Ansible playbook (ansible-aws-inventory-main.yml) can be downloaded from https://gist.github.com/nivleshc/64ea7201fb0ba8cb6f87d06adc6152de.
The worker playbook (ansible-aws-inventory-worker.yml) has the following format.
- go through each of the defined resource types and confirm that it is to be checked (checks for a particular resource type are enabled using the boolean variable that is defined in the main playbook)
- If checks are enabled for that particular resource type, find all provisioned resources of that type in the region provided by the main Ansible playbook
- write the results to the respective output file
- if verbose is enabled, write the results to screen
The worker file (ansible-aws-inventory-worker.yml) can be downloaded from https://gist.github.com/nivleshc/bedd2c440c816ebc86dbaeddef50d500.
Running the Ansible playbooks
Use the following steps to run the above mentioned Ansible playbooks to perform an inventory of your AWS account.
1. On a computer that has Ansible installed, create a folder and name it appropriately (for instance inventory)
2. Download ansible-aws-inventory-main.yml from https://gist.github.com/nivleshc/64ea7201fb0ba8cb6f87d06adc6152de and put it in the folder that was created in step 1 above
3. Download ansible-aws-inventory-worker.yml from https://gist.github.com/nivleshc/bedd2c440c816ebc86dbaeddef50d500 and put it in the folder that was created in step 1 above
4. Download the Ansible inventory file from https://gist.github.com/nivleshc/bc2e300fe1d2779ecc15c0876fc4db62, rename it to hosts and put it in the folder that was created in step 1 above
5. Customise ansible-aws-inventory-main.yml by adding your account id as the owner_id and change the output folder by updating the output_root_folder variable. If you need to disable inventory for certain resource types, you can set the respective boolean variable to false.
6. Create a user account with access keys enabled within your AWS account. For checking all the resources defined in the playbook, at a minimum, the account must have the following permissions assigned.
AmazonVPCReadOnlyAccess AmazonEC2ReadOnlyAccess ElasticLoadBalancingReadOnly AmazonRDSReadOnlyAccess AmazonS3ReadOnlyAccess
7. Open a command line and then run the following to configure environment variables with credentials of the account that was created in step 6 above (the following commands are specific to MacOS).
export AWS_ACCESS_KEY_ID="xxxxx" export AWS_SECRET_ACCESS_KEY="xxxxxxx"
8. There is a possibility that you might encounter an error with boto complaining that it is unable to access region us-west-3. To fix this, define the following environment variable as well.
export BOTO_USE_ENDPOINT_HEURISTICS=True
9. Run the Ansible playbook using the following command line.
ansible-playbook -i hosts ansible-aws-inventory-main.yml
Depending on how many resources are being inventoried, the playbook can take anywhere from five to ten minutes to complete. So, sit back and relax, while the playbook runs.
I found a bug with the Ansible “AWS S3 bucket facts” module. It ignores the region parameter and instead of returning S3 buckets in a specific region, it returns buckets in all regions. Due to this, the s3 buckets .CSV file will have the same buckets repeated in all the regions.
Hope you enjoy the above Ansible playbooks and they make your life much easier when trying to find all resources that are deployed within your AWS account.
Till the next time, enjoy!