Automating AWS Resource Discovery with AWS CloudShell and AWS CLI

Managing AWS resources efficiently requires automation and the ability to quickly identify the relationships between different services. In this blog post, we’ll explore how to create a script that identifies all AWS services linked to a specific EC2 instance, including IAM accounts, S3 buckets, RDS databases, VPCs, routes, gateways, Elastic IPs, and more. We’ll provide versions of the script for both AWS CloudShell and AWS CLI on Windows.

Introduction

When working with AWS, it’s often necessary to audit and document the resources associated with an EC2 instance, especially during decommissioning or troubleshooting. Manually gathering this information can be time-consuming. Automating this process not only saves time but also reduces the likelihood of missing critical dependencies.

Prerequisites

Before we begin, ensure you have the following:

• An AWS account with sufficient permissions to describe resources (EC2, IAM, VPC, etc.).
• Basic knowledge of AWS services and the command-line interface.
• For the AWS CLI version:
  • A Windows machine with AWS CLI installed and configured.
  • PowerShell installed (comes default with Windows 10 and later).

AWS CloudShell Version

AWS CloudShell is a browser-based shell that makes it easy to securely manage, interact with, and explore your AWS resources. Below is the script designed to run in AWS CloudShell:

Script: identify_resources.sh

#!/bin/bash

# Define the instance ID variable (modify this as needed)
INSTANCE_ID="i-0e3562863ae619460"

# Output file
OUTPUT_FILE="instance_associated_resources.json"

# Create temp directory
TEMP_DIR=$(mktemp -d)

# Get instance description
aws ec2 describe-instances --instance-ids $INSTANCE_ID > $TEMP_DIR/instance_desc.json

# Parse VPC ID
VPC_ID=$(jq -r '.Reservations[0].Instances[0].VpcId' $TEMP_DIR/instance_desc.json)

# Parse Subnet ID
SUBNET_ID=$(jq -r '.Reservations[0].Instances[0].SubnetId' $TEMP_DIR/instance_desc.json)

# Parse Security Group IDs
SG_IDS=$(jq -r '.Reservations[0].Instances[0].SecurityGroups[].GroupId' $TEMP_DIR/instance_desc.json | paste -sd " " -)

# Parse IAM Instance Profile ARN
IAM_INSTANCE_PROFILE_ARN=$(jq -r '.Reservations[0].Instances[0].IamInstanceProfile.Arn' $TEMP_DIR/instance_desc.json)

# Get IAM Instance Profile Name
if [ "$IAM_INSTANCE_PROFILE_ARN" != "null" ]; then
    IAM_INSTANCE_PROFILE_NAME=$(echo $IAM_INSTANCE_PROFILE_ARN | awk -F'/' '{print $NF}')
else
    IAM_INSTANCE_PROFILE_NAME=""
fi

# Parse EBS Volume IDs
EBS_VOLUME_IDS=$(jq -r '.Reservations[0].Instances[0].BlockDeviceMappings[].Ebs.VolumeId' $TEMP_DIR/instance_desc.json | paste -sd " " -)

# Parse Network Interface IDs
NETWORK_INTERFACE_IDS=$(jq -r '.Reservations[0].Instances[0].NetworkInterfaces[].NetworkInterfaceId' $TEMP_DIR/instance_desc.json | paste -sd " " -)

# Now, collect details of the associated resources

# Collect Security Group details
if [ -n "$SG_IDS" ]; then
    aws ec2 describe-security-groups --group-ids $SG_IDS > $TEMP_DIR/sg_desc.json
else
    echo "[]" > $TEMP_DIR/sg_desc.json
fi

# Collect IAM Role details
if [ -n "$IAM_INSTANCE_PROFILE_NAME" ]; then
    aws iam get-instance-profile --instance-profile-name $IAM_INSTANCE_PROFILE_NAME > $TEMP_DIR/iam_instance_profile_desc.json
    IAM_ROLE_NAME=$(jq -r '.InstanceProfile.Roles[0].RoleName' $TEMP_DIR/iam_instance_profile_desc.json)
    if [ -n "$IAM_ROLE_NAME" ]; then
        aws iam get-role --role-name $IAM_ROLE_NAME > $TEMP_DIR/iam_role_desc.json
    else
        echo "{}" > $TEMP_DIR/iam_role_desc.json
    fi
else
    echo "{}" > $TEMP_DIR/iam_instance_profile_desc.json
    echo "{}" > $TEMP_DIR/iam_role_desc.json
fi

# Collect EBS Volume details
if [ -n "$EBS_VOLUME_IDS" ]; then
    aws ec2 describe-volumes --volume-ids $EBS_VOLUME_IDS > $TEMP_DIR/ebs_volume_desc.json
else
    echo "[]" > $TEMP_DIR/ebs_volume_desc.json
fi

# Collect Network Interface details
if [ -n "$NETWORK_INTERFACE_IDS" ]; then
    aws ec2 describe-network-interfaces --network-interface-ids $NETWORK_INTERFACE_IDS > $TEMP_DIR/network_interface_desc.json
else
    echo "[]" > $TEMP_DIR/network_interface_desc.json
fi

# Collect Elastic IPs associated with the network interfaces
ELASTIC_IPS=$(jq -r '.NetworkInterfaces[].Association.PublicIp' $TEMP_DIR/network_interface_desc.json | grep -v null | paste -sd " " -)

if [ -n "$ELASTIC_IPS" ]; then
    # Describe Elastic IPs
    aws ec2 describe-addresses --public-ips $ELASTIC_IPS > $TEMP_DIR/elastic_ip_desc.json
else
    echo "[]" > $TEMP_DIR/elastic_ip_desc.json
fi

# Collect VPC details
aws ec2 describe-vpcs --vpc-ids $VPC_ID > $TEMP_DIR/vpc_desc.json

# Collect Subnet details
aws ec2 describe-subnets --subnet-ids $SUBNET_ID > $TEMP_DIR/subnet_desc.json

# Collect Route Tables associated with the VPC
aws ec2 describe-route-tables --filters "Name=vpc-id,Values=$VPC_ID" > $TEMP_DIR/route_tables_desc.json

# Collect Internet Gateways associated with the VPC
aws ec2 describe-internet-gateways --filters "Name=attachment.vpc-id,Values=$VPC_ID" > $TEMP_DIR/internet_gateways_desc.json

# Collect NAT Gateways in the VPC
aws ec2 describe-nat-gateways --filter "Name=vpc-id,Values=$VPC_ID" > $TEMP_DIR/nat_gateways_desc.json

# Collect RDS instances in the same VPC
aws rds describe-db-instances > $TEMP_DIR/rds_instances.json
jq --arg VPC_ID "$VPC_ID" '[.DBInstances[] | select(.DBSubnetGroup.VpcId == $VPC_ID)]' $TEMP_DIR/rds_instances.json > $TEMP_DIR/rds_instances_in_vpc.json

# Prepare final JSON output
jq -n \
  --slurpfile instanceDetails $TEMP_DIR/instance_desc.json \
  --slurpfile securityGroups $TEMP_DIR/sg_desc.json \
  --slurpfile iamInstanceProfile $TEMP_DIR/iam_instance_profile_desc.json \
  --slurpfile iamRole $TEMP_DIR/iam_role_desc.json \
  --slurpfile ebsVolumes $TEMP_DIR/ebs_volume_desc.json \
  --slurpfile networkInterfaces $TEMP_DIR/network_interface_desc.json \
  --slurpfile elasticIPs $TEMP_DIR/elastic_ip_desc.json \
  --slurpfile vpc $TEMP_DIR/vpc_desc.json \
  --slurpfile subnet $TEMP_DIR/subnet_desc.json \
  --slurpfile routeTables $TEMP_DIR/route_tables_desc.json \
  --slurpfile internetGateways $TEMP_DIR/internet_gateways_desc.json \
  --slurpfile natGateways $TEMP_DIR/nat_gateways_desc.json \
  --slurpfile rdsInstancesInVPC $TEMP_DIR/rds_instances_in_vpc.json \
  '{
    "InstanceDetails": $instanceDetails[0],
    "SecurityGroups": $securityGroups[0],
    "IAMInstanceProfile": $iamInstanceProfile[0],
    "IAMRole": $iamRole[0],
    "EBSVolumes": $ebsVolumes[0],
    "NetworkInterfaces": $networkInterfaces[0],
    "ElasticIPs": $elasticIPs[0],
    "VPC": $vpc[0],
    "Subnet": $subnet[0],
    "RouteTables": $routeTables[0],
    "InternetGateways": $internetGateways[0],
    "NATGateways": $natGateways[0],
    "RDSInstancesInVPC": $rdsInstancesInVPC[0]
  }' > $OUTPUT_FILE

# Clean up temp directory
rm -rf $TEMP_DIR

echo "Output saved to $OUTPUT_FILE"

Instructions to Run in AWS CloudShell

1. Open AWS CloudShell from the AWS Management Console.
2. Create a new script file:

nano identify_resources.sh

1. Paste the script into the editor.
2. Modify the INSTANCE_ID variable at the top of the script to target your specific EC2 instance.
3. Save and exit the editor (Ctrl+O to save, Ctrl+X to exit).
4. Make the script executable:

chmod +x identify_resources.sh

1. Run the script:

./identify_resources.sh

1. The script will generate a JSON file named instance_associated_resources.json containing all the linked AWS resources.

AWS CLI Windows Version

For Windows users, we can convert the script to a PowerShell script to run using AWS CLI. Below is the PowerShell version:

Script: identify_resources.ps1

# Define the instance ID variable (modify this as needed)
$INSTANCE_ID = "i-0e3562863ae619460"

# Output file
$OUTPUT_FILE = "instance_associated_resources.json"

# Create temp directory
$TEMP_DIR = Join-Path -Path ([System.IO.Path]::GetTempPath()) -ChildPath ([System.IO.Path]::GetRandomFileName())
New-Item -ItemType Directory -Path $TEMP_DIR | Out-Null

# Get instance description
aws ec2 describe-instances --instance-ids $INSTANCE_ID | Out-File "$TEMP_DIR\instance_desc.json"

# Parse VPC ID
$instanceData = Get-Content "$TEMP_DIR\instance_desc.json" | ConvertFrom-Json
$VPC_ID = $instanceData.Reservations[0].Instances[0].VpcId

# Parse Subnet ID
$SUBNET_ID = $instanceData.Reservations[0].Instances[0].SubnetId

# Parse Security Group IDs
$SG_IDS = $instanceData.Reservations[0].Instances[0].SecurityGroups | ForEach-Object { $_.GroupId }
$SG_IDS = $SG_IDS -join " "

# Parse IAM Instance Profile ARN
$IAM_INSTANCE_PROFILE_ARN = $instanceData.Reservations[0].Instances[0].IamInstanceProfile.Arn

# Get IAM Instance Profile Name
if ($IAM_INSTANCE_PROFILE_ARN -ne $null) {
    $IAM_INSTANCE_PROFILE_NAME = ($IAM_INSTANCE_PROFILE_ARN -split '/')[1]
} else {
    $IAM_INSTANCE_PROFILE_NAME = ""
}

# Parse EBS Volume IDs
$EBS_VOLUME_IDS = $instanceData.Reservations[0].Instances[0].BlockDeviceMappings | ForEach-Object { $_.Ebs.VolumeId }
$EBS_VOLUME_IDS = $EBS_VOLUME_IDS -join " "

# Parse Network Interface IDs
$NETWORK_INTERFACE_IDS = $instanceData.Reservations[0].Instances[0].NetworkInterfaces | ForEach-Object { $_.NetworkInterfaceId }
$NETWORK_INTERFACE_IDS = $NETWORK_INTERFACE_IDS -join " "

# Now, collect details of the associated resources

# Collect Security Group details
if ($SG_IDS -ne "") {
    aws ec2 describe-security-groups --group-ids $SG_IDS | Out-File "$TEMP_DIR\sg_desc.json"
} else {
    "[]" | Out-File "$TEMP_DIR\sg_desc.json"
}

# Collect IAM Role details
if ($IAM_INSTANCE_PROFILE_NAME -ne "") {
    aws iam get-instance-profile --instance-profile-name $IAM_INSTANCE_PROFILE_NAME | Out-File "$TEMP_DIR\iam_instance_profile_desc.json"
    $iamProfileData = Get-Content "$TEMP_DIR\iam_instance_profile_desc.json" | ConvertFrom-Json
    $IAM_ROLE_NAME = $iamProfileData.InstanceProfile.Roles[0].RoleName
    if ($IAM_ROLE_NAME -ne "") {
        aws iam get-role --role-name $IAM_ROLE_NAME | Out-File "$TEMP_DIR\iam_role_desc.json"
    } else {
        "{}" | Out-File "$TEMP_DIR\iam_role_desc.json"
    }
} else {
    "{}" | Out-File "$TEMP_DIR\iam_instance_profile_desc.json"
    "{}" | Out-File "$TEMP_DIR\iam_role_desc.json"
}

# Collect EBS Volume details
if ($EBS_VOLUME_IDS -ne "") {
    aws ec2 describe-volumes --volume-ids $EBS_VOLUME_IDS | Out-File "$TEMP_DIR\ebs_volume_desc.json"
} else {
    "[]" | Out-File "$TEMP_DIR\ebs_volume_desc.json"
}

# Collect Network Interface details
if ($NETWORK_INTERFACE_IDS -ne "") {
    aws ec2 describe-network-interfaces --network-interface-ids $NETWORK_INTERFACE_IDS | Out-File "$TEMP_DIR\network_interface_desc.json"
} else {
    "[]" | Out-File "$TEMP_DIR\network_interface_desc.json"
}

# Collect Elastic IPs associated with the network interfaces
$networkInterfacesData = Get-Content "$TEMP_DIR\network_interface_desc.json" | ConvertFrom-Json
$ELASTIC_IPS = $networkInterfacesData.NetworkInterfaces | ForEach-Object { $_.Association.PublicIp } | Where-Object { $_ -ne $null }
$ELASTIC_IPS = $ELASTIC_IPS -join " "

if ($ELASTIC_IPS -ne "") {
    aws ec2 describe-addresses --public-ips $ELASTIC_IPS | Out-File "$TEMP_DIR\elastic_ip_desc.json"
} else {
    "[]" | Out-File "$TEMP_DIR\elastic_ip_desc.json"
}

# Collect VPC details
aws ec2 describe-vpcs --vpc-ids $VPC_ID | Out-File "$TEMP_DIR\vpc_desc.json"

# Collect Subnet details
aws ec2 describe-subnets --subnet-ids $SUBNET_ID | Out-File "$TEMP_DIR\subnet_desc.json"

# Collect Route Tables associated with the VPC
aws ec2 describe-route-tables --filters "Name=vpc-id,Values=$VPC_ID" | Out-File "$TEMP_DIR\route_tables_desc.json"

# Collect Internet Gateways associated with the VPC
aws ec2 describe-internet-gateways --filters "Name=attachment.vpc-id,Values=$VPC_ID" | Out-File "$TEMP_DIR\internet_gateways_desc.json"

# Collect NAT Gateways in the VPC
aws ec2 describe-nat-gateways --filter "Name=vpc-id,Values=$VPC_ID" | Out-File "$TEMP_DIR\nat_gateways_desc.json"

# Collect RDS instances in the same VPC
aws rds describe-db-instances | Out-File "$TEMP_DIR\rds_instances.json"
$rdsInstancesData = Get-Content "$TEMP_DIR\rds_instances.json" | ConvertFrom-Json
$rdsInstancesInVPC = $rdsInstancesData.DBInstances | Where-Object { $_.DBSubnetGroup.VpcId -eq $VPC_ID }
$rdsInstancesInVPC | ConvertTo-Json | Out-File "$TEMP_DIR\rds_instances_in_vpc.json"

# Prepare final JSON output
$jsonObject = @{
    "InstanceDetails" = $instanceData
    "SecurityGroups" = (Get-Content "$TEMP_DIR\sg_desc.json" | ConvertFrom-Json)
    "IAMInstanceProfile" = (Get-Content "$TEMP_DIR\iam_instance_profile_desc.json" | ConvertFrom-Json)
    "IAMRole" = (Get-Content "$TEMP_DIR\iam_role_desc.json" | ConvertFrom-Json)
    "EBSVolumes" = (Get-Content "$TEMP_DIR\ebs_volume_desc.json" | ConvertFrom-Json)
    "NetworkInterfaces" = $networkInterfacesData
    "ElasticIPs" = (Get-Content "$TEMP_DIR\elastic_ip_desc.json" | ConvertFrom-Json)
    "VPC" = (Get-Content "$TEMP_DIR\vpc_desc.json" | ConvertFrom-Json)
    "Subnet" = (Get-Content "$TEMP_DIR\subnet_desc.json" | ConvertFrom-Json)
    "RouteTables" = (Get-Content "$TEMP_DIR\route_tables_desc.json" | ConvertFrom-Json)
    "InternetGateways" = (Get-Content "$TEMP_DIR\internet_gateways_desc.json" | ConvertFrom-Json)
    "NATGateways" = (Get-Content "$TEMP_DIR\nat_gateways_desc.json" | ConvertFrom-Json)
    "RDSInstancesInVPC" = (Get-Content "$TEMP_DIR\rds_instances_in_vpc.json" | ConvertFrom-Json)
}

# Output the final JSON to a file
$jsonObject | ConvertTo-Json -Depth 10 | Out-File $OUTPUT_FILE

# Clean up temp directory
Remove-Item -Recurse -Force $TEMP_DIR

Write-Host "Output saved to $OUTPUT_FILE"

Instructions to Run in AWS CLI on Windows

1. Ensure that AWS CLI and PowerShell are installed on your Windows machine.
2. Save the script as identify_resources.ps1.
3. Modify the $INSTANCE_ID variable at the top of the script to target your specific EC2 instance.
4. Open PowerShell and navigate to the directory containing the script.
5. If running the script for the first time, you might need to change the execution policy:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

This allows you to run local scripts.

1. Run the script:

.\identify_resources.ps1

1. The script will generate a JSON file named instance_associated_resources.json containing all the linked AWS resources.

Script Explanation

The script performs the following steps:

1. Defines the EC2 instance ID to target.
2. Creates a temporary directory to store intermediate JSON files.
3. Retrieves detailed information about the EC2 instance using aws ec2 describe-instances.
4. Extracts associated resource IDs such as VPC ID, Subnet ID, Security Group IDs, IAM Instance Profile ARN, EBS Volume IDs, and Network Interface IDs.
5. Collects detailed information about each associated resource using AWS CLI commands like describe-security-groups, describe-volumes, etc.
6. Aggregates all the collected data into a single JSON object.
7. Outputs the final JSON to a file named instance_associated_resources.json.
8. Cleans up the temporary directory used for storing intermediate files.

This automation ensures that all resources linked to an EC2 instance are documented, which is especially useful for decommissioning processes or auditing.

Conclusion

Automating resource discovery in AWS can significantly streamline your workflows and reduce the risk of leaving behind orphaned resources. Whether you prefer using AWS CloudShell or running scripts locally via AWS CLI on Windows, the scripts provided offer a comprehensive solution to identify all resources associated with an EC2 instance.

Feel free to customize and extend the scripts to suit your specific needs, such as integrating with other AWS services or incorporating additional resource types.