Working with Amazon EC2 instances by using the CLI/API

I had a need to administrate AWS EC2 instances with the API using the CLI tool set or Python. To administrate AWS EC2 instances there are many options, some of them are listed below.

• Using the AWS EC2 console is of course an option
• Using the AWS CLI
• Using the AWS API Python module(s)

Below I will be discussing option 2 – Using the AWS CLI, the steps needed to install configure and use the CLI.

Installing AWS-cli

The below commends will download and install the aws-cli.

curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip -qq awscli-bundle.zip
./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

Note: Make sure to add /usr/local/bin to your PATH to use aws command directly.

Configuring AWS – Adding AWS Access Key

Note: While there are many things you can do with the aws cli. I would advice you to first add/configure your aws access keys (it would make life easier in the long 🙂 ).
To configure AWS remote keys just run aws configure an example is below.

AWS Access Key ID [None]: GTIAIDXGIUVZ7M4VDLRT
AWS Secret Access Key [None]: xoDExBBLJ7Bl7RR8c62i6Dou79lk/KgeJUAYOcY9
Default region name [None]: us-east-1
Default output format [None]: text or json

To get a Secret Access Key, login to your AWS EC2 console and go to Security > IAM. Create a Policy (or use a per-defined) Create a new User and use the new Access Key and Secret Access Key, optional download and save the key in a safe place. Attach the policy to the new user, to be able to access remotely. Note: Once the you close the window, you wont be able to retrieve the Secret Access Key you can of course re-generate the key which might not always be the best option.

Attaching Policy to the new user

To be able to use the new user, you need to apply(attach) rights (i.e policy) to new user. An example Policy is below, just copy and paste in AWS, or use one of the existing policy’s.

# Example Policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "iam:GetUser",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "iam:ListInstanceProfiles",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "iam:ListServerCertificates",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "iam:PassRole",
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "NotAction": "iam:*",
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Accessing instances with the aws cli

The simplest form might be to use the describe-instances, an example output is below

{
    "Reservations": [
        {
            "OwnerId": "111000999111", 
            "ReservationId": "r-0daa5212gg224d5a2", 
            "Groups": [], 
            "Instances": [
                {
                    "Monitoring": {
                        "State": "enabled"
                    }, 
                    "PublicDnsName": "ec2-11-22-44-150.compute-1.amazonaws.com", 
                    "State": {
                        "Code": 16, 
                        "Name": "running"
                    }, 
                    "EbsOptimized": false, 
                    "LaunchTime": "2016-09-22T19:07:18.000Z", 
                    "PublicIpAddress": "54.54.45.160", 
                    "PrivateIpAddress": "123.45.48.99", 
                    "ProductCodes": [
                        {
                            "ProductCodeId": "7eyp7o9i99afqvpvvh5gujt8w", 
                            "ProductCodeType": "marketplace"
                        }
                    ], 
                    "VpcId": "vpc-f7a99999", 
                    "StateTransitionReason": "", 
                    "InstanceId": "i-0815ed98b0333e2c7", 
                    "ImageId": "ami-7c099d6b", 
                    "PrivateDnsName": "ip-172-31-48-54.ec2.internal", 
                    "KeyName": "My-Key-Name-Key", 
                    "SecurityGroups": [
                        {
                            "GroupName": "Security group", 
                            "GroupId": "sg-403e444a"
                        }
                    ], 
[..] truncated                     

Anther example can be monitoring remote instances. If AWS cloudwatch is disabled and not configured, you would get the below output.

aws ec2 monitor-instances --instance-ids i-0915ed9990554e2c9
{
    "InstanceMonitorings": [
        {
            "InstanceId": "i-0915ed9990554e2c9", 
            "Monitoring": {
                "State": "enabled"
            }
        }
    ]
}

Administrating AWS EC2 by using the Python API’s

First we need to install the AWS modules. Note: The AWS module is installed using pip, make sure to have pip installed before installing. Installing AWS Python modules.

pip install --upgrade --user awscli

Next, lets create an /etc/boto.cfg (or ~/.boto), with the aws access key

[Credentials]
aws_access_key_id = BKIAIDXGIUXXXXXXXXX
aws_secret_access_key = abCDeFG7Bl7RR8c62i6Dou79lk/KSeABCDOfG9

Now, we can list all running AWS instances. run the below.

#!/bin/python

from pprint import pprint
from boto import ec2

# specify the aws access key(s) (only needed if there is no /etc/boto.cfg, uncomment next 3 lines).
# access_key_id = 'BKIAIDXGIUXXXXXXXXX'
# secret_access_key = 'abCDeFG7Bl7RR8c62i6Dou79lk/KSeABCDOfG9'
# c = ec2.connection.EC2Connection(access_key_id, secret_access_key)

# use if you have /etc/boto.cfg with the access key's 
c = ec2.connection.EC2Connection()

reservations = c.get_all_instances();
for reservation in reservations:
 for instances in reservation.instances:
   # print just the instance ip address
   print instances.ip_address
   # print all the instance parameters / values
   #print pprint(instances.__dict__)

Sample output is below.

# Just instance ip address
10.10.10.10

# if full print is used
{'_in_monitoring_element': False,
 '_placement': us-east-1b,
 '_previous_state': None,
[..] snip

The above example is just a small sample of what can be done with aws api. almost the full ec2 manamgment is possible with the api’s.
Next will explorer how to install and use azure’s API.

References

Installing AWS python module AWS Python Developer Center

Boto Reference

Boto documentation Boto on github Boto cheat sheet