Configure a Python Virtual Environment for Ansible AWS - ansible collection amazon.aws

How to configure a Python Virtual Environment for Ansible AWS?

Using a Python Virtual Environment is a convenient way to maintain up-to-date Python dependency of the Ansible collection amazon.aws without interfering with your Linux system.

This initial configuration sometimes is a roadblock for some AWS users to start using Ansible.

I’m Luca Berton and welcome to today’s episode of Ansible Pilot.

See also: Ansible AWS EC2: Automate Ubuntu Instance Creation & Data Collection

Links

• Ansible collection amazon.aws • Python boto3

Playbook

Configure a Python Virtual Environment for Ansible AWS: • boto3

How to Python Virtual Environment for Ansible AWS.

I’m going to show you how to configure a Python Virtual Environment for Ansible AWS to successfully use the Ansible collection amazon.aws of modules and plugins to manage various operations related to AWS infrastructure such as EC2, VPC, Security Groups, etc.

Ansible AWS modules are written on top of boto3. Boto3 is the Python SDK for the AWS that allows users to manage AWS infrastructure: EC2, VPC, Security Groups, etc.

code

$ python3.8 -m venv venv
$ source venv/bin/activate
(venv) $ pip3.8 install --upgrade pip
(venv) $ pip3.8 install boto3
(venv) $ ansible-galaxy collection install amazon.aws

execution

$ ansible --version
ansible [core 2.12.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/devops/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  ansible collection location = /home/devops/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]
  jinja version = 2.10.3
  libyaml = True
[devops@demo ~]$ whereis python
python: /usr/bin/python3.6 /usr/bin/python3.6m /usr/bin/python3.8 /usr/lib/python3.6 /usr/lib/python3.8 /usr/lib64/python3.6 /usr/lib64/python3.8 /usr/local/lib/python3.8 /usr/include/python3.6m /usr/include/python3.8 /usr/share/man/man1/python.1.gz
[devops@demo ~]$ python3.8 -m venv venv
[devops@demo ~]$ source venv/bin/activate
(venv) [devops@demo ~]$ pip3.8 install --upgrade pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/96/2f/caec18213f6a67852f6997fb0673ae08d2e93d1b81573edb93ba4ef06970/pip-22.1.2-py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.3.1
    Uninstalling pip-19.3.1:
      Successfully uninstalled pip-19.3.1
Successfully installed pip-22.1.2
(venv) [devops@demo ~]$ pip3.8 install boto3
Collecting boto3
  Downloading boto3-1.24.27-py3-none-any.whl (132 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.5/132.5 kB 392.0 kB/s eta 0:00:00
Collecting botocore<1.28.0,>=1.27.27
  Downloading botocore-1.27.27-py3-none-any.whl (9.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.0/9.0 MB 2.7 MB/s eta 0:00:00
Collecting jmespath<2.0.0,>=0.7.1
  Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting s3transfer<0.7.0,>=0.6.0
  Downloading s3transfer-0.6.0-py3-none-any.whl (79 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.6/79.6 kB 3.3 MB/s eta 0:00:00
Collecting python-dateutil<3.0.0,>=2.1
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 4.0 MB/s eta 0:00:00
Collecting urllib3<1.27,>=1.25.4
  Downloading urllib3-1.26.10-py2.py3-none-any.whl (139 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.2/139.2 kB 3.8 MB/s eta 0:00:00
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3
Successfully installed boto3-1.24.27 botocore-1.27.27 jmespath-1.0.1 python-dateutil-2.8.2 s3transfer-0.6.0 six-1.16.0 urllib3-1.26.10
(venv) [devops@demo ~]$ pip3.8 list | grep boto
boto3           1.24.27
botocore        1.27.27
(venv) [devops@demo ~]$ pip3.8 freeze > requirements.txt
(venv) [devops@demo ~]$ deactivate
[devops@demo ~]$ source venv/bin/activate
(venv) [devops@demo ~]$ pip list
Package         Version
--------------- -------
boto3           1.24.27
botocore        1.27.27
jmespath        1.0.1
pip             22.1.2
python-dateutil 2.8.2
s3transfer      0.6.0
setuptools      41.6.0
six             1.16.0
urllib3         1.26.10
(venv) [devops@demo ~]$ ansible-galaxy collection install amazon.aws
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading https://galaxy.ansible.com/download/amazon-aws-4.0.0.tar.gz to /home/devops/.ansible/tmp/ansible-local-5083_hekoyln/tmpawwwduwa/amazon-aws-4.0.0-_wa4znqr
Installing 'amazon.aws:4.0.0' to '/home/devops/.ansible/collections/ansible_collections/amazon/aws'
amazon.aws:4.0.0 was installed successfully
(venv) [devops@demo ~]$ ansible-galaxy collection list amazon.aws
# /usr/lib/python3.8/site-packages/ansible_collections
Collection Version
---------- -------
amazon.aws 2.1.0
# /home/devops/.ansible/collections/ansible_collections
Collection Version
---------- -------
amazon.aws 4.0.0  
(venv) [devops@demo ~]$

• requirements.txt

boto3==1.24.27
botocore==1.27.27
jmespath==1.0.1
python-dateutil==2.8.2
s3transfer==0.6.0
six==1.16.0
urllib3==1.26.10

code with ❤️ in GitHub

See also: Configuring Ansible for AWS: Setup Guide & Playbook

Conclusion

Now you know how to Configure a Python Virtual Environment for Ansible AWS.

Related Articles

• publishing collections to Ansible Galaxy • the Ansible inventory deep-dive • EC2 provisioning with Ansible

See also: Search for AWS EC2 AMI ID by Region - Ansible module ec2_ami_info

Complete Setup

# Create virtual environment
python3 -m venv ~/ansible-aws-venv
source ~/ansible-aws-venv/bin/activate
# Install Ansible and AWS dependencies
pip install ansible boto3 botocore
# Install AWS collection
ansible-galaxy collection install amazon.aws community.aws
# Verify
python3 -c "import boto3; print(f'boto3 {boto3.__version__}')"
ansible-galaxy collection list | grep aws

AWS Authentication

# Option 1: Environment variables
export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
export AWS_DEFAULT_REGION="us-east-1"
# Option 2: AWS CLI profile
aws configure --profile ansible
export AWS_PROFILE=ansible

Test Playbook

- name: Test AWS connectivity
  hosts: localhost
  connection: local
  tasks:
    - name: List EC2 instances
      amazon.aws.ec2_instance_info:
        region: us-east-1
      register: ec2_info
- name: Show instances
      ansible.builtin.debug:
        msg: "{{ item.instance_id }} — {{ item.state.name }}"
      loop: "{{ ec2_info.instances }}"
- name: List S3 buckets
      amazon.aws.s3_bucket_info:
      register: s3_info
- name: Show buckets
      ansible.builtin.debug:
        msg: "{{ s3_info.buckets | map(attribute='name') | list }}"

Using Ansible Vault for AWS Credentials

# group_vars/all/vault.yml (encrypted)
vault_aws_access_key: AKIAIOSFODNN7EXAMPLE
vault_aws_secret_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# In playbook
- hosts: localhost
  environment:
    AWS_ACCESS_KEY_ID: "{{ vault_aws_access_key }}"
    AWS_SECRET_ACCESS_KEY: "{{ vault_aws_secret_key }}"

FAQ

Why use a virtual environment for AWS?

Virtual environments isolate dependencies. Different projects may need different boto3 versions. venvs prevent system Python conflicts and make requirements reproducible.

Which Python packages do AWS modules need?

boto3 and botocore are required. Some modules may also need awscli for advanced features. Install with pip install boto3 botocore.

Can I use AWS SSO with Ansible?

Yes. Configure AWS SSO via aws configure sso, then set AWS_PROFILE to the SSO profile name. Ansible will use the SSO session token.