Ansible Run Playbooks in Parallel: Async, Forks & Concurrent Execution

By Luca Berton · Published 2024-01-01 · Category: interviews

How to run Ansible playbooks and tasks in parallel. Use async, forks, free strategy, and GNU parallel to execute multiple playbooks simultaneously.

Introduction

By default, Ansible runs tasks sequentially — one task across all hosts before moving to the next. But many scenarios require parallel execution: running two independent playbooks simultaneously, launching long tasks without waiting, or maximizing throughput across hundreds of hosts.

This guide covers every approach to parallel execution in Ansible, from simple shell backgrounding to advanced async patterns.

Method 1: Run Two Playbooks Simultaneously (Shell)

The simplest approach — run separate ansible-playbook processes:

# Background both playbooks
ansible-playbook deploy-web.yml &
ansible-playbook deploy-db.yml &
wait  # Wait for both to finish
echo "Both playbooks completed"

With output logging:

ansible-playbook deploy-web.yml > /tmp/web.log 2>&1 &
WEB_PID=$!

ansible-playbook deploy-db.yml > /tmp/db.log 2>&1 &
DB_PID=$!

# Wait and check results
wait $WEB_PID
WEB_RC=$?
wait $DB_PID
DB_RC=$?

echo "Web: exit $WEB_RC, DB: exit $DB_RC"

Method 2: GNU Parallel

# Run multiple playbooks in parallel
parallel ansible-playbook {} ::: deploy-web.yml deploy-db.yml configure-monitoring.yml

# With limit on concurrent jobs
parallel -j 3 ansible-playbook {} ::: playbook1.yml playbook2.yml playbook3.yml playbook4.yml

Method 3: Async Tasks (Within a Playbook)

Launch tasks without waiting, then poll for results:

---
- hosts: all
  tasks:
    # Launch long-running tasks in parallel
    - name: Run database backup
      command: /opt/scripts/backup-db.sh
      async: 3600    # Max runtime: 1 hour
      poll: 0         # Don't wait — fire and forget
      register: backup_job

    - name: Run log rotation
      command: /opt/scripts/rotate-logs.sh
      async: 600
      poll: 0
      register: logs_job

    - name: Run cache warmup
      command: /opt/scripts/warm-cache.sh
      async: 300
      poll: 0
      register: cache_job

    # Do other work while async tasks run
    - name: Deploy new config
      template:
        src: app.conf.j2
        dest: /etc/myapp/app.conf

    # Now wait for all async tasks
    - name: Wait for backup
      async_status:
        jid: "{{ backup_job.ansible_job_id }}"
      register: backup_result
      until: backup_result.finished
      retries: 60
      delay: 30

    - name: Wait for logs
      async_status:
        jid: "{{ logs_job.ansible_job_id }}"
      register: logs_result
      until: logs_result.finished
      retries: 30
      delay: 10

    - name: Wait for cache
      async_status:
        jid: "{{ cache_job.ansible_job_id }}"
      register: cache_result
      until: cache_result.finished
      retries: 30
      delay: 10

Method 4: Free Strategy

Default strategy (linear) waits for all hosts to finish a task before moving on. free lets each host proceed independently:

---
- hosts: all
  strategy: free
  tasks:
    - name: Update packages (slow on some hosts)
      apt:
        upgrade: dist
      become: true

    - name: Restart service
      service:
        name: myapp
        state: restarted
      become: true

With free, fast hosts don't wait for slow ones.

Method 5: Increase Forks

# ansible.cfg
[defaults]
forks = 50  # Default is 5!

# Or per-run
ansible-playbook site.yml -f 50

This controls how many hosts are processed simultaneously per task.

Method 6: Multiple Plays in One Playbook

---
# These run sequentially (plays are always sequential)
- name: Deploy web servers
  hosts: webservers
  roles:
    - webserver

- name: Deploy databases
  hosts: dbservers
  roles:
    - database

To make them truly parallel, split into separate files and use shell backgrounding (Method 1).

Method 7: AWX/AAP Workflow Templates

In AWX or Ansible Automation Platform, use Workflow Templates to run playbooks in parallel:

[Provision VMs] → [Deploy Web] ─┐
                                 ├→ [Run Tests]
                  [Deploy DB]  ──┘

Workflow nodes can run simultaneously when they don't depend on each other.

Method 8: Import with Async

---
- hosts: webservers
  tasks:
    - name: Deploy app (async on each host)
      include_tasks: deploy-tasks.yml
      vars:
        async_mode: true

# deploy-tasks.yml
- command: /opt/deploy.sh
  async: 600
  poll: "{{ 0 if async_mode else 15 }}"
  register: deploy_result

Comparison

Method	Scope	Complexity	Best For
Shell `&`	Playbooks	Low	2-3 independent playbooks
GNU parallel	Playbooks	Low	Many independent playbooks
Async tasks	Tasks within play	Medium	Long-running tasks
Free strategy	Hosts within play	Low	Heterogeneous host speeds
Forks	Hosts per task	Low	Large inventories
AWX Workflows	Playbooks	Medium	Production orchestration

Pitfalls to Avoid

Race Conditions

# DANGEROUS — both tasks modify the same file
- name: Task A writes config
  template: { src: a.j2, dest: /etc/app.conf }
  async: 60
  poll: 0

- name: Task B writes config
  template: { src: b.j2, dest: /etc/app.conf }  # Overwrites A!
  async: 60
  poll: 0

Resource Contention

# Too many forks can overload the controller
# Monitor controller CPU/memory when increasing forks
forks = 100  # May need 4+ GB RAM on controller

Async + become

# Async tasks with become work, but the job runs as the become user
- command: /opt/backup.sh
  async: 3600
  poll: 0
  become: true
  become_user: backup

FAQ

Can I run two plays in the same playbook in parallel?

No — plays within a playbook always run sequentially. To run plays in parallel, put them in separate files and launch with & or GNU parallel.

What happens if an async task fails?

Check the result with async_status. If the task fails, async_status returns finished: true and failed: true. Handle it with failed_when or ignore_errors.

How many forks should I use?

Start with forks = 20-50. Each fork uses ~100MB on the controller. Monitor controller resources and adjust. For SSH, also consider pipelining = true to reduce connections.

Can I mix async and synchronous tasks?

Yes — async tasks run in the background while subsequent synchronous tasks run normally. Use async_status to collect results when needed.

Category: interviews

Browse all Ansible tutorials · AnsiblePilot Home

AnsiblePilot — Master Ansible Automation

Popular Topics

About Luca Berton