Distribute Task Execution (Nx Agents)

Nx Agents lets you distribute your CI across many machines with minimal configuration. It comes with features such as dynamically allocating agents based on the size of the PR, flaky task re-running, and intelligent task splitting and distribution. Keep reading to learn more.

Distribute Task Execution with Nx Agents

For a more thorough explanation of how Nx Agents optimizes your CI pipeline, read this guide to parallelization and distribution in CI.

Enabling Nx Agents

To enable task distribution with Nx Agents, there are two requirements:

  1. Enable version control system integration. The integrations currently available are GitHub, GitLab and Bitbucket. These integrations can be enabled from your Nx Cloud dashboard.
  2. Add a single line to your CI pipeline configuration.

Add the start-ci-run command to your CI pipeline configuration after checking out the repository and before installing node_modules:

.github/workflows/main.yaml
1# After checkout repository 2- name: Start CI run 3 run: 'npx nx-cloud start-ci-run --distribute-on="8 linux-medium-js" --stop-agents-after="e2e-ci"' 4# Before install node_modules 5# Run any nx commands as if running on a single machine 6

The --distribute-on flag instructs Nx Cloud to distribute tasks across 8 agents of type linux-medium-js. linux-medium-js is the name of the launch template that will be used to provision the agent. The default launch templates can be found here

Launch Templates

You can also define your own "launch templates" (here's an example from the Nx repo):

.nx/workflows/agents.yaml
1launch-templates: 2 linux-medium: 3 resource-class: 'docker_linux_amd64/medium+' 4 init-steps: 5 - name: Pnpm Install 6 script: | 7 pnpm install --frozen-lockfile 8 9 - name: Install Cypress 10 script: pnpm exec cypress install 11 12 - name: Install Rust 13 - ... 14

Here are the available resource classes.

Dynamically Allocate Agents

Assign a different number of agents to a pipeline based on the size of the PR

Automatically Split E2E Tasks

Split large e2e tasks into separate tasks for each spec file

Identify and Re-run Flaky Tasks

Re-run flaky tasks in CI whenever they fail

CI/CD Guides

Every organization manages their CI/CD pipelines differently, so the guides don't cover org-specific aspects of CI/CD (e.g., deployment). They mainly focus on configuring Nx correctly using Nx Agents and Nx Replay.

Note that only cacheable operations can be distributed because they have to be replayed on the main job.

Relevant Repositories and Examples