Snakemake executor plugin: aws-batch
This is the Snakemake plugin for AWS Batch. This plugin is used to distribute Snakemake jobs to AWS Batch EC2 instances.
This plugin supports Snakemake 8 and 9.
Installation
Install this plugin by installing it with pip or mamba directly, e.g.:
pip install snakemake-executor-plugin-aws-batch
Or, if you are using pixi, add the plugin to your pixi.toml. Be careful to put it under the right dependency type based on the plugin’s availability, e.g.:
snakemake-executor-plugin-aws-batch = "*"
Usage
In order to use the plugin, run Snakemake (>=8.6) in the folder where your workflow code and config resides (containing either workflow/Snakefile or Snakefile) with the corresponding value for the executor flag:
snakemake --executor aws-batch --default-resources --jobs N ...
with N being the number of jobs you want to run in parallel and ... being any additional arguments you want to use (see below).
The machine on which you run Snakemake must have the executor plugin installed, and, depending on the type of the executor plugin, have access to the target service of the executor plugin (e.g. an HPC middleware like slurm with the sbatch command, or internet access to submit jobs to some cloud provider, e.g. azure).
The flag --default-resources ensures that Snakemake auto-calculates the mem and disk resources for each job, based on the input file size.
The values assumed there are conservative and should usually suffice.
However, you can always override those defaults by specifying the resources in your Snakemake rules or via the --set-resources flag.
Depending on the executor plugin, you might either rely on a shared local filesystem or use a remote filesystem or storage. For the latter, you have to additionally use a suitable storage plugin (see section storage plugins in the sidebar of this catalog) and eventually check for further recommendations in the sections below.
All arguments can also be persisted via a profile, such that they don’t have to be specified on each invocation. Here, this would mean the following entries inside of the profile
executor: aws-batch
default_resources: []
For specifying other default resources than the built-in ones, see the docs.
Settings
The executor plugin has the following settings (which can be passed via command line, the workflow or environment variables, if provided in the respective columns):
Further details
AWS Credentials
This plugin assumes you have setup AWS CLI credentials in ~/.aws/credentials. For more information see aws cli configuration.
AWS Infrastructure Requirements
The snakemake-executor-plugin-aws-batch requires an EC2 compute environment and a job queue to be configured. The plugin repo contains terraform used to setup the requisite AWS Batch infrastructure.
Assuming you have terraform installed and aws cli credentials configured, you can deploy the required infrastructure as follows:
cd terraform
terraform init
terraform plan
terraform apply
Resource names can be updated by including a terraform.tfvars file that specifies
variable name overrides of the defaults defined in vars.tf. The outputs variables from
running terraform apply can be exported as environment variables for snakemake-executor-plugin-aws-batch to use.
SNAKEMAKE_AWS_BATCH_REGION SNAKEMAKE_AWS_BATCH_JOB_QUEUE SNAKEMAKE_AWS_BATCH_JOB_ROLE
Required IAM permissions
The principal that runs Snakemake (the executor role) needs the permissions below. They are grouped so you can grant only what your submission path uses: a small always-required core, an add-on for the default (dynamic) job-definition path, and an add-on for tags.
Always required
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BatchCore",
"Effect": "Allow",
"Action": [
"batch:SubmitJob",
"batch:DescribeJobs",
"batch:TerminateJob"
],
"Resource": "*"
}
]
}
These three actions are exercised on every submission path: the plugin submits
each job, polls it with DescribeJobs, and terminates it on cancellation or
shutdown.
Add-on: default (dynamic) job-definition path
These permissions are used unless you point the plugin at a pre-existing job
definition (--aws-batch-job-definition or the per-rule
aws_batch_job_definition resource). On the default path the plugin inspects
the queue, registers a job definition per job, and deregisters it afterwards; a
pre-existing definition is submitted against directly with containerOverrides
and needs none of these.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BatchDynamicJobDefinition",
"Effect": "Allow",
"Action": [
"batch:RegisterJobDefinition",
"batch:DeregisterJobDefinition",
"batch:DescribeJobQueues",
"batch:DescribeComputeEnvironments"
],
"Resource": "*"
},
{
"Sid": "PassJobRole",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::<account-id>:role/<job-role-name>"
}
]
}
batch:DescribeJobQueuesandbatch:DescribeComputeEnvironmentslet the plugin detect whether a queue is backed by EC2 or Fargate so it builds a compatible job definition.batch:RegisterJobDefinitionandbatch:DeregisterJobDefinitionregister the per-job definition and deregister it afterwards.The
PassJobRolestatement covers passing the job role (--aws-batch-job-role/SNAKEMAKE_AWS_BATCH_JOB_ROLE) into the definition the plugin registers. AWS enforcesiam:PassRolewhere the role is supplied — atRegisterJobDefinition— so it is only needed on this path, scoped to the job-role ARN. The plugin refuses--aws-batch-job-roletogether with a pre-existing definition: there the job role is baked into the definition by whoever registered it (who needediam:PassRoleat that point), and the executor never passes it, soiam:PassRolecan be omitted. If aSubmitJobis ever denied citingiam:PassRole, grant it scoped to the definition’s role ARN.
Rule-Specific Container Images
By default, all jobs use the global container image specified via --container-image. However, you can specify a different container image for individual rules using the aws_batch_container_image resource parameter:
rule my_rule:
input:
"input.txt"
output:
"output.txt"
resources:
aws_batch_container_image="my-custom-image:tag"
shell:
"process_data.sh {input} {output}"
This allows you to use different containers with specialized tools for different rules within the same workflow, rather than requiring all tools to be present in a single container.
Each rule-specific image must still satisfy the same requirements as the global
container image (see Container Image Requirements
below): the worker runs snakemake inside the container, so the image must have
Snakemake and a compatible storage plugin installed. A plain tool image that does
not include Snakemake will fail to launch the job.
Container Image Map
A deployment that copies every image a workflow uses into its own registry (for example, pinned by digest in a private ECR repository) can make jobs run those copies without editing the workflow. Pass a JSON object mapping image references to the images to run:
{
"quay.io/biocontainers/samtools:1.21--h50ea8bc_0": "123456789012.dkr.ecr.us-east-1.amazonaws.com/tools@sha256:…",
"my-custom-image:tag": "123456789012.dkr.ecr.us-east-1.amazonaws.com/custom@sha256:…"
}
snakemake --executor aws-batch --aws-batch-container-image-map images.json ...
# or: SNAKEMAKE_AWS_BATCH_CONTAINER_IMAGE_MAP=images.json
Every job’s image, whether the global --container-image or a rule’s
aws_batch_container_image (including one computed by a function), is looked up in the
map as written and then in its normalized form: ubuntu is
docker.io/library/ubuntu:latest, registry names are case-insensitive, and a reference
with a digest matches with or without a tag. The map is strict: a job whose image is not
in it fails without being submitted, like any other failed job (jobs already running
finish, and --keep-going and --retries apply), so no job pulls an image the
deployment has not copied. To let an image through unchanged, map it to itself.
A map that names an image twice, or maps two spellings of one image to different images without an entry for the normalized spelling, is refused at startup.
The map cannot check the image of a pre-existing job definition,
so it cannot be combined with --aws-batch-job-definition (refused at startup) or the
per-rule aws_batch_job_definition resource (that job fails).
Example
Create environment
Install snakemake and the AWS executor and storage plugins into an environment. We recommend the use of mamba package manager which can be installed using miniforge, but these dependencies can also be installed using pip or other python package managers.
mamba create -n snakemake-example \
snakemake snakemake-storage-plugin-s3 snakemake-executor-plugin-aws-batch
mamba activate snakemake-example
Clone the snakemake tutorial repo containing the example workflow:
git clone https://github.com/snakemake/snakemake-tutorial-data.git
Setup and run tutorial workflow on the the executor
cd snakemake-tutorial-data
export SNAKEMAKE_AWS_BATCH_REGION=
export SNAKEMAKE_AWS_BATCH_JOB_QUEUE=
export SNAKEMAKE_AWS_BATCH_JOB_ROLE=
snakemake --jobs 4 \
--executor aws-batch \
--aws-batch-region us-west-2 \
--default-storage-provider s3 \
--default-storage-prefix s3://snakemake-tutorial-example \
--verbose
Container Image Requirements
The plugin supports Snakemake 8 and 9 and does not auto-deploy the default
storage provider to workers (auto_deploy_default_storage_provider=False):
workers do not run pip install snakemake-storage-plugin-s3 at startup, because
an unpinned install can pull a storage-plugin version whose
snakemake-interface-storage-plugins requirement does not match the Snakemake
major baked into the image, breaking the worker (e.g. Snakemake 8 needs a plugin
built against snakemake-interface-storage-plugins 3.x, while recent Snakemake 9
releases need one built against 4.x). The exact interface major tracks the
installed Snakemake release, so the container image used for jobs must pre-install
a storage-plugin version compatible with its Snakemake version. Image maintainers
are responsible for pinning a compatible pair.
Per-Rule Job Queues
By default all jobs are submitted to the queue given by
--aws-batch-job-queue. A rule can override this with the batch_queue
resource, e.g. to route jobs to a queue wired to a different compute
environment (ARM vs x86, GPU vs CPU):
rule align:
resources:
batch_queue="arn:aws:batch:us-west-2:123456789012:job-queue/arm-queue"
...
Platform detection and job submission both use the resolved per-rule queue.
Task Timeout
By default jobs have no timeout. Set --aws-batch-task-timeout to impose a
workflow-wide limit (in seconds; minimum 60). A rule can override this with the
aws_batch_task_timeout resource, e.g. to give a long-running alignment step
more time while keeping a tight limit on bookkeeping rules:
rule align:
resources:
aws_batch_task_timeout=14400 # 4 h
...
The per-rule resource takes precedence over --aws-batch-task-timeout. When
neither is set, AWS Batch imposes no timeout. When set, the value must be at
least 60 seconds (the AWS minimum). See
Spot Reclaim Retries for how it applies to a job that
AWS Batch retries in place.
Spot Reclaim Retries
When AWS reclaims an EC2 Spot instance, AWS Batch fails the jobs running on it
(their statusReason starts with Host EC2). By default the plugin treats that like
any other failure, so it spends one of the rule’s --retries on a new Batch job.
Set --aws-batch-spot-attempts (1-10) to have AWS Batch retry a host
termination itself instead, inside the same Batch job:
snakemake --executor aws-batch --aws-batch-spot-attempts 3 ...
Every submitted job then carries a retryStrategy with that many attempts that
retries only a host termination and exits on anything else, so a real failure
(bad input, out of memory, a bug) still reaches Snakemake and --retries as
before. This keeps the two apart: reclaims no longer use up --retries, and
--retries need not be raised to absorb them, which would re-run every real
failure that many more times. When the attempts run out, the job fails as it
does today and --retries takes over.
A rule can override the setting with the aws_batch_spot_attempts resource. For
example, aws_batch_spot_attempts=1 sends a reclaim straight to Snakemake, so a
rule whose queue depends on the attempt can move to an on-demand queue after one
reclaim:
rule align:
resources:
aws_batch_spot_attempts=1,
batch_queue=lambda wildcards, attempt: (
SPOT_QUEUE if attempt == 1 else ON_DEMAND_QUEUE
),
...
The per-rule resource takes precedence over --aws-batch-spot-attempts. When
neither is set, the plugin sends no retryStrategy, so a pre-existing job
definition’s own, if any, applies. The strategy is set on SubmitJob, so it
also applies to (and overrides the retryStrategy of) a pre-existing job
definition. A group job runs as one Batch job and gets the smallest of its
members’ values, so a member with aws_batch_spot_attempts=1 opts the group
out. A member whose resource is a function of inputs that do not exist yet when
the group is submitted counts as not setting it (it uses the setting). An invalid --aws-batch-spot-attempts fails at startup. An invalid
resource stops the workflow, cancelling its running jobs, when a job of that
rule is submitted (like an invalid aws_batch_task_timeout), so a value
computed from attempt must stay within 1-10 on every attempt.
Only an EC2 host termination is retried. A Fargate Spot interruption (possible
with a pre-existing job definition on a Fargate Spot queue) is reported
differently, so it is not retried in the job and reaches --retries as before.
A task timeout (--aws-batch-task-timeout / aws_batch_task_timeout) applies
to each attempt, so a job that is reclaimed and retried can run for up to
attempts times the timeout in all.
Task Logs
Unless told otherwise, AWS Batch sends every job’s output to the CloudWatch
Logs group /aws/batch/job, which all Batch jobs in the account and region
share. Because CloudWatch Logs grants read access per log group, anyone who may
read one workflow’s job logs there may read every workflow’s. Retention and KMS
encryption are also set per group. To send a workflow’s jobs to their own group,
set --aws-batch-log-group:
snakemake --executor aws-batch \
--aws-batch-log-group /snakemake/my-project \
...
This sets the awslogs log driver on each job definition the plugin registers,
with awslogs-group set to the given group and awslogs-region to
--aws-batch-region. It takes the group’s name, not its ARN. The group must
already exist (the plugin does not create it), and the compute environment’s ECS
instance role needs logs:CreateLogStream and logs:PutLogEvents on it. The
executor role needs no additional permissions; with logs:DescribeLogGroups it
also checks at startup that the group exists (see Preflight
Validation).
Pre-existing Job Definitions
By default the plugin registers a fresh AWS Batch job definition for every job
and deregisters it afterward. Accounts where job definitions are managed by
infrastructure tooling (Terraform, CloudFormation) can opt out of this with
--aws-batch-job-definition. When set, the plugin skips
RegisterJobDefinition/DeregisterJobDefinition entirely and instead submits
with the supplied definition, pushing per-job specifics (command, environment
variables, vcpu/mem/gpu) through containerOverrides.
Because the pre-existing definition supplies its own role, --aws-batch-job-role
is not required in this mode (it is required in the default register-per-job
mode). Providing it alongside --aws-batch-job-definition is rejected — see
Incompatible combinations below.
snakemake --executor aws-batch \
--aws-batch-job-definition my-snakemake-def:3 \
...
The value can be a bare name (my-def), a name:revision pair (my-def:3),
or a full ARN
(arn:aws:batch:us-east-1:123456789012:job-definition/my-def:3).
A rule can override the setting for a specific job with the
aws_batch_job_definition resource (mirrors the batch_queue pattern):
rule align:
resources:
aws_batch_job_definition="gpu-enabled-def:2"
...
Incompatible combinations — the following are rejected with a
WorkflowError because they are only meaningful when the plugin builds the
definition:
--aws-batch-job-role(job_role): the job role is baked into the definition at registration time and cannot be overridden viacontainerOverrides. Combined with the global--aws-batch-job-definitionit is rejected at startup (preflight); combined with a per-ruleaws_batch_job_definitionresource it is rejected when that job is submitted.The per-rule
shared_memory_size_mbresource:linuxParameters.sharedMemorySizeis a definition-level field (rejected at job submission).--aws-batch-log-group(log_group): the log configuration is a definition-level field (rejected at job submission).
The --aws-batch-container-image (container_image) setting and the per-rule
aws_batch_container_image resource are both silently ignored in this mode — the
container image is taken from the pre-existing definition.
For the same reason a container image map cannot be used with a
pre-existing definition.
Task timeout, scheduling priority and spot attempts are still honored:
--aws-batch-task-timeout (and the per-rule aws_batch_task_timeout resource),
the scheduling priority (--aws-batch-scheduling-priority / the per-rule
aws_batch_scheduling_priority resource) and the spot attempts
(--aws-batch-spot-attempts / the per-rule aws_batch_spot_attempts resource)
travel as SubmitJob‘s top-level timeout, schedulingPriorityOverride and
retryStrategy fields, so they apply to pre-existing definitions just as they
do to dynamically registered ones.
Preflight Validation
The executor sanity-checks the AWS Batch configuration so you don’t wait on jobs that could never start, in two stages.
At startup (before submitting any job) it verifies that the job role exists
(when --aws-batch-job-role is set and iam:GetRole is available) and that the
log group exists (when --aws-batch-log-group is set and
logs:DescribeLogGroups is available). A confirmed-missing role, or a malformed
or non-existent log group, fails fast with a clear error.
Job-queue and compute-environment health is checked per effective queue at
submission time, not at startup, because a rule can override the queue with the
batch_queue resource — so validating only the global --aws-batch-job-queue
at startup would both miss an unhealthy per-rule queue and wrongly reject a
workflow that never submits to an unhealthy global one. The first time a job is
submitted to a given queue, the executor verifies that the queue is ENABLED
and not in a failed/deleting state (status INVALID/DELETING/DELETED) and
that at least one of its compute environments is usable (ENABLED, not in a
failed/deleting state, and maxvCpus > 0 — AWS Batch falls back across the
queue’s computeEnvironmentOrder, so one healthy environment is enough). A
confirmed misconfiguration (a disabled/failed queue, or a queue with no usable
compute environment) fails fast before the job is submitted. Each distinct queue
is checked once and the result cached, so a workflow that fans jobs across
several queues validates every queue it uses without re-describing any of them.
Jobs that submit against a pre-existing job definition
(--aws-batch-job-definition or the per-rule aws_batch_job_definition
resource) skip this queue check: like the platform lookup, it is deliberately
omitted so those jobs keep the smaller IAM surface that static definitions are
meant to provide (no batch:DescribeJobQueues /
batch:DescribeComputeEnvironments required).
All of these checks are deliberately conservative about uncertainty: a
transient API error, a queue mid-update (status CREATING/UPDATING), or a missing
iam:GetRole or logs:DescribeLogGroups permission is logged as a warning and
the check is skipped rather than failing the workflow. The queue check reuses the
batch:DescribeJobQueues / batch:DescribeComputeEnvironments permissions the
executor already needs for platform detection (so a run missing those is not
blocked by validation, but will still fail later when the job definition is
built), plus the optional iam:GetRole for the job-role check and
logs:DescribeLogGroups for the log-group check.
When tags are configured (via --aws-batch-tags or the
SNAKEMAKE_AWS_BATCH_JOB_TAGS environment variable), the executor additionally
runs a tag/untag round-trip on the job queue as a best-effort proxy for the
batch:TagResource permission, so a missing permission surfaces as a warning at
startup rather than only as an opaque AccessDenied an hour into the run. It
probes the queue, whereas jobs are tagged on the job and job-definition
resources, so an IAM policy that scopes batch:TagResource per resource cannot
be fully verified this way — a denial is therefore logged as a warning (it does
not block the run) and a pass is a strong hint, not a guarantee. The probe
needs batch:TagResource (and batch:UntagResource to remove the throwaway
snakemake-preflight tag it writes; if that cleanup untag is denied, the tag is
left on the queue). Depending on your account’s tag-authorization settings you
may additionally need ecs:TagResource.