Snakemake executor plugin: aws-batch-flex

Warning

No repository URL found in Pypi metadata. The plugin should specify a repository URL in its pyproject.toml (key 'repository'). It is unclear whether the plugin is maintained and reviewed by the official Snakemake organization (https://github.com/snakemake).

Installation

Install this plugin by installing it with pip or mamba directly, e.g.:

pip install snakemake-executor-plugin-aws-batch-flex

Or, if you are using pixi, add the plugin to your pixi.toml. Be careful to put it under the right dependency type based on the plugin’s availability, e.g.:

snakemake-executor-plugin-aws-batch-flex = "*"

Usage

In order to use the plugin, run Snakemake (>=8.6) in the folder where your workflow code and config resides (containing either workflow/Snakefile or Snakefile) with the corresponding value for the executor flag:

snakemake --executor aws-batch-flex --default-resources --jobs N ...

with N being the number of jobs you want to run in parallel and ... being any additional arguments you want to use (see below). The machine on which you run Snakemake must have the executor plugin installed, and, depending on the type of the executor plugin, have access to the target service of the executor plugin (e.g. an HPC middleware like slurm with the sbatch command, or internet access to submit jobs to some cloud provider, e.g. azure).

The flag --default-resources ensures that Snakemake auto-calculates the mem and disk resources for each job, based on the input file size. The values assumed there are conservative and should usually suffice. However, you can always override those defaults by specifying the resources in your Snakemake rules or via the --set-resources flag.

Depending on the executor plugin, you might either rely on a shared local filesystem or use a remote filesystem or storage. For the latter, you have to additionally use a suitable storage plugin (see section storage plugins in the sidebar of this catalog) and eventually check for further recommendations in the sections below.

All arguments can also be persisted via a profile, such that they don’t have to be specified on each invocation. Here, this would mean the following entries inside of the profile

executor: aws-batch-flex
default_resources: []

For specifying other default resources than the built-in ones, see the docs.

Settings

The executor plugin has the following settings (which can be passed via command line, the workflow or environment variables, if provided in the respective columns):

Settings

CLI argument	Description	Default	Choices	Required	Type
--aws-batch-flex-queue VALUE	AWS Batch job queue to submit to. A rule can override this with the resource batch_queue.	None		✓
--aws-batch-flex-region VALUE	AWS region. Defaults to the standard boto3 resolution (AWS_REGION, AWS_PROFILE, ~/.aws/config).	None		✗
--aws-batch-flex-job-role-arn VALUE	IAM role assumed by the job container (needs access to the storage bucket). If omitted, the compute environment’s instance role is used (EC2 only, and only if the instances let containers reach it: jobs_use_instance_role in the Terraform setup).	None		✗
--aws-batch-flex-execution-role-arn VALUE	ECS task execution role. Required for Fargate and for pulling from private registries.	None		✗
--aws-batch-flex-head-role-arn VALUE	Only for smkflex run: IAM role of the head job that runs Snakemake on AWS (Terraform prints it as head_role_arn). Snakemake itself ignores it.	None		✗
--aws-batch-flex-platform VALUE	Compute platform of the main queue: ‘ec2’ (also covers EC2 Spot) or ‘fargate’. Default: detected per queue from its compute environment (EC2 and Fargate queues can be mixed via the batch_queue resource); falls back to ec2 when the queue cannot be inspected. A rule can force it with the batch_platform resource.	None		✗
--aws-batch-flex-default-image VALUE	Image for rules without a container: directive. Defaults to Snakemake’s –container-image.	None		✗
--aws-batch-flex-bootstrap VALUE	How to get Snakemake into job containers. ‘host’ (EC2, recommended; Nextflow-style): mount a Snakemake env pre-installed on the host (see snakemake-batch-flex-host-env), so images need only /bin/sh. ‘auto’: install at job start only if the image lacks the matching version; ‘always’: always install at job start; ‘never’: the image must already contain Snakemake and the storage plugins.	'auto'		✗
--aws-batch-flex-host-env-path VALUE	Path of the host-installed Snakemake env, mounted read-only at the same path in every container (bootstrap=host).	'/opt/snakemake-env'		✗
--aws-batch-flex-host-env-wait-seconds VALUE	bootstrap=host: how long a job waits for the host env to become ready (instances installing it via user-data). Use 0 on a baked AMI.	120		✗
--aws-batch-flex-install-prereqs VALUE	Install-at-start modes (auto/always): if the image has no curl, wget or python3+pip, install curl/ca-certificates/bash with the image’s package manager (apt/apk/dnf/yum). Needs root in the image and access to the distro mirrors.	False		✗
--aws-batch-flex-script-python VALUE	Which python runs Python script: rules. ‘image’: the image’s python if it is 3.10 or newer (so the script sees the image’s packages), else Snakemake’s. ‘host’: always Snakemake’s own python. Per rule: the batch_script_python resource.	'image'		✗
--aws-batch-flex-mem-overhead-mib VALUE	MiB added to each rule’s mem_mb/mem_mib for the Snakemake process that runs inside the job and shares its memory limit. 0 = request exactly mem_mb.	256		✗
--aws-batch-flex-skip-preflight VALUE	Skip the startup checks of the queue, its compute environment and the job role.	False		✗
--aws-batch-flex-conda-path VALUE	Host-installed conda (see snakemake-batch-flex-host-env conda / bake-ami.sh –conda), mounted read-only at the same path. Lets any image run conda: rules, including existing named environments in <path>/envs. EC2 only.	None		✗
--aws-batch-flex-volumes VALUE	Extra host bind mounts, HOST:CONTAINER[:ro] (EC2 only), like Nextflow’s aws.batch.volumes. Example: /scratch:/scratch	None		✗
--aws-batch-flex-extra-packages VALUE	Extra pip requirements installed alongside Snakemake when bootstrapping (e.g. additional Snakemake plugins).	None		✗
--aws-batch-flex-python-version VALUE	Python version for the bootstrap venv. Defaults to the version running this Snakemake.	None		✗
--aws-batch-flex-spot-retries VALUE	How many times AWS Batch itself retries a job whose host was reclaimed (Spot interruption), 0-9; 0 = none. Independent of Snakemake’s –retries. Per rule: the batch_spot_retries resource.	2		✗
--aws-batch-flex-scheduling-priority VALUE	Scheduling priority (0-9999, higher runs first) for every job. Only works on queues with a fair-share scheduling policy; skipped with a warning otherwise. Per rule: the batch_priority resource.	None		✗
--aws-batch-flex-log-group VALUE	CloudWatch log group for job logs. The default needs no setup.	'/aws/batch/job'		✗
--aws-batch-flex-job-definition VALUE	Use this existing job definition (name, name:revision or ARN) instead of registering one per image; per rule: the batch_job_definition resource. It fixes the image, job role, mounts and /dev/shm; vCPUs, memory, timeout and tags still come from the rules. Without the host-env mount, jobs install Snakemake at start.	None		✗
--aws-batch-flex-job-definition-prefix VALUE	Prefix for auto-registered job definitions.	'snakemake'		✗
--aws-batch-flex-assign-public-ip VALUE	Fargate only: ‘ENABLED’ or ‘DISABLED’. DISABLED needs a NAT gateway or VPC endpoints for image pulls and bootstrapping.	'ENABLED'		✗
--aws-batch-flex-ephemeral-storage-gib VALUE	Fargate only: minimum scratch disk in GiB (21-200); a rule’s disk_mb can raise it for that rule.	None		✗
--aws-batch-flex-registry-credentials VALUE	REGISTRY=SECRET_ARN pairs for private registries outside AWS, e.g. docker.io=arn:aws:secretsmanager:…:secret:dockerhub (the secret holds {“username”: …, “password”: …}). Only images from that registry use it. Needs –aws-batch-flex-execution-role-arn with secretsmanager:GetSecretValue on the secret. ECR images need no credentials (the instance/execution role pulls them).	None		✗
--aws-batch-flex-secrets VALUE	NAME=ARN pairs: environment variables jobs get from an SSM parameter (arn:aws:ssm:…:parameter/…) or a Secrets Manager secret. ECS reads them when the job starts, so unlike –envvars the value is never in the job’s settings, the Batch console or CloudTrail. Needs –aws-batch-flex-execution-role-arn allowed to read them.	None		✗
--aws-batch-flex-orphan-jobs VALUE	What to do with jobs still running from an earlier, interrupted run of the same workflow (same directory, Snakefile and storage prefix), which Snakemake would otherwise run a second time: ‘cancel’ (default), ‘warn’ (default with –nolock) or ‘ignore’. Needs batch:ListJobs.	None		✗
--aws-batch-flex-tags VALUE	KEY=VALUE tags applied to every job and to job definitions (cost allocation). Rules can add or override tags for their own jobs with the resource batch_tags=”k1=v1,k2=v2”.	None		✗
--aws-batch-flex-conda-cache VALUE	Reuse conda environments instead of building them in every job: ‘instance’ (jobs on the same EC2 machine share them), an S3 URL such as s3://BUCKET/snakemake/.smkflex/conda-cache (also kept there for later jobs and runs), or ‘none’.	'instance'		✗
--aws-batch-flex-oom-memory-factor VALUE	When a job ran out of memory and Snakemake retries it (–retries, or a rule’s retries:), ask for this many times more memory on each retry. 1 = never.	2.0		✗
--aws-batch-flex-no-usage-summary VALUE	Don’t print the per-rule summary of what the run used (vCPU-hours, memory, CPUs, disk, and the resources to paste into the profile) at the end of the run.	False		✗
--aws-batch-flex-no-auto-tags VALUE	Disable the automatic job tags snakemake-run-id, snakemake-workflow, snakemake-rule and snakemake-job-id (on by default, for per-rule cost reporting). Also turns off the check for jobs left over from an interrupted run.	False		✗
--aws-batch-flex-max-sources-mb VALUE	Every job gets the workflow folder (what git would track, or everything .gitignore doesn’t exclude, and config/; single files over 10 MB are left out). Stop with a message when that is more than this many MiB (as du -sh shows sizes): data belongs in storage.	100		✗
--aws-batch-flex-runnable-warn-minutes VALUE	Warn when a job has been RUNNABLE (waiting for capacity) this long; usually means the request exceeds what the compute environment can provide.	15		✗