Snakemake executor plugin: lsf-sanger

Warning

This plugin is not maintained and reviewed by the official Snakemake organization.

LSF is common high performance computing batch system. This is a version of the generic LSF executor plugin modified to better integrate with the Sanger compute environment and simplify pipeline execution.

Rule time and memory requirements are used to select an appropriate farm queue automatically, so users don’t have to explicitly specify LSF queues for rules with resource declarations.

Installation

Install this plugin by installing it with pip or mamba directly, e.g.:

pip install snakemake-executor-plugin-lsf-sanger

Or, if you are using pixi, add the plugin to your pixi.toml. Be careful to put it under the right dependency type based on the plugin’s availability, e.g.:

snakemake-executor-plugin-lsf-sanger = "*"

Usage

In order to use the plugin, run Snakemake (>=8.6) in the folder where your workflow code and config resides (containing either workflow/Snakefile or Snakefile) with the corresponding value for the executor flag:

snakemake --executor lsf-sanger --default-resources --jobs N ...

with N being the number of jobs you want to run in parallel and ... being any additional arguments you want to use (see below). The machine on which you run Snakemake must have the executor plugin installed, and, depending on the type of the executor plugin, have access to the target service of the executor plugin (e.g. an HPC middleware like slurm with the sbatch command, or internet access to submit jobs to some cloud provider, e.g. azure).

The flag --default-resources ensures that Snakemake auto-calculates the mem and disk resources for each job, based on the input file size. The values assumed there are conservative and should usually suffice. However, you can always override those defaults by specifying the resources in your Snakemake rules or via the --set-resources flag.

Depending on the executor plugin, you might either rely on a shared local filesystem or use a remote filesystem or storage. For the latter, you have to additionally use a suitable storage plugin (see section storage plugins in the sidebar of this catalog) and eventually check for further recommendations in the sections below.

All arguments can also be persisted via a profile, such that they don’t have to be specified on each invocation. Here, this would mean the following entries inside of the profile

executor: lsf-sanger
default_resources: []

For specifying other default resources than the built-in ones, see the docs.

Further details

Workflow Resource Specifications

A workflow rule may support a number of resource specifications. For a LSF cluster, a mapping between Snakemake and LSF needs to be performed.

You can use the following specifications:

LSF	Snakemake	Description
-q	lsf_queue	the queue a rule/job is to use
--W	walltime	the walltime per job in minutes
-R "rusage[mem=<memory_amount>]"	mem, mem_mb	memory a cluster node must provide
		(mem: string with unit, mem_mb: i)
-R "rusage[ngpus_physical=<gpus>"]	gpu	GPUs to request for the job
-R "rusage[mem=<memory_amount>]"	mem_mb_per_cpu	memory per reserved CPU
omit -R span[hosts=1]	mpi	Allow splitting across nodes for MPI
-R span[ptile=<ptile>]	ptile	Processors per host. Reqires mpi
Other bsub arguments	lsf_extra	Other args to pass to bsub (str)

Each of these can be part of a rule, e.g.:

rule:
    input: ...
    output: ...
    resources:
        partition: <partition name>
        walltime: <some number>

walltime and runtime are synonyms.

Please note: as --mem and --mem-per-cpu are mutually exclusive, their corresponding resource flags mem/mem_mb and mem_mb_per_cpu are mutually exclusive, too. You can only reserve memory a compute node has to provide or the memory required per CPU (LSF does not make any distintion between real CPU cores and those provided by hyperthreads). The executor will convert the provided options based on cluster config.

Additional custom job configuration

There are various bsub options not directly supported via the resource definitions shown above. You may use the lsf_extra resource to specify additional flags to bsub:

rule myrule:
    input: ...
    output: ...
    resources:
        lsf_extra="-R a100 -gpu num=2"

You can use a configuration profile to specify such resources outside of the rule definition.

An example profile may look like this:

jobs: <max concurrent jobs>
executor: lsf-sanger
default-resources:
  - 'lsf_project=<your LSF project>'

set-resources:
  your_rule_name:
    walltime: 1000
    gpu: 2

Specifying Project and Queue

LSF clusters can have mandatory resource indicators for accounting and scheduling, Project and Queue, respectively. These resources are usually omitted from Snakemake workflows in order to keep the workflow definition independent from the platform. However, it is also possible to specify them inside of the workflow as resources in the rule definition (see the Resources document).

To specify them at the command line, define them as default resources:

$ snakemake --executor lsf --default-resources lsf_project=<your LSF project> lsf_queue=<your LSF queue>

If individual rules require e.g. a different queue, you can override the default per rule:

$ snakemake --executor lsf --default-resources lsf_project=<your LSF project> lsf_queue=<your LSF queue> --set-resources <somerule>:lsf_queue=<some other queue>

Usually, it is advisable to persist such settings via a configuration profile, which can be provided system-wide, per user, and in addition per workflow.

Ordinary SMP jobs

Most jobs will be carried out by programs which are either single core scripts or threaded programs, hence SMP (shared memory programs) in nature. Any given threads and mem_mb requirements will be passed to LSF:

rule a:
    input: ...
    output: ...
    threads: 8
    resources:
        mem_mb=14000

This will give jobs from this rule 14GB of memory and 8 CPU cores. It is advisable to use resonable default resources, such that you don't need to specify them for every rule. Snakemake already has reasonable defaults built in, which are automatically activated when using any non-local executor (hence also with lsf). Use mem_mb_per_cpu to give the standard LSF type memory per CPU

MPI jobs

Snakemake's LSF backend also supports MPI jobs, see the MPI support document for details.

rule calc_pi:
  output:
      "pi.calc",
  log:
      "logs/calc_pi.log",
  threads: 40
  resources:
      tasks=10,
      mpi='mpirun,
  shell:
      "{resources.mpi} -np {resources.tasks} calc-pi-mpi > {output} 2> {log}"

$ snakemake --set-resources calc_pi:mpi="mpiexec" ...

Per-job vs per-core

By default, this plugin keeps the specified memory request as a per-job, as expected by the Sanger LSF cluster. If for some reason you want the request to be per-CPU core (i.e. -R rusage[mem=<mem_mb/threads>]) then set the environment variable SNAKEMAKE_LSF_MEMFMT to percpu.

The executor automatically detects the request unit from cluster configuration, so if your cluster does not use MB, you do not need to do anything.