Snakemake Executor Kueue

This is provided as example documentation. This comes from snakemake-executor-kueue. This is a snakemake executor plugin that enables interaction with Kueue. The plugin will install Python dependencies that are needed, and it’s assumed that you have installed Kueue and have queues configured.

Usage

You will need to create a cluster first. For local development we recommend kind:

$ kind create cluster

You will then need to install Kueue and create your local queues with cluster quotas.

E.g., here is an example:

VERSION=v0.4.0
kubectl apply -f https://github.com/kubernetes-sigs/kueue/releases/download/$VERSION/manifests.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/kueue/main/site/static/examples/single-clusterqueue-setup.yaml

You’ll also need kubernetes python installed. We recommend a virtual environment (also with snakemake)

python -m venv env
source env/bin/activate
pip install kubernetes requests snakemake

And of course install the plugin! From the cloned repository you can do:

pip install .

Job Resources

Operator

By default, Kueue will use a batchv1/Job for each step. However, you can customize this to a different operator with the job resources via the kueue.operator attribute:

Note this does not work yet, as the resource names are checked

rule a:
    input:     ...
    output:    ...
    resources:
        kueue.operator=flux-operator
    shell:
        "..."

We currently support the following operators:

• flux-operator: deploy using the Flux Operator
• mpi-operator: deploy using the MPI Operator
• job: (the default) either unset, or set to “job”

Note that you are in charge of installing and configuring the various operators on your cluster! See the Kueue tasks for more details.

Container

If you are using the Flux operator, you need a container with Flux and your software! We have prepared a container with Flux, Snakemake, and Mamba for you to get started. The Dockerfile is here and you can use our build as follows:

rule a:
    input:     ...
    output:    ...
    resources:
        container=ghcr.io/rse-ops/mamba:app-mamba
    shell:
        "..."

Memory

The memory defined for Kubernetes is in a string format, and while we could ideally do a conversion for now we are lazy and ask you to define it directly:

rule a:
    input:     ...
    output:    ...
    resources:
        kueue.memory=200M1
    shell:
        "..."

Tasks

The Flux Operator can handle tasks for MPI, so you can set them as follows:

rule a:
    input:     ...
    output:    ...
    resources:
        kueue.tasks=1
    shell:
        "..."

Want to write a plugin?

If you are interested in writing your own plugin, instructions are provided via the snakemake-executor-plugin-interface. If you need to include large blocks of code, we recommend a details section:

rule a:
    input:     ...
    output:    ...
    resources:
        kueue.tasks=1
    shell:
        "..."