Snakemake storage plugin: xrootd

https://img.shields.io/badge/repository-unknown-blue?color=%23022c22 https://img.shields.io/badge/author-Chris%20Burr%20%3Cchristopher.burr%40cern.ch%3E-purple?color=%23064e3b https://img.shields.io/badge/author-Johannes%20Koester%20%3Cjohannes.koester%40uni--due.de%3E-purple?color=%23064e3b https://img.shields.io/badge/author-Matthew%20Monk%20%3Cmatthew.david.monk%40cern.ch%3E-purple?color=%23064e3b PyPI - Version PyPI - License

Warning

No repository URL found in Pypi metadata. The plugin should specify a repository URL in its pyproject.toml (key 'repository'). It is unclear whether the plugin is maintained and reviewed by the official Snakemake organization (https://github.com/snakemake).

Installation

Install this plugin by installing it with pip or mamba, e.g.:

pip install snakemake-storage-plugin-xrootd

Usage

Queries

Queries to this storage should have the following format:

Query type

Query

Description

any

root://eosuser.cern.ch//eos/user/s/someuser/somefile.txt

A file on a XrootD instance not specifying any arguments.

any

root://eos/user/s/someuser/somefile.txt

A file on an XrootD instance where the host has beenspecified in the storage object.

As default provider

If you want all your input and output (which is not explicitly marked to come from another storage) to be written to and read from this storage, you can use it as a default provider via:

snakemake --default-storage-provider xrootd --default-storage-prefix ...

with ... being the prefix of a query under which you want to store all your results. You can also pass custom settings via command line arguments:

snakemake --default-storage-provider xrootd --default-storage-prefix ... \
    --storage-xrootd-max-requests-per-second ... \
    --storage-xrootd-host ... \
    --storage-xrootd-port ... \
    --storage-xrootd-username ... \
    --storage-xrootd-password ... \
    --storage-xrootd-url-decorator ...

Within the workflow

If you want to use this storage plugin only for specific items, you can register it inside of your workflow:

# register storage provider (not needed if no custom settings are to be defined here)
storage:
    provider="xrootd",
    # optionally add custom settings here if needed
    # alternatively they can be passed via command line arguments
    # starting with --storage-xrootd-..., see
    # snakemake --help
    # Maximum number of requests per second for this storage provider. If nothing is specified, the default implemented by the storage plugin is used.
    max_requests_per_second=...,
    # The XrootD host to connect to
    host=...,
    # The XrootD port to connect to
    port=...,
    # The username to use for authentication
    username=...,
    # The password to use for authentication. NOTE: Only use this setting in trusted environments! Snakemake will print the password in plaintext as part of the XRootD URLs used in the inputs/outputs of jobs.
    password=...,
    # A Python expression (given as str) to decorate the URL (which is available as variable `url` in the expression) e.g. to decorate it with an auth token.
    url_decorator=...,

rule example:
    input:
        storage.xrootd(
            # define query to the storage backend here
            ...
        ),
    output:
        "example.txt"
    shell:
        "..."

Using multiple entities of the same storage plugin

In case you have to use this storage plugin multiple times, but with different settings (e.g. to connect to different storage servers), you can register it multiple times, each time providing a different tag:

# register shared settings
storage:
    provider="xrootd",
    # optionally add custom settings here if needed
    # alternatively they can be passed via command line arguments
    # starting with --storage-xrootd-..., see below
    # Maximum number of requests per second for this storage provider. If nothing is specified, the default implemented by the storage plugin is used.
    max_requests_per_second=...,
    # The XrootD host to connect to
    host=...,
    # The XrootD port to connect to
    port=...,
    # The username to use for authentication
    username=...,
    # The password to use for authentication. NOTE: Only use this setting in trusted environments! Snakemake will print the password in plaintext as part of the XRootD URLs used in the inputs/outputs of jobs.
    password=...,
    # A Python expression (given as str) to decorate the URL (which is available as variable `url` in the expression) e.g. to decorate it with an auth token.
    url_decorator=...,

# register multiple tagged entities
storage foo:
    provider="xrootd",
    # optionally add custom settings here if needed
    # alternatively they can be passed via command line arguments
    # starting with --storage-xrootd-..., see below.
    # To only pass a setting to this tagged entity, prefix the given value with
    # the tag name, i.e. foo:max_requests_per_second=...
    # Maximum number of requests per second for this storage provider. If nothing is specified, the default implemented by the storage plugin is used.
    max_requests_per_second=...,
    # The XrootD host to connect to
    host=...,
    # The XrootD port to connect to
    port=...,
    # The username to use for authentication
    username=...,
    # The password to use for authentication. NOTE: Only use this setting in trusted environments! Snakemake will print the password in plaintext as part of the XRootD URLs used in the inputs/outputs of jobs.
    password=...,
    # A Python expression (given as str) to decorate the URL (which is available as variable `url` in the expression) e.g. to decorate it with an auth token.
    url_decorator=...,

rule example:
    input:
        storage.foo(
            # define query to the storage backend here
            ...
        ),
    output:
        "example.txt"
    shell:
        "..."

Settings

The storage plugin has the following settings (which can be passed via command line, the workflow or environment variables, if provided in the respective columns):

Settings

CLI argument

Description

Default

Choices

Required

Type

--storage-xrootd-max-requests-per-second VALUE

Maximum number of requests per second for this storage provider. If nothing is specified, the default implemented by the storage plugin is used.

None

--storage-xrootd-host VALUE

The XrootD host to connect to

None

--storage-xrootd-port VALUE

The XrootD port to connect to

None

--storage-xrootd-username VALUE

The username to use for authentication

None

--storage-xrootd-password VALUE

The password to use for authentication. NOTE: Only use this setting in trusted environments! Snakemake will print the password in plaintext as part of the XRootD URLs used in the inputs/outputs of jobs.

None

--storage-xrootd-url-decorator VALUE

A Python expression (given as str) to decorate the URL (which is available as variable url in the expression) e.g. to decorate it with an auth token.

None