Snakemake storage plugin: irods

https://img.shields.io/badge/repository-github-blue?color=%23022c22 https://img.shields.io/badge/author-Oliver%20Stolpe-purple?color=%23064e3b PyPI - Version PyPI - License

A plugin for handling input and output on iRODS.

Installation

Install this plugin by installing it with pip or mamba, e.g.:

pip install snakemake-storage-plugin-irods

Usage

Queries

Queries to this storage should have the following format:

Query type

Query

Description

any

irods://folder/myfile.txt

A file in a folder on the iRODS server.

As default provider

If you want all your input and output (which is not explicitly marked to come from another storage) to be written to and read from this storage, you can use it as a default provider via:

snakemake --default-storage-provider irods --default-storage-prefix ...

with ... being the prefix of a query under which you want to store all your results. You can also pass custom settings via command line arguments:

snakemake --default-storage-provider irods --default-storage-prefix ... \
    --storage-irods-max-requests-per-second ... \        --storage-irods-host ... \        --storage-irods-port ... \        --storage-irods-username ... \        --storage-irods-password ... \        --storage-irods-zone ... \        --storage-irods-home ... \        --storage-irods-authentication-scheme ... \        --storage-irods-encryption-algorithm ... \        --storage-irods-encryption-key-size ... \        --storage-irods-encryption-num-hash-rounds ... \        --storage-irods-encryption-salt-size ... \        --storage-irods-client-server-negotiation ... \        --storage-irods-client-server-policy ... \        --storage-irods-ssl-verify-server ... \        --storage-irods-ssl-ca-certificate-file ... \        --storage-irods-use-ssl ...

Within the workflow

If you want to use this storage plugin only for specific items, you can register it inside of your workflow:

# register storage provider (not needed if no custom settings are to be defined here)
storage:
    provider="irods",
    # optionally add custom settings here if needed
    # alternatively they can be passed via command line arguments
    # starting with --storage-irods-..., see
    # snakemake --help
    # Maximum number of requests per second for this storage provider. If nothing is specified, the default implemented by the storage plugin is used.
    max_requests_per_second=...,
    # The host name of the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    host=...,
    # The port of the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    port=...,
    # The user name for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    username=...,
    # The password for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.  If no password is provided in the settings or environment file, the password in ~/.irods/.irodsA will be used with native authentication.
    password=...,
    # The zone for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    zone=...,
    # The home parameter for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    home=...,
    # The authentication scheme for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.  If no password is provided in the settings or environment file, the password in ~/.irods/.irodsA will be used with native authentication.
    authentication_scheme=...,
    # Encryption algorithm for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.
    encryption_algorithm=...,
    # Key size for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.
    encryption_key_size=...,
    # Number of hash rounds for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.
    encryption_num_hash_rounds=...,
    # Salt size for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.
    encryption_salt_size=...,
    # Set to 'request_server_negotiation' to enable SSL/TLS. Value in ~/.irods/irods_environment.json has higher priority, if present.
    client_server_negotiation=...,
    # CS_NEG_REFUSE: no SSL/TLS, CS_NEG_REQUIRE: enforce SSL/TLS, CS_NEG_DONT_CARE: let server decide. Value in ~/.irods/irods_environment.json has higher priority, if present.
    client_server_policy=...,
    # none: do not verify certificate, cert: verify certificate validity (but not hostname), hostname: verify certificate validity and hostname. Value in ~/.irods/irods_environment.json has higher priority, if present.
    ssl_verify_server=...,
    # Path to file with trusted CA certificates in PEM format. Used in conjunction with system default trusted certificates. Value in ~/.irods/irods_environment.json has higher priority, if present.
    ssl_ca_certificate_file=...,
    # Whether to use SSL when connecting to the server.
    use_ssl=...,

rule example:
    input:
        storage.irods(
            # define query to the storage backend here
            ...
        ),
    output:
        "example.txt"
    shell:
        "..."

Using multiple entities of the same storage plugin

In case you have to use this storage plugin multiple times, but with different settings (e.g. to connect to different storage servers), you can register it multiple times, each time providing a different tag:

# register shared settings
storage:
    provider="irods",
    # optionally add custom settings here if needed
    # alternatively they can be passed via command line arguments
    # starting with --storage-irods-..., see below
    # Maximum number of requests per second for this storage provider. If nothing is specified, the default implemented by the storage plugin is used.
    max_requests_per_second=...,
    # The host name of the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    host=...,
    # The port of the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    port=...,
    # The user name for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    username=...,
    # The password for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.  If no password is provided in the settings or environment file, the password in ~/.irods/.irodsA will be used with native authentication.
    password=...,
    # The zone for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    zone=...,
    # The home parameter for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    home=...,
    # The authentication scheme for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.  If no password is provided in the settings or environment file, the password in ~/.irods/.irodsA will be used with native authentication.
    authentication_scheme=...,
    # Encryption algorithm for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.
    encryption_algorithm=...,
    # Key size for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.
    encryption_key_size=...,
    # Number of hash rounds for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.
    encryption_num_hash_rounds=...,
    # Salt size for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.
    encryption_salt_size=...,
    # Set to 'request_server_negotiation' to enable SSL/TLS. Value in ~/.irods/irods_environment.json has higher priority, if present.
    client_server_negotiation=...,
    # CS_NEG_REFUSE: no SSL/TLS, CS_NEG_REQUIRE: enforce SSL/TLS, CS_NEG_DONT_CARE: let server decide. Value in ~/.irods/irods_environment.json has higher priority, if present.
    client_server_policy=...,
    # none: do not verify certificate, cert: verify certificate validity (but not hostname), hostname: verify certificate validity and hostname. Value in ~/.irods/irods_environment.json has higher priority, if present.
    ssl_verify_server=...,
    # Path to file with trusted CA certificates in PEM format. Used in conjunction with system default trusted certificates. Value in ~/.irods/irods_environment.json has higher priority, if present.
    ssl_ca_certificate_file=...,
    # Whether to use SSL when connecting to the server.
    use_ssl=...,

# register multiple tagged entities
storage foo:
    provider="irods",
    # optionally add custom settings here if needed
    # alternatively they can be passed via command line arguments
    # starting with --storage-irods-..., see below.
    # To only pass a setting to this tagged entity, prefix the given value with
    # the tag name, i.e. foo:max_requests_per_second=...
    # Maximum number of requests per second for this storage provider. If nothing is specified, the default implemented by the storage plugin is used.
    max_requests_per_second=...,
    # The host name of the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    host=...,
    # The port of the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    port=...,
    # The user name for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    username=...,
    # The password for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.  If no password is provided in the settings or environment file, the password in ~/.irods/.irodsA will be used with native authentication.
    password=...,
    # The zone for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    zone=...,
    # The home parameter for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.
    home=...,
    # The authentication scheme for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.  If no password is provided in the settings or environment file, the password in ~/.irods/.irodsA will be used with native authentication.
    authentication_scheme=...,
    # Encryption algorithm for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.
    encryption_algorithm=...,
    # Key size for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.
    encryption_key_size=...,
    # Number of hash rounds for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.
    encryption_num_hash_rounds=...,
    # Salt size for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.
    encryption_salt_size=...,
    # Set to 'request_server_negotiation' to enable SSL/TLS. Value in ~/.irods/irods_environment.json has higher priority, if present.
    client_server_negotiation=...,
    # CS_NEG_REFUSE: no SSL/TLS, CS_NEG_REQUIRE: enforce SSL/TLS, CS_NEG_DONT_CARE: let server decide. Value in ~/.irods/irods_environment.json has higher priority, if present.
    client_server_policy=...,
    # none: do not verify certificate, cert: verify certificate validity (but not hostname), hostname: verify certificate validity and hostname. Value in ~/.irods/irods_environment.json has higher priority, if present.
    ssl_verify_server=...,
    # Path to file with trusted CA certificates in PEM format. Used in conjunction with system default trusted certificates. Value in ~/.irods/irods_environment.json has higher priority, if present.
    ssl_ca_certificate_file=...,
    # Whether to use SSL when connecting to the server.
    use_ssl=...,

rule example:
    input:
        storage.foo(
            # define query to the storage backend here
            ...
        ),
    output:
        "example.txt"
    shell:
        "..."

Settings

The storage plugin has the following settings (which can be passed via command line, the workflow or environment variables, if provided in the respective columns):

CLI setting

Workflow setting

Envvar setting

Description

Default

Choices

Required

Type

--storage-irods-max-requests-per-second VALUE

max_requests_per_second

Maximum number of requests per second for this storage provider. If nothing is specified, the default implemented by the storage plugin is used.

None

str

--storage-irods-host VALUE

host

The host name of the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.

None

str

--storage-irods-port VALUE

port

The port of the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.

None

str

--storage-irods-username VALUE

username

SNAKEMAKE_STORAGE_IRODS_USERNAME

The user name for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.

None

str

--storage-irods-password VALUE

password

SNAKEMAKE_STORAGE_IRODS_PASSWORD

The password for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present. If no password is provided in the settings or environment file, the password in ~/.irods/.irodsA will be used with native authentication.

None

str

--storage-irods-zone VALUE

zone

The zone for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.

None

str

--storage-irods-home VALUE

home

The home parameter for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present.

None

str

--storage-irods-authentication-scheme VALUE

authentication_scheme

The authentication scheme for the iRODS server. Value in ~/.irods/irods_environment.json has higher priority, if present. If no password is provided in the settings or environment file, the password in ~/.irods/.irodsA will be used with native authentication.

'native'

str

--storage-irods-encryption-algorithm VALUE

encryption_algorithm

Encryption algorithm for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.

'AES-256-CBC'

str

--storage-irods-encryption-key-size VALUE

encryption_key_size

Key size for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.

32

str

--storage-irods-encryption-num-hash-rounds VALUE

encryption_num_hash_rounds

Number of hash rounds for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.

16

str

--storage-irods-encryption-salt-size VALUE

encryption_salt_size

Salt size for parallel transfer encryption. Value in ~/.irods/irods_environment.json has higher priority, if present.

8

str

--storage-irods-client-server-negotiation VALUE

client_server_negotiation

Set to ‘request_server_negotiation’ to enable SSL/TLS. Value in ~/.irods/irods_environment.json has higher priority, if present.

None

str

--storage-irods-client-server-policy VALUE

client_server_policy

CS_NEG_REFUSE: no SSL/TLS, CS_NEG_REQUIRE: enforce SSL/TLS, CS_NEG_DONT_CARE: let server decide. Value in ~/.irods/irods_environment.json has higher priority, if present.

'CS_NEG_REFUSE'

str

--storage-irods-ssl-verify-server VALUE

ssl_verify_server

none: do not verify certificate, cert: verify certificate validity (but not hostname), hostname: verify certificate validity and hostname. Value in ~/.irods/irods_environment.json has higher priority, if present.

'hostname'

str

--storage-irods-ssl-ca-certificate-file VALUE

ssl_ca_certificate_file

Path to file with trusted CA certificates in PEM format. Used in conjunction with system default trusted certificates. Value in ~/.irods/irods_environment.json has higher priority, if present.

None

str

--storage-irods-use-ssl VALUE

use_ssl

Whether to use SSL when connecting to the server.

False

str