ETH-NEXUS/gExcite_pipeline

single-cell Gene and AnTibody Expression pipeline

Overview

Topics:

Latest release: None, Last update: 2023-05-24

Linting: linting: failed, Formatting: formatting: failed

Deployment

Step 1: Install Snakemake and Snakedeploy

Snakemake and Snakedeploy are best installed via the Mamba package manager (a drop-in replacement for conda). If you have neither Conda nor Mamba, it is recommended to install Miniforge. More details regarding Mamba can be found here.

When using Mamba, run

mamba create -c conda-forge -c bioconda --name snakemake snakemake snakedeploy

to install both Snakemake and Snakedeploy in an isolated environment. For all following commands ensure that this environment is activated via

conda activate snakemake

Step 2: Deploy workflow

With Snakemake and Snakedeploy installed, the workflow can be deployed as follows. First, create an appropriate project working directory on your system and enter it:

mkdir -p path/to/project-workdir
cd path/to/project-workdir

In all following steps, we will assume that you are inside of that directory. Then run

snakedeploy deploy-workflow https://github.com/ETH-NEXUS/gExcite_pipeline . --tag None

Snakedeploy will create two folders, workflow and config. The former contains the deployment of the chosen workflow as a Snakemake module, the latter contains configuration files which will be modified in the next step in order to configure the workflow to your needs.

Step 3: Configure workflow

To configure the workflow, adapt config/config.yml to your needs following the instructions below.

Step 4: Run workflow

The deployment method is controlled using the --software-deployment-method (short --sdm) argument.

To run the workflow with automatic deployment of all required software via conda/mamba, use

snakemake --cores all --sdm conda

Snakemake will automatically detect the main Snakefile in the workflow subfolder and execute the workflow module that has been defined by the deployment in step 2.

For further options such as cluster and cloud execution, see the docs.

Step 5: Generate report

After finalizing your data analysis, you can automatically generate an interactive visual HTML report for inspection of results together with parameters and code inside of the browser using

snakemake --report report.zip

Configuration

The following section is imported from the workflow’s config/README.md.

Configuration

Before using the pipeline the following files need to be provided/adapted:

  1. config.yaml
  2. samplemap
  3. HashingFile
  4. featureReferenceFile

config.yaml

Before running the pipeline the config.yaml file needs to be adapted to contain the input and output paths for the intended analysis.

Adaptation necessary for a default run:

  • In section [inputOutput] the input directories input_fastqs_gex, input_fastqs_adt need to point to the location of the respective FASTQ files. This location needs to be specified relative to the gExcite working directory (usually gExcite_pipeline/).
  • In section [resources], reference_transcriptome needs to point to the location of the genomic reference used for the cellranger mapping
  • In sections [tools][cellranger_count_gex] and [tools][cellranger_count_adt], call needs to point to the the path to the cellranger installation
  • Section [computingResources] needs to list the resources that can be assigned to the analysis steps your data and batch system. The example resources specify memory per job, not thread.
  • Section [scampi][resources] needs to be filled with the cell type information, selected genes to show in expression plots, and gene sets for the GSVA analysis.

The default example config file is pointing to the example data input files.

samplemap

Further, a "samplemap" must be provided specifying sample-specific parameters in a tab-delimited text file. A pre-configured samplemap ready to run on the test data that can be adapted is provided in this directory.

Example samplemap:

sample    HashingFile     SeqRunName      nTargetCells    featureReferenceFile
sampleA   HashingFileA    SeqRunNameA     10000           featureReferenceFileA
sampleB   HashingFileB    SeqRunNameB     15000           featureReferenceFileB

With one line per set of samples

  • sample contains the sample identifier that is used throughout the pipeline
  • HashingFile contains the full path to the comma separated text file containing the hashtag barcodes and their assignment to individual sample names (see HashingFile).
  • SeqRunName corresponds to the sequencing sample name of the ADT sample; this parameter is only required for the Cellranger run of ADT data. It can be retrieved from the FASTQ file names as follows:
[SeqRunName]_S[Number]_L00[Lane Number]_[Read Type]_001.fastq.gz

Where Read Type is one of: I1, R1, R2.

  • nTargetCells corresponds to the number of targeted cells for the sample.
  • featureReferenceFile corresponds to the ADT feature reference file for the sample set. For further information please consult the Cellranger tool documentation.
    NOTE: the path to the FeatureReferenceFile must be relative to the gExcite working directory (usually gExcite_pipeline/).

HashingFile

In case of hashed samples, the hashtag barcodes, the hashtag names, and the corresponding sample names must be associated with the sample set. To do so, we need a comma-separated file with the following structure:

Barcode1,TagName1,sampleA
Barcode2,TagName2,sampleB

 featureReferenceFile

The "featureReferenceFile" is a comma-separated text file describing all ADT antibodies used in the experiment at hand. For further information please consult the Cellranger tool documentation.
An example feature_reference.txt ready to run on the test data is available in the testdata directory.

Linting and formatting

Linting results

  1/home/runner/micromamba-root/envs/snakemake-workflow-catalog/lib/python3.11/site-packages/google/protobuf/internal/api_implementation.py:110: UserWarning: Selected implementation cpp is not available.
  2  warnings.warn(
  3Lints for snakefile /tmp/tmp7di3duuu/workflow/rules/misc_snake.smk:
  4    * Absolute path "/" + samples[" in line 111:
  5      Do not define absolute paths inside of the workflow, since this renders
  6      your workflow irreproducible on other machines. Use path relative to the
  7      working directory instead, or make the path configurable via a config
  8      file.
  9      Also see:
 10      https://snakemake.readthedocs.io/en/latest/snakefiles/configuration.html#configuration
 11    * Path composition with '+' in line 59:
 12      This becomes quickly unreadable. Usually, it is better to endure some
 13      redundancy against having a more readable workflow. Hence, just repeat
 14      common prefixes. If path composition is unavoidable, use pathlib or
 15      (python >= 3.6) string formatting with f"...".
 16      Also see:
 17
 18    * Path composition with '+' in line 8:
 19      This becomes quickly unreadable. Usually, it is better to endure some
 20      redundancy against having a more readable workflow. Hence, just repeat
 21      common prefixes. If path composition is unavoidable, use pathlib or
 22      (python >= 3.6) string formatting with f"...".
 23      Also see:
 24
 25    * Path composition with '+' in line 66:
 26      This becomes quickly unreadable. Usually, it is better to endure some
 27      redundancy against having a more readable workflow. Hence, just repeat
 28      common prefixes. If path composition is unavoidable, use pathlib or
 29      (python >= 3.6) string formatting with f"...".
 30      Also see:
 31
 32
 33Lints for snakefile /tmp/tmp7di3duuu/workflow/rules/gex_cellranger.smk:
 34    * Absolute path "/" + config[" in line 11:
 35      Do not define absolute paths inside of the workflow, since this renders
 36      your workflow irreproducible on other machines. Use path relative to the
 37      working directory instead, or make the path configurable via a config
 38      file.
 39      Also see:
 40      https://snakemake.readthedocs.io/en/latest/snakefiles/configuration.html#configuration
 41    * Path composition with '+' in line 11:
 42      This becomes quickly unreadable. Usually, it is better to endure some
 43      redundancy against having a more readable workflow. Hence, just repeat
 44      common prefixes. If path composition is unavoidable, use pathlib or
 45      (python >= 3.6) string formatting with f"...".
 46      Also see:
 47
 48
 49Lints for snakefile /tmp/tmp7di3duuu/workflow/rules/adt_cellranger.smk:
 50    * Absolute path "/results/pooled_samples/cellranger_adt/{sample_set}.adt_library.txt" in line 18:
 51      Do not define absolute paths inside of the workflow, since this renders
 52      your workflow irreproducible on other machines. Use path relative to the
 53      working directory instead, or make the path configurable via a config
 54      file.
 55      Also see:
 56      https://snakemake.readthedocs.io/en/latest/snakefiles/configuration.html#configuration
 57    * Absolute path "/results/pooled_samples/cellranger_adt/{sample_set}.adt_library.txt" in line 39:
 58      Do not define absolute paths inside of the workflow, since this renders
 59      your workflow irreproducible on other machines. Use path relative to the
 60      working directory instead, or make the path configurable via a config
 61      file.
 62      Also see:
 63      https://snakemake.readthedocs.io/en/latest/snakefiles/configuration.html#configuration
 64
 65Lints for snakefile /tmp/tmp7di3duuu/workflow/rules/adt_hashing.smk:
 66    * Path composition with '+' in line 13:
 67      This becomes quickly unreadable. Usually, it is better to endure some
 68      redundancy against having a more readable workflow. Hence, just repeat
 69      common prefixes. If path composition is unavoidable, use pathlib or
 70      (python >= 3.6) string formatting with f"...".
 71      Also see:
 72
 73
 74Lints for rule cellranger_count_gex (line 9, /tmp/tmp7di3duuu/workflow/rules/gex_cellranger.smk):
 75    * Specify a conda environment or container for each rule.:
 76      This way, the used software for each specific step is documented, and the
 77      workflow can be executed on any machine without prerequisites.
 78      Also see:
 79      https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#integrated-package-management
 80      https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#running-jobs-in-containers
 81    * Shell command directly uses variable config from outside of the rule:
 82      It is recommended to pass all files as input and output, and non-file
 83      parameters via the params directive. Otherwise, provenance tracking is
 84      less accurate.
 85      Also see:
 86      https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
 87    * Param cr_out is a prefix of input or output file but hardcoded:
 88      If this is meant to represent a file path prefix, it will fail when
 89      running workflow in environments without a shared filesystem. Instead,
 90      provide a function that infers the appropriate prefix from the input or
 91      output file, e.g.: lambda w, input: os.path.splitext(input[0])[0]
 92      Also see:
 93      https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
 94      https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html#tutorial-input-functions
 95
 96Lints for rule create_library_file_adt (line 10, /tmp/tmp7di3duuu/workflow/rules/adt_cellranger.smk):
 97    * Specify a conda environment or container for each rule.:
 98      This way, the used software for each specific step is documented, and the
 99      workflow can be executed on any machine without prerequisites.
100      Also see:
101      https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#integrated-package-management
102      https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#running-jobs-in-containers
103
104Lints for rule cellranger_count_adt (line 66, /tmp/tmp7di3duuu/workflow/rules/adt_cellranger.smk):
105    * Specify a conda environment or container for each rule.:
106      This way, the used software for each specific step is documented, and the
107      workflow can be executed on any machine without prerequisites.
108      Also see:
109      https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#integrated-package-management
110      https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#running-jobs-in-containers
111    * Shell command directly uses variable config from outside of the rule:
112      It is recommended to pass all files as input and output, and non-file
113      parameters via the params directive. Otherwise, provenance tracking is
114      less accurate.
115      Also see:
116      https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
117    * Param cr_out is a prefix of input or output file but hardcoded:
118      If this is meant to represent a file path prefix, it will fail when
119      running workflow in environments without a shared filesystem. Instead,
120      provide a function that infers the appropriate prefix from the input or
121      output file, e.g.: lambda w, input: os.path.splitext(input[0])[0]
122      Also see:
123      https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
124      https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html#tutorial-input-functions
125
126Lints for rule create_tag_file (line 9, /tmp/tmp7di3duuu/workflow/rules/adt_hashing.smk):
127    * Migrate long run directives into scripts or notebooks:
128      Long run directives hamper workflow readability. Use the script or
129      notebook directive instead. Note that the script or notebook directive
130      does not involve boilerplate. Similar to run, you will have direct access
131      to params, input, output, and wildcards.Only use the run directive for a
132      handful of lines.
133      Also see:
134      https://snakemake.readthedocs.io/en/latest/snakefiles/rules.html#external-scripts
135      https://snakemake.readthedocs.io/en/latest/snakefiles/rules.html#jupyter-notebook-integration
136    * Param outdir is a prefix of input or output file but hardcoded:
137      If this is meant to represent a file path prefix, it will fail when
138      running workflow in environments without a shared filesystem. Instead,
139      provide a function that infers the appropriate prefix from the input or
140      output file, e.g.: lambda w, input: os.path.splitext(input[0])[0]
141      Also see:
142      https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
143      https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html#tutorial-input-functions
144
145Lints for rule gzip_files_hashingInput (line 154, /tmp/tmp7di3duuu/workflow/rules/adt_hashing.smk):
146    * Specify a conda environment or container for each rule.:
147      This way, the used software for each specific step is documented, and the
148      workflow can be executed on any machine without prerequisites.
149      Also see:
150      https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#integrated-package-management
151      https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#running-jobs-in-containers
152    * Param root_out is a prefix of input or output file but hardcoded:
153      If this is meant to represent a file path prefix, it will fail when
154      running workflow in environments without a shared filesystem. Instead,
155      provide a function that infers the appropriate prefix from the input or
156      output file, e.g.: lambda w, input: os.path.splitext(input[0])[0]
157      Also see:
158      https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
159      https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html#tutorial-input-functions
160
161Lints for rule Rscript_analyseHashing (line 216, /tmp/tmp7di3duuu/workflow/rules/adt_hashing.smk):
162    * Param adt_folder is a prefix of input or output file but hardcoded:
163      If this is meant to represent a file path prefix, it will fail when
164      running workflow in environments without a shared filesystem. Instead,
165      provide a function that infers the appropriate prefix from the input or
166      output file, e.g.: lambda w, input: os.path.splitext(input[0])[0]
167      Also see:
168      https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
169      https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html#tutorial-input-functions
170    * Param output_prefix is a prefix of input or output file but hardcoded:
171      If this is meant to represent a file path prefix, it will fail when
172      running workflow in environments without a shared filesystem. Instead,
173      provide a function that infers the appropriate prefix from the input or
174      output file, e.g.: lambda w, input: os.path.splitext(input[0])[0]
175      Also see:
176      https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
177      https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html#tutorial-input-functions
178
179Lints for rule scampi_identify_doublets (line 163, https://:ghs_3a0fPKZqqPJB4FSmcjSUUi10DO6r9l0JTDLQ@raw.githubusercontent.com/ETH-NEXUS/scAmpi_single_cell_RNA/v2.0.7/workflow/rules/scAmpi_basic_rules.smk):
180    * Param outdir is a prefix of input or output file but hardcoded:
181      If this is meant to represent a file path prefix, it will fail when
182      running workflow in environments without a shared filesystem. Instead,
183      provide a function that infers the appropriate prefix from the input or
184      output file, e.g.: lambda w, input: os.path.splitext(input[0])[0]
185      Also see:
186      https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
187      https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html#tutorial-input-functions
188
189Lints for rule scampi_filter_genes_and_cells (line 228, https://:ghs_3a0fPKZqqPJB4FSmcjSUUi10DO6r9l0JTDLQ@raw.githubusercontent.com/ETH-NEXUS/scAmpi_single_cell_RNA/v2.0.7/workflow/rules/scAmpi_basic_rules.smk):
190    * Param outDir is a prefix of input or output file but hardcoded:
191      If this is meant to represent a file path prefix, it will fail when
192      running workflow in environments without a shared filesystem. Instead,
193      provide a function that infers the appropriate prefix from the input or
194      output file, e.g.: lambda w, input: os.path.splitext(input[0])[0]
195      Also see:
196      https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
197      https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html#tutorial-input-functions
198
199Lints for rule scampi_sctransform_preprocessing (line 322, https://:ghs_3a0fPKZqqPJB4FSmcjSUUi10DO6r9l0JTDLQ@raw.githubusercontent.com/ETH-NEXUS/scAmpi_single_cell_RNA/v2.0.7/workflow/rules/scAmpi_basic_rules.smk):
200    * Param outDir is a prefix of input or output file but hardcoded:
201
202... (truncated)

Formatting results

 1[DEBUG] 
 2[DEBUG] 
 3[DEBUG] 
 4[DEBUG] 
 5[ERROR] In file "/tmp/tmp7di3duuu/workflow/rules/cohort_adt_analysis.smk":  NameError: name 'gatherCellrangerADTFolder' is not defined
 6[DEBUG] In file "/tmp/tmp7di3duuu/workflow/rules/cohort_adt_analysis.smk":  
 7[DEBUG] 
 8[DEBUG] In file "/tmp/tmp7di3duuu/workflow/rules/gex_cellranger.smk":  Formatted content is different from original
 9[DEBUG] 
10[DEBUG] In file "/tmp/tmp7di3duuu/workflow/rules/misc_snake.smk":  Formatted content is different from original
11[DEBUG] 
12[INFO] 1 file(s) raised parsing errors 🤕
13[INFO] 2 file(s) would be changed 😬
14[INFO] 5 file(s) would be left unchanged 🎉
15
16snakefmt version: 0.8.4