nodrogluap/tamor

Illumina Dragen cancer genome and transcriptome analysis automation using Snakemake

Overview

Topics: cancer-genomics dragen mutation-analysis pcgr ruo sciworkflows bioinformatics-pipeline

Latest release: None, Last update: 2025-04-09

Linting: linting: failed, Formatting: formatting: failed

Deployment

Step 1: Install Snakemake and Snakedeploy

Snakemake and Snakedeploy are best installed via the Mamba package manager (a drop-in replacement for conda). If you have neither Conda nor Mamba, it is recommended to install Miniforge. More details regarding Mamba can be found here.

When using Mamba, run

mamba create -c conda-forge -c bioconda --name snakemake snakemake snakedeploy

to install both Snakemake and Snakedeploy in an isolated environment. For all following commands ensure that this environment is activated via

conda activate snakemake

Step 2: Deploy workflow

With Snakemake and Snakedeploy installed, the workflow can be deployed as follows. First, create an appropriate project working directory on your system and enter it:

mkdir -p path/to/project-workdir
cd path/to/project-workdir

In all following steps, we will assume that you are inside of that directory. Then run

snakedeploy deploy-workflow https://github.com/nodrogluap/tamor . --tag None

Snakedeploy will create two folders, workflow and config. The former contains the deployment of the chosen workflow as a Snakemake module, the latter contains configuration files which will be modified in the next step in order to configure the workflow to your needs.

Step 3: Configure workflow

To configure the workflow, adapt config/config.yml to your needs following the instructions below.

Step 4: Run workflow

The deployment method is controlled using the --software-deployment-method (short --sdm) argument.

To run the workflow with automatic deployment of all required software via conda/mamba, use

snakemake --cores all --sdm conda

Snakemake will automatically detect the main Snakefile in the workflow subfolder and execute the workflow module that has been defined by the deployment in step 2.

For further options such as cluster and cloud execution, see the docs.

Step 5: Generate report

After finalizing your data analysis, you can automatically generate an interactive visual HTML report for inspection of results together with parameters and code inside of the browser using

snakemake --report report.zip

Configuration

The following section is imported from the workflow’s config/README.md.

NOTA BENE!!! When running snakemake for the first time with this repository, it may take many hours, as it will download both all the software environment needed to run PCGR mutation impact reports, and all the large public resource files needed for the same (by automatically running workflow/scripts/download_resources.py). If you intererupt the downloading and unpacking of these files, you will need to rerun the download script manually.

Configuring Tamor to Analyze Your Cancer Cases

Cases are organized into logical units: Projects (a.k.a. cohorts), that have Subjects (a.k.a. patients) that have Samples (e.g. biopsy or blood).

Cases from the same cohort will be outputted into the same output folder, for organizational purposes.

A Subject must have at least one normal/germline sample. We may expand to support tumor-only analysis in the future.

A Subject can have one or more tumor samples (e.g. primary and refractory). Each tumor must have a DNA sample, and optionally an RNA sample.

The default config files are preconfigured for didactic purposes with a public leukaemia genome+transcriptome case from the NCBI Short Read Archive. This case is part of the cohort PR-TEST-CLL, with the patient labelled as PR-TEST-CLL-SAMN08512283, and there are three sets of input FASTQ files downloaded/generated by running workflow/scripts/download_testdata.py. The three samples are PR-TEST-CLL-SAMN08512283-SRR6702602-T (tumor DNA) , PR-TEST-CLL-SAMN08512283-SRR6702602-N (pseudonormal DNA generated by the script since no actual normal is available), and PR-TEST-CLL-SAMN08512283-SRR6702601-T (tumor RNA). Such long systematic names are not necessary, but in practice we have found them very useful as you start accumulating larger cohorts.

Sequencing instrument run IDs and sample IDs are typically rather opaque and automatically assigned by the sequencing lab. These are not part of the config files, nor reported out by Tamor, but rather linked to designated Subjects and Samples via the Illumina Samplesheets.

config/config.yaml

config/config.yaml is the file that you can customize for your site-specific settings. By default the config is set up to read input files from the resources folder, and write result files under the results folder. By default the genome index and annotation files, as well as the PCGR data bundle, are expected in resources. This is where workflow/scripts/download_resources.py puts those files.

Tamor’s default config has the input lists of paired tumor-normal samples (with minimal metadata, described below) in files called config/dna_samples.tsv and config/rna_samples.tsv. These TSVs are the main config files that you will need to edit to run your own samples through the workflow.

config/dna_samples.tsv

Has 8 columns to be specified:

subjectID<tab>
tumorSampleID<tab>
TrueOrFalseTumorHasPCRDuplicates<tab>
germlineSampleID<tab>
TrueOrFalseGermlineHasPCRDuplicates<tab>
TrueOrFalse_germline_contains_some_tumor<tab>
OncoTreeCode<tab>
ProjectID

The subjectID, tumorSampleID and germlineSampleID must:

  • CONTAIN NO UNDERSCORES

  • The subjectID must be between 6 and 35 characters (due to a PCGR naming limitation)

  • tumorSampleID and germlineSampleID must be the exact Sample_Name values you used in your Illumina sequencing sample spreadsheets (see samplesheet section below for details).

The third and fifth column tell Dragen whether to consider (in tumor and germline respectively) as PCR duplicates read pairs that map to the same start and end in the reference genome. If you used a PCR-free library prep, set this to False, otherwise set it to True.

The eighth column is a unique project ID to which the subject belongs. For example if you have two cohorts of lung and breast cancer, assigning individuals to two projects would be logical. All project output files go into their own output folders, even if they were sequenced together on the same Illumina sequencing runs.

The sixth column of the paired input sample TSV file is usually False, unless your germline sample is from a leukaemia or perhaps a poor quality histology section from a tumor, in which case use True. This instructs Dragen to consider low frequency variants in the germline sample to still show up as somatic variants in the tumor analysis output (see default of 0.05 under tumor_in_normal_tolerance_proportion in config.yaml)

For the seventh column, the type of cancer the tumor represent must be coded. This is preferably an OncoTree code. Those codes can be found here: https://oncotree.mskcc.org/ This information will be used to customize some parts of the variant, gene expression, and immune profiling reports. If no cancer type information is available at all, you can use the top-level code in OncoTree: “TISSUE”. While OncoTree codes are preferred, Tamor will also attempt to uniquely map codes from the ICD-O, NCIt, UMLS and HemeOnc systems.

config/rna_samples.tsv

Has 5 columns to be specified:

subjectID<tab>
tumorRNASampleID<tab>
matchedTumorDNASampleID<tab>
ProjectID<tab>
CohortNameForExpressionAnalysis

If you have both normal and tumor RNA samples available, it is critical to list the tumor RNA sample first.
The first RNA sample listed in the file is the one that will be included on the PCGR report for matchedTumorDNASampleID, and typically you want to report out regarding the tumor RNA.

The last column CohortNameForExpressionAnalysis is used for Djerba cohort reporting, e.g. to identify Z-score and percentile rank outliers genes in this sample compared to others being processed at the same time and nominally of the same cancer/tissue type as defined by the user.

Samplesheets

These sample sheets are the only other metadata to which Tamor has access. Place all the Illumina experiment sample sheets for your project into resources/spreadsheets by default (see the samplesheets_dir setting in config/config.yaml). They must be called runID.csv, where runID is typically the Illumina folder name in the format YYMMDD_machineID_SideFlowCellID.

Note that a sample can actually be sequenced across multiple runs, Tamor will aggregate the sequence data across the runs to generate a single report (e.g. a primary run and some top-up sequencing due to unexpected low read count on the first run). The same sample name can have the same sample ID or different sample IDs across runs, they will be aggregated regardless. This allows for a single tumor sample to be prepared using two different sequencing library preps for example.

If you are providing the FASTQs directly as input to Tamor, they must also be in the resources/analysis/primary/sequencerName/runID directory, with a corresponding Illumina Experiment Manager samplesheet resources/spreadsheets/runID.csv. Why? This is required because Tamor reads the sample sheet to find the correspondence between Sample_Name and Sample ID for each sequencing library, also analysis for DNA samples differs from that for RNA samples, so the sample sheet must also contain a Sample_Project column. Sample projects with names that contain “RNA” in them will be processed as such, all others are assumed to be DNA. The Sample_Project is not used for any other purpose than distinguishing RNA and DNA, and does not need to be the same as the ProjectIDs listed in the config folder files.

The samplesheet is also used to determine if Unique Molecular Indices were used to generate the sequencing libraries, which requires different handling in Dragen during genotyping downstream.

If you provide FASTQ files directly, they must be timestamped later than the corresponding Illumina Experiment Manager spreadsheet, otherwise Snakemake will assume you’ve consequentially changed the spreadsheet and try to automatically regenerated all FASTQs for that run – from potentially non-existent BCLs.

If you are starting with BCLs, the full Illumina experiment output folders (which contain the requisite Data/Intensities/Basecalls subfolder) are expected by in resources/bcls/runID (see bcl_dir setting inconfig.yaml). Tamor will perform BCL to FASTQ conversion, with the FASTQ output into results/analysis/primary/sequencer/runID (see analysis_dir setting in config.yaml, and the default sequencer is HiSeq per the test data mentioned earlier).

Linting and formatting

Linting results

Lints for snakefile /tmp/tmphc7s8pav/workflow/Snakefile:
    * Absolute path "/{project_name}/{subject}/rna/{sample_combo}.rna.quant.genes.hugo.tpm.txt" in line 37:
      Do not define absolute paths inside of the workflow, since this renders
      your workflow irreproducible on other machines. Use path relative to the
      working directory instead, or make the path configurable via a config
      file.
      Also see:
      https://snakemake.readthedocs.io/en/latest/snakefiles/configuration.html#configuration
    * Absolute path "/{project_name}/{subject}/rna/{sample_combo}.rna.quant.genes.fpkm.txt" in line 38:
      Do not define absolute paths inside of the workflow, since this renders
      your workflow irreproducible on other machines. Use path relative to the
      working directory instead, or make the path configurable via a config
      file.
      Also see:
      https://snakemake.readthedocs.io/en/latest/snakefiles/configuration.html#configuration
    * Absolute path "/{project_name}/{subject}/rna/{sample_combo}.rna.fusion_candidates.features.csv" in line 39:
      Do not define absolute paths inside of the workflow, since this renders
      your workflow irreproducible on other machines. Use path relative to the
      working directory instead, or make the path configurable via a config
      file.

... (truncated)

Formatting results

[DEBUG] 
[DEBUG] In file "/tmp/tmphc7s8pav/workflow/rules/fastq_list.smk":  Formatted content is different from original
[DEBUG] 
[ERROR] In file "/tmp/tmphc7s8pav/workflow/rules/data_export.smk":  InvalidPython: Black error:

Cannot parse for target version Python 3.12: 14:0: shell(“gzip -cd {input.somatic_snv_vcf} | perl -F\t -ane ‘$F[0] =~ s/^chr//; print join(“\t”, $F[0], $F[1], $F[1]+length($F[3])-1, $F[3], $F[4], $2, $1, “{wildcards.tumor}”), “\n” if not /^#/ and $F[$#F] =~ /^[^:]+:[^:]+:(\d+),(\d+)/’ > {output.maf}”)

(Note reported line number may be incorrect, as snakefmt could not determine the true line number)


[DEBUG] In file "/tmp/tmphc7s8pav/workflow/rules/data_export.smk":  
[ERROR] In file "/tmp/tmphc7s8pav/workflow/rules/pcgr.smk":  InvalidPython: Black error:

Cannot parse for target version Python 3.12: 1:146: “workflow/scripts/generate_pcgr.py {input.tumor_site_code_file} {input.cpsr} {input.cpsr_yaml} {input.somatic_snv_vcf} {input.somatic_cnv_vcf} “ +

(Note reported line number may be incorrect, as snakefmt could not determine the true line number)


[DEBUG] In file "/tmp/tmphc7s8pav/workflow/rules/pcgr.smk":  
[DEBUG] In file "/tmp/tmphc7s8pav/workflow/rules/metrics.smk":  Formatted content is different from original

... (truncated)