SarahSaadain/pastForward

A Snakemake-based pipeline for processing and quality-controlling ancient DNA data, including contamination assessment and comparative genomic analysis across time points, producing damage-rescaled, mapped reads ready for downstream analyses.

Overview

Latest release: v2.0.1, Last update: 2026-08-13

Share link: https://snakemake.github.io/snakemake-workflow-catalog?wf=SarahSaadain/pastForward

Quality control: linting: passed formatting: failed

Topics: adna dna-sequencing genomics pipeline snakemake ancient-dna ancient-dna-analysis ancientdna genome genome-mapping multiqc raw-reads bioinformatics bioinformatics-pipeline short-read-mapping short-reads bam fasta fastq

Deployment

Step 1: Install Snakemake and Snakedeploy

Snakemake and Snakedeploy are best installed via the Conda package manager. It is recommended to install conda via Miniforge. Run

conda create -c conda-forge -c bioconda -c nodefaults --name snakemake snakemake snakedeploy

to install both Snakemake and Snakedeploy in an isolated environment. For all following commands ensure that this environment is activated via

conda activate snakemake

For other installation methods, refer to the Snakemake and Snakedeploy documentation.

Step 2: Deploy workflow

With Snakemake and Snakedeploy installed, the workflow can be deployed as follows. First, create an appropriate project working directory on your system and enter it:

mkdir -p path/to/project-workdir
cd path/to/project-workdir

In all following steps, we will assume that you are inside of that directory. Then run

snakedeploy deploy-workflow https://github.com/SarahSaadain/pastForward . --tag v2.0.1

Snakedeploy will create two folders, workflow and config. The former contains the deployment of the chosen workflow as a Snakemake module, the latter contains configuration files which will be modified in the next step in order to configure the workflow to your needs.

Step 3: Configure workflow

To configure the workflow, adapt config/config.yml to your needs following the instructions below.

Step 4: Run workflow

The deployment method is controlled using the --software-deployment-method (short --sdm) argument.

To run the workflow with automatic deployment of all required software via conda/mamba, use

snakemake --cores all --sdm conda

Snakemake will automatically detect the main Snakefile in the workflow subfolder and execute the workflow module that has been defined by the deployment in step 2.

For further options such as cluster and cloud execution, see the docs.

Step 5: Generate report

After finalizing your data analysis, you can automatically generate an interactive visual HTML report for inspection of results together with parameters and code inside of the browser using

snakemake --report report.zip

Configuration

The following section is imported from the workflow’s config/README.md.

Setup Guide

This guide walks through everything you need to set up and configure pastForward.

What You’ll Need

pastForward runs on two free tools:

  • Conda installs and manages all the other software the pipeline needs.

  • Snakemake runs the pipeline itself and can be installed using conda. Version 9.9.0 or newer is required.

Step 1: Install Conda

If you don’t already have conda, download and install it Miniforge. Follow the instructions for your operating system.

Step 2: Install Snakemake

Open a terminal and type each of these lines, pressing Enter after each one:

conda create -c conda-forge -c bioconda -c nodefaults -n snakemake snakemake
conda activate snakemake
snakemake --help

What each line does:

  1. Creates a separate, self-contained conda environment called snakemake and installs Snakemake into it. You only need to do this once.

  2. Switches your terminal into that environment. Run this line every time you open a new terminal window, before using pastForward.

  3. Checks that the install worked. You should see the helppage print out.

For more installation options, see the Snakemake documentation.

Step 3: Get pastForward

Download or clone this repository into a folder on your computer using:

git clone https://github.com/SarahSaadain/pastForward.git

That folder becomes your project folder. pastForward, your data, and your results will all live inside it. See Project Structure below for what this folder should contain.

Step 4: Add Your Species and Data

Project Structure

A pastForward project is a single folder containing the workflow/ and config/ folders (the pipeline code you just downloaded) plus one folder per species you want to process:

my_project/                  <- project folder — run `snakemake` from here
├── workflow/                <- pastForward pipeline code (do not edit)
├── config/                  <- config.yaml, config_designer.html
├── Dmel/                    <- one folder per species; name must match the `species:` key in config.yaml
│   ├── input/
│   ├── processed/
│   └── results/
└── Dsim/
    ├── input/
    ├── processed/
    └── results/

The pipeline code and your data live side by side in this one folder. There’s no separate install location.

One project can handle one species or many. Which you choose depends on how you want to work:

  • Combine several species in one project folder if you just want a quick look across many species at once, sharing a single command and config. For example, checking data quality across a batch from a low-depth trial run.

  • Give each species its own project folder if you want to start, re-run, and configure each one independently without affecting the others. This is the better choice for a full production run.

Add a Species

To add a new species:

  1. Create a folder for it in the project root (next to workflow/ and config/). The folder name must exactly (case sensitive) match the species key you’ll use under species: in config.yaml (see Configuration below).

  2. Put your raw read files and reference genome inside that folder (see below).

Providing Your Data

The simplest option: drop your raw read files and reference genome inside the <species> folder. The first time you run pastForward, it automatically finds them and moves them to its destination. This shortcut only works for reads and the reference genome. REVEAL input files (feature library, and optionally SCG) must go in their specific folders, not just anywhere in <species>.

IF you want to place the files directly in their final place, put your files here:

  • raw reads in <species>/input/read_module/

  • the reference genome(s) in <species>/input/reference_module/

  • (optional) a feature library — a FASTA of TE or other genomic feature sequences to compare across samples — in <species>/input/reveal_module/feature_library/, needed only if you’re using the REVEAL comparison stage

  • (optional) a pre-built SCG (single-copy gene) FASTA in <species>/input/reveal_module/scg/. If you skip this, pastForward determines SCGs automatically via BUSCO, as long as pipeline.reveal_module.scg_selector.execute is true (the default) and species.<key>.lineage is set to a BUSCO lineage name (e.g. drosophilidae_odb12, see busco.ezlab.org). No lineage configured and no FASTA provided means SCG determination is skipped.

If your files are large, shared with other tools, or already live somewhere else on disk, you don’t need to copy them. Place a symlink (a shortcut/pointer file) in the expected location instead, and pastForward will use it directly. The symlink’s name must follow pastForward’s naming convention (below), but the real file it points to can keep its own name and live anywhere.

Storing Species Data Elsewhere

This is an optional, advanced feature. Skip this section if your data lives inside the project folder as shown above. That’s the default, and most people don’t need to change it.

If you’d rather keep some or all of a species’ data elsewhere (a different disk, a shared network drive, or a folder outside the project entirely), set one or more of the following optional settings under species.<key> in config.yaml. If you don’t set any of these, nothing changes from the default behavior described above.

Setting

Overrides

species_dir

The whole species root. Must contain the same input/{read_module,reference_module,reveal_module/{scg,feature_library,competition}}, processed/, results/ layout as a normal species folder. Used as the default target for every setting below.

reads_dir

<species>/input/read_module/

reference_dir

<species>/input/reference_module/

scg_dir

<species>/input/reveal_module/scg/

feature_library_dir

<species>/input/reveal_module/feature_library/

competition_dir

<species>/input/reveal_module/competition/

processed_dir

<species>/processed/

results_dir

<species>/results/

If you set both species_dir and one of the more specific settings, the specific setting wins.

species:
  Dmel:
    name: "Drosophila melanogaster"
    # Everything for Dmel lives on a different disk...
    species_dir: "/mnt/big_disk/pastforward_data/Dmel"
    # ...except processed/, which should go to fast local scratch instead.
    processed_dir: "/scratch/pastforward_processed/Dmel"

At startup, pastForward creates a shortcut (symlink) at the usual in-project location (e.g. Dmel/input/read_module) pointing at your configured target, so every part of the pipeline keeps working normally. A few things to know:

  • This happens automatically, once per run, before pastForward looks for any input files.

  • If something already exists at the usual location (a real folder, or a shortcut to somewhere else), pastForward will stop and show an error instead of overwriting it. Fix the config, or move/remove the conflicting folder, then try again.

  • processed_dir and results_dir are additionally protected against two different projects accidentally writing to the same place at the same time. See FAQ.md if you run into a lock error.

What Gets Created Automatically

You don’t need to create any folders beyond your species folder. Everything else is created as the pipeline runs:

  • <species>/processed/ holds files created while the pipeline is working. Most are temporary files and are deleted automatically once they’re no longer needed. Some are kept so the pipeline can pick up from a failed step without starting over.

  • <species>/results/ holds your final results and reports. This is what you’ll actually look at.

Everything related to a reference is grouped under a <reference> folder inside processed/ or results/. For most purposes, only results/ matters. If you need more detail, the processed/ folder usually has it. A couple of large intermediate file types (.sam and unsorted .bam files) are always deleted to save disk space. If you ever need to redo a step, just delete its output files and re-run pastForward.

Naming Your Read Files

pastForward needs your raw read filenames to follow one consistent pattern, so it can tell which files belong to which sample and which read pair they are. A few correct examples:

Dmel01_DabneyProtocol_R1_006.fastq.gz
Dmel01_DabneyProtocol_1.fastq.gz
Dmel01_DabneyProtocol_R1.fq.gz

The pattern, piece by piece:

<Individual>_[<FreeText>_]<ReadNumber>[_<FreeText>].fastq.gz

  • <Individual> is a unique ID for the sample, e.g. Dmel01. It’s everything before the first underscore, and pastForward uses it to group files that belong together.

  • <FreeText> (optional, can appear before or after the read number) is any extra label you want, e.g. a protocol name. Useful when the same individual was extracted twice with different methods.

  • <ReadNumber> marks which read of the pair this file is: R1/R2, or a plain 1/2. A plain 1 or 2 must stand on its own between underscores or right before the file extension. It won’t be picked up inside a longer number like _10_ or _21. In case you provide single end data, use 1 or R1 as well.

  • The file must end in .fastq.gz or .fq.gz (compressed FASTQ). Uncompressed .fastq/.fq files are not supported.

Configuration (config.yaml)

config.yaml tells pastForward which species to process and which pipeline options to use.

*If you don’t want to change the config in the terminal, open config_designer.html in your web browser. This interactive tool walks you through every option with a graphical interface and generates a ready-to-use config.yaml for you. No code required.

If you’d rather write it yourself, every pipeline stage is turned on by default, so a minimal config only needs a project name and species list:

project_name: "pastForward_Project"

species:
  Dmel:
    name: "Drosophila melanogaster"

Once your data is in place and your config is ready, head back to the main README to start the pipeline.

Linting and formatting

Linting results
All tests passed!
Formatting results
 1[DEBUG] 
 2[DEBUG] 
 3[DEBUG] 
 4[DEBUG] In file "/tmp/tmp8tvffu03/SarahSaadain-pastForward-aad8168/workflow/rules/initialize.smk":  Formatted content is different from original
 5[DEBUG] 
 6[DEBUG] 
 7[DEBUG] 
 8[DEBUG] 
 9[INFO] 1 file(s) would be changed 😬
10[INFO] 6 file(s) would be left unchanged 🎉
11
12snakefmt version: 0.11.5