milesroberts-123/custom-rag-cli
playing around with ollama and llama index to pass files to an LLM
Overview
Latest release: None, Last update: 2026-11-02
Share link: https://snakemake.github.io/snakemake-workflow-catalog?wf=milesroberts-123/custom-rag-cli
Quality control: linting: failed formatting: passed
Deployment
Step 1: Install Snakemake and Snakedeploy
Snakemake and Snakedeploy are best installed via the Conda package manager. It is recommended to install conda via Miniforge. Run
conda create -c conda-forge -c bioconda -c nodefaults --name snakemake snakemake snakedeploy
to install both Snakemake and Snakedeploy in an isolated environment. For all following commands ensure that this environment is activated via
conda activate snakemake
For other installation methods, refer to the Snakemake and Snakedeploy documentation.
Step 2: Deploy workflow
With Snakemake and Snakedeploy installed, the workflow can be deployed as follows. First, create an appropriate project working directory on your system and enter it:
mkdir -p path/to/project-workdir
cd path/to/project-workdir
In all following steps, we will assume that you are inside of that directory. Then run
snakedeploy deploy-workflow https://github.com/milesroberts-123/custom-rag-cli . --tag None
Snakedeploy will create two folders, workflow and config. The former contains the deployment of the chosen workflow as a Snakemake module, the latter contains configuration files which will be modified in the next step in order to configure the workflow to your needs.
Step 3: Configure workflow
To configure the workflow, adapt config/config.yml to your needs following the instructions below.
Step 4: Run workflow
The deployment method is controlled using the --software-deployment-method (short --sdm) argument.
To run the workflow with automatic deployment of all required software via conda/mamba, use
snakemake --cores all --sdm conda
Snakemake will automatically detect the main Snakefile in the workflow subfolder and execute the workflow module that has been defined by the deployment in step 2.
For further options such as cluster and cloud execution, see the docs.
Step 5: Generate report
After finalizing your data analysis, you can automatically generate an interactive visual HTML report for inspection of results together with parameters and code inside of the browser using
snakemake --report report.zip
Configuration
The following section is imported from the workflow’s config/README.md.
Configuration
The workflow reads two files from this directory:
config/config.yaml— workflow parameters.config/samples.tsv— one SRA run per row.
All relative paths are resolved from the repository root. Override the sample
sheet without editing config.yaml:
snakemake -n --config samples=config/samples.test.tsv
Sample sheet
config/samples.tsv must be tab-separated with these columns:
Column |
Description |
|---|---|
|
SRA run accession ( |
|
NCBI BioSample accession ( |
|
NCBI BioProject accession ( |
|
Scientific name, for example |
Example:
run biosample bioproject species
SRR41450469 SAMN64520521 PRJNA1552792 Arabidopsis thaliana
SRR41450470 SAMN64520520 PRJNA1552792 Arabidopsis thaliana
The workflow fetches BioSample and SRA metadata by accession. If a BioProject has no linked PubMed ID, the paper stage is skipped for that project; the sample is still labeled from metadata only.
samples
Path to the sample sheet. Default: config/samples.tsv.
ncbi
Key |
Default |
Description |
|---|---|---|
|
required |
Contact email sent with NCBI E-utilities requests. |
|
|
Environment variable containing an NCBI API key. Optional, but raises the rate limit. |
|
|
BioSample accessions per efetch request. |
|
|
SRA run accessions per efetch request. |
|
|
Retry attempts for NCBI requests. |
peco
Key |
Default |
Description |
|---|---|---|
|
PECO OWL URL |
Ontology download location. |
|
|
Candidate terms passed to the exposure model. |
|
|
Sentence-transformers model for retrieval. |
|
|
Root of the plant exposure branch. |
|
|
Root of the plant study condition branch. |
|
|
Root of growth-medium exposure terms used for |
paper
Key |
Default |
Description |
|---|---|---|
|
|
Maximum excerpt length sent to the paper-summary model. |
tissue
Key |
Description |
|---|---|
|
Allowed tissue categories. |
|
Keyword lists used by the rules fallback, keyed by category. |
|
BioSample attribute names searched by the rules fallback. |
|
Description of the |
germplasm
Key |
Description |
|---|---|
|
Allowed germplasm categories. |
|
BioSample attribute names used as evidence. |
|
Accession names that count as natural accessions. |
|
Regex lists keyed by germplasm category. |
|
Category descriptions sent to the decision model. |
The default categories are natural_accession, mutant, transgenic,
hybrid, grafted, cell_culture, polyploid, aneuploid, and unknown.
rna_type
Key |
Description |
|---|---|
|
RNA types that continue to labeling. Defaults to |
|
BioSample and SRA attribute names searched for RNA-type evidence. |
|
Minimum model confidence for a fallback RNA-type call; lower values become |
The classifier uses SRA library_strategy, library_selection,
library_source, the library construction protocol, and selected BioSample or
SRA attributes. It does not use the paper. Other RNA types are written to
results/labels.excluded.tsv.
review
Key |
Default |
Description |
|---|---|---|
|
|
Flag tissue calls below this confidence. |
|
|
Flag germplasm calls below this confidence. |
|
|
Flag exposure calls with a lower top retrieval score. |
|
|
Flag study-condition calls below this confidence. |
|
|
Flag included RNA-type calls below this confidence. |
|
|
Flag projects without a linked paper. |
|
|
Flag abstract-only or low-confidence paper summaries. |
models
Key |
Default |
Description |
|---|---|---|
|
|
|
|
|
Chat fallback model for RNA type. |
|
|
Tev1 decision model for RNA type. |
|
|
|
|
|
Chat fallback model for tissue. |
|
|
Tev1 decision model for tissue. |
|
|
Backend for germplasm. |
|
|
Chat fallback model for germplasm. |
|
|
Tev1 decision model for germplasm. |
|
|
Backend for study condition. |
|
|
Chat fallback model for study condition. |
|
|
Tev1 decision model for study condition. |
|
|
Chat model for exposure labeling. |
|
|
Chat model for paper summarization. |
Each *_params key is passed to the corresponding model request. The defaults
set temperature: 0 for deterministic calls.
ollama
Key |
Default |
Description |
|---|---|---|
|
|
Ollama server used by the rule scripts. |
|
|
Placeholder key for the OpenAI-compatible client. |
Environment variables override these values:
OLLAMA_SERVERoverridesollama.server_url.OLLAMA_API_KEYoverridesollama.api_key.NCBI_API_KEYis read according toncbi.api_key_env.
Running
# Parse the workflow and check the DAG.
snakemake -n
# Run with Conda-deployed rule environments.
snakemake --sdm conda --cores 8
On Savio, workflow/savio/run.sh starts a shared Ollama server on an H200 and
runs the CPU rules with the Slurm profile. See the repository README for the
one-time setup.
Workflow parameters
The following table is automatically parsed from the workflow’s config.schema.y(a)ml file.
Parameter |
Type |
Description |
Required |
Default |
|---|---|---|---|---|
samples |
string |
path to the SRA run sample sheet |
yes |
|
ncbi |
yes |
|||
string |
contact email sent with NCBI E-utilities requests |
yes |
||
. api_key_env |
string |
NCBI_API_KEY |
||
. batch_size |
integer |
200 |
||
. sra_batch_size |
integer |
50 |
||
. max_retries |
integer |
5 |
||
peco |
yes |
|||
. url |
string |
yes |
||
. top_k |
integer |
30 |
||
. embed_model |
string |
BAAI/bge-base-en-v1.5 |
||
. exposure_branch |
string |
PECO:0001001 |
||
. study_branch |
string |
PECO:0007231 |
||
. growth_system_root |
string |
PECO:0007147 |
||
paper |
yes |
|||
. max_chars |
integer |
yes |
12000 |
|
tissue |
yes |
|||
. categories |
array |
yes |
||
. keywords |
{} |
|||
. attribute_keys |
array |
[‘tissue’, ‘organism part’, ‘organ’, ‘tissue type’, ‘sample tissue’, ‘cell type’] |
||
. other_description |
string |
None of the listed categories fits. |
||
germplasm |
yes |
|||
. categories |
array |
yes |
||
. attribute_keys |
array |
[‘genotype’, ‘genotype/variation’, ‘ecotype’, ‘cultivar’, ‘strain’, ‘sample name’, ‘description’, ‘tissue’] |
||
. natural_accessions |
array |
[] |
||
. patterns |
{} |
|||
. descriptions |
{} |
|||
rna_type |
yes |
|||
. accepted_rna_types |
array |
yes |
[‘mrna’, ‘ribo_depleted_total_rna’] |
|
. attribute_keys |
array |
[‘rna type’, ‘rna_type’, ‘library type’, ‘library_type’, ‘library preparation’, ‘library_preparation’, ‘rna extraction’, ‘rna_extraction’, ‘extraction method’, ‘assay type’, ‘assay_type’, ‘sequencing method’, ‘nucleic acid extraction’] |
||
. min_confidence |
number |
0.6 |
||
review |
yes |
|||
. tissue_confidence |
number |
yes |
0.6 |
|
. germplasm_confidence |
number |
yes |
0.6 |
|
. exposure_retrieval_score |
number |
yes |
0.55 |
|
. study_confidence |
number |
yes |
0.6 |
|
. rna_type_confidence |
number |
yes |
0.6 |
|
. flag_missing_paper |
boolean |
yes |
true |
|
. flag_low_confidence_paper |
boolean |
yes |
true |
|
models |
yes |
|||
. rna_type_backend |
string |
yes |
decision |
|
. rna_type |
string |
yes |
||
. rna_type_decision |
string |
yes |
||
. rna_type_params |
{} |
|||
. tissue_backend |
string |
yes |
decision |
|
. tissue |
string |
yes |
||
. tissue_decision |
string |
yes |
||
. germplasm_backend |
string |
yes |
decision |
|
. germplasm |
string |
yes |
||
. germplasm_decision |
string |
yes |
||
. exposure |
string |
yes |
||
. study_backend |
string |
yes |
decision |
|
. study |
string |
yes |
||
. study_decision |
string |
yes |
||
. summary |
string |
yes |
||
. tissue_params |
{} |
|||
. germplasm_params |
{} |
|||
. exposure_params |
{} |
|||
. study_params |
{} |
|||
. summary_params |
{} |
|||
ollama |
yes |
|||
. server_url |
string |
yes |
||
. api_key |
string |
yes |
Linting and formatting
Linting results
1/home/runner/work/snakemake-workflow-catalog/snakemake-workflow-catalog/.pixi/envs/default/lib/python3.13/site-packages/google/auth/transport/grpc.py:43: FutureWarning: grpcio < 1.83.0 does not support Post-Quantum Cryptography (PQC). Support for non-PQC environments is deprecated. In April 2027, google-auth will raise its minimum requirements to enforce grpcio >= 1.83.0. For more details on Google Cloud's post-quantum security migration, visit: https://cloud.google.com/security/resources/post-quantum-cryptography
2 warnings.warn(
3Using workflow specific profile workflow/profiles/default for setting default command line arguments.
4usage: snakemake [-h] [--dry-run] [--profile PROFILE]
5 [--workflow-profile WORKFLOW_PROFILE] [--cache [RULE ...]]
6 [--snakefile FILE] [--cores N] [--jobs N] [--local-cores N]
7 [--resources NAME=INT [NAME=INT ...]]
8 [--set-threads RULE=THREADS [RULE=THREADS ...]]
9 [--max-threads MAX_THREADS]
10 [--set-resources RULE:RESOURCE=VALUE [RULE:RESOURCE=VALUE ...]]
11 [--set-scatter NAME=SCATTERITEMS [NAME=SCATTERITEMS ...]]
12 [--set-resource-scopes RESOURCE=[global|local] [RESOURCE=[global|local] ...]]
13 [--default-resources [NAME=INT ...]]
14 [--preemptible-rules [PREEMPTIBLE_RULES ...]]
15 [--preemptible-retries PREEMPTIBLE_RETRIES]
16 [--configfile FILE [FILE ...]] [--config [KEY=VALUE ...]]
17 [--replace-workflow-config] [--envvars VARNAME [VARNAME ...]]
18 [--directory DIR] [--touch] [--keep-going]
19 [--rerun-triggers {code,input,mtime,params,software-env} [{code,input,mtime,params,software-env} ...]]
20 [--force] [--executor {local,dryrun,touch}] [--forceall]
21 [--forcerun [TARGET ...]]
22 [--consider-ancient RULE=INPUTITEMS [RULE=INPUTITEMS ...]]
23 [--prioritize TARGET [TARGET ...]]
24 [--batch RULE=BATCH/BATCHES] [--until TARGET [TARGET ...]]
25 [--omit-from TARGET [TARGET ...]] [--rerun-incomplete]
26 [--shadow-prefix DIR]
27 [--strict-dag-evaluation {cyclic-graph,functions,periodic-wildcards} [{cyclic-graph,functions,periodic-wildcards} ...]]
28 [--scheduler [{greedy,ilp}]]
29 [--conda-base-path CONDA_BASE_PATH] [--no-subworkflows]
30 [--precommand PRECOMMAND] [--groups GROUPS [GROUPS ...]]
31 [--group-components GROUP_COMPONENTS [GROUP_COMPONENTS ...]]
32 [--report [FILE]] [--report-after-run]
33 [--report-stylesheet CSSFILE] [--report-metadata FILE]
34 [--reporter PLUGIN] [--draft-notebook TARGET]
35 [--edit-notebook TARGET] [--notebook-listen IP:PORT]
36 [--lint [{text,json}]] [--generate-unit-tests [TESTPATH]]
37 [--containerize [{dockerfile,apptainer}]] [--export-cwl FILE]
38 [--list-rules] [--list-target-rules]
39 [--dag [{dot,mermaid-js}]] [--rulegraph [{dot,mermaid-js}]]
40 [--filegraph] [--d3dag] [--summary] [--detailed-summary]
41 [--archive FILE] [--cleanup-metadata FILE [FILE ...]]
42 [--cleanup-shadow] [--skip-script-cleanup] [--unlock]
43 [--list-changes {params,input,code}] [--list-input-changes]
44 [--list-params-changes] [--list-untracked]
45 [--delete-all-output | --delete-temp-output]
46 [--keep-incomplete] [--drop-metadata] [--version]
47 [--printshellcmds] [--debug-dag] [--nocolor]
48 [--quiet [{all,host,progress,reason,rules} ...]]
49 [--print-compilation] [--verbose] [--force-use-threads]
50 [--allow-ambiguity] [--nolock] [--ignore-incomplete]
51 [--max-inventory-time SECONDS] [--trust-io-cache]
52 [--max-checksum-file-size SIZE] [--latency-wait SECONDS]
53 [--wait-for-free-local-storage WAIT_FOR_FREE_LOCAL_STORAGE]
54 [--wait-for-files [FILE ...]] [--wait-for-files-file FILE]
55 [--runtime-source-cache-path PATH]
56 [--queue-input-wait-time SECONDS]
57 [--omit-flags OMIT_FLAGS [OMIT_FLAGS ...]] [--notemp]
58 [--all-temp] [--unneeded-temp-files FILE [FILE ...]]
59 [--keep-storage-local-copies] [--not-retrieve-storage]
60 [--target-files-omit-workdir-adjustment]
61 [--allowed-rules ALLOWED_RULES [ALLOWED_RULES ...]]
62 [--max-jobs-per-timespan MAX_JOBS_PER_TIMESPAN]
63 [--max-status-checks-per-second MAX_STATUS_CHECKS_PER_SECOND]
64 [--seconds-between-status-checks SECONDS_BETWEEN_STATUS_CHECKS]
65 [--retries RETRIES] [--wrapper-prefix WRAPPER_PREFIX]
66 [--default-storage-provider DEFAULT_STORAGE_PROVIDER]
67 [--default-storage-prefix DEFAULT_STORAGE_PREFIX]
68 [--local-storage-prefix LOCAL_STORAGE_PREFIX]
69 [--remote-job-local-storage-prefix REMOTE_JOB_LOCAL_STORAGE_PREFIX]
70 [--shared-fs-usage {input-output,persistence,software-deployment,software-deployment-cache,source-cache,sources,storage-local-copies,none} [{input-output,persistence,software-deployment,software-deployment-cache,source-cache,sources,storage-local-copies,none} ...]]
71 [--scheduler-greediness SCHEDULER_GREEDINESS]
72 [--scheduler-subsample SCHEDULER_SUBSAMPLE] [--no-hooks]
73 [--debug] [--runtime-profile FILE]
74 [--local-groupid LOCAL_GROUPID] [--attempt ATTEMPT]
75 [--show-failed-logs] [--logger {} [{} ...]]
76 [--job-deploy-sources] [--benchmark-extended]
77 [--persistence-backend {db,file}]
78 [--persistence-backend-db-url PERSISTENCE_BACKEND_DB_URL]
79 [--container-image IMAGE] [--immediate-submit]
80 [--jobscript SCRIPT] [--jobname NAME]
81 [--software-deployment-method {apptainer,conda,env-modules} [{apptainer,conda,env-modules} ...]]
82 [--container-cleanup-images] [--use-conda]
83 [--conda-not-block-search-path-envvars] [--list-conda-envs]
84 [--conda-prefix DIR] [--conda-cleanup-envs]
85 [--conda-cleanup-pkgs [{tarballs,cache}]]
86 [--conda-create-envs-only] [--conda-frontend {conda,mamba}]
87 [--use-apptainer] [--apptainer-prefix DIR]
88 [--apptainer-args ARGS] [--use-envmodules]
89 [--deploy-sources QUERY CHECKSUM]
90 [--target-jobs TARGET_JOBS [TARGET_JOBS ...]]
91 [--mode {default,remote,subprocess}]
92 [--scheduler-solver-path SCHEDULER_SOLVER_PATH]
93 [--max-jobs-per-second MAX_JOBS_PER_SECOND]
94 [--storage-azure-max-requests-per-second [TAG::]VALUE [[TAG::]VALUE ...]]
95 [--storage-azure-account-name [TAG::]VALUE [[TAG::]VALUE ...]]
96 [--storage-ftp-max-requests-per-second [TAG::]VALUE [[TAG::]VALUE ...]]
97 [--storage-ftp-username [TAG::]VALUE [[TAG::]VALUE ...]]
98 [--storage-ftp-password [TAG::]VALUE [[TAG::]VALUE ...]]
99 [--storage-ftp-active-mode]
100 [--storage-gcs-max-requests-per-second [TAG::]VALUE [[TAG::]VALUE ...]]
101 [--storage-gcs-project [TAG::]VALUE [[TAG::]VALUE ...]]
102 [--storage-gcs-stay-on-remote [TAG::]VALUE [[TAG::]VALUE ...]]
103 [--storage-gcs-retries [TAG::]VALUE [[TAG::]VALUE ...]]
104 [--storage-http-max-requests-per-second [TAG::]VALUE [[TAG::]VALUE ...]]
105 [--storage-http-auth [TAG::]AUTH_TYPE=ARG1,ARG2,... [[TAG::]AUTH_TYPE=ARG1,ARG2,... ...]]
106 [--storage-http-allow-redirects [TAG::]VALUE [[TAG::]VALUE ...]]
107 [--storage-http-supports-head [TAG::]VALUE [[TAG::]VALUE ...]]
108 [--storage-irods-max-requests-per-second [TAG::]VALUE [[TAG::]VALUE ...]]
109 [--storage-irods-host [TAG::]VALUE [[TAG::]VALUE ...]]
110 [--storage-irods-port [TAG::]VALUE [[TAG::]VALUE ...]]
111 [--storage-irods-username [TAG::]VALUE [[TAG::]VALUE ...]]
112 [--storage-irods-password [TAG::]VALUE [[TAG::]VALUE ...]]
113 [--storage-irods-zone [TAG::]VALUE [[TAG::]VALUE ...]]
114 [--storage-irods-home [TAG::]VALUE [[TAG::]VALUE ...]]
115 [--storage-irods-authentication-scheme [TAG::]VALUE [[TAG::]VALUE ...]]
116 [--storage-irods-encryption-algorithm [TAG::]VALUE [[TAG::]VALUE ...]]
117 [--storage-irods-encryption-key-size [TAG::]VALUE [[TAG::]VALUE ...]]
118 [--storage-irods-encryption-num-hash-rounds [TAG::]VALUE [[TAG::]VALUE ...]]
119 [--storage-irods-encryption-salt-size [TAG::]VALUE [[TAG::]VALUE ...]]
120 [--storage-irods-client-server-negotiation [TAG::]VALUE [[TAG::]VALUE ...]]
121 [--storage-irods-client-server-policy [TAG::]VALUE [[TAG::]VALUE ...]]
122 [--storage-irods-ssl-verify-server [TAG::]VALUE [[TAG::]VALUE ...]]
123 [--storage-irods-ssl-ca-certificate-file [TAG::]VALUE [[TAG::]VALUE ...]]
124 [--storage-irods-use-ssl]
125 [--storage-s3-max-requests-per-second [TAG::]VALUE [[TAG::]VALUE ...]]
126 [--storage-s3-endpoint-url [TAG::]VALUE [[TAG::]VALUE ...]]
127 [--storage-s3-region [TAG::]VALUE [[TAG::]VALUE ...]]
128 [--storage-s3-access-key [TAG::]VALUE [[TAG::]VALUE ...]]
129 [--storage-s3-secret-key [TAG::]VALUE [[TAG::]VALUE ...]]
130 [--storage-s3-token [TAG::]VALUE [[TAG::]VALUE ...]]
131 [--storage-s3-signature-version [TAG::]VALUE [[TAG::]VALUE ...]]
132 [--storage-s3-retries [TAG::]VALUE [[TAG::]VALUE ...]]
133 [--storage-xrootd-max-requests-per-second [TAG::]VALUE [[TAG::]VALUE ...]]
134 [--storage-xrootd-host [TAG::]VALUE [[TAG::]VALUE ...]]
135 [--storage-xrootd-port [TAG::]VALUE [[TAG::]VALUE ...]]
136 [--storage-xrootd-username [TAG::]VALUE [[TAG::]VALUE ...]]
137 [--storage-xrootd-password [TAG::]VALUE [[TAG::]VALUE ...]]
138 [--storage-xrootd-protocol [TAG::]VALUE [[TAG::]VALUE ...]]
139 [--storage-xrootd-url-decorator [TAG::]VALUE [[TAG::]VALUE ...]]
140 [--storage-xrootd-glob-wildcards-max-depth [TAG::]VALUE [[TAG::]VALUE ...]]
141 [--storage-zenodo-max-requests-per-second [TAG::]VALUE [[TAG::]VALUE ...]]
142 [--storage-zenodo-access-token [TAG::]VALUE [[TAG::]VALUE ...]]
143 [--storage-zenodo-restricted-access-token [TAG::]VALUE [[TAG::]VALUE ...]]
144 [--storage-zenodo-sandbox] [--report-html-path VALUE]
145 [--report-html-stylesheet-path VALUE]
146 [--scheduler-greedy-greediness VALUE]
147 [--scheduler-greedy-omit-prioritize-by-temp-and-input]
148 [--scheduler-ilp-solver VALUE]
149 [--scheduler-ilp-solver-path VALUE]
150 [targets ...]
151snakemake: error: argument --executor/-e: invalid choice: 'slurm' (choose from 'local', 'dryrun', 'touch')
Formatting results
All tests passed!