clami66/AF_server
None
Overview
Topics:
Latest release: None, Last update: 2023-01-02
Linting: linting: failed, Formatting: formatting: failed
Deployment
Step 1: Install Snakemake and Snakedeploy
Snakemake and Snakedeploy are best installed via the Mamba package manager (a drop-in replacement for conda). If you have neither Conda nor Mamba, it is recommended to install Miniforge. More details regarding Mamba can be found here.
When using Mamba, run
mamba create -c conda-forge -c bioconda --name snakemake snakemake snakedeploy
to install both Snakemake and Snakedeploy in an isolated environment. For all following commands ensure that this environment is activated via
conda activate snakemake
Step 2: Deploy workflow
With Snakemake and Snakedeploy installed, the workflow can be deployed as follows. First, create an appropriate project working directory on your system and enter it:
mkdir -p path/to/project-workdir
cd path/to/project-workdir
In all following steps, we will assume that you are inside of that directory. Then run
snakedeploy deploy-workflow https://github.com/clami66/AF_server . --tag None
Snakedeploy will create two folders, workflow
and config
. The former contains the deployment of the chosen workflow as a Snakemake module, the latter contains configuration files which will be modified in the next step in order to configure the workflow to your needs.
Step 3: Configure workflow
To configure the workflow, adapt config/config.yml
to your needs following the instructions below.
Step 4: Run workflow
The deployment method is controlled using the --software-deployment-method
(short --sdm
) argument.
To run the workflow with automatic deployment of all required software via conda
/mamba
, use
snakemake --cores all --sdm conda
Snakemake will automatically detect the main Snakefile
in the workflow
subfolder and execute the workflow module that has been defined by the deployment in step 2.
For further options such as cluster and cloud execution, see the docs.
Step 5: Generate report
After finalizing your data analysis, you can automatically generate an interactive visual HTML report for inspection of results together with parameters and code inside of the browser using
snakemake --report report.zip
Configuration
The following section is imported from the workflow’s config/README.md
.
config.yaml
# E-mail settings
server_address: "user@server.com" ### server mailbox address
mail_server: "server.com" ### server mailbox domain
whitelist: "config/whitelist" ### The whitelist contains one email per line, only senders from the list will be allowed
# AF settings
AF_install_dir: "/proj/apps/alphafold" ### AlphaFold installation directory
monomer_flagfile: "/proj/apps/alphafold/flagfiles/monomer_full_dbs.flag" # monomer and multimer flag files paths
multimer_flagfile: "/proj/apps/alphafold/flagfiles/multimer_full_dbs.flag"
# resources settings, these override the slurm profile settings
mem_mb: 240000
walltime: 4320
max_gpus: 8
gpus_per_node: 8
n_cores_per_gpu: 16
# CASP settings, can be ignored if it is not a CASP server. Will show in PDB headers
CASP_groupn: "xxxx-yyyy-zzzz-wwww" #
CASP_groupn_multi: "wwww-xxxx-yyyy-zzzz"
CASP_groupname: "af2-standard"
CASP_groupname_multi: "af2-multimer"
CASP_sender: "casp-meta@predictioncenter.org"
# for uploading results somewhere (passwordless).
# Results in invoking `rsync -av results/AF_models/{target}/[models,msas].tar.gz user@server.com:/path`
data_server_user: "user"
data_server_address: "server.com"
data_server_folder: "/path"
envmodules.yaml
envmodules:
run_alphafold: # specifies envmodules to load when executing rule `run_alphafold`
- Anaconda/2021.05-nsc1
- AlphaFold/2.2.1
- other envmodules...
whitelist
allowed_sender@mail.com # one email per line
another_allowed_sender@server.net
Linting and formatting
Linting results
1Lints for snakefile /tmp/tmpzw509uh7/workflow/rules/common.smk:
2 * Path composition with '+' in line 144:
3 This becomes quickly unreadable. Usually, it is better to endure some
4 redundancy against having a more readable workflow. Hence, just repeat
5 common prefixes. If path composition is unavoidable, use pathlib or
6 (python >= 3.6) string formatting with f"...".
7 Also see:
8
9
10Lints for rule add_headers (line 82, /tmp/tmpzw509uh7/workflow/rules/alphafold.smk):
11 * No log directive defined:
12 Without a log directive, all output will be printed to the terminal. In
13 distributed environments, this means that errors are harder to discover.
14 In local environments, output of concurrent jobs will be mixed and become
15 unreadable.
16 Also see:
17 https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#log-files
18 * Specify a conda environment or container for each rule.:
19 This way, the used software for each specific step is documented, and the
20 workflow can be executed on any machine without prerequisites.
21 Also see:
22 https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#integrated-package-management
23 https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#running-jobs-in-containers
24 * Param model_dir is a prefix of input or output file but hardcoded:
25 If this is meant to represent a file path prefix, it will fail when
26 running workflow in environments without a shared filesystem. Instead,
27 provide a function that infers the appropriate prefix from the input or
28 output file, e.g.: lambda w, input: os.path.splitext(input[0])[0]
29 Also see:
30 https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
31 https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html#tutorial-input-functions
32
33Lints for rule send_acknowledgement (line 7, /tmp/tmpzw509uh7/workflow/rules/submit.smk):
34 * No log directive defined:
35 Without a log directive, all output will be printed to the terminal. In
36 distributed environments, this means that errors are harder to discover.
37 In local environments, output of concurrent jobs will be mixed and become
38 unreadable.
39 Also see:
40 https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#log-files
41 * Migrate long run directives into scripts or notebooks:
42 Long run directives hamper workflow readability. Use the script or
43 notebook direcive instead. Note that the script or notebook directive does
44 not involve boilerplate. Similar to run, you will have direct access to
45 params, input, output, and wildcards.Only use the run direcive for a
46 handful of lines.
47 Also see:
48 https://snakemake.readthedocs.io/en/latest/snakefiles/rules.html#external-scripts
49 https://snakemake.readthedocs.io/en/latest/snakefiles/rules.html#jupyter-notebook-integration
50
51Lints for rule send_results (line 40, /tmp/tmpzw509uh7/workflow/rules/submit.smk):
52 * No log directive defined:
53 Without a log directive, all output will be printed to the terminal. In
54 distributed environments, this means that errors are harder to discover.
55 In local environments, output of concurrent jobs will be mixed and become
56 unreadable.
57 Also see:
58 https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#log-files
59 * Migrate long run directives into scripts or notebooks:
60 Long run directives hamper workflow readability. Use the script or
61 notebook direcive instead. Note that the script or notebook directive does
62 not involve boilerplate. Similar to run, you will have direct access to
63 params, input, output, and wildcards.Only use the run direcive for a
64 handful of lines.
65 Also see:
66 https://snakemake.readthedocs.io/en/latest/snakefiles/rules.html#external-scripts
67 https://snakemake.readthedocs.io/en/latest/snakefiles/rules.html#jupyter-notebook-integration
68
69Lints for rule data_upload (line 8, /tmp/tmpzw509uh7/workflow/rules/upload.smk):
70 * Specify a conda environment or container for each rule.:
71 This way, the used software for each specific step is documented, and the
72 workflow can be executed on any machine without prerequisites.
73 Also see:
74 https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#integrated-package-management
75 https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#running-jobs-in-containers
76 * Param data_dir is a prefix of input or output file but hardcoded:
77 If this is meant to represent a file path prefix, it will fail when
78 running workflow in environments without a shared filesystem. Instead,
79 provide a function that infers the appropriate prefix from the input or
80 output file, e.g.: lambda w, input: os.path.splitext(input[0])[0]
81 Also see:
82 https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
83 https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html#tutorial-input-functions
84
85Lints for rule msa_upload (line 45, /tmp/tmpzw509uh7/workflow/rules/upload.smk):
86 * Specify a conda environment or container for each rule.:
87 This way, the used software for each specific step is documented, and the
88 workflow can be executed on any machine without prerequisites.
89 Also see:
90 https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#integrated-package-management
91 https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#running-jobs-in-containers
92 * Param data_dir is a prefix of input or output file but hardcoded:
93 If this is meant to represent a file path prefix, it will fail when
94 running workflow in environments without a shared filesystem. Instead,
95 provide a function that infers the appropriate prefix from the input or
96 output file, e.g.: lambda w, input: os.path.splitext(input[0])[0]
97 Also see:
98 https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
99 https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html#tutorial-input-functions
100
101Lints for rule model_upload (line 80, /tmp/tmpzw509uh7/workflow/rules/upload.smk):
102 * Specify a conda environment or container for each rule.:
103 This way, the used software for each specific step is documented, and the
104 workflow can be executed on any machine without prerequisites.
105 Also see:
106 https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#integrated-package-management
107 https://snakemake.readthedocs.io/en/latest/snakefiles/deployment.html#running-jobs-in-containers
108 * Param data_dir is a prefix of input or output file but hardcoded:
109 If this is meant to represent a file path prefix, it will fail when
110 running workflow in environments without a shared filesystem. Instead,
111 provide a function that infers the appropriate prefix from the input or
112 output file, e.g.: lambda w, input: os.path.splitext(input[0])[0]
113 Also see:
114 https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules
115 https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html#tutorial-input-functions
116
117Lints for rule pkl_reduction (line 116, /tmp/tmpzw509uh7/workflow/rules/upload.smk):
118 * No log directive defined:
119 Without a log directive, all output will be printed to the terminal. In
120 distributed environments, this means that errors are harder to discover.
121 In local environments, output of concurrent jobs will be mixed and become
122 unreadable.
123 Also see:
124 https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#log-files
125 * Migrate long run directives into scripts or notebooks:
126 Long run directives hamper workflow readability. Use the script or
127 notebook direcive instead. Note that the script or notebook directive does
128 not involve boilerplate. Similar to run, you will have direct access to
129 params, input, output, and wildcards.Only use the run direcive for a
130 handful of lines.
131 Also see:
132 https://snakemake.readthedocs.io/en/latest/snakefiles/rules.html#external-scripts
133 https://snakemake.readthedocs.io/en/latest/snakefiles/rules.html#jupyter-notebook-integration
Formatting results
1[DEBUG]
2[DEBUG] In file "/tmp/tmpzw509uh7/workflow/rules/alphafold.smk": Formatted content is different from original
3[DEBUG]
4[DEBUG]
5[DEBUG] In file "/tmp/tmpzw509uh7/workflow/rules/common.smk": Formatted content is different from original
6[DEBUG]
7[DEBUG] In file "/tmp/tmpzw509uh7/workflow/rules/upload.smk": Formatted content is different from original
8[DEBUG]
9[DEBUG] In file "/tmp/tmpzw509uh7/workflow/Snakefile": Formatted content is different from original
10[INFO] 4 file(s) would be changed 😬
11[INFO] 1 file(s) would be left unchanged 🎉
12
13snakefmt version: 0.8.0