Ovewh/Climaso
The CLimate IMpact of AeroSOls: forcing and feedbacks
Overview
Topics:
Latest release: piClim-2xdust-v1.1.0, Last update: 2023-05-03
Linting: linting: failed, Formatting: formatting: failed
Deployment
Step 1: Install Snakemake and Snakedeploy
Snakemake and Snakedeploy are best installed via the Mamba package manager (a drop-in replacement for conda). If you have neither Conda nor Mamba, it is recommended to install Miniforge. More details regarding Mamba can be found here.
When using Mamba, run
mamba create -c conda-forge -c bioconda --name snakemake snakemake snakedeploy
to install both Snakemake and Snakedeploy in an isolated environment. For all following commands ensure that this environment is activated via
conda activate snakemake
Step 2: Deploy workflow
With Snakemake and Snakedeploy installed, the workflow can be deployed as follows. First, create an appropriate project working directory on your system and enter it:
mkdir -p path/to/project-workdir
cd path/to/project-workdir
In all following steps, we will assume that you are inside of that directory. Then run
snakedeploy deploy-workflow https://github.com/Ovewh/Climaso . --tag piClim-2xdust-v1.1.0
Snakedeploy will create two folders, workflow
and config
. The former contains the deployment of the chosen workflow as a Snakemake module, the latter contains configuration files which will be modified in the next step in order to configure the workflow to your needs.
Step 3: Configure workflow
To configure the workflow, adapt config/config.yml
to your needs following the instructions below.
Step 4: Run workflow
The deployment method is controlled using the --software-deployment-method
(short --sdm
) argument.
To run the workflow with automatic deployment of all required software via conda
/mamba
, use
snakemake --cores all --sdm conda
Snakemake will automatically detect the main Snakefile
in the workflow
subfolder and execute the workflow module that has been defined by the deployment in step 2.
For further options such as cluster and cloud execution, see the docs.
Step 5: Generate report
After finalizing your data analysis, you can automatically generate an interactive visual HTML report for inspection of results together with parameters and code inside of the browser using
snakemake --report report.zip
Configuration
The following section is imported from the workflow’s config/README.md
.
Describe how to configure the workflow (using config.yaml and maybe additional files). All of them need to be present with example entries inside of the config folder.
Linting and formatting
Linting results
1/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:410: SyntaxWarning: invalid escape sequence '\D'
2 vmax=90,
3/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:457: SyntaxWarning: invalid escape sequence '\D'
4 vmax=70,
5/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:503: SyntaxWarning: invalid escape sequence '\D'
6 'MIROC6' ,
7/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:542: SyntaxWarning: invalid escape sequence '\D'
8 model = ['GISS-E2-1-G', 'MIROC6', 'MPI-ESM-1-2-HAM','EC-Earth3-AerChem',
9/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:587: SyntaxWarning: invalid escape sequence '\D'
10
11/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:972: SyntaxWarning: invalid escape sequence '\D'
12/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:1019: SyntaxWarning: invalid escape sequence '\D'
13/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:1066: SyntaxWarning: invalid escape sequence '\D'
14Lints for snakefile /tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/Snakefile:
15 * Absolute path "/cluster/shared/ESGF" in line 18:
16 Do not define absolute paths inside of the workflow, since this renders
17 your workflow irreproducible on other machines. Use path relative to the
18 working directory instead, or make the path configurable via a config
19 file.
20 Also see:
21 https://snakemake.readthedocs.io/en/latest/snakefiles/configuration.html#configuration
22 * Path composition with '+' in line 50:
23 This becomes quickly unreadable. Usually, it is better to endure some
24 redundancy against having a more readable workflow. Hence, just repeat
25 common prefixes. If path composition is unavoidable, use pathlib or
26 (python >= 3.6) string formatting with f"...".
27 Also see:
28
29 * Path composition with '+' in line 54:
30 This becomes quickly unreadable. Usually, it is better to endure some
31 redundancy against having a more readable workflow. Hence, just repeat
32 common prefixes. If path composition is unavoidable, use pathlib or
33 (python >= 3.6) string formatting with f"...".
34 Also see:
35
36 * Path composition with '+' in line 72:
37 This becomes quickly unreadable. Usually, it is better to endure some
38 redundancy against having a more readable workflow. Hence, just repeat
39 common prefixes. If path composition is unavoidable, use pathlib or
40 (python >= 3.6) string formatting with f"...".
41 Also see:
42
43 * Path composition with '+' in line 74:
44 This becomes quickly unreadable. Usually, it is better to endure some
45 redundancy against having a more readable workflow. Hence, just repeat
46 common prefixes. If path composition is unavoidable, use pathlib or
47 (python >= 3.6) string formatting with f"...".
48 Also see:
49
50 * Path composition with '+' in line 76:
51 This becomes quickly unreadable. Usually, it is better to endure some
52 redundancy against having a more readable workflow. Hence, just repeat
53 common prefixes. If path composition is unavoidable, use pathlib or
54 (python >= 3.6) string formatting with f"...".
55 Also see:
56
57 * Path composition with '+' in line 78:
58 This becomes quickly unreadable. Usually, it is better to endure some
59 redundancy against having a more readable workflow. Hence, just repeat
60 common prefixes. If path composition is unavoidable, use pathlib or
61 (python >= 3.6) string formatting with f"...".
62 Also see:
63
64
65Lints for snakefile /tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/compute_ERF.smk:
66 * Path composition with '+' in line 25:
67 This becomes quickly unreadable. Usually, it is better to endure some
68 redundancy against having a more readable workflow. Hence, just repeat
69 common prefixes. If path composition is unavoidable, use pathlib or
70 (python >= 3.6) string formatting with f"...".
71 Also see:
72
73 * Path composition with '+' in line 53:
74 This becomes quickly unreadable. Usually, it is better to endure some
75 redundancy against having a more readable workflow. Hence, just repeat
76 common prefixes. If path composition is unavoidable, use pathlib or
77 (python >= 3.6) string formatting with f"...".
78 Also see:
79
80 * Path composition with '+' in line 78:
81 This becomes quickly unreadable. Usually, it is better to endure some
82 redundancy against having a more readable workflow. Hence, just repeat
83 common prefixes. If path composition is unavoidable, use pathlib or
84 (python >= 3.6) string formatting with f"...".
85 Also see:
86
87 * Path composition with '+' in line 94:
88 This becomes quickly unreadable. Usually, it is better to endure some
89 redundancy against having a more readable workflow. Hence, just repeat
90 common prefixes. If path composition is unavoidable, use pathlib or
91 (python >= 3.6) string formatting with f"...".
92 Also see:
93
94 * Path composition with '+' in line 109:
95 This becomes quickly unreadable. Usually, it is better to endure some
96 redundancy against having a more readable workflow. Hence, just repeat
97 common prefixes. If path composition is unavoidable, use pathlib or
98 (python >= 3.6) string formatting with f"...".
99 Also see:
100
101 * Path composition with '+' in line 123:
102 This becomes quickly unreadable. Usually, it is better to endure some
103 redundancy against having a more readable workflow. Hence, just repeat
104 common prefixes. If path composition is unavoidable, use pathlib or
105 (python >= 3.6) string formatting with f"...".
106 Also see:
107
108 * Path composition with '+' in line 134:
109 This becomes quickly unreadable. Usually, it is better to endure some
110 redundancy against having a more readable workflow. Hence, just repeat
111 common prefixes. If path composition is unavoidable, use pathlib or
112 (python >= 3.6) string formatting with f"...".
113 Also see:
114
115 * Path composition with '+' in line 135:
116 This becomes quickly unreadable. Usually, it is better to endure some
117 redundancy against having a more readable workflow. Hence, just repeat
118 common prefixes. If path composition is unavoidable, use pathlib or
119 (python >= 3.6) string formatting with f"...".
120 Also see:
121
122 * Path composition with '+' in line 137:
123 This becomes quickly unreadable. Usually, it is better to endure some
124 redundancy against having a more readable workflow. Hence, just repeat
125 common prefixes. If path composition is unavoidable, use pathlib or
126 (python >= 3.6) string formatting with f"...".
127 Also see:
128
129 * Path composition with '+' in line 149:
130 This becomes quickly unreadable. Usually, it is better to endure some
131 redundancy against having a more readable workflow. Hence, just repeat
132 common prefixes. If path composition is unavoidable, use pathlib or
133 (python >= 3.6) string formatting with f"...".
134 Also see:
135
136 * Path composition with '+' in line 8:
137 This becomes quickly unreadable. Usually, it is better to endure some
138 redundancy against having a more readable workflow. Hence, just repeat
139 common prefixes. If path composition is unavoidable, use pathlib or
140 (python >= 3.6) string formatting with f"...".
141 Also see:
142
143
144Lints for snakefile /tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/utils.smk:
145 * Mixed rules and functions in same snakefile.:
146 Small one-liner functions used only once should be defined as lambda
147 expressions. Other functions should be collected in a common module, e.g.
148 'rules/common.smk'. This makes the workflow steps more readable.
149 Also see:
150 https://snakemake.readthedocs.io/en/latest/snakefiles/modularization.html#includes
151 * Path composition with '+' in line 79:
152 This becomes quickly unreadable. Usually, it is better to endure some
153 redundancy against having a more readable workflow. Hence, just repeat
154 common prefixes. If path composition is unavoidable, use pathlib or
155 (python >= 3.6) string formatting with f"...".
156 Also see:
157
158 * Path composition with '+' in line 91:
159 This becomes quickly unreadable. Usually, it is better to endure some
160 redundancy against having a more readable workflow. Hence, just repeat
161 common prefixes. If path composition is unavoidable, use pathlib or
162 (python >= 3.6) string formatting with f"...".
163 Also see:
164
165 * Path composition with '+' in line 105:
166 This becomes quickly unreadable. Usually, it is better to endure some
167 redundancy against having a more readable workflow. Hence, just repeat
168 common prefixes. If path composition is unavoidable, use pathlib or
169 (python >= 3.6) string formatting with f"...".
170 Also see:
171
172 * Path composition with '+' in line 119:
173 This becomes quickly unreadable. Usually, it is better to endure some
174 redundancy against having a more readable workflow. Hence, just repeat
175 common prefixes. If path composition is unavoidable, use pathlib or
176 (python >= 3.6) string formatting with f"...".
177 Also see:
178
179 * Path composition with '+' in line 127:
180 This becomes quickly unreadable. Usually, it is better to endure some
181 redundancy against having a more readable workflow. Hence, just repeat
182 common prefixes. If path composition is unavoidable, use pathlib or
183 (python >= 3.6) string formatting with f"...".
184 Also see:
185
186
187Lints for snakefile /tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/diagnose_cloud_changes.smk:
188 * Path composition with '+' in line 11:
189 This becomes quickly unreadable. Usually, it is better to endure some
190 redundancy against having a more readable workflow. Hence, just repeat
191 common prefixes. If path composition is unavoidable, use pathlib or
192 (python >= 3.6) string formatting with f"...".
193 Also see:
194
195
196Lints for snakefile /tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:
197 * Path composition with '+' in line 4:
198 This becomes quickly unreadable. Usually, it is better to endure some
199 redundancy against having a more readable workflow. Hence, just repeat
200 common prefixes. If path composition is unavoidable, use pathlib or
201
202... (truncated)
Formatting results
1[DEBUG]
2[DEBUG] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/compute_ERF.smk": Formatted content is different from original
3[DEBUG]
4[DEBUG] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/dust_analysis.smk": Formatted content is different from original
5[DEBUG]
6[WARNING] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/diagnose_cloud_changes.smk": Keyword "input" at line 5 has comments under a value.
7 PEP8 recommends block comments appear before what they describe
8(see https://www.python.org/dev/peps/pep-0008/#id30)
9[DEBUG] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/diagnose_cloud_changes.smk": Formatted content is different from original
10[DEBUG]
11[WARNING] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/utils.smk": Keyword "input" at line 39 has comments under a value.
12 PEP8 recommends block comments appear before what they describe
13(see https://www.python.org/dev/peps/pep-0008/#id30)
14[DEBUG] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/utils.smk": Formatted content is different from original
15[DEBUG]
16[WARNING] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/Snakefile": Keyword "input" at line 76 has comments under a value.
17 PEP8 recommends block comments appear before what they describe
18(see https://www.python.org/dev/peps/pep-0008/#id30)
19[DEBUG] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/Snakefile": Formatted content is different from original
20[DEBUG]
21[ERROR] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk": InvalidPython: Black error:
Cannot parse: 138:0: EOF in multi-line statement
[INFO] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk": 1 file(s) raised parsing errors 🤕
[INFO] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk": 5 file(s) would be changed 😬
snakefmt version: 0.8.5