Ovewh/Climaso

The CLimate IMpact of AeroSOls: forcing and feedbacks

Overview

Latest release: piClim-2xdust-v1.1.0, Last update: 2023-05-03

Linting: linting: failed, Formatting: formatting: failed

Deployment

Step 1: Install Snakemake and Snakedeploy

Snakemake and Snakedeploy are best installed via the Mamba package manager (a drop-in replacement for conda). If you have neither Conda nor Mamba, it is recommended to install Miniforge. More details regarding Mamba can be found here.

When using Mamba, run

mamba create -c conda-forge -c bioconda --name snakemake snakemake snakedeploy

to install both Snakemake and Snakedeploy in an isolated environment. For all following commands ensure that this environment is activated via

conda activate snakemake

Step 2: Deploy workflow

With Snakemake and Snakedeploy installed, the workflow can be deployed as follows. First, create an appropriate project working directory on your system and enter it:

mkdir -p path/to/project-workdir
cd path/to/project-workdir

In all following steps, we will assume that you are inside of that directory. Then run

snakedeploy deploy-workflow https://github.com/Ovewh/Climaso . --tag piClim-2xdust-v1.1.0

Snakedeploy will create two folders, workflow and config. The former contains the deployment of the chosen workflow as a Snakemake module, the latter contains configuration files which will be modified in the next step in order to configure the workflow to your needs.

Step 3: Configure workflow

To configure the workflow, adapt config/config.yml to your needs following the instructions below.

Step 4: Run workflow

The deployment method is controlled using the --software-deployment-method (short --sdm) argument.

To run the workflow with automatic deployment of all required software via conda/mamba, use

snakemake --cores all --sdm conda

Snakemake will automatically detect the main Snakefile in the workflow subfolder and execute the workflow module that has been defined by the deployment in step 2.

For further options such as cluster and cloud execution, see the docs.

Step 5: Generate report

After finalizing your data analysis, you can automatically generate an interactive visual HTML report for inspection of results together with parameters and code inside of the browser using

snakemake --report report.zip

Configuration

The following section is imported from the workflow’s config/README.md.

Describe how to configure the workflow (using config.yaml and maybe additional files). All of them need to be present with example entries inside of the config folder.

Linting and formatting

Linting results

  1/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:410: SyntaxWarning: invalid escape sequence '\D'
  2  vmax=90,
  3/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:457: SyntaxWarning: invalid escape sequence '\D'
  4  vmax=70,
  5/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:503: SyntaxWarning: invalid escape sequence '\D'
  6  'MIROC6' ,
  7/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:542: SyntaxWarning: invalid escape sequence '\D'
  8  model = ['GISS-E2-1-G', 'MIROC6', 'MPI-ESM-1-2-HAM','EC-Earth3-AerChem',
  9/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:587: SyntaxWarning: invalid escape sequence '\D'
 10  
 11/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:972: SyntaxWarning: invalid escape sequence '\D'
 12/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:1019: SyntaxWarning: invalid escape sequence '\D'
 13/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:1066: SyntaxWarning: invalid escape sequence '\D'
 14Lints for snakefile /tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/Snakefile:
 15    * Absolute path "/cluster/shared/ESGF" in line 18:
 16      Do not define absolute paths inside of the workflow, since this renders
 17      your workflow irreproducible on other machines. Use path relative to the
 18      working directory instead, or make the path configurable via a config
 19      file.
 20      Also see:
 21      https://snakemake.readthedocs.io/en/latest/snakefiles/configuration.html#configuration
 22    * Path composition with '+' in line 50:
 23      This becomes quickly unreadable. Usually, it is better to endure some
 24      redundancy against having a more readable workflow. Hence, just repeat
 25      common prefixes. If path composition is unavoidable, use pathlib or
 26      (python >= 3.6) string formatting with f"...".
 27      Also see:
 28
 29    * Path composition with '+' in line 54:
 30      This becomes quickly unreadable. Usually, it is better to endure some
 31      redundancy against having a more readable workflow. Hence, just repeat
 32      common prefixes. If path composition is unavoidable, use pathlib or
 33      (python >= 3.6) string formatting with f"...".
 34      Also see:
 35
 36    * Path composition with '+' in line 72:
 37      This becomes quickly unreadable. Usually, it is better to endure some
 38      redundancy against having a more readable workflow. Hence, just repeat
 39      common prefixes. If path composition is unavoidable, use pathlib or
 40      (python >= 3.6) string formatting with f"...".
 41      Also see:
 42
 43    * Path composition with '+' in line 74:
 44      This becomes quickly unreadable. Usually, it is better to endure some
 45      redundancy against having a more readable workflow. Hence, just repeat
 46      common prefixes. If path composition is unavoidable, use pathlib or
 47      (python >= 3.6) string formatting with f"...".
 48      Also see:
 49
 50    * Path composition with '+' in line 76:
 51      This becomes quickly unreadable. Usually, it is better to endure some
 52      redundancy against having a more readable workflow. Hence, just repeat
 53      common prefixes. If path composition is unavoidable, use pathlib or
 54      (python >= 3.6) string formatting with f"...".
 55      Also see:
 56
 57    * Path composition with '+' in line 78:
 58      This becomes quickly unreadable. Usually, it is better to endure some
 59      redundancy against having a more readable workflow. Hence, just repeat
 60      common prefixes. If path composition is unavoidable, use pathlib or
 61      (python >= 3.6) string formatting with f"...".
 62      Also see:
 63
 64
 65Lints for snakefile /tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/compute_ERF.smk:
 66    * Path composition with '+' in line 25:
 67      This becomes quickly unreadable. Usually, it is better to endure some
 68      redundancy against having a more readable workflow. Hence, just repeat
 69      common prefixes. If path composition is unavoidable, use pathlib or
 70      (python >= 3.6) string formatting with f"...".
 71      Also see:
 72
 73    * Path composition with '+' in line 53:
 74      This becomes quickly unreadable. Usually, it is better to endure some
 75      redundancy against having a more readable workflow. Hence, just repeat
 76      common prefixes. If path composition is unavoidable, use pathlib or
 77      (python >= 3.6) string formatting with f"...".
 78      Also see:
 79
 80    * Path composition with '+' in line 78:
 81      This becomes quickly unreadable. Usually, it is better to endure some
 82      redundancy against having a more readable workflow. Hence, just repeat
 83      common prefixes. If path composition is unavoidable, use pathlib or
 84      (python >= 3.6) string formatting with f"...".
 85      Also see:
 86
 87    * Path composition with '+' in line 94:
 88      This becomes quickly unreadable. Usually, it is better to endure some
 89      redundancy against having a more readable workflow. Hence, just repeat
 90      common prefixes. If path composition is unavoidable, use pathlib or
 91      (python >= 3.6) string formatting with f"...".
 92      Also see:
 93
 94    * Path composition with '+' in line 109:
 95      This becomes quickly unreadable. Usually, it is better to endure some
 96      redundancy against having a more readable workflow. Hence, just repeat
 97      common prefixes. If path composition is unavoidable, use pathlib or
 98      (python >= 3.6) string formatting with f"...".
 99      Also see:
100
101    * Path composition with '+' in line 123:
102      This becomes quickly unreadable. Usually, it is better to endure some
103      redundancy against having a more readable workflow. Hence, just repeat
104      common prefixes. If path composition is unavoidable, use pathlib or
105      (python >= 3.6) string formatting with f"...".
106      Also see:
107
108    * Path composition with '+' in line 134:
109      This becomes quickly unreadable. Usually, it is better to endure some
110      redundancy against having a more readable workflow. Hence, just repeat
111      common prefixes. If path composition is unavoidable, use pathlib or
112      (python >= 3.6) string formatting with f"...".
113      Also see:
114
115    * Path composition with '+' in line 135:
116      This becomes quickly unreadable. Usually, it is better to endure some
117      redundancy against having a more readable workflow. Hence, just repeat
118      common prefixes. If path composition is unavoidable, use pathlib or
119      (python >= 3.6) string formatting with f"...".
120      Also see:
121
122    * Path composition with '+' in line 137:
123      This becomes quickly unreadable. Usually, it is better to endure some
124      redundancy against having a more readable workflow. Hence, just repeat
125      common prefixes. If path composition is unavoidable, use pathlib or
126      (python >= 3.6) string formatting with f"...".
127      Also see:
128
129    * Path composition with '+' in line 149:
130      This becomes quickly unreadable. Usually, it is better to endure some
131      redundancy against having a more readable workflow. Hence, just repeat
132      common prefixes. If path composition is unavoidable, use pathlib or
133      (python >= 3.6) string formatting with f"...".
134      Also see:
135
136    * Path composition with '+' in line 8:
137      This becomes quickly unreadable. Usually, it is better to endure some
138      redundancy against having a more readable workflow. Hence, just repeat
139      common prefixes. If path composition is unavoidable, use pathlib or
140      (python >= 3.6) string formatting with f"...".
141      Also see:
142
143
144Lints for snakefile /tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/utils.smk:
145    * Mixed rules and functions in same snakefile.:
146      Small one-liner functions used only once should be defined as lambda
147      expressions. Other functions should be collected in a common module, e.g.
148      'rules/common.smk'. This makes the workflow steps more readable.
149      Also see:
150      https://snakemake.readthedocs.io/en/latest/snakefiles/modularization.html#includes
151    * Path composition with '+' in line 79:
152      This becomes quickly unreadable. Usually, it is better to endure some
153      redundancy against having a more readable workflow. Hence, just repeat
154      common prefixes. If path composition is unavoidable, use pathlib or
155      (python >= 3.6) string formatting with f"...".
156      Also see:
157
158    * Path composition with '+' in line 91:
159      This becomes quickly unreadable. Usually, it is better to endure some
160      redundancy against having a more readable workflow. Hence, just repeat
161      common prefixes. If path composition is unavoidable, use pathlib or
162      (python >= 3.6) string formatting with f"...".
163      Also see:
164
165    * Path composition with '+' in line 105:
166      This becomes quickly unreadable. Usually, it is better to endure some
167      redundancy against having a more readable workflow. Hence, just repeat
168      common prefixes. If path composition is unavoidable, use pathlib or
169      (python >= 3.6) string formatting with f"...".
170      Also see:
171
172    * Path composition with '+' in line 119:
173      This becomes quickly unreadable. Usually, it is better to endure some
174      redundancy against having a more readable workflow. Hence, just repeat
175      common prefixes. If path composition is unavoidable, use pathlib or
176      (python >= 3.6) string formatting with f"...".
177      Also see:
178
179    * Path composition with '+' in line 127:
180      This becomes quickly unreadable. Usually, it is better to endure some
181      redundancy against having a more readable workflow. Hence, just repeat
182      common prefixes. If path composition is unavoidable, use pathlib or
183      (python >= 3.6) string formatting with f"...".
184      Also see:
185
186
187Lints for snakefile /tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/diagnose_cloud_changes.smk:
188    * Path composition with '+' in line 11:
189      This becomes quickly unreadable. Usually, it is better to endure some
190      redundancy against having a more readable workflow. Hence, just repeat
191      common prefixes. If path composition is unavoidable, use pathlib or
192      (python >= 3.6) string formatting with f"...".
193      Also see:
194
195
196Lints for snakefile /tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk:
197    * Path composition with '+' in line 4:
198      This becomes quickly unreadable. Usually, it is better to endure some
199      redundancy against having a more readable workflow. Hence, just repeat
200      common prefixes. If path composition is unavoidable, use pathlib or
201
202... (truncated)

Formatting results

 1[DEBUG] 
 2[DEBUG] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/compute_ERF.smk":  Formatted content is different from original
 3[DEBUG] 
 4[DEBUG] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/dust_analysis.smk":  Formatted content is different from original
 5[DEBUG] 
 6[WARNING] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/diagnose_cloud_changes.smk":  Keyword "input" at line 5 has comments under a value.
 7	PEP8 recommends block comments appear before what they describe
 8(see https://www.python.org/dev/peps/pep-0008/#id30)
 9[DEBUG] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/diagnose_cloud_changes.smk":  Formatted content is different from original
10[DEBUG] 
11[WARNING] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/utils.smk":  Keyword "input" at line 39 has comments under a value.
12	PEP8 recommends block comments appear before what they describe
13(see https://www.python.org/dev/peps/pep-0008/#id30)
14[DEBUG] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/utils.smk":  Formatted content is different from original
15[DEBUG] 
16[WARNING] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/Snakefile":  Keyword "input" at line 76 has comments under a value.
17	PEP8 recommends block comments appear before what they describe
18(see https://www.python.org/dev/peps/pep-0008/#id30)
19[DEBUG] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/Snakefile":  Formatted content is different from original
20[DEBUG] 
21[ERROR] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk":  InvalidPython: Black error:

Cannot parse: 138:0: EOF in multi-line statement


[INFO] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk":  1 file(s) raised parsing errors 🤕
[INFO] In file "/tmp/tmpv8xaqw8w/Ovewh-Climaso-5db1376/workflow/rules/plotAerChemMIP.smk":  5 file(s) would be changed 😬

snakefmt version: 0.8.5