Andrew Stiles
11/01/2024, 8:29 PM# This file should exist in the root of your pipeline directory
def has_label(directives):
return len(directives.label) > 0
def has_cpus(directives):
return len(directives.cpus) > 0
def rule_has_label_or_cpus(module):
for process in module.processes:
if not (has_label(process.directives) or has_cpus(process.directives)):
fatal("process %s has no label or cpus directive" % process.name)
Feedback is much appreciated!Arthur
11/02/2024, 5:16 PMMaxime Garcia
11/04/2024, 8:34 AMBen Sherman
11/04/2024, 9:04 AMBen Sherman
11/04/2024, 9:05 AMAndrew Stiles
11/07/2024, 9:42 PMreft lint nfcore
Here's more info
I focused on module rules, specifically container and label rules. They have full test coverage.Arthur
11/07/2024, 9:46 PMmypy and pyright)Andrew Stiles
11/07/2024, 9:47 PMreft lint nfcore functionality. I did some git surgery to remove a jar from the repo, so you have to re-clone if building from source. (sorry -- figured it was still small enough)Andrew Stiles
11/07/2024, 10:01 PMInteresting. I guess my main concern here is unification/concordance of the different lexers/syntax parsers if the two projects are separate and people want the linting but also the formatting.Definitely -- I'd like for RefTrace to be conservative here. As in, it shouldn't reject programs that are actually correct. It doesn't do any formatting.
Arthur
11/07/2024, 10:15 PMrules.py because we would want to be able to update/disable rules selectively (as we currently allow using the config yml) and would not want to rely on updates to this tool in order to update nf-core tooling. The barrier of yet another language (Go) on top of groovy/python would also be a barrier to devs making new rules easily I suspect (which is precisely the point of reftrace AFAICT!)
Calling out to remote modules is definitely a necessity for some of the tests but I think much of the linting can be done without and the rest with a single clone/pull of the remote repo.
Can rules.py contain any arbitrary Python? (Can it use third-party libraries?)
It would also possibly be useful if rules could take extra arguments (must have a default) through some form of configuration as this could allow the same rule-set to be used over multiple repositories/remotes with ease.
RE: the comparison to the new language server, I am more concerned that they both lex/tokenize the same code the same way. My naive hope (before this RefTrace was announced) was the language server could be used by nf-core tools to parse code in the way reftrace does here. Ideally it's best if tools are in sync (ie nf-core tools/other tooling could all depend on the language servers tokenization).Andrew Stiles
11/08/2024, 8:32 PMfrom reftrace import Module
module = Module("path/to/nextflow.nf")
print(f"File: {module.path}")
print(f"DSL Version: {module.dsl_version}")
for process in module.processes:
print(f"\nProcess: {process.name}")
for directive in process.directives:
if directive.format == "simple":
print(f" Container: {directive.simple_name}")
else: # ternary
print(f" Container: {directive.condition} ? {directive.true_name} : {directive.false_name}")
and some output:
File: /home/andrew/circrna/modules/local/annotation/main.nf
DSL Version: 2
Process: ANNOTATION
Container: ((workflow.containerEngine == singularity) && !(task.ext.singularity_pull_docker_container)) ? https://depot.galaxyproject.org/singularity/pandas:1.5.2 : biocontainers/pandas:1.5.2
I need to flesh out the bindings a bit and publish on PyPI.
And yes -- the goal of RefTrace is to make it easier for people to write correct pipelines! Hopefully having native Python access to the DSL helps. As far as integration and speeding things up, would you prefer a separate pip-installable tool or a PR to nf-core/tools?Arthur
11/08/2024, 10:08 PMArthur
11/08/2024, 10:10 PMAndrew Stiles
11/09/2024, 9:00 PMAndrew Stiles
11/11/2024, 8:36 AMfor process in module.processes:
for container in process.containers:
names = get_names(container)
for name in names:
if ' ' in name:
results.errors.append(ModuleError(
line=1, # TODO: use the actual line number
error=f"container name '{name}' contains spaces, which is not allowed"
))
The core reftrace library docs and API docs are updated. I still need to do more here.
It's annoying building everything from source. Once I publish the core library on PyPI, that example will run with pip install reftrace.