https://nf-co.re logo
Join Slack
Powered by
Hi! We’re working with a <custom executor>, loosel...
# configs
m
Hi! We’re working with a custom executor, loosely based on K8S. In our config we have several mountpoints predefined, which are mounted on each container when running a task. This is giving us some issues lately, especially when running a massive number of small jobs, e.g. during variant calling. Is there a way to add a closure to the executor block and make some config changes based on the name of the workflow?
e.g: 
workflow.name == nf-core/demultiplex :? mount sequencer_upload
m
Wouldn't you use a pipeline specific organisation config? https://github.com/nf-core/configs/tree/master/conf/pipeline/demultiplex Or am I missing something?
m
it's a secret
demultiplex was just an example, we can’t disclose the config
the configuration is going into 
$NXF_HOME/config
m
You can't set in the pipeline itself?
m
The pipeline is public, so we can’t add it in the config there
m
And then use 
.call()
to make it evaluate
m
Yes, I was looking mainly for a way to access the workflow name (not the run name) and if that variable would be available in the executor scope
but seqera AI helped me along already. I can access the 
manifest.name
property and get the workflow name from there in a closure
m
nice
m
Seems like an AI hallucination though 😞 
ERROR ~ Unknown config attribute nomad.jobs.manifest.name
redacted config
Copy code
nomad {
            client {
                address = System.getenv("NOMAD_ADDR")
                token   = System.getenv("NOMAD_TOKEN")
            }

            jobs {
                deleteOnCompletion = false
                namespace = System.getenv("NOMAD_NAMESPACE")
                datacenters = System.getenv("NOMAD_DC")

                volumes = [
                    manifest.name == "nf-core/demultiplex" ? { type "csi" name "bcl_share" path "/media/cmgg_upload" readOnly true } : ""
                ]
            }
        }
m
What if you tried to use 
"grep 'name:' $projectDir/.nf-core.yml".execute().text.contains('demultiplex')
in a closure which you call?
I think the configuration has access to very few variables unless they're in the file itself.
m
Nice one! This works 😄 although it seems a but dodgy to execute a command from a config file 😅
m
If it works, it works. Does it fail silently too?
m
The exclusion seems to work when using nf-core/demo. I’m trying out the inclusion right now
m
I'd test using a non-nf-core wf for failure
m
you mean a workflow without the 
.nf-core.yml
file?
m
ya
m
that’s the next step!
Tested all scenario’s, inclusion, exclusion and non nf-core pipelines. All seem to work fine 👍🏻
Thanks for the help, as always!
👍🏽 1
n
I love some good workarounds 🤣
Couldn't you also just check the last two directories of 
projectDir
? These should also contain the full pipeline name 😛
m
You mean the 
workflows
sub directory ?
n
no just the path 
$NXF_HOME/assets/nf-core/demultiplex
for example 😛
(yeah okay this won't work with manually downloaded pipelines, but we don't do that)
Open in Slack
PreviousNext
Powered by