Can someone help me with a small issue I have in a...
# configs
l
Can someone help me with a small issue I have in adapting/updating an institutional config file? Based on the queue (partition) that is selected, I want to either use the
--account
that is set through an env variable or use a different one. A small snippet of this is:
Copy code
// Define the queue closure
            queue = {
                task.memory >= 239.GB ?
                    (task.time >= 72.h ? 'dedicated_big_bigmem' : 'bigmem,hugemem') :
                    (task.time >= 72.h ? 'batch_long,batch_icelake_long,batch_sapphirerapids_long' : 'batch,batch_sapphirerapids,batch_icelake')
            }
			// Set clusterOptions, changing account based on queue
            clusterOptions = queue =~ /dedicated/ ? "--clusters=wice --account=lp_big_wice_cpu" : "--clusters=wice --account=$tier1_project"
So basically, if 'dedicated' is in the queue name, it should take
lp_big_wice_cpu
else it should take the
$tier1_project
variable. However when I try this, even if my queue contains 'dedicated' it always takes the latter option. Can someone spot a clear mistake in this or am I misunderstanding something perhaps? For example in a
.command.run
script I can see it takes (lp_lvibccbeu is the env variable account):
Copy code
#SBATCH -p dedicated_big_bigmem
#SBATCH --clusters=wice --account=lp_lvibccbeu
p
Your
queue
is uncontroversial. As for
clusterOptions
. If you don't make it a closure somehow, it will only be called once (whenever the file is parsed, I assume). I don't know (and can't be bothered to check myself right now) if accessing other directives (
queue
). work like this. I have a fairly strong preference for putting the decision tree in a function and having the closure being only a call with the desired inputs. And if the access works as one could possibly expect, I believe you'd see a closure rather than the result of the evaluation, you likely need to call the closure to get the result.
l
Hmm not sure if I fully understand (still relatively new with groovy etc), but using
clusterOptions
as a closure would make it so that it is dynamically evaluated. So this 'should' work:
Copy code
clusterOptions = { 
                queue =~ /dedicated/ ? "--clusters=wice --account=lp_big_wice_cpu" : "--clusters=wice --account=$tier1_project" 
            }
However, it ends up giving me the same it gives me the option for the False evaluation even if the queue contains 'dedicated'..
p
Assuming you can access
queue
at all, I would expect it to yield the closure. To get the value you'd need to evaluate it if that's what's happening.
l
Ah I see. I'll have another look tomorrow, thanks for the suggestions!
t
Have you resolved the issue? If not, have you tried being more explicit when referring to
queue
(I think I had to do this for other things inside closures)? Try
... *process*.queue =~ /dedicated/ ? ...
.
l
Hi Thomas, Thanks for the suggestion. I did not resolve it yet and just got back working on it after the holidays. Your suggestion of being more explicit does using
process.queue
does not seem to change anything. It just seems that my expression
Copy code
process.queue =~ /dedicated/ ? "--clusters=wice --account=lp_big_wice_cpu" : "--clusters=wice --account=$tier1_project"
Always returns False. Even though I can see in the
.command.run
file the final partition contains
dedicated
.
Copy code
#SBATCH -p dedicated_big_bigmem
I'm not familiar with groovy at all so maybe I am simply doing something wrong with the expression?
p
If you are still having issues with that, I'd recommend restructuring so the issue with evaluating
queue
never appears.
t
I have no experience with regex expression in Nextflow/Groovy, but I don't see a problem with it. Could it be that this is evaluated before the
queue
is set (e.g. statically, rather than dynamically)? You could see if you can debug this with print statements, or somethings, or try to find some work-around like Pontus suggests. If you can nail this down to some regex-evaluation issue that is reproducible (maybe even with things other than
queue
), it moves from being a #CGRTMASKY issues to something more general, and you might get more attention to that over in the Nextflow Slack, e.g. in #nextflow-help. 🤞
l
Managed to get it to work by restructuring some bits. Basically just re-evaluated the
queue
value within the
clusterOptions{}
enclosure and then it works! Thanks for all the suggestions everyone!
👍 1