https://nf-co.re logo
Join Slack
Powered by
I have a short question. I have both CAT and GTDBT...
# mag
c
I have a short question. I have both CAT and GTDBTk set up on a local 
/scratch
space on one of our worker nodes on our cluster (this is local to the node, not visible from the cluster file system). Our worker nodes do not have network access but we’re launching 
mag
from a login node we have set up for workflows which does (for Singularity, etc). We’re seeing validation issues, likely because the I’m guessing validation is occurring in the main nextflow job on the login node, where it can’t see the 
/scratch
space. Any ideas?
I may try offline mode but wanted to see if this has popped up before.
I also noticed the warnings:
Copy code
$ nextflow run nf-core/mag -r 5.4.2 -profile singularity -c config.json -params-file params.mag.yml -qs 6 -with-report -with-trace -resume

 N E X T F L O W   ~  version 26.04.3

Launching `<https://github.com/nf-core/mag>` [voluminous_dubinsky] revision: 5dabb0159a [5.4.2]

WARN: Unrecognized config option 'validation.defaultIgnoreParams'
WARN: Unrecognized config option 'validation.monochromeLogs'

------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/mag 5.4.2
------------------------------------------------------

Input/output options
  input                     : samplesheet.csv
  outdir                    : results                                                                                                                                                                                    email                     : <mailto:cjfields@illinois.edu|cjfields@illinois.edu>

Generic options
  trace_report_suffix       : 2026-06-16_00-07-29
...
Core Nextflow options
  revision                  : 5.4.2
  runName                   : voluminous_dubinsky
  containerEngine           : singularity
  launchDir                 : XXXX
  workDir                   : XXXX
  projectDir                : XXXX
  userName                  : cjfields
  profile                   : singularity
  configFiles               : /home/groups/hpcbio/apps/nextflow/nextflow-26.04.3/.nextflow/assets/.repos/nf-core/mag/clones/5dabb0159ac0104885e09f301db22126e8fcb394/nextflow.config, /home/groups/hpcbio/projects/vddonk/2026-June-Metagenomes/results/2026-06-11-mag/config.json

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------

* The pipeline
    <https://doi.org/10.1093/nargab/lqac007>

* The nf-core framework
    <https://doi.org/10.1038/s41587-020-0439-x>

* Software dependencies
    <https://github.com/nf-core/mag/blob/main/CITATIONS.md>

ERROR ~ Validation of pipeline parameters failed!

 -- Check '.nextflow.log' file for details
The following invalid input values have been detected:

* --cat_db (/scratch/CAT/20241212_CAT_nr_website/): the file or directory '/scratch/CAT/20241212_CAT_nr_website/' does not exist
j
We’re seeing validation issues, likely because the I’m guessing validation is occurring in the main nextflow job on the login node, where it can’t see the /scratch space
I think this is the most likely cause, yes
I think offline and submitting your command as a job is the only way here :
😕
d
Maybe we should probably update the url validations to make it possible to run mag totally offline
c
Yeah I was worried about that. We can try the offline workaround
@Diego Alvarez Saravia that would def be needed, still quite a few clusters like ours which firewall outside network on worker nodes.
d
I'm thinking on a new parameter, kinda 
--skip-path-validations
or even more direct: 
--offline-mode
. How do you manage this in other pipelines @James Fellows Yates?
c
In my opinion those are two different issues. For example I can run Nextflow from our custom login node (online), it just can’t see the staged databases on the worker node (path validation fails). Running offline would essentially be a workaround to the path validation issue
I did notice the path issue bit someone who stages their databases on AWS
j
On the move but: maybe try 
validation { ignoreParams = [param_to_ingore]}
?
I can't remember the exact syntax but we already do it a lot on nf-core/cpnfigs
@Nicolas Vannieuwkerke help if you are here
c
I’ll test, but note the warnings in the above:
Copy code
WARN: Unrecognized config option 'validation.defaultIgnoreParams'
WARN: Unrecognized config option 'validation.monochromeLogs'
That was using nextflow 26.04.3
n
yes 
validation.ignoreParams = [<list of params>]
is the correct syntax
The warnings are a known issue, I need to find the time to fix them properly 😁
c
Okay, that bypasses the validation step, but it fails with gtdb:
Copy code
* Software dependencies
    <https://github.com/nf-core/mag/blob/main/CITATIONS.md>

ERROR ~ No such file or directory: /scratch/gtdbtk/release226

-- Check script '/home/groups/hpcbio/apps/nextflow/nextflow-26.04.3/.nextflow/assets/.repos/nf-core/mag/clones/5dabb0159ac0104885e09f301db22126e8fcb394/workflows/mag.nf' at line: 113 or see '.nextflow.log' file for more details
Different error but likely related to the same problem
(added the traceback)
n
Maybe you should just disable validation in general 😅 
--validate_params false
c
Like so?
Copy code
nextflow run nf-core/mag -r 5.4.2 -profile singularity -c config.json -params-file params.mag.yml -qs 6 -with-report -with-trace -resume --validate_params false
Doesn’t work:
Copy code
ERROR ~ Validation of pipeline parameters failed!

 -- Check '.nextflow.log' file for details
The following invalid input values have been detected:

* --validate_params (false): Value is [string] but should be [boolean]

 -- Check script '/home/groups/hpcbio/apps/nextflow/nextflow-26.04.3/.nextflow/assets/.repos/nf-core/mag/clones/5dabb0159ac0104885e09f301db22126e8fcb394/subworkflows/nf-core/utils_nfschema_plugin/main.nf' at line: 68 or see '.nextflow.log' file for more details
n
can you try via a parameters file? This is an 'issue' in the latest Nextflow version
c
Yes, that works but still ends up with the above failure, which is noticably different (not a validation problem but a literal missing file)
I may try moving gtdb back to the shared file system. But uncompressed that is about 140GB (CAT uncompressed is ~300GB).
Tons of small files on our shared file system (GPFS) are an issue hence caching it to a local node scratch space, but this is generally a problem across any NFS
So there are a couple of overlapping issues. Moving GTDBTk to the NFS works in general (bypasses the above issue), but then the workflow fails later when it can’t find CAT:
Copy code
ERROR ~ No such file or directory: /scratch/CAT/20241212_CAT_nr_website

 -- Check '.nextflow.log' file for details
n
Yeah that seems like the only solution at the moment 🥲
c
Correct, but unfortunately not great, our campus cluster admins hate this approach since it causes I/O on the shared system to spike and affects other users. Localizing the data avoids this.
The other option is to point to the tar.gz, but this then requires untarring it for each job.
n
Would running the pipeline as a job on the same system as the processes not work?
c
Yes it should but has to be run offline (no network access). And it sounds like it may fail for other reasons
But will test
n
I'll leave it up to @James Fellows Yates to help you further on that 😛. Good luck!
j
I'm pretty sure a combination of offline mode plus skip validation should work?
(thanks though @Nicolas Vannieuwkerke!)
n
No problem, ping me if you need more help!
c
Thx @Nicolas Vannieuwkerke! Getting this setup for offline mode, will report back if there are issues 🤞
🎉 2
👍 1
So far offline is working fine with the combination of the two (bypassed all of the prior issues)! Hopefully there aren’t any downstream steps trying outside access; I had this issue w/ BUSCO v6 just recently
🤞 1
Just a note, I am seeing this warning in the run (
-r 5.4.2
):
Copy code
WARN: Access to undefined parameter `skip_tiara` -- Initialise it to a default value eg. `params.skip_tiara = some_value`
Funny enough I set it explicitly to false in the params and it went away 🤷
n
It might be some issue with the default of that parameter, not sure from the warning itself
j
Uhhh 
skip_tiara
doesn't exist at all in the code base?
If you want to skip tiara it should be: 
--bin_domain_classification false
(whic is default)
c
A couple of places in 
utils_nfcore_mag_pipeline
actually
Copy code
$ grep -r 'skip_tiara' .
./subworkflows/local/utils_nfcore_mag_pipeline/main.nf:        !params.skip_tiara ? text_tiara : "",
./subworkflows/local/utils_nfcore_mag_pipeline/main.nf:    if (!params.skip_tiara) {
Just an update though, it’s all running fine using offline mode. Might be something to note if the issue pops up for others
🙌 1
d
Yeah, that is already fixed on dev 😁. https://github.com/nf-core/mag/pull/1010/changes
👍 1
c
I’m being good and using 
-r 5.4.2
😉
d
Yeah, I know. We've been acumulating many fixes on dev now. 5.5 will be quite big 😅
👀 1
c
I did find that BUSCO v6 has a new ‘phone home’ feature you need to opt out of, even if you select 
--offline
(I’m guessing this is a bug and that it will eventually be fixed). That opt-out param is 
--opt-out-run-stats
. I think it should still finish if the network is available, but it does attempt connection several times before giving up
facepalm 1
Ran into this with another workflow, the Sanger ToL genomeassembly one
j
Could you make an issue about the BUSCO nonsense @Chris Fields?
👍 1
c
Just running it through to make sure it’s not an issue in this case. I noticed our local run didn’t have the BUSCO database set up for offline on the first pass, so they very slowly silently failed (but didn’t kill the workflow). Running it again to see if the ‘call home’ is still triggered with 
busco_db
set
Just follow this through, I didn’t see any issues with our local BUSCO run in 
mag
when it was set for a local database. The issue we encountered elsewhere was with the Sanger Tree of Life workflow for genome assembly, which took some effort to figure out.
2 Views
Open in Slack
PreviousNext
Powered by