Hi everyone! I’m currently running the

nf-core/mag

5.4.0 pipeline on a large-scale dataset (82 samples, ~1 TB). I encountered a blocking

null object

error during the

get_software_versions

/ MultiQC metadata collection step. It seems the pipeline crashes when trying to parse Bowtie2 versions/logs on this scale. I managed to bypass it by locally patching the code to skip the specific version check, allowing the binning processes to complete. Has anyone else experienced this bottleneck with Terabyte-scale runs? I’d be happy to share my logs or the specific lines I modified if it helps improving the pipeline's robustness!

If you could share the

.nextflow.log

file that would be much appreciated ♥️

Hi @James Fellows Yates. I started with a generic 'Could not find which method load() to invoke' error. [Actor Thread 56] DEBUG nextflow.Session - Session aborted -- Cause:objectERROR ~ Could not find which method load() to invoke from this list: public java.lang.Object org.yaml.snakeyaml.Yaml#load(java.io.InputStream) public java.lang.Object org.yaml.snakeyaml.Yaml#load(java.io.Reader) public java.lang.Object org.yaml.snakeyaml.Yaml#load(java.lang.String) public java.lang.Object org.yaml.snakeyaml.Yaml#load(java.io.File) public java.lang.Object org.yaml.snakeyaml.Yaml#load(java.nio.file.Path). After some debugging, I captured the underlying NullPointerException: 'Cannot invoke method getText() on null object' at

processVersionsFromYAML

. Attached is the most detailed log I have."

Aha, ok the same `could not find which method load()`e rror has been reported last week and this week a couple of times Unfortunately the context around the error is missing so it's hard to easily debug. I have a suspicion it might be related to the new module version system that's being rolled out at the moment But unfortunately I'm currently dealing with lots of non-pipeline prjenct deadlines and a sick kid at home so I can't dig deeper currently, and the current lead dev Diego is on holiday (I don't know if any of <!subteam^S08MYEKMNTX> has a bit of time to look into it, but everyone seems quiet atm) but otherwise we will need a bit of time to get around to to investigateong, so thank you in advance for your patience 🙏

Hi! I totally understand the situation and family/work priorities definitely come first. Please don't feel any pressure to reply to this now. Since I'm working on my thesis with a very large dataset (1 TB), I had to find a quick workaround to keep my run going. I'm sharing the few lines I added to

processVersionsFromYAML

just in case they might be a helpful starting point for the team whenever you'll have the chance to look into this. _I simply added a null check for

yaml_file

to prevent the crash. Wishing a quick recovery to the kid! Thanks again for all the work you do on nf-core/mag. line 82 from_ .nextflow/assets/nf-core/mag/subworkflows/nf-core/utils_nfcore_pipeline/main.nf // // Get software versions for pipeline // def processVersionsFromYAML(yaml_file) { def yaml = new org.yaml.snakeyaml.Yaml() if (!yaml_file) { println "Skipping null yaml_file" return "{}" } String yaml_text = yaml_file.getText('UTF-8') def raw_map = yaml.load(yaml_text) def clean_map = raw_map.collectEntries { k, v -> [k.tokenize(':')[-1], v] } return yaml.dump(clean_map).trim()