I'm trying to integrate all the framework code generated by

nf-core pipeline create

into my baby workflow. I'm getting a stack overflow. I tried debugging using these instructions and I could connect to the process but IntelliJ doesn't stop at my breakpoint. Via many log messages and gradual incorporation of changes, I've narrowed my problem down to the following: Having

params {
    validate_params            = true
}

in my nextflow.config. If I comment this line out, problem does not occur. Then, even if I pass

--validate_params false

on the command line. with setting confirmed by

Other parameters
...
  validate_params       : false

I've added log messages to subworkflows/nf-core/utils_nfschema_plugin/main.nf

<http://log.info|log.info>("Before validateParameters: validate_params=${validate_params}, parameters_schema=${parameters_schema}")
    if(validate_params) {
        validateOptions = [:]
        if(parameters_schema) {
            validateOptions << [parametersSchema: parameters_schema]
        }
        <http://log.info|log.info>("Before validateParameters")
        validateParameters(validateOptions)
        <http://log.info|log.info>("After validateParameters")
    }

And I get

Before validateParameters: validate_params=false, parameters_schema=null
Before validateParameters
ERROR ~ Unexpected error [StackOverflowError]

So it's weird that despite validate_params being false, the if statement is executed. Log file contains the stack trace, which is lots of

at groovy.json.DefaultJsonGenerator.writeIterator(DefaultJsonGenerator.java:416)
	at groovy.json.DefaultJsonGenerator.writeObject(DefaultJsonGenerator.java:206)
	at groovy.json.DefaultJsonGenerator.writeObject(DefaultJsonGenerator.java:168)
	at groovy.json.DefaultJsonGenerator.writeIterator(DefaultJsonGenerator.java:416)
	at groovy.json.DefaultJsonGenerator.writeObject(DefaultJsonGenerator.java:206)
	at groovy.json.DefaultJsonGenerator.writeObject(DefaultJsonGenerator.java:168)
	at groovy.json.DefaultJsonGenerator.writeIterator(DefaultJsonGenerator.java:416)

Given that nf-schema is a plugin that I guess is downloaded on demand, I don't even see how to debug with log message. Help!

what version of nf-core cli, nextflow and nf-schema?

N E X T F L O W ~ version 26.02.0-edge nf-schema: 2.6.1 (I don't have this configured anywhere but I see in .nextflow.log

{"plugins":[{"id":"nf-schema","releases":[{"version":"2.6.1"

nf-core 3.5.2

the nf-schema plugin version should be pinned in you

nextflow.config

but doesn't matter. could you try with a non-edge version (

NXF_EDGE=1 nextflow self-update

)

You are using typed params, right?

Ah, adding parameter type for

validate_params

causes

--validate_params false

to be respected. However if I leave it as the default of true, I still get the stack overflow. And IntelliJ doesn't work to set breakpoints.

Would you be able to create an MRE (minimal reproducible example)? I can take a look at it tomorrow

Maybe. I realize that the nextflow_schema.json that was created by

nf-core pipelines create

has lots of stuff that is not created by

nf-core pipelines schema build

. I'm going to try including all that gorp and see what happens.

the nextflow_schema.json comes from our template here https://github.com/nf-core/tools/blob/main/nf_core/pipeline-template/nextflow_schema.json

Hi @Nicolas Vannieuwkerke, ok, I've done some debugging. The problem is triggered by having a process with some

path

inputs:

process MAPMYCELLS_FROMSPECIFIEDMARKERS {
    label 'process_medium'
    label 'mapmycells'
    
    container 'us-docker.pkg.dev/mccarroll-scrna-seq/us.gcr.io/mapmycells:current'
    memory '16 GB'
    scratch true // use local disk for input files because sqlite is slow over gcsfuse
    stageInMode 'copy' // ensure files are copied to local scratch, rather than symlinked

    input:
        path query_markers_json
        path precomputed_stats_h5ad
        path dge_h5ad
        path gene_mapping
        val mmc_args
        val analysis_identifier

When I run with the debugger, I can see that the stack overflow is when precomputed_stats_h5ad value is being converted to json. If I change the process definition so that input param is no longer a path, then I get the stack overflow for the next path. Stack trace:

"main" #1 [4867] prio=5 os_prio=31 cpu=2580.15ms elapsed=143.14s tid=0x000000012c008200 nid=4867 at breakpoint [0x000000016b8d5000]
   java.lang.Thread.State: RUNNABLE
	at groovy.json.DefaultJsonGenerator.writeIterator(DefaultJsonGenerator.java:408)
	at groovy.json.DefaultJsonGenerator.writeObject(DefaultJsonGenerator.java:206)
	at groovy.json.DefaultJsonGenerator.writeObject(DefaultJsonGenerator.java:168)
	at groovy.json.DefaultJsonGenerator.toJson(DefaultJsonGenerator.java:102)
	at groovy.json.JsonOutput.toJson(JsonOutput.java:146)
	at nextflow.trace.ResourcesAggregator.renderSummaryJson(ResourcesAggregator.groovy:120)
	at nextflow.trace.ReportObserver.renderSummaryJson(ReportObserver.groovy:201)
	at nextflow.trace.ReportObserver.renderPayloadJson(ReportObserver.groovy:205)
	at nextflow.trace.ReportObserver.renderHtml(ReportObserver.groovy:216)
	at nextflow.trace.ReportObserver.onFlowComplete(ReportObserver.groovy:129)
	at nextflow.Session$_notifyFlowComplete_lambda38.doCall(Session.groovy:1082)
	at nextflow.Session$$Lambda/0x0000000800729a08.accept(Unknown Source)
	at nextflow.Session.notifyEvent(Session.groovy:1112)
	at nextflow.Session.notifyFlowComplete(Session.groovy:1082)
	at nextflow.Session.shutdown0(Session.groovy:779)
	at nextflow.Session.abort(Session.groovy:834)
	at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:139)
	at nextflow.cli.CmdRun.run(CmdRun.groovy:435)
	at nextflow.cli.Launcher.run(Launcher.groovy:515)
	at nextflow.cli.Launcher.main(Launcher.groovy:675)

Thanks for the debugging work! We unfortunately have to use

Path

objects for file path validation as these use the extended classes built-in to Nextflow

The bug is triggered by parameter validation in code created by

nf-core pipelines create

. It generates 160 files. I'm a nextflow novice so I don't think I'd be able to prune that down. If ~160 files is considered a small workflow, maybe I could do it. I'm not suggesting you don't use Path. Rather, I'm suggesting that Paths be converted to strings in

nextflow.trace.ResourcesAggregator.renderSummaryJson

. Also, I'm new to groovy. I don't know how to report bugs in the groovy library nor see if a bug has been fixed. It's obviously a groovy library bug,. Is nextflow using the latest groovy library? Maybe it's already been fixed.

I think I know how to fix this on our side, but yeah might be nice trying to fix this in groovy itself. I don't know how to submit a fix either though...

Thanks!