https://nf-co.re logo
Join Slack
Powered by
Hello, first I'd like to thank you all for the awe...
# eager
a
Hello, first I'd like to thank you all for the awesome pipeline. Second, I need some help. I am trying to run eager on a small subset of our data and I'm getting the following error:
Copy code
Mar-24 16:01:16.033 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:mc_small` matches labels `mc_small` for process with name fastp
Mar-24 16:01:16.034 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Mar-24 16:01:16.034 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Mar-24 16:01:16.042 [Actor Thread 16] ERROR nextflow.processor.TaskProcessor - Error executing process > 'makeBT2Index (GCF_003254395.2_Amel_HAv3.1_genomic.fna)'

Caused by:
  No signature of method: Script_ee11debf.check_max() is applicable for argument types: (Integer, String) values: [8, cpus] -- Check script '/ceph/hpc/home/marinca/.nextflow/assets/nf-core/eager/main.nf' at line: 541

Source block:
  """
      bowtie2-build --threads ${task.cpus} $fasta $fasta
      mkdir BT2Index && mv ${fasta}* BT2Index
      """

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
Mar-24 16:01:16.047 [Actor Thread 16] INFO  nextflow.Session - Execution cancelled -- Finishing pending tasks before exit
The runner slurm job keeps running, but does not submit any new jobs. 
scancel
has no effect and I have to log in the compute node and manually kill the process. I'm using singularity, the cluster runs slurm and I've made some changes to the nextflow config; I changed the number of cores per attempt in the 
withLabel
block. Reason being, that I want to get the test run into the 
cpu
slurm partition where queue times are lower on our cluster. If the ratio of cores to memory is too low the cluster complains. I took 
eager/conf/base.config
and modified it. The 
nextflow.config
file in my project dir now looks like this:
Copy code
profiles {
  slurm {
    executor {
      name = 'slurm'
      submitRateLimit = '190/60min'
    }
    
    process {
      // Vega specific
      queue = 'cpu'	
      container = 'eager/eager_2.5.3.sif'

      // Modified eager base config 
      cpus = { check_max( 7 * task.attempt, 'cpus' ) }
      memory = { check_max( 7.GB * task.attempt, 'memory' ) }
      time = { check_max( 24.h * task.attempt, 'time' ) }

      errorStrategy = { task.exitStatus in [143,137,104,134,139, 140] ? 'retry' : 'finish' }
      maxRetries = 3
      maxErrors = '-1'

      // Process-specific resource requirements
      // NOTE - Only one of the labels below are used in the fastqc process in the main script.
      //        If possible, it would be nice to keep the same label naming convention when
      //        adding in your processes.
      // See <https://www.nextflow.io/docs/latest/config.html#config-process-selectors>

      // Generic resource requirements - s(ingle)c(ore)/m(ulti)c(ore)

      withLabel: 'sc_tiny' {
        // queue = 'largemem'
        cpus   = { check_max( 1 * task.attempt, 'cpus' ) }
        memory = { check_max( 1.GB * task.attempt, 'memory' ) }
        time   = { check_max( 4.h * task.attempt, 'time' ) }
      }

      withLabel: 'sc_small' {
        // queue = 'largemem'
        cpus   = { check_max( 4 * task.attempt, 'cpus' ) }
        memory = { check_max( 4.GB * task.attempt, 'memory' ) }
        time   = { check_max( 4.h * task.attempt, 'time' ) }
      }

      withLabel: 'sc_medium' {
        // queue = 'largemem'
        cpus   = { check_max( 8 * task.attempt, 'cpus' ) }
        memory = { check_max( 8.GB * task.attempt, 'memory' ) }
        time   = { check_max( 4.h * task.attempt, 'time' ) }
      }

      withLabel: 'mc_small' {
        cpus   = { check_max( 4 * task.attempt, 'cpus' ) }
        memory = { check_max( 4.GB * task.attempt, 'memory' ) }
        time   = { check_max( 4.h * task.attempt, 'time'   ) }
      }

      withLabel: 'mc_medium' {
        cpus   = { check_max( 8 * task.attempt, 'cpus'   ) }
        memory = { check_max( 8.GB * task.attempt, 'memory' ) }
        time   = { check_max( 4.h * task.attempt, 'time' ) }
      }

      withLabel: 'mc_large' {
        cpus   = { check_max( 16 * task.attempt, 'cpus' ) }
        memory = { check_max( 16.GB * task.attempt, 'memory' ) }
        time   = { check_max( 4.h * task.attempt, 'time'   ) }
      }

      // This would require 256 cores. We can save resources (billing) if we 
      // run this on largemem partition.
      withLabel: 'mc_huge' {
        queue = 'largemem'
        cpus   = { check_max( 32 * task.attempt, 'cpus' ) }
        memory = { check_max( 256.GB * task.attempt,'memory' ) }
        time   = { check_max( 4.h   * task.attempt, 'time' ) }
      }

      // Process-specific resource requirements (others leave at default, e.g. Fastqc)
      withName:get_software_versions {
        cache = false
      }

      withName:qualimap{
        errorStrategy = { task.exitStatus in [1,143,137,104,134,139, 140] ? 'retry' : task.exitStatus in [255] ? 'ignore' : 'finish' }
      }

      withName:preseq {
        errorStrategy = 'ignore'
      }

      withName:damageprofiler {
        errorStrategy = { task.exitStatus in [1,143,137,104,134,139, 140] ? 'retry' : 'finish' }
      }

      // Add 1 retry for certain java tools as not enough heap space java errors gives exit code 1
      withName: dedup {
        errorStrategy = { task.exitStatus in [1,143,137,104,134,139, 140] ? 'retry' : 'finish' } 
      }
  
      withName: markduplicates {
        errorStrategy = { task.exitStatus in [143,137, 140] ? 'retry' : 'finish' } 
      }

      // Add 1 retry as not enough heapspace java error gives exit code 1
      withName: malt {
        errorStrategy = { task.exitStatus in [1,143,137,104,134,139, 140] ? 'retry' : 'finish' } 
      }

      // other process specific exit statuses
      withName: nuclear_contamination {
        errorStrategy = { task.exitStatus in [143,137,104,134,139, 140] ? 'ignore' : 'retry' }
      }

      // Custom settigns for adapter removal ahead.
      // Adapter removal likes to run for more than 4 hours a lot of the time. 
      // Resubmissions are sometimes sketchy.
      //withName: 'adapter_removal' {
      //  time = { check_max( Math.min(8.h * task.attempt, 24.h), 'time' ) }
      //}
    } 
  } 
}

params {
  // Defaults only, expecting to be overwritten
  // Override spcific to Vega starts here.
  max_memory = 256.GB
  max_cpus = 128
  max_time = 48.h
  igenomes_base = '<s3://ngi-igenomes/igenomes/>'
}
I'd be grateful for any pointers on how to proceed. Probably something is wrong with the way I'm trying to change the amount of resources each process gets depending on their label, however, I have no idea what is wrong.
🎉 1
t
Hi Andraz! I believe the issue is with the 
check_max
function, which does not get loaded from one config to another. If you are specifying a custom config that uses it, you also need to define the function within that config. See: https://github.com/nf-core/configs/blob/master/conf%2Fpipeline%2Feager%2Feva.config#L627-L659 This is overall a deprecated way to do this, since nextflow now lets you define max resources more cleanly, but nf-core/eager is still using the old syntax till version 3.0.0 releases :)
a
Thanks for a quick reply. I see, so I just need to copy the function definition to my config. Will try that. Thanks again.
t
No problem!
a
I've now added the 
check_max
function to my config. I'm getting a new error:
Copy code
Mar-24 17:05:11.887 [Actor Thread 12] ERROR nextflow.processor.TaskProcessor - Error executing process > 'makeSeqDict (GCF_003254395.2_Amel_HAv3.1_genomic.fna)'

Caused by:
  Cannot invoke method toMega() on null object -- Check script '/ceph/hpc/home/marinca/.nextflow/assets/nf-core/eager/main.nf' at line: 624

Source block:
  """
      picard -Xmx${task.memory.toMega()}M CreateSequenceDictionary R=$fasta O="${fasta.baseName}.dict"
      """

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
I'm looking at 
<http://main.nf|main.nf>
and it seems like my config should assign 8GB RAM to the task: 
<http://main.nf|main.nf>
:
Copy code
process makeSeqDict {
    label 'sc_medium'
    tag "${fasta}"
    publishDir path: "${params.outdir}/reference_genome/seq_dict", mode: params.publish_dir_mode, saveAs: { filename -> 
            if (params.save_reference) filename 
            else if(!params.save_reference && filename == "where_are_my_files.txt") filename
            else null
    }
    
    when: !params.seq_dict && params.fasta

    input:
    path fasta from ch_fasta_for_seqdict
    path where_are_my_files

    output:
    path "*.dict" into ch_seq_dict
    path "where_are_my_files.txt"

    script:
    """
    picard -Xmx${task.memory.toMega()}M CreateSequenceDictionary R=$fasta O="${fasta.baseName}.dict"
    """
}
Snippet of my config:
Copy code
withLabel: 'sc_medium' {
        // queue = 'largemem'
        cpus   = { check_max( 8 * task.attempt, 'cpus' ) }
        memory = { check_max( 8.GB * task.attempt, 'memory' ) }
        time   = { check_max( 4.h * task.attempt, 'time' ) }
      }
The command being run:
Copy code
conda run -n nextflow_22 nextflow -log "eager_${dt}.log" run -work-dir "work_test" nf-core/eager \
    -r 2.5.3 \
    -profile singularity,slurm \
    -resume \
    -with-trace \
    --outdir ../data/processed/test/eager/ 
    *** further eager parameters omitted here***
I suspect I'm using a nextflow config layering anti-pattern of some sort. Edit: Now I've also tried adding 
-c nextflow.config
to the command. The error persists.
j
I don't see anything obvious 🤔
Unless 
check_max
has not be added properly (so the function isn't found)
This is how we added it for the EVA config
a
Copy code
// Function to ensure that resource requirements don't go beyond
// a maximum limit
// FOR DSL1 PIPELINE ONLY!
def check_max(obj, type) {
    if (type == 'memory') {
        try {
            if (obj.compareTo(params.max_memory as nextflow.util.MemoryUnit) == 1)
                return params.max_memory as nextflow.util.MemoryUnit
            else
                return obj
        } catch (all) {
            println "   ### ERROR ###   Max memory '${params.max_memory}' is not valid! Using default value: $obj"
            return obj
        }
    } else if (type == 'time') {
        try {
            if (obj.compareTo(params.max_time as nextflow.util.Duration) == 1)
                return params.max_time as nextflow.util.Duration
            else
                return obj
        } catch (all) {
            println "   ### ERROR ###   Max time '${params.max_time}' is not valid! Using default value: $obj"
            return obj
        }
    } else if (type == 'cpus') {
        try {
            return Math.min( obj, params.max_cpus as int )
        } catch (all) {
            println "   ### ERROR ###   Max cpus '${params.max_cpus}' is not valid! Using default value: $obj"
            return obj
        }
    }
}

profiles {
  slurm {
     *** same as prevously *** 
  }
}
This is how I've added it to my config. I pretty much jost copy-pasted it at the top of my config file.
j
To confirm: completely outside all other scopes ?
a
Indeed, outside all scopes.
j
I'm not sure then... what happens if you remove the 
check_max
for the medium albel memory thingy?
a
Ok, thanks, I'll report back when it runs.
I've replaced check_max call with a fixed memory value like so:
Copy code
withLabel: 'sc_medium' {
        // queue  = 'largemem'
        cpus      = { check_max( 8 * task.attempt, 'cpus' ) }
        // memory = { check_max( 8.GB * task.attempt, 'memory' ) }
        memory    = 24.GB
        time      = { check_max( 4.h * task.attempt, 'time' ) }
      }
Now adapter removal fails.
Copy code
Mar-25 10:18:05.121 [Actor Thread 5] ERROR nextflow.processor.TaskProcessor - Error executing process > 'adapter_removal (SRR14703781_L1)'

Caused by:
  Cannot invoke method toGiga() on null object -- Check script '/ceph/hpc/home/marinca/.nextflow/assets/nf-core/eager/main.nf' at line: 838

Source block:
  def base = "${r1.baseName}_L${lane}"
  def adapters_to_remove = !params.clip_adapters_list ? "--adapter1 ${params.clip_forward_adaptor} --adapter2 ${params.clip_reverse_adaptor}" : "--adapter-list ${adapterlist}"
  def preserve5p = params.preserve5p ? '--preserve5p' : ''
  if ( seqtype == 'PE'  && !params.skip_collapse && !params.skip_trim  && !params.mergedonly && !params.preserve5p ) {
      """
      mkdir -p output
  
      AdapterRemoval --file1 ${r1} --file2 ${r2} --basename ${base}.pe --gzip --threads ${task.cpus} --qualitymax ${params.qualitymax} --collapse ${preserve5p} --trimns --trimqualities ${adapters_to_remove} --minlength ${params.clip_readlength} --minquality ${params.clip_min_read_quality} --minadapteroverlap ${params.min_adap_overlap}
  
      cat *.collapsed.gz *.collapsed.truncated.gz *.singleton.truncated.gz *.pair1.truncated.gz *.pair2.truncated.gz > output/${base}.pe.combined.tmp.fq.gz
      
      mv *.settings output/
  
      ## Add R_ and L_ for unmerged reads for DeDup compatibility
      AdapterRemovalFixPrefix -Xmx${task.memory.toGiga()}g output/${base}.pe.combined.tmp.fq.gz | pigz -p ${task.cpus - 1} > output/${base}.pe.combined.fq.gz
      
      """
      //PE mode, collapse and trim, outputting all reads, preserving 5p
      } else if (seqtype == 'PE'  && !params.skip_collapse && !params.skip_trim  && !params.mergedonly && params.preserve5p) {
      """
      mkdir -p output
  
      AdapterRemoval --file1 ${r1} --file2 ${r2} --basename ${base}.pe --gzip --threads ${task.cpus} --qualitymax ${params.qualitymax} --collapse ${preserve5p} --trimns --trimqualities ${adapters_to_remove} --minlength ${params.clip_readlength} --minquality ${params.clip_min_read_quality} --minadapteroverlap ${params.min_adap_overlap}
  
      cat *.collapsed.gz *.singleton.truncated.gz *.pair1.truncated.gz *.pair2.truncated.gz > output/${base}.pe.combined.tmp.fq.gz
  
      mv *.settings output/
  
      ## Add R_ and L_ for unmerged reads for DeDup compatibility
      AdapterRemovalFixPrefix -Xmx${task.memory.toGiga()}g output/${base}.pe.combined.tmp.fq.gz | pigz -p ${task.cpus - 1} > output/${base}.pe.combined.fq.gz
  
      """
      // PE mode, collapse and trim but only output collapsed reads
      } else if ( seqtype == 'PE'  && !params.skip_collapse && !params.skip_trim && params.mergedonly && !params.preserve5p ) {
      """
      mkdir -p output
      AdapterRemoval --file1 ${r1} --file2 ${r2} --basename ${base}.pe  --gzip --threads ${task.cpus} --qualitymax ${params.qualitymax} --collapse ${preserve5p} --trimns --trimqualities ${adapters_to_remove} --minlength ${params.clip_readlength} --minquality ${params.clip_min_read_quality} --minadapteroverlap ${params.min_adap_overlap}
      
      cat *.collapsed.gz *.collapsed.truncated.gz > output/${base}.pe.combined.tmp.fq.gz
          
      ## Add R_ and L_ for unmerged reads for DeDup compatibility
      AdapterRemovalFixPrefix -Xmx${task.memory.toGiga()}g output/${base}.pe.combined.tmp.fq.gz | pigz -p ${task.cpus - 1} > output/${base}.pe.combined.fq.gz
  
      mv *.settings output/
      """
      // PE mode, collapse and trim but only output collapsed reads, preserving 5p
      } else if ( seqtype == 'PE'  && !params.skip_collapse && !params.skip_trim && params.mergedonly && params.preserve5p ) {
      """
      mkdir -p output
      AdapterRemoval --file1 ${r1} --file2 ${r2} --basename ${base}.pe  --gzip --threads ${task.cpus} --qualitymax ${params.qualitymax} --collapse ${preserve5p} --trimns --trimqualities ${adapters_to_remove} --minlength ${params.clip_readlength} --minquality ${params.clip_min_read_quality} --minadapteroverlap ${params.min_adap_overlap}
      
      cat *.collapsed.gz > output/${base}.pe.combined.tmp.fq.gz
      
      ## Add R_ and L_ for unmerged reads for DeDup compatibility
      AdapterRemovalFixPrefix -Xmx${task.memory.toGiga()}g output/${base}.pe.combined.tmp.fq.gz | pigz -p ${task.cpus - 1} > output/${base}.pe.combined.fq.gz
  
      mv *.settings output/
      """
      // PE mode, collapsing but skip trim, (output all reads). Note: seems to still generate `truncated` files for some reason, so merging for safety.
      // Will still do default AR length filtering I guess
      } else if ( seqtype == 'PE'  && !params.skip_collapse && params.skip_trim && !params.mergedonly ) {
      """
      mkdir -p output
      AdapterRemoval --file1 ${r1} --file2 ${r2} --basename ${base}.pe --gzip --threads ${task.cpus} --qualitymax ${params.qualitymax} --collapse ${preserve5p} --adapter1 "" --adapter2 ""
      
      cat *.collapsed.gz *.pair1.truncated.gz *.pair2.truncated.gz > output/${base}.pe.combined.tmp.fq.gz
          
      ## Add R_ and L_ for unmerged reads for DeDup compatibility
      AdapterRemovalFixPrefix -Xmx${task.memory.toGiga()}g output/${base}.pe.combined.tmp.fq.gz | pigz -p ${task.cpus - 1} > output/${base}.pe.combined.fq.gz
  
      mv *.settings output/
      """
      // PE mode, collapsing but skip trim, and only output collapsed reads. Note: seems to still generate `truncated` files for some reason, so merging for safety.
      // Will still do default AR length filtering I guess
      } else if ( seqtype == 'PE'  && !params.skip_collapse && params.skip_trim && params.mergedonly ) {
      """
      mkdir -p output
      AdapterRemoval --file1 ${r1} --file2 ${r2} --basename ${base}.pe --gzip --threads ${task.cpus} --qualitymax ${params.qualitymax} --collapse ${preserve5p}  --adapter1 "" --adapter2 ""
      
      cat *.collapsed.gz > output/${base}.pe.combined.tmp.fq.gz
      
      ## Add R_ and L_ for unmerged reads for DeDup compatibility
      AdapterRemovalFixPrefix -Xmx${task.memory.toGiga()}g output/${base}.pe.combined.tmp.fq.gz | pigz -p ${task.cpus - 1} > output/${base}.pe.combined.fq.gz
  
      mv *.settings output/
      """
      // PE mode, skip collapsing but trim (output all reads, as merging not possible) - activates paired-end mapping!
      } else if ( seqtype == 'PE'  && params.skip_collapse && !params.skip_trim ) {
      """
      mkdir -p output
      AdapterRemoval --file1 ${r1} --file2 ${r2} --basename ${base}.pe --gzip --threads ${task.cpus} --qualitymax ${params.qualitymax} ${preserve5p} --trimns --trimqualities ${adapters_to_remove} --minlength ${params.clip_readlength} --minquality ${params.clip_min_read_quality} --minadapteroverlap ${params.min_adap_overlap}
      
      mv ${base}.pe.pair*.truncated.gz *.settings output/
      """
      } else if ( seqtype != 'PE' && !params.skip_trim ) {
      //SE, collapse not possible, trim reads only
      """
      mkdir -p output
      AdapterRemoval --file1 ${r1} --basename ${base}.se --gzip --threads ${task.cpus} --qualitymax ${params.qualitymax} ${preserve5p} --trimns --trimqualities ${adapters_to_remove} --minlength ${params.clip_readlength} --minquality ${params.clip_min_read_quality} --minadapteroverlap ${params.min_adap_overlap}
      mv *.settings *.se.truncated.gz output/
      """
      } else if ( seqtype != 'PE' && params.skip_trim ) {
      //SE, collapse not possible, trim reads only
      """
      mkdir -p output
      AdapterRemoval --file1 ${r1} --basename ${base}.se --gzip --threads ${task.cpus} --qualitymax ${params.qualitymax} ${preserve5p} --adapter1 "" --adapter2 ""
      mv *.settings *.se.truncated.gz output/
      """
      }

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
Mar-25 10:18:05.125 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:sc_tiny` matches labels `sc_tiny` for process with name lanemerge
Mar-25 10:18:05.164 [Actor Thread 12] DEBUG nextflow.util.CacheHelper - [WARN] Unknown hashing type: class nextflow.util.MemoryUnit
j
Ugh i guess it's because smae issue but with the adapteremoval lable which is differnet
So for some reason 
check_max()
isn't working in this context, but I'm not sure why 😕
Maybe @Thiseas C. Lamnidis has more ideas/
I'm assuming you're runing nextflow 22.10 or something @Andraz Marinč?
a
It's version 22.10.6.
j
YEah, as it should be. I'm honestly a bit confused why the check_max thing isn't being propgated from the root 
nextflow.confg
a
It seems I managed to set things up in a completely novel and unreasonably broken way. 😂
j
Not necessaily, I seem to remember going through the same process when I faffed around the the 
eva.config
but that was a while ago and I'm a bit stuck now
I guess you COULD try and host it remotely on a github repo and use 
--custom_config_base
to test it 🤔
a
If I understand correctly, I should host my custom config on github and then use --custom_config_base to pull it from there? Will try that.
This part of the log looks normal, right?
Copy code
Mar-25 10:17:54.230 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /ceph/hpc/home/marinca/.nextflow/assets/nf-core/eager/nextflow.config
Mar-25 10:17:54.247 [main] DEBUG nextflow.config.ConfigBuilder - User config file: /ceph/hpc/data/s26o01-10-users/MuseumSamples/scripts/nextflow_checkmax_test.config
Mar-25 10:17:54.248 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /ceph/hpc/home/marinca/.nextflow/assets/nf-core/eager/nextflow.config
Mar-25 10:17:54.248 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /ceph/hpc/data/s26o01-10-users/MuseumSamples/scripts/nextflow_checkmax_test.config
Mar-25 10:17:54.268 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `singularity,slurm`
Mar-25 10:17:55.635 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `singularity,slurm`
Mar-25 10:17:55.938 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [bih, test_tsv_fna, cfc_dev, uzl_omics, ifb_core, embl_hd, denbi_qbic, alice, mjolnir_globe, uppmax, giga, incliva, ilifu, ki_luria, uge, icr_alma, rosalind_uge, lugh, mccleary, unibe_ibu, vai, test_tsv_kraken, czbiohub_aws, jax, roslin, iris, tes, scw, unc_longleaf, tigem, tubingen_apg, apollo, params, ipop_up, vsc_calcua, pdc_kth, ceci_nic5, ccga_cau, humantechnopole, stjude, daisybio, eddie, medair, biowulf, mahuika, test_direct, bi, bigpurple, adcra, test_bam, cedars, pawsey_setonix, vsc_kul_uhasselt, pawsey_nimbus, rki, ucl_myriad, utd_ganymede, charliecloud, fred_hutch, seattlechildrens, icr_davros, test_tsv_humanbam, ceres, test_kraken, munin, rosalind, hasta, cfc, uzh, shu_bmrc, ebi_codon_slurm, ebc, ccga_dx, crick, ku_sund_danhead, marvin, lrz_cm4, biohpc_gen, shifter, mana, york_viking, unc_lccc, wehi, awsbatch, wustl_htcf, arcc, ceci_dragon2, imperial, maestro, test_tsv_bam, software_license, cannon, genotoul, nci_gadi, abims, eva_grace, janelia, nu_genomics, googlebatch, oist, sahmri, benchmarking_human, kaust, alliance_canada, mpcdf, test_stresstest_human, leicester, vsc_ugent, create, sage, cambridge, jex, podman, ebi_codon, cheaha, xanadu, nyu_hpc, test, marjorie, computerome, ucd_sonic, seg_globe, mssm, sanger, dkfz, bluebear, pasteur, einstein, ethz_euler, m3c, test_full, imb, ucl_cscluster, tuos_stanage, test_fna, azurebatch, test_pretrim, seadragon, crukmi, csiro_petrichor, qmul_apocrita, docker, engaging, gis, hypatia, psmn, eva, unity, cropdiversityhpc, benchmarking_vikingfish, nygc, test_humanbam, fgcz, conda, crg, singularity, mpcdf_viper, dirac, self_hosted_runner, tufts, uw_hyak_pedslabs, binac2, hki_genie, debug, genouest, cbe, test_tsv_pretrim, unsw_katana, utd_juno, test_tsv_complex, slurm, phoenix, seawulf, uod_hpc, fub_curta, lovelace, uct_hpc, aws_tower, fsu_draco]
Mar-25 10:17:56.069 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=1 from script declararion
j
Yeah I think so.
t
hmmm. this is odd indeed. are you setting the params 
max_memory
, 
max_cpus
, 
max_time
at all? Could it be that those params are not set correctly? I am wondering if maybe the comparisons within 
check_max
end up returning the max, and that is not a correctly typed object or sth 🤔
👀 1
a
@Thiseas C. Lamnidis I'm embarrassed to report that I screwed up the config file when I copied the check_max definition. In the config I first posed I had the params defined at the top level. However, when I copied the 
check_max
function to the file, I accidentally moved the 
params
block one level, to the profiles scope (it was on the same level as 
slurm
profile). I suppose this created a new profile called "params". The nesting then looked like this:
Copy code
def check_max(obj, type) {
  *** check_max code ***
}

profiles {
  slurm {
    *** slurm profile code ***
  }

  params {
    // Defaults only, expecting to be overwritten
    // Override spcific to Vega starts here.
    max_memory = 256.GB
    max_cpus = 128
    max_time = 24.h
    igenomes_base = '<s3://ngi-igenomes/igenomes/>'
  }
}
I now moved the params block back to the outermost scope and the workflow is currently running with no errors reported so far. Sorry to have wasted your time with this. 🫠 I still don't understand how this produced the error, though.. Shouldn't the misplaced params block be ignored? Anyway, thanks for your help everyone.
t
good to hear you found the solution! 😃
2 Views
Open in Slack
PreviousNext
Powered by