https://nf-co.re logo
Join Slack
Powered by
My nf-test fails with `java.lang.OutOfMemoryError:...
# nf-test
f
My nf-test fails with 
java.lang.OutOfMemoryError: Required array size too large
probably because I have too many files in my outputDir but when I comment out the 
stable_name
and just assert the 
stable_path
it works. Did anyone encounter this before and could give me hint for a good way forward because I want to assert the 
stable_name
part aswell.
shakefist 1
m
how many files do you get?
f
not sure but I load vep cache (downsized) and another database
but maybe its a weird bug related to nested filepaths or something?
m
oh, maybe don't select theses
these are input in any case, you don't really want to capture them as output
f
I tried but excluding this 
reference
folder that I create did not solve the issue
m
interesting, could you share the PR?
f
(but current state is not here)
m
could you push even current state?
f
The test data I only have locally atm
but yes
m
oh
ok, I'm AFK for a bit, I'll check when I'm back
n
snapshots have a maximum size allowed, this just means you went over that threshold, I can't find the limit anymore but it's somewhere
f
so maximum of lines?
but even if I exclude the 
references
folder I run into this issue and without that folder I just have a handfull of files as far as I can tell 😞
m
It was 255 elements to snapshots if I remember well
party this 1
n
Maximum amounf of items
m
But using an array as we do with this function bypass the limit
We have 800+ files per snapshots in rnaseq
f
this needs to be noted down somewhere haha
ok 
stable_name
does not work even if I use 
.nftignore
but 
stable_path
works
so getting the md5sum works but getting the dir paths does not?
that seems like a bug to me
Copy code
then {
            // stable_name: All files + folders in ${params.outdir}/ with a stable name
            def stable_name = getAllFilesFromDir(params.outdir, relative: true, includeDir: true, ignoreFile: 'tests/.nftignore') //ignore: ['pipeline_info/*.{html,json,txt}', 'reference/**/*.{tsv.gz,tsv,html,fa.gz,fa.gz.fai,fa.gz.gzi,rds,vcf.gz.vcf.gz.tbi}'])
            // stable_path: All files in ${params.outdir}/ with stable content
            //def stable_path = getAllFilesFromDir(params.outdir, ignoreFile: 'tests/.nftignore')
            // vcf_files: All vcf files
            //def vcf_files = getAllFilesFromDir(params.outdir, include: ['**/*.vcf{,.gz}'])
            assert workflow.success
            assertAll(
                { assert snapshot(
                    // pipeline versions.yml file for multiqc from which Nextflow and pipeline versions are removed (all from the workflow key)
                    //removeFromYamlMap("${outputDir}/pipeline_info/nf_core_rnavar_software_mqc_versions.yml", "Workflow"),
                    // All stable path name, with a relative path
                    stable_name,
                    // All files with stable contents
                    //stable_path.isEmpty() ? 'No stable content' : stable_path,
                    // All vcf files
                    //vcf_files.isEmpty() ? 'No VCF files' : vcf_files.collect { file -> file.getName() + ":md5," + path(file.toString()).vcf.variantsMD5 }
                ).match() }
            )
        }
leads to error
Copy code
then {
            // stable_name: All files + folders in ${params.outdir}/ with a stable name
            //def stable_name = getAllFilesFromDir(params.outdir, relative: true, includeDir: true, ignoreFile: 'tests/.nftignore') //ignore: ['pipeline_info/*.{html,json,txt}', 'reference/**/*.{tsv.gz,tsv,html,fa.gz,fa.gz.fai,fa.gz.gzi,rds,vcf.gz.vcf.gz.tbi}'])
            // stable_path: All files in ${params.outdir}/ with stable content
            def stable_path = getAllFilesFromDir(params.outdir, ignoreFile: 'tests/.nftignore')
            // vcf_files: All vcf files
            //def vcf_files = getAllFilesFromDir(params.outdir, include: ['**/*.vcf{,.gz}'])
            assert workflow.success
            assertAll(
                { assert snapshot(
                    // pipeline versions.yml file for multiqc from which Nextflow and pipeline versions are removed (all from the workflow key)
                    //removeFromYamlMap("${outputDir}/pipeline_info/nf_core_rnavar_software_mqc_versions.yml", "Workflow"),
                    // All stable path name, with a relative path
                    //stable_name,
                    // All files with stable contents
                    stable_path.isEmpty() ? 'No stable content' : stable_path,
                    // All vcf files
                    //vcf_files.isEmpty() ? 'No VCF files' : vcf_files.collect { file -> file.getName() + ":md5," + path(file.toString()).vcf.variantsMD5 }
                ).match() }
            )
        }
works
Yeah confirmed by testing the issue comes from 
includeDir: true
. When I remove it 
stable_name
also works without OOM error
m
Sorry, got sidelined
I wouldn't use includeDir when snapshoting files content
I think nf-test then tries to snapshot the content and it behaves weirdly
f
but why was that in the template then 🤔 anyway I removed it now and it works yay
m
it was in the template?
???
f
not sure but I think so? Because I did not put it there.
m
f
yes
and that produced the error for me
m
confused dog
ok, I'm confused now
ok, I misread the whole thread
that's interesting
🐕 1
are you using a disk large?
f
the problem occurs on my linux machine locally
m
ok
I noticed some not enough space message on the CI
and you have pushed the latest changes to the PR?
I think something like that should work
Copy code
def stable_name = getAllFilesFromDir(params.outdir, relative: true, includeDir: true, ignore: ['pipeline_info/*.{html,json,txt}', 'reference/{**/*,*}'])
Open in Slack
PreviousNext
Powered by