Hi, I have two modules that takes quite a lot of t...
# team-maintainers
l
Hi, I have two modules that takes quite a lot of times and resources to compute the nf-test due to big databases. •
IPHOP_DOWNLOAD
and
IPHOP_PREDICT
download a 4.5Gb database (it's a small one compare to the real on 109Gb) •
HLALA_TYPING
similarly the graph is quite huge if we want it to work I was thinking to only have a nf-test for the stub and comment the real test to not run it on the github action. For
IPHOP_DOWNLOAD
I have
java.lang.OutOfMemoryError: Required array size too large
when I try to check the md5sum of 44 files. Do you know how I could avoid this error ?
m
44 files should be less than the array limit
l
Maybe it's the computation of all the md5sum that takes the memory ? I had the same issue with the stub test with
Copy code
{ assert snapshot(
    process.out,
    path(process.out.versions[0]).yaml
).match() }
But the error disappeared in the stub test when setting the assertion to
Copy code
{ assert snapshot(
    process.out.iphop_db,
    path(process.out.versions[0]).yaml
).match() }
I'm on wsl2 so there might be some limitation ?
m
Interesting. Have you checked in GitHub codespaces?
l
I can try but this might fail due to the storage limitation.
I've used the 4-core 16GB RAM 32GB on codespace and it seems that it silently kills the nf-test snapshot...
Copy code
Test Process IPHOP_DOWNLOAD

  Test [3e40beb0] 'iPHoP_db_rw_for-test' 
    > N E X T F L O W  ~  version 24.10.5
    > Launching `/workspaces/modules/.<http://nf-test-3e40beb08eb3ff6e65a2e2936ad8ef38.nf|nf-test-3e40beb08eb3ff6e65a2e2936ad8ef38.nf>` [jolly_tesla] DSL2 - revision: 09ba4c5830
    > WARN: Unknown directive `params` for process `IPHOP_DOWNLOAD`
    > WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /workspaces/modules/.nf-test/tests/3e40beb08eb3ff6e65a2e2936ad8ef38/work/singularity -- Use the environment variable NXF_SINGULARITY_CACHEDIR to specify a different location
    > Pulling Singularity image <https://depot.galaxyproject.org/singularity/iphop:1.3.2--pyhdfd78af_0> [cache /workspaces/modules/.nf-test/tests/3e40beb08eb3ff6e65a2e2936ad8ef38/work/singularity/depot.galaxyproject.org-singularity-iphop-1.3.2--pyhdfd78af_0.img]
    > [31/ea2883] Submitted process > IPHOP_DOWNLOAD
gitpod /workspaces/modules (iphop/download) $
I got no error message but the snapshot isn't updated
When I snapshot only the versions it works
{ assert snapshot(process.out.versions).match() }
m
Interesting
Any idea on how many files are created?
I don't see why that is an issue
l
Any idea on how many files are created?
There is 36 files created.
m
Weird that we cannot just snapshot their names...
I'll try to work on nft-utils so we can capture that more easily
l
getAllFilesFromDir(process.out.iphop_db[0], true, null, null, ['*', '**/*'])*.name
did work !
m
Seriously 😆
I'm impressed
l
But
getAllFilesFromDir(process.out.iphop_db[0], true, null, null, ['*', '**/*'])
give the same error
java.lang.OutOfMemoryError: Required array size too large
at least it is not silent ^^'
👀 1
I'm ready to PR the nf-test migration of
iphop/download
but I wonder if non stub test should be added or commented out ? It does download a huge file... What's your point of view ?
m
if it's too painful and takes too long, stub is enough for me
but if possible, let's add some tests and comment them out, that way we can always test locally whenever
what was done before?
l
It was fully tested 😕