Hi Team, I have found an issue related to Conda in...
# team-maintainers
i
Hi Team, I have found an issue related to Conda in pipeline tests in #C06R4FC4C1E, and I believe it's a more general issue. In some cases, custom modules in pipelines don't support Conda, usually because they use some bespoke custom package that can be put in a Docker container, but not published to conda-forge. While it's easy to handle such cases in the pipeline, and the effect of disabling such modules is often minor, it's enough to entirely ruin standard nf-tests: it affects the number of processes, the version list, and the output file list. When that happens, the developer is faced with a tragic choice: • skip these parts in the test and have untested code in the pipeline, • or leave them in place and struggle with tests. Ideally, it could be possible to detect Conda in nf-test and handle it accordingly. While an anti-pattern in general, it could make life significantly easier in particular cases. I have thoroughly investigated, and there does not seem to be any way (not even hacky) to achieve this. I have opened an issue in the nf-test repo, but it's still awaiting response. For the time being, I see 4 reasonable solutions: 1. Opt out of Conda tests for the affected pipelines until this is fixed and document it properly in the README, 2. Create separate tests for Conda and Docker/Singularity, and add Conda tests to the small CI suite (for
dev
PRs), 3. A compromise: run a strongly reduced (possibly stub-only) Conda test in the small CI, and full Conda CI in release CI, 4. Set up an AI-powered bot to bother nf-test devs with increasingly angry messages until they release a fix (for legal reasons, please don't). I invite you to discuss this issue. I am happy to implement any of the solutions. You can also vote with number reactions of you support one of the options as is.
s
Could we set an environment variable inside the CI runner?
which could then be referenced in some way?
i
I have tried this, and it seems to work correctly in the runners, but then the tests become pretty hard to work with locally, as the dev has to remember to set/unset this variable as required before running a particular test
And there's no obvious alternative for local runs, as all Conda detection methods check for the presence of Conda in the environment of nf-test rather than of the pipeline, which means it almost always detects Conda (both my machine and the runners have Conda enabled)
s
Yes, I saw your discussion on that. Basically we need functionality in nf-test, so we need to get a PR to them
i
I've looked into the codebase and it looks tricky to navigate. It might be time to install Cursor to find the relevant code.
s
I suspect that AI could manage it yes