I'm starting to work with Sarek, and it seems I will have as many questions as every beginner. IT has not allowed me to install Slack on the company computer and I'm writing this from my phone, so apologies in advance for limited logs. I am trying to run the test profile with Conda (Docker is not an option), and I keep getting the same error:

Exception in thread ...: exception while reading process stream

, with

Caused by: Stream closed

later in the traceback. It seems Conda-related, since exiting at this stage causes a "command not found" error on resume. Creating the environment by running the correct command manually fixes the issue, but it's annoying to have to do it for many modules. I'm not sure this is Sarek-specific, but perhaps someone here has encountered this issue before. Any ideas on how to solve it?

can you test if the demo pipeline works with conda?

It does not, fastqc and seqtk modules work, but the same error appears in the MultiQC module. After waiting another long while, the pipeline terminates with an error that says

Failed to create Conda environment

and

Status: 120

. I see no more useful information in the terminal output, I still need to investigate the log file.

hm maybe somehing for a help channel. don't really know what is going on there

Alright, I'll go ask there

Slack can be used through web as well 🙂 Can you use Singularity or Apptainer? There should be packages allowing for installation as a normal user on modern Linuxes.

It might be tricky, as the terminal of this particular cluster is running in a Docker container. I know Docker in Docker is bad practice, less sure about Singularity in Docker.

"terminal"? If you can run jobs outside of that environment, singularity/apptainer (or presumably docker) should be fine. I would expect the configuration would not allow for (easily) running singularity/apptainer inside Docker (although it's not impossible).

It's a strange configuration, the cluster is accessed through a VM running in Docker, and normal users are not allowed to directly access the main machine. I might be able to run jobs outside of the container using HTCondor, but I have not been able to set it up with Nextflow yet (due to more quirks of the setup).

Normally when I talk about cluster, I mean a bunch of computers with a scheduler that you can submit jobs to that will then be run on one of the computers. Do you mean the same here or something else? If you mean the same, are jobs also executed in a Docker environment?

Yes, I do mean a cluster in this sense. I am trying to figure out the details of how everything works, as it is extremely atypical compared to all clusters I've worked with before and the documentation is not great. For the moment, I am trying to run the pipeline in the VM (which is allowed, it has ample resources), and I know Docker is not an option there (IT has explicitly rejected a request from another user). I am not sure about Singularity, as seemingly nobody has tried it yet.

you have chosen the way of pain

It is what it is