Hi All, just rechecking if anybody has had any experience with the problem I am encountering, described above. Thank you. : https://nfcore.slack.com/archives/CGFUX04HZ/p1724352875744429

In your log you have

com.amazonaws.SdkClientException: Failed to connect to service endpoint

and (a bit further down)

Caused by: java.net.SocketException: Network is unreachable

, suggesting a (temporary?) network connection problem, apparently while trying to download the snpEff cache. Not sure there is a relationship to the error that kill the pipeline, but for both a shaky internet connection could be an explanation. The error you are getting is strange --

END_VERSIONS

is the tag signalling the end of a "here-document" in bash that is used to create the

versions.yml

file of each process. Somehow that seems to have been incorporated into the file and the YAML parser chokes on it. No idea how that can happen, but I would make sure it was not some one-time glitch. You are downloading the pipeline on-the-fly, I could image something got corrupted there. I would try renaming (or deleting)

~/.nextflow

, that contains by default all of the pipeline code (under

assets

[if you just want to remove the code], as well as various caches), to force a redownload. If your internet connection is indeed less than perfectly reliable, I would recommend predownloading everything you can. The

nf-core download

command from tools can download the pipeline code and all containers, and running the pipeline with the parameters

--download_cache --build_only_index --tools vep,snpeff --genome $mygenome

(and not much else) lets you download the VEP and/or snpEff caches before an actual run (you could even download on another machine and then copy them over); you can then use the respective parameters to let the pipeline know where to find them. That way any internet glitch will not kill your runs with actual data.

Could your problem be somewhat similar to this one?

Thank you both @Anders Sune Pedersen and @Thomas Danhorn. @Thomas Danhorn: there is no internet connectivity issue, but that error message has been showing up randomly with zero effect on the other processes. As for the pipeline running on the fly, I have previously pulled it, but based on your reasonable suggestion, I parked the previous cache and pulled 3.4.3 as the stable version, just in case something has gone wrong with the asset cache and I did not notice that. I rally had a hard time understand whether this "END_VERSIONS" should be in the file or not, and by checking under successfully running modules, my intuition was also that it should not be in the versions.yml files. I tried to remove all the instances of that from versions.yml files when the pipeline crashed, but upon resume, #CGFUX04HZ overwrites the files with yet another problematic copy of versions.yml. I am currently running the newly pulled Sarek pipeline (3.4.3) and I will report back. Thanks a lot for your help. @Anders Sune Pedersen Thanks for checking for this issue. It seems the unnecessary presence of the phrase "END_VERSIONS" in the last line of the versions.yml along with the added indentation points the YML interpreter to expect this to be a key-value pair. As I explained above for Thomas, it seems my intervention to remove these lines before resuming the pipeline does not help, as the resume overwrites the files with a similarly problematic content. I will keep you posted.

The

END_VERSIONS

should not be in any file, it signifies the end of the "here-document" that is used in creating the file. And in functional code I can't see how it ends up in the file, hence my hypothesis that the code was corrupted. Manual removal does not solve the problem, because at that point the process has failed and Nextflow will rerun it (upon

-resume

) and overwrite anything that was there before (including your edits). The fact that this seems to be reproducible is consistent with my hypothesis of corrupted code. Hope the new download solves it.🤞