Is there a way to limit the number of foreign file...
# helph
Henry Webel
03/12/2026, 7:00 PMIs there a way to limit the number of foreign files to be staged? I continously hit the rate limit of an FTP. I tried to pass these to the config, but with no effect. Only one job gets executed, but this does not hinder the staging of files in the background to be triggered directly.
Copy code
// Optional: Also add rate limiting for job submissions
executor {
queueSize = 1
submitRateLimit = '1/5s' // Max 1 job submission every 5 seconds
}Henry Webel
03/12/2026, 7:07 PMbrief summary. two files stared to be stated, the third failed
Copy code
Caused by:
Can't stage file <http://ftp.ebi.ac.uk/pride-archive/2015/12/PXD001819/UPS1_12500amol_R3.raw> -- reason: Connection reset
(...)
-- Check '.nextflow.log' file for details
Staging foreign file: <http://ftp.ebi.ac.uk/pride-archive/2015/12/PXD001819/UPS1_12500amol_R1.raw>
Staging foreign file: <http://ftp.ebi.ac.uk/pride-archive/2015/12/PXD001819/UPS1_12500amol_R2.raw>
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: <https://nf-co.re/docs/usage/troubleshooting>t
Thomas Danhorn
03/21/2026, 7:10 PMCan you download the files and cache them on a local disk? That would probably be the most efficient way, especially if some files would be used by more than one process.
h
Henry Webel
03/23/2026, 2:13 PMI agree. will setup a test folder with the data and maybe add a local testing to the test configuration.
👍 1
2 Views