https://nf-co.re logo
Join Slack
Powered by
Can someone else look at this: <github.com/nf-core...
# team-maintainers
s
Can someone else look at this: github.com/nf-core/modules/pull/12899 suggested issue with a 
mv
command on AWS / Platform, but it is inside a module, so I would have thought it would only be on the local machine before it gets to pushed to s3
p
Yes if using Nextflow s3 support that stages files out at the end of the task only
No if you're using any kind of smart filesystem: Fusion, but also Mountpoint, yada yada yada (there are many of them)
These sync to s3 on the fly so that it's syncing during the whole task. Means you don't waste time at the end of the task waiting for all the data to be transferred
s
So this is a potential Fusion issue then
p
Fusion should be clever enough to know what files are outputs though
So I would hypothesise that in this case it's all the others except Fusion which are a potential issue
s
Well the issue is happening on the AWS megatests
Which presumably use Fusion?
p
I don't know what the issue is, so I have no idea
Fusion tends to get blamed a lot, these days it's usually not Fusion
But whatever is going on here, I would generally think that avoiding file manipulation is A Good Thing. Usually file moves can be easily avoided with stagein/out, a bit of care. So why not?
m
yeah, not moving files is better
m
That's a strange consideration. I also would have thought 
mv
was fine here. I guess an alternative is to 
ln
symlink, but I'm still not quite sure what the issue is.
p
Ok I dug into this a bit more, and yes.. in this time it is Fusion that's the issue 🫣 logo icon fusion color I was right that Fusion is being clever and avoiding uploading / downloading the intermediate files when they're not marked as inputs and outputs. However, when you do 
mv
Fusion still has to record a new S3 address for each file, so it knows where to write the bytes later. It gets that address by asking S3 about the destination key. It's a bit pointless here because the destination is a fresh path Fusion just made itself so ever request gets a 404, but the requests still take time. It has a timeout budget and when you have sufficient numbers of files in a directory it can't do all the checks in time and it times out and crashes.
🚫 2
I've written this up and reported it to the Fusion team so that hopefully we can be a bit more clever and avoid this class of error (the intention is that you can always treat the working dir as a local file system and not worry about 
mv
).
👍 1
But even if fixed, it's still probably better to avoid the rename. If we create the correct filename to begin with then Fusion can start uploading the files immediately as the task progresses. If you do a 
mv
at the end then it has to wait until the compute is finished before it can start uploading anything.
Hope that makes sense!
m
What about symlinks? How are they treated?
p
Looks like symlinks are ok: Fusion deliberately skips the existence check for new symlinks already. But symlinks are difficult for other reasons 😅 Because the directory that they're pointing to isn't uploaded (it's not an output), I'm not totally sure that Fusion would properly upload the source files. But I'm not sure that it wouldn't either. Might be fine..
Symlinks can be awkward with pure Nextflow as well though so should be avoided I think
Trying to remember the issue with that.. I think it's when you have 
scratch true
and then nextflow moves the symlink out of the workdir but doesn't know about the files that the symlink points to. If the next task can't access the scratch from the previous then it then fails..
💯 1
Basically the same thing I was worried about for Fusion, I guess
Open in Slack
PreviousNext
Powered by