When running VEP or SNPEFF (and probably other modules) there is a

cache

option with let’s you pass the path to a local cache of already downloaded data. The actual directory which is staged in these cases is the parent directory of the cache under which there might be several genomes. This is fine when running on a local filesystem as the directory is just staged in via a mount and the size of the cache doesn’t really matter. However if you are running on a cloud executor like Batch then the data will be staged onto the worker instance via copy. This means that even though you might only need one specific genome from your cache, the staging will wait to copy across the entire directory. To me this seems to undermine the usefulness of having pre-fetched the data from some other source. Do we think there is a better way for these modules to pass and stage these reference data so that we do not end up copying potentially unused cache items? (I don’t want an answer that is just “use fusion”, that isn’t viable for many people)

Depends on what the structure it is looking for looks like. If it just wants one subfolder with the appropriate genome name, then can we just stage the cache into that folder?

Well both of these are structured AFAIK.

snpEff

is

<cachedir>/<genome>

which is simple.

VEP

is more involved and possibly not solvable because it depends on (at least)

--merged

which may or may not be set in

args

. I know Genomics England used mountpoints for reference data but I really don’t like that solution because it: • reduces the portability (need to always manually mount in your references now) • removes the file from staging controls • requires modification of existing modules (my suspicion is they patch, but they might just be using all their own modules now) This might be one of those situations without a good fix honestly, but I thought I’d put it out there.

That's why I designed #C05DACM4HPV that way. Every genome cache version is actually in its own separate cache. And Sarek recompute the path back to it if you use annotation cache or just point to the local cache with multiple versions and genomes if you have it

Sarek is using it