Kolby Wilson
05/29/2024, 5:42 PM"/home/ec2-user/reference_genomes" . I have my AWS Batch compute environment configured to use the AMI when launching instances, and I've configured nextflow to mount the volume to the containers for the jobs.
My goal was that when Batch would create the EC2 instances for the jobs, everything would be mounted and the jobs would be able to access the files from there.
Here is my nextflow config file:
process {
executor = 'awsbatch'
queue = 'nextflow_job_queue'
}
aws{
region = 'ca-central-1'
batch{
cliPath = '/home/ec2-user/miniconda/bin/aws'
volumes = '/home/ec2-user/reference_genomes/'
}
}
workDir = 's3://nextflow-wd/workdir/rnaseq/'
docker.enabled = true
docker.runOptions = '-u $(id -u):$(id -g)'
and here is my params file:
{
"input": "s3://nextflow-wd/samples/samplesheet.csv",
"outdir": "s3://nextflow-wd/output/rnaseq/",
"gtf": "/home/ec2-user/reference_genomes/homo_sapiens/Homo_sapiens.GRCh38.111.gtf",
"fasta": "/home/ec2-user/reference_genomes/homo_sapiens/Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa",
"skip_alignment": true,
"pseudo_aligner": "kallisto",
"kallisto_index": "/home/ec2-user/reference_genomes/homo_sapiens/homo_sapiens.idx",
"igenomes_ignore": true,
"genome": null
}
I launch the pipeline using this command:
nextflow run nf-core/rnaseq -c nextflow.config -params-file params.json
but when I do, I get the following error:
N E X T F L O W ~ version 23.10.1
Launching `https://github.com/nf-core/rnaseq` [disturbed_jennings] DSL2 - revision: b89fac3265 [master]
ERROR ~ ERROR: Validation of pipeline parameters failed!
-- Check '.nextflow.log' file for details
The following invalid input values have been detected:
* --kallisto_index: the file or directory '/home/ec2-user/reference_genomes/homo_sapiens/homo_sapiens.idx' does not exist.
* --gtf: the file or directory '/home/ec2-user/reference_genomes/homo_sapiens/Homo_sapiens.GRCh38.111.gtf' does not exist.
* --fasta: the file or directory '/home/ec2-user/reference_genomes/homo_sapiens/Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa' does not exist.
-- Check script '/home/kolby/.nextflow/assets/nf-core/rnaseq/main.nf' at line: 52 or see '.nextflow.log' file for more details
After investigating the logs, it appears that the SchemaValidator is causing the error and the reason it is unhappy is because those files are not present on the EC2 launching nextflow (different AMI). So I tried running the command again from a new EC2 that is running the same AMI with the files present, that solves the error- but in the logs I notice this:
May-29 10:41:33.474 [FileTransfer-2] DEBUG nextflow.file.FilePorter - Copying foreign file /home/ec2-user/reference_genomes/homo_sapiens/Homo_sapiens.GRCh38.111.gtf to work dir: s3://nextflow-wd/workdir/rnaseq/stage-7b6b7742-1c53-4c19-911b-12c621db69a7/f7/09aa2bfb70a3252fad0895b8ed234f/Homo_sapiens.GRCh38.111.gtf
From there on out it seems to be using the staged files from the work directory when running the jobs, so while it is working, its not working how I intended and unnecessarily uploading the local files to S3. It does make sense why this happens, I just figured once the SchemaValidator was happy- it would proceed- then the jobs would just be looking to that path in the container once they run, but this doesn't seem to be the case.
I realize I could forego storing the files on the custom AMI and instead store them in S3. I've tested this and updated my params file to reference the S3 URL and it works. I don't see any staging of the files or calls from FilePorter. But I wanted to see if this custom AMI approach with mounted volumes would work as I thought it might be useful for larger databases (like Kraken2 for use with nf-core/taxprofiler) and I could avoid the jobs from having to pull the files from S3 each time since if they were just readily available, it could just use the path to the database.
Is there a way to get this approach to work, or is this just a flawed approach in general?
Follow up question, if I used a shared file system like FSx or EFS and that shared file system were mounted to both the EC2 launching Nextflow and the containers launched by Batch, would that run into the same issue? With a shared file system is there a way in which the jobs will reference the files in the mounted file system on the container?
Also, to clarify, I have sanity checked that Batch is working as expected with this configuration, I've confirmed both that the EC2s launched are running the expected AMI,and that the volume is being mounted in each job. So everything is there, I just can't seem to make Nextflow utilize it.
Thank youPhil Ewels
05/29/2024, 8:04 PMPhil Ewels
05/29/2024, 8:05 PMPhil Ewels
05/29/2024, 8:06 PM--validate_params false