https://nf-co.re logo
Join Slack
Powered by
Hello! I am fairly new to nf-test and am running i...
# nf-test
e
Hello! I am fairly new to nf-test and am running into an issue with AWS. I have written a pipeline level test for a custom pipeline. My samplesheet is composed of a large private validation dataset's s3 URIs. Due to how large the dataset is, running locally is not an option. There is a specific validation reason I need to use such a large test dataset for this pipeline test. I am attempting to run through AWSBatch. I have added these "options" to my 
${baseDir}/nf-test.config
with
Copy code
options "--custom_config_base <https://raw.githubusercontent.com/config/path> -bucket-dir <s3://path/to/work>"
I am encountering an error where it seems as if my aws credentials are not being passed to nextflow when running through nf-test. This is the error I have been hit with:
Copy code
ERROR ~ Access Denied (Service: S3, Status Code: 403, Request ID: ID, Extended Request ID: ID (SDK Attempt Count: 1)
I have tried assigning my rotating credentials in my 
${baseDir}/nf-test.config
files like so:
Copy code
profiles {
    awsdev {
        process.executor        = 'awsbatch'
        process.queue           = 'dev_nextflow_nextflow_job'
        aws.region              = 'us-east-2'
        aws.accessKey           = System.getenv('AWS_ACCESS_KEY_ID')
        aws.secretKey           = System.getenv('AWS_SECRET_ACCESS_KEY')
        aws.sessionToken        = System.getenv('AWS_SESSION_TOKEN')
        aws.batch.cliPath       = '/home/ec2-user/miniconda/bin/aws'
        docker.enabled          = true
    }
}
This failed. I then tried setting them on the command line like so:
Copy code
AWS_ACCESS_KEY_ID="ID" AWS_SECRET_ACCESS_KEY="ID" AWS_SESSION_TOKEN="ID" nf-test test tests/default.nf.test --profile awsdev
This failed as well. I added them to the 
{baseDir}/tests/config/nextflow.config
with:
Copy code
config {
    env {
        AWS_ACCESS_KEY_ID = System.getenv('AWS_ACCESS_KEY_ID')
        AWS_SECRET_ACCESS_KEY = System.getenv('AWS_SECRET_ACCESS_KEY')
        AWS_DEFAULT_REGION = System.getenv('AWS_DEFAULT_REGION')
        AWS_SESSION_TOKEN = System.getenv('AWS_SESSION_TOKEN')
    }
}
This has also failed. I refreshed my credentials and redid all of the steps above to make sure it wasn't my creds. I am able to run the pipeline normally with my awsdev profile with
Copy code
nextflow run ../main.nf -profile awsdev etc.
I feel quite dumb and am a bit lost on what to try next. Thank you for your time! 😄
🤔 1
n
Do you see any more information in the log files?
e
The 
nextflow.log
isn't terribly helpful, but I could always be missing something!
Copy code
Apr-13 15:00:05.964 [main] DEBUG n.script.dsl.ProcessConfigBuilder - Config settings `withName:SAMPLESHEET_CHECK` matches process WSLH_BIO_SPNTYPEID:SPNTYPEID:INPUT_CHECK:SAMPLESHEET_CHECK
Apr-13 15:00:05.967 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: awsbatch
Apr-13 15:00:05.967 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'awsbatch'
Apr-13 15:00:05.971 [main] DEBUG nextflow.executor.Executor - [warm up] executor > awsbatch
Apr-13 15:00:05.978 [main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-executor, limiter:RateLimiter[stableRate=50.0qps], poolSize:100, maxPoolSize:100, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
Apr-13 15:00:05.981 [main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-reaper, limiter:RateLimiter[stableRate=50.0qps], poolSize:100, maxPoolSize:100, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
Apr-13 15:00:05.982 [main] DEBUG n.cloud.aws.batch.AwsBatchExecutor - Creating parallel monitor for executor 'awsbatch' > pollInterval=10s; dumpInterval=5m
Apr-13 15:00:05.985 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: awsbatch)
Apr-13 15:00:06.925 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  TOOLS

Apr-13 15:00:06.926 [main] DEBUG nextflow.Session - Session aborted -- Cause: Access Denied (Service: S3, Status Code: 403, Request ID: D6E9X2C1FJNJCC19, Extended Request ID: 6EAQ+wZ93T3Mfj0+kuSX8yLvZNHtC7hFL2BsjT5m01xpFa73zw3zgt0h8CXMFcWtx4R0/gmTlZA8hJfmEaClI71sPFM781nh) (SDK Attempt Count: 1)
Apr-13 15:00:06.942 [main] INFO  nextflow.Nextflow - -[0;35m[wslh-bio/spntypeid][0;31m Pipeline completed with errors[0m-
Apr-13 15:00:06.947 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=0; peakCpus=0; peakMemory=0; ]
Apr-13 15:00:06.956 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Apr-13 15:00:07.441 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Apr-13 15:00:07.486 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Apr-13 15:00:07.490 [main] ERROR nextflow.cli.Launcher - @unknown
software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: D6E9X2C1FJNJCC19, Extended Request ID: 6EAQ+wZ93T3Mfj0+kuSX8yLvZNHtC7hFL2BsjT5m01xpFa73zw3zgt0h8CXMFcWtx4R0/gmTlZA8hJfmEaClI71sPFM781nh) (SDK Attempt Count: 1)
n
Hmmm that's very weird, I'm also not entirely sure AWS works with nf-test 😕
e
Ahh, I assumed it did work! I apologize. Do you happen to know if there is any documentation we can refer to regarding what nf-test is compatible with? I've been struggling to find some. 😅
n
nf-test itself is a purely local tool written to execute small tests. It might be possible to let it submit jobs to a cluster or AWS but I've not tested this myself so I can't say that for sure
I'm also no expert in using AWS batch so I'm not sure if you are doing something wrong or not
e
Interesting! I completely understand. I just posted a general question in the no stupid questions channel, so hopefully someone knows for sure there! 😄
🙌 1
n
Allright fingers crossed!
Open in Slack
PreviousNext
Powered by