Hi all, I'm looking to confirm my understanding of...
# help
s
Hi all, I'm looking to confirm my understanding of how Nextflow integrates with AWS Batch. My current understanding is: • Nextflow dynamically registers AWS Batch job definitions when using the AWS Batch executor. • The
aws.batch.platformType
setting is global for the entire workflow (
ec2
or
fargate
). • There does not appear to be a per-process
platformType
configuration. • As a result, all dynamically created job definitions inherit the same platform capability (EC2 or Fargate). My question is: Is it possible to run a single Nextflow workflow across both Fargate and EC2 compute environments without using pre-registered AWS Batch job definitions (
job-definition://
) ?
For example: • Lightweight processes run on a Fargate-backed queue. • High-resource processes (e.g., requiring 48 vCPUs) run on an EC2-backed queue. • Nextflow manages job-definition creation automatically. • No manually created AWS Batch job definitions are used. From what I've been able to determine, this is not possible, because the dynamically registered job definitions are created with a single platform capability based on the global
aws.batch.platformType
setting, and AWS Batch queues require the job definition's platform capability to match the underlying compute environment. Can anyone confirm whether this understanding is correct, or if there is another supported approach for mixing EC2 and Fargate within the same workflow run without manually managing job definitions? Thanks!
n
Hi, @SINRASU ALAGUDURAI! This looks like a general question and not something specific to an nf-core workflow. Please re-post your question on the Nextflow Community Forum - https://community.seqera.io 🎉
p
That seems essentially correct (generally, only
process
directives supports dynamic values through closures). With that said, the
platformType
is mostly used to dress up fargate jobs in a way that will help them get through - I don't recall anything there that should always break other queue types (but I may be forgetting something). I don't think it's a sensible thing to do, but it seems if nothing else you should be able to create your job definitions yourself and use them Just In Time from a
process.container
closure (combined with
process.queue
to direct to the correct job queue).