Hi, potentially a very quick and simple question: How long is NFCORE_RNASEQRNASEQFASTQ_QC_TRIM_FILTER_SETSTRANDEDNESSFASTQ SUBSAMPLE FQ SALMONSALMON_QUANT expected to take to complete for a fastq.gz of about 1GB with GRCh38 human genome assembly? My guess was that this should only take something like 10-20 minutes, but these tasks have been running for 5 hrs 20 min. For context, I'm running the nf-core/rnaseq pipeline with default configuration, but via Google Batch. I've pasted the contents of .nextflow.log in a comment below.

Those processes should be super-quick (it's only running salmon on a tiny sample of reads). There's something wrong with your infra, but I couldn't tell you what. Claude's take if you're interested:

Short version

  No, 5+ hours is not normal - this step should take single-digit minutes per sample, so your 10-20 min guess was already generous. Something is wrong; these tasks are effectively stuck, not legitimately busy.

  What the log shows

  That process is the strandedness auto-detection step (FASTQ_SUBSAMPLE_FQ_SALMON:SALMON_QUANT). It is not the real quantification - it subsamples a small slice of reads and runs Salmon purely to infer library strandedness. It's meant to be cheap.

  Timeline from your .nextflow.log:

  - 09:09:06 - all 6 SALMON_QUANT tasks submitted to Google Batch (one per sample), all go to RUNNING.
  - 14:34:06 - end of the log: all 6 are still RUNNING, no exit code, none completed. That's ~5h25m with zero progress.
  - The 7 -> 6 task-count drop at 09:59 was not a Salmon task finishing - it was STAR_GENOMEGENERATE (building the full human STAR index, a far heavier job needing ~38 GB RAM) completing in ~48 min (09:09 -> 09:57).

  So the compute environment itself is fine - STAR genome generation ran and finished normally. It's specifically these 6 Salmon strandedness tasks that are hung. Six identical tasks all stalling indefinitely points to an environmental/IO problem, not the data.

  Also worth noting: SALMON_INDEX and FQ_SUBSAMPLE were both resumed from cache, so the index is already built and the reads are already subsampled. By the time SALMON_QUANT starts there is very little real work left to do.

  The catch

  The Nextflow log only shows Google Batch's view (state=RUNNING = VM is up and the container started). It does not show what's happening inside the VM, so I can't tell from this file alone whether the tasks are:

  1. slowly staging the large decoy-aware human Salmon index from GCS onto each of the 6 VMs (most likely culprit - default Google Batch boot disk is small/slow pd-standard, and 6 VMs each pulling a ~15-20 GB index over the network can crawl),
  2. stuck pulling the container,
  3. genuinely hung (a zombie salmon process), or
  4. quantifying far more reads than intended (if subsampling didn't actually reduce the count).

  To pin it down

  - Look at the task's .command.log / .command.err / .command.out in the work dir, e.g.
  <gs://processing-wb-lambent-jalapeno-9597/scratch/57/a911190acb4c718d8c80f66d7a0969/> - if Salmon has printed its "processed N fragments" progress it's running; if it's silent it's stuck in staging/pull.
  - Check Google Cloud Logging / Batch for the job IDs in the log (e.g. nf-57a91119-1782378546129) to see disk/network/CPU activity on the VM.
  - Quick mitigations if it's the index-staging theory: bump the boot/scratch disk type and size for this process (or use Fusion + NVMe local SSD), which dramatically speeds up large-index staging on Google Batch.

Thanks for the help Jonathan. I thought something was fishy. The work dirs hadn't even produced

.command.log / .command.err / .command.out

yet after 5 hours! Would that be consistent with any of claude's 4 scenarios?

Probably excludes option 4, means nothing has even started running yet. Staging files or containers seems likely culprits

Copy code

- Check Google Cloud Logging / Batch for the job IDs in the log (e.g. nf-57a91119-1782378546129) to see disk/network/CPU activity on the VM.

CloudWatch might be able to give you a clue when you check the cloud console.

I may be misremembering but thought outgoing content was only synced at the end with cloud (likely excluding fusion uses) so that looking in the work directory while the task is running won't give much.

Correct, Google Batch runs everything on a local drive then copies it out at the end. However, you may be able to see Google Batch related issue while running, e.g., failure to pull the container, local node running out of storage