Hello! I am trying to run taxprofiler in Azure Bat...
# taxprofiler
a
Hello! I am trying to run taxprofiler in Azure Batch and I keep getting this error:
Copy code
ERROR ~ Error executing process > 'NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS (SAMN05826489)'

Caused by:
  Process `NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS (SAMN05826489)` terminated with an error exit status (1)


Command executed:

  samtools \
      stats \
      --threads 1 \
      --reference human_plus_phix.fna \
      SAMN05826489_SAMN05826489.bam \
      > SAMN05826489_SAMN05826489.stats
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS":
      samtools: $(echo $(samtools --version 2>&1) | sed 's/^.*samtools //; s/Using.*$//')
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Work dir:
  <az://rawdata/work/6d/d9aa8bd4032fd6ab43e3e53801b19a>

Container:
  quay.io/biocontainers/samtools:1.20--h50ea8bc_0

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: <https://nf-co.re/docs/usage/troubleshooting>

 -- Check '.nextflow.log' file for details
The .command.log in the work folder is like this:
Copy code
mkdir: can't create directory 'SAMN05826489_SAMN05826489.bam': File exists
mkdir: can't create directory 'human_plus_phix.fna': File exists
I've tried to delete the work folder and run the pipeline again but the same error occurs. At the end, I get only one sample out of five being profiled by Metaphlan in my tests. I really don't know how to solve this without turning off host reads removal.
j
That's bizarre... Trying to create directorires of the file name?!
Can you send the
nextflow.config
?
a
This is the config file I'm using. Just omitted some ID info
Copy code
process {
    executor = 'azurebatch'
    container = 'docker'
}
    azure {
      managedIdentity {
        clientId = 'myid'
      }

      storage {
        accountName = 'mystoragename'
      }

      batch {
        accountName = 'mybatchaccountname'
        location = 'westeurope'

        autoPoolMode = true
        allowPoolCreation = true

        pools {
          auto {
            autoScale = true
            vmType = 'Standard_D32d_v4'
            virtualNetwork = 'myvnet'

            identity {
              type = 'UserAssigned'
              userAssignedIdentities = [
                'myid'
              ]
            }
          }
        }
      }
    }
https://github.com/nf-core/sarek/issues/1790 I found this person with a similar problem and environment as me, but running a different pipeline
j
Ok and your
.nextflow.log
?
a
nextflow.log
j
Sorry for the delay (weekend and all that) So I think tactually your issue is slightly upstream in teh working directroy creation, your log file just before the process erorr is:
Copy code
May-24 07:04:20.845 [TaskFinalizer-3] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS (SAMN05826489); work-dir=<az://rawdata/work/6d/d9aa8bd4032fd6ab43e3e53801b19a>
  error [nextflow.exception.ProcessFailedException]: Process `NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS (SAMN05826489)` terminated with an error exit status (1)
May-24 07:04:20.873 [TaskFinalizer-3] DEBUG nextflow.processor.TaskRun - Unable to dump output of process 'NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS (SAMN05826489)' -- Cause: java.nio.file.NoSuchFileException: <az://rawdata/work/6d/d9aa8bd4032fd6ab43e3e53801b19a/.command.out>
May-24 07:04:20.881 [TaskFinalizer-3] DEBUG nextflow.processor.TaskRun - Unable to dump error of process 'NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS (SAMN05826489)' -- Cause: java.nio.file.NoSuchFileException: <az://rawdata/work/6d/d9aa8bd4032fd6ab43e3e53801b19a/.command.err>
So it can't even make the `.command.out`/`.command.err` messages. This happens in the start up step before the actual command is executed too, so is happening to what is being recorded in
.command.log
. You say you 'keep having this error', did you try running from scratch the pipeline multiple times, and/or did you try with
-resume
? Instinctively this to me feels like that particular cloud node has a problem (but I don't know if I can get any more info on that from this log file).
The only person I know who has run stuff on azure is @Adam Talbot... I dunno if he has any suggestions (from Monday ofc)
a
Try to separate the azure stuff from the pipeline. Can you run hello world? Can you run other pipelines?
pools.auto.userAssignedIdentities
isn’t a config item so won’t do anything
Ah wait I see some of the pipeline worked:
At the end, I get only one sample out of five being profiled by Metaphlan in my tests.
Gonna have to look on my computer on Monday
❤️ 2
j
Yeah exactly, that's why I have a feeling it might be a node failure
a
I tried running it fresh and resuming afterwards but the same error occurs. This time I didn't kill the process so I'm attaching here the new nextflow.log files. Thank you for the replies! I will also get back into this on Monday. Just for the record: every time I run the pipeline I get the same error, could a node failure be reproducible like this?
j
No, if it's reproducible then unlikely not a node failure
🤔 1
Sorry bad typing, too many negatives 😆
It's likely not a failure (as in it's something else)
I've never seen this error before though
a
Do you get the same error with
-profile test
?
a
Actually it runs completely fine with
-profile test
j
Ok that's strange then
a
Is the
NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS
output actually used for host removal? Can I bypass it?
j
No it's not used for it - it's for reporting the number of reads moved. I think you can add something like
Copy code
withName: *.SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS {
    when: false
}
To a config file and give it to the pipeline with
-c
. But I've not tried it myself before. But it is VERY strange that the test profile works with the same process and not in your full run...
I still find that suspicious
a
Now it's working with
Copy code
withName: 'SAMTOOLS_STATS' {
	when = false
}
Honestly I have no clue what is going on with that error since it's the first time I run Nextflow in Azure Batch. I feel like it's Azure-related but it's beyond my knowledge 😔
a
So the issue is specific to samtools stats? Could you share a config where we can recreate the issue using test data? Perhaps it only fails with data that is a certain size?
j
Could it be its running out of space on the node but for some reason nxf doesn't see? But I would be extremely surprised unless it's running multiple jobs on the same node?
a
Thats likely. So most batch processors pack multiple jobs onto larger nodes for efficiency but each one is isolated. Azure comes with fixed disk sizes which are a bit annoying because when they run out of space the node blows up. There’s not really a good way around it other than don’t run a million 1 CPU jobs.
🤔 1
I’ve added the disk directive to the azure plugin so if you specify disk = 128GB it will try and reserve that much per task
👍 1
j
@Adam Talbot do you have a link to some docs about that? Might be useful for @André Ferreira Cunha to try?
a
Just the stuff on task packing
A Standard_D32d_v4 comes with 32 cores and 1.2TB of local storage. Each SAMTOOLS_STATS task only uses 1 CPU, so Azure Batch will schedule 32 tasks onto each node. If you're BAM files are larger than 37.5GB on average this will fail. Because you're using the most recent nextflow version, you should be able to do:
Copy code
withName: 'SAMTOOLS_STATS' {
	disk = 128GB
}
To reserve a chunk of disk on the node at runtime.
Looking at the logs now: 1. SAMTOOLS_STATS failed for an unknown reason:
Copy code
Process `NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS (SAMN05826487)` terminated with an error exit status (1)
2. After completion, it did not upload the .command.out and .command.log to remote storage
Copy code
May-24 17:43:59.262 [TaskFinalizer-2] DEBUG nextflow.processor.TaskRun - Unable to dump error of process 'NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS (SAMN05826487)' -- Cause: java.nio.file.NoSuchFileException: <az://rawdata/work/f8/3f88df384251b69bd065cb0b96d34b/.command.err>
May-24 17:43:59.271 [TaskFinalizer-2] DEBUG nextflow.processor.TaskRun - Unable to dump output of process 'NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_HOSTREMOVAL:SAMTOOLS_STATS (SAMN05826487)' -- Cause: java.nio.file.NoSuchFileException: <az://rawdata/work/f8/3f88df384251b69bd065cb0b96d34b/.command.out>
3. Nextflow was unable to see what happened because it can't see the log files and exitcode I have a PR open to catch these situations better: https://github.com/nextflow-io/nextflow/pull/5803
❤️ 2
a
Thank you for the explanation! As soon as I can I will test it reserving a disk chunk. If you want to try to reproduce my error by any chance, I was running some tests with samples from this study. More specifically, this samples:
Copy code
SAMN05826487
SAMN05826489
SAMN05827028
SAMN05826517
SAMN05826516
All associated runs had been already merged into sample FASTQs since I downloaded everything using fastq-dl with
--group-by-sample
. I was using the config file I sent earlier in this thread and I think you can see the pipeline parameters in the nextflow.log files.
Please let me know if you need something else to reproduce the error