I am running a large batch of samples and to me it seems that the processes do not properly finish:

INFO	2026-05-19 09:50:13	MarkDuplicates	Writing complete. Closing input iterator.
INFO	2026-05-19 09:50:13	MarkDuplicates	Duplicate Index cleanup.
INFO	2026-05-19 09:50:13	MarkDuplicates	Getting Memory Stats.
INFO	2026-05-19 09:50:13	MarkDuplicates	Before output close freeMemory: 451581368; totalMemory: 503316480; maxMemory: 25769803776
INFO	2026-05-19 09:50:13	MarkDuplicates	Closed outputs. Getting more Memory Stats.
INFO	2026-05-19 09:50:13	MarkDuplicates	After output close freeMemory: 237398632; totalMemory: 285212672; maxMemory: 25769803776
[Tue May 19 09:50:13 GMT 2026] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 9.63 minutes.
Runtime.totalMemory()=285212672
Tool returned:
0
[nahbu450@login1 f86e58199d8afbf31543daae53c3b1]$ ls -la
total 173
drwxr-sr-x.   2 nahbu450 nahnsen 37888 May 19 11:40 .
drwxr-sr-x. 102 nahbu450 nahnsen 72704 May 19 11:39 ..
-rw-------.   1 nahbu450 nahnsen     0 May 19 11:40 .command.begin
-rw-------.   1 nahbu450 nahnsen 33209 May 19 11:50 .command.log
-rw-r--r--.   1 nahbu450 nahnsen 15351 May 19 11:40 .command.run
-rw-r--r--.   1 nahbu450 nahnsen  1190 May 19 11:40 .command.sh

Above you can see part of the output of the

.command.log

and then the contents of the work folder. There is no exitcode despite the log of the tool clearly saying that its finished. I am running on a slurm hpc. Did anyone see something similar? Can I do something about that?

can you please give some details about the workflow/process? I've seen it some times and it was always something going on with the hpc EDIT: I see it's markDuplicates, but my question was if you are running it from a workflow.

Yes, I am running sarek 🙂

INFO	2026-05-19 09:50:13	MarkDuplicates	After output close freeMemory: 237398632; totalMemory: 285212672; maxMemory: 25769803776
[Tue May 19 09:50:13 GMT 2026] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 9.63 minutes.
Runtime.totalMemory()=285212672
Tool returned:
0
ls: cannot access '*.bai': No such file or directory
[nahbu450@login1 f86e58199d8afbf31543daae53c3b1]$ ls -la
total 11622881
drwxr-sr-x.   2 nahbu450 nahnsen      37888 May 19 11:53 .
drwxr-sr-x. 181 nahbu450 nahnsen      84480 May 19 14:27 ..
-rw-------.   1 nahbu450 nahnsen          0 May 19 11:40 .command.begin
-rw-------.   1 nahbu450 nahnsen      33245 May 19 11:53 .command.err
-rw-------.   1 nahbu450 nahnsen      33262 May 19 11:53 .command.log
-rw-------.   1 nahbu450 nahnsen         17 May 19 11:53 .command.out
-rw-r--r--.   1 nahbu450 nahnsen      15351 May 19 11:40 .command.run
-rw-r--r--.   1 nahbu450 nahnsen       1190 May 19 11:40 .command.sh
-rw-------.   1 nahbu450 nahnsen        295 May 19 11:53 .command.trace
-rw-------.   1 nahbu450 nahnsen          1 May 19 11:53 .exitcode

looking at the finalized thing it seems as if it took from around 9:50 (last entry in .command.log) until 11:53 (creation time of .exitcode).

do you use the scratch on slurm? maybe it failed to write back from there?

command.sh

#!/usr/bin/env bash -C -e -u -o pipefail
gatk --java-options "-Xmx24576M -XX:-UsePerfData" \
    MarkDuplicates \
    --INPUT NA18562.0001.bam --INPUT NA18562.0004.bam --INPUT NA18562.0003.bam --INPUT NA18562.0002.bam --INPUT NA18562.0008.bam --INPUT NA18562.0009.bam --INPUT NA18562.0010.bam --INPUT NA18562.0005.bam --INPUT NA18562.0011.bam --INPUT NA18562.0007.bam --INPUT NA18562.0006.bam --INPUT NA18562.0012.bam \
    --OUTPUT NA18562.md.bam \
    --METRICS_FILE NA18562.md.cram.metrics \
    --TMP_DIR . \
    --REFERENCE_SEQUENCE Homo_sapiens_assembly38.fasta \
    --REMOVE_DUPLICATES false --VALIDATION_STRINGENCY LENIENT

# If cram files are wished as output, the run samtools for conversion
if [[ NA18562.md.cram == *.cram ]]; then
    samtools view -Ch -T Homo_sapiens_assembly38.fasta -o NA18562.md.cram NA18562.md.bam
    rm NA18562.md.bam
    samtools index NA18562.md.cram
fi

cat <<-END_VERSIONS > versions.yml
"NFCORE_SAREK:SAREK:FASTQ_PREPROCESS_GATK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES":
    gatk4: $(echo $(gatk --version 2>&1) | sed 's/^.*(GATK) v//; s/ .*$//')
    samtools: $(echo $(samtools --version 2>&1) | sed 's/^.*samtools //; s/Using.*$//')
END_VERSIONS

.command.err

INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
Using GATK jar /opt/conda/share/gatk4-4.6.1.0-0/gatk-package-4.6.1.0-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx24576M -XX:-UsePerfData -jar /opt/conda/share/gatk4-4.6.1.0-0/gatk-package-4.6.1.0-local.jar MarkDuplicates --INPUT NA18562.0001.bam --INPUT NA18562.0004.bam --INPUT NA18562.0003.bam --INPUT NA18562.0002.bam --INPUT NA18562.0008.bam --INPUT NA18562.0009.bam --INPUT NA18562.0010.bam --INPUT NA18562.0005.bam --INPUT NA18562.0011.bam --INPUT NA18562.0007.bam --INPUT NA18562.0006.bam --INPUT NA18562.0012.bam --OUTPUT NA18562.md.bam --METRICS_FILE NA18562.md.cram.metrics --TMP_DIR . --REFERENCE_SEQUENCE Homo_sapiens_assembly38.fasta --REMOVE_DUPLICATES false --VALIDATION_STRINGENCY LENIENT
09:40:35.391 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/opt/conda/share/gatk4-4.6.1.0-0/gatk-package-4.6.1.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
[Tue May 19 09:40:35 GMT 2026] MarkDuplicates --INPUT NA18562.0001.bam --INPUT NA18562.0004.bam --INPUT NA18562.0003.bam --INPUT NA18562.0002.bam --INPUT NA18562.0008.bam --INPUT NA18562.0009.bam --INPUT NA18562.0010.bam --INPUT NA18562.0005.bam --INPUT NA18562.0011.bam --INPUT NA18562.0007.bam --INPUT NA18562.0006.bam --INPUT NA18562.0012.bam --OUTPUT NA18562.md.bam --METRICS_FILE NA18562.md.cram.metrics --REMOVE_DUPLICATES false --TMP_DIR . --VALIDATION_STRINGENCY LENIENT --REFERENCE_SEQUENCE Homo_sapiens_assembly38.fasta --MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP 50000 --MAX_FILE_HANDLES_FOR_READ_ENDS_MAP 8000 --SORTING_COLLECTION_SIZE_RATIO 0.25 --TAG_DUPLICATE_SET_MEMBERS false --REMOVE_SEQUENCING_DUPLICATES false --TAGGING_POLICY DontTag --CLEAR_DT true --DUPLEX_UMI false --FLOW_MODE false --FLOW_DUP_STRATEGY FLOW_QUALITY_SUM_STRATEGY --FLOW_USE_END_IN_UNPAIRED_READS false --FLOW_USE_UNPAIRED_CLIPPED_END false --FLOW_UNPAIRED_END_UNCERTAINTY 0 --FLOW_UNPAIRED_START_UNCERTAINTY 0 --FLOW_SKIP_FIRST_N_FLOWS 0 --FLOW_Q_IS_KNOWN_END false --FLOW_EFFECTIVE_QUALITY_THRESHOLD 15 --ADD_PG_TAG_TO_READS true --ASSUME_SORTED false --DUPLICATE_SCORING_STRATEGY SUM_OF_BASE_QUALITIES --PROGRAM_RECORD_ID MarkDuplicates --PROGRAM_GROUP_NAME MarkDuplicates --READ_NAME_REGEX <optimized capture of last three ':' separated fields as numeric values> --OPTICAL_DUPLICATE_PIXEL_DISTANCE 100 --MAX_OPTICAL_DUPLICATE_SET_SIZE 300000 --VERBOSITY INFO --QUIET false --COMPRESSION_LEVEL 2 --MAX_RECORDS_IN_RAM 500000 --CREATE_INDEX false --CREATE_MD5_FILE false --help false --version false --showHidden false --USE_JDK_DEFLATER false --USE_JDK_INFLATER false
[Tue May 19 09:40:35 GMT 2026] Executing as nahbu450@c002.m3c on Linux 5.14.0-611.30.1.el9_7.x86_64 amd64; OpenJDK 64-Bit Server VM 17.0.11-internal+0-adhoc..src; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: Version:4.6.1.0
INFO	2026-05-19 09:40:35	MarkDuplicates	Start of doWork freeMemory: 189912344; totalMemory: 234881024; maxMemory: 25769803776
INFO	2026-05-19 09:40:35	MarkDuplicates	Reading input file and constructing read end information.
INFO	2026-05-19 09:40:35	MarkDuplicates	Will retain up to 93368854 data points before spilling to disk.

Do you have enough quota? Just trying to figure out if it's a problem with the hpc

I mean the jobs are somethings started later because I cannot use that many cores at the same time, but if its a started job then it should finish without impact

this command should give you the job id of all processes, given the name of run:

nextflow log <name_of_run> -f hash,native_id,name,status

the second column is the job id

ok nice, thanks I will investigate further and might contact the cluster admins