❌ sarek/test_full_germline_aws failed 😿 Run Name sarek_test_full_germline_aws_7 Duration 7m 8s Status ❌ Failed Error Message

8:50AM INF [seqera-fusion] setting up AWS Batch spot instance monitoring provider=aws-batch
8:50AM INF [seqera-fusion] instance metadata ami-id=ami-0acc5dd6b31082def instance-id=i-03f433cc3841fc327 instance-type=c6id.16xlarge
8:50AM INF [seqera-fusion] starting incremental snapshot timer interval=300000
2026-06-25 08:50:51.104334: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-06-25 08:50:51.104670: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2026-06-25 08:50:51.107259: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2026-06-25 08:50:51.143525: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/usr/local/lib/python3.10/dist-packages/google/api_core/_python_version_support.py:275: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
2026-06-25 08:50:51.964837: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
I0625 08:50:52.896066 140684845830592 run_deepvariant.py:623] Creating a directory for intermediate results in tmp
***** Intermediate results will be written to tmp in docker. ****
I0625 08:50:52.896928 140684845830592 run...

Command Line

nextflow run '<https://github.com/nf-core/sarek>' -name sarek_test_full_germline_aws_7 -params-file '<https://api.cloud.seqera.io/ephemeral/7r1h9c6h9R4Ev04tHmPW4w.json>' -with-tower -r worktree-bump-deepvariant -profile test_full_germline_aws -resume f26ab704-e630-4ff0-96b8-1d08771c7354

Jun 25, 2026 at 8:51 AM

oh no

Is there a link to this actual run? So one can inspect the full logs?

We have a second one that worked

👍 So the deepvariant thing is solved?

Yeah, have a look at https://nfcore.slack.com/archives/C05V9FRJYMV/p1782405994053009?thread_ts=1782405994.053009&cid=C05V9FRJYMV

Great!

think it might be fusion after all sad cowboy following up with the team internally now