https://nf-co.re logo
Join Slack
Powered by
I’m having trouble with timeout issues using the t...
# seqera-platform-community-help
m
I’m having trouble with timeout issues using the tower agent when trying to run a lot of jobs. Is there a way to increase the timeout time? (See thread)
Some of these Jobs on my HPC start to run, but then others fail do to timeout communicating with my HPCs slurm
Copy code
Launching run: KAUST061_plastid_from_FDB_20260304_175035
  Workflow 3oCzCbGhutW75D submitted at user workspace.

    <https://cloud.seqera.io/user/markpampuch/watch/3oCzCbGhutW75D>


Launching run: KAUST061_mito_from_FDB_20260304_175035
  Workflow 4e7t2QDOSgU34q submitted at user workspace.

    <https://cloud.seqera.io/user/markpampuch/watch/4e7t2QDOSgU34q>


Launching run: KAUST061_nuclear_from_FDB_20260304_175035
  Workflow 59GbxWUv9UwoT9 submitted at user workspace.

    <https://cloud.seqera.io/user/markpampuch/watch/59GbxWUv9UwoT9>


Launching run: KAUST063_plastid_from_FDB_20260304_175035
  Workflow 3HEZhTE2IKi8qg submitted at user workspace.

    <https://cloud.seqera.io/user/markpampuch/watch/3HEZhTE2IKi8qg>


Launching run: KAUST063_mito_from_FDB_20260304_175035
  Workflow 4MFUgQm7EmOyYH submitted at user workspace.

    <https://cloud.seqera.io/user/markpampuch/watch/4MFUgQm7EmOyYH>


Launching run: KAUST063_nuclear_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST065_plastid_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST065_mito_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST065_nuclear_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST066_plastid_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST066_mito_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST066_nuclear_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST068_plastid_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST068_mito_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST068_nuclear_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST070_euk1_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST070_euk2_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST070_plastid_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST070_nuclear_from_FDB_20260304_175035
  Workflow 2g37oiHeo9bq5u submitted at user workspace.

    <https://cloud.seqera.io/user/markpampuch/watch/2g37oiHeo9bq5u>


Launching run: KAUST070_mito_from_FDB_20260304_175035
  Workflow 3MNy5UQDBrvxCA submitted at user workspace.

    <https://cloud.seqera.io/user/markpampuch/watch/3MNy5UQDBrvxCA>


Launching run: KAUST070_mito2_from_FDB_20260304_175035
 ERROR: Failed HPC grid availability check - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST071_plastid_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST071_mito_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected


Launching run: KAUST071_nuclear_from_FDB_20260304_175035
 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected
The workaround I’ve done is to wrap the execution into a retry block
Copy code
Launching run: KAUST065_mito_from_FDB_20260304_182055, attempt 1 of 10

 ERROR: Failed storage access check - Timeout waiting for command response - Check Tower Agent is running and connected

tw launch failed (attempt 1 of 10). Retrying...
Launching run: KAUST065_mito_from_FDB_20260304_182055, attempt 2 of 10

 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected

tw launch failed (attempt 2 of 10). Retrying...
Launching run: KAUST065_mito_from_FDB_20260304_182055, attempt 3 of 10

 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected

tw launch failed (attempt 3 of 10). Retrying...
Launching run: KAUST065_mito_from_FDB_20260304_182055, attempt 4 of 10

 ERROR: Unable to connect - Timeout waiting for command response - Check Tower Agent is running and connected

tw launch failed (attempt 4 of 10). Retrying...
Launching run: KAUST065_mito_from_FDB_20260304_182055, attempt 5 of 10

  Workflow BC4pYCUz872mM submitted at user workspace.

    <https://cloud.seqera.io/user/markpampuch/watch/BC4pYCUz872mM>
But ideally it would be nice if we could modify the 
tw-agent
behaviour through CLI flags or ENV variables
5 Views
Open in Slack
PreviousNext
Powered by