:question: Is there a 15 minute builtin cooldown i...
# atmos
z
❓ Is there a 15 minute builtin cooldown if atmos fails to download a source when JIT-ing?
I had a pipeline that basically looked like it froze then 15 minutes later I get an error
Copy code
05:54:24   INFO  Running hooks event=before.terraform.apply
06:09:24  # Error
06:09:24
06:09:24  **Error:** provisioner failed auto-provision component source: source
06:09:24  provisioning failed: source provisioning failed: failed to download file: error
06:09:24  downloading '<https://gitlab>...
 git command
06:09:24  exited
06:09:24  with non-zero status: /usr/bin/git exited with -1: Cloning into
06:09:24  '/tmp/getter362016375/temp'... fatal: unable to access
06:09:24  '<https://gitlab>...
The requested URL returned error:
06:09:24  504
06:09:26   INFO  Running hooks event=before.terraform.apply
06:09:28  ✓ Auto-provisioned source to .workdir/terraform/...
06:09:28
06:09:28  Initializing the backend...
Something with go-getter I'm guessing
asked copilot
Copy code
The 15 minutes comes from 3 sequential git commands in pkg/downloader/get_git.go (git ls-remote, git clone, git submodule update), each of which can hang for the full 5-minute context timeout set in internal/exec/terraform_execute_helpers.go when the remote is unreachable. There's no per-command timeout, so if all 3 hang you get exactly 15 minutes before atmos gives up. The fix would be adding git-level timeout flags (e.g. http.lowSpeedTime for HTTPS or ConnectTimeout for SSH) to each command so git itself enforces a shorter deadline rather than relying solely on the Go context.
might be something worth throwing in some retries or throwing more logging anyway
e
I do believe we have retries
Did it eventually succeed?
It should have been fatal if it didn't
My guess is the gitlab instance was unreachable
z
It wasn't unreachable for 15 minutes - but sometimes it is flaky
It eventually succeeded due to workflow retries, but I think that specific atmos command failed
It was just interesting it was exactly 15 minutes and hung silently
e
Do you have this configured?
z
I think only at the workflow level. Will need to double check. Probably not
Do you think it'll be configurable to help it fail faster?
e
Yes, that is my assertion
Alternatively we might need to expose a timeout
You know the drill - happy to harden atmos for this failure condition
z
Yeah yeah. Just wanted to make sure it wasn't our flaky environment in general
e
Well, I do think it is a flaky environment :-) but I haven’t seen that particular error before
if you can
--use-version
to help validate this alleviates problems we can merge
z
Yeah...... not sure how I will replicate. Maybe hostfile