https://cloudposse.com logo
Join Slack
Powered by
<@U05Q5J0GQJ1> any insights on what the OpenTofu p...
# terraform
e
@James Humphries any insights on what the OpenTofu provider rate limits are?
this 1
j
What do you mean by provider rate limits?
OpenTofu doesn't throttle or apply rate limits to any provider. So I'd like to understand the situation a bit more if possible
e
We were talking with someone yesterday, and they said their deployments were failing due to getting rate limited downloading providers. I was surprised to hear it.
Is there a chance that even using the opentofu binary that they could be pulling from the hashicorp registry?
(I thought provider registry were dynamically rewritten)
j
OpenTofu should only ever use the opentofu registry. When you run 
tofu init
we re-write your statefile to use registry.opentofu.org I can talk a bit about how it works Our OpenTofu registry (registry.opentofu.org etc) is hosted as static files in cloudflare r2 (their s3-like service) and we do not put any servers in front of it. It's just a r2 bucket serving on a custom domain. We have turned off all rate limiting rules from cloudfare, all DDOS protection, anything that could rate limit people. For all providers in the OpenTofu registry(hashicorp namespace AND everything else) we serves a json file that points to the actual binaries for the providers in github releases, and this is where the 
tofu
binary will download the providers from. (This is most likely where rate limiting is happening, although I've never seen anyone hit it). If for some reason you were still downloading providers from the terraform registry, that works in 2 ways: ā€¢ For 
hashicorp/*
namespaced providers, hashicorp serves the binaries up directly from their own servers ā€¢ For any other namespaced provider, its the same as opentofu, a json file from the registry that points to github releases.
In the future if this happens, you can set the env var 
TF_LOG=debug
and look at which http requests are failing to get an idea of what is being rate limited. I also recommend that if you have an OTEL-compatible tool to turn on OTEL tracing for 
tofu
and that will show you exactly what http requests are happening under the hood, and which are failing. You can find docs about how to enable this here: https://opentofu.org/docs/v1.10/internals/tracing/
e
@James Humphries this is extremely helpful! We can narrow things down knowing how this is supposed to work, and it makes things easier.
j
No problem šŸ™‚ If you hit any issues feel free to DM me. I try and keep an eye on a bunch of slacks. I may be more responsive to the CNCF slack or the Spacelift workspaces because I don't have this sweetops on my phone. I'm always happy to look at things on my side too to check if everything is okay. I just checked our cloudflare logs and cant find any issues there.
šŸ‘ 2
e
So with your help we were able to track down the issue and it turns out to be based on stale information.
The last attempt was in june/july of last year, prior to the rate limit issue resolution.
j
Ahh okay, let me know if they still hit this issue or not. thanks for letting me know
2 Views
Open in Slack
PreviousNext
Powered by