Hello everyone. I'm having a strange failure with ...
# replication-troubleshooting
r
Hello everyone. I'm having a strange failure with Airbyte deployed to Kubernetes. When trying to setup any connection or source, the
check_connection
calls are retuning a "upstream request timeout" error and I'm currently not able to understand where is it coming from because there are absolutely no errors in the logs, even with DEBUG enabled. I've logged an issue here, any help is appreciated: https://github.com/airbytehq/airbyte/issues/20963. Can anyone tell me how can I further debug this issue? I've exec'd into the server container and it can communicate and connect to all the other hosts (temporal and webapp, for example). I'm stuck and have no idea how to debug it further.
n
Hi Renato! Thanks for your patience - I was out last week for the holidays and am catching up with messages now. Let me look into this a bit further and get back to you. Thanks for all the details - they are really helpful!
r
Thank you @Nataly Merezhuk (Airbyte)! I'm happy to keep debugging, I just need a pointer because I'm stuck at this stage.
Hi @Nataly Merezhuk (Airbyte)! I noticed you've closed the issue on Github, I'm happy to continue the conversation here. Do you have any idea how can I investigate this further? There's nothing special with my setup, I simply installed the latest version of the helmchart, pointed it to a working and reachable Postgres and all the pods are healthy and able to communicate with each other. I really don't know where to go from here, the logs are set to DEBUG and I still don't see any error.
Hi @Nataly Merezhuk (Airbyte), I'm sorry for bothering you but I really don't know how to proceed on debugging this problem. Is there any way you can help me?
n
Yes - there are a few things to check here: 1. The server might be timing out, take a look at this: https://kitcharoenp.github.io/nginx/2021/10/21/upstream-timed-out.html 2. If that doesn't help - could you attach the full logs? 3. Look in the network tab and see what you're getting there
r
Hi Nataly. I've attached the full logs to the github issue, should I copy it here? The call is timing out after 3 seconds which is way below any timeout set anywhere through the system. From the network tab, the request to
check_connection
endpoint is returning a 504 Gateway Timeout with this in the payload:
upstream request timeout
n
Gotcha, I think this maybe a configuration issue with your instance. Where are you hosting the cluster?