Hello Channel , I have airbyte oss(appVersion: 0.50.56) on gke running from last one year ,this is the only application running on the whole cluster Suddenly i am getting 504 and not able to open the webapp all the cpu and memory request are within limit , workder nodes are 4cpu and 16gb ram My Traefik logs get 200 for initial few request for the assets and the later are retuning 499 for all the other requests My ingressroute is pointed to the webapp on port 80 kubectl port-forwarding it to local i am able to access my app perfectly and my jobs are still running This setup is behind VPN Any leads on debugging this issue will be helpful Also tried adding below to traefik spec: forwardingTimeouts: dialTimeout: 120s responseHeaderTimeout: 300s idleConnTimeout: 300s

Based on the information provided in the knowledge sources, I can offer some insights and suggestions for your issue with Airbyte OSS on GKE experiencing 504 errors. The 504 Gateway Timeout error you're encountering is likely related to timeout settings in your ingress or proxy configuration. Here are some steps you can try to debug and potentially resolve the issue: 1. Increase timeout settings: As mentioned in [this GitHub issue](https://github.com/airbytehq/airbyte/issues/45156), you can try increasing the timeout settings on your ingress. For nginx ingress, you can add the following annotations to your ingress definition:

yaml
   apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
   kind: Ingress
   metadata:
     annotations:
       <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "120"
       <http://nginx.ingress.kubernetes.io/proxy-connect-timeout|nginx.ingress.kubernetes.io/proxy-connect-timeout>: "300"
       <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "300"

You can edit your ingress using the following command:

kubectl -n <your-airbyte-namespace> edit <http://ingresses.networking.k8s.io|ingresses.networking.k8s.io> <your-ingress-name>

2. Check server logs: Since you're able to access the app via port-forwarding, the issue might be related to the ingress or networking. Check the server logs for any error messages that might provide more insight into the problem. 3. Environment variables: Ensure that your environment variables are correctly set. As mentioned in [this GitHub discussion](https://github.com/airbytehq/airbyte/discussions/40738), you might want to check the following environment variables:

global:
     env_vars:
       HTTP_IDLE_TIMEOUT: 1800s

For the webapp ingress:

webapp:
     ingress:
       annotations:
         <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: internal
         <http://nginx.ingress.kubernetes.io/proxy-body-size|nginx.ingress.kubernetes.io/proxy-body-size>: 16m
         <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: 1800
         <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: 1800

4. Check resource usage: Although you mentioned that CPU and memory usage are within limits, it might be worth double-checking the resource consumption of your pods, especially during high load periods. 5. Investigate network issues: Since your setup is behind a VPN, there might be network-related issues causing the timeouts. Ensure that all necessary ports and connections are properly configured in your VPN setup. 6. Consider upgrading: If possible, consider upgrading to the latest version of Airbyte OSS, as newer versions might have improvements or fixes for similar issues. 7. Check for failed jobs: As suggested in [this forum post](https://discuss.airbyte.io/t/376), try deleting any failed jobs using

kubectl delete

as they might impact the resources and scheduling capabilities of the cluster.

Based on the information provided in the knowledge sources, I can offer some insights and suggestions for your issue with Airbyte OSS on GKE experiencing 504 errors. The 504 Gateway Timeout error you're encountering is likely related to timeout settings in your ingress or proxy configuration. Here are some steps you can try to debug and potentially resolve the issue: 1. Increase timeout settings: As mentioned in [this GitHub issue](https://github.com/airbytehq/airbyte/issues/45156), you can try increasing the timeout settings on your ingress. For nginx ingress, you can add the following annotations to your ingress definition:

yaml    apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>    kind: Ingress    metadata:      annotations:        <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "120"        <http://nginx.ingress.kubernetes.io/proxy-connect-timeout|nginx.ingress.kubernetes.io/proxy-connect-timeout>: "300"        <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "300"

You can edit your ingress using the following command:

kubectl -n <your-airbyte-namespace> edit <http://ingresses.networking.k8s.io|ingresses.networking.k8s.io> <your-ingress-name>

2. Check server logs: Since you're able to access the app via port-forwarding, the issue might be related to the ingress or networking. Check the server logs for any error messages that might provide more insight into the problem. 3. Environment variables: Ensure that your environment variables are correctly set. As mentioned in [this GitHub discussion](https://github.com/airbytehq/airbyte/discussions/40738), you might want to check the following environment variables:

global:      env_vars:        HTTP_IDLE_TIMEOUT: 1800s

For the webapp ingress:

webapp:      ingress:        annotations:          <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: internal          <http://nginx.ingress.kubernetes.io/proxy-body-size|nginx.ingress.kubernetes.io/proxy-body-size>: 16m          <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: 1800          <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: 1800

4. Check resource usage: Although you mentioned that CPU and memory usage are within limits, it might be worth double-checking the resource consumption of your pods, especially during high load periods. 5. Investigate network issues: Since your setup is behind a VPN, there might be network-related issues causing the timeouts. Ensure that all necessary ports and connections are properly configured in your VPN setup. 6. Consider upgrading: If possible, consider upgrading to the latest version of Airbyte OSS, as newer versions might have improvements or fixes for similar issues. 7. Check for failed jobs: As suggested in [this forum post](https://discuss.airbyte.io/t/376), try deleting any failed jobs using

kubectl delete

as they might impact the resources and scheduling capabilities of the cluster. If none of these steps resolve the issue, you may need to dive deeper into your specific setup and configurations. Consider opening a new issue on the [Airbyte GitHub repository](https://github.com/airbytehq/airbyte) with detailed information about your setup, logs, and the steps you've already taken to troubleshoot.

Sorry @Atharva Gunjkar, I'm not sure what you mean. Please try again. 👋