https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> i'm trying to make check on databri...
# replication-ask-ai
s
@kapa.ai i'm trying to make check on databricks destination but fails with An unknown error occurred. (HTTP 504), and no actual error message. by looking at console log i see this Failed to load resource: the server responded with a status of 504 (Gateway Time-out). and credentials are correct what may have happened and went wrong? are there any logs for this i can check? i'm running airbyte OSS with abctl on azure vm
k
HTTP 504 Gateway Timeout on Databricks Destination Check The 504 Gateway Timeout you're seeing is a common issue in Airbyte, and it typically means the connection check is taking longer than the proxy/ingress timeout allows — not necessarily that your credentials are wrong. Likely Causes 1. Databricks cluster spin-up time: If your Databricks cluster is not already running when Airbyte tries to connect, it needs to spin up first. Airbyte may time out while waiting for the cluster to become ready. The connector works fine once the cluster is already in a running state. [Databricks timeout forum] 2. Nginx ingress timeout: When running Airbyte with 
abctl
(which uses an nginx ingress), the default nginx timeout is 60 seconds. If the connection check takes longer than that, you'll get a 504 even if the underlying operation eventually succeeds. [nginx timeout issue] How to Fix Option 1: Ensure your Databricks cluster is already running before triggering the connection check in Airbyte. This is the simplest workaround. [Databricks timeout forum] Option 2: Increase the nginx ingress timeout by adding annotations to your ingress resource:
Copy code
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: airbyte-ingress
  namespace: airbyte
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: '600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '600'
spec:
  ingressClassName: ingress-nginx-internal
  ...
[nginx timeout fix] Option 3: Set the 
BASIC_AUTH_PROXY_TIMEOUT
environment variable (for Docker-based deployments):
Copy code
BASIC_AUTH_PROXY_TIMEOUT=600
[proxy timeout env var] Where to Find Logs Since you're on Airbyte OSS, you can check logs in several ways: • In the UI: Navigate to the connection's Job History tab → click the three grey dots next to a sync → select View logs. Search for 
Error
, 
Exception
, or 
Fail
. • Download logs: Use the Download logs option from the same menu to get the full log file locally. • Pod logs: Since 
abctl
runs on Kubernetes, you can also check the logs of the relevant pods (e.g., 
airbyte-worker
, 
airbyte-server
) using 
kubectl logs
. [browsing logs docs] The 504 is most likely a timeout issue rather than a credentials problem, so I'd start by making sure your Databricks cluster is already running before testing the connection. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai databricks is running, can you give me commands for option 2 & 3. where to look forfiles too
k
Fixing the 504 Timeout — Commands & File Locations Option 2: Increase Nginx Ingress Timeout (Kubernetes/abctl) Since you're using 
abctl
, first get access to your cluster:
Copy code
# Get cluster name
kind get clusters

# Export kubeconfig
kind export kubeconfig --name airbyte-abctl

# Check current ingress
kubectl get ingress -n airbyte-abctl
Then patch the ingress with increased timeouts:
Copy code
kubectl annotate ingress airbyte-ingress \
  -n airbyte-abctl \
  nginx.ingress.kubernetes.io/proxy-read-timeout="600" \
  nginx.ingress.kubernetes.io/proxy-send-timeout="600" \
  --overwrite
Or apply a full ingress YAML. Create a file (e.g., 
airbyte-ingress-patch.yaml
) with:
Copy code
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: airbyte-ingress
  namespace: airbyte-abctl
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: '600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '600'
spec:
  ingressClassName: ingress-nginx-internal
Then apply it:
Copy code
kubectl apply -f airbyte-ingress-patch.yaml
[nginx timeout fix] Option 3: Set BASIC_AUTH_PROXY_TIMEOUT (Docker .env file) > ⚠️ This option applies to Docker Compose deployments, not 
abctl
. Since you're using 
abctl
, Option 2 is the more relevant fix for your setup. If you were on a Docker deployment, the 
.env
file is located at:
Copy code
~/airbyte/.env
You would edit it and set:
Copy code
BASIC_AUTH_PROXY_TIMEOUT=600
Then restart Airbyte:
Copy code
docker-compose down && docker-compose up -d
[proxy timeout env var] Summary For your abctl on Azure VM setup, focus on Option 2. The key commands are:
Copy code
kind get clusters
kind export kubeconfig --name airbyte-abctl
kubectl get ingress -n airbyte-abctl
kubectl annotate ingress airbyte-ingress -n airbyte-abctl \
  nginx.ingress.kubernetes.io/proxy-read-timeout="600" \
  nginx.ingress.kubernetes.io/proxy-send-timeout="600" \
  --overwrite
Note that the exact ingress and namespace names may differ slightly — use 
kubectl get ingress -n airbyte-abctl
to confirm the actual names in your deployment. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai my ai agent told me to 1. Raise the limit via values.yaml + reinstall
Copy code
cat > ~/values.yaml <<'EOF'
jobs:
  resources:
    requests:
      cpu: "250m"
      memory: "1Gi"
    limits:
      cpu: "2"
      memory: "4Gi"
EOF
abctl local install --values ~/values.yaml
do you think it's a good idea and might solve the problem? because looking at logs it said OOM killed on databricks check
k
On the values.yaml Resource Fix for OOM The suggestion is partially correct but incomplete, and may not fully solve the problem based on known issues in the knowledge sources. The Problem with That Config The 
jobs.resources
path in 
values.yaml
only sets resources for sync jobs, not for 
check
, 
discover
, or 
spec
jobs. Since your OOM is happening during a Databricks destination check, those limits won't apply. [GitHub Issue] More Complete Fix You need to set all job-type-specific variables. A more complete 
values.yaml
would be:
Copy code
global:
  env_vars:
    JOB_MAIN_CONTAINER_CPU_REQUEST: "250m"
    JOB_MAIN_CONTAINER_CPU_LIMIT: "2"
    JOB_MAIN_CONTAINER_MEMORY_REQUEST: "1Gi"
    JOB_MAIN_CONTAINER_MEMORY_LIMIT: "4Gi"
    CHECK_JOB_MAIN_CONTAINER_CPU_REQUEST: "250m"
    CHECK_JOB_MAIN_CONTAINER_CPU_LIMIT: "2"
    CHECK_JOB_MAIN_CONTAINER_MEMORY_REQUEST: "1Gi"
    CHECK_JOB_MAIN_CONTAINER_MEMORY_LIMIT: "4Gi"
    DISCOVER_JOB_MAIN_CONTAINER_CPU_REQUEST: "250m"
    DISCOVER_JOB_MAIN_CONTAINER_CPU_LIMIT: "2"
    DISCOVER_JOB_MAIN_CONTAINER_MEMORY_REQUEST: "1Gi"
    DISCOVER_JOB_MAIN_CONTAINER_MEMORY_LIMIT: "4Gi"
    SPEC_JOB_MAIN_CONTAINER_CPU_REQUEST: "250m"
    SPEC_JOB_MAIN_CONTAINER_CPU_LIMIT: "2"
    SPEC_JOB_MAIN_CONTAINER_MEMORY_REQUEST: "1Gi"
    SPEC_JOB_MAIN_CONTAINER_MEMORY_LIMIT: "4Gi"
Then reinstall:
Copy code
abctl local install --values ~/values.yaml
[GitHub Issue] ⚠️ Additional Caveat There is also a known open bug where resource limits set via 
values.yaml
and env vars are ignored by the 
workload-launcher
in newer Airbyte versions (including 
abctl
deployments), with pods defaulting to hardcoded values regardless of configuration. [GitHub Issue] If the above doesn't work, a confirmed workaround is to set resources directly in the Airbyte database:
Copy code
update actor_definition
set resource_requirements = '{
  "jobSpecific": [
    {
      "jobType": "check_connection",
      "resourceRequirements": {
        "memory_limit": "4Gi",
        "memory_request": "2Gi"
      }
    }
  ]
}'::jsonb,
updated_at = now()
where id = '<your-databricks-actor-definition-id>';
[GitHub Issue] To find the Databricks actor definition ID, connect to the Airbyte internal database and run:
Copy code
select * from actor_definition where docker_repository like '%databricks%';
[Connector Resources Docs] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by