@kapa.ai I get a 504 Gateway Time-out error during the

discover_schema

step using the

airbyte/source-sftp-bulk

connector on Airbyte 2.0.1. Can you help ?

504 Gateway Timeout During Schema Discovery with source-sftp-bulk A 504 error during

discover_schema

typically means the schema discovery process is taking longer than your proxy/ingress timeout allows. Here are the relevant fixes: 1. Increase Airbyte Server Timeouts (Helm/Kubernetes) If you're running Airbyte on Kubernetes via Helm, update your `values.yaml`:

server:
  httpIdleTimeout: 20m
  extraEnv:
    - name: READ_TIMEOUT
      value: 30m

[Schema Discovery Timeouts] 2. Increase Nginx Ingress Timeouts If Airbyte sits behind an nginx ingress controller, the default timeout is 60 seconds, which is often too short for schema discovery. Add these annotations to your Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: airbyte-ingress
  namespace: airbyte
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: '600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '600'

[GA4 Discovery Timeout Fix] 3. Docker Deployment: Increase Proxy Timeout If you're on a Docker-based deployment, you can set the

BASIC_AUTH_PROXY_TIMEOUT

environment variable:

BASIC_AUTH_PROXY_TIMEOUT=600

[Nginx 504 Forum Post] 4. Known SFTP-Specific Issue: SSH Keepalive There is a known issue with

source-sftp-bulk

where large file transfers (e.g., ~375 MB) can cause a

paramiko.buffered_pipe.PipeTimeout

because intermediate network devices (firewalls, NAT, load balancers) drop idle SSH connections. A fix has been submitted that adds SSH keepalive packets every 60 seconds to prevent this. [SFTP Keepalive Fix] In the meantime, as a workaround: • Check if your SFTP server or network has aggressive idle/session timeout settings and adjust them server-side. • Verify whether smaller files transfer successfully through the same pipeline. • Ensure there are no firewalls or proxies between Airbyte and your SFTP server that might drop long-running connections. [SFTP Bulk Issue] Summary If the issue persists after applying these changes, check your browser's network tab for the exact API response on the

discover_schema

call, and review Airbyte server logs for more detail. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai Is it possible to skip the

discover_schema

instead ?

The knowledge sources do not contain enough information to confirm a direct "skip discover" option for

source-sftp-bulk

specifically. However, based on what the knowledge sources do describe, there is a relevant alternative: Use the "Schemaless" Schema Option For file-based sources (which

source-sftp-bulk

is), Airbyte offers a schemaless schema option that avoids a time-consuming

discover

phase entirely. Instead of inferring the schema from your files, it uses a hardcoded schema of

{"data": object}

, wrapping each record under a single

data

key. > "This option avoids a time-consuming or inaccurate discover phase and guarantees that everything ends up in your destination, at the expense of Airbyte being able to structure the data into different columns." [Schemaless Sources] Trade-offs to be aware of: • ❌ No typed columns in the destination — all data lands in a single

data

field • ❌ Column selection and schema evolution features are not available • ✅ Discovery is fast and reliable • ✅ All records are guaranteed to reach the destination Whether this option is exposed in the

source-sftp-bulk

connector's configuration UI is something you'd need to verify directly in the connector settings. If it is available, it would be the closest thing to "skipping" schema discovery. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.