@kapa.ai If we have 10GB + data in mongo then while making connection its giving 504 error, any limitations of airbyte?

The 504 Gateway Timeout error you're experiencing with a large MongoDB source is a known issue in Airbyte, and it's primarily related to schema discovery performance, not a hard data size limit. Here's what's happening and how to address it: Why It Happens When creating a connection, Airbyte runs a schema discovery (Discover) phase that executes

$sample

aggregation pipelines concurrently across all collections in your MongoDB database. With large collections (10GB+), these parallel queries are I/O-intensive and can take a long time, causing the request to exceed Airbyte's timeout and return a 504 error. [MongoDB troubleshooting] Relevant Airbyte Cloud Limits For reference, Airbyte Cloud enforces these limits (not directly related to your 504, but worth knowing): • Max streams returned by a source in a discover call: 1,000 • Max streams configured to sync in a single connection: 1,000 • Max size of a single record: 20MB (MongoDB's own BSON limit is 16MB) [Cloud limits] How to Fix the 504 Timeout Reduce Discovery Workload 1. Disable schema enforcement — Set Schema Enforced to

false

. This skips the heavy sampling and only checks for the

_id

field, dramatically reducing load. [MongoDB troubleshooting] 2. Reduce the Discovery Sample Size — Lower the default from 10,000 documents to something like 1,000 to reduce query load. [MongoDB troubleshooting] Isolate Discovery Traffic 3. Route reads to a secondary node — Add

readPreference=secondaryPreferred

to your MongoDB connection string to avoid hitting the primary. [MongoDB alternatives] 4. Limit the number of databases/collections — Remove unnecessary databases from your source config to reduce the total collections discovered. [MongoDB alternatives] Increase Airbyte's Timeout (Self-Managed) If you're self-hosting Airbyte, you can increase the server timeout via Helm chart values:

# Helm chart V2
server:
  httpIdleTimeout: 20m
  extraEnv:
    - name: READ_TIMEOUT
      value: 30m

[Scaling Airbyte] The most immediate fix is typically disabling schema enforcement or reducing the sample size, as these directly address the root cause of the timeout. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.