i have been trying to add another stream source to my s3 source, but i am gettting the following error when trying to define parquet files: An unknown error occurred. (HTTP 504)

HTTP 504 Error When Adding Parquet Stream to S3 Source This is a known issue. The HTTP 504 Gateway Timeout when configuring an S3 source with Parquet files is typically caused by one of the following: 1. Ingress Timeout Too Short If you're running Airbyte on Kubernetes, your ingress controller's backend timeout may be too short. Schema discovery and connection checks for large Parquet files can take longer than default timeout values. > Set appropriate backend timeout values for the Airbyte server ingress. Timeout values that are too short can lead to 504 errors in the UI when creating new sources or destinations. [Implementation Guide] 2. OOM During CHECK on Large Parquet Files If your Parquet files are large, the S3 connector may run out of memory during the CHECK phase (which tries to parse a full record), causing the process to be killed and resulting in a timeout. [S3 OOM PR] Workaround: A new opt-in config option

skip_full_check_for_parquet: true

has been introduced (in connector version

4.15.3

) that skips the full record-parse step during CHECK for Parquet streams. This prevents OOM errors on large Parquet datasets. Note the trade-off: with this flag enabled, schema mismatches or corrupt files will only surface at sync time rather than during CHECK. [S3 OOM PR] Recommended Steps 1. Check your Parquet file sizes — if they are large (hundreds of MB or more), the OOM issue is likely the cause. 2. Upgrade to S3 connector version

4.15.3

or later and enable

skip_full_check_for_parquet: true

on the affected stream if OOM is the issue. 3. Increase ingress timeout on your Kubernetes ingress controller if the timeout is hitting before the check completes. 4. Allocate more memory to the connector job if possible. Could you share more details about your deployment type (Docker/Kubernetes) and the approximate size of your Parquet files? That would help narrow down the root cause further. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.