@kapa.ai It seems like when syncing certain sources they tend to output way too many parquets. Is there any way I can configure my destinations to output less parquet files but larger ones?

Unfortunately, based on the available knowledge sources, there is no user-configurable option to control the number of output files or set a larger chunk/part size for Parquet (or other format) destinations like S3 or GCS. A few relevant points: • Airbyte automatically partitions output files, targeting a size of ~200 MB compressed or lower per file. This behavior is fixed and not configurable. [S3 setup guide] • The

part_size_mb

option that previously existed was deprecated to enable parallel stream syncs to S3 and other bucket destinations. [S3 parquet parts forum] • A community member asking the same question about chunk size was confirmed by Airbyte support that there is no way to configure this. [S3 chunk size forum] What you can configure for Parquet output are internal Parquet formatting parameters (not file count/size), such as: | Parameter | Default | Description | | - | - | - | |

block_size_mb

| 128 MB | Row group size buffered in memory | |

max_padding_size_mb

| 8 MB | Max padding to align row groups | |

page_size_kb

| 1024 KB | Page size for compression | [S3 Parquet config; GCS Parquet config] These parameters affect internal Parquet structure but will not reduce the number of output files. If reducing file count is important to your workflow, you would need to handle file consolidation downstream (e.g., using a post-processing step in your data warehouse or pipeline). 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.