Hi team, has there been investigation into using parquet (instead of CSV) for AWS S3 Staging COPY e.g. when syncing from source to Snowflake - particularly with respect to compressed storage space as read/write efficiencies
u
user
09/20/2021, 9:08 PM
that a great idea airbyte rocket I think no one had request this before. I created this feature request to extend the staging https://github.com/airbytehq/airbyte/issues/6329 can you add more info if you want 😄
u
user
09/21/2021, 11:50 AM
@Liren Tu this might be able to re-use how you implemented parquet staging for databricks?