Liren, I didn't realise we were using the S3 client under the hood for the GCS destination. Is that because the blob apis are actually compatible?
09/17/2021, 2:23 AM
Yes. Since S3 is so successful, it becomes kind of a protocol.
GCS and DigitalOcean both support it.
By reusing the S3 client, we was able to reuse all the S3 writers and support all four formats on GCS. Otherwise, Avro and Parquet formats can be tricky.