Kriti (Postman)
08/31/2021, 6:43 AMuser
08/31/2021, 7:16 AMWe have a bunch of s3 files at a prefix - I am assuming the Sync would iterate over them and write the data to destination.Yes. You can specify a path pattern: https://docs.airbyte.io/integrations/sources/s3#path-pattern
user
08/31/2021, 7:18 AMuser
08/31/2021, 7:18 AMHow would Airbyte handle failure in sync of 1 (or more) s3 file?It depends on the type of failure. For errors showing up in the schema detection stage, if there are files with inconsistent schema, the S3 source will fail immediately. For errors showing up in the data syncing stage: https://docs.airbyte.io/faq/data-loading#what-happens-to-data-in-the-pipeline-if-the-destinat[…]with-duplicate-data-when-the-pipeline-is-reconnected
user
08/31/2021, 7:21 AMuser
08/31/2021, 7:23 AMuser
08/31/2021, 7:23 AMsome error
user
08/31/2021, 7:26 AM