Hi everyone, I’m using S3 connector to sync data from S3 into Postgresql, but in case some csv file doesn’t correct format ( delimiter & quote character doesn’t not match with S3 source config) so the sync is failed. Do we have any option to skip the file error and continue to process the rest of file in S3 ?

Hi, not yet, as wrong delimiter or quote does not necessarily mean that we can not read the file. May I ask you to provide some example?

I ask about that issue because we use S3 share folder to difference user can update csv file, in case the user use difference setting when export from excel to csv for example then the format is not consistent in S3 folder, then Airbyte can’t process whenever one file is not correct format.

@Artem Inzhyyants @kelvin I have a similar issue. The first row in the CSV contains the value as 123 for the price column but the next row contains 199.5 as the value The sync fails with the error as

CSV conversion error to int64: invalid value '199.5'

Same goes with timezone and time format

Invalid date-time: Invalid timezone offset: +0000

Anything the we can do here to avoid these issues?

in your case, I was tried to add

Manually enforced data schema

and mostly all column I use string format, then I load it into DB and do some transform later. By that way at least we can load all raws data

@Arjunsingh Yadav @kelvin is right, you can provide schema manually

@kelvin Thanks a lot for the help 🙂 Any way to enforce data schema on existing connectors(S3 as the source in this case)?

As you can see in the S3 setup guide, If set

Infer Datatypes

to False and don’t provide data schema then it will force all fields as String by default.

Gotcha party parrot Thanks @kelvin

I use DBT

cool… thanks 🙂