Hey Airbyte! I'm looking to test Airbyte as a replacement / companion to an ETL framework we've built in-house. First question I have is how does Airbyte handle bad input data (e.g. csvs with malformed rows)?

@George Claireaux — in the general case this behavior is up to the connector implementations. In some cases it may make sense for the connector to drop bad data from the underlying source (e.g: if a CSV row is malformed or is just whitespace), in other cases it may make sense to fail the sync entirely (e.g: the CSV file is not actually a CSV file). However, it is expected that all data output by the connector is well formed. Any data that is not well formed will be considered “debugging” output that Airbyte displays in logs, but will not be sent to the destination. are you asking specifically about what the file source connector does int he case that a CSV with malformed rows is encountered?

Thanks @s, very helpful answer. I was asking more generally but I think you covered that example well also. The question arose because this is something we have built into our bespoke tool in order to define a tolerance for bad data before failing over completely so just wanted to understand how/if there was a similar implementation here. Thanks!