Why in the universe does Airbyte completely abandon 50K+ rows of data (syncing from Slack to MotherDuck) just because ONE FREAKING MESSAGE fails to meet the schema that Airbyte setup in the Destination?! This makes zero sense.

Sorry @Jay Stevens, I'm not enabled for this channel yet. Please contact your workspace admins to help get me set up. 👋

Hi Jay, I understand your concerns regarding this. The reason Airbyte fails the whole sync rather than skipping the bad record comes down to data integrity. If Airbyte silently skipped records, you could end up with missing data in your destination with no clear indication of what was dropped. That said, you can reduce how often this happens by configuring your schema change handling settings on the connection. Setting it to automatically propagate schema changes means Airbyte adapts when new fields appear rather than failing

Let me try that. I guess one thing I'd like to be able to do re: bad records is to have an option (regardless of the specific integration) to save/keep the good records and throw the bad ones into a separate bucket. Seems crazytown to extract 100K records (just revisited the logs and it was eventually twice what it was when I stopped watching it yesterday) and then they just go nowhere (bandwidth wasted, etc.) only to have it do the same thing 19 more times (there were 20 total attempts at the sync). I'll revisit the schema changes, but based on the message I was seeing, I'm a little sus that this will work.