Hello everyone, and expanding on the above ^ ... Now that we've launched Destinations V2 for Snowflake and Bigquery (other destinations coming soon), we are able to more easily make adjust how data is represented in your data warehouse's final tables, based on your feedback. The first area of investigation is how we represent deletions from CDC sources (e.g. from

source-postgres

). We are conducting a survey to learn if Airbyte users prefer: • CDC deletes should be removed from the final tables • CDC deletes should remain in the final table, but with a 'soft-detele' or 'tombstone' column (e.g.

deleted_at=2022-01-01

or

null

Please check out the Github Discussion on the topic (https://github.com/airbytehq/airbyte/discussions/31242) and vote! Please add comments to explain your preference! 📝 Thanks!

@King Ho, FYI: new BQ connector.

@Evan Tahler is there a timeline for the other destinations yet? Really hoping mssql and postgres are on the cards soon, looking forwards to testing these out as i think it will massively help with some custom dbt's we are doing.

@Koen Verburg niceeee will read.

is this related to the error I get when attempting to configure a mysql destination?

{"type":"<https://reference.airbyte.com/reference/errors#resource-not-found>","title":"resource-not-found","status":400,"detail":"Could
│ not find a resource for: 50ac2a60-fb51-4512-a12c-73765a042f8e"}

io.airbyte.workers.exception.WorkerException: Could not find image: airbyte/destination-mysql:0.2.0

@Josh Jeffries - We are aiming to move all the JDBC destinations to Destinations V2 by the end of the year. No firm commitments, but that is the goal.

@Evan Tahler... Done. Soft deletes but with a tweak. Not in final table but a similar table with just deleted logs.

tombstone

I observe a different behaviour than explained here: I have hubspot source and Bigquery Destionation V2. I am using “Incremental Append + Deduped” strategy, however when the object in Deals/Contacts is deleted in the source it remains in the destination unchanged. If I do complete refresh then data is correct. What I am doing wrong and how to get the expected behaviour?

@Denis Lazarenko the discussion is about CDC sources to Destinations. Hubspot is an API source and I’m not sure if right now we want to discuss the future behavior of these. As you describe it looks the correct behavior for the connection you have.

Thank you for your response!

But the difference between cdc sources and other sources that handle deletes is /strange/. That’s called out in the github discussion as well.