<!channel> Hey everyone :wave:!
here are the updat...
# announcementsj
John (Airbyte)
04/13/2021, 6:55 PM
<!channel> Hey everyone 👋!
here are the updates for the past week:
• Welcome to @Mason Wheeler who's joining our team as senior software engineer 🎉 🎉 ! We're so excited to have him on board!
• Airbyte has become an official package provider of Airflow!
• Progress on connectors:
◦ Oracle DB is now supported as a source connector, thanks to @[DEPRECATED] Marcos Marx
◦ Some fixes
• Progress on Core (0.20.0):
◦ Change Data Capture (CDC) is now supported for Postgres, thanks to @Jared Rhizor (Airbyte) and @charles. We will now expand it to MySQL and MSSQL in the coming weeks.
◦ When displaying the schema for a source, you can now search for table names, thanks to @Artem Astapenko
◦ Better feedback UX when manually triggering a sync with “Sync now”
• 3 months ago, we published our OKRs for Q1 2021. Here’s an article about how we performed on those OKRs, and also about our goals for Q2. You will see how much the community has grown, which is super exciting!
• Don’t forget to register to our 2nd community call on Wednesday 05/05 at 3pm PST with @Jins Kadwood, who will show us how AgriDigital is leveraging Airbyte for their caching strategy.
🚀 3
airbyte 7
airbyte star 4
🔥 11
😮 5
👍 15
airbyte rocket 14
h
Harsh M.
04/13/2021, 7:06 PMQuestion on CDC for PostgreSQL, does it not support tables with unique key only? Does it have to have primary key in order to support CDC?
j
Jared Rhizor (Airbyte)
04/13/2021, 7:23 PM
Our current implementation only supports primary keys for CDC. We’ve considered a few options for non-pk tables but decided to release our initial release without it.
Jared Rhizor (Airbyte)
04/13/2021, 7:25 PM
If you create an issue on GitHub and describe your use case we’ll have a place to track interest/progress. It’s not on our short term roadmap (within CDC we’re prioritizing adding support for tables with primary keys in other databases), but if there’s a large demand for non-pk tables we may end up prioritizing it differently.
Jared Rhizor (Airbyte)
04/13/2021, 8:32 PM
Also, if anyone is setting up Postgres CDC and has questions or want to pair with an engineer during the setup process, please reach out! We’d love to get more feedback on the Postgres implementation before launching other CDC implementations.
👍 6
h
Harsh M.
04/13/2021, 9:05 PMOur database is a mess 😄 Majority of our tables are designed without primary key and have multiple unique keys instead 🙄. I was able to test CDC briefly for a table with PK and it seems to be working as expected. Unfortunately, I can't use this feature until unique key support is added. I will create an issue. Thank you!
👍 2
c
charles
04/13/2021, 9:08 PM
Hi Harsh. "turning on" support for tables without PKs isn't that big a deal.
charles
04/13/2021, 9:09 PM
The reason we didn't do it by default was that while we can pull the data out in a CDC-like manner pretty easily, it's not obvious that we will be able to de-dupe well for you.
charles
04/13/2021, 9:09 PM
There are some strategies we could use, like falling back on postgres' row id.
charles
04/13/2021, 9:10 PM
Or just letting you specify the columns that we should treat as a primary key yourself.
👍 2
charles
04/13/2021, 9:10 PM
Can you describe what output you expect in these cases? Are you expecting your records to be deduped? and if so how would you want to go about configuring that?
d
Devon Seitz
04/13/2021, 9:17 PMI am assuming when its a core update we need to update airbyte as a whole
Devon Seitz
04/13/2021, 9:17 PMupdating the postgres package is not enough
c
charles
04/13/2021, 9:40 PM
correct.
charles
04/13/2021, 9:40 PM
you need to upgrade core and the the postgres package.
c
Cody K.
04/14/2021, 3:16 AMGreat work @Jared Rhizor (Airbyte) on:
• Progress on Core (0.20.0):
◦ Change Data Capture (CDC) is now supported for Postgres, thanks to @Jared Rhizor (Airbyte) and @charles. We will now expand it to MySQL and MSSQL in the coming weeks.
Looking forward to implementing MSSQL CDC when available!
👍 1
m
Marco Sollie
04/14/2021, 8:44 AMWe're interested in testing out MySQL CDC into BQ. Looking forward to this and definitely will reach out to you during set up. Thanks so much!
h
Harsh M.
04/14/2021, 12:45 PMI like the idea of specifying columns to be treated as a primary key, may be MD5 hash can be used? I understand you guys have other high priority items in your roadmap so I'll create a ticket.
3 Views