Hey <!everyone>! :octavia-wave: :octavia-wave:
:o...
# announcementsj
John (Airbyte)
09/07/2023, 5:58 PM
Hey <!everyone>! octavia wave octavia wave
octavia rocket Postgres source's significant improvements octavia rocket
Our Postgres source connector has seen a series of transformative enhancements:
• Performance Benchmarking Against Fivetran: We took on the challenge of optimizing our Postgres connector, which was initially 5x slower than Fivetran. Through rigorous optimization and fine-tuning, we're proud to announce that recent benchmarks show Airbyte surpassing Fivetran in performance. Read the full benchmarking analysis here.
• Reading Very Large Postgres Tables: We've delved deep into handling large tables, optimizing data replication for even the most massive tables. Our approach focuses on reading data in its natural order and efficient pagination strategies. Discover our top lessons learned here.
• Replicating Datasets of Any Size: Our Postgres connector is now more versatile than ever. From handling modest datasets to colossal databases, we've ensured seamless data replication. With an impressive throughput of up to 11 MB per second, our Postgres Source connector stands out in the industry. Learn more about our approach here.
Our journey with the Postgres connector exemplifies our broader vision at Airbyte: relentless improvement, commitment to excellence, and passion for innovation. We invite you to explore the articles above for a deeper understanding of our Postgres advancements and other innovations.
All this will be coming on MySQL in a few weeks, so stay tuned!
octavia partying Events octavia partying
• New demo for the connector builder on 09/14
• Find us at our booth at Big Data London on 09/20-21 (we also host a happy hour there) and at Big Data & AI in Paris on 09/25-26
💰 Hackathon prizes 💰
We will announce the winners next Friday!
---
As always, thank you for being a part of our community! 🚀
In the thread, you will find the changelogs.
airbyte heart 3
🦜 2
🤯 9
airbyte rocket 27
👋 24
fiesta parrot 1
💯 16
octavia partying 24
octavia muscle 1
🍾 10
👍 38
John (Airbyte)
09/07/2023, 5:58 PM
✨ New and improved features
• New Sources and Promotions
◦ 🎉 New Source: Outbrain-Amplify (#27008)
◦ 🤖 Add Clockify connector to cloud (#29146)
◦ ✨Source Drift: Migrate Python CDK to Low-code CDK (#29121)
◦ 🎉 Migrate Chargify Source Connector to Low Code (#29130)
◦ ✨ Source MailGun: Migrate Python CDK to Low-code CDK (#29122)
◦ ✨ Source Plaid: Migrate Python CDK to No-code CDK (#29127)
◦ ✨ Source BrainTree: Migrate Python CDK to LOW-code CDK (#29200)
◦ ✨ Source FreshServices : Migrate Python CDK to Low-code CDK (#29126)
◦ 🎉 Migrate Confluence Source Connector to Low Code (#29125
• New Features for Existing Connectors
◦ ✨ Source Mongodb POC: Ignore data type for discovered field uniqueness (#29168)
◦ ✨ Source Hubspot: handle connection errors (#28909)
◦ ✨ Source Zendesk Support: New Stream - Topics (#27208)
◦ ✨ Source LinkedIn Ads: Add new analytics streams (#29175)
◦ ✨ Destination BigQuery - Add v1v2 Migration (#28962)
◦ 🎉 Source Shopify: updated
OAuth Spec*.<http://myshopify.com|myshopify.com>input configurationorderitemsexpected records.jsonlDatePickerStart DateHTTP-500Internal Server Error❤️ 1
j
Jordan Fox
09/07/2023, 6:03 PMPostgres, then Mysql, what's the roadmap look for the rest of the rdbms connectors like mssql, oracle, etc?
j
John (Airbyte)
09/07/2023, 9:54 PM
MySQL should become GA in the next couple of weeks, so pretty fast.
MongoDB will be in the next month or SO.
MS SQL will be a focus in Q4, don't know exactly when it will be ready.
Oracle should be next year though, most likely 1st half.
a
Andrzej Lewandowski
09/08/2023, 6:08 AMHi, great news but what about this issue: https://github.com/airbytehq/airbyte/issues/24659. We can’t event use mysql connector with CDC mode and only way to work with it it’s a use 1.x version which is old, so all of new features are not for us 😞
a
Ayush Singh
09/08/2023, 6:51 AM@John (Airbyte) Question :-
My team is looking for a connector that can sync data from postgres to snowflake in near real time (fivetran does it for us in like 10 mins max). Have we in airbyte reached/optimized that speed ? and do we also replicate toast data ? If yes, I am going to convince my team to use Aibyte cloud instead of fivetran, please do let me know
j
John (Airbyte)
09/08/2023, 4:33 PM
@Andrzej Lewandowski @Ayush Singh will both keep you posted, checking with the team.
m
Matheus Barbosa
09/08/2023, 6:15 PMHi guys, very happy to read this!! Great job! What about the ClickHouse destination? You guys have some plans for it?
j
John (Airbyte)
09/08/2023, 8:13 PM
to be clear, we have plans for all 🙂 . but no timeline yet on Clickhouse destination
p
Prateek Mukhedkar
09/09/2023, 12:03 AM@Andrzej Lewandowski we will fix the connector. It is likely that when the 2.0 release which brought in a new Debezium version also introduced the issue of server timezone not being applied correctly when connecting to the database.
❤️ 1
Prateek Mukhedkar
09/11/2023, 4:23 AM@Ayush Singh We support TOAST columns if you allow replication identity full while configuring CDC (doc link). We have not optimized the connector for reading toast data, so you may run into out-of-memory problems during replication. But if that happens, we can give the connector extra memory. Also, we checkpoint when reading your data, both during initial and incremental syncs, so that any failure does not cause us to lose progress. And you can set the replication frequency to 15 minutes, bringing us closer to your near real-time requirements. Give it a shot and let me know how it goes!
l
Luan Carvalho
09/11/2023, 6:54 PMAre all these improvements already applied to the latest version of the Postgres source connector?
p
Prateek Mukhedkar
09/11/2023, 9:32 PM@Luan Carvalho, yes.
l
Luan Carvalho
09/11/2023, 9:35 PMTy @Prateek Mukhedkar !
r
Rocky Appiah
09/12/2023, 4:10 PMWhat version of postgres is required to take advantage of the new connect initial sync improvements?
Will pg 10.x work?
p
Prateek Mukhedkar
09/12/2023, 6:10 PM@Rocky Appiah As of now Postgres 14 and higher and we are bringing the same set of improvements to v12,13 also.
r
Rocky Appiah
09/12/2023, 6:53 PMok - should I be able to just upgrade the postgres source connector on an OSS installation? Or do I need to remove the connection and re-add
Rocky Appiah
09/12/2023, 6:53 PMThanks @Prateek Mukhedkar
g
Gregor
09/14/2023, 9:43 AMDoes GA include SSL (TPCS) support for sources and destinations? I was looking at the Oracle Source and found that there is no support for TCPS. I also haven't found an issue on Github regarding this feature.
p
Prateek Mukhedkar
09/15/2023, 7:57 PM@Gregor can't commit for Oracle just now, however other database sources including Postgres, MySQL have full SSL support
Prateek Mukhedkar
09/15/2023, 7:58 PM@Andrzej Lewandowski can you get the latest MySQL source connector and give it a go? We released a fix for 24659.
g
Gregor
09/19/2023, 12:26 PM@Prateek Mukhedkar do you know, by any chance, what's the issue with the Oracle SSL support? Just looking at the code in Github it seems like it is already implemented.
a
Andrzej Lewandowski
09/21/2023, 8:14 AM@Prateek Mukhedkar
Sync Succeeded👍 1
p
Prateek Mukhedkar
09/21/2023, 5:06 PM@Gregor it is about ensuring that we have tested the support properly for all SSL variants.
👍 1