Hey <!everyone>! :octavia-wave: :octavia-wave: :o...
# announcements
j
Hey <!everyone>! octavia wave octavia wave octavia rocket Postgres source's significant improvements octavia rocket Our Postgres source connector has seen a series of transformative enhancements: • Performance Benchmarking Against Fivetran: We took on the challenge of optimizing our Postgres connector, which was initially 5x slower than Fivetran. Through rigorous optimization and fine-tuning, we're proud to announce that recent benchmarks show Airbyte surpassing Fivetran in performance. Read the full benchmarking analysis here. • Reading Very Large Postgres Tables: We've delved deep into handling large tables, optimizing data replication for even the most massive tables. Our approach focuses on reading data in its natural order and efficient pagination strategies. Discover our top lessons learned here. • Replicating Datasets of Any Size: Our Postgres connector is now more versatile than ever. From handling modest datasets to colossal databases, we've ensured seamless data replication. With an impressive throughput of up to 11 MB per second, our Postgres Source connector stands out in the industry. Learn more about our approach here. Our journey with the Postgres connector exemplifies our broader vision at Airbyte: relentless improvement, commitment to excellence, and passion for innovation. We invite you to explore the articles above for a deeper understanding of our Postgres advancements and other innovations. All this will be coming on MySQL in a few weeks, so stay tuned! octavia partying Events octavia partyingNew demo for the connector builder on 09/14 • Find us at our booth at Big Data London on 09/20-21 (we also host a happy hour there) and at Big Data &amp; AI in Paris on 09/25-26 💰 Hackathon prizes 💰 We will announce the winners next Friday! --- As always, thank you for being a part of our community! 🚀 In the thread, you will find the changelogs.
airbyte heart 3
🦜 2
🤯 9
airbyte rocket 27
👋 24
fiesta parrot 1
💯 16
octavia partying 24
octavia muscle 1
🍾 10
👍 38
New and improved featuresNew Sources and Promotions🎉 New Source: Outbrain-Amplify (#27008) ◦ 🤖 Add Clockify connector to cloud (#29146) ◦ Source Drift: Migrate Python CDK to Low-code CDK (#29121) ◦ 🎉 Migrate Chargify Source Connector to Low Code (#29130) ◦ Source MailGun: Migrate Python CDK to Low-code CDK (#29122) ◦ Source Plaid: Migrate Python CDK to No-code CDK (#29127) ◦ Source BrainTree: Migrate Python CDK to LOW-code CDK (#29200) ◦ Source FreshServices : Migrate Python CDK to Low-code CDK (#29126) ◦ 🎉 Migrate Confluence Source Connector to Low Code (#29125New Features for Existing Connectors Source Mongodb POC: Ignore data type for discovered field uniqueness (#29168) ◦ Source Hubspot: handle connection errors (#28909) ◦ Source Zendesk Support: New Stream - Topics (#27208) ◦ Source LinkedIn Ads: Add new analytics streams (#29175) ◦ Destination BigQuery - Add v1v2 Migration (#28962) ◦ 🎉 Source Shopify: updated
OAuth Spec
, allow
*.<http://myshopify.com|myshopify.com>
shop names for
input configuration
(#28291) ◦ Source S3 (v4): Set decimal_as_float to True for parquet files (#29342) ◦ [file-based cdk] S3 file format adapter (#29353) ◦ Switch redshift staging to async mode (#28619) ◦ 🎉 Source Facebook Marketing - add new Custom Audience stream (#29412) ◦ Amazon Seller Partner: new stream
orderitems
(#29210) 🚨 Security & Breaking changes🚨 Destination Snowflake: Remove GCS/S3 Staging. (#29236) 🐛 Bug fixes🐛 Source Close-Com: fix
expected records.jsonl
formatting (#29161) • 🐛 Source Close-Com: fix
DatePicker
format issue with
Start Date
(#29206) • 🐛 Source Google Sheets: added logic to skip modified sheet (#29246) • 🐛 Source Marketo: fix expected records for CAT (#29303) • 🐛 Source Shopify: handle the
HTTP-500
Internal Server Error
(#29302) • 🐛 Source Greenhouse, Xero: fix expected records (#29245) • 🐛 Source Prestashop: fix build (#29395) • 🐛 Source Google Sheets: updated handling 404 and 403 error while discover (#29327) • 🐛 Source S3 (V4): Ensure all files are not resync'd when migrating from v3 to v4 (#29418) • 🐛 Source Mongo Internal POC: Add Debezium MongoDB connector (#29588) • 🐛 Destination Bigquery - ensure raw dataset created with migration (#29522) • 🐛 Destination bigquery: Properly fix per-stream state handling (#29498) • 🐛 Destination Bigquery - fix migration logic (#29461) • 🐛 Destinations v2: handle streams with no columns (#29381) • 🐛 Source Mongo Internal POC: Use sampling for schema discovery (#29607) • 🐛 Source Google Ads: fix build (#29653) • 🐛 Source Linkedin Ads: fix build (#29655) • 🐛 Source Hubspot: update expected records (#29648) • 🐛 Source Google Sheets: added stop reading in case of 429 error (#29427)
❤️ 1
j
Postgres, then Mysql, what's the roadmap look for the rest of the rdbms connectors like mssql, oracle, etc?
j
MySQL should become GA in the next couple of weeks, so pretty fast. MongoDB will be in the next month or SO. MS SQL will be a focus in Q4, don't know exactly when it will be ready. Oracle should be next year though, most likely 1st half.
a
Hi, great news but what about this issue: https://github.com/airbytehq/airbyte/issues/24659. We can’t event use mysql connector with CDC mode and only way to work with it it’s a use 1.x version which is old, so all of new features are not for us 😞
a
@John (Airbyte) Question :- My team is looking for a connector that can sync data from postgres to snowflake in near real time (fivetran does it for us in like 10 mins max). Have we in airbyte reached/optimized that speed ? and do we also replicate toast data ? If yes, I am going to convince my team to use Aibyte cloud instead of fivetran, please do let me know
j
@Andrzej Lewandowski @Ayush Singh will both keep you posted, checking with the team.
m
Hi guys, very happy to read this!! Great job! What about the ClickHouse destination? You guys have some plans for it?
j
to be clear, we have plans for all 🙂 . but no timeline yet on Clickhouse destination
p
@Andrzej Lewandowski we will fix the connector. It is likely that when the 2.0 release which brought in a new Debezium version also introduced the issue of server timezone not being applied correctly when connecting to the database.
❤️ 1
@Ayush Singh We support TOAST columns if you allow replication identity full while configuring CDC (doc link). We have not optimized the connector for reading toast data, so you may run into out-of-memory problems during replication. But if that happens, we can give the connector extra memory. Also, we checkpoint when reading your data, both during initial and incremental syncs, so that any failure does not cause us to lose progress. And you can set the replication frequency to 15 minutes, bringing us closer to your near real-time requirements. Give it a shot and let me know how it goes!
l
Are all these improvements already applied to the latest version of the Postgres source connector?
p
@Luan Carvalho, yes.
l
Ty @Prateek Mukhedkar !
r
What version of postgres is required to take advantage of the new connect initial sync improvements? Will pg 10.x work?
p
@Rocky Appiah As of now Postgres 14 and higher and we are bringing the same set of improvements to v12,13 also.
r
ok - should I be able to just upgrade the postgres source connector on an OSS installation? Or do I need to remove the connection and re-add
Thanks @Prateek Mukhedkar
g
Does GA include SSL (TPCS) support for sources and destinations? I was looking at the Oracle Source and found that there is no support for TCPS. I also haven't found an issue on Github regarding this feature.
p
@Gregor can't commit for Oracle just now, however other database sources including Postgres, MySQL have full SSL support
@Andrzej Lewandowski can you get the latest MySQL source connector and give it a go? We released a fix for 24659.
g
@Prateek Mukhedkar do you know, by any chance, what's the issue with the Oracle SSL support? Just looking at the code in Github it seems like it is already implemented.
a
@Prateek Mukhedkar
Sync Succeeded
with new mysql connector version. Thanks!
👍 1
p
@Gregor it is about ensuring that we have tested the support properly for all SSL variants.
👍 1