Hey everyone! :octavia-wave::octavia-wave: July h...
# announcements
j
Hey everyone! octavia waveoctavia wave July has been a month of significant advancements at Airbyte! Here's a quick rundown of what's new: airbyte code *Airbyte API and Terraform Provider Are Now Available in Open Source!* airbyte code Transitioning from the Configuration API? The new Airbyte API offers a REST interface for a smoother experience. šŸ¤– Harnessing AI at Airbyte šŸ¤– We're pushing the boundaries with AI! Check out our recent articles and tutorials: 1. Why AI Shouldn't Reinvent ETL 2. Chat with Your Data using OpenAI, Pinecone, Airbyte, and Langchain 3. Measure Customer Support Sentiment Analysis with GPT, Airbyte and MindsDB 4. Airbyte and LlamaIndex: ELT and Chat with your data warehouse without writing SQL šŸ“š Enhancements in Postgres Integration šŸ“š We've refined our Postgres Source connector, optimizing for large tables and ensuring reliable data syncs. Our journey with Postgres has been about continuous improvement, and we're excited to share our learnings with you. Dive deeper into our Postgres journey: Top 4 Lessons We Learned Reading Very Large Postgres Tables šŸ• Introducing Airbyte's Datadog Integration šŸ• Elevate your monitoring game with our latest integration with Datadog! This integration is designed for Airbyte Enterprise users, offering a seamless way to monitor and analyze data pipelines. Get started with: • Datadog's official documentation • Airbyte's documentation octavia partying Events octavia partying • Quarterly Product Review - August 16th: This webinar will demonstrate last quarter's newest features: Checkpointing, Column Selection, and Schema Propagation. Our engineering team will display how these features improved Airbyte and how they can make data movement easier for you. | 9:00 AM PT| 6:00 PM CET | • Airbyte powers AI through vector databases - August 17th: This demo will show how our new vector database destination can help with embeddings for your unstructured data. We will also cover using LLMs to implement a conversational interface on top of LangChain. | 9:00 AM PT | 12:00 PM ET | 6:00 PM CET | • Key Takeaways from Enhancing our Postgres Connector - August 30th: Join us for a live Q&A session on how we improved our Postres source connector from 4 mb/s to 11 mb/s, unlocking the ability to read tables in the terabyte range. | 10:00 AM PT | 1:00 PM ET | 7:00 PM CET | ---- Stay tuned for more updates, and as always, thank you for being a part of our community! šŸš€
šŸ’Æ 6
✨ New and improved features • New Sources and Promotions ā—¦ šŸ¤– Add Datadog connector to cloud (#27906) ā—¦ šŸŽ‰ New Destination Vector Database (powered by LangChain) (#26184) ā—¦ ✨ New Source: Gainsight-px [Low Code CDK] (#26998) ā—¦ ✨ Destination Iceberg: Bump Iceberg from 1.0.0 to 1.1.0 (#23201) • New Features for Existing Connectors ā—¦ šŸŽ‰ Source Shopify-OAuth: make
OAuth
the only option for CANDIDATE for
Cloud
(#6457) ā—¦ ✨ Source Datadog: Multiple query streams and datadog site parameter (#27804) ā—¦ ✨Source Twilio: New streams VerifyServices (#27231) ā—¦ ✨Source Zendesk Support: New Stream - Posts (#27156) ā—¦ ✨ Source Zendesk Support: add stream Ticket Skips (#27436) ā—¦ ✨ Source Zendesk Support: add Skills Based Routing streams (#27450) ā—¦ ✨ Destination Google Sheets: migrate to advanced auth (#28035) ā—¦ šŸŽ‰ Source RD Station Marketing: migrated to advancedOAuth (#28009) ā—¦ šŸŽ‰ Source Instagram: Improve, refactor
STATE
management (#27908) ā—¦ ✨ Source Amazon-Ads: Add streams for portfolios and sponsored brands v3 (#27607) ā—¦ ✨Source Stripe: New Stream - Persons (#26208) ā—¦ ✨ Source Google Ads: add new streams (#28246) ā—¦ ✨Source Twilio: New streams UserConversations with parent Users (#27221) ā—¦ ✨ Source Monday: Add incremental sync for Items and Boards stream (#27944) ā—¦ šŸŽ‰ Source Facebook Marketing - add action report time enum to adinsights class (#26345) ā—¦ ✨ Source Coda: Add rows to codaio source (#27797) ā—¦ šŸŽ‰ source-postgres : Enable xmin replication (#26723) ā—¦ šŸŽ‰ Source Google Ads: add new stream
Campaign Budget
(#28078) ā—¦ šŸŽ‰ Source Shopify-OAuth: release for Cloud Review (#28507) ā—¦ ✨ Destination BigQuery - Handle Schema Changes (#28382) • New Features in Airbyte Platform ā—¦ šŸŽØ Dark Mode 🌚 (#6632) ā—¦ šŸŽ‰ Add sync mode dropdown to stream details panel (#7688) ā—¦ šŸŽ‰ Connector Builder: Global requests UI (#7699) 🚨 Security & Breaking changes • 🚨 Source Hubspot - fixed incremental for engagement stream (#27161) • 🚨 Source MetaBase: Remove
Activity
Stream (#27777) • 🚨 Source Okta: fix group members and resource sets streams (#27533) • 🚨 Source Facebook Marketing: use SDK v17 (#27563) • 🚨 Source Pipedrive: Remove
followers_count
field from
Products
stream (#27832) • 🚨 Source PayPal Transactions: Rename field in
Balances
schema (#27916) • 🚨 Source Recharge: fix stream schema (#27612) • šŸšØšŸ› Source Coda: Update Schemas and Fix Tests (#28093) • 🚨 Low code CDK: Add session token authenticator (#28050) šŸ› Bug fixes • šŸ› Fix custom connector creation flow (#8018) • šŸ› Connector Builder: Fix copy to/from button behavior • šŸ› Source Square: update following state management changes in the CDK (#27762) • šŸ› Source Greenhouse: update following state management changes in the CDK (#27773) • šŸ› Source Posthog: update following state management changes in the CDK (#27764) • šŸ› Source Zenloop: update CDK version to avoid bug introduced during data feed release (#27838) • šŸ› Source Quickbooks: update following state management changes in the CDK (#27803) • šŸ› Source Zenloop: update following state management changes in the CDK (#27761) • šŸ› Source GA Data API: change page size when calling
check()
(#27718) • šŸ› Source Google Ads: refactor date slicing (#27711) • šŸ› Source Datadog: Fix schemas (#27784) • šŸ› Source Recharge: migrated to 2021-11 api version (#27503) • šŸ› Source Typeform: add form_id field to records of responses stream (#27653) • šŸ› Source Okta: Fix Infinite Loop in GroupMembers Stream (#20833) • šŸ› Source Jira: update expected records (#28005) • šŸ› Source Facebook Marketing: restore reduced request record limit after
successful
response (#27979) • šŸ› Destination Snowflake: Merge old snowflake work (#27935) • šŸ› Source Tiktok Marketing: Update ad_groups schema (#27988) • šŸ› Source Pinterest: update expected records (#27964) • šŸ› Source Pipedrive: update expected records (#27967) • šŸ› Source Jira: update expected records (#27951) • šŸ› Source Google Analytics Data API: limit custom report page size (#27909) • šŸ› Source Google Ads: fix query template (#27835) • šŸ› Source Hubspot: use fully qualified name for custom objects (#27658) • šŸ› Destination Snowflake: Pull in async minor bug fix. (#28315) • šŸ› Source Datadog - Additional stream and query details in response (#28089) • šŸ› Source Hubspot: update expected records (#28040) • šŸ› Source Twilio: update expected records (#28216) • šŸ› Source Zendesk Support: use cursor pagination only (#28096) • šŸ› Source Github: update schema for Organizations stream (#28144) • šŸ› Source Apple Search Ads: manifest.yaml duplicate key error (#28153) • šŸ› Source Monday: Add new streams and update old (#27410) • šŸ› Source Square: fix display order of spec fields (#28019) • šŸ›Source Iterable: fix type error (#28457) • šŸ› Source Google Ads: include
today
in the date range (#28365) • šŸ› Source Mixpanel: reimplement backoff strategy (#27752) • šŸ› Source apify-dataset: Fix broken sync (#28290) • šŸ›Source PostHog: fixed EventsSimpleRetriever declaration (#28461) • šŸ› Destination bigquery 1s1t: respect dataset location (#28580) • šŸ› Release Snowflake Async State Bug. (#28581)
m
Super cool updates, especially to the Postgres Connector! From reading the Postgres blogpost I see a lot of similar issues in the CockroachDB source Connector. Are there any plans to get this connector up to date as well? šŸ¤ž
j
we'll get there, it will take a bit of time. our priorities on databases are MySQL (should in great shape already now, and even better in the next 2 weeks), MongoDB (within the next 4 weeks), then MS SQL (in Q4). database replication is a major priority, so you can be sure we'll tackle CockroachDB too. just a question of timing...
šŸ™ 1
m
Hi John! Just following up here, what is the status on this? I see Cockroach is still not available in Airbyte Cloud 😢
r
Thanks for the CockroachDB source! What about destination, is that on the roadmap?