Hey <!everyone> octavia wave ! 🚀 Exciting news! move(data) 2023, our 2nd virtual data engineering conference, is just around the corner on December 6th and 7th. Best part? It's online and free! 🔍 Why Join? • Hear from top voices in data engineering like Benn Stancil (ThoughtSpot), Joe Reis (Ternary Data), Benjamin Rogojan (Seattle Data Guy), Michel Tricot (Airbyte), Drew Banin (dbt Labs), Pete Hunt (Dagster Labs), Ken Wong (Databricks), Mehdi Ouazza (MotherDuck), and many more. • Only 10-minute 🌩️ talks for maximum learning! • Discover the latest trends and tools in our field. • Network with over 5,000 data professionals. 🔗 Sign Up Now! Don't miss out on this opportunity to connect and grow. Register here in just a few clicks! If you can’t at that time, consider still registering, as we’ll share recordings to the people that signed up. 👍 Drop a '✅' below if you're in! Questions? Shoot them here, and let's get the conversation started!

Hey <!everyone> 👋, Today is the day! move(data) will be s*tarting in 10 minutes* with the Keynote from Michel Tricot, CEO of Airbyte. If you haven't registered yet you can sign up here. You won't want to miss any the exciting lightning talks. To chat about the conference join the #C03UHAYGYCB channel. Looking forward to seeing you all at the conference today!

redshift Calling all AWS Redshift users! We need beta testers for

destination-redshift

V2! New features include: • _One-to-one mapping_: Data from one stream (endpoint or table) will now create one table in the destination - gone are the days where you will need to join multiple tables to learn about your data. • _Improved error handling_: Typing errors will no longer fail your sync, and are now easily visible in a new

_airbyte_meta

column, allowing for better tracking of inconsistencies and resolution of issues. • _Incremental data delivery in final tables_: Data will be incrementally delivered in final tables for all initial snapshots. No more waiting hours to see the first rows in your destination table. Learn more at the Github Discussion and then join us the linked slack channel. Airbyte OSS and Cloud users are welcome!

Happy New Year, <!everyone> and welcome back! 🥳 As we step into 2024, let's take a moment to reflect on our incredible journey in 2023 and look forward to what's ahead. Last year was remarkable for us at Airbyte, with significant achievements and growth: • We reached 127,000 deployments and received over 12,000 stars on GitHub. help us reach 15k 😃 • 350+ connectors available to the community, keeping Airbyte the platform with the most connectors in the industry! • Our community expanded to 17,000+ members, which is now supported by an AI chatbot to help us answer the growing number of questions on #C01AHCD885S. • Launched the No Code Connector Builder UI in addition to our Low Code CDK, so you can build connectors in minutes. Over 2,000 connectors have been built with it. • Hacktoberfest: We joined forces with Hashnode this year, and the community migrated 70 connectors to our low-code framework, fixed 42 bugs, and wrote 24 quickstarts! • Hosted our 2nd annual virtual conference, move(data). 5,000+ folks signed up virtually to hear 35 speakers over 8 hours of lightning talks. • AI use cases unlocked with unstructured data sources and vector DB destinations. • The official Airbyte API and Terraform Provider became available on Airbyte Cloud and Open Source. • Speed - we 10x’d our database replication speed to support larger datasets. Looking ahead to 2024, we're excited about: • further improving speed and reliability, • aiming for 500 high-quality connectors, • focusing on ELTP (extract, load, transform, publish) to close the data movement loop. • covering data lakehouses • enhancing Airbyte Enterprise for self-managed solutions. Our community remains a cornerstone of our success, and we plan to invest more in programs to facilitate contributions and scale support too. Thank you for your passion and commitment. Together, we're building the largest community around data integration. Here's to a fantastic 2024! 🥂 Read more in our CEO's article 🙂

You asked, we listened. <!everyone> Good news, we're in the process of building a certified NetSuite ODBC connector! We are looking for users with SuiteAnalytics enabled on their NetSuite accounts to help us verify this connector. Please respond in this thread if you're interested in helping us with this effort. The goal is to make this available to everybody later in Q1!

Hi Everyone octavia wave! We are excited to announce our Partnership with Vectara as a destination to help build GenAI applications In this blog post, we will show how you can ingest text data from google drive using Airbyte to Vectara showing you how you can develop GenAI applications that are scalable enterprise-grade. You can also sign up to attend our joint webinar where we will demo this live.

python Calling all Python developers! python <!channel> - Join us in shaping the future of Python-based data integration! We're building an exciting new product, codename "AirbyteLib", which allows seamless execution of Airbyte jobs directly from Python. This new python library will support two modes: a local mode (no Airbyte workspace needed) and a hosted mode with deep integrations to Airbyte Cloud, OSS, and Enterprise. This project aims to break down silos and make it easy to incorporate Airbyte with the broader ecosystem of Python-based AI, LLM, and data engineering technologies. 🚀 Why Join Our Private Beta? • Get early access to innovative features. • Influence the product's development to suit your needs. • Solve your specific data integration challenges with our support. 👋 What Will You Build? • ⚙️ Data Engineers & Analysts - design and build Airbyte jobs locally in Python, as code. • jupyter Jupyter & Google Colab Devs - streamline notebook development with instant access to high-quality data sources. • ✨ AI Developers - integrate diverse data into your AI and LLM projects. Interested? We'd love your input to help make this product outstanding. To be considered for the private beta access, please drop me a note in this thread or DM me (@AJ Steers (Airbyte)) with a quick description of what problems you'd solve with this library. Looking forward to collaborating with you! octavia muscle 🚨 UPDATE: Due to tremendous response, our initial private beta is now full. Sign up here to get on the waitlist. 🚨

👋 Hello <!everyone>! If you utilize dbt within your stack for data transformation, we’d love to learn more. • Extra context: related github discussion on custom normalization (dbt) in Airbyte OSS. Our question to you: As a user of Airbyte OSS, what method or tools are you using to trigger dbt jobs on your data? Please let us know in the thread! 🙌

👋 In 2024, we are heavily investing in the Airbyte OSS experience. ➡️ Please complete our (30-second) multiple-choice survey (no writing required!) We’d love to learn about your experience and make Airbyte OSS amazing for you ✨

👋 Hi <!everyone> - Our last reminder to complete the 30-second survey this week for improving your Airbyte OSS product experience! 🙏

Hey <!everyone>! January was full of updates so let's go over them! octavia rocketoctavia rocketoctavia rocket Airbyte x python We have something to show you on 2.28 👀 Be sure to add our 2024 Winter Release event to your calendars to see what’s coming! 📝 State of Data & AI Survey 2024 Last year we had the largest data engineering with State of Data 2023. So we’re back again for the 2024 edition! Be sure to fill this out 🙏🏽 🎉 New Connectors 3 new connectors are now live! • OneDrive: Thanks to you, the community, we are able to add the OneDrive connector into the catalog as a community build connector! • SharePoint: SharePoint is also currently in progress to be certified soon! • Vectara: for GenAI and semantic search applications! 🔧 Certifications and Upgrades Some of our current connectors also got an upgrade: • Our Redshift destination now supports Destinations v2 • The S3 source connector now supports IAM Authentication • Stripe and Salesforce connectors are now able to run processes concurrently with full refresh and some incremental streams • The S3 and Bing Ads source connectors got a facelift: ◦ We’ve verified a reliable experience for our customers using the connector ◦ Validated the UX and docs continue to be high quality and accessible ◦ Solved any outstanding bugs surfaced during the beta period 👨🏽‍💻 What’s Coming? • Shopify: We have a major revamp of our Shopify connector coming! This is going to be a performance upgrade where we switch Shopify to a bulk GraphQL implementation! • SharePoint: Our team is working hard to upgrade this new connector to our rigorous certification standards. • Netsuite: We are building a certified NetSuite ODBC connector! This connector delivers even more functionality than the Community Connector which relies on Webhooks. In order to get started, you must have a Netsuite account with Suite Analytics and Suite Analytics Connect enabled.

🌟👋 Calling all MSSQL users! We need beta testers for our updated

source-mssql

! New features include: • Customer Support: Connector support from the internal Airbyte team (we plan to certify this source shortly!) • _Terabyte sized tables_: source-mssql can now do incremental reads of terabyte-sized tables without incurring any performance degradation. • _Reliability improvements_: Failed reads, (for any reason e.g connection disruption), can now resume reading from a last known checkpoint, rather than having to start over. • _Various bug fixes_: fixes have been implemented for many issues discovered in the community connector, including full support of datetime data types. Learn more in the Github Discussion and comment here if you'd like to be added to our private slack channel. Airbyte OSS and Cloud users are welcome!

Hey <!everyone>! We still need beta testers with that are syncing 500GB+ of data for our updated

source-mssql

! New features include: • Customer Support: Connector support from the internal Airbyte team (we plan to certify this source shortly!) • _Terabyte sized tables_: source-mssql can now do incremental reads of terabyte-sized tables without incurring any performance degradation. • _Reliability improvements_: Failed reads, (for any reason e.g connection disruption), can now resume reading from a last known checkpoint, rather than having to start over. • _Various bug fixes_: fixes have been implemented for many issues discovered in the community connector, including full support of datetime data types. Learn more in the Github Discussion and comment here if you'd like to be added to our private slack channel. Airbyte OSS and Cloud users are welcome! Let us know down in this thread!

Airbyte's Winter Release is a day away, and to celebrate the newest Pythonic member of the Airbyte family, we built a little game called SnakeByte! Give it a spin and see if you can beat our team's high score! 😄 🐍 🎮

<!channel> python PyAirbyte has entered public beta! python We have been building out PyAirbyte and it is ready for all of you to start building with! It has never been easier than ever to execute Airbyte jobs and utilize our connectors, all directly from Python. Join the #C06FZ238P8W channel for resources, links, and to share your use cases. In case you missed it, you can watch our launch announcement here - where we will walk you through the basics of PyAirbyte as well as what is coming soon! https://airbyte.com/blog/announcing-pyairbyte

hey everyone!! octavia wave octavia wave A lot of things are happening at Airbyte. This year, we will start doing meetups octavia rocket . And we'd love your input about our meetup group name, it only takes 1 minute! Thank you!! 🙏 https://form.typeform.com/to/QcnjBtFK

We have have been hard at work here at Airbyte and we are working on a lot of big improvements! Among those improvements is the implementation of per-stream features like checkpointing, which is integral to Airbyte’s data correctness strategy. However, there are a handful of community connectors that do not support per-stream features and as such Airbyte cannot ensure the reliability and correctness of this source and has made the decision to discontinue these connectors. Below are the list of connectors that will be discontinued and will not be compatible with the improvements to ensure reliability and correctness: •

source-talkdesk-explore

•

source-kustomer-singer

•

source-courier

•

source-zuora

•

source-dv-360

•

source-search-metrics

•

source-appstore-singer

•

source-google-workspace-admin-reports

Given that these sources will no longer be compatible, we recommend creating a custom connector if you are still in need to pull data from these community sources. For further details on custom connectors, please refer to our Custom Connector documentation.

Hello Everyone! We're excited to announce our first meetup in our new office, featuring panelists from Airbyte, LangChain and Pinecone! Eat food, drink beverages, and discuss the intersection of data, LLMs and infrastructure engineering. What's not to love? Reserve your spot now for March 28th! https://www.meetup.com/data-bytes/events/299752878/

Hello everyone! octavia wave Great news for those who want to learn more about Airbyte!!! You might have heard about Marc Lamberti’s great courses 👨‍🏫 . Today, he’s launched a hands-on introduction course on using Airbyte. In this introduction to Airbyte, you’ll learn all of these essential concepts as you build a complete data engineering project from start to finish airbyte rocket : • ✅ How to set up Airbyte OSS locally • ✅ The core concepts and components • ✅ Best practices around data ingestion • ✅ How to use the different syncs • ✅ How to sync data with CDC • ✅ How to detect and act on schema changes Here is the link to the Udemy course: https://www.udemy.com/course/the-complete-hands-on-introduction-to-airbyte/ You’re also invited to join the channel #C06QPNMFD0W to chat with fellow learners about questions, ideas, and what’s next in your data engineering journey!

We just released a big update for PyAirbyte! Check the details below, drop questions or feedback in #C06FZ238P8W, and stay tuned for more updates over the next two weeks! Happy Friday! octavia wave

We’re live on Youtube hosting the Data Bytes! Meetup

Hello everyone and happy Friday octavia wave This is a heads up that Airbyte Platform Version 0.58.0 contains a bug fix for an upcoming connector feature we call Resumable Full Refresh. Today Full Refresh connections always start from scratch on each job attempt. This is very inefficient. Sources supporting Resumable Full Refresh will be able to start from where the previous attempt ended, with the goal of increasing the success rate of long running Full Refresh jobs. Running a Full Refresh stream with a Resumable Full Refresh Source on pre-0.58.0 versions will cause all jobs after the first job to behave as Incremental. While Platform support for Resumable Full Refresh is coming later, Resumable Full Refresh supported Sources will be released in the next few weeks. These Sources will contain a

min_platform_version

to highlight Platform version compatibility. If upgrading the Platform isn’t feasible now, please check the Changelog before upgrading your Source to avoid this bug. More info about 0.58.0 release you can get here.

x-post if you are using the Airbyte API with Airbyte OSS, please be aware of future breaking changes that may affect your applications: https://airbytehq.slack.com/archives/C023W76QGE4/p1714654107541709

⚠️ Security alert <!channel>: We have identified a malicious website airbyte<dash>usa<dot>com that are impersonating to Airbyte. Do not attempt to visit the website. What You Should Do: • If you receive any communications claiming to be from Airbyte that seem suspicious, please do not engage or provide any personal information. • Contact us directly at security@airbyte.io or @Tobias Wennergren (Airbyte) if you suspect you have been approached by anyone associated with this fake site. We’re Here to Help: Your security is our top priority. If you have any concerns or believe you may have been contacted by these impersonators, please reach out to us immediately. We are here to assist you and ensure your continued safety.

Please visit the #C023W76QGE4 channel if you haven’t yet 🥺 octavia thanks . The Airbyte Team posts updates about new features, early demos, deprecations, and release notes there. x-posting the Release Notes for April 2024: https://airbytehq.slack.com/archives/C023W76QGE4/p1715267589767089

👋 Hey <!everyone>! Here’s a wrap up of updates we’ve had over the last 2 months! 🍃 Spring Release 2024 We have some exciting announcements coming in our Spring Release, launching on May 30th! Head to this page and add the even to your calendar to see what’s coming. 💻 PyAirbyte Hackathon Starting in June We are thrilled to announce the Airbyte PyAirbyte Tutorial Hackathon for June 2024! Join us on an exciting journey to revolutionize data integration. This hackathon is a great opportunity for developers worldwide to showcase their skills using Airbyte’s Python Library to address common AI use cases. python PyAirbyte 0.10 Update: Airbyte Cloud Integration We’ve released PyAirbyte 0.10, featuring direct management of Airbyte Cloud jobs, status checks for previous jobs, and access to data from Snowflake and BigQuery. It also includes several experimental features. This update simplifies the orchestration and creation of data pipelines using Airbyte connectors. For more details, check out the PyAirbyte 0.10 release notes! easy button Simplified Connection Flow (Cloud only, coming to OSS) We’ve introduced a declarative way to select sync modes, simplifying your data sync process. All settings are now consolidated in one place for easier setup. A new connection setting will backfill historical data for new columns detected through schema propagation. 🔧 Connector Builder Updates There have been a substantial amount of updates to the Connector Builder in order to enhance your experience! Here are some highlights: • Persistent Testing Values ◦ Values and credentials entered into the Testing Values menu of the Connector Builder will now persist across sessions. ◦ Sensitive data is stored using secure secret persistence configured for connector configurations. ◦ This enhancement eliminates the need to re-enter testing values every time the Builder is reopened. • Concise YAML Manifests ◦ Child streams now reference parent streams using $ref, avoiding duplication. ◦ All streams point to a shared base URL and Authenticator definition using

$ref

. ◦ Schemas are placed at the bottom of the YAML and referenced with

$ref

, improving clarity. ◦ YAML keys are consistently ordered by the Builder. ◦ Top-level fields in the YAML are separated by empty lines for better readability. ◦ Unused optional fields are omitted from the YAML instead of being set to empty values. • YAML Components and Local Custom Component Support ◦ The Connector Builder supports YAML component definitions for low-code CDK features such as paginator, errorHandler, transformations, and incremental sync, with more components to be supported soon. ◦ You can develop connectors with custom components using the Connector Builder with the Open Source version of the platform. ◦ Instructions are provided for developing and publishing connectors as Docker connectors. • User Inputs ◦ The Connector Builder now manages user inputs more efficiently with unique IDs for auto-created inputs to avoid collisions. ◦ Users can edit the IDs and titles of all inputs, including auto-created ones, independently. • Incremental State ◦ When Incremental Sync is configured, the state output by each partition can be viewed in the testing panel. ◦ Users can set the Input state for the next test read, allowing verification of incremental state compatibility and testing responses to different input states. ◦ The output state tab includes a “Use as Input State” button for easy reuse of output state as input state in subsequent test reads. • Record Filtering and Casting ◦ The Record Selector component now includes a Field Path inside a full Record Selector card. ◦ Users can define conditions for filtering records in the Record Filter input. ◦ Record fields can be cast to types declared in the stream schema if the values are compatible. ✅ Certified Database Sources Resilience We have enhanced the resilience of our certified database source connectors to ensure that a single row containing an error will no longer disrupt the entire synchronization process. You will now have visibility on how the data was modified (

NULLED

,

TRUNCATED

) and the reason for the change (e.g.,

SOURCE_SERIALIZATION_ERROR

) through the

airbyte_meta.changes

log. 🔒 JWT Authentication Added to the Low-Code CDK JWT authentication support is now available in the Low-Code CDK, enabling access to connectors that require JWT for authentication. This includes connectors for services like JIRA, Google Cloud, Apple, and Zoom, among others. 📆 Upcoming Events • Data Cloud Summit 24: June 3-6, 2024, in San Francisco - We invite you to attend Airbyte CEO, Michel Tricot's talk at the Data Cloud Summit about Building GenAI-powered applications using Airbyte and Snowflake Cortex. The entire week, we will have a booth in Basecamp South 1210, where you can meet with our team, get a demo, and pick up your own Octavia plushy. • Data Oasis Virtual Reality Edition: June 4th, San Francisco - Test your skills, whether you're a seasoned VR enthusiast or a curious newcomer. We will also have food & drinks for when you're not playing. • Electric Data Carnival: June 5th, San Francisco - Join us for good music, great people, and an unforgettable night at the Summit after-party of the year.

⚠️ Security alert: We have identified another malicious website airbyte<dash>worksite<dot>com that are impersonating to Airbyte. Do not attempt to visit the website. What You Should Do: • If you receive any communications claiming to be from Airbyte that seem suspicious, please do not engage or provide any personal information. • Contact us directly at security@airbyte.io or @Tobias Wennergren (Airbyte) if you suspect you have been approached by anyone associated with this fake site. We’re Here to Help: Your security is our top priority. If you have any concerns or believe you may have been contacted by these impersonators, please reach out to us immediately. We are here to assist you and ensure your continued safety.

Hey everyone! octavia wave On May 30th, our Spring Release for 2024 will be going live! It'll go over our newest updates and previews of upcoming features that will enhance your user experience with Airbyte. Add the Spring Release to your calendars to be notified when the post goes live.

🎉 Exciting news! The Spring Release is now live! 🎉 Read all about AI assistance, GenAI, Connector Builder + PyAirbyte updates, and Airbyte 1.0 here!

octavia musclepython Exciting new feature in PyAirbyte! You can now run "no-code" sources directly in PyAirbyte, without a separate installation or virtual environment needed for the connector. You can also use this to instantly test and run your own custom connectors built using the Connector Builder. Check out this quick loom demo for more info - and here is the Google Colab Notebook if you want to follow along!