👋 Hey <!everyone>! Here’s a wrap up of updates we’ve had over the last 2 months! 🍃 Spring Release 2024 We have some exciting announcements coming in our Spring Release, launching on May 30th! Head to this page and add the even to your calendar to see what’s coming. 💻 PyAirbyte Hackathon Starting in June We are thrilled to announce the Airbyte PyAirbyte Tutorial Hackathon for June 2024! Join us on an exciting journey to revolutionize data integration. This hackathon is a great opportunity for developers worldwide to showcase their skills using Airbyte’s Python Library to address common AI use cases. python PyAirbyte 0.10 Update: Airbyte Cloud Integration We’ve released PyAirbyte 0.10, featuring direct management of Airbyte Cloud jobs, status checks for previous jobs, and access to data from Snowflake and BigQuery. It also includes several experimental features. This update simplifies the orchestration and creation of data pipelines using Airbyte connectors. For more details, check out the PyAirbyte 0.10 release notes! easy button Simplified Connection Flow (Cloud only, coming to OSS) We’ve introduced a declarative way to select sync modes, simplifying your data sync process. All settings are now consolidated in one place for easier setup. A new connection setting will backfill historical data for new columns detected through schema propagation. 🔧 Connector Builder Updates There have been a substantial amount of updates to the Connector Builder in order to enhance your experience! Here are some highlights: • Persistent Testing Values ◦ Values and credentials entered into the Testing Values menu of the Connector Builder will now persist across sessions. ◦ Sensitive data is stored using secure secret persistence configured for connector configurations. ◦ This enhancement eliminates the need to re-enter testing values every time the Builder is reopened. • Concise YAML Manifests ◦ Child streams now reference parent streams using $ref, avoiding duplication. ◦ All streams point to a shared base URL and Authenticator definition using

$ref

. ◦ Schemas are placed at the bottom of the YAML and referenced with

$ref

, improving clarity. ◦ YAML keys are consistently ordered by the Builder. ◦ Top-level fields in the YAML are separated by empty lines for better readability. ◦ Unused optional fields are omitted from the YAML instead of being set to empty values. • YAML Components and Local Custom Component Support ◦ The Connector Builder supports YAML component definitions for low-code CDK features such as paginator, errorHandler, transformations, and incremental sync, with more components to be supported soon. ◦ You can develop connectors with custom components using the Connector Builder with the Open Source version of the platform. ◦ Instructions are provided for developing and publishing connectors as Docker connectors. • User Inputs ◦ The Connector Builder now manages user inputs more efficiently with unique IDs for auto-created inputs to avoid collisions. ◦ Users can edit the IDs and titles of all inputs, including auto-created ones, independently. • Incremental State ◦ When Incremental Sync is configured, the state output by each partition can be viewed in the testing panel. ◦ Users can set the Input state for the next test read, allowing verification of incremental state compatibility and testing responses to different input states. ◦ The output state tab includes a “Use as Input State” button for easy reuse of output state as input state in subsequent test reads. • Record Filtering and Casting ◦ The Record Selector component now includes a Field Path inside a full Record Selector card. ◦ Users can define conditions for filtering records in the Record Filter input. ◦ Record fields can be cast to types declared in the stream schema if the values are compatible. ✅ Certified Database Sources Resilience We have enhanced the resilience of our certified database source connectors to ensure that a single row containing an error will no longer disrupt the entire synchronization process. You will now have visibility on how the data was modified (

NULLED

,

TRUNCATED

) and the reason for the change (e.g.,

SOURCE_SERIALIZATION_ERROR

) through the

airbyte_meta.changes

log. 🔒 JWT Authentication Added to the Low-Code CDK JWT authentication support is now available in the Low-Code CDK, enabling access to connectors that require JWT for authentication. This includes connectors for services like JIRA, Google Cloud, Apple, and Zoom, among others. 📆 Upcoming Events • Data Cloud Summit 24: June 3-6, 2024, in San Francisco - We invite you to attend Airbyte CEO, Michel Tricot's talk at the Data Cloud Summit about Building GenAI-powered applications using Airbyte and Snowflake Cortex. The entire week, we will have a booth in Basecamp South 1210, where you can meet with our team, get a demo, and pick up your own Octavia plushy. • Data Oasis Virtual Reality Edition: June 4th, San Francisco - Test your skills, whether you're a seasoned VR enthusiast or a curious newcomer. We will also have food & drinks for when you're not playing. • Electric Data Carnival: June 5th, San Francisco - Join us for good music, great people, and an unforgettable night at the Summit after-party of the year.