[DEPRECATED] Marcos Marx
John (Airbyte)
$ref
.
◦ Schemas are placed at the bottom of the YAML and referenced with $ref
, improving clarity.
◦ YAML keys are consistently ordered by the Builder.
◦ Top-level fields in the YAML are separated by empty lines for better readability.
◦ Unused optional fields are omitted from the YAML instead of being set to empty values.
• YAML Components and Local Custom Component Support
◦ The Connector Builder supports YAML component definitions for low-code CDK features such as paginator, errorHandler, transformations, and incremental sync, with more components to be supported soon.
◦ You can develop connectors with custom components using the Connector Builder with the Open Source version of the platform.
◦ Instructions are provided for developing and publishing connectors as Docker connectors.
• User Inputs
◦ The Connector Builder now manages user inputs more efficiently with unique IDs for auto-created inputs to avoid collisions.
◦ Users can edit the IDs and titles of all inputs, including auto-created ones, independently.
• Incremental State
◦ When Incremental Sync is configured, the state output by each partition can be viewed in the testing panel.
◦ Users can set the Input state for the next test read, allowing verification of incremental state compatibility and testing responses to different input states.
◦ The output state tab includes a “Use as Input State” button for easy reuse of output state as input state in subsequent test reads.
• Record Filtering and Casting
◦ The Record Selector component now includes a Field Path inside a full Record Selector card.
◦ Users can define conditions for filtering records in the Record Filter input.
◦ Record fields can be cast to types declared in the stream schema if the values are compatible.
✅ Certified Database Sources Resilience
We have enhanced the resilience of our certified database source connectors to ensure that a single row containing an error will no longer disrupt the entire synchronization process. You will now have visibility on how the data was modified (NULLED
, TRUNCATED
) and the reason for the change (e.g., SOURCE_SERIALIZATION_ERROR
) through the airbyte_meta.changes
log.
🔒 JWT Authentication Added to the Low-Code CDK
JWT authentication support is now available in the Low-Code CDK, enabling access to connectors that require JWT for authentication. This includes connectors for services like JIRA, Google Cloud, Apple, and Zoom, among others.
📆 Upcoming Events
• Data Cloud Summit 24: June 3-6, 2024, in San Francisco - We invite you to attend Airbyte CEO, Michel Tricot's talk at the Data Cloud Summit about Building GenAI-powered applications using Airbyte and Snowflake Cortex. The entire week, we will have a booth in Basecamp South 1210, where you can meet with our team, get a demo, and pick up your own Octavia plushy.
• Data Oasis Virtual Reality Edition: June 4th, San Francisco - Test your skills, whether you're a seasoned VR enthusiast or a curious newcomer. We will also have food & drinks for when you're not playing.
• Electric Data Carnival: June 5th, San Francisco - Join us for good music, great people, and an unforgettable night at the Summit after-party of the year.[DEPRECATED] Marcos Marx
Justin Chau (Airbyte)
Justin Chau (Airbyte)
AJ Steers
06/04/2024, 5:48 PMNatalie
Legacy
state for connectors, as all connectors should utilize either Stream
or Global
states. Your connector's syncs may be affected if it is on an older version of the CDK. We've outlined migration steps here: https://github.com/airbytehq/airbyte/discussions/39430Natalie
Natalie
Natalie
run_ab_platform.sh
script will end in August 2024. With the release of abctl, we've provided a best-in-class tool that improves the initial user experience and ensures you are set up for a production-ready instance of Airbyte from the start.
To support the migration of existing Docker Compose deployments, we’ve created a migration path that we are beginning to test with external users, and looking for volunteers from the Community to help ensure our migration paths can adequately cover the ways in which users currently use Docker Compose. This beta group of testers will work directly with our Engineering team to migrate their existing Airbyte deployments to abctl
. You can read more and sign up for the beta testing group here: https://github.com/airbytehq/airbyte/discussions/40599[DEPRECATED] Marcos Marx
[DEPRECATED] Marcos Marx
Davin Chia (Airbyte)
0.390.0
.
Workloads is Airbyte's next generation Worker architecture. It is more scalable, reliable and maintainable than the previous Worker architecture. It performs particularly well in low-resource environments. Workloads is an Airbyte-internal job abstraction decoupling the number of running jobs (including those in queue), from the number of jobs that can be started. Jobs stay queued until more resources are available or canceled. This allows for better back pressure and self-healing in resource constrained environments.
Careful users will notice two new Airbyte services 1) the Workload Api Server and the 2) Workload Launcher. Both services function together to enable Workloads. While we do not expect OSS users to interact with these services often, or at all, we still want to note the additional services to minimise surprises.
Workloads is automatically enabled for Community users as of Airbyte Helm Chart version 0.390.0
. We recommend users upgrading from existing deployments to pause existing connections before upgrading to minimise errors.
Users interested in learning more can check out this blogpost and the public docs.
While we expect users to not need to interact with Workloads on a day-to-day basis, or at all, we welcome feedback/improvement on this Github Discussion. Thank you!Kat Wilson (Airbyte)
08/13/2024, 5:30 PMdestination-databricks
!
New features include:
• Per-record error handling
• Typing and deduping
• Refreshes
• Resumable Full Fresh
• All the great benefits of Destinations Version 2.0!
Please note: you must be using Unity Catalog to use this connector.
Learn more in the Github Discussion and comment here if you'd like to be added to our private slack channel.
Airbyte OSS and Cloud users are welcome!
Note: For OSS you must upgrade your Airbyte Platform version to at least 0.63.7 before upgrading to this connector version.[DEPRECATED] Marcos Marx
abctl local credentials
returns the email. This was a common problem for folks who migrated and didn’t remember which email they used in the legacy Docker deployment. Also the command now supports --email
and --password
to set email and password.
• ⛳ New flag --insecure-cookies
to anyone trying to access the instance in a non-localhost setup and didn’t configure a TLS certificate. Example: create a simple network rule to allow your private IP to access an EC2 instance.
• ⛳ New flag --low-resource
in case you don’t have the minimal requirements to run Airbyte through abctl. This can reduce performance but unlock you to use Airbyte for PoC or quick demos.
We hope everyone has a smooth transition migrating from Docker Compose or deploying a fresh install using abctl. If you have any troubles, don’t hesitate to contact @[DEPRECATED] Marcos Marx
👩💻 You can now contribute connectors in Connector Builder!
Both on Airbyte Cloud, and starting Airbyte OSS 0.63.19, you can now contribute your connectors to the Marketplace via Connector Builder!
Once you’re happy with your connector, if you’d like to make it available with everyone else who is using Airbyte, you can choose to contribute it to the Marketplace! . Builder will make the pull request with your new connector, we’ll review and help get it merged. We’re aiming for under 2 weeks review and merge time, but the first connector contributions that we see are merged in just 2 days.
☎️ Next Community Call
Join Joey Benamy from OncoLens to learn how they use Airbyte for seamless data integration. Discover insights on handling sensitive data, regulatory compliance, and deploying Airbyte in healthcare settings.
If you’re interested to learn about it, register here for the Community Call happening on September 9th at 8:30AM PT!
Do you have an interesting use case with Airbyte? We’d love to hear from you! Reach out to @[DEPRECATED] Marcos Marx and share your story! 🎈John (Airbyte)
John (Airbyte)
[DEPRECATED] Marcos Marx
[DEPRECATED] Marcos Marx
[DEPRECATED] Marcos Marx
[DEPRECATED] Marcos Marx
[DEPRECATED] Marcos Marx
[DEPRECATED] Marcos Marx
destination-s3-datalake
now supports refreshes / resumable full refresh.
• Iceberg support: Iceberg is the first datalake format this connector can write, with support for the Glue catalog and the Nessie catalog (new!) .
• Reliability improvements: We now leverage Iceberg’s snapshot branching capabilities to checkpoint and recover from failed syncs.
• Strongly-typed tables: The new connector writes data with true Iceberg types, instead of a single JSON string.
• Sync mode support: Dedup mode (via merge-on-read) is now available.
Learn more in the Github Discussion and reply here if you’d like to be added to our private slack channel. We’ll kick-off after the holidays are over.
Airbyte OSS users are welcome! Contact us if you’re interested in testing on Airbyte Cloud.
☘️ Airbyte & Motherduck Hackathon
The Airbyte + Motherduck hackathon is happening, and there is still time to participate. The event will run until 15 of January.
In this event, you will showcase your creativity by building a data app.
More details here or join #C05DVB06458 to discuss with other participants.[DEPRECATED] Marcos Marx
Chris Rose (Airbyte)
Kat Wilson (Airbyte)
02/10/2025, 6:25 PMChris Rose (Airbyte)
[DEPRECATED] Marcos Marx
[DEPRECATED] Marcos Marx
[DEPRECATED] Marcos Marx