Sharing some highlights of impactful user-facing u...
# releases
n
Sharing some highlights of impactful user-facing updates from the last few weeks! 💪 Building the best connectors • MySQL has been enhanced to ensure large historical syncs are successful. During an initial sync, Airbyte now gracefully consumes WAL and acknowledges logs periodically to ensure disk space is freed in a timely manner. • Source S3 has become faster! With CDK 2.0, the protocol now uses Pydantic V2 and removes the serialization of each record. As connectors are updated to the new CDK version, additional connectors will see varying levels of speed improvements. • Destination Redshift (3.3.0) and Snowflake (3.11.0) support Refreshes, which reduces data downtime when resyncing historical data. They join BigQuery in also supporting Resumable Full Refresh, which ensures the success of large Full Refresh syncs. octavia loves Making Airbyte Intuitive and Easy to Use • Native authentication by username and password is available in Airbyte OSS so that any instance of Airbyte is secure by default. • Managing and obtaining API keys is now consistent across all our products. We have brought our API portal in-house and API keys can now be generated and found in the “Applications” page of our UI. Along with this, the Terraform provider and Java/Python SDKs have been updated to automatically retrieve tokens between each action, so that users don’t need to worry about fetching tokens themselves. • As we prepare to deprecate Docker Compose, we published a migration guide for those migrating from Docker Compose to abctl. • The progress of active syncs is now reported every 10 seconds for increased visibility and tracking (available in OSS v.0.63.5) • Schema discovery is now always run as a part of the sync for Airbyte Cloud. This ensures we always run the sync with the latest schema, which ensures the auto-propagation and backfill features works as expected. • Cloud customers will now get email notifications when their syncs are running for an unusually long amount of time. 🤝 Improving connector & community leverage • Community Connectors are now found in the Marketplace. In the UI, we now show three tabs on the source/destination selection page: Airbyte Connectors (formerly "certified"), Marketplace (formerly "community"), and Custom. Connectors in the Marketplace tab now also contain success rate and usage metric info to help users understand how the connector is used. • Connector Builder has added support for regex_search to ease filtering for substrings, originating from this community request • Connector Builder has improved warnings to help with testing & validation while building. We’ve made clearer warnings to indicate when API requests fail, no response is received from the API, no records could be found, or issues are found with the primary key. We’ll also indicate exactly which streams are affected. • For OSS users, the Airbyte platform will notify users in the UI when upgrading a connector requires a specific corresponding platform version. This is helpful when a connector version upgrade relies on a platform feature to be available for the sync to succeed. • pyAirbyte now integrates with Apache Arrow and supports Pydantic 2.0 and CDK 2.0! Pydantic 2.0 support was a frequently-requested feature for PyAirbyte users and the benefit for users is (1) improved performance, and (2) improved compatibility with modern Python tools.
🔥 4
airbyte rocket 4
j
@Natalie Lots of great stuff in this, thanks for getting the notes out! One this one:
The progress of active syncs is now reported every 10 seconds for increased visibility and tracking (available in OSS v.0.63.5)
. . . one thing I wanted to point out is that since the addition of the update interval on active syncs, we've noticed an increase in the number of HTTP 500 errors we see when working in the UI and having these pages on screen. It seems like these just happen occasionally (maybe related to the auth proxy Airbyte sits behind? not sure if everyone is seeing these), but with more requests being sent it comes up more often . . . it may be good for these background checks to be treated more opportunistically and not trigger UI errors (at least not until it hits some threshold of missed requests, e.g. 3 or 6) to help avoid transient errors that users likely wouldn't even notice otherwise. Just an idea that might help reduce user-impacting error noise 🙂
👀 1
c
it may be good for these background checks to be treated more opportunistically and not trigger UI errors (at least not until it hits some threshold of missed requests, e.g. 3 or 6) to help avoid transient errors that users likely wouldn't even notice otherwise.
Great thing to note, I'll make sure its discussed in our frontend sync. I'm seeing ~10/day error responses for this endpoint, so not a huge increase but definitely something to watch. There are some other background calls we make to keep page data fresh, (and also error at times) and we should find a way to keep them from crashing the page.
👍 1
t
@Justin Beasley Do you happen to have a chance getting us the actual 500 error content (with the stacktrace) in case it happens again? I'd be curious to see the exact error that's happening there.
j
@Tim Roes (Airbyte) Yep, happy to grab it next time I'm in the UI and see one. Anything else you want me to grab or check at the same time? (I know stuff like this can be a pain to repro.)
t
I think just the error (as long as it has a stacktrace in it) would be a good start. Since it could be anything at that point, can't really point to anything else to grab before seeing the error, sorry 😄
👍 1
j
@Tim Roes (Airbyte) Here's one, but you're not going to like it 😂:
Copy code
{
  "url": "https://[REDACTED]/workspaces/1ad4ae36-9516-4549-9a2b-b161029b4411/connections/29344389-b2bb-43a1-9fff-abfbbccf4833/replication",
  "airbyteVersion": "0.63.13",
  "errorType": "HttpError",
  "errorConstructor": "Th",
  "error": {
    "i18nKey": "errors.http.internalServerError",
    "i18nParams": {
      "status": 500
    },
    "name": "HttpError",
    "requestId": "kAGoJSntvmMeXXm5s9qhQN",
    "request": {
      "url": "/api/v1/actor_definition_versions/get_for_destination",
      "method": "post",
      "headers": {
        "Content-Type": "application/json"
      },
      "data": {
        "destinationId": "02e9b259-639f-44f6-9732-e1a889d6ee2a"
      }
    },
    "status": 500,
    "response": "An internal server error occurred."
  },
  "stacktrace": "HttpError: errors.http.internalServerError\n    at Vpe (https://[REDACTED]/assets/core-lx5fw6bdzu.js:119:9851)",
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
  "featureFlags": {}
}
Not particularly helpful there. This one wasn't the endpoint I normally see, so I'll keep an eye out for another. But it did pop up on a sync screen sitting idle, so I'm guessing it was still a background request of some type.
t
Interesting 🤔 Given we don't have any stack trace on the
response
but it's a plain string, this feels like it's not coming from the airbyte server even, but is caused by something else within the infrastructure?
j
hm, it's pretty basic . . . GKE Autopilot behind a GCP HTTP/S LB ingress. The only other thing I can think of is Cloud IAP (which we're using to lock down the UI by Google Account, and the API by service account JWT). I'll see if there's anything in the logs for IAP that might correlate. I also wonder if auth proxies in general may be a problem with the background polling due to the short token lifetimes . . . maybe they're expiring and not being handled in a way that will catch a new one. but I would imagine that would affect just about any auth proxy like this, not just IAP.
t
Regarding the polling: We discussed this yesterday, and we'll make the webapp "ignore" errors in this polling mostly. That way we'll not fail out the whole stream screen once an individual attempt to get stats fails. So even if that's caused by the infrastructure the webapp should be more resilient towards that in the future (since we'd consider the "progress data" not be crucial enough to cause the webapp to fail on).
👍 1
j
Sounds good. It's quite noisy overall, about 1-4 requests per second across `list`/`latest_per_run_state`/`status`/`health`/`sync_progress`/`get_for_destination` (thinking of that with multiple users/multiple tabs/etc. that could add up quickly—I wonder if there's some opportunity to batch, bundle, or otherwise optimize those). Seems like sometimes there's a big spike in requests sent at once which I think may be when a tab has been background-idled by Chrome, and hundreds get dispatched at once which could put undue load on the backend depending on how long the tab has been idle . . . seems like Chrome isn't very smart and sends a lot of redundant requests. Not something that looks like it's urgently breaking, but I imagine on a larger deployment with enough users in the UI the impact could become significant. But I digress 🙂 I'll let you know what I find on the IAP side, as it may apply to experience with other auth/proxy situations too
t
I know that we want to look again on how some of those calculation for stream statuses work. cc @Chandler Prall (Airbyte) I wonder if consolidating some of that information from/into the same API was a part of this
nope 1
For the health API we btw have an (currently iceboxed) issue to remove the check in OSS. We already removed it for Cloud since... it put up too much pressure on the servers 😄 So we should still remove this for OSS as well. It's mostly there to show the "the webapp might still be booting up" screen. However that feels like a too small win for all those health API requests.
🙃 1