Wanted to jump in again and say how much I enjoy your product. We have been syncing daily for 9 months and started adding other sources and the process is so smooth and the UI is chefskiss . No complaints but fantastic product!

Spammer / DM advertiser

Just out of curiosity, did you know the ready-to-launch version of the app on the Digitalocean marketplace is VERSION=0.30.25-alpha, and that it seemingly has no basic auth config? It would be nice if the license was changed to support platforms like Cloudron, like other open source apps ala Mattermost

The low-code declarative framework seems awesome, but has some rough edges. Specifically, trying to troubleshoot JSON schema validation errors is brutal currently as only the top level anyOf error is returned in the builder. I got stuck and ended up hacking something together to use the jsonschema library to iterate through error context and get some kind of actionable feedback Would be awesome to see this improved 🙂

from airbyte_cdk.sources.declarative.parsers.manifest_reference_resolver import (
    ManifestReferenceResolver,
)
from airbyte_cdk.sources.declarative.parsers.manifest_component_transformer import (
    ManifestComponentTransformer,
)

with open("my_declarative_manifest.yaml") as file:
    manifest = yaml.safe_load(file)

resolved_source_config = ManifestReferenceResolver().preprocess_manifest(manifest)
propagated_source_config = (
    ManifestComponentTransformer().propagate_types_and_parameters(
        "", resolved_source_config, {}
    )
)

validator = jsonschema.Draft7Validator(declarative_component_schema)

errors = validator.iter_errors(propagated_source_config)

# error_tree = jsonschema.ErrorTree(validator.iter_errors(propagated_source_config))

for error in sorted(errors, key=lambda e: e.path):
    print(f"Error in {list(error.path)}: {error.message}")
    print(f"  Schema path: {list(error.schema_path)}")
    print(f"  Instance: {error.instance}")
    for suberror in sorted(error.context, key=jsonschema.exceptions.relevance):
        print(f"  Suberror in {list(suberror.path)}: {suberror.message}")
        print(f"    Schema path: {list(suberror.schema_path)}")
        print(f"    Instance: {suberror.instance}")

error_tree = jsonschema.ErrorTree(validator.iter_errors(propagated_source_config))

if "streams" in error_tree:
    print("Errors in 'streams':")
    for path, error in error_tree["streams"].errors.items():
        print(f"  {path}: {error.message}")
        print(f"    Schema path: {list(error.schema_path)}")
        print(f"    Instance: {error.instance}")

with open("resolved_manifest.json", "w") as file:
    json.dump(resolved_source_config, file, indent=2)

with open("propagated_manifest.json", "w") as file:
    json.dump(propagated_source_config, file, indent=2)

👋 In my company, we plan to introduce airbyte, with the databricks as a destination. The problem is we cannot enable unity catalog, at least for now. Are the any plans to implement a databricks destination without requiring it to have the unity catalog enabled? If not, what would be the estimated effort to implement it as an OSS contribution?

hi – with the Snowflake destination connector, could performance be sped up if the files were staged in a more performant file format (parquet?), or is there more to this?

Is there a plan for Airbyte to support multi step authentication in the future without requiring python CDK?

API reference is down 😞 https://reference.airbyte.com/reference/getting-started

Hi all regarding the s3 data lake destination, i really like it and grateful for it one improvement that i would like to see is that it will create the table with a daily partition on the cursor field 🙏

I would like a way to download the list of fields of any expanded streams when selecting stream fields while building a connector. We are a consulting firm that helps other companies with their data. They often want to know fields are available so they can select what they want. Getting a list of available fields across multiple streams seems very cumbersome right now in the UI.

Hi all, I'm wondering if there are any plans to support AWS Systems Manager Parameter Store as an alternative to AWS Secrets Manager? It's used by many teams and has its place especially in cost conscious smaller setups. Thanks in advance!

Hi everyone - I'm running into a recurring friction point when deploying Airbyte on GCP that could significantly impact adoption. Specifically, the current authentication model relies on long-lived service account JSON keys to access GCP APIs (e.g., Secret Manager) This creates some tough trade-offs: • Many orgs now enforce short-lived credentials by policy—blocking them from using Airbyte entirely. • Others are forced to bypass best practices, increasing their risk posture just to get started. • GCP now disables SA JSON key creation by default in new orgs, which may unintentionally steer new users away from Airbyte before they’ve even begun. From what I can tell, AWS support includes short-lived credentials—are there blockers or constraints preventing similar support for GCP? Would love to hear from anyone who's tackled this, or if there's appetite to collaborate on a path forward. Related GitHub issues for context: https://github.com/airbytehq/airbyte/issues/12437 https://github.com/airbytehq/airbyte/issues/13081

It would be nice to have endpoints on Airbyte OSS documented 🙂 https://github.com/airbytehq/airbyte/issues/56965

Is there any workaround for copying raw files larger than 1GB? Or is that on the development roadmap? Thanks!

I believe this statement is accurate - Airbyte effectively doesn't work with a Redshift destination that makes use of Redshifts dynamic data masking (DDM) because Airbyte turns on case sensitivity to true. DDM doesn't let changing the case sensitivity configuration from the default for targets that have DDM enabled. And most Redshift I believe this condition is applied overly broadly, as I can't think of why it would matter for our use case. My yet unverified suspicion is that it is happening when we have a json datatype in our Postgres getting mapped to a varchar type in Redshift, and that varchar has masking applied. Or, it could just be unrelated to the datatype, and it could just be low velocity tables that didn't have updates. The relevant airbyte code This associated airbyte issue

I would have expected the migration guides to be linked in the change log, but could not see where the S3 migration guide was linked: https://airbytehq.slack.com/archives/C01AHCD885S/p1743766606708199

Any reason the today_with_timezone macro is not documented in YAML reference? Would have saved a bit of time if I knew this existed

when we click "upgrade all" in the connectors pages in OSS, does it just tell the Airbyte instance to fetch the newer version if and when it's used, or will it actually download 449 connector images in this case, even if they'll never be needed?

Given that Shopify has deprecated and removed support for the REST admin API since April 1 for all new apps, do we have a timeline for when

orders

and

customers

streams will move to GraphQL in

source-shopify

?

There are a couple of helm-chart-related PRs that are mostly straightforward, and you also have integration tests. It would help if you would add GHA workflow to run those tests against PRs https://github.com/airbytehq/airbyte-platform/tree/main/charts/helm-tests

Something weird happened with diagram https://docs.airbyte.com/platform/understanding-airbyte/high-level-view There are no labels on Chrome, Firefox or Safari Diagram here is fine https://github.com/airbytehq/airbyte/blob/master/docs/platform/understanding-airbyte/high-level-view.md

Airbyte Observability - Alerting Problem "Unfortunately, there doesn't appear to be a built-in solution in Airbyte to automatically generate alerts for parsing errors that don't cause a sync to fail. This is a limitation of the current notification system, which is primarily triggered by sync failure rather than by the presence of errors within a successful sync." Proposed Solution I think Airbyte should create a new status for a sync "PARTIAL_SUCCESS" to indicate a sync completed successfully but could not replicate all of the records e.g., due to a parsing error. This status type should then be included in the failed sync webhook to generate an alert e.g., to Slack. This will resolve the gap in alerting for syncs that did not replicate 100% of records. The user then knows that some form of remediation is required.

One feature that would be great to see and I don't think it would be so hard to put in place with what you have already done is the possibility to retrieve existing secrets in the secretStore of our choice. Instead of having airbyte creating a new secret in the secret store everytime we create a destination/source, we would be able to retrieve an existing one. 🙏 In my company we already have all of our secrets in a secretStore and we like to keep a single copy of this secret. It gives us the possibility to rotate the password more easily. For example in the case of AWS Secret Manager, have the possibility to give the ARN of the secret and it would retrieve it.

Hey folks octavia wave. We’re a little concerned about an upcoming deprecation to the V17 version of the Google Ads API that is currently used by Airbyte as part of their official supported connector. I opened an issue last week but there’s been no sign of movement on it. What are the chances that this will get picked up before the 4th and the connection breaks?

It would be helpful to allow memory customization for each container within a JOB, not just for the

JOB_MAIN_CONTAINER_*

settings. https://github.com/airbytehq/airbyte-platform/blob/main/charts/airbyte/templates/env-configmap.yaml#L98-L101

Hello, we are using Airbyte Cloud with BigQuery with GCS destination. After recent upgrade to 2.11 version for destination something went wrong and data is no longer stored in BigQuery tables at all, even though from logs it seems that all operations were successful. I have tested this with completely fresh sync setup, which extracted around 4k records, but this data table does not appear on Google BigQuery. Please contact me for further details.

Hi Airbyte, I would love to have a synch button in the connection page. This will help to save time when i need to run manual connectors. https://github.com/airbytehq/airbyte/discussions/61336

added an integration to this channel: Pointagram

The helm chart deploys a k8s secret

airbyte-airbyte-secrets

with db information and other secrets, but we still expect the user to manually deploy another secret with similar information into

airbyte-config-secrets

what’s the point of that? imo it just makes it confusing and harder to understand

I find nifi is really fast and more configurable table Airbyte. Eg. If i just want to load 1 day, one month of data using filters. It's forcing me to create view at source. i used nifi it's loading the data in 30 minutes however using Airbyte - it's 8 hours that's strange Source : oracle data compared with same cpu / memory specs !! (edited) We are dropping Airbyte and moving to nifi Please test the performance before adapting Airbyte. It wasted our efforts and impression to clients