We run airbyte open source version and we recently upgraded to 1.4.0. The source catalog contains 500+ source definitions now which is good. The public api however does not allow creating some of the newer sources by using their names because this file wasn't updated https://github.com/airbytehq/airbyte-platform/blob/main/airbyte-server/src/main/ko[…]airbyte/server/apis/publicapi/mappers/NameToDefinitionMapper.kt I guess the workaround would be to use definition id

If I'm understanding this correctly, I can't configure syncs to timeout faster than a day? https://docs.airbyte.com/operator-guides/configuring-airbyte#jobs

Hi team! We installed Airbyte for the first time to kick the tires on a local VM, and abctl has been a decent experience. I think we'd still strongly prefer a docker compose architecture because the added k8s (kind) complexity is already seemingly causing us headaches on the first day. Airbyte doesn't seem to be able to reach servers on our local network, despite the fact that host VM can reach them all fine. My suspicion is that abctl could be abstracting some network settings that cause the container network calls to go out to the open internet and not using our local DNS to resolve hosts first. It's also possible I have some very simple misconfiguration on the host VM, but I can't seem to think of what it could be. Is there some trick when installing via abctl to understand and troubleshoot networking issues? I would hope that with a tool designed for local deployments that it would natively be able to reach our local servers without deep diving k8s logs. Thanks for any insights you could share with someone new to k8s and Airbyte (but well versed in docker compose). Getting this error when trying to set up a MS SQL source that is reachable by the host VM and can connect to it with a database IDE without issue: "Connect timed out. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".

It would be really nice if there was a way to set the node selector globally in the helm chart. Edit: Asking kappa as well

What happened to the documentation around monitoring and OpenTelemetry? The only information seems to be related to DataDog: https://docs.airbyte.com/operator-guides/collecting-metrics#airbyte-opentelemetry-integration

We submitted a severity 1 issue last Thursday morning and have not heard anything back. This is not the first time Airbyte support has been unresponsive, which gives a poor impression.

Hi, is there a reason there is no filter for instagram business account id on the official airbyte instagram connector like exists for customer_id on google ads, facebook etc? I can see at some point there was discussion about it.

HiBob documentation appears to be dated and the connector fields are different from what I presume is required for a connection. A service user needs to be created but the connector just asks for a username and password (optional). A service user has an ID and token (not username password)

A way to increase the number of workers from the UI to help with a sync while it's running

Wanted to jump in again and say how much I enjoy your product. We have been syncing daily for 9 months and started adding other sources and the process is so smooth and the UI is chefskiss . No complaints but fantastic product!

Spammer / DM advertiser

Just out of curiosity, did you know the ready-to-launch version of the app on the Digitalocean marketplace is VERSION=0.30.25-alpha, and that it seemingly has no basic auth config? It would be nice if the license was changed to support platforms like Cloudron, like other open source apps ala Mattermost

The low-code declarative framework seems awesome, but has some rough edges. Specifically, trying to troubleshoot JSON schema validation errors is brutal currently as only the top level anyOf error is returned in the builder. I got stuck and ended up hacking something together to use the jsonschema library to iterate through error context and get some kind of actionable feedback Would be awesome to see this improved 🙂

from airbyte_cdk.sources.declarative.parsers.manifest_reference_resolver import (
    ManifestReferenceResolver,
)
from airbyte_cdk.sources.declarative.parsers.manifest_component_transformer import (
    ManifestComponentTransformer,
)

with open("my_declarative_manifest.yaml") as file:
    manifest = yaml.safe_load(file)

resolved_source_config = ManifestReferenceResolver().preprocess_manifest(manifest)
propagated_source_config = (
    ManifestComponentTransformer().propagate_types_and_parameters(
        "", resolved_source_config, {}
    )
)

validator = jsonschema.Draft7Validator(declarative_component_schema)

errors = validator.iter_errors(propagated_source_config)

# error_tree = jsonschema.ErrorTree(validator.iter_errors(propagated_source_config))

for error in sorted(errors, key=lambda e: e.path):
    print(f"Error in {list(error.path)}: {error.message}")
    print(f"  Schema path: {list(error.schema_path)}")
    print(f"  Instance: {error.instance}")
    for suberror in sorted(error.context, key=jsonschema.exceptions.relevance):
        print(f"  Suberror in {list(suberror.path)}: {suberror.message}")
        print(f"    Schema path: {list(suberror.schema_path)}")
        print(f"    Instance: {suberror.instance}")

error_tree = jsonschema.ErrorTree(validator.iter_errors(propagated_source_config))

if "streams" in error_tree:
    print("Errors in 'streams':")
    for path, error in error_tree["streams"].errors.items():
        print(f"  {path}: {error.message}")
        print(f"    Schema path: {list(error.schema_path)}")
        print(f"    Instance: {error.instance}")

with open("resolved_manifest.json", "w") as file:
    json.dump(resolved_source_config, file, indent=2)

with open("propagated_manifest.json", "w") as file:
    json.dump(propagated_source_config, file, indent=2)

👋 In my company, we plan to introduce airbyte, with the databricks as a destination. The problem is we cannot enable unity catalog, at least for now. Are the any plans to implement a databricks destination without requiring it to have the unity catalog enabled? If not, what would be the estimated effort to implement it as an OSS contribution?

hi – with the Snowflake destination connector, could performance be sped up if the files were staged in a more performant file format (parquet?), or is there more to this?

Is there a plan for Airbyte to support multi step authentication in the future without requiring python CDK?

API reference is down 😞 https://reference.airbyte.com/reference/getting-started

Hi all regarding the s3 data lake destination, i really like it and grateful for it one improvement that i would like to see is that it will create the table with a daily partition on the cursor field 🙏

I would like a way to download the list of fields of any expanded streams when selecting stream fields while building a connector. We are a consulting firm that helps other companies with their data. They often want to know fields are available so they can select what they want. Getting a list of available fields across multiple streams seems very cumbersome right now in the UI.

Hi all, I'm wondering if there are any plans to support AWS Systems Manager Parameter Store as an alternative to AWS Secrets Manager? It's used by many teams and has its place especially in cost conscious smaller setups. Thanks in advance!

Hi everyone - I'm running into a recurring friction point when deploying Airbyte on GCP that could significantly impact adoption. Specifically, the current authentication model relies on long-lived service account JSON keys to access GCP APIs (e.g., Secret Manager) This creates some tough trade-offs: • Many orgs now enforce short-lived credentials by policy—blocking them from using Airbyte entirely. • Others are forced to bypass best practices, increasing their risk posture just to get started. • GCP now disables SA JSON key creation by default in new orgs, which may unintentionally steer new users away from Airbyte before they’ve even begun. From what I can tell, AWS support includes short-lived credentials—are there blockers or constraints preventing similar support for GCP? Would love to hear from anyone who's tackled this, or if there's appetite to collaborate on a path forward. Related GitHub issues for context: https://github.com/airbytehq/airbyte/issues/12437 https://github.com/airbytehq/airbyte/issues/13081

It would be nice to have endpoints on Airbyte OSS documented 🙂 https://github.com/airbytehq/airbyte/issues/56965

Is there any workaround for copying raw files larger than 1GB? Or is that on the development roadmap? Thanks!

I believe this statement is accurate - Airbyte effectively doesn't work with a Redshift destination that makes use of Redshifts dynamic data masking (DDM) because Airbyte turns on case sensitivity to true. DDM doesn't let changing the case sensitivity configuration from the default for targets that have DDM enabled. And most Redshift I believe this condition is applied overly broadly, as I can't think of why it would matter for our use case. My yet unverified suspicion is that it is happening when we have a json datatype in our Postgres getting mapped to a varchar type in Redshift, and that varchar has masking applied. Or, it could just be unrelated to the datatype, and it could just be low velocity tables that didn't have updates. The relevant airbyte code This associated airbyte issue

I would have expected the migration guides to be linked in the change log, but could not see where the S3 migration guide was linked: https://airbytehq.slack.com/archives/C01AHCD885S/p1743766606708199

Any reason the today_with_timezone macro is not documented in YAML reference? Would have saved a bit of time if I knew this existed

when we click "upgrade all" in the connectors pages in OSS, does it just tell the Airbyte instance to fetch the newer version if and when it's used, or will it actually download 449 connector images in this case, even if they'll never be needed?

Given that Shopify has deprecated and removed support for the REST admin API since April 1 for all new apps, do we have a timeline for when

orders

and

customers

streams will move to GraphQL in

source-shopify

?

There are a couple of helm-chart-related PRs that are mostly straightforward, and you also have integration tests. It would help if you would add GHA workflow to run those tests against PRs https://github.com/airbytehq/airbyte-platform/tree/main/charts/helm-tests

Something weird happened with diagram https://docs.airbyte.com/platform/understanding-airbyte/high-level-view There are no labels on Chrome, Firefox or Safari Diagram here is fine https://github.com/airbytehq/airbyte/blob/master/docs/platform/understanding-airbyte/high-level-view.md