Is there any workaround for copying raw files larger than 1GB? Or is that on the development roadmap? Thanks!

I believe this statement is accurate - Airbyte effectively doesn't work with a Redshift destination that makes use of Redshifts dynamic data masking (DDM) because Airbyte turns on case sensitivity to true. DDM doesn't let changing the case sensitivity configuration from the default for targets that have DDM enabled. And most Redshift I believe this condition is applied overly broadly, as I can't think of why it would matter for our use case. My yet unverified suspicion is that it is happening when we have a json datatype in our Postgres getting mapped to a varchar type in Redshift, and that varchar has masking applied. Or, it could just be unrelated to the datatype, and it could just be low velocity tables that didn't have updates. The relevant airbyte code This associated airbyte issue

I would have expected the migration guides to be linked in the change log, but could not see where the S3 migration guide was linked: https://airbytehq.slack.com/archives/C01AHCD885S/p1743766606708199

Any reason the today_with_timezone macro is not documented in YAML reference? Would have saved a bit of time if I knew this existed

when we click "upgrade all" in the connectors pages in OSS, does it just tell the Airbyte instance to fetch the newer version if and when it's used, or will it actually download 449 connector images in this case, even if they'll never be needed?

Given that Shopify has deprecated and removed support for the REST admin API since April 1 for all new apps, do we have a timeline for when

orders

and

customers

streams will move to GraphQL in

source-shopify

?

There are a couple of helm-chart-related PRs that are mostly straightforward, and you also have integration tests. It would help if you would add GHA workflow to run those tests against PRs https://github.com/airbytehq/airbyte-platform/tree/main/charts/helm-tests

Something weird happened with diagram https://docs.airbyte.com/platform/understanding-airbyte/high-level-view There are no labels on Chrome, Firefox or Safari Diagram here is fine https://github.com/airbytehq/airbyte/blob/master/docs/platform/understanding-airbyte/high-level-view.md

Airbyte Observability - Alerting Problem "Unfortunately, there doesn't appear to be a built-in solution in Airbyte to automatically generate alerts for parsing errors that don't cause a sync to fail. This is a limitation of the current notification system, which is primarily triggered by sync failure rather than by the presence of errors within a successful sync." Proposed Solution I think Airbyte should create a new status for a sync "PARTIAL_SUCCESS" to indicate a sync completed successfully but could not replicate all of the records e.g., due to a parsing error. This status type should then be included in the failed sync webhook to generate an alert e.g., to Slack. This will resolve the gap in alerting for syncs that did not replicate 100% of records. The user then knows that some form of remediation is required.

One feature that would be great to see and I don't think it would be so hard to put in place with what you have already done is the possibility to retrieve existing secrets in the secretStore of our choice. Instead of having airbyte creating a new secret in the secret store everytime we create a destination/source, we would be able to retrieve an existing one. 🙏 In my company we already have all of our secrets in a secretStore and we like to keep a single copy of this secret. It gives us the possibility to rotate the password more easily. For example in the case of AWS Secret Manager, have the possibility to give the ARN of the secret and it would retrieve it.

Hey folks octavia wave. We’re a little concerned about an upcoming deprecation to the V17 version of the Google Ads API that is currently used by Airbyte as part of their official supported connector. I opened an issue last week but there’s been no sign of movement on it. What are the chances that this will get picked up before the 4th and the connection breaks?

It would be helpful to allow memory customization for each container within a JOB, not just for the

JOB_MAIN_CONTAINER_*

settings. https://github.com/airbytehq/airbyte-platform/blob/main/charts/airbyte/templates/env-configmap.yaml#L98-L101

Hello, we are using Airbyte Cloud with BigQuery with GCS destination. After recent upgrade to 2.11 version for destination something went wrong and data is no longer stored in BigQuery tables at all, even though from logs it seems that all operations were successful. I have tested this with completely fresh sync setup, which extracted around 4k records, but this data table does not appear on Google BigQuery. Please contact me for further details.

Hi Airbyte, I would love to have a synch button in the connection page. This will help to save time when i need to run manual connectors. https://github.com/airbytehq/airbyte/discussions/61336

added an integration to this channel: Pointagram

The helm chart deploys a k8s secret

airbyte-airbyte-secrets

with db information and other secrets, but we still expect the user to manually deploy another secret with similar information into

airbyte-config-secrets

what’s the point of that? imo it just makes it confusing and harder to understand

I find nifi is really fast and more configurable table Airbyte. Eg. If i just want to load 1 day, one month of data using filters. It's forcing me to create view at source. i used nifi it's loading the data in 30 minutes however using Airbyte - it's 8 hours that's strange Source : oracle data compared with same cpu / memory specs !! (edited) We are dropping Airbyte and moving to nifi Please test the performance before adapting Airbyte. It wasted our efforts and impression to clients

Hello Airbyte Team, I’m quite disappointed with how Airbyte’s product and support have been recently. We’ve been using Airbyte for a couple of years now, and while it works well for data sources with high volumes, it has been a constant issue on critical pipelines. Specifically, the Intercom source has been a pain point, with catastrophic failures roughly once per quarter. This is not an acceptable threshold for a production pipeline. In the past, we at least felt supported, even if the suggested solution was often to rerun and reload everything on a new connection, which, of course, comes with extra costs and long waiting times. Now, we can’t even get a response after a pipeline breaks – even for what should be an “innocent” change. The Cloud UI lacks sufficient information about the stage a run fails or the error itself, making it impossible to understand how to fix the issue efficiently. We picked Airbyte to handle ingestion, not to spend our time trying to debug its logs and errors on our own. It’s frustrating to feel abandoned when we need support, especially when relying on Airbyte for business-critical pipelines. *Please let us know how this is going to be addressed and when we can expect meaningful support on our open ticket(12736).* Looking forward to any reply as we will need to evaluate our options for alternatives. Thank you

In airbyte-platform repository it would be nice to have tags also to patch versions (not only major.minor) https://github.com/airbytehq/airbyte-platform/tags E.g. there are tags for versions 1.6.0, 1.7.0, but not 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.7.1

Until recently, I touted AirByte as the greatest product of all time. I was able to pin the version of our custom sources and destinations, and could literally forget about it for years. Everything just worked. As of 7/2, our unmapped fields stopped working, which was a feature we relied upon to ingest the dynamic properties in our constantly evolving CRM. Sure, I got an email about AB dropping support for unmapped fields a little while ago, but it didn’t even cross my mind that this change would affect custom connectors/destinations with pinned legacy versions. Now a critical feed is broken, and I can’t even save changes to our connector (409 “unknown error occurred”, ticket 12947 if anyone’s listening) so that I can properly debug. What’s the point of allowing users to pin connector/destination versions if not to ensure consistent behavior?

Hey Airbyters, we recently encountered a scenario where the applySchemaChanger helper appears to have incorrectly applied a breaking schema change when the discover operation for a connector returned the default schemaless schema. As a result, we had to rollback the destination table and retrigger the sync. Is it possible to investigate this a bit further to determine how this is possible? It looks like the FieldTransform model has a method to create transforms for when a field is removed from a stream. Presumably this should normally be considered a breaking change.

Hi Airbyte, We’ve observed that Airbyte is consuming a high amount of CPU, memory, and swap, even on a powerful server with ample resources. This excessive resource utilization is causing unnecessary performance issues and system instability. We believe this behavior could be optimized further, especially for large-scale or production environments. Any improvements or guidance to reduce resource usage would be greatly appreciated.

Hello airbyters, we have been experiencing issues with our ELT pipeline since this morning. Our pipeline is airflow trigger 8 airbyte (airbyte cloud) connections through the airbyte API and monitor until done, then DBT etc. Before this morning, the 8 connections would trigger at the same time. Since this morning, the 8 connections are taking a lot of time to trigger. It looks like the airbyte API requires a lot of time to "process" the trigger. I tried to trigger some connections manually by clicking on the button "sync now" and it took 5 minutes before "registering" the trigger. Does anyone else has been facing similar issues ? Any insight would be greatly appreciated 🙏

Hey guys, Just want to report we are experiencing a critical issue with a connection to the Postgres connector since Monday. The connection has been stable for months, but is now returning errors we have never seen before. The errors supposedly seem to have to do with long running queries in Postgres that are causing the sync to fail and get stuck (it will run for hours doing nothing until we manually stop it). There have been no recent changes to our Postgres read replica, so we aren't sure what may have caused this issue all of a sudden. We've opened 2 support tickets (one yesterday, one today) but have not heard back from support yet. Any support here would be greatly appreciated!

hey Airbyte, We're migrating data from our old database to a new one. To save costs, we set up Airbyte to only sync recent data and moved the older data directly to the new database. However, we're finding that after each Airbyte sync, the data we moved manually keeps disappearing. Does Airbyte delete existing data that doesn't match its metadata expectations?

Hey Airbyte, We have been experiencing a breaking issue with the Snowflake source connector ever since got bumped from

1.0.1

to

1.0.2

. We are now getting a missing certificate JDBC error and are not able to rectify it on our side. I have confirmed this occurs on our organisation account as well as on a separate Snowflake trial account. As we use cloud, we are unable to roll back to the previous release.

java.lang.RuntimeException: Column name discovery query failed: JDBC driver internal error: Max retry reached for the download of chunk#0 (Total chunks: 1) retry: 7, 
error: net.snowflake.client.jdbc.SnowflakeSQLLoggedException: JDBC driver encountered communication error. Message: No trusted certificate found 
Verify that the hostnames and portnumbers in SYSTEM$ALLOWLIST are added to your firewall's allowed list. To troubleshoot your connection further, you can refer to this article: <https://docs.snowflake.com/en/user-guide/client-connectivity-troubleshooting/overview>.

I have contacted support 5 days ago but have not received a reply. Would love some sort of guidance or assistance as this has completely broken some of our critical syncs.

chart version:

v1.7.2

Hello good people, I'm having this issue in deploying Airbyte, when I try to deploy it using the helm chart I'm getting this error:

Could not determine release state: unable to determine cluster state: Deployment/data-pipelines/airbyte-workload-api-server dry-run failed: failed to create typed patch object (data-pipelines/airbyte-workload-api-server; apps/v1, Kind=Deployment): .spec.template.spec.containers[name="airbyte-workload-api-server-container"].env: duplicate entries for key [name="AB_JWT_SIGNATURE_SECRET"]

the issue is the charts has duplicate entries for

AB_JWT_SIGNATURE_SECRET

env variable which is inside workload-api-server chart -> templates -> deployment.yaml:

- name: WORKLOAD_API_BEARER_TOKEN
          valueFrom:
            secretKeyRef:
              name: {{ index .Values "workloadApi" "bearerTokenSecretName" | default (printf "%s-airbyte-secrets" .Release.Name ) }}
              key: {{ index .Values "workloadApi" "bearerTokenSecretKey" | default "WORKLOAD_API_BEARER_TOKEN" }}
        - name: AB_JWT_SIGNATURE_SECRET ---------------------------------------> First Entry here <----------------------------------------------------------------------
          valueFrom:
            secretKeyRef:
              name: {{ .Values.global.auth.secretName | default "airbyte-auth-secrets" | quote }}
              key: {{ .Values.global.auth.jwtSignatureSecretKey | default "jwt-signature-secret" | quote }}
        - name: MICRONAUT_ENVIRONMENTS
          valueFrom:
            configMapKeyRef:
              name: {{ .Release.Name }}-airbyte-env
              key: WORKERS_MICRONAUT_ENVIRONMENTS
        - name: AB_JWT_SIGNATURE_SECRET ---------------------------------------> Second Entry here <---------------------------------------------------------------------
          valueFrom:
            secretKeyRef:
              name: {{ .Values.global.auth.secretName | default "airbyte-auth-secrets" | quote }}
              key: {{ .Values.global.auth.jwtSignatureSecretKey | default "jwt-signature-secret" | quote }}

Pagination on the connections page doesn't seem to be working. When you have a lot of connections, some don't show. Not even if you apply filters. Cloud.

Hi all - Does anyone know if there is an update on the Google Ads connector upgrade to Google Ads API v19? I received a notification from Google Ads this morning that said v18 will stop accepting requests starting August 20, 2025.

Starting in v1.7 the webapp container was rolled into the server container, but neither helm chart v1 or v2 provide an equivalent to

webapp.ingress

in

values.yaml

for the server container. I see no github issues about this (open or closed). Surely I am not the only person in the world who was using this portion of the helm chart? Now it seems I have to separately deploy an ingress resource to make the webapp available again, which is a rather large breaking change in the chart itself. This could have easily been migrated seamlessly in the chart by reading

webapp.ingress

and using those values to make an ingress resource for the server container. Then this backfill functionality could be removed in the v2 helm chart in place of a dedicated

server.ingress

section in

values.yaml

I created the following issue for this: https://github.com/airbytehq/airbyte/issues/64941