Using connector builder, can i return a list of codes that i want to use as paramters into the child stream - but it only accepts 50 codes at a time

When running a newer version of abctl, do i need to also provide the values/secrets files? Or are these settings maintained,even on local install.

Has anyone created a Hubspot destination on Airbyte? Since I am running Airbyte in a private EC2 which I connect to it via VPN, Hubspot cannot connect to

/auth_flow

because my Airbyte is not exposed to public. Is there a work around to it?

I am running Airbyte 1.8 using helm ,, it was working but now I am getting below error while check_connection "failureOrigin": "airbyte_platform", "externalMessage": "Workload failed, source: airbyte_platform", "internalMessage": "Unable to persist the job Output, check the document store credentials.", While create source is successful

Hi. We are using airbyte 1.8 here. How to streaming data from postgres to iceberg+nessie table, but following the existing iceberg table partitioning? We pre-created the iceberg table with partitioning. Then when the sync is over, there is new folder on the object storage, containing the data, but there is no partitioning. Meanwhile if we use normal INSERT query to the iceberg table, it automatically apply the partitioning (The folder location and hierarchy level are identical)

For those who have encountered problems with Zscaler SSL Inspection (or similar endpoint MITM tech) preventing Airbyte Docker images from spinning up; What domains / URLs did you whitelist in order to unblock?

Hi All - has anyone had any success getting past OOMKilled errors for larger tables to postgres destination? An example problem stream I have is initial load for MySQL CDC source -> Postgres destination, where the table contains 3.7m records / 5.6GB data. I have tried tuning the resources and confirmed my settings are applying, but my syncs reliably fail whenever i’m trying to sync a table where the entire dataset does not fit into memory, logs show that it is getting past fetching data and the “typing and deduping” steps, and is moving on to copying data to the destination. According to documentation, Airbyte is supposed to batch into 10000 row sets - but if my math is correct Airbyte should be batching the data in approximately 15MB per batches. This seems contrary to what im actually seeing where the only way to get a stream to succeed is to allocate at least 2x the data size in memory. Im currently running Airbyte 2.0.0 (Helm chart 2.0.18), but this behavior has been consistently a problem in older versions too. Over the past weeks I have tested pretty much every version between 1.6.2 and 2.0.0. Happy to share more info if it is helpful.

Hi team! 👋 I'm running into an issue with increasing memory for connector pods in Airbyte 2.0 on Kubernetes (Helm v2 charts). ***Problem:*** My postgres source connector is being OOMKilled with the default 2GB limit, and I need to increase it to 4GB. ***Documentation gap:*** The "Configuring Connector Resources" page (https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources) says to "set the following variables in your values.yaml file" and lists: - JOB_MAIN_CONTAINER_MEMORY_REQUEST - JOB_MAIN_CONTAINER_MEMORY_LIMIT - etc. But it doesn't specify where in values.yaml these should go. After digging through GitHub issues and the Helm chart structure, I think they go under

workloadLauncher.extraEnv

, but I'm not certain. ***Questions:*** 1. Is

workloadLauncher.extraEnv

the correct place for these variables in Helm v2? 2. Is there newer documentation or a different approach I should be using instead? 3. Would it be possible to update the docs to show the full values.yaml structure? Currently using: - Airbyte 2.0 with Helm v2 charts - K3s cluster - RDS Postgres → Snowflake connection Thanks for any guidance! The product is great, just trying to get the resource configuration sorted. 🙏 Thanks!

Hi I want to upgrade to airbyte version 2.0.0 but still using helm v1. So the chart version pointing to

version          = "1.8.5"

Then in values.yaml, should I change the global.version to "2.0.0" and also put the image name and tag in global.image.tag? I dont want to change to helm v2 because it will required some effort, and if I force it it will give error like

Message: Could not resolve placeholder ${AIRBYTE_URL}

Message: Could not resolve placeholder ${STORAGE_BUCKET_AUDIT_LOGGING}

Hello there 🙂 I created a sync from BigQuery (source) to HubSpot (Destination). I works fine. However, I added a column in my BigQuery model, but have not seen this column appear in the Airbyte mapping tab.

Hello everyone 🙂 I'm testing a self hosted installation and I don't seem to be able to create new workspaces via the UI. I ran the

abctl install local

and I have the

organization-admin

permission but no way to manage org/workspaces in the UI. I'm attaching a screenshot of what I see in the UI. I'm able to create workspaces over API but they don't seem to be available for my user either.

Hi folks! Has anybody experienced something like this on a kubernetes deployment (EKS, AB 1.8, Helm v2)? Seeing every nth sync fail due to

An internal transient Airbyte error has occurred. The sync should work fine on the next retry.

Same pipeline configurations worked fine on an EC2 based deployment. It doesn't give much to go on but sounds like it could be a resource availability issue. I've been trying to tweak some configs, but have not been able to resolve yet.

Hi, I want to create OAuth override credentials for a workspace and source type. I'm using this guide: https://reference.airbyte.com/reference/workspaceoauthcredentials The endpoint returns 200ok, but I still can't use my own credentials. Any ideas on where I'm missing something? I already have the redirects and my credentials configured in the Google console?

Anyone ever have trouble with creating sources via the api - specifically when I create a source, the delivery method is not set, which makes the source break everything else it is used for unless I go into the ui and manually set it myself

Can i use a partition router in version 1.8.5 by using custom components - show me

still having this issue after trying to install airbyte using the latest abctl version after trying to access it // HttpError { "i18nKey": "errors.http.notFound", "i18nParams": { "status": 404 }, "name": "HttpError", "requestId": "4jqEJZXqfhVtwdBDxQv3p2", "request": { "url": "/api/v1/workspaces/get", "method": "POST", "headers": { "Content-Type": "application/json" }, "data": { "workspaceId": "48e3540c-07e9-403d-a5b3-a4190b6362d7" } }, "status": 404, "response": { "message": "Internal Server Error: Could not find configuration for STANDARD_WORKSPACE: 48e3540c-07e9-403d-a5b3-a4190b6362d7.", "exceptionClassName": "io.airbyte.commons.server.errors.IdNotFoundKnownException",

Hello my airbyte still not running any sync on 2.0 version. (original posted on releases date https://airbytehq.slack.com/archives/C021JANJ6TY/p1760524222333319) Everything was working well before the update, but after upgrading to 2.0, the sync won't start. The error I get is 'Airbyte could not start the process within time limit. The workload was never claimed.' Airbyte runned without low-resource mode on GCP VM with 8 vCPUs, 16 GB Memory. Any suggestions to fix it?

@kapa.ai airbyte losing DB connection due to authentication issue

m running AIrbyte 1.8.5 and am trying to use GroupingPartitionRouter but cant seem to get it to work. Is it possible?

Hello, our Facebook Marketing sync has recently started failing with:

Warning from destination: Input was fully read, but some streams did not receive a terminal stream status message. If the destination did not encounter other errors, this likely indicates an error in the source or platform.

Connector is on the latest version

4.1.0

Any guidance on how to best troubleshoot / fix this?

Hey guys, I setup the airbyte open source version and im running into a issue with from db2 to snowflake is stalling after 21 million records and mssql to snowflake is stalling at 4 million. I checked the logs for db2 and im only seeing infor not any warnings. Let me know if i can provide any logs or other info. Thanks!

Hey all, we have a custom MS Entra ID connector that we forked and seems to not accept creating a new source via the API (I'm using Postman for the request in this case). The reason seems to be that it does not accept

microsoft-entra-id

as the

sourceType

, I got that value from the source code of the connector. The request body is as follows:

{
  "name": "microsoft_entra_id_test",
  "workspaceId": "{{WORKSPACE_ID}}",
  "configuration": {
    "sourceType": "microsoft-entra-id",
    "start_date": "2017-01-25T00:00:00Z",
    "enable_experimental_streams": false,
    "credentials": {
      "credentials_title": "OAuth Credentials",
      "client_id": "{{CLIENT_ID}}",
      "client_secret": "{{CLIENT_SECRET}}",
      "refresh_token": "{{REFRESH_TOKEN}}"
    }
  }
}

Response:

{
  "status": 400,
  "type": "[<https://reference.airbyte.com/reference/errors](https://reference.airbyte.com/reference/errors)>",
  "title": "value-not-found",
  "detail": "The submitted value could not be found.",
  "documentationUrl": null,
  "data": {
    "value": "microsoft-entra-id"
  }
}

This throws an error, saying it cannot find the

sourceType

of that value. This can be right since the connector we are using is a custom one so I requested the source details to see what kind of

sourceType

it needs to work, it said it was set as an empty string:

{
  "sourceId": "8cf5d308-7056-45e5-8948-11d217607019",
  "name": "Microsoft Entra Id",
  "sourceType": "",
  "definitionId": "9e1d5c73-b7a2-4642-b224-f78379fdf4d5",
  "workspaceId": "7d4c5653-bd89-454a-b520-173e77c68fc8",
  "configuration": {
    "user_id": "**********",
    "client_id": "**********",
    "tenant_id": "**********",
    "client_secret": "**********",
    "__injected_declarative_manifest": {},
    "__injected_components_py_checksums": {}
  },
  "createdAt": 1761225116
}

That's fine, however, setting it to an empty string yields the same response:

{
  "name": "microsoft_entra_id_test",
  "workspaceId": "{{WORKSPACE_ID}}",
  "configuration": {
    "sourceType": "",
    "start_date": "2017-01-25T00:00:00Z",
    "enable_experimental_streams": false,
    "credentials": {
      "credentials_title": "OAuth Credentials",
      "client_id": "{{CLIENT_ID}}",
      "client_secret": "{{CLIENT_SECRET}}",
      "refresh_token": "{{REFRESH_TOKEN}}"
    }
  }
}

Response:

{
  "status": 400,
  "type": "[<https://reference.airbyte.com/reference/errors](https://reference.airbyte.com/reference/errors)>",
  "title": "value-not-found",
  "detail": "The submitted value could not be found.",
  "documentationUrl": null,
  "data": {
    "value": ""
  }
}

Is there any idea what I need to add to

sourceType

? I thought it was

microsoft-entra-id

but that does not work. Many thanks!

Hi All! I’m running the Shopify connector in airbyte cloud, and just trying to sync the orders table for now… I’m curious what kind of speeds to expect? It’s running 8/9 hours for 1-2M rows, and I’m not sure there’s any more optimizations to make. Does oss / pro / enterprise enable higher speeds?

GA4 connector version shows up as 2.9.14 in https://docs.airbyte.com/integrations/sources/google-analytics-data-api, how do I know when the previous versions of the connector were released (i.e history of releases)

Hey everyone, We’re currently seeing an issue where Airbyte pipelines aren’t starting properly, and no new logs are appearing after restart attempts. Several connections have been stuck in a running state for over two hours with no progress, and stopping/restarting them doesn’t help. It seems to affect multiple connections. We’ve already opened a support ticket with Airbyte, is anyone else here experiencing the same issue?

Facing an issue - my use case - Incremental sync from db1 to db2 (cursor based) and use DB2 for some analytics. (so indexign and primary keys on db2 tables is needed) When I incremental append + dedupe sync (cursor based) a table from Db1 to db2 , 1. It doesnt create primary keys on destination - which is ok . I can do manually. 2. I create the primary key then next time when i do incremental - append + dedup. It starts giving duplicate id issues. I am surprised if this is not supported by airbyte as this is a pretty common use case.

Hi all, I'm running Airbyte on Docker, and when setting up CDC for PostgreSQL, no streams are appearing. Any idea what could be wrong?

can someone give me the requirements for airbyte open source?

This message was deleted.

neeraj@airbyte-prod:~$ docker run -d --name airbyte -p 8080:8080 airbyte/airbyte:1.7.0 Unable to find image 'airbyte/airbyte:1.7.0' locally docker: Error response from daemon: pull access denied for airbyte/airbyte, repository does not exist or may require 'docker login': denied: requested access to the resource is denied Run 'docker run --help' for more information neeraj@airbyte-prod:~$ sudo docker run -d --name airbyte -p 8080:8080 airbyte/airbyte:1.7.0 Unable to find image 'airbyte/airbyte:1.7.0' locally docker: Error response from daemon: pull access denied for airbyte/airbyte, repository does not exist or may require 'docker login': denied: requested access to the resource is denied Run 'docker run --help' for more information neeraj@airbyte-prod:~$