@kapa.ai where to find the airbyte cdk versions

I am trying to move a google sheet to snowflake but the google sheet name has a space in it. I would like to remove the space to make downstream processing easier, as it is currently creating a table with a space in the name. Is there a way I can accomplish this? It appears to be easy for column names, but I don't see an easy option for the sheet name itself. Thanks!

@kapa.ai Hello, I have upgraded to Airbyte v1 using

abctl

but now, the sync wont start. Looking at the kubernetes pods, there seems to be an error, though it not clear what the issue might be.

ubuntu@ip-10-0-60-1:~$ kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig -n airbyte-abctl get pods
NAME                                                     READY   STATUS             RESTARTS       AGE
airbyte-abctl-airbyte-bootloader                         0/1     Completed          0              19h
airbyte-abctl-connector-builder-server-f7b8c4d7b-6w7qh   1/1     Running            0              19h
airbyte-abctl-cron-7f6dc9484-2d98n                       1/1     Running            1 (18h ago)    19h
airbyte-abctl-pod-sweeper-pod-sweeper-856cf644f9-zf2tl   1/1     Running            0              19h
airbyte-abctl-server-8477ffd8d6-97xrx                    1/1     Running            12 (18h ago)   19h
airbyte-abctl-temporal-857dd9f6f4-xkwtq                  1/1     Running            0              19h
airbyte-abctl-webapp-cccb469b8-b95rb                     1/1     Running            0              19h
airbyte-abctl-worker-54df9686cc-qnvzq                    1/1     Running            0              19h
airbyte-abctl-workload-api-server-6867bffd8b-b2mvf       1/1     Running            0              19h
airbyte-abctl-workload-launcher-5f6bc4cd55-nnpl4         1/1     Running            0              19h
airbyte-minio-0                                          1/1     Running            0              19h
source-postgres-check-22394-1-ikhqn                      0/2     Error              0              120m
source-postgres-check-22394-2-oyrpg                      0/2     Error              0              110m
source-postgres-check-22394-3-mpzgz                      0/2     Error              0              99m
source-postgres-check-22394-4-dmcio                      0/2     Error              0              86m
source-postgres-check-22395-0-wvktf                      1/2     ImagePullBackOff   0              85s

It there a way to dig deeper into the issue? As for the sync job logs, they stay in a loop like this:

2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: CLAIM — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:49 INFO i.a.c.i.LineGobbler(voidCall):166 - 
2024-11-01 14:39:38 INFO i.a.w.l.c.WorkloadApiClient(claim):75 - Claimed: true for decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check via API for local
2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: CHECK_STATUS — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:38 INFO i.a.w.l.p.s.CheckStatusStage(applyStage):59 - No pod found running for workload decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check
2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: BUILD — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: MUTEX — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:38 INFO i.a.w.l.p.s.EnforceMutexStage(applyStage):50 - No mutex key specified for workload: decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check. Continuing...
2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: LAUNCH — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:49 INFO i.a.c.i.LineGobbler(voidCall):166 - ----- START CHECK -----
2024-11-01 14:39:49 INFO i.a.c.i.LineGobbler(voidCall):166 -

hi @kapa.ai, I need to know what airbyte's behavior is when the schema of a table changes. For example, adding a new field. How does it behave by default?

This message was deleted.

@kapa.ai On a fresh installation of airbyte 1.1.0, when trying to create a new postgres source. The following error pops up on the screen, and it’s not possible to create the source:

{
  "url": "<http://localhost:8888/workspaces/2f396b03-501a-42fa-9b98-6cb6e302b709/connections/new-connection?search=pos&sourceType=new&sourceDefinitionId=decd338e-5647-4c0b-adf4-da0e75f5a750>",
  "airbyteVersion": "1.1.0",
  "errorType": "TypeError",
  "errorConstructor": "TypeError",
  "error": {},
  "stacktrace": "TypeError: Cannot read properties of undefined (reading 'split')\n    at ne (<http://localhost:8888/assets/DocumentationPanel-m4r58wojbs.js:2:739>)\n    at cR (<http://localhost:8888/assets/core-icl96mliue.js:38:19551>)\n    at AB (<http://localhost:8888/assets/core-icl96mliue.js:40:44099>)\n    at TB (<http://localhost:8888/assets/core-icl96mliue.js:40:39820>)\n    at ate (<http://localhost:8888/assets/core-icl96mliue.js:40:39747>)\n    at pb (<http://localhost:8888/assets/core-icl96mliue.js:40:39597>)\n    at o3 (<http://localhost:8888/assets/core-icl96mliue.js:40:35959>)\n    at xB (<http://localhost:8888/assets/core-icl96mliue.js:40:34906>)\n    at R (<http://localhost:8888/assets/core-icl96mliue.js:25:1559>)\n    at MessagePort.N (<http://localhost:8888/assets/core-icl96mliue.js:25:1922>)",
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
  "featureFlags": {}
}

Is there a way to skip the initial load and only read cdc logs in sql server source connector?

@kapa.ai When a column is added or removed is there an approach to prevent airbyte from dropping the destination table? Or do we need to stop the use of views against airbyte tables.

@kapa.ai the build.log in airbyte-worker container is getting too big, how to handle?

Looking for an expert with extensive experience and in-depth knowledge of Airbyte. I need to enable real-time querying across multiple data sources using SQL. Could Airbyte be a viable solution for this?

Having trouble with no-code connector settings step and granularity. The api I contact accepts date ranges with a start_date and end_date, where the returned records have a create_date, where start_date <= create_date < end_date. So for each increment I need the cursor start_time to become the value of the previous increment end_time? What am I missing

I am facing the following error while testing the API using the connector builder UI:

Caused by: com.fasterxml.jackson.databind.JsonMappingException: String value length (20032516) exceeds the maximum allowed (20000000, from `StreamReadConstraints.getMaxStringLength()`) (through reference chain: io.airbyte.connectorbuilderserver.api.client.model.generated.StreamRead["slices"]->java.util.ArrayList[0]->io.airbyte.connectorbuilderserver.api.client.model.generated.StreamReadSlicesInner["pages"]->java.util.ArrayList[0]->io.airbyte.connectorbuilderserver.api.client.model.generated.StreamReadSlicesInnerPagesInner["response"]->io.airbyte.connectorbuilderserver.api.client.model.generated.HttpResponse["body"]

@kapa.ai How to perform transformation before loading to destination

@kapa.ai How do I resolve Could not find image: airbyte/source-s3:4.9.1

@kapa.ai How to delete the source using sourceId from airbyte self hosted version. I have username and password

@kapa.ai, if i have 10 connections how many resources should i have on airbyte deployed on kubernetes

Randomly started getting the below error on postgres to postgres connection:

2024-11-04 05:50:57 ERROR i.a.c.Application(run):80 - Killing orchestrator because of an Exception
java.lang.IllegalStateException: baseUrl is invalid.

Airbyte helm chart version: airbyte-0.94.1

@kapa.ai Error message ‘Cannot read properties of undefined (reading ‘toLowerCase’)’ encountered when creating a source mysql

@kapa.ai I am using MS SQL Connector to fetch data & dump it to S3 Bucket , all works fine but when i go to Bigger schema while fetching Schema Airbyte throws 504 Error I am using Airbyte open Source on Kuberntes , MS SQL connector Version 4.1.15 , Can you please guide me where to increase connection timeout parameter in helm file

getting the below error after upgrading to latest airbyte helm chart 1.1.1 io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: <https://<URL>/api/v1/namespaces/warehouse/pods?labelSelector=auto_id%3Dea69839d-df66-4ab4-9f54-a39d498096b5>. Message: pods is forbidden: User "systemserviceaccountwarehouse:airbyte-admin" cannot list resource "pods" in API group "" in the namespace "warehouse". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods is forbidden: User "systemserviceaccountwarehouse:airbyte-admin" cannot list resource "pods" in API group "" in the namespace "warehouse", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}). 11

@kapa.ai how to migrate a table from one connector to another and preserve data

I am trying to deploy airbyte onto aws eks. But we are coming across an error when deploying via helm, stating that the airbyte-db and airbyte-minio pod has unbound immediate PersistentVolumeClaims. How can we configure to get at least this part working for now?

@kapa.ai how to create a custom source connector by using api. please give me sample command.

Which replication modes does airbyte to support

how to upgrade version airbyte in localhost? @kapa.ai

why do I get the error: "i.m.s.r.u.Loggers$Slf4JLogger(warn):299 - [64b03b9c, L/127.0.0.146489 - Rlocalhost/127.0.0.18125] An exception has been observed post termination, use DEBUG level to see the full stack: java.net.PortUnreachableException: recvAddress(..) failed: Connection refused"?

@kapa.ai I'm trying to setup a new connection with PartnerStack, but I'm getting this error:

Configuration check failed
"Unable to connect to stream groups - stream_slices() missing 2 required positional arguments: 'sync_mode' and 'stream_state'"

On Airbyte 1.1.0 I have a job which loops forever, only 5 of the streams start. I can see they show as complete in the logs:

│ orchestrator 2024-11-04 12:54:04 replication-orchestrator > Updating status: brands - COMPLETE                                                                                                                                                                                 │

However other streams in the job stay queued and do not start. The log also shows some unusual negative numbers for the state manager:

Queue `zendesk_brands`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 9 MB, Used: -1110 bytes, percentage Used -1.0586905619167069E-4

What could be the issue here?

@kapa.ai can you please redirect me to where I can find resources to set a subpath like https://{url}.com/airbyte instead of using airbyte on the root path?

Hi Team, i am getting this error on my aks cluster "*Failed to pull image "airbyte/bootloader:0.50.35": failed to pull and unpack image "docker.io/airbyte/bootloader:0.50.35": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/airbyte/bootloader/manifests/sha256:1c8ebffaeab00e60fd139c1ee5d3ea2a180d64e58bb6201b3e2f7f5e2f478554: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit*"