An API returns data [{ id1,attributes{...}} I,...] would like to collect both the id and each of the attributes, can i do this in the low code cdk selector, currently it is pointing toward the attributes and working. Or do I need to write the full connector?

One of my interests in Airbyte, was to consume data from Firebase analytics, and I was quite certain I saw a connector I could use as a source, but I can’t find it anymore in the OSS catalog, and the website also points to a 404 -> https://docs.airbyte.com/integrations/sources/Firebase-Realtime-Database

Is this connector removed?

Hi, happy day, do you know how I can solve this error?

Hi guys, Our airbyte version deployed on EKS using helm charts is 0.43.29 and :- 1. We are trying to copy a csv file from S3 source to an avro file on S3 destination using airbyte and it looks like the sync operation is taking forever to complete. 2. However when we stop the sync and re-run it then it completes quickly. 3. This is what we get when the sync is complete on airbyte

2023-02-28 13:54:05 destination > INFO i.a.i.b.a.AdaptiveDestinationRunner$Runner(run):88 Completed destination: io.airbyte.integrations.destination.s3.S3Destination

4. We see the same thing on the current sync as well but it has not terminated yet 5. The file that is being copied it 1.3KB in size so it should not take this long 6. Also we are using the full refresh| overwrite sync mode with replication frequency set to manual and Non-breaking schema updates detected set to ignore 7. Another interesting thing to note is that we are copying the same csv file again and again but with a different name and it seems to be increasing the number of synced or emitted or committed records in the log . Could this be because the csv file is missing headers or because we haven’t specified any schema for airbyte source S3 / destination s3?

Is anyone experiencing k8s auth issues on airbyte 0.41.0? I'm using an older helm chart (0.43.22), but I don't see any recent changes to the service account permissions, so I can't imagine that's causing this problem.

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: <https://10.100.0.1/api/v1/namespaces/airbyte/pods>. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:682) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:661) ~[kubernetes-client-5.12.2.jar:?]

Has anyone loaded txt files from SFTP? I have a txt file that is separated by tabs. I wonder if it will be an issue since Airbyte only supports csv and json. I didn't see an option for a delimiter in the configuration. (edited)

Does Airbyte Cloud provide API Access?

Hello ! I am trying use a airbyte in local deploy and trying connect a Zendesk Support, but i receive a error from SSL. My setup is: Ubuntu 20.04 in WSL Docker This scenario is in the corporation, have a firewall, etc ...

Hey guys, I am brand new to the workspace and I am looking to use Airbyte at a startup. My question is: should I be too concerned if a connected is in

alpha

? I am a little surprised the DynamoDb connector isn't more tested, and I assume it will be in the future, but just wanted to see how much caution I need when deploying one of these connectors in production

Hi, I'm trying to load data from api, but sometime the stream pull data to the same avro file (

xxx/data/dump_2/2023/02/27/15/72247da1-e4bc-4cbf-8b65-dc112b34c8ee/2.avro

) at the same sync. As the result part of data drops. How can i fix it?

airbyte version 0.40.18

log:

2023-02-27 16:22:52 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):383 - Records read: 2680000 (284 MB)
2023-02-27 16:22:52 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Initiated multipart upload to xxx/data/dump_2/2023/02/27/15/72247da1-e4bc-4cbf-8b65-dc112b34c8ee/2.avro with full ID ABPnzm4Wgmepg2bXrbpotHV4Dyl0vznhMwUX2Z40BoayH0Zzwu8k2MQnRzEzkgD1W6XIh0E
2023-02-27 16:22:53 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):383 - Records read: 2681000 (285 MB)
2023-02-27 16:22:53 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Called close() on [MultipartOutputStream for parts 1 - 10000]
2023-02-27 16:22:54 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - [Manager uploading to xxx/data/dump_2/2023/02/27/15/72247da1-e4bc-4cbf-8b65-dc112b34c8ee/2.avro with id ABPnzm4Wg...D1W6XIh0E]: Finished uploading [Part number 1 containing 10.01 MB]

...............

2023-02-27 23:46:12 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):383 - Records read: 41597000 (4 GB)
2023-02-27 23:46:13 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Initiated multipart upload to xxx/data/dump_2/2023/02/27/15/72247da1-e4bc-4cbf-8b65-dc112b34c8ee/2.avro with full ID ABPnzm46bz3VtfUl2WWDwmkyFDgPGbR3hNS3p6ocoNppt3-L9G7NcvEF3Lr85psBci9cncc
2023-02-27 23:46:13 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):383 - Records read: 41598000 (4 GB)
2023-02-27 23:46:13 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Called close() on [MultipartOutputStream for parts 1 - 10000]
2023-02-27 23:46:15 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - [Manager uploading to xxx/data/dump_2/2023/02/27/15/72247da1-e4bc-4cbf-8b65-dc112b34c8ee/2.avro with id ABPnzm46b...sBci9cncc]: Finished uploading [Part number 1 containing 10.01 MB]
2023-02-27 23:46:15 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - [Manager uploading to xxx/data/dump_2/2023/02/27/15/72247da1-e4bc-4cbf-8b65-dc112b34c8ee/2.avro with id ABPnzm46b...sBci9cncc]: Finished uploading [Part number 3 containing 10.01 MB]

Hi folks, we’re attempting to set up a netsuite connector and are running into the following error:

HTTPError('400 Client Error: Bad Request for url: <https://7074563.suitetalk.api.netsuite.com/services/rest/record/v1/contact?limit=1>')

this error was also posted in this thread, but i didn’t see a resolution. the netsuite logs don’t point towards a networking issue as we get back some successful login messages when we test the connector. we’ve tested this on other object types and have received the same error message. how can we solve this issue?

A quick announcement about Kustomize versus Helm and why we’ve sunsetted Kustomize as part of Airbyte: In the process of planning our development cycle, we realized that we couldn’t support both helm and kustomize at the same time and provide the quality of experience that we wanted to for our users. We chose Helm over Kustomize because most of our users use it and it has much more community mindshare. We had planned to give a longer sunset period for Kustomize, but at the same time we were planning and executing our platform code migration, and we realized that to move kustomize as part of this migration was going to be a big lift for something we were going to sunset no matter what. So we settled for this middle ground where we sunset it with very little warning but made it possible for existing users to keep using it. This particular decision was driven by a confluence of events that we don’t anticipate happening again, and we will endeavour to give more warning and a longer sunset process in the future. Thank you for your time and attention.

Marvel api connector. So I'm trying to create a new connector using the marvel api as a study case (https://developer.marvel.com/docs#!/public/getComicsCollection_get_6) However I'm getting this error when I try to read the data.

{"type": "LOG", "log": {"level": "FATAL", "message": "1 validation error for ConfiguredAirbyteCatalog\nstreams -> 0 -> stream -> supported_sync_modes\n  field required (type=value_error.missing)\nTraceback (most recent call last):\n

Hi, I’m looking to integrate Airbyte into our EKS cluster and usually we do so for other applications / services with helm charts. I see on the documents that there is a structured way to download and customize the helm charts + templates needed however the hyperlink to those helm charts is broken. I was wondering if there’s a mirror for the value files + templates or if there’s another workaround we could use to keep a customizable version of the helm charts in our own deployment repo? Thanks!

I have installed Airbyte in K8s using helm chart, how can I set a password using the values.yaml file? or maybe even better use an environment variable to do it?

Hi everyone! It looks like the new helm charts, taking into consideration the switch to the airbyte-platform repo, are not being pushed. We are not able to deploy Airbyte through Helm at all currently. This comment seems to have identified the problem already. Should I open a PR to fix it or is it something the team is already looking at?

Any recommendations on extending a current destination/source connector so that it can receive updates to the base connector but custom changes can remain separated? I figured theres options like symlinks, git sub modules, import paths, but didnt know if there was a best approach. For example, if I wanted to modify the source-file connector to have additional features, but still receive updates from the main branch, whats the best way to handle that.

Hello, I have a question if I point two different airbyte connections to the same destination in S3 (sync mode is Incremental|Append) would airbyte sync still work properly without any data being accidentally deleted in S3? Thank you!

I am using Airbyte v0.41.0 and the WooCommerce connector (v0.2.0). I am running into the following error:

DateTimeValidator(tryParse):82 - Invalid date-time: No zone offset information found

Attached you can find the full logs. Grateful vor any support!

I'm reading the docs for the DB2 connector and it does not mention an SSH Tunnel option. Does that mean I can only connect to DB2 if it is accessible to the public internet? https://docs.airbyte.com/integrations/sources/db2/

(cross-posted to the Slack forums) We upgraded to Airbyte OSS v0.41 today in the hopes of getting past an issue with the Iterable connector. But now I’m wondering if the issue titled "Incremental dedup sync hangs until timeout after sync is complete" is still around in v0.41, and if so should we go back to v.0.40.30. What was working fine under v0.40.15 for Google Analytics is now hanging with the dedup sync problem (

create or update state error

starting at line 634 of the attached)

Dear team: Can Airbyte connect with dolphinscheduler？ I had check source and destination. I can’t find anything about it. Thanks for your answer！

Hi Airbyte team, I am wondering if duped + history work properly in raw json format, as we have found duplicate record in our db, details ⬇️

Need help with this stack overflow question: https://stackoverflow.com/questions/75600159/airbyte-role-role-specified-in-the-connect-string-does-not-exist-or-not-autho

Hi evereyone, i got below error after migration Airbyte(Helm) on k8s to another namespace. We have a task sync GA4 to S3&Redshift. config:

airbyte version = 0.40.25
helm chart version = 0.43.3
K8s cluster = 1.24
airbyte/source-google-analytics-data-api=0.0.3
airbyte/destination-s3=0.3.17

Failure during reporting of activity result to the server. ActivityId = e3ac2ed6-65b9-32f4-81cf-c8a394012c8f, ActivityType = Persist, WorkflowId=sync_58, WorkflowType=SyncWorkflow, RunId=722e9263-fad1-4c80-9b69-0d5a8dfbf7d1 io.grpc.StatusRuntimeException: NOT_FOUND: workflow execution already completed

2023-03-01 06:24:43 destination > Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-03-01 06:24:57 destination > Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-03-01 06:35:35 WARN i.t.i.w.ActivityWorker$TaskHandlerImpl(logExceptionDuringResultReporting):365 - Failure during reporting of activity result to the server. ActivityId = e3ac2ed6-65b9-32f4-81cf-c8a394012c8f, ActivityType = Persist, WorkflowId=sync_58, WorkflowType=SyncWorkflow, RunId=722e9263-fad1-4c80-9b69-0d5a8dfbf7d1
io.grpc.StatusRuntimeException: NOT_FOUND: workflow execution already completed
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.50.2.jar:1.50.2]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.50.2.jar:1.50.2]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.50.2.jar:1.50.2]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondActivityTaskCompleted(WorkflowServiceGrpc.java:3840) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.lambda$sendReply$0(ActivityWorker.java:303) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.lambda$retry$0(GrpcRetryer.java:52) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:67) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retry(GrpcRetryer.java:50) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.sendReply(ActivityWorker.java:298) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:252) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

All source and destination connections tested OK, new connections task works fine but all the existing task are failed to sync.

Hi everyone, need help with airbyte, I have setup a connection from redshift to snowlake. But my syncs are having 0 records :- Please help

logs link :- https://cloud.airbyte.com/workspaces/49a87173-c8ac-43fa-a3a8-2ea62bd9c627/connections/bd19e725-e004-4fb8-b575-be3ece45c671/status#1418722::0

hey folks, im thinking about using airbyte for mysql binlog, does somebody know if its possible to deploy airbyte open source on aws and schedule by airflow

Hi Everyone, i have deployed airbyte using Docker-compose, but unable to connect to my mysql-server, can anybody help on this.