can i export all the connection/sources/destinations?

Is anyone using the S3 source to load parquet files from AWS? Been trying for a while setup the access to NY Taxis dataset at ‘arnawss3:::nyc-tlc’ and it doesn’t work. I have created the IAM policy with the ListBucket and GetObject permissions for the above resource. When I perform the connection testing I always get errors. If on the bucket I set the ARN then it complains the format is not correct, if I just add the bucket name it mentions it cannot list files… I am lost trying to understand what is exactly failing. Any assistance is appreciated :)

I have a salesforce to snowflake sync running and everything seems to be working except I've noticed that my user roles table is missing a couple rows. I don't think it's a permission issue because about 90% of the data is syncing over. I changed it to a full refresh overwrite and still no luck. Has anyone else had this? I'm running on helm right now

Hello, One of our sync jobs suddenly started failing with a not so clear error message. It simply says

Source process read attempt failed

. I have refreshed the connection multiple times, but it always fails on the next sync. Can someone help me figure out what the problem is? Attached you’ll finde the attempt logs. Thank you Airbyte version: v0.63.13 Source: Postgres - v3.6.22 Destination: Postrges - v2.4.0 (edited)

Does the Airbyte API allow for creation of streams? Looked at https://github.com/airbytehq/airbyte-api-python-sdk/blob/main/docs/sdks/streams/README.md but can only see get_stream_properties

Hi Team, Is there a way to add certain debezium configurations to the Sql Server connector in Airbyte? We would like to experiment with configs like

incremental.snapshot.chunk.size

to see if it improves the performance for inital snapshots for us which is taking much longer then expected

Hi Team, I'm having issues with pyairbyte in airflow. Trying to run the source-google-sheets but keep getting the connection state manager (same as this PR: https://github.com/airbytehq/airbyte/issues/46283). Due to the circumstances I can't run a docker container in my airflow task because airflow is running form a docker compose in EC2. When trying to run the following script I get the ConnectorStateManager issue mentioned in the above PR. I have tried to force an older version which can work locally on my test python files but in production airflow does not as it appears unable to downgrade the airbyte-cdk version in the venv that it is running. I have also tried forcing the pip version field but it seems it does nothing to downgrade the connector (error also included below with mismatch on target)

Traceback (most recent call last):
  File "/Users/teddykosciuszek/Documents/CodeRepos/data-infra-airflow/dags/airbyte/test.py", line 3, in <module>
    sheet = get_source(
            ^^^^^^^^^^^
  File "/opt/anaconda3/envs/airbyte_py311/lib/python3.11/site-packages/airbyte/sources/util.py", line 107, in get_source
    executor=get_connector_executor(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/airbyte_py311/lib/python3.11/site-packages/airbyte/_executors/util.py", line 261, in get_connector_executor
    executor.ensure_installation()
  File "/opt/anaconda3/envs/airbyte_py311/lib/python3.11/site-packages/airbyte/_executors/python.py", line 275, in ensure_installation
    raise exc.AirbyteConnectorInstallationError(
airbyte.exceptions.AirbyteConnectorInstallationError: Connector's reported version does not match the target version. (AirbyteConnectorInstallationError)
------------------------------------------------------------
AirbyteConnectorInstallationError: Connector's reported version does not match the target version.
    Connector Name: 'source-google-sheets'
    Venv Name: '.venv-source-google-sheets'
    Target Version: '0.7.0'
    Original Installed Version: None
    Version After Reinstall: '0.7.4'

from airbyte import get_source

sheet = get_source(
    "source-google-sheets",
    config={
        "spreadsheet_id": "",
        "credentials": {
            "auth_type": "Service",
            "service_account_info": {{ var.google_sheets_gcp_sa_key}}
        },
        
    },
    version= "0.7.0",
    # docker_image= "airbyte/source-google-sheets:0.7.0",
    install_if_missing=True,
    # pip_url= "git+<https://github.com/pixisai/airbyte.git@SOURCE_GOOGLE_SHEET_CDK_VERSION>",
    streams= "Sheet1",
)

read_result = sheet.read()

for record in read_result["Sheet1"]:
    print(record)

Hello, I have upgraded to Airbyte v1 using

abctl

but now, the sync wont start. Looking at the kubeters pods there seems to be an error, though it not clear what the issue might be.

ubuntu@ip-10-0-60-1:~$ kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig -n airbyte-abctl get pods
NAME                                                     READY   STATUS             RESTARTS       AGE
airbyte-abctl-airbyte-bootloader                         0/1     Completed          0              19h
airbyte-abctl-connector-builder-server-f7b8c4d7b-6w7qh   1/1     Running            0              19h
airbyte-abctl-cron-7f6dc9484-2d98n                       1/1     Running            1 (18h ago)    19h
airbyte-abctl-pod-sweeper-pod-sweeper-856cf644f9-zf2tl   1/1     Running            0              19h
airbyte-abctl-server-8477ffd8d6-97xrx                    1/1     Running            12 (18h ago)   19h
airbyte-abctl-temporal-857dd9f6f4-xkwtq                  1/1     Running            0              19h
airbyte-abctl-webapp-cccb469b8-b95rb                     1/1     Running            0              19h
airbyte-abctl-worker-54df9686cc-qnvzq                    1/1     Running            0              19h
airbyte-abctl-workload-api-server-6867bffd8b-b2mvf       1/1     Running            0              19h
airbyte-abctl-workload-launcher-5f6bc4cd55-nnpl4         1/1     Running            0              19h
airbyte-minio-0                                          1/1     Running            0              19h
source-postgres-check-22394-1-ikhqn                      0/2     Error              0              120m
source-postgres-check-22394-2-oyrpg                      0/2     Error              0              110m
source-postgres-check-22394-3-mpzgz                      0/2     Error              0              99m
source-postgres-check-22394-4-dmcio                      0/2     Error              0              86m
source-postgres-check-22395-0-wvktf                      1/2     ImagePullBackOff   0              85s

It there a way to dig deeper into the issue? As for the sync job logs, they stay in a loop like this:

2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: CLAIM — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:49 INFO i.a.c.i.LineGobbler(voidCall):166 - 
2024-11-01 14:39:38 INFO i.a.w.l.c.WorkloadApiClient(claim):75 - Claimed: true for decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check via API for local
2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: CHECK_STATUS — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:38 INFO i.a.w.l.p.s.CheckStatusStage(applyStage):59 - No pod found running for workload decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check
2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: BUILD — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: MUTEX — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:38 INFO i.a.w.l.p.s.EnforceMutexStage(applyStage):50 - No mutex key specified for workload: decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check. Continuing...
2024-11-01 14:39:38 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: LAUNCH — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_22395_0_check) — (dataplaneId = local)
2024-11-01 14:39:49 INFO i.a.c.i.LineGobbler(voidCall):166 - ----- START CHECK -----
2024-11-01 14:39:49 INFO i.a.c.i.LineGobbler(voidCall):166 -

Hi Community. I have been tasked to build an application that can run on Kubernetes and be as scalable as possible. Basically the challenge is that the application should receive requests to make certain processing tasks and these tasks are basically a group of queries that run against a database. Now there are two things, these queries should run in parallel as much as possible and different users might trigger many processes at same time. Considering the bottleneck is not the database… What is the best design to ensure that the application scales in Kubernetes by managing new pods etc? If I am running a parallel loop lets say in python, can Kubernetes break it down into multiple pods? Thank you for the help while I try to learn these new concepts to app design.

@airbyte-cloud-source-slack Hey I am trying to refresh my schema and am receiving this error.

Server temporarily unavailable (http.502.re3bRSk3RYzJdNXr9Kztvo)

Hello team, I am trying to configure airbyte using EC2 instance on aws , I want it to be https using ALB and ACM , it seems I am missing something but not sure what is it. @Bryce Groff, I saw your video on deploying airbyte on EC2 and you mentioned at the end of your video that you will teach how to do it, is there any link to that video or any instructions on the documentation, because I am not able to find something that specifies how to do it . Thanks 🙂

Is anyone aware of whether the documented behaviour cursor step and granularity as shown in the example https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#split-up-interval here is correct? I need the same type of new start_time is the old end_time?

👋 Hello, team!,I'm doing some automation stuff where i need Airbyte,i facing issue .

In locally setup Airbyte ,oauth client crendentials working when credentials passed as json but not working when passed as urlencoded content type. But in airbyte cloud one it is working with both urlencoded and json one when credentials are send in one of the content-type

In Airbyte cloud both content type is working

But in local setup it is giving error when using urlencoded one as content type

message has been deleted

Has anyone successfully used Bing Ads connector in Airbyte Cloud successfully? Getting errors related to redirect_url..

#48125 Deeply nested JSON data issues New discussion created by paulswanson22 Hello, I'm trying to parse out a JSON response from a connector for my school, but I'm getting some strange responses. The data that comes back is for academic years and terms, and looks like: "body": { "academic_years": { "diploma": { "academic_years": [ { "id": 5309, "name": "January – June 2011", "starts_on": "2011-01-22", "ends_on": "2011-06-03", "updated_at": "2015-09-22T033835.000-04:00", "academic_terms": [ { "id": 1592, "name": "Quarter 3", "starts_on": "2011-01-22", "ends_on": "2011-04-01", "updated_at": "2015-09-22T033835.000-04:00", "exam_grade": false }, { "id": 1593, "name": "Quarter 4", "starts_on": "2011-04-20", "ends_on": "2011-06-03", "updated_at": "2015-09-22T033835.000-04:00", "exam_grade": false } ] }, etc The problem is that the term "academic_years" appears twice in the JSON and it seems to be giving Airbyte some difficulties. When I use a Field Path of "academic_years", "diploma" it also strips out the second "academic_years", and gives a response of: [ { "id": 5309, "name": "January – June 2011", "starts_on": "2011-01-22", "ends_on": "2011-06-03", "updated_at": "2015-09-22T033835.000-04:00", "academic_terms": [ { "id": 1592, "name": "Quarter 3", "starts_on": "2011-01-22", "ends_on": "2011-04-01", "updated_at": "2015-09-22T033835.000-04:00", "exam_grade": false }, { "id": 1593, "name": "Quarter 4", "starts_on": "2011-04-20", "ends_on": "2011-06-03", "updated_at": "2015-09-22T033835.000-04:00", "exam_grade": false } ] }, When I try to further specify within this, I get no records. If I try a field path of "academic_years", "diploma", "academic_years" I get a Python error. Has anyone encountered this issue before? airbytehq/airbyte

Hi everyone! Airbyte OSS does not have direct monitoring over prometheus and grafana? It needs OpenTelemetry? Any good guidance on how to set ip up when using helm charts?

Please post your weekly update in thread🧵. Thanks, team!

Hello everyone! We use Airbyte to extract data from multiple sources for our clients, with over 5,000 connections running daily on an Airbyte cluster deployed on Kubernetes. Our sources include • Amazon Ads • Google Ads • Facebook Ads • Shopify • WooCommerce • Amazon SP. Currently, we're on version : 0.39.42-alpha. Could anyone help with the following questions? 1. What is the latest stable Airbyte version recommended for migration? 2. Is there any standard documentation to guide the migration process? 3. What are common challenges during migration, and is there documentation on resolutions for these? Thanks!

Hello everyone, I'm using airbyte to push data from postgres to big query Initially I enebaled WAL(postgres) to track deletion of the records but the amount of logs generated was so huge I was not able to bare the server size expansion, hence I have decided to go ahead based on cursor that is based on timestamp and serially generated id. The issue I'm facing right now is on syncing big table, I have table of size 500gb which needs to be synced to big query I'm getting following error Stack Trace: java.lang.RuntimeException: org.postgresql.util.PSQLException: Result set exceeded maxResultBuffer limit. Received: 161061290; Current limit: 161061273 Has anyone faced this error? Any advice on how to get it solved?

#48132 Marketplace SFTP Connector New discussion created by allamsudha Hi, I have recently started using Airbyte and was elated to see that Salesforce connector could be made to work in a short time. However, I got stuck with SFTP market place connector due to the reason that it does not have PGP decryption along with SSH password/ley authorization. Could you please let me know if this could be taken up for enhancement? If somebody has customized this connector, you may please point me to the procedure and I will try doing the same. airbytehq/airbyte

Hey everyone! can someone please redirect me to where I can find resources to set a subpath like https://{url}.com/airbyte instead of using airbyte on the root path?

Hi Team, Please help me on this issue deployed on AKS by helm "*Failed to pull image "airbyte/bootloader:0.50.35": failed to pull and unpack image "docker.io/airbyte/bootloader:0.50.35": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/airbyte/bootloader/manifests/sha256:1c8ebffaeab00e60fd139c1ee5d3ea2a180d64e58bb6201b3e2f7f5e2f478554: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit*"

Hi everyoneoctavia helloo, recently I'm trying to deploy Airbyte on EKS. When doing

helm install airbyte airbyte/airbyte --namespace airbyte --values ./values.yaml

I'm getting the following error.

coalesce.go:286: warning: cannot overwrite table with non table for airbyte.keycloak-setup.initContainers.keycloakReadinessCheck.image (map[])
coalesce.go:286: warning: cannot overwrite table with non table for airbyte.keycloak.initContainers.initDb.image (map[])
coalesce.go:286: warning: cannot overwrite table with non table for airbyte.keycloak-setup.initContainers.keycloakReadinessCheck.image (map[])
coalesce.go:286: warning: cannot overwrite table with non table for airbyte.keycloak.initContainers.initDb.image (map[])
Error: INSTALLATION FAILED: failed pre-install: 1 error occurred:
	* timed out waiting for the condition

During the process

airbyte-airbyte-bootloader

starts but after the error it get stopped. I double checked the yaml and seems ok. And I didn't change any properties relating to keycloak or keycloak-setup. Also I'm using helm version "v3.16.2". Any idea what is causing this? Thanks.

After various back and forth in this Slack channel, following various GitHub discussions and making changes to various parameters, at mssql connector level, Airbyte platform level and postgres DB level, we are still facing source > Terminating due to java.lang.OutOfMemoryError: Java heap space error in one of our connections. Anyone able to help us out here as failure of this ingest pipeline is affecting critical data delivery for end-user reports?

This commit fixes broken tolerations on job pods, when can we expect this to be in a release? https://github.com/airbytehq/airbyte-platform/commit/2ca3c4192793b15a1ccc2bfd644dd725c3a2903c#diff-3555dc77946bb010495d[…]1e0f5b54b3c8a37394c3b0R227

Hi , I am getting HTTP 504 error while selecting Streams / Discovering Schemas in Airbyte , Its able to connect & fetch the data from same source, while creating new connection it throws error 1. Airbyte Deployed on Kubernetes , Airbyte Version :- 1.1.0 , Helm Chart Version :- 1.1.1 2. MS SQL Connector Version :- 4.1.15 I have tried this parameters in https://docs.airbyte.com/enterprise-setup/scaling-airbyte#schema-discovery-timeouts server: extraEnvs: - name: HTTP_IDLE_TIMEOUT value: 10m - name: READ_TIMEOUT value: 10m Still I got HTTP 504 Error exactly around 60 Sec , In Same Database different schema which have lesser number of Tables its working Fine, In this schema I have around 900 Tables Can anyone please help me out as its pretty urgent , I am at Production stage in Dev it was working fine as it has lesser number of Tables