Hello everyone. I wanted to ask if it possible to create an ETL connection between Microsoft MSSQL Data warehouse and Postgres DB using Airbyte. There's data in the MSSQL I need put in Postgres

Hi, I'm trying to create a new connector from Connector Builder UI locally. When I'm trying to test a stream, I get

"Could not perform read with with error: [ErrorWrapper(exc=DictError(), loc=('body',))]"

Any help with debugging this?

Hello. Anyone managed to get a google ads oauth token that lasts more than 7 days without having to go through an app verification process that requires privacy policy / youtube video / etc ?

Hi, our company's security admin has set up a sandbox Airbyte/Snowflake environment for me to test the tool. However, I have encountered a problem setting up a Snowflake destination as it requires only following three methods: Username/password, Oauth or Key pair authentication. Our company is using authenticator=externalbrowser parameter for security reason thus I don't have direct username/password access. Is there way to setup Snowflake as destination using that jdbc parameter? Thank you in advance!

Hi Folks, I trying to create a new Salesforce connection in Airbyte and it is failing. Is this your first time deploying Airbyte: No Memory / Disk: 8Gb / 16GB Deployment: EKS Airbyte Version: 0.40.29 Source name/version: Salesforce Destination name/version: S3 Step: Creating the connection Description: The connection creation fail with message: The connection tests failed. NetworkError when attempting to fetch resource.

Hi Team. Are source connectors streams should be made strictly per API only? When does it become applicable to manipulate and join two API outputs?

Hi Team, We have a MySQL(V1.0.21) to BigQuery(V1.2.13) connection for a one-time dump of a large table containing over 170 crore records(Airbyte Version 0.40.32). At the time of completion of 3 days, it automatically fails. so we changed the default timeout values of Airbyte to 7 days. even after, it fails after reaching 72 hours. Can someone help me to solve this?

Hi, I’m experiencing issues in long running sync. I have a shopify sync that is running for multiple days, but eventually fails with the following error:

2023-02-16 23:27:22 WARN i.a.w.s.LauncherWorker(killRunningPodsForConnection):253 - There are currently running pods for the connection: [orchestrator-repl-job-10235-attempt-1]. Killing these pods to enforce one execution at a time.
2023-02-16 23:27:22 INFO i.a.w.s.LauncherWorker(killRunningPodsForConnection):256 - Attempting to delete pods: [orchestrator-repl-job-10235-attempt-1]
2023-02-16 23:27:22 INFO i.a.w.s.LauncherWorker(killRunningPodsForConnection):261 - Waiting for deletion...
2023-02-16 23:27:23 INFO i.a.w.p.AsyncOrchestratorPodProcess(computeExitValue):129 - State Store missing status. Orchestrator pod orchestrator-repl-job-10235-attempt-1 non-existent. Assume failure.
2023-02-16 23:27:23 INFO i.a.w.s.LauncherWorker(lambda$run$3):227 - Destroying process due to cancellation.
2023-02-16 23:27:23 WARN i.a.w.p.AsyncOrchestratorPodProcess(destroy):186 - Wasn't able to delete pod orchestrator-repl-job-10235-attempt-1 from namespace default

this is a k8s env with 0.40.32 airbyte version, shopify source connector and snowflake dest. this causes the sync to never complete, because after a few days the pod is pausing again. Why isn’t the state persistency mechanism ensuring that the records yielded by the source are at least committed so that future syncs will continue from the point it failed?

Hi, I have a general/technical question for the BigQuery destination connector. In the case of a data ingestion in a BigQuery table, if the “schema” or “structure” of the data source has been modified and it is then added in the BigQuery table. Would there be a schema error with Airbyte? To be more precise, 1. if my source data has new columns, will my Bigquery table have its schema updated? 2. If my source data has less columns, will my BigQuery table have a schema that is also update? Thanks for your time !

Hi folks I had a query on the S3 destination connector. I have created am IAM profile with the necessary permissions. How do I pass it to the s3-destination pod that gets created by the worker? Thanks ~ashwin

Hi, I am new to Airbyte, when i run this to start Airbyte, it show errors. Can anyone help me with that?

D:\>cd airbyte

D:\airbyte>docker compose up
no configuration file provided: not found

Hello folks, I’m currently using Airbyte open-source to sync some data I am trying to sync data at lower intervals ( < 1hour ) How should I go about this, high level steps would work as well ( just point me to where to look ) Thank you 🎉

Hi Team, I am creating the source as MongoDB and destination is Local Json. I am passing my database name in the source, but due to the huge volume of collections in this database, it keeps scanning the schema of all the collections and takes lot of time to create the connection. Is there any way i can pass the collection name as well in the Mongo DB Source along with database name?

Hey Team, a new-bee here. I’m trying to deloy airbyte in GCP and while following the official documentation, I get following error. It seems like there is no files in github. Any tips would be really helpful. Thanks.

Hi Team, I am very new to airbyte and I am trying to create a connector between s3 bucket and postgresql on AWS RDS, not able to understand why my connector getting failed with error, also that table _(temp_customers_segment_scd)_ does not exists in the database. Also why it is needed to give alter permission to these tables manually, why connector is not able to alter these tables that it is creating ?

cannot delete from table "temp_customers_segment_scd" because it does not have a replica identity and publishes deletes
2023-02-20 09:36:26 normalization >   HINT:  To enable deleting from the table, set REPLICA IDENTITY using ALTER TABLE.

Hi, Are there docs or a recommended way to rename a HTTP python connector generated with the airbyte code generator script? Apart from find and replace

Hi there. I got an error during initial sync:

Stack Trace: java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Buffer length is less than expected payload length.

any thoughts on what I can do about it?

API unresponsive

Hello there, I deployed the helm chart (0.43.29) on a EKS cluster. I have a problem where the API don't respond even for very simple calls

POST /api/v1/sources/list HTTP/2.0" 499 0 "<https://airbyte-production.mydomain.com/workspaces/fdf5cd09-c49c-4cd3-ac91-82a6eddeceba/source>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" 101 31.188 [core-production-airbyte-airbyte-webapp-svc-80] [] 10.0.2.139:80 0 31.188 - 5a7c2de61eaa07ad6ffec93a339b287e

Did someone already have the case of error 499 and have successfully solved it?

Hello there, I'm trying to deploy airbyte to digital ocean droplets but when trying to hit the following command "wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,flags.yml,docker-compose.yaml}" give me not found !!. anyone here can help in that, please?

Hello airbyte ! I'm trying to ingest all our Mailchimp data (campaigns, email_activity and lists) using API key to S3. Their API is pretty slow and connection aborted each 10 minutes. It take me about 1h to ingest 30MB and 30000 rows. Then connection restart and the process continue but always like a turtle 🐢 😄 Example

2023-02-20 12:30:59 INFO i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):321 - Records read: 52000 (53 MB)
2023-02-20 12:31:20 source > Backing off _send(...) for 5.0s (requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')))
2023-02-20 12:31:20 source > Caught retryable error '('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))' after 1 tries. Waiting 5 seconds then retrying...
2023-02-20 12:34:10 INFO i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):321 - Records read: 53000 (54 MB)

Did anyone struggle using their API ?

Hi All.........i pulled the latest repository of airbyte from GitHub.........I tried to ran the server this was the following error Previously this files were used to be the present , i think they were omitted in the latest version.......Can someone please help........ i think mine is the same problem as @Wajdi M

I hope that someone could help me figuring out what goes wrong with my instagram connection. I have one app that I use to connect to Facebook ads and instagram. I have added a whole list of advertising accounts in that app it, some of which are also linked to an instagram account. The strange thing is that at least one instagram account is missing in the data stream! The corresponding facebook connector has no issue. The instagram account is connected to the ad account and the ad account id has been added to the facebook ad. If someone has any good idea where to check, that would be great, since I can’t see the reason for this difference in behaviour

I'm trying to run Airbyte on EKS with an external RDS Database. It looks like everything is working, but the server component restarts quite often due to timeouts in the Liveness/Readiness probe. I've increased this value from 1s to 10s but it still times out and still shuts down the server. I'm don't understand why

Hello 👋 Is there a way to set SYNC_JOB_MAX_ATTEMPTS env var when deploying airbyte in EKS with helm chart?

👋 Hello! What triggers a

Non-breaking schema updates detected

in the UI ? When clicking on

Review changes

I get a

No changes were detected in the source schema.

Is this a bug?

Hi, I am ingesting data from a database through the MySQL connector using the "Logical Replication (CDC)" replication method and the data being ingested is several days out of date. For example, it has been executed today(02/20) and there is data up to (02/15). The status of the synchronizations is "Sync Succeeded". • Airbyte version = 0.40.29 • MySQL source version = 1.0.21 • BigQuery destination version = 1.2.14 Can you help me? Thanks!

Hey folks 👋 Regarding octavia cli — has anyone managed to get around self signed certificates on their airbyte instance? i.e. is there a config option somewhere that instructs the api client to ignore unsafe ssl My dev instance runs with self signed certs and so octavia refuses to interact with it. Long term I’ll go about getting the octavia image to trust my certs but in the meantime would be nice if I didn’t have to do that.

Hey guys, one question, what is the procedure to backup and restore all the configuration in case airbyte crash, deleted, or tshoot ?

Airbyte-Platform Architecture

Hello there 👋, I'm using Airbyte deployed on EKS via the helm chart. I have several questions regarding the platform architecture • First, I find it very weird that the REST API is handled by the WebApp component. I think the REST API should be the main access point to communicate with the server and should be completed decoupled from the frontend. We should be able to run Airbyte without a frontend. • Second, I suspect a strongly coupled architecture between jobs and the server. I run a multi-tenant architecture where I have one connection per tenant. I clearly while they are syncs in progress, the API is very unresponsive and I have a lot of timeouts. I think the REST API should be completely decoupled from whatever happens in the background and highly available. Do someone know if there is a roadmap / slack channel regarding Airbyte architecture? (and not about connectors themselves)

Hi, I'm trying to find the files at https://github.com/airbytehq/airbyte.git kube/overlays/stable... as per the docs in the screenshot. I see they were removed in this commit: https://github.com/airbytehq/airbyte/commit/2dc839acb8428e53930fce9a09b40f9a16f03d74 Is kubernetes(via kustomize) still an option. Whats the story with this change?