hi all, what is the best way to retrieve the connection_id of airbyte? this is for the use case of using dagster to perform airbyte syncing operations

hey all - attempting to use airbyte to replicate data from s3 to postgres. I was able to successfully do this with the raw json but it fails on normalization. in the logs I found this: 2022-06-24 024406 [32mINFO[m i.a.w.DefaultNormalizationWorker(run):46 - Running normalization. 2022-06-24 024406 [32mINFO[m i.a.w.n.DefaultNormalizationRunner(runProcess):122 - Running with normalization version: airbyte/normalization:0.1.71 2022-06-24 024406 [32mINFO[m i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/normalization:0.1.71 exists... 2022-06-24 024406 [32mINFO[m i.a.c.i.LineGobbler(voidCall):82 - airbyte/normalization:0.1.71 not found locally. Attempting to pull the image... 2022-06-24 024417 [32mINFO[m i.a.c.i.LineGobbler(voidCall):82 - Image does not exist. 2022-06-24 024417 [32mINFO[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):158 - Completing future exceptionally... io.airbyte.workers.WorkerException: Normalization Failed. Should airbyte/normalization have been part of the install or is it something else that needs to be added seperately?

Hello there, cross-posting from #C038KM9NF7Z! I am using Airbyte (0.39.20-alpha) to do a CDC against a MySQL table to Kafka. I am actually only interested in the latest updates to the table, but using the MySQL CDC connector, it looks like it is starting to read the table updates from when the database was created, in May 2021. As I don't particularly care about going that far back in the past, I was wondering if there was a setting to use to just pull the updates as they are happening now. Thanks in advance!

Hi all, I understand that we can get information about a job from the API like this: https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#tag--jobs . But while the job is running, the call doesn't return the bytes downloaded or any other indication of progress. How can we get an indication of number of rows downloaded.. or bytes downloaded etc.

Hi All, just need some advice on connector development. I am working on a python source connector for customer io. I saw there is already a connector in typescript that is not working as I don't really get replies on there github issues pages, no longer maintained. As the low quality of the plugin( url links to beta url, no difference in US vs EU zones) I 've decided it was for me faster to restart from scratch. Is there any way once finished to share this with airbyte community?

I have deployed airbyte on my system, now I want to publish that using dedicated ip for others to use. But I dont want it to be open and want some authentication for that. What's the way to achieve that?

Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.

Hi All, I have been going through the documentation of airbyte. 1. Is it possible to use Facebook/Snapchat or any other advertising connectors as destination? It is only listed as source on the connectors page of airbyte. 2. Is there APIs to use the connectors?

Can octavia cli auto populate the yml from existing connections

Hi, I'm in a situation whereby replication between MySQL and Kafka has stopped. Is there a way to restart this? On the UI, it says that last sync was 2 hours ago (though set at every 5 minutes) but other than that there's no visible error. Below is what I think is a relevant log:

2022-06-26 06:19:40 source > 2022-06-26 06:19:39 INFO i.a.i.s.m.MySqlCdcStateHandler(saveState):41 - debezium state: {"mysql_db_history":"{\"source\":{\"server\":\"database_prod\"},\"position\":{\"file\":\"mysql-bin.001125\",\"pos\":
2022-06-26 06:19:40 source > 2022-06-26 06:19:39 WARN i.a.i.b.IntegrationRunner(watchForOrphanThreads):239 - The main thread is exiting while children non-daemon threads from a connector are still active.
2022-06-26 06:19:40 source > Ideally, this situation should not happen...
2022-06-26 06:19:40 source > Please check with maintainers if the connector or library code should safely clean up its threads before quitting instead.
2022-06-26 06:19:40 source > The main thread is: main (RUNNABLE)
2022-06-26 06:19:40 source >  Thread stacktrace: java.base/java.lang.Thread.getStackTrace(Thread.java:1610)

2022-06-26 06:04:31 source > 2022-06-26 06:04:30 INFO i.a.i.s.m.MySqlCdcTargetPosition(reachedTargetPosition):83 - Signalling close because record's binlog file : mysql-bin.001130 , position : 66010983 is after target file : mysql-bin.001130 , target position : 66010115
2022-06-26 06:04:31 source > 2022-06-26 06:04:30 INFO i.d.e.EmbeddedEngine(stop):996 - Stopping the embedded engine
2022-06-26 06:04:31 source > 2022-06-26 06:04:30 INFO i.d.e.EmbeddedEngine(stop):1004 - Waiting for PT5M for connector to stop
2022-06-26 06:06:00 source > Jun 26, 2022 6:06:00 AM com.github.shyiko.mysql.binlog.BinaryLogClient$5 run
2022-06-26 06:06:00 source > INFO: Trying to restore lost connection to 199.299.199.199:3306

Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.

Hi all When trying to run Standard Tests from airbyte root repository, I see an error that the

RepositoryHandler.jcenter()

method has been deprecated has anyone seen this error before?

Hello, we are running a POC to see If we can replace Fivetran and use Airbyte managed version. To start with we set up a VM with a few connectors to test the performance etc. One of the key databases is an MSSQL and we are facing some issues when setting up the connector. When clicking on set up we get these log:

2022-06-27 12:40:23 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30003ms.

We don’t have much flexibility to update that MSSQL box and the JVM vars. The same connector in Fivetran with the same configuration works correctly. Also sshing into the box works fine

ssh -i airbyte_nandos_rsa airbyte@xx.xx.xx.xx

Has anyone experienced similar problem? Any advice? Thanks. in advance

Hellllo Is there any way to make a connector faster? for example, if every sync I’m refreshing the last 7 days. can I spread it for 7 concurrent requests?

Hey guys! If I have DB table with account list that constantly updating let’s name it list_acc = [acc1,acc2..accn]. I need to pass this list to “Customer Id(s)” field of Google Ads connector - is there exists an elegant and airbyte way of doing this?

Hi. I have a question about scaling. I’m reading https://docs.airbyte.com/operator-guides/scaling-airbyte and see

As a reference point, the typical Airbyte user has 5 - 20 connectors and 10 - 100 connections configured. Almost all of these connections are scheduled, either hourly or daily, resulting in at most 100 concurrent jobs.

I would like to use the Postgres CDC source connector, but our source DB has over 550 tables to sync. Would Airbyte have any issues with this. We plan to deploy OSS Airbyte to Kubernetes.

Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.

Hello! is it possible to use a custom GAQL as the only source stream in a google ads connector? When I make a connection it syncs the selected google data type first and then the custom queries pointing to its destination table names

I installed Airbyte on a VPS using this tutorial - https://docs.airbyte.com/deploying-airbyte/on-digitalocean-droplet Anyone have a tutorial to enable secured access username/password to the insatnce deployed in this way?

Hi Guys, Preliminary informations : I am using airbyte (using helm) and installed it inside minikube, My goal is to transfer few files from s3 bucket to a destination database. Issue : While checking source connection it is giving me an error representing that my AWS_ACCESS_KEY is invalid , even though i have provided proper endpoint url , access key and secret key. Then I test it back and launch the application using

docker compose up

and with the exact same values for access key , secret key and endpoint for the same source it passes the connection check. Can someone guide me where I might go wrong when running application with docker v/s helm charts. Thanks in advance.

Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.

Hello, I posted on airbyte forum my issue https://discuss.airbyte.io/t/config-validation-error-todo-is-a-required-property/1643?u=berkanecyl if you guys have time, can you please help me out. Thanks in advance.

Is there a way to resync just a single table from scratch? I have Connection with multiple tables, and I'm using

Incremental | Deduped history

for most of them. However, one of the source tables was manually edited by a user, and we need to resync it from scratch. If I delete the destination table (in BigQuery), will Airbyte be smart enough to recreate the table and sync it from the start?

Hi, my name is Bart Baeyens and I’m new to AirByte. I’m trying to work through the onboarding “Sync MySQL CDC to Kafka using Change Data Capture”, using a local instance of MySQL. I can create the necessary tables using the provided script, fill them with the data but when I try to make a connector this always fails, whatever the replication method: standard or CDC. I’m using: mysql Ver 8.0.29 for macos11.6 on x86_64 (Homebrew) airbyte docker, version 0.39.28-alpha. Could not connect with provided configuration. Error: HikariPool-1 - Connection is not available, request timed out after 60003ms.

Hello, I am trying to build latest master with

./gradlew build

from project root but it fails with an error:

> Task :airbyte-webapp:test
/home/ryermilov/development/airbyte/airbyte-webapp/node_modules/react-dom/cjs/react-dom.development.js:3879
          var evt = document.createEvent('Event');
                             ^

TypeError: Cannot read properties of null (reading 'createEvent')
    at Object.apply (/home/ryermilov/development/airbyte/airbyte-webapp/node_modules/react-dom/cjs/react-dom.development.js:3905:26)
    at invokeGuardedCallback (/home/ryermilov/development/airbyte/airbyte-webapp/node_modules/react-dom/cjs/react-dom.development.js:4056:31)
    at eventHandler (/home/ryermilov/development/airbyte/airbyte-webapp/node_modules/react-dom/cjs/react-dom.development.js:23543:11)
    at Scheduler_runWithPriority (/home/ryermilov/development/airbyte/airbyte-webapp/node_modules/scheduler/cjs/scheduler.development.js:468:12)
    at runWithPriority$1 (/home/ryermilov/development/airbyte/airbyte-webapp/node_modules/react-dom/cjs/react-dom.development.js:11276:10)
    at flushPassiveEffects (/home/ryermilov/development/airbyte/airbyte-webapp/node_modules/react-dom/cjs/react-dom.development.js:23447:14)
    at flushWork (/home/ryermilov/development/airbyte/airbyte-webapp/node_modules/react-dom/cjs/react-dom-test-utils.development.js:992:10)
    at Immediate.<anonymous> (/home/ryermilov/development/airbyte/airbyte-webapp/node_modules/react-dom/cjs/react-dom-test-utils.development.js:1003:11)
    at processImmediate (node:internal/timers:466:21)
FAIL src/views/Connector/ServiceForm/ServiceForm.test.tsx

One more error:

> Task :airbyte-integrations:connectors:destination-kafka:airbyteDocker
... <package setup list>
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
hologram 0.0.13 requires jsonschema<3.2,>=3.0, but you have jsonschema 3.2.0 which is incompatible.

And one more:

> Task :airbyte-integrations:connectors:destination-kvdb:airbyteDocker
...
#5 [2/7] RUN apt-get update && apt-get install -y bash && rm -rf /var/lib/apt/lists/*
#5 sha256:f27f81c253422d48efe2f63b6fb16cc5b65d2b4137c5b1c9d233843ce7ffb069
#5 0.847 /bin/sh: apt-get: not found
#5 ERROR: executor failed running [/bin/sh -c apt-get update && apt-get install -y bash && rm -rf /var/lib/apt/lists/*]: exit code: 127
------
 > [2/7] RUN apt-get update && apt-get install -y bash && rm -rf /var/lib/apt/lists/*:
------
executor failed running [/bin/sh -c apt-get update && apt-get install -y bash && rm -rf /var/lib/apt/lists/*]: exit code: 127

> Task :airbyte-integrations:connectors:destination-kvdb:airbyteDocker FAILED

Am I doing something wrong or those errors are bugs?

hey all, i've added a custom connector to airbyte open source but it doesnt appear in the ui - does anyone know why?

Is this still the best approach to secure airbyte using a htpasswd via nginx? https://shadabshaukat.medium.com/deploy-and-secure-airbyte-with-nginx-reverse-proxy-basic-authentication-lets-encrypt-ssl-72bee223a4d9

or is there a way to build

airbyte/webapp

to allow for auth?

Hello! does the “start_date” parameter in the google ads source creation affect custom GAQL queries too? there are some data types (eg. AdGroup) that don’t have that column

Hello! Just wanted to know if there is an option for "SFTP" in destination like a sources type "SFTP". I see there's an option of SFTP JSON, but I'm looking for only SFTP.