After deploying Airbyte on a cloud droplet and docker-compose up, how can I open it? Should the server have a GUI? Because I couldn't open it from server terminal. I was able to open http://serverIP:8000 from my local computer browser but it is not secure I think.

Will logical replication mode of postgres will increase the size of database and load on cpu?

Is it bad to have my connections IDs in a public repo?

Hello, is there any way to use “powered by airbyte” but for reverse ETL? I’d like to create integrations with tools but to push information rather than to pull

Hi there, I was wondering is there anyway to not replicate some columns while using CDC? The recommended workaround is to use a view but that won’t work with CDC. We want to use CDC in order to capture deletes. We’ve also been looking at PipelineWise and there you’re able to transform the data in order to anonymize it. Is there anything similar in Airbyte?

I just pulled from the github to my local and after running docker-compose up. This is the error that I get. All containers are running with the exception of the server which keeps restarting

Hey I have bunch of questions regarding Airbyte: 1. Is there a convenient way of changing workspaces on UI (some kind of list of workspaces or something like that)? 2. I have updated airbyte on GKE from 0.35.28-alpha to 0.35.46-alpha. I'm using GCP Cloud SQL as Database. I do not want purge the table and recreate as I have lots of connections already created. I have received this error trying to reach Airbyte UI error on server occurred:

SQL [select * from "public"."actor_definition" where "public"."actor_definition"."actor_type" = ?::"public"."actor_type"]; Error while reading field: "public"."actor_definition"."resource_requirements", at JDBC index: 15 at org.jooq_3.13.4.POSTGRES.debug(Unknown Source)

3. I was developing new connector and in logs I see error, but data is saved in destination:

2022-03-07 11:52:04 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: null
2022-03-07 11:52:06 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed.

Hi 👋 I'm trying to set up Google Ads as a connector, and I'm going through the google AdWords API application to get my tokens. They have asked for a design document for Airbyte. Does anyone know what I could use for this?

Hello there, Is it possible to execute not an incremental sync, nor a full-refresh one but something in the middle? I have a use case when I only need the last 30 days of data of a source (shopify and Google Analytics in my case) Is it doable?

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9.983793757s. what does this error means

Hi everyone, we currently use airbyte and now want to set up a new connection (as we are deploying a new database). My question is if we need a new service account (GCP) or can we use the existing one?

^ Actually it seems like clickhouse is not even supported as a destination, is this correct?

Also something weird is happening, so I wasn't able to setup on my local machine. I tried to do that on GCP and after trying to setup a connection, it failed. But weird enough, I decided to peek at my dataset created on BigQuery. Lo and behold I could see airbyte has written to the warehouse. Quite buggy for my first experience.

Hi 👋 is there any tutorial on how to create airbyte connections dynamically? in detail: I want to deploy airbyte on AWS and setup connections, sources, destinations dynamically from my backend service (also deployed on AWS). By dynamically I mean without using the GUI.

Hi There, I have a mysql table I wanted to replicate onto my data warehouse, would it be possible to limit this to certain number of rows, or by date. Let's say for the past x day/month/year? My use case is I have data dating back from 2013 and I would like to replicate only the past year. I guess I can do this by creating another table that only pulls what I wanted from the main table. Does anyone else have any recommendations?

Hey all, I just started to look into airbyte and it's caps to check if it will be good match for my needs. for now still running locally. I am getting

Cannot reach server. The server may still be starting up.

error in airflow's UI. logs:

airbyte-webapp      | 172.20.0.1 - - [08/Mar/2022:09:35:36 +0000] "GET / HTTP/1.1" 200 1151 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.2 Safari/605.1.15" "-"
airbyte-webapp      | 172.20.0.1 - - [08/Mar/2022:09:35:36 +0000] "POST /undefinedworkspaces/list HTTP/1.1" 405 460 "<http://localhost:8000/>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.2 Safari/605.1.15" "-"
airbyte-webapp      | 2022/03/08 09:35:39 [error] 28#28: *411 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.20.0.1, server: localhost, request: "POST /api/v1/workspaces/list HTTP/1.1", upstream: "<http://172.20.0.8:8001/api/v1/workspaces/list>", host: "localhost:8000", referrer: "<http://localhost:8000/>"
airbyte-webapp      | 172.20.0.1 - - [08/Mar/2022:09:35:39 +0000] "POST /api/v1/workspaces/list HTTP/1.1" 502 494 "<http://localhost:8000/>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.2 Safari/605.1.15" "-"

can you help me?

Hi team, I'm configuring postgres db as data source which is deployed in EC2 instance. My airbyte is deployed as k8s pod in

dev

namespace in EKS cluster. Can anyone let me know what's the issue..

Hello All, Good morning. Has anyone tried Hubspot to google BigQuery (denormalized typed struct) sync. I am facing this issue at destination > java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid format: “2021-11-03 135501" is malformed at ” 135501"

Hi Folks - I'm going through the quickstart and had a question about security with GBQ destinations -- is a service account key the only way to authenticate? Is it possible to use the role of the host instance?

i want the dbt tables to be used by the bi tools and they refresh independently. So, dbt models which i create needs to be incremental right so that data is always present for bi tool

Hi Team, I'm trying to add Postgres as a source in Airbyte using SSH tunnel. I've correct username/IP and SSH private key

RSA

which has

-----BEGIN RSA PRIVATE KEY-----
<content>
-----END RSA PRIVATE KEY-----

It tries to make connection but eventually fails. Here are the logs. It's urget, response will be highly appriciated!!

2022-03-09 05:23:53 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$8):523 - Setting stdout...
2022-03-09 05:23:53 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):534 - Setting stderr...
2022-03-09 05:23:54 INFO i.a.w.p.KubePodProcess(<init>):502 - Reading pod IP...
2022-03-09 05:23:54 INFO i.a.w.p.KubePodProcess(<init>):504 - Pod IP: 10.0.2.114
2022-03-09 05:23:54 INFO i.a.w.p.KubePodProcess(<init>):511 - Using null stdin output stream...
2022-03-09 05:23:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Class path contains multiple SLF4J bindings.
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 INFO i.a.i.s.p.PostgresSource(main):361 - starting source: class io.airbyte.integrations.source.postgres.PostgresSource
2022-03-09 05:23:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Found binding in [jar:file:/airbyte/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {check=null, config=source_config.json}
2022-03-09 05:23:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-03-09 05:23:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: See <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation.
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 INFO i.a.i.b.IntegrationRunner(runInternal):105 - Running integration: io.airbyte.integrations.base.ssh.SshWrappedSource
2022-03-09 05:23:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Command: CHECK
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 INFO i.a.i.b.IntegrationRunner(runInternal):107 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 INFO i.a.i.b.s.SshTunnel(getInstance):170 - Starting connection with method: SSH_KEY_AUTH

Hi 😄 I'm also now having a bit of trouble setting up Facebook Marketing as a connector. I'm getting this error in the screenshot. Has anyone else had this problem? I'm not sure whether it's something to do with the set up of the app on Facebook Marketing, as they were asking for a screen recording of someone using the app. I could still get the API token so I tried without, but maybe I do need to provide the screen recordings to facebook and get this approved - has anyone had to do this?

I don’t understand how I can filter out all the

_airbyte_tmp_

and

_airbyte_raw_

tables from my migration. Is there a way to have those go somewhere else so my schema isn’t filled with all these tables cluttering up the tables we actually use?

Hello everyone, We are seriously considering switching to Airbyte after running into issues with the script we built ourself to move data from MongoDB to Bigquery for reporting. The main question I have before I can commit to using Airbyte is, does it support multidocument transactions in Mongo? We used to use Fivetran until we ran into this issue (which is why we had to build our own script).

Hey! I’ve been trying to find information about the release date of CDC support for Oracle DB but I can’t find it. When will it be released?

hi, im new to Airbyte and just wondering if it is possible to config the source/destination/connection and run the sync jobs without using the WebApp, but by providing relative config json files/commands only? i guess it is possible but couldn't find the documents

Hello, I want to perform join queries in the destination side. I am using full refresh append so there will be many copies of the same data at different intervals. I want to join them based on the interval or sync number at which they were synced. So, Is there a way to add sync number or sync ID kind of thing or something like this which will help me in this case?

Hello Team, I am getting wrong timestamp at

_airbyte_emitted_at

column if I am using bigquery denormalized destination whereas If I am using bigquery destination it was giving correct value. Can you help me out on why am I getting wrong value. The value I am getting while using bigquery denormalized is

54155-07-08 16:40:49 UTC

.

in mysql to mysql connection datetime columns are replicated as string. Ex: '2018-02-09 094500' is replicated as '2018-02-09T064500.000000Z'. Is there a solution in airbyte for this?

Hi, I'm having issues deploying Airbyte on a windows 10 machine in docker desktop. The volumes appear to create properly but it goes into an endless failure loop on startup which appears to be due to the SchedulerApp waiting on the server and timing out.

airbyte-server      |   at io.airbyte.server.ServerApp.getServer(ServerApp.java:185) [io.airbyte-airbyte-server-0.35.49-alpha.jar:?]
airbyte-server      |   at io.airbyte.server.ServerApp.main(ServerApp.java:236) [io.airbyte-airbyte-server-0.35.49-alpha.jar:?]
airbyte-scheduler   | 2022-03-11 15:47:10 INFO i.a.s.a.SchedulerApp(waitForServer):220 - Waiting for server to become available...
airbyte-worker      | 2022-03-11 15:47:11 WARN i.t.i.r.GrpcSyncRetryer(retry):56 - Retrying after failure
airbyte-worker      | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.998877400s.
airbyte-worker      |   at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.44.1.jar:1.44.1]