Hello, I'm have a question regarding the Conversion Window for the GoogleAds source. From my understanding, this would fetch duplicate data when using the Incremental Append sync. Is there a column value we can use to query only the latest data? For example, Stitch provides a specific column to allow this -

_sdc_report_datetime

(https://www.stitchdata.com/docs/integrations/saas/google-ads#query-for-the-latest-data) Is this something we would have to do with a custom transformation? I noticed only the

_airbyte_emitted_at

column, but this is record specific, not batch specific.

Hello folks! I am interested in populate a datalake on S3 with Jira data. I saw that Airbyte can does it. Q: Airbyte needs an orchestrator like Airflow, Prefect or Dagster or works alone better? 👋

Hello folks, I was trying out airbyte today. I've setup airbyte on my local machine and try to use Clickhouse connector as a source and S3 bucket as my destination. I am getting this error`The AWS Access Key Id you provided does not exist in our records.` I am able to upload files to s3 from aws cli from my mac. Could you please help? Thanks!

Hello, I am on a mission to connect Mongo DB cluster to Big Query. • My airbyte is hosted locally with Docker. • Airbyte Source test from Mongo DB cluster was successful • Airbyte destination setup test to Big Query was successful. • Connection setup test between source and destination was successful. • Syncing started and kept running for a long time but no data was in my big query dataset. • When I checked the run log on airbyte, there was an error that database name of Mongo DB cluster must be alpha numeric. (Although the syncing did not stop). • Please how can I bypass this without changing DB cluster name as it will take a lot of effort for the engineering team to change the name of the DB cluster as it is a live production DB? Please help me resolve this. Thank you very much.

Soa, Airbyte vs other tools anyy comparison ?

Hi all, I'm trying to get things setup so that my destination files are saved in folders specific to the connection. I'm loading into Azure Blob Storage. I've looked at this document but it seems to only explain custom naming of the files. https://docs.airbyte.com/understanding-airbyte/namespaces#custom-format Anyone know how I can update my connection settings to put the extract files into a folder?

Hi folks - I’m starting to play around with the Airbyte API. Where can I get the API hostname for the endpoints (cloud)?

Hey guys, I have added clickhouse as a source, and S3 as a destination and tried to make a connection, I am getting

Failed to fetch schema. Please try again

. Does anyone know why?

Hello Team, I am trying to deploy airbtye today and followed below instructions git clone https://github.com/airbytehq/airbyte.git cd airbyte docker-compose up It fails to start and throws the following message airbyte-worker | 2022-01-11 061336 INFO i.a.c.EnvConfigs(getEnvOrDefault):577 - Using default value for environment variable WORKER_ENVIRONMENT: 'DOCKER' airbyte-server | 2022-01-11 061340 ERROR i.a.s.ServerApp(main):274 - Server failed airbyte-server | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.998932000s. airbyte-server | at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.42.1.jar:1.42.1] airbyte-server | at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.42.1.jar:1.42.1] airbyte-server | at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.42.1.jar:1.42.1] Any suggestions on how to resolve this error? airbyte-server | at io.grpc.health.v1.HealthGrpc$HealthBlockingStub.check(HealthGrpc.java:252) ~[grpc-services-1.42.1.jar:1.42.1] airbyte-server | at io.temporal.serviceclient.WorkflowServiceStubsImpl.lambda$checkHealth$2(WorkflowServiceStubsImpl.java:282) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.temporal.serviceclient.WorkflowServiceStubsImpl.checkHealth(WorkflowServiceStubsImpl.java:275) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.temporal.serviceclient.WorkflowServiceStubsImpl.<init>(WorkflowServiceStubsImpl.java:182) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:51) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:41) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:40) ~[io.airbyte-airbyte-workers-0.35.4-alpha.jar:?] airbyte-server | at io.airbyte.server.ServerApp.getServer(ServerApp.java:186) ~[io.airbyte-airbyte-server-0.35.4-alpha.jar:?] airbyte-server | at io.airbyte.server.ServerApp.main(ServerApp.java:272) [io.airbyte-airbyte-server-0.35.4-alpha.jar:?] airbyte-server exited with code 1 airbyte-worker | 2022-01-11 061342 WARN i.t.i.r.GrpcSyncRetryer(retry):56 - Retrying after failure airbyte-worker | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.973427100s. airbyte-worker | at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.42.1.jar:1.42.1] airbyte-worker | at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.42.1.jar:1.42.1] airbyte-worker | at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.42.1.jar:1.42.1] airbyte-worker | at io.grpc.health.v1.HealthGrpc$HealthBlockingStub.check(HealthGrpc.java:252) ~[grpc-services-1.42.1.jar:1.42.1] airbyte-worker | at io.temporal.serviceclient.WorkflowServiceStubsImpl.lambda$checkHealth$2(WorkflowServiceStubsImpl.java:282 airbyte-worker | 2022-01-11 061336 INFO i.a.c.EnvConfigs(getEnvOrDefault):577 - Using default value for environment variable WORKER_ENVIRONMENT: 'DOCKER' airbyte-server | 2022-01-11 061340 ERROR i.a.s.ServerApp(main):274 - Server failed airbyte-server | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.998932000s. airbyte-server | at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.42.1.jar:1.42.1] airbyte-server | at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.42.1.jar:1.42.1] airbyte-server | at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.42.1.jar:1.42.1] airbyte-server | at io.grpc.health.v1.HealthGrpc$HealthBlockingStub.check(HealthGrpc.java:252) ~[grpc-services-1.42.1.jar:1.42.1] airbyte-server | at io.temporal.serviceclient.WorkflowServiceStubsImpl.lambda$checkHealth$2(WorkflowServiceStubsImpl.java:282) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.temporal.serviceclient.WorkflowServiceStubsImpl.checkHealth(WorkflowServiceStubsImpl.java:275) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.temporal.serviceclient.WorkflowServiceStubsImpl.<init>(WorkflowServiceStubsImpl.java:182) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:51) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:41) ~[temporal-serviceclient-1.6.0.jar:?] airbyte-server | at io.airbyte.workers.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:40) ~[io.airbyte-airbyte-workers-0.35.4-alpha.jar:?] airbyte-server | at io.airbyte.server.ServerApp.getServer(ServerApp.java:186) ~[io.airbyte-airbyte-server-0.35.4-alpha.jar:?] airbyte-server | at io.airbyte.server.ServerApp.main(ServerApp.java:272) [io.airbyte-airbyte-server-0.35.4-alpha.jar:?] airbyte-server exited with code 1 airbyte-worker | 2022-01-11 061342 WARN i.t.i.r.GrpcSyncRetryer(retry):56 - Retrying after failure airbyte-worker | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.973427100s. airbyte-worker | at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.42.1.jar:1.42.1] airbyte-worker | at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.42.1.jar:1.42.1] airbyte-worker | at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.42.1.jar:1.42.1] airbyte-worker | at io.grpc.health.v1.HealthGrpc$HealthBlockingStub.check(HealthGrpc.java:252) ~[grpc-services-1.42.1.jar:1.42.1] airbyte-worker | at io.temporal.serviceclient.WorkflowServiceStubsImpl.lambda$checkHealth$2(WorkflowServiceStubsImpl.java:282

Hi All, Is there a way to use airbyte with snowflake and to create the tables without being trasient table?

Howdy y'all! My name is Farman and I'm a software engineer at Mailchimp. I'm doing some vendor research for my team, Data Integrations, and I'm really intrigued by what Airbyte has to offer. I'd like to present a use case and make sure that I understand what Airbyte does so that I can further present this to my stakeholders and use this platform. So I need to import data from our big query table into a data science tool that uses a GCS bucket. What happens in between is data transformation so that the results we get from our big query table are turned into a format that is easily accessible for the data science tool. Can I leverage airbyte to do all of this? We expect to have similar work and want to be able to "productize" my team. I think this will also be a good use case to contribute to your open source projects but need to make sure I understand all of this. I've read the documentation and worked with it a little bit but can never be too sure by myself, haha.

Hi. I was exploring different ETL tools for my work. We have a need to extract data from a few sources and drop them into S3. One of the APIs is time sensitive. Does airbyte open source have built in logging or state if we want to make sure we have not missed any windows?

Hi team, how are you? I'm exploring airbyte with another ETL tools, and I got a doubt. When I try to get data from a source and put it into an S3 bucket it doesn't show the field to use dbt transform scripts? There is a reason why ? I meam if I choose to do some transformation like a traditional ETL instead of an ELT it won't be possible with S3 destination? My case is that I need to anonymize some data before get it into my Lake.

Hey guys! Nice to e-meet you! I was wondering if someone running Postgres->Redshift CDC successfully and could answer few potentially silly questions? 🙂 We're looking for a working solution to copy our prod database to BI, tried different tools (all of those we checked had some shortcomings) and now landed to airbyte, but I am a bit confused that it seems a lot of data types are not fully supported/converted to varchar? (https://docs.airbyte.com/integrations/sources/postgres#data-type-mapping) It sounds like it's not a very rare case to copy PG to Redshift, so I'm curious if it's docs which are obsolete (cause this PR seems to be adding more types? https://github.com/airbytehq/airbyte/pull/7339) or there is some proven way to get more types into Redshift (some dbt scripts?) Thanks in advance! 🙂

Hi Team, I'm new to Airbyte, planning to carry out an evaluation. Just wanted to know on authentication mechanisms supported on Airbyte UI console

Hi, what’s the easiest way to deploy airbyte ? Do you have a click to deploy droplet ?

Hi Team, I'm new to Airbyte. Currently we use Airflow (Google Cloud Composer) SFTPtoGCS operator to pull raw files (Parquet, Avro, txt, csv, json) from SFTP locations to GCS (Google Cloud Storage), and later load into BigQuery (SFTP->GCS->BigQuery) based on schema/ data dictionary; but checking if Airbyte can do SFTP->GCS. I tried using

File

source with SFTP and

Google Cloud Storage

as destination within Airbyte, but it requires for me to 1. Specify File schema Reader Options for each file 2. Create a source for each file I am checking if Airbyte can get all raw files from SFTP and dump into GCS location on scheduled bases without supplying reader option/ schema.

Is there a way to delete the _raw tables when it’s done syncing data? They are taking up a lot of space and cluttering our schemas.

Evaluating Airbyte 0.35.4-alpha docker image with destination as Redshift. I get airbyte/destination-redshift-0.3.23 image does not exist. Not sure if I missed any pre-requisite steps or if the image has a defect

Hello!! I have one question, airbyte growth 🔥 How Airbyte scale ? on AWS I can see only deploy using EC2. What happen when it is necessary scaling at the moment? when the load of ingest is bigger https://docs.airbyte.com/deploying-airbyte

Hello, I have made a connection from mysql to big query. the schema shown is right but I cant sync and get an error. any ideas is appreciated

Hi Team, I'm new to Airbyte, I was trying to build the airbyte component separately and build is failing with the following error

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/kshitijchaurasiya/SandboxDev/airbyte/airbyte-cdk/python/build.gradle' line: 3

* What went wrong:
An exception occurred applying plugin request [id: 'airbyte-docker']
> Failed to apply plugin 'airbyte-docker'.
   > A problem occurred starting process 'command 'docker''

And docker is running properly on the system, so I'm not sure why its failing, is anyone else faced the same issue.

➜  airbyte git:(master) ✗ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Or Is there is some documentation of building the component separately. Any help will be appreciated.

Hello!, from Yesterday I am trying to deploy Airbyte on EC2 following Quickstart page. I am having problems, when I try to access to the UI using ssh tunnel. the instance does not respond and the command docker ps does not work. I tried to deployed 3 times and the same complications

I couldn’t find it but I was wondering if there was any way to delete specific data for RTBF requests. So like where we can plug in a user id or something and it will delete all associated data from a warehouse.

Connectors are open source correct? We are able to extend them as needed for data we might need that isn’t currently implemeented?

What does the migration from self-hosted to cloud look like? Is there a way to import existing connections? Does migrating require a full re-sync of the destination?

Hey team! Thanks for all great stuff you're creating🚀 Love Airbyte UI ❤️ I wanted to dig a bit deeper in this conversation (https://airbytehq.slack.com/archives/C021JANJ6TY/p1641999469109500?thread_ts=1641983077.104500&amp;cid=C021JANJ6TY). In short question is - what is the proper way to preserve types in Postgres to Redshift exports? (details in the thread)

Hello, I have a couple questions. Overall been very pleased with the experience with Airbyte. However, I'm testing setting up various connections and am a bit confused. As we are frequently adding and modifying tables in our source Postgres application database and need to sync them to Snowflake, we'll constantly be updating the connections. When I select the 'Update to latest schema' button it is telling me it's going to reset ALL of my data and start all over... Some of our tables are hundreds of GBs, so this will take days. The source schema for most tables hasn't changed.. I'm just looking to add an additional table. In this case is it recommended to create a connection per table? In that case is there a way to 'name' a connection so we're not looking at a list of 75+ connections all with the same exact information (source / dest) shown on the page? Thanks for the help!

Does anyone have examples of using airbyte and dbt together and a walk-through on how to configure it and what the prerequisites are?

Hello folks!!! 👋 I am new on Airbyte, i was looking for a generic connector to consume a generic API, like a python request. Is there this option on Airbyte?