Based on our real data test, airbyte is quite strong ETL tool for oracle to Kafka. 👍 . May I know what is the mechanism of periodically sync between the source and destination(e.g. every 30 minutes), sometime the table may not have datetime column which may indicate the current day. Thanks, the outstanding airbyte team.

Is it possible to connect to the Airbyte DB? is there any doc for that? thanks.

What would the price per month be for 2TB or data being synced over to a Google Bigquery instance?

Are we able to use an incrementing ID as the cursor for an incremental strategy for data ingestion on a database?

Is there a way to disable usage data collection in Airbyte?

This message was deleted.

Hi! Can I somewhere change the timestamp format? The original column is a timestamp without timezone, and looks like this "2021-12-16 110029", After synchronise I got this format "2021-12-16T110029Z"

Hi there, is there an easy way to change sync mode in bulk?

Can Airbyte anonymize the data when transferring it? For example convert usernames from postgresql to a hash before sending it to the final DB

Hi, I’m using Postgres as my source and destination db, hosted on Azure, but the sync of merely 10-12 records is taking over a minute. Is there a way I could boost this process?

Hey everyone, is there a way to dynamically set the date while setting a custom namespace in the connection?

Hey, I am trying to deploy airbyte by follow the following command: git clone https://github.com/airbytehq/airbyte.git cd airbyte docker-compose up checked in the airbyte-server encountered the following error: Exception in thread “main” io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.998906743s.  at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) at

Hi, I am new here. I just setup Airbytes and also configured hubspot as source and mysql (8.0.27) as destination. I am using the Basic Normalization option. After syncing data, I have always been getting this error: Caused by: io.airbyte.workers.WorkerException: Normalization Failed. Please find the logs below. Please I need help

Hi, quick question. I created test table and put X number of rows. I synced and they went to destination. Now, If I add more rows to the table, either Airbyte does the full load again with full refresh/append option or it doesn't pickup the new records with incremental/append. Any idea?

Hello, I'm new to the channel and hoping to get started with working with Airbyte on k8s via Helm. Would anyone happen to have an example of a working set of

values

. I keep running into a problem where the server fails to start because it can't connect to the postgres installation.

Hi everyone, I am trying to deploy Airbyte on AWS and am following these instructions. https://docs.airbyte.io/deploying-airbyte/on-aws-ec2 I was able to install docker on AWS and also managed to install Airbyte on Docker, but I can't really Connect to Airbyte. (To be exact below attached instructions are not working for me) Can anyone guide me how can I get started with Airbyte on AWS ?🙏

Hi All , i'm done below steps and open this url in browser http://localhost:8000/ then i tried to setup first freshsale source connector is not connecting .i have fill the name, select the source type it automatically refresh the screen the got this error message " Cannot reach server. The server may still be starting up." git clone https://github.com/airbytehq/airbyte.git cd airbyte docker-compose up

Any idea how I can integrate Airbyte with EventBridge in S3 to have Airbyte Connections run on specific EventBridge triggers, like when something is uploaded to an S3 bucket? @[DEPRECATED] Marcos Marx

I want to collect data from branch.io (branch as destination). I can see this connector is not available right now. However, is there any other option?Branch allows webhook or Data export via API. How can I still be able to collect branch data using Airbyte. Please help clarify.

I am syncing data from hubspot to ms sql. The last log report was 2021-12-22 003815 and nothing seems to be happening since. This is like 9hours after. Please is this normal or is there anything I need to do? I also notice it’s blocking the new scheduled job to run as this ETL is configured to run every 3hours. The next one has not run since this has been blocked. Please help

Hi, is there anyway that I can add Host Aliases (add hosts in /etc/hosts) for airbyte ETL job? We have a kafka cluster that using hostname to install, and we don't setup the dns server. Thanks.

What is the recommended method for moving connector configurations through environments (dev, QA, prod)? - promoting the changes in the config db is an option, but I wasn't sure if there's preferred approach via API

Helloo, my Name Ian, Im new on Airbyte S3, is there any way to customize folder name like create new folder ( YYYY/MM/DD/HH/file_name.csv) every hour/day or both? the purpose is to have historical backup. Thank you.

Hi, trying out Airbyte for our data pipelines and I like what I am seeing so far, this is the product that I've always wanted! I do have a question -- one of the MySQL tables am trying to sync is quite big, with transactions going back 5 years. Syncing invariably fails because of timeout/ closed connection. I don't really need to go back to the beginning, so is it possible to set the cursor to some other value for the first sync? Target is GCS CSV file.

Hi everyone, I'm new to airbyte. Exciting possibilities. I'm doing some testing with Kafka and Postgres destinations. It looks like with the Postgres destination you can apply a transformation as part of the flow. I don't see that option with the Kafka destination. Is that correct? If so, is that something that is on the roadmap?

Hello Everyone! new to Airbyte. While attempting to configure s3 access (as source and via redshift) it works fine with Access key and Secret but when attempted to configure s3 as destination, the connection is failing (with same access and secret which is really strange). Documentation looks straight forward and I’m not sure what is missing here..

Hey everyone, Im new to Airbyte, I’m trying to send data from a postgres RDS to snowflake (both source and destination works correctly) but when the Destination thread complete. The Sync worker failed. Any suggestions?

hi team, I am trying to install airbyte locally for a small POC but the server is failing to start. I am using (https://docs.airbyte.io/deploying-airbyte/local-deployment#setup-and-launch-airbyte) as a guide and have cloned successfully but when I am using the docker-compose up command in the directory the window keeps on giving me the atttached error message. Please guide me.

Hi all, wondering if anyone here has experience using the embed API (https://airbyte.io/embed-airbyte-connectors-with-api) with the Shopify connector? I am specifically curious about the experience w/ getting approved for public apps on the Shopify App Store and whether there were any issues with that

Hi there, I consider using Airbyte to transfer my "final" tables from a PostgreSQL database where all the data transformations are performed to an only read target database dedicated for a BI tool. Do you know if it's a good practice ? Or it's more appropriate to create an RDS database and a read replica for instance ? Thank you in advance for your help !