Hey guys, been playing around with airbyte and it looks really cool. Couple of things that I see as roadblocks to adoption: 1) Scheduling with CRON: Can see the GitHub discussions on this so hoping this would be closed sometime soon. 2) Can't figure out how to sync only particular columns of a selected table to help keep the data warehouse lean. Was wondering if anyone had solved any of these or had some workarounds

Hi , I have two questions about using Airbyte. 1. What are the best practice for switching from historical sync to incremental sync ? 2. How does Airbyte keep track of state, is it stored somewhere? It seems that Airbyte is reading our db before it writes to it, is this how it is keeping track of state?

root@airbyteone:/mnt/programs/airbyte# docker --version Docker version 20.10.7, build f0df350 root@airbyteone:/mnt/programs/airbyte# docker-compose up ERROR: Invalid interpolation format for "environment" option in service "scheduler": "AIRBYTE_ROLE=${AIRBYTE_ROLE:-}"

Hi all have a question regarding loading data from s3 to snowflake. I have a folder in s3 where I store 100+ tables each with a respective folder. While I trying out the File connector with s3, the config is forcing me to have a specific table meaning that for my case I would need 100+ connections to feed all of these tables in my Warehouse. Is there a way around this?

Hey Guys, quick check in. For postgres to postgres data transfer would a db sync / replication be better (it's realtime and lighter as it would sync only the changes) or would airbyte have any advantages (usability definitely is one)

Hi Team, i am getting started with Airbyte. Have setup Airbyte on my local windows PC and installed a MySQL database. Tried to create a source with MySQL database. And get the below error. "Could not connect with provided configuration. Error: Cannot create PoolableConnectionFactory (Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)"

@Ranjith Kumar @Davin Chia (Airbyte) I have faced the similar issue

Hi mates.. I was having some troubles at upgrading about

versionMismatchServer

, so I decided to erase whole project and clone it again from your repo, and I'm having the same error. Thank You

Hola, This is Ram.. I have trouble connecting my dbt git repo and doing a transformation.. Is there any simple documentation that helps me understand how to connect Airbyte with git repo (dbt code) and perform the transformation? Also have a doubt with Airbyte transformation (dbt).. does dbt connect to source, does the transformation and loads to target (ETL way) or does dbt do transformations from target (ELT approach) or can we do both??.. Appreciate your comments and views, Thanks in advance!

Hello! I am Panos from Lavanda. I am looking at Airbyte and I was trying to understand how the different

Sync settings

work when setting up a connection between a source and a destination. Is there a document that would describe the different options? Thank you very much in advance.

Hey folks, just onboarding to the tool. Love how simple it is to get started. Can you confirm if this is best practice? 1. I have an orders table that gets updated if it changes, no updated_at field unfortunately. Should I use

full refresh - overwrite

, since I need the latest and greatest, and the primary key will not be enough as a cursor field? 2. I have an order_carts table that gets updated if there's a new event, with a created_at field. Existing rows will never be changed. In this case, I think I can use

incremental - append

, since it would just add the new fields since the last sync.

Hi everyone, I'm new to Airbyte I have a task to migrate a db from AWS Ec2 to BigQuery. I'm trying to do testrun before working on the actual project by setting up a basic MySQL db on EC2 and try to use Airbyte to migrate it to BigQuery. I'm currently getting this error when i try running the last instruction on the guide # In your workstation terminal ssh -i $SSH_KEY -L 8000localhost8000 -N -f ec2-user@$INSTANCE_IP Permission denied (publickey,gssapi-keyex,gssapi-with-mic)

Since there isn't a way to connect via ssh key in the UI, I'm running into connection errors.

Could not connect with provided configuration. Error: Cannot create PoolableConnectionFactory (The connection attempt failed.)

How do you recommend I connect to my postgres db on ec2?

Hello, I'm trying to setup my Postgres DB as a destination with Airbyte running locally. It works fine when I use a super user, but doesn't work with my dedicated Airbyte user. I can connect without an issue when accessing the DB directly with that user, but Airbyte gets a "Could not connect with provided configuration. ERROR: permission denied for database <database>". Any idea? The DB is hosted on AWS RDS, but the IP is allowed (and again, it works when using a super user, and when accessing it via other tools). Thanks in advance!

Hello, anybody ever made a source connector to deal with SOAP APIs? I'm trying to adapt the HTTP connector to do that, but was just curious to know if somebody already went through this and has some tips/wants to share the experience.

Hello, we are just getting started with Airbyte, we have some data loads going between databases and Redshift and those are great. We are running into some problems though in loading csv files from SFTP sources. The normalization process is failing on them as it is casting the fields to the wrong data types automatically. I understand that the theory behind Airbyte is really focused on the EL layers and not so much transformation, but I’m trying to figure out the easiest way to modify the field types. Is the only way within Airbyte to build a dbt transformation that the connection can use, or are their any options that are easier to just change a few fields that are automatically detected as an incorrect type? Thanks for any help you can provide.

Hi again, from https://docs.airbyte.io/integrations/custom-connectors#adding-your-connectors-in-the-ui I read

1.Get the

Docker

coordinate of a custom connector from

Dockerhub

(or any image repository that Airbyte can access).

Can somebody tell me which image repositories that includes? Specifically, is there a way to use private repo like AWS? Thanks!

I recently did a sample local MySQL data to BigQuery as test-run before implementing on my actual EC2 MySQL migration project. I'm curious to know why extra copies of data with _*airbyte_raw*_ were added in addition to the main tables from the source. What does this extra tables host?

Hello! Creating a custom source connector and running into some trouble I am having the hardest of times trying to jump from "discover" to "read" Following the Python tutorial and I can't understand WHERE configured airbyte catalog comes from?? The output from discover generates a Airbyte Catalog, but then during the "read" command, it takes a --catalog option to read from a file??? Even in the tutorials you never write to anyfile.. so where is this configured airbyte catalog supposed to come from??\

The only logical thing I can think of is that you manually define the Configured Airbyte Catalog that "read" uses but then what is the point of the output from discover??

also there is no "main_dev.py" when using the generator.sh

Hi Team, We are just getting started with airbyte by connecting amplitude (source) and redshift (destination). I keep a historical sync date as of a month back..but it is only syncing data since today. Would need help on how can we export historical data from Amplitude using airbyte

I’m unable to run Airbyte locally because I’m using an M1 Mac but i’ve got it up and running on EC2… What would be the best way to develop a custom-source connector?

Hi Everyone, I really do apologise in advance for my dumb questions but I just needed some clarification. I’m trying to setup air byte on AWS EC2 following the documentation here https://docs.airbyte.io/deploying-airbyte/on-aws-ec2, so I was wondering once I complete the setup following the instruction and since I need to use my local browser/ terminal to connect to the Airbyte UI & browser to complete the sync-source/destination for the data migration , does the synchronisation job continue to run in the background even after I close my terminal/restart my local machine or is any setup required to fix this? Thank you

What’s the difference between an AirbyteStream and ConfiguredAirbyteStream?

Hi Everyone, I was trying to understand mysql as connector, under the integration runner  i couldn't locate where is the read method defined. https://github.com/airbytehq/airbyte/blob/766ddab895b625d49927d0ed7baa7ad9ae55b607[…]c/main/java/io/airbyte/integrations/base/IntegrationRunner.java Line 107, i could see multiple definitions of read() method, but if someone can point me to the exact definition it would be great. Have  been hustling since days.

check this

Hi all, I am just about to get started with Airbyte and my expected deployment is on an EC2 VM. I just have a question with respect to how and where the data is moving from source to destination. My source is likely an AWS s3 bucket or Amazon RDS living in a VPC and the destination is Snowflake setup with AWS PrivateLink to our source. If we have our EC2 VM deployed on the same AWS network will Airbyte be moving the data across the public internet or will it leverage the PrivateLink? Due to security risks we don't want this data in particualr to move across the public internet

I'm going through the tutorial on this page: https://docs.airbyte.io/contributing-to-airbyte/building-new-connector/tutorials/building-a-python-source When I run the code for step 1 (run the generate.sh file to create a new source template). The template file runs, but I am never prompted to give the source a name as the instructions on the webpage indicate. Has something changed in the .sh file? Or could I be missing some vital step?

Hey again, is there a way to select which fields/columns to replicate in the UI? I only see the option to include/exclude tables. Thanks in advance!