Cross posting some great news 🎉 https://airbytehq.slack.com/archives/C01PCV47CTY/p1615041229009400

I am okay with providing what I have inputed but let me know if I should do it in other chanell

Hello everyone! I have good news, i have made some progress with using Airbyte and now the sync of some pretty large tables in our company is almost working: The last bit of problem happening now is the follow: After Airbyte completes loading data from source to destination (ie Snowflake), it triggers the finalization procedure, moving data from table

AIRBYTE_STG._airbyte_1615200259762__airbyte_raw_Campaign

to

AIRBYTE_STG._airbyte_raw_Campaign

. This query is started in Snowflake, but it gets killed exactly after 5 minutes. My guess is that there’s a timeout happening. Airbyte logs are:

What is the ETA for sql server deletes/soft deletes/log based replication?

Hi everyone! I want to know if there is any python API for airbyte or not? For example, a basic task would be to create source/destination definitions using a python interface.

vedant@ved:~/airbyte$ docker-compose up WARNING: The API_URL variable is not set. Defaulting to a blank string. ERROR: Couldn't connect to Docker daemon - you might need to run

docker-machine start default

. vedant@ved:~/airbyte$

Hey all! We're discussing with Databricks, and wanted to know who would use a Delta Lake connector? Not saying it will be a priority for us, but to evaluate how it should be one. 🙂

Hello I have a tough love question, I’m currently comparing Meltano vs Airbyte. I like the answer that you guys provided in Hacker News. I am willing to put the extra effort for the connector to be compliant with Airbyte Protocol vs Singer. However the deeper question is in terms of your future business model. If I spend hundreds of hours creating connectors that are specific to Airbyte protocol and then suddenly I have to pay to use them I will be very very frustrated because that same time I would have spent it using Meltano (or Singer for that matter) instead. Thanks for you help

anyone else seeing slow load times on the UI when loading the source/destination configuration during the onboarding process?

Is there an option for normalized tables to be created as normal tables in Snowflake? Right now Airbyte is creating them as transient tables and we're not able to clone them. We use cloning to separate our dev and prod environments in the data development life-cycle. @Nate

Could not connect with provided configuration

Mariadb is working and connecting to the db with Mysql workbench

Hi I’m evaluating airbyte and meltano. Airbyte looking great so far, but wondering about high availability/more reliable way of production deploying that an ec2 instance running docker-compose

Just joined slack and this may have been answered before, but can we add a

custom destination

(an API call to my internal service) for data being ingested from sources?

Do i need to completely redeploy to update version, or is there a smoother way? i.e: Export configuration -> Terminate VM in GCP -> Create new VM with latest version -> Load Configuration I think i heard something about this in a podcast.

Hi, I'm evaluating Airbyte. Need to know whether I can map source columns to destination columns in airbyte? If not, Are there any references available on how to do it.

is the demo for airbyte down right now? https://demo.airbyte.io/

-------------- We have a question for our beloved community! 👇 We will be building out support for declarative configuration in the next couple of weeks. Do you think we should build it 1️⃣ using Terraform, 2️⃣ building our own configuration DSL or 3️⃣ another option? Please vote with emoji reactions! And don’t hesitate to detail your thoughts in the thread.

Hello For the integration with Airflow, if for example we have Airflow in Kubernetes, how to connect to Airbyte when the recommended way to connect to it is still the SSH ? The documentation shows the connection in localhost but in reality that’s rarely the case

Is it possible to replace the (godawful) airbyte redshift target with one from singer or pipelinewise?

hey airbyte team! is there anywhere i can track the progress on the hosted solution for airbyte? (https://docs.airbyte.io/company-handbook/business-model#2-hosted-solution) asking for a friend

👋 Hi everyone!

Hi! how are all? 😁

Hello!

Hi everyone, I love Airbyte so far. I have a use case where I have a large, inefficient db of data scraped directly from the web, and I would like to use Airbyte to sync to a production db while also performing some transformations on the data - i.e. converting into 3NF. I was looking at the tutorial here and thought this might be possible by pulling out and editing the SQL file generated by DBT. Is there a way to push and execute the modified SQL back to airbyte?

Hey everyone, I’m Luke from www.daybreakhealth.com and I’m trying to understand what it would take for AirByte to replace some of our in house “replicators”. These replicators run on a cron and make HTTP requests against rest and non-rest APIs, parse the response JSON, execute a few minor transforms and then upsert the resultant records into tables in postgresql. From what I can tell it’s a pretty standard extract transform and load concept which should, in theory, be readily replaceable by AirByte, or at least matches a use case AirByte is intended to solve. Am I correct in my understanding so far? I’ve spent a few hours this weekend trying to figure out what it would take to replace our existing replicators and I’ve landed on the five below extensions: 1. Make it possible for AirByte to consume and execute user provided dbt transforms on a per-destination basis. 2. Enhance the HTTP Request source with the ability to “recur” through all pages, in a response. 3. Enhance the HTTP Request source with the ability to query a DB for a set of IDs and then use those IDs as part of the query string. 4. Enhance the HTTP Request source with the ability to query a DB for a set of IDs and then use those IDs as part of the query params. 5. Enhance the HTTP Request source with the ability to query a DB for a set of IDs and then use those IDs as part of the request body. Would changes such as these be welcome?

Hey everyone, We are evaluating Airbyte and have a few questions : 1. How does the current Airbyte RDBMS source connectors deal with loading of very large source tables (Table sizes in 100s of GBs ). Any splits based approach? 2. Any support for custom queries (support for filters, select specific columns, functions, etc) in the RDBMS source connectors. 3. Any plans for supporting cron tab expressions in scheduling?

Hey everyone, I was evaluating Airbyte to move data to Snowflake and have a few questions related to the MySQL connector: • I have a few tables that uses VARIANT as MySQL Type for some json that are saved on it, this is a supported type on Airbyte or its saved as and string value?

👋 Hi everyone! Where can I read more about how changes at the source are replicated to destination while syncing with airbyte?

❓ by the way, we're thinking about adding a Connections tab in the left column. Here is what the design would look like: https://www.figma.com/file/pO3Ob5W0yKUFOQzvg8TR3z/Airbyte?node-id=2099%3A1690 Would that add: 1️⃣ lots of value (must have), 2️⃣ some value (nice to have), 3️⃣ no value ?