👋 Hi everyone! YCW19 company here... love what you guys are building. I was wondering where webhooks are on the roadmap? I tried poking around but could not find anything

Hello, I don't know which Chanel to put this in: I'm trying to test airbyte locally but once I've downloaded Docker I open my terminal and I get an error message when I execute the command. Where do I have to look?

are there plans to support using an external database? also, an all-in-one docker container (bundled server, scheduler, webapp) that points to an external database would be ideal for us to to deploy to ecs

Hey 👋 How does one go about estimating EC2 instance size needed to run Airbyte? We set it up on t3.micro, though our syncs fail and the logs include java error messages around running out of memory — assuming this is because the instance does not have enough memory?

The team is off tomorrow, so exceptionally there won’t be any office hours unfortunately. They will resume after Thanksgivings. Thanks for your understanding airbyte heart !

If anyone is around, can you tell me if you have seen this? Running locally, the sources list is empty

Another framework (Meltano) that uses Singer taps advocates / requires separate Python virtual environment for each tap. Does the Airbyte application model resolve this? I vaguely recall reading that each tap is its own container. Sorry for a newbie question but this came up on conversation recently and hoping someone might shed some light.

Hello ! I just launched Airbyte on my local desktop, it is great ! 👻 I saw in the doc that there is a REST API source but I can't find it on the app 😞

For Airbyte deployment, has anyone or would anyone recommend deploying airbyte as an ECS Fargate? Any tips or processes to get it working properly?

Also in reference to https://docs.airbyte.io/deploying-airbyte/on-aws-ec2, Sorry for the dumb questions in advance…. 1. Once i set up a sync using this install approach, does it continue to run in the background (even after I kill my terminal) or do I need to set up the service? 2. The docs use t2-micro - how much load can this handle? 3. Any tips on how to secure this instance so that it can be accessed only via SSL?

Has anyone here found S3 as a good alternative to warehouses like Redshift? If yes, what approach was used to sync data from a DB like Postgres into S3? And what BI tool was used to analyse the data?

<!channel> Hey everyone 👋! here are the updates for last week. • new source connectors: Sendgrid, Recurly, Freshdesk, Twilio • new destination connector: Redshift is going through final tests and should come this week • contributions WIP: Hive (source), DynamoDB (source), Trello (source), Mailgun (source), MySQL (destination), Kafka (destination) • progress on incremental: we want to let you choose whether to replicate data incrementally or with full refreshs (incremental being the default option). here is what we’re building. It should be released within 2-3 weeks. let us know if you see anything missing! https://www.figma.com/file/pO3Ob5W0yKUFOQzvg8TR3z/Airbyte?node-id=1131%3A595 • more info on sync jobs / attempts: some users were having a hard time debugging, as several sync attempts can be made if sync fails. So we want to give you visibility to the attempt level. Here is what is coming shortly. let us know any feedback you have! https://www.figma.com/file/pO3Ob5W0yKUFOQzvg8TR3z/Airbyte?node-id=1197%3A7927 • just last week we got 30 companies that started syncing data with us, even though we had Thanksgiving! The more adoption, the more contribution, the more well-maintained connectors airbyte rocket

Hi, I have a very generic question. When data from a source is transferred to a target which guarantees are given that no data gets lost? This question relates to guaranteed message delivery in messaging systems like Kafka or Pulsar. Is the transfer of data "transaction safe"? Where are "tombstones" written that in the case of a failure the process of transferring data continues where it left off? The same question but now related to "exactly once". How is assured that data is not written twice?

Kubernetes is deprecating Docker as a container runtime after v1.20. https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/

Reminder: This is office hour time!! Join now: https://us02web.zoom.us/j/84317290190?pwd=S0E5RzZRV2NrTGw1VDM5cFZoV2JCUT09

Hello everyone, My name is Diego. I am a geographer and I work on geospatial data. I usually use postgres and postgis to store my data. Sometimes I use elasticsearch too. In the last project, finished this week, I used a soup of strategies to extract and to load all the data I needed (shell, python, PDI). I used the rundeck to build and schedule the pipelines. For the next projects I intend to use a more modern approach with new tools. Googling for modern FOSS ETL tools I found the Airbyte. I like its approach but I have some doubts. The main doubt concern the support for geospatial data. Reading the postgres docs, I couldn't find any mention to spatial data. So I'd like to know if I lost something or if this feature is yet not completely developed.

<!channel> Hey everyone 👋 ! here are the updates for last week. • Our first Connector Contest from today to 12/23: 2 winners for mechanical keyboards, and surprises for all participants. Learn more here! The team is excited to help you! • 2 new destination connectors: Redshift🔥🔥 ! You can use Redshift to connect your Panoply, it will work too (we'll work on a dedicated configuration process soon). • 5 new source connector: Slack*,* Zendesk Support,* *Braintree, Redshift, Greenhouse • progress on incremental: we will alpha test our 1st version of incremental (append-only) on one connector this week. But we're in shape to release on all connectors within the next 2 weeks!

👋 Hi everyone!

Hello everyone! I've joined this slack community as I'm about to write some connectors. I've been reading the docs and this page seems to be the reference guide on how to write connectors using airbyte's best practices, is that right? • https://github.com/airbytehq/airbyte/tree/master/docs/contributing-to-airbyte/building-new-connector • https://github.com/airbytehq/airbyte/blob/master/docs/architecture/airbyte-specification.md

it seems we now have access to the "Discussions" tab on our GitHub repo (Discourse-like). don't hesitate to ask your questions / ideas / comments there! https://github.com/airbytehq/airbyte/discussions

I see now its getting added as "public.<table_name>_raw"

Hi, I have a question regarding the sync of data from a relational database. If more than one table should be synced how is a consistent snapshot created and/or kept/stored for all the tables? Or can they be out of sync?

Hi all, Quick question for deployment options: is Azure being considered?

<!channel> 👋 Hey everyone! We had to share this with you 🙂. We've released our first version of incremental replication 🔥🔥🔥! Incremental Append is now available for 10 sources (including Postgres, MySQL, Intercom, Zendesk, Stripe, Twilio, Marketo, Shopify, GitHub, etc) and all destinations. We will expand it to all sources in the next couple of weeks. We know a lot of you were waiting for it to start using Airbyte on prod! Let us know how it goes airbyte rocket

Hi, everyone! I'm new with Airbyte , so I just did the quickstart but when I go to http://localhost:8000/onboarding the source type list is empty. Can someone help me with this issue? Thanks!!!

git clone <https://github.com/airbytehq/airbyte.git>
cd airbyte
docker-compose up

Hey Airbyte team, I’ve used in my previous projects: Stitchdata, Airflow, Meltano, Singer and DBT, and now I’m trying to grasp all the details about Airbyte to decide if I use it for a new project I have.  I read all articles from Michel and John (about the differences between Singer, Airbyte and Meltano, also about the unified Data Infrastructure Architecture, etc) they make a lot of sense and I really enjoyed them :) I agree with all the problems you guys mentioned about Singer (and also recognize their importance in laying a foundation for the EL protocol). But I still have some questions that I wanted to double check with you: 1. When we use Airbyte and I want to run in production, I should install it on a GCP compute engine (in my case), access the UI and do all the connector’s setup via UI, similar to what I would do if I was using Stitchdata. But when you guys deploy a new version with new connectors, or change any connector, how do I deploy them to my server? Do I just pull from the github master branch? If I do that, what happens to: a) My configuration and state files? (I don’t lose it because Airbyte is using named volumes? There is no database for that like Airflow usually uses, right?) b) Could there be any compatibility issue if you deploy changes to a connector that I’m using? 2. If I want to develop a new connector that is not in your roadmap (for example a niche brazilian platform that we use here), a) Should I follow your guidelines and submit a pull request? (I read the documentation that mentions I don’t need to maintain anything). b) I could use a separate branch without sending a pull request, but this is not the desired scenario, correct? 3. As Airbyte’s goal is to handle only EL and basic normalization (equivalent to what Stitchdata does), for the ‘T’ part I should use something else, for example, DBT. a) And to run DBT I’ll need to use a separate orchestrator: it can be a simple cronjob, or airflow or their own platform, correct? So, differently from Meltano, Airbyte's goal is not to unify ELT frameworks under the same roof, but to be the best open source tool for EL correct? b) Do you plan using Kubernetes to parallelize processing and decrease server costs?

Merry Christmas 🎄 everyone!!

Hello

Question, how do i make sure pagination work for a http source?

@Thomas Giardina - Thank you for taking part in our Connector Contest! He contributed the Plaid source. This source also happens to be our first source written in JavaScript! This is also likely the last new source of 2020 (unless anyone else has one they are going to knock out in the next few hours 🤣).