Hey everyone, I’m Luke from www.daybreakhealth.com and I’m trying to understand what it would take for AirByte to replace some of our in house “replicators”. These replicators run on a cron and make HTTP requests against rest and non-rest APIs, parse the response JSON, execute a few minor transforms and then upsert the resultant records into tables in postgresql. From what I can tell it’s a pretty standard extract transform and load concept which should, in theory, be readily replaceable by AirByte, or at least matches a use case AirByte is intended to solve. Am I correct in my understanding so far? I’ve spent a few hours this weekend trying to figure out what it would take to replace our existing replicators and I’ve landed on the five below extensions: 1. Make it possible for AirByte to consume and execute user provided dbt transforms on a per-destination basis. 2. Enhance the HTTP Request source with the ability to “recur” through all pages, in a response. 3. Enhance the HTTP Request source with the ability to query a DB for a set of IDs and then use those IDs as part of the query string. 4. Enhance the HTTP Request source with the ability to query a DB for a set of IDs and then use those IDs as part of the query params. 5. Enhance the HTTP Request source with the ability to query a DB for a set of IDs and then use those IDs as part of the request body. Would changes such as these be welcome?

Hi Luke! I’ll bring this up with the rest of the team in our next sync meeting. should have an answer for you within a day

Sure, give me a half hour or so to put together a doc we can center the conversation on.

Got it! Thanks Luke!

@Luke Mercado isn’t our call on the 29th?

Ooph, you’re right

Luke is too excited about Airbyte 🙂

Hey Davin, glad to hear the DBT work is top of mind. I was thinking that would be a bit outside of my team’s capability. Expanding the generic HTTP request seems a bit more doable though. We don’t have control over the API itself and wrapping it in the manner suggested for points 3,4 & 5 would greatly reduce the value we gain from AirByte. That said, perhaps creating a new source capable of such things would be a worthwhile compromise. I’ll mull that over. Is https://github.com/airbytehq/airbyte/issues/2566 the issue I should watch to track the DBT work? Once that’s done, assuming it suits our purposes it’ll shift the calculus on using AirByte dramatically. I’d also love it if you could link the pagination issue so I can track that as well.

Yes, #2566 is the right issue to track to keep up with DBT work. I’ve created this issue to track the pagination work. Feel free to add missing details, if any!

@s this might be related to some of your future tickets?

yup, will take a look/respond on those tickets directly.

Thanks, I’ve commented on #2573 and will follow along on both.