Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Question, how do i make sure pagination work for a http source?

Hi <@U01CQT00Q72> the http source does not currently offer pagination 

So the only method is that i would create a custom connector ?

Do you have any tips on how to get started with that?

Sounds like it. Which API would you be pulling data from?

This page: <https://docs.airbyte.io/contributing-to-airbyte/building-new-connector> contains the basic requirements needed to spin up a connector
This page: <https://docs.airbyte.io/tutorials/toy-connector> runs through an example of building a connector start to finish. It’s not 100% how you’d build a connector since it doesn’t follow best practices (just to make the tutorial easier to follow) e.g: it puts everything in one file and uses only the standard python lib. But we’re more than happy to provide all the needed support/Q&amp;A to make any connector you build follow these best practices.

Which language would you be using to build this? FWIW I’d be more than happy to do a pairing session to get started/support as needed :slightly_smiling_face:

Hi Shrif! HNY! I would build the connector in Python, i would love to have a pairing session!

I would love to hear how i can manage secrets! Most of the api's i will want to connect with use Oauth2.0 in some form.

Lets schedule something, do you use google calendar?

<@U01A6TEQXRU> please book a time in my calendar via Calendly

<https://calendly.com/thomas-gcompany/30min>

Hi Thomas, the first viable time I could find was 7 days from now (next tuesday) at 7:30am PST

LMK if something at around the same time opens up this week for you