Hi folks, I'm looking into using Airbyte to captur...
# help-connector-developments
Steven Murphy
06/07/2023, 9:18 AMHi folks, I'm looking into using Airbyte to capture data from an Async API. As in, I submit a job request to the API, then either poll that API to check when the job is completed, or have that API send it's result to a webhook/callback.
Does it seem like a fit for Airbyte? I've found a couple of posts along similar lines they didn't get much engagement. It appears Airbyte is centered around the assumption that the web APIs it hits will return responses synchronously
https://discuss.airbyte.io/t/handling-async-apis/3501
https://github.com/airbytehq/airbyte/issues/2787
Thank you
k
kapa.ai
06/07/2023, 9:18 AMA message from kapa.ai
kapa.ai
06/07/2023, 9:19 AMAirbyte can handle async APIs to some extent. In the [forum discussion](https://discuss.airbyte.io/t/3501) you mentioned, the user suggested using substreams and slices to handle the async API. However, the support response indicated that it might not work perfectly for all cases. To handle async APIs, you can experiment with substreams and slices, but be aware that it might not cover all scenarios. Airbyte is indeed more focused on synchronous API interactions, so handling async APIs might require some customizations in your connector implementation. If you decide to give it a try, you can refer to the [tutorial on building a custom Airbyte source connector](https://airbyte.com/tutorials/extract-data-from-the-webflow-api) for guidance on creating your own connector to handle the specific async API requirements.
j
Joe Reuter (Airbyte)
06/07/2023, 9:56 AMThe interface connectors have to implement is very generic and it can definitely handle this case of API.
Joe Reuter (Airbyte)
06/07/2023, 9:57 AMWhat's true is that there are not as many helpers as for sync APIs so it's a little more manual work to get it done
Joe Reuter (Airbyte)
06/07/2023, 9:57 AMYou can check out the amazon seller partner connector for example: https://github.com/airbytehq/airbyte/blob/4e86231e7f078b962a1340fce34a5df882af8c43[…]e-amazon-seller-partner/source_amazon_seller_partner/streams.py
s
Steven Murphy
06/07/2023, 9:58 AMI will check that out, thanks very much
j
Joe Reuter (Airbyte)
06/07/2023, 9:58 AMThe low code CDK / connector builder doesn't support this yet, but it's on our radar.
Joe Reuter (Airbyte)
06/07/2023, 10:00 AMJust re-read your original question and what's not supported yet is the "the API is calling you back via webhook" part - so it would need to be polling
s
Steven Murphy
06/07/2023, 10:07 AMThe API does have the option of us polling it for a response, though utilising the APIs webhook/callback capability would be preferable. I'd be submitting thousands of job requests and would like to avoid hammering the API with polling job status requests. (and getting 429'd or using up compute/memory unneccessarily)
I could roll my own orchestration, job tracking etc. My concern was that I'd be shoehorning this into Airbyte when it might not be appropriate. Though I do have other simpler cases where we hit sync endpoints, for which I almost certainly plan to use Airbyte
j
Joe Reuter (Airbyte)
06/07/2023, 10:11 AMMakes sense, thanks for the context. There are no immediate plans to add a webhook channel to the Airbyte platform for the source to report back, I added a note about your use case though
👍 1
Joe Reuter (Airbyte)
06/07/2023, 10:12 AMWhat's imaginable is to solve this problem outside of Airbyte - have a simple server that receives webhooks and stores the invocations and have the connector poll this server in a cheap way
Joe Reuter (Airbyte)
06/07/2023, 10:12 AMBut I realize that it complicates things quite a bit on the operational side
s
Steven Murphy
06/07/2023, 10:17 AMWhat's imaginable is to solve this problem outside of Airbyte - have a simple server that receives webhooks and stores the invocations and have the connector poll this server in a cheap wayThat's what I'm leaning towards at the moment. Assuming I did, do you think the use of Airbyte would still be appropriate for polling that server and handling the responses? I'm thinking (with my next to zero Airbyte experience), I'd be shoehorning Airbyte in.
j
Joe Reuter (Airbyte)
06/07/2023, 10:19 AMYeah, I think it would be appropriate - there's a bunch of connectors polling for reports to complete already.
s
Steven Murphy
06/07/2023, 10:20 AMWould that amazon connector you pointed me to earlier be a good place to start?
j
Joe Reuter (Airbyte)
06/07/2023, 10:21 AMYes, there's also the facebook marketing manager that's similar: https://github.com/airbytehq/airbyte/blob/2e099acc52aeddc172c5ee66ed98426c69449a4a[…]arketing/source_facebook_marketing/streams/async_job_manager.py
Joe Reuter (Airbyte)
06/07/2023, 10:22 AMWe have plans to add support for this kind of connector (request report, poll for completion, download) to low code cdk / the connector builder in the mid term
s
Steven Murphy
06/07/2023, 10:22 AMI'm curious, those github links you gave me, are those considered low code?
j
Joe Reuter (Airbyte)
06/07/2023, 10:23 AMNo, those are using the Python CDK - see this documentation page for the different ways to build a connector: https://docs.airbyte.com/connector-development/
s
Steven Murphy
06/07/2023, 10:25 AMCool, appreciate the info thank you. It's given me some food for thought.