I wrote a custom connector to pull data from an http api as a source. I used the requests library. This connector has to go 4 levels deep: For each record in level 1, it calls each record in level 2 and for each record of level 2, it calls each record of level 3, etc... Because of this, it takes probably an hour to go through all the calls that it has to make, even though it'll only bring back about 1,200 records and 127 MB of data. I would like to think that if I could make the api calls asynchronous, that it would speed up the process dramatically. I was thinking about using the httpx library. Maybe I was doing it wrong, but I tried to override the "read_records" method and create the async client there, but couldn't figure out how to get it to return the necessary generator for the rest of the cdk to run with. Has anyone done something like that before or is there any documentation that I missed that someone can point me to?

@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2456 to ensure timely resolution!

Hey @Craig Condie, at the moment we don’t support async streams. It’s a feature we want to add to our cdk. In the past, users have used the javascript cdk which to my knowledge uses async operations. Hope this helps.

Thanks Saj for letting me know!