Hello everyone, I am about to develop a connector ...
# connector-development
g
Hello everyone, I am about to develop a connector for a Rest api that should fetch the data and then store them an sql database. Some endpoints require a parameter which is contained in another endpoint response. In details: • call and endpoint to retrieve all the ids • use those ids to call another endpoint What is the best way to tackle this kind of problem? Should I fetch the parent endpoint each time? Or should I get the ids from the sql table?
o
loading...
loading...
loading...
a
Hi @Giunio De Luca, leveraging the
use_cache
attribute + HttpSubStream is your way to go.
Feel free to check out this similar conversation
You can also get inspiration from the following connector that make use of substreams: • Github • GreenHouse • Harvest • Linnworks • Notion • Pinterest • OneSignal • BigCommerce • Stripe • SurveyMonkeys • YouTube
We also have section in our doc about nested streams and caching: https://docs.airbyte.com/connector-development/cdk-python/http-streams#nested-streams-and-caching
g
thanks you 🙂