hello everyone, I am creating a new connector for ...
# ask-community-for-troubleshootingk
Kevin Phan
07/28/2022, 2:19 PMhello everyone, I am creating a new connector for my Select Star source and in total from the api, there are 1750 tables, where the tables differ in col names and schema. Id want to ingest all those tables in S3 via airbyte. I am stuck on the discover method, more specifically, how would I define the discover method, and json schemas for all of these tables? Seems like I could have a function that generates json schema dynamically on the fly in order to be used for the airbyte catalog -> configured catalog. Maybe I am misunderstanding something? Thanks for the help. @Alex Marquardt (Airbyte) @Alexandre Airvault @ anyone else
a
Alex Marquardt (Airbyte)
07/28/2022, 4:13 PMThis tutorial shows how to dynamically generate schemas by querying an API (Webflow in the tutorial case) to pull back the schema - https://airbyte.com/tutorials/extract-data-from-the-webflow-api
If you select star source has a way of telling you the schema for each table, perhaps you could use a similar approach dynamically generate your schemas.
k
Kevin Phan
07/28/2022, 4:22 PMok sound good, so in general, every table that is being generated needs a corresponding schema for airbyte to reference in the discover method?
Kevin Phan
07/28/2022, 4:26 PMi am reading:
Copy code
This is a simple task with the Airbyte CDK. For each stream in our connector we'll need to:
Create a python class in source.py which extends HttpStream.
Place a <stream_name>.json file in the source_<name>/schemas/ directory. The name of the file should be the snake_case name of the stream whose schema it describes, and its contents should be the JsonSchema describing the output from that stream.Kevin Phan
07/28/2022, 4:27 PMby each
streama
Alex Marquardt (Airbyte)
07/28/2022, 6:25 PMHave a look in that article for the description of what a stream is.
3 Views