I have a question about dynamically generating streams/catalog. I have an MSSQL database source that the base MSSQL Connector cannot interact with. I have a means of auto generating schemas based on the table DDLs. I use this query within Discover and receive back a valid catalog. I am a little confused as to how Airbyte will identify/render the configured catalog. The documentation has you build a discover method, but there is also a need for a "Schemas" folder as well as a "configured_catalog.json" file. Do I still need to have a "configured_catalog.json" file and/or schema folder since I am auto generating the json at runtime? The CLI requires a catalog file to be passed along with "Read", yet I wasn't planning on having a physical file. I have been using the https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-azure-table code as a guide along with the Documentation. Also of note, they do not have a schema's folder at all either and their configured_catalog.json is essentially empty. Is there any reason to avoid imitating this approach?

Yeah, you can definitely imitate their approach if it suits your source data! There's some documentation on dynamic schemas if you didn't see yet: https://docs.airbyte.com/connector-development/cdk-python/schemas#dynamic-schemas The azure table source connector is an interesting one, it has a generic schema for all streams. If you haven't seen there's a note about why in the doc:

Azure Table storage is a service that stores non-relational structured data (also known as structured NoSQL data). There is no efficient way to read schema for the given table. We use

data

property to have all the properties for any given row.

• data - This property contain all values

• additionalProperties - This property denotes that all the values are in

data

property.

The

configured_catalog.json

file also contains configuration on sync mode, cursor field, etc for each stream in the connector (and it is used for testing) so it is still necessary, even if it looks empty. I found more examples of python connectors that don't use static schemas, for your reference: source-google-sheets, source-airtable, source-convex