Hi everyone, I am working on a source connector th...
# help-connector-development
t
Hi everyone, I am working on a source connector that pulls data from a system with a varying amount of streams. I created a generator function to create the stream objects and overwrote the
get_json_schema
function to fetch the correct json - which seems to work fine. I am now having issues to dynamically adjust the catalog. I checked the
streams
list in the
def streams(self, config: Mapping[str, Any]) -> List[Stream]:
function and it contains the object for the generated streams. This might be just an issue when testing it in the CLI and specifying a catalog for testing through the
--catalog integration_tests/configured_catalog.json
parameter but I am wondering if I missed anything. Thanks in advance for any help!
k
A message from kapa.ai
It seems like you're on the right track with dynamically generating streams and their schemas. To dynamically adjust the catalog, you can follow the approach mentioned in the [dynamic schemas documentation](https://docs.airbyte.com/connector-development/cdk-python/schemas#dynamic-schemas). You can also refer to the [Smartsheets source](https://docs.airbyte.com/integrations/sources/smartsheets) as an example of a connector with a dynamic schema based on the spreadsheet structure. When testing with the CLI, you might need to adjust the
configured_catalog.json
file to match the dynamic streams you're generating. The
configured_catalog.json
file is necessary for testing as it contains information about sync modes, cursor fields, etc., for each stream in the connector. You can refer to the [Python CDK Speedrun tutorial](https://docs.airbyte.com/connector-development/tutorials/cdk-speedrun) for more information on how to use the CLI for testing. If you're using the
high
test strictness level, you might need to adjust your test configuration as well. In this level, the
basic_read
test is run on a configured catalog created from the discovered catalog, and declaring
configured_catalog_path
in the test configuration is not allowed. You can find more information about this in the [Acceptance Tests Reference](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference). Make sure to review your implementation and compare it with the examples provided in the extracts to ensure you haven't missed anything.