Hey, I have a use-case to download ZIP files (with...
# help-connector-development
a
Hey, I have a use-case to download ZIP files (without extracting) from an HTTPS website. The current Files source connector does not support syncing zip files. So I guess the next step would be to create a custom connector. I've gone through the Python CDK docs & tutorial but would like to get some ideas from the community. • Can we modify the current Files connector to also ingest zip files? If not, what's the reason? • How to define a schema for this use case?
k
A message from kapa.ai
It's great that you've gone through the Python CDK docs and tutorial. To address your questions: 1. Modifying the current Files connector to ingest ZIP files might be possible, but it would require diving into the connector's code and making the necessary changes. However, creating a custom connector using the Airbyte Python CDK might be a more straightforward approach, as it provides a flexible way to implement a source connector tailored to your specific use case [source](https://docs.airbyte.com/connector-development/cdk-python/basic-concepts#the-source-class). 2. Defining a schema for your use case might be a bit different than the usual JSON schema, as you're dealing with ZIP files. You can start by creating a static schema that describes the structure of the ZIP files you want to download. However, if you need to dynamically define schemas, you can refer to the [schema docs](https://docs.airbyte.com/connector-development/cdk-python/full-refresh-stream#defining-the-streams-schema) for more information, although it's beyond the scope of the tutorial. To create a custom connector, you can follow these steps: 1. Generate an empty connector using the code generator by following the instructions in the [Getting Started](https://docs.airbyte.com/connector-development/cdk-python#getting-started) section. 2. Complete all the `TODO`s in the generated project directory, as mentioned in the [Connector Development Kit](https://docs.airbyte.com/connector-development/cdk-python) documentation. 3. Implement the necessary logic to download ZIP files from the HTTPS website in your custom connector. 4. Define the schema for your use case, as discussed earlier. For more guidance and support, you can join the [Airbyte community Slack Channel](https://airbyte.io/community) and participate in discussions on [Airbyte’s discourse](https://discuss.airbyte.io/).