I need to create a custom connector that will allo...
# ask-ai
m
I need to create a custom connector that will allow me to download data from REST endpoints. I want to manage this connector via IaC (Terraform) or another automated method (many environments). I managed to make an example using the
No-Code Connector builder
[https://docs.airbyte.com/connector-development/connector-builder-ui/overview], and I assumed that from the import/export YAML level, there would be access to possibly recreation and modification of it. However, an attempt to use the Terraform provider [https://registry.terraform.io/providers/airbytehq/airbyte/latest/docs] showed that there are no such possibilities. There is only the resource "airbyte_source_custom," which requires that it be already created and does not allow modification. Is it possible to manage the connector from this level in the way I mentioned (automation and parameterization)? The second attempt was to use PythonCDK [https://docs.airbyte.com/connector-development/cdk-python/]. However, from what I understand from the documentation, I then have to: 1. Create a connector according to the steps here: https://docs.airbyte.com/connector-development/tutorials/custom-python-connector/environment-setup 2. Publish the result using https://docs.airbyte.com/operator-guides/using-custom-connectors 3. Automate the process of uploading the image to Airbyte Are these all the steps, and is there a defined path for PythonCDK to create my connector and reuse it on several environments?
k
Based on the knowledge sources provided, I can offer some insights into your questions about creating and managing custom connectors in Airbyte. Let me address your points: 1. Regarding the No-Code Connector Builder and Terraform: You're correct that the Connector Builder allows you to create custom connectors through a UI. However, the current Terraform provider for Airbyte has limited capabilities for managing custom connectors. The
airbyte_source_custom
resource you mentioned indeed requires the connector to be already created and doesn't allow for direct modification through Terraform. According to the [Github Issue about Custom Connectors via API & Terraform](https://github.com/airbytehq/airbyte/issues/32744), there have been recent developments to improve this situation. The issue states that "Custom Connectors are available through Airbyte API & Terraform Provider for Cloud, OSS and Self-Managed." However, the exact implementation details and limitations are not fully clear from the provided knowledge sources. 2. Regarding the Python CDK approach: Your understanding of the process is generally correct, but there are some important points to consider: - The Python CDK is being phased out for new connector contributions. According to the [Connector Development Kit documentation](https://docs.airbyte.com/connector-development/cdk-python), "Over the next few months, the project will only accept connector contributions that are made using the Low-Code CDK or the Connector Builder." - For your specific use case of creating a connector for REST endpoints, the Low-Code CDK or Connector Builder is recommended over the Python CDK. - The steps you outlined (creating the connector, publishing it, and automating the upload) are generally correct, but may not be the most efficient approach given the current Airbyte recommendations. 3. Recommended approach: Based on the information available, here's what I would recommend: a. Use the Connector Builder to create your custom REST API connector. This aligns with Airbyte's current direction and should be suitable for most REST API scenarios. b. For managing the connector across multiple environments, you might be able to use a combination of the Airbyte API and Terraform. According to a [comment on the Github issue](https://github.com/airbytehq/airbyte/issues/32744), you can: - Get the
sourceDefinitionId
of your custom connector from the Airbyte UI or API. - Use this ID in your Terraform configuration to create and manage the connector across environments. c. For parameterization and automation, you might need to combine Terraform with custom scripts that interact with the Airbyte API. The exact process isn't fully detailed in the provided knowledge sources, but it seems to be the direction Airbyte is moving towards.
Based on the knowledge sources provided, I can offer some insights into your questions about creating and managing custom connectors in Airbyte. Let me address your points: 1. Regarding the No-Code Connector Builder and Terraform: You're correct that the Connector Builder allows you to create custom connectors through a UI. However, the current Terraform provider for Airbyte has limited capabilities for managing custom connectors. The
airbyte_source_custom
resource you mentioned indeed requires the connector to be already created and doesn't allow for direct modification through Terraform. According to the [Github Issue about Custom Connectors via API & Terraform](https://github.com/airbytehq/airbyte/issues/32744), there have been recent developments to improve this situation. The issue states that "Custom Connectors are available through Airbyte API & Terraform Provider for Cloud, OSS and Self-Managed." However, the exact implementation details and limitations are not fully clear from the provided knowledge sources. 2. Regarding the Python CDK approach: Your understanding of the process is generally correct, but there are some important points to consider: - The Python CDK is being phased out for new connector contributions. According to the [Connector Development Kit documentation](https://docs.airbyte.com/connector-development/cdk-python), "Over the next few months, the project will only accept connector contributions that are made using the Low-Code CDK or the Connector Builder." - For your specific use case of creating a connector for REST endpoints, the Low-Code CDK or Connector Builder is recommended over the Python CDK. - The steps you outlined (creating the connector, publishing it, and automating the upload) are generally correct, but may not be the most efficient approach given the current Airbyte recommendations. 3. Recommended approach: Based on the information available, here's what I would recommend: a. Use the Connector Builder to create your custom REST API connector. This aligns with Airbyte's current direction and should be suitable for most REST API scenarios. b. For managing the connector across multiple environments, you might be able to use a combination of the Airbyte API and Terraform. According to a [comment on the Github issue](https://github.com/airbytehq/airbyte/issues/32744), you can: - Get the
sourceDefinitionId
of your custom connector from the Airbyte UI or API. - Use this ID in your Terraform configuration to create and manage the connector across environments. c. For parameterization and automation, you might need to combine Terraform with custom scripts that interact with the Airbyte API. The exact process isn't fully detailed in the provided knowledge sources, but it seems to be the direction Airbyte is moving towards. It's worth noting that Airbyte is actively working on improving these processes. As mentioned in a [forum post about source control and multiple environments](https://discuss.airbyte.io/t/209), Airbyte is developing the Octavia CLI, which aims to provide better source control and resource creation capabilities. Given the evolving nature of these features, I recommend checking the latest Airbyte documentation or reaching out to their support channels for the most up-to-date information on managing custom connectors across multiple environments.
m
Can you describe to me the possibilities of development in the direction described in this point?
Copy code
c. For parameterization and automation, you might need to combine Terraform with custom scripts that interact with the Airbyte API. The exact process isn't fully detailed in the provided knowledge sources, but it seems to be the direction Airbyte is moving towards.