Michal Lebida
02/07/2025, 11:58 AMNo-Code Connector builder
[https://docs.airbyte.com/connector-development/connector-builder-ui/overview], and I assumed that from the import/export YAML level, there would be access to possibly recreation and modification of it.
However, an attempt to use the Terraform provider [https://registry.terraform.io/providers/airbytehq/airbyte/latest/docs] showed that there are no such possibilities. There is only the resource "airbyte_source_custom," which requires that it be already created and does not allow modification. Is it possible to manage the connector from this level in the way I mentioned (automation and parameterization)?
The second attempt was to use PythonCDK [https://docs.airbyte.com/connector-development/cdk-python/]. However, from what I understand from the documentation, I then have to:
1. Create a connector according to the steps here: https://docs.airbyte.com/connector-development/tutorials/custom-python-connector/environment-setup
2. Publish the result using https://docs.airbyte.com/operator-guides/using-custom-connectors
3. Automate the process of uploading the image to Airbyte
Are these all the steps, and is there a defined path for PythonCDK to create my connector and reuse it on several environments?kapa.ai
02/07/2025, 11:59 AMairbyte_source_custom
resource you mentioned indeed requires the connector to be already created and doesn't allow for direct modification through Terraform.
According to the [Github Issue about Custom Connectors via API & Terraform](https://github.com/airbytehq/airbyte/issues/32744), there have been recent developments to improve this situation. The issue states that "Custom Connectors are available through Airbyte API & Terraform Provider for Cloud, OSS and Self-Managed." However, the exact implementation details and limitations are not fully clear from the provided knowledge sources.
2. Regarding the Python CDK approach:
Your understanding of the process is generally correct, but there are some important points to consider:
- The Python CDK is being phased out for new connector contributions. According to the [Connector Development Kit documentation](https://docs.airbyte.com/connector-development/cdk-python), "Over the next few months, the project will only accept connector contributions that are made using the Low-Code CDK or the Connector Builder."
- For your specific use case of creating a connector for REST endpoints, the Low-Code CDK or Connector Builder is recommended over the Python CDK.
- The steps you outlined (creating the connector, publishing it, and automating the upload) are generally correct, but may not be the most efficient approach given the current Airbyte recommendations.
3. Recommended approach:
Based on the information available, here's what I would recommend:
a. Use the Connector Builder to create your custom REST API connector. This aligns with Airbyte's current direction and should be suitable for most REST API scenarios.
b. For managing the connector across multiple environments, you might be able to use a combination of the Airbyte API and Terraform. According to a [comment on the Github issue](https://github.com/airbytehq/airbyte/issues/32744), you can:
- Get the sourceDefinitionId
of your custom connector from the Airbyte UI or API.
- Use this ID in your Terraform configuration to create and manage the connector across environments.
c. For parameterization and automation, you might need to combine Terraform with custom scripts that interact with the Airbyte API. The exact process isn't fully detailed in the provided knowledge sources, but it seems to be the direction Airbyte is moving towards.kapa.ai
02/07/2025, 11:59 AMairbyte_source_custom
resource you mentioned indeed requires the connector to be already created and doesn't allow for direct modification through Terraform. According to the [Github Issue about Custom Connectors via API & Terraform](https://github.com/airbytehq/airbyte/issues/32744), there have been recent developments to improve this situation. The issue states that "Custom Connectors are available through Airbyte API & Terraform Provider for Cloud, OSS and Self-Managed." However, the exact implementation details and limitations are not fully clear from the provided knowledge sources. 2. Regarding the Python CDK approach: Your understanding of the process is generally correct, but there are some important points to consider: - The Python CDK is being phased out for new connector contributions. According to the [Connector Development Kit documentation](https://docs.airbyte.com/connector-development/cdk-python), "Over the next few months, the project will only accept connector contributions that are made using the Low-Code CDK or the Connector Builder." - For your specific use case of creating a connector for REST endpoints, the Low-Code CDK or Connector Builder is recommended over the Python CDK. - The steps you outlined (creating the connector, publishing it, and automating the upload) are generally correct, but may not be the most efficient approach given the current Airbyte recommendations. 3. Recommended approach: Based on the information available, here's what I would recommend: a. Use the Connector Builder to create your custom REST API connector. This aligns with Airbyte's current direction and should be suitable for most REST API scenarios. b. For managing the connector across multiple environments, you might be able to use a combination of the Airbyte API and Terraform. According to a [comment on the Github issue](https://github.com/airbytehq/airbyte/issues/32744), you can: - Get the sourceDefinitionId
of your custom connector from the Airbyte UI or API. - Use this ID in your Terraform configuration to create and manage the connector across environments. c. For parameterization and automation, you might need to combine Terraform with custom scripts that interact with the Airbyte API. The exact process isn't fully detailed in the provided knowledge sources, but it seems to be the direction Airbyte is moving towards. It's worth noting that Airbyte is actively working on improving these processes. As mentioned in a [forum post about source control and multiple environments](https://discuss.airbyte.io/t/209), Airbyte is developing the Octavia CLI, which aims to provide better source control and resource creation capabilities. Given the evolving nature of these features, I recommend checking the latest Airbyte documentation or reaching out to their support channels for the most up-to-date information on managing custom connectors across multiple environments.Michal Lebida
02/07/2025, 12:24 PMc. For parameterization and automation, you might need to combine Terraform with custom scripts that interact with the Airbyte API. The exact process isn't fully detailed in the provided knowledge sources, but it seems to be the direction Airbyte is moving towards.