<@U04S8PLJWMT> I wish to use terraform to create a...
# ask-ai
j
@kapa.ai I wish to use terraform to create a custom source connector, I have an exported builder yaml file.
k
You can manage custom connectors via Terraform, but the detailed HCL examples for using a Builder YAML manifest are not included in the provided knowledge. I’ll outline what is clearly documented and where the gaps are. 1. Terraform provider supports custom connectors Airbyte’s Terraform provider has specific resources for custom sources and destinations, which you then reference from
airbyte_source_custom
/
airbyte_destination_custom
using
definition_id
.[\[Strong/weak typing\]](https://docs.airbyte.com/developers/terraform-documentation#typing) For a custom source instance (once the definition exists), the pattern is:
Copy code
hcl    resource "airbyte_source_custom" "my_custom_source" {      configuration = jsonencode({        # your connector-specific config here        # e.g. api_key, base_url, etc.      })      name         = "My Custom Source"      workspace_id = var.workspace_id      definition_id = "YOUR_CUSTOM_SOURCE_DEFINITION_ID"    }
You obtain
definition_id
from the URL of the custom connector in the UI.[\[Create source\]](https://docs.airbyte.com/developers/terraform-documentation#step-2-create-a-source) 2. Creating the custom connector definition from YAML (Builder) Airbyte 1.6 introduced API and Terraform support for creating custom connectors using a Connector Builder YAML manifest or Docker image.[\[1.6 release notes\]](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform) The release notes list new API endpoints for YAML source definitions (declarative sources), e.g.: -
POST /workspaces/{workspaceId}/definitions/declarative_sources/
– Create YAML source definitions -
GET /workspaces/{workspaceId}/definitions/declarative_sources/{definitionId}
– Get a YAML source definition …and similar CRUD endpoints for Docker-based source and destination definitions.[\[1.6 release notes\]](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform) The notes state that there are corresponding Terraform resources (“new endpoints and resources you can use to create and update custom connectors using a Connector Builder YAML manifest or Docker image”) and refer you to the Terraform registry for details.[\[1.6 release notes\]](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform) However, the provided knowledge does not show the exact Terraform resource names or an example of how to point a resource at your exported Builder YAML file. It just points to the registry docs for that detail, which are outside what I can quote here. 3. What you can confidently do from the docs you have - Use the API endpoints above directly to create a custom YAML-based source definition from your exported Builder YAML manifest, then: - Use
airbyte_source_custom
in Terraform, referencing the
definition_id
of the custom source definition.[\[Create source\]](https://docs.airbyte.com/developers/terraform-documentation#step-2-create-a-source) 4. Where to look next Because the precise Terraform syntax for “YAML-based source definition from Builder export” is not present in the included documents, the knowledge sources do not contain enough information to show the exact Terraform code you need. The release notes explicitly direct you to the Terraform provider docs for those new resources.[\[1.6 release notes\]](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform) To finish your setup, you’ll need to: 1. Open the Airbyte Terraform registry docs for the custom definition resources referenced in the release notes. 2. Use the resource for declarative (YAML) source definitions, supplying your exported Builder YAML as the manifest. 3. Then, in the same Terraform project, define an
airbyte_source_custom
using the resulting
definition_id
. If you share the specific Terraform resource snippet from the registry you’re planning to use, I can help you wire it together with
airbyte_source_custom
based on the patterns in these docs.
A message from kapa.ai
You’re on the right track—Terraform can manage custom connectors created from a Builder YAML, but the knowledge here is missing the exact Terraform resource syntax for wiring the YAML file itself. From the docs we have: 1. Airbyte now supports custom connectors from Builder YAML via API and Terraform Airbyte 1.6 introduced API endpoints and Terraform resources that let you create and update custom connectors from a Connector Builder YAML manifest or Docker image.[[1.6 release notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] For YAML (declarative) source definitions, the available API endpoints include: -
POST /workspaces/{workspaceId}/definitions/declarative_sources/
– create YAML source definitions -
GET /workspaces/{workspaceId}/definitions/declarative_sources/{definitionId}
– get one YAML definition - plus list/update/delete variants.[[1.6 release notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] The same release notes say there are matching Terraform resources for this, and point you to the Terraform registry, but those specific resource names and HCL examples are not shown in the provided materials.[[1.6 release notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] 2. Creating a source instance in Terraform once the custom definition exists Once your custom connector definition (from the YAML) exists in Airbyte, you can create source instances with Terraform using
airbyte_source_custom
, referencing the custom definition by `definition_id`:
Copy code
hcl    resource "airbyte_source_custom" "my_custom_source" {      configuration = jsonencode({        # your connector’s config fields here        # e.g. api_key, base_url, etc.      })      name          = "My Custom Source"      workspace_id  = var.workspace_id      definition_id = "YOUR_CUSTOM_SOURCE_DEFINITION_ID"    }
The docs show this pattern for a custom Stripe connector, where
definition_id
comes from the URL of the custom connector in the Airbyte UI.[[Terraform create source](https://docs.airbyte.com/developers/terraform-documentation#step-2-create-a-source)] 3. What’s missing / what you need to do The knowledge provided does not include: - The Terraform resource name or HCL example for “create a declarative (YAML) source definition from a Builder YAML file”. - How exactly to point that resource at the exported YAML file (e.g.,
file("...")
, field names, etc.). The docs explicitly say to use the Terraform registry for those new resources.[[1.6 release notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] Since that content isn’t included here, I can’t safely provide the exact Terraform block without guessing. 4. Practical path forward Based on the available docs, your high‑level workflow is: 1. Create the custom connector definition from the exported Builder YAML, using either: - The new API endpoint
POST /workspaces/{workspaceId}/definitions/declarative_sources/
, or - The corresponding Terraform resource documented in the Airbyte Terraform registry (required if you want this fully managed via Terraform).[[1.6 release notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] 2. Get the
definition_id
of that custom source definition (via UI URL or API). 3. Create the source instance via Terraform using
airbyte_source_custom
and that
definition_id
, as in the pattern above.[[Terraform create source](https://docs.airbyte.com/developers/terraform-documentation#step-2-create-a-source)] Because the included knowledge does not show the Terraform resource that directly consumes the Builder YAML file, I’m unable to give you a complete HCL example for that part without making it up. You’ll need to open the Airbyte Terraform provider docs on the registry and look for the resource that manages declarative (YAML) source definitions, then combine it with
airbyte_source_custom
as shown.