<@U04S8PLJWMT> I created a custom API source using...
# replication-ask-ai
a
@kapa.ai I created a custom API source using connector builder, , and exported manifest json using API How can i use that manifest to recreate that source on another Airbyte core local instance (v2.0.1) using terraform provider (v0.13.0)? what terraform resource template should I use, "airbyte_source_custom", "airbyte_declarative_source_definition" or other? how should be the airbyte/terraform/modules/sources/main.tf file?
k
The knowledge sources don’t describe, step‑by‑step, how to take a Connector Builder manifest JSON and recreate that exact custom API source on another instance via Terraform, so the answer below is based only on what is documented and there are gaps. From the docs, there are two relevant Terraform patterns: 1. Custom / JSON‑based sources via
airbyte_source_custom
This is the generic, weakly‑typed resource that takes a JSON configuration and can be used with any connector (including custom ones). [[Typing docs](https://docs.airbyte.com/developers/terraform-documentation#typing)] Example pattern (from the Stripe tutorial, but the structure is what matters): [[Step 2 source](https://docs.airbyte.com/developers/terraform-documentation#step-2-create-a-source)]
Copy code
hcl    resource "airbyte_source_custom" "my_source" {      configuration = jsonencode({        # For public / marketplace connectors:        # "source_type" : "stripe",         # For your own custom connector, you instead reference it by definition_id        # and omit source_type:        # (this ID comes from the custom connector definition in the target instance)        "account_id"           : "YOUR_ACCOUNT_ID",        "client_secret"        : "YOUR_SECRET",        "start_date"           : "2023-07-01T00:00:00Z",        "lookback_window_days" : 0,        "slice_range"          : 365      })       name         = "My Custom API Source"      workspace_id = var.workspace_id       # For custom connectors only:      # definition_id = "e094cb9a-26de-4645-8761-65c0c425d1de"    }
For a custom connector, the docs say you must use
definition_id
instead of
source_type
. [[Step 2 source](https://docs.airbyte.com/developers/terraform-documentation#step-2-create-a-source)] 2. Declarative / YAML‑based custom connectors via new API & Terraform resources Release notes for 1.6 mention new API and Terraform support for creating custom connectors from a Connector Builder YAML manifest or Docker image. [[Custom connectors APIs](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] However, the provided docs do not show the Terraform HCL for
airbyte_declarative_source_definition
(or similar) nor how to feed a Builder manifest JSON directly into it. They only list the REST endpoints. Because of that: - I cannot confidently show a correct
airbyte_declarative_source_definition
example or confirm its exact usage with a Builder manifest JSON from the available sources. - The only fully documented Terraform pattern for sources is
airbyte_source_custom
(JSON config) and the connector‑specific strongly‑typed resources (e.g.
airbyte_source_stripe
), which don’t apply to your arbitrary custom API connector. [[Typing docs](https://docs.airbyte.com/developers/terraform-documentation#typing); [Step 2 source](https://docs.airbyte.com/developers/terraform-documentation#step-2-create-a-source)] ### What this means for your use case Based on the docs: - Resource to use for the source instance itself: Use
airbyte_source_custom
with a JSON
configuration
. [[Typing docs](https://docs.airbyte.com/developers/terraform-documentation#typing)] - But: to use
airbyte_source_custom
with a custom connector, you still need the custom source definition (the connector itself) to exist on the target instance, and you must know its
definition_id
. [[Step 2 source](https://docs.airbyte.com/developers/terraform-documentation#step-2-create-a-source)] - The docs do say you can create custom connectors from YAML/Docker via API and Terraform, but they don’t show the Terraform resource syntax, so I can’t reliably give you a
<http://main.tf|main.tf>
that recreates the definition from your exported manifest JSON. [[Custom connectors APIs](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] ### Minimal
airbyte/terraform/modules/sources/main.tf
pattern (what is documented) Within
airbyte/terraform/modules/sources/main.tf
, the documented pattern looks like:
Copy code
hcl resource "airbyte_source_custom" "my_custom_api_source" {   configuration = jsonencode({     # This JSON must match the connector’s config schema.     # For a custom connector, you omit "source_type" and just pass the fields.     # You would derive these fields from the config part of your manifest.     "api_key"      : var.my_api_key,     "base_url"     : "<https://api.example.com>",     "start_date"   : "2023-07-01T00:00:00Z"     # ... other fields from your connector’s config   })    name         = "My Custom API Source"   workspace_id = var.workspace_id    # This must be the definition_id of the custom connector   # already present on the target instance.   definition_id = var.custom_source_definition_id }
And in
<http://variables.tf|variables.tf>
for that module:
Copy code
hcl variable "workspace_id" {   type = string }  variable "custom_source_definition_id" {   type = string }  variable "my_api_key" {   type      = string   sensitive = true }
### Gaps / what the docs don’t cover - How to directly take a Connector Builder manifest JSON and: - create the custom source definition on the new instance via Terraform, and - then use that definition in
airbyte_source_custom
. - Exact Terraform syntax for
airbyte_declarative_source_definition
(or equivalent) and how to pass the manifest JSON/YAML into it. Given the current knowledge sources, I can’t provide a correct, end‑to‑end
<http://main.tf|main.tf>
that: 1. Creates the declarative source definition from your exported manifest, and 2. Instantiates a source from that definition, because the necessary Terraform examples are not present in the docs provided. If you need that full flow, you’ll likely have to: - Consult the Terraform provider reference directly for
airbyte_declarative_source_definition
(or similar) and its arguments, and/or - Use the documented REST endpoints to create the declarative source definition from your manifest, then use
airbyte_source_custom
in Terraform with the resulting
definition_id
. [[Custom connectors APIs](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)]