What's the general developer workflow when using airbyte in combination of its terraform modules? How do I know what attributes to put in a connection terraform resource? If I were to use airbyte UI to make a connection first, how do I export it back into a terraform resource? @kapa.ai

1. General Developer Workflow with the Airbyte Terraform Provider The typical developer workflow involves setting up provider credentials, defining sources and destinations, creating connections between them, and testing via Terraform CLI [Getting Started; Aggregating Data]: 1. Configure Provider and Variables: Set up

main.tf

with the

airbytehq/airbyte

provider and configure credentials (such as

client_id

,

client_secret

, and optionally

server_url

or

workspace_id

) in

variables.tf

or environment variables [Getting Started]. 2. Define Sources and Destinations: Use the

airbyte_source

and

airbyte_destination

resources. You can pair them with the

airbyte_connector_configuration

data source to validate configurations against the connector schema and separate secrets [Getting Started; airbyte_connector_configuration]. 3. Define Connections: Link sources and destinations using the

airbyte_connection

resource [Create a Connection]. 4. Initialize and Plan: Run

terraform init

to download provider plugins, then

terraform plan

to validate the syntax and schema against connector specifications [Getting Started; airbyte_connector_configuration]. 5. Apply Configuration: Run

terraform apply

to create or update the Airbyte resources [Getting Started]. 6. Verify: Confirm the created connectors and connections in the Airbyte UI [Setting Up Connectors]. 2. Identifying Attributes for an airbyte_connection Resource To configure an

airbyte_connection

, the resource schema specifies the required and optional attributes [airbyte_connection Schema]: • Required Attributes: ◦ `source_id`: The ID of the source (e.g.,

airbyte_source.postgres.source_id

). ◦ `destination_id`: The ID of the destination (e.g.,

airbyte_destination.bigquery.destination_id

). • Optional Attributes: ◦ `name`: Connection name. ◦ `schedule`: Sync scheduling (such as

cron

expressions and

schedule_type

). ◦ `status`: Set connection state (

active

,

inactive

,

deprecated

, or

locked

). ◦

namespace_definition

& `namespace_format`: Location rules for data storage in the destination (

source

,

destination

, or

custom_format

). ◦ `prefix`: String prefix prepended to destination stream names. ◦ `non_breaking_schema_updates_behavior`: Handling for upstream schema updates (

ignore

,

disable_connection

,

propagate_columns

,

propagate_fully

). ◦ `configurations`: Detailed stream settings, including

streams

list, sync modes (

sync_mode

), cursor fields, primary keys, and field transformations/mappers [airbyte_connection Example]. ◦ `tags`: Workspace tags attached to the connection. 3. Exporting a UI-Created Connection into Terraform The documentation does not provide an automated tool to directly export a UI-created connection into full Terraform HCL code. However, you can bring an existing connection created in the UI under Terraform management using one of the following methods: Method A: Using Terraform Import You can declare the resource block in your

.tf

file and import the existing connection by its connection ID [airbyte_connection Import]. • *Terraform 1.5+

import

block:*import { to = airbyte_connection.my_connection id = "your-connection-id" } • *CLI Import:*terraform import airbyte_connection.my_connection "your-connection-id" Method B: Reading Configuration with the Data Source You can query the settings of an existing UI-created connection using the

airbyte_connection

data source by passing its

connection_id

to inspect its stream configurations, schedules, and namespace settings [airbyte_connection Data Source]:

data "airbyte_connection" "existing_connection" {
  connection_id = "<your-connection-id>"
}

👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.