@kapa.ai We don’t want to allow syncing for streams that don’t have a primary key. How can we enforce this? We’re provisioning resources with Terraform.

When provisioning Airbyte connections with Terraform, you can enforce the requirement for a primary key by ensuring that any stream configured with a deduplication sync mode always has the

primary_key

field set. In the Terraform

airbyte_connection

resource, the

primary_key

field is defined per stream and is required when

destination_sync_mode

is

*_dedup

(e.g.,

incremental_deduped_history

,

full_refresh_overwrite_deduped

). [Terraform connection resource] Here's what the relevant schema looks like:

resource "airbyte_connection" "example" {
  # ...
  configurations = {
    streams = [
      {
        name      = "your_stream_name"
        sync_mode = "incremental_deduped_history"
        primary_key = [["id"]]  # REQUIRED for *_dedup modes
      }
    ]
  }
}

Key points to enforce this in your Terraform setup: 1. Always use a dedup sync mode for streams where you want to require a primary key (e.g.,

incremental_deduped_history

or

full_refresh_overwrite_deduped

). The

primary_key

field is required for these modes unless already supplied by the source schema. [Terraform connection resource] 2. Explicitly set

primary_key

in your Terraform configuration for every stream. If the source doesn't define a primary key, you can specify a user-defined one — including composite primary keys using multiple fields (e.g.,

[["field1"], ["field2"]]

). [Incremental Append+Deduped] 3. Avoid non-dedup sync modes (like

incremental_append

or

full_refresh_append

) for streams that must have a primary key, since those modes do not enforce the

primary_key

requirement. By structuring your Terraform configurations to only use dedup sync modes and always explicitly providing

primary_key

, you can effectively prevent streams without a primary key from being synced. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

The knowledge sources provided do not contain enough information to definitively answer whether Airbyte has built-in catalog validation that rejects a configured stream with both an empty

source_defined_primary_key

and an empty user-supplied

primary_key

regardless of sync mode. What the knowledge sources do confirm is: • The Airbyte protocol specifies that

primary_key

is required when

destination_sync_mode

is

*_dedup

. For non-dedup modes (e.g.,

append

,

overwrite

), the

primary_key

field is ignored. [Airbyte Protocol] • There is a

containsAtLeastOneNonNullPk

validation in the platform worker (

BasicAirbyteMessageValidator

) that checks records at runtime — but this is a record-level validator, not a catalog-level validator that rejects stream configuration upfront. [GitHub Issue #67711] So, based on the available sources: • There is no evidence of a catalog-level validation that proactively rejects a stream configuration where both

source_defined_primary_key

and user-supplied

primary_key

are empty, regardless of sync mode. • The protocol only enforces

primary_key

presence for

*_dedup

destination sync modes. For a definitive answer, I'd recommend checking the Airbyte GitHub repository directly or the Airbyte community Slack/forum, as this level of internal validation detail is not covered in the provided knowledge sources. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

1. @kapa.ai *At what point in the sync lifecycle is the missing-PK error raised when

*_dedup

is used without a

primary_key

?* Is it at: ◦ Catalog discovery time? ◦ Connection save time (Terraform apply)? ◦ First sync attempt? ◦ Per-record write time? ◦ We need to know if a misconfigured connection can be*applied via Terraform*but only fail later (operator pages out at 3 AM vs. fails fast at deploy time).

The knowledge sources provided do not contain enough information to definitively answer at which specific point in the sync lifecycle the missing-PK error is raised when

*_dedup

is used without a

primary_key

. What the knowledge sources do confirm is limited to: • The Airbyte protocol specification states that

primary_key

is required when

destination_sync_mode

is

*_dedup

, but it does not describe when or where this requirement is enforced at runtime. [Airbyte Protocol] • The Terraform provider marks

primary_key

as optional (not required) in the resource schema, with a note that it is "REQUIRED if

destination_sync_mode

is

*_dedup

unless it is already supplied by the source schema." This strongly implies the Terraform provider itself does not enforce this at

terraform apply

time — it is documented as a semantic requirement, not a schema-level constraint. [Terraform connection resource] • One GitHub PR (

#74270

) shows that for the Iceberg destination, a missing-PK error was previously raised as a cryptic

IllegalStateException

at write time (per-record/sync time), and the fix added an earlier

ConfigErrorException

— but this is specific to the Iceberg destination, not a general platform behavior. [GitHub PR #74270] Practical implication based on available evidence: The Terraform provider will likely not fail at

terraform apply

time for a missing PK on a dedup stream. The failure would occur later — most likely at sync time — but the exact moment (connection save vs. first sync vs. per-record write) is not confirmed by the available knowledge sources. For a definitive answer on this operational risk, I'd recommend testing directly or consulting the Airbyte GitHub repository or community Slack, as this level of lifecycle detail is not covered in the provided sources. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Does the Airbyte Terraform provider call

discover

on every

terraform apply

, or only on the first one (and reuse a cached catalog thereafter)?

The knowledge sources do not contain enough information to definitively answer whether the Airbyte Terraform provider calls

discover

on every

terraform apply

or only on the first one. What the knowledge sources do touch on, tangentially, is how Airbyte handles catalog discovery caching in general: • Connector acceptance tests cache discovered catalogs by default for performance and reuse the same discovered catalog through all tests. This can be disabled with

cached_discovered_catalog: False

. [Acceptance Tests Reference] However, this is specific to the connector testing framework, not the Terraform provider. •

DISCOVER_REFRESH_WINDOW_MINUTES

is a platform-level setting that controls how often discover is re-run during syncs. A GitHub issue notes that in v1.8.0+, discover runs at least once (on the first sync) and subsequent syncs reuse the cached result when this is set to

0

. [GitHub Issue #72306] Again, this is about sync-time behavior, not Terraform provider behavior. • The Terraform provider documentation does not mention any

discover

call being triggered during

terraform apply

. [Terraform Getting Started] Bottom line: The knowledge sources do not describe the Terraform provider triggering

discover

at all during

terraform apply

. The provider interacts with Airbyte via the Airbyte API, and whether that API call internally triggers a discover is not addressed in the available sources. For a definitive answer, I'd recommend checking the Airbyte Terraform provider source code or the Airbyte community Slack. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.