@kapa.ai I have a postgres -> Kafka CDC (JSON) connection, now I'm setting up a separate Kafka -> Postgres connection Second connection asks for "primary key" and "cursor", although it only offers to chosse from one field "value" (string), which isn't correct since it's a JSON string. How do I resolve this

Airbyte’s docs and the provided forum thread don’t describe this exact Kafka→Postgres JSON scenario in detail, so the guidance below is partially inferred; the knowledge sources do not explicitly document how to handle a JSON-encoded

value

field as the only available cursor/primary-key choice. From the docs: • A cursor is the field used to determine which records are new/updated since the last sync. [Cursor concept] • For many connectors, you can only choose from fields that the source exposes in the stream schema; source-defined cursors/PKs are auto-selected and not editable in the UI. [Configuring schema] • CDC sources often use an internal, source-defined cursor (e.g., replication log position) rather than a column, so no “cursor field” is shown. [CDC cursor forum] In your Kafka→Postgres connection, the Kafka source is exposing a single field

value

(string). Airbyte therefore only lets you pick

value

as cursor/primary key, because that is the only field it “sees” in the stream schema. The docs do not describe a built‑in way to tell Airbyte “inside this JSON string there are proper fields like id, updated_at; use those instead.” Given the limitations in the knowledge sources: • I’m not seeing any documented option to: ◦ Parse the JSON inside

value

at the source level, or ◦ Define a nested cursor/primary key inside that JSON for the Kafka connector via the UI. So, based only on the docs you provided: 1. Airbyte expects the cursor/primary key to be an actual field in the stream schema, not a subfield inside an opaque JSON string. [Cursor concept; Configuring schema] 2. If the Kafka connector exposes only

value

as a string, Airbyte will only offer

value

as cursor/PK, which is what you’re seeing. 3. The knowledge sources do not explain how to change the Kafka stream schema so that the JSON is broken out into separate fields, nor how to configure a cursor/PK on those inner JSON properties. Because of that, I can’t give you a documented, step‑by‑step fix from these sources alone. The most that can be concluded from the docs is: • To have a meaningful cursor/primary key in the Kafka→Postgres connection, the Kafka source stream would need to expose proper fields (e.g.,

id

,

updated_at

) instead of a single JSON

value

string. • With the current schema (only

value

), any cursor/PK you choose will necessarily be that

value

field, which is not semantically correct for your use case. If you can adjust how messages are written to Kafka (e.g., so the connector exposes structured fields instead of a single JSON string), that would align with how Airbyte expects cursors/PKs to be defined, but this approach is not explicitly documented in the provided sources. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.