@kapa.ai How to handle column-oriented/ split format JSON data in Builder Sample data: { "columns": [ {"columnName": "Subtotal Level", "outputType": "Integer"}, {"columnName": "Legal Entity ID", "outputType": "Integer"} ], "data": [ [0, 3, "Strategic Investors Fund VI, L.P.", ...], [0, 4, "Strategic Investors Fund VI-A, L.P.", ...] ] }

The knowledge sources don’t describe a built‑in way in Connector Builder to “zip” a

columns

array together with a parallel

data

array and emit one record per row, so anything beyond simple array field-path selection would require custom logic outside the no‑code Builder. What is documented: • Connector Builder’s record selection can only traverse JSON structures via a field path, with support for wildcards like

*

to iterate over array elements (e.g.

data

,

*

,

record

to get each

record

object in an array). [Fields in arrays] • For more complex reshaping (like turning a matrix into row objects), community guidance is to either: ◦ store the whole response “as is” without parsing, or ◦ move to the low‑code/Python CDK and implement custom components/logic. [Issue parsing API] Applied to your example:

{
  "columns": [
    {"columnName": "Subtotal Level", "outputType": "Integer"},
    {"columnName": "Legal Entity ID", "outputType": "Integer"}
  ],
  "data": [
    [0, 3, "Strategic Investors Fund VI, L.P.", ...],
    [0, 4, "Strategic Investors Fund VI-A, L.P.", ...]
  ]
}

In pure Connector Builder you can: • Use a field path like

data

,

*

to emit each inner array as a record (you’ll get arrays, not named fields). [Fields in arrays] • Or set the field path to the root (or

[]

) to store the entire response (including

columns

and

data

) as a single object and reshape it downstream (e.g., with dbt), which is similar to the “store whole response without parsing” suggestion. [Issue parsing API] What you cannot do with current documented Builder capabilities: • Dynamically map

columns[i].columnName

onto

data[row][i]

to output records like:

{
      "Subtotal Level": 0,
      "Legal Entity ID": 3,
      ...
    }

There is no documented transformation in Builder that joins two arrays in this way. If you need that fully normalized row‑per‑record structure inside the connector, the documented path is to: • Switch to the low‑code/Python CDK and implement a custom extractor/transformer that: ◦ reads

columns

to get the field names, and ◦ zips each

data

row with those names into an object, then ◦ yields those objects as records. [Issue parsing API] So, with the current docs: • Builder only: either emit

data

rows as arrays, or store the whole response and reshape later. • For proper column‑oriented → row‑oriented mapping: use a custom component in low‑code/Python CDK. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai In airbyte cloud?

The knowledge sources indicate that handling this kind of column‑oriented JSON (where you must “zip”

columns

with

data

rows) requires custom components / custom logic, not just the standard record selector. [Issue parsing API] They also state that: • One option is to store the whole response without parsing and reshape it downstream. • Another is to use the Low‑Code CDK with custom components or the Python CDK to implement the transformation you need. [Issue parsing API] However, the docs and forum thread do not clearly state whether these custom‑component / Low‑Code options are available in Airbyte Cloud specifically, versus only in self‑hosted deployments. The only explicit note about custom components mentions running the Builder with support for custom components on local deployments. [Nested data unwrapping] So, based on the provided sources: • In Cloud, using only the no‑code Connector Builder, you’re limited to: ◦ selecting

data

and emitting each inner array as a record, or ◦ storing the whole response and transforming it later. • Whether you can run custom components / Low‑Code CDK with custom extractors in Airbyte Cloud is not clearly documented in these sources; they only document that setup for local deployments, so I can’t confirm Cloud support from this material. [Nested data unwrapping] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Iam getting error for second sync for iceberg destination

Could you share more details about the error message you're seeing? That said, based on the knowledge sources, here are some known issues with the Iceberg destination that could cause errors on a second (incremental/dedup) sync: Known Iceberg Destination Issues 1. Dedup with Number-type primary keys If your primary key column has an Airbyte

Number

type, older versions of the connector would fail with a cryptic

IllegalStateException

from the Iceberg SDK. This happens because: •

Number

types map to Iceberg

DoubleType

, which Iceberg disallows as identifier fields (floating-point equality is unreliable). • The PK columns were silently excluded, leaving an empty identifier set, which caused the SDK to throw an unhelpful error at write time. A fix was added to surface a clear

ConfigErrorException

explaining the incompatibility. [fix(iceberg): dedup error] Workaround: If your PK is a

Number

type, consider switching to

append

or

overwrite

sync modes, or use a non-float primary key. [S3 Data Lake deduplication] 2. Dedup assumptions about primary keys The Iceberg destination (S3 Data Lake / GCS Data Lake) assumes: • The source never emits the same primary key twice in a single sync attempt, OR • If it does, records are always emitted in cursor order (oldest to newest). If these conditions aren't met, you may see inaccurate data. Known sources that violate this include Stripe and Monday. [S3 Data Lake deduplication; GCS Data Lake deduplication] If you can share the exact error message, I can help narrow down the cause further. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Here is the erroe Schema evolution for column "_airbyte_meta" between struct<59: sync_id: optional long, 60: changes: optional list<struct<62: field: optional string, 63: change: optional string, 64: reason: optional string>>> and struct<-740212846: sync_id: required long, -87793063: changes: required list<struct<1683807146: field: required string, -1218224811: change: required string, 967543765: reason: required string>>> is not allowed.

This error is related to Iceberg schema evolution on the

_airbyte_meta

column. Specifically, the error occurs because the field IDs embedded in the Iceberg schema for

_airbyte_meta

differ between the existing table (with positive field IDs like

59

,

60

,

62

, etc.) and the new schema being written (with negative/hashed field IDs like

-740212846

,

-87793063

, etc.). Iceberg treats these as incompatible structs and refuses the schema evolution. This is a known issue related to how Airbyte's Iceberg destination handles the

_airbyte_meta

metadata column. The

_airbyte_meta

field is a required Airbyte metadata column that contains

sync_id

and

changes

sub-fields. [S3 Data Lake output schema; GCS Data Lake output schema] The Iceberg destination documentation explicitly states that it never rewrites existing Iceberg data files, which means it can only handle specific schema changes. An incompatible struct field ID mismatch like this falls outside what the connector can automatically resolve. [S3 Data Lake output schema] How to Fix The documented options for handling unsupported schema changes in the Iceberg destination are: 1. Run a Refresh in Airbyte — This will rebuild the table from scratch with the correct schema. 2. Run a Clear in Airbyte — This will drop the existing data and re-sync from the beginning. 3. Manually edit your table schema in Iceberg directly — You could attempt to reconcile the field IDs manually, though this is complex. 4. Use Full Refresh - Overwrite sync mode — This handles unsupported schema changes automatically by overwriting the table each sync. [S3 Data Lake output schema; GCS Data Lake output schema] The simplest and most recommended path is to trigger a Refresh (removing existing records) from the Airbyte UI, which will recreate the table with the correct schema. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I updated my postgres to iceberg connection from full refresh overwrite to incremental append sync mode and iam getting the below error Failure in destination: Schema evolution for column "_airbyte_meta" between struct<27: sync_id: optional long, 28: changes: optional list<struct<30: field: optional string, 31: change: optional string, 32: reason: optional string>>> and struct<-594594512: sync_id: required long, 93739844: changes: required list<struct<-274296128: field: required string, 2067092581: change: required string, 1559849202: reason: required string>>> is not allowed. why is it and how to resolve it

Why This Error Occurs This is the same

_airbyte_meta

schema evolution error you encountered before, but now triggered by switching sync modes from Full Refresh Overwrite to Incremental Append. Here's what's happening: • Your existing Iceberg table was created under Full Refresh Overwrite mode. The

_airbyte_meta

column was written with specific Iceberg field IDs (e.g.,

27

,

28

,

30

,

31

,

32

) and with

optional

nullability. • When you switched to Incremental Append, Airbyte is now trying to write

_airbyte_meta

with a different set of field IDs (negative/hashed IDs like

-594594512

,

93739844

, etc.) and

required

nullability. • Iceberg treats these as incompatible struct schemas and refuses the evolution, since the field IDs don't match and the optionality changed. The Iceberg destination never rewrites existing data files, so it can only handle specific schema changes — and a struct field ID mismatch like this is not one of them. [S3 Data Lake output schema; GCS Data Lake output schema] How to Resolve The documented options are: 1. Run a Refresh (removing existing records) from the Airbyte UI — this rebuilds the table from scratch with the correct schema for the new sync mode. This is the simplest fix. 2. Run a Clear — drops existing data and re-syncs from the beginning. 3. Manually edit the Iceberg table schema to reconcile the field IDs — this is complex and error-prone. Since you intentionally switched to Incremental Append (and likely want to keep historical data going forward), the recommended path is to trigger a Refresh in Airbyte, which will recreate the table with the correct schema and then resume incremental syncs from there. [S3 Data Lake output schema; GCS Data Lake output schema] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.