I have a custom cvent connector in airbyte .. and ouput format is parquet and storage is GCS but I am getting Warning from destination: Something went wrong in the connector. See the logs for more details. See less tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field group is expected to be one of these: NULL, STRING. If this is a complex type, check if offending field (path: group) adheres to schema: {id=50bafb55-a0e7-41b0-8d7c-7b0f4ca9446d} @kapa.ai

JSON to Avro Conversion Error: group Field Type Mismatch The error you're seeing is a classic JSON-to-Avro type mismatch. The Avro schema expects the

group

field to be either

NULL

or

STRING

, but the actual data coming from your Cvent connector contains a JSON object (

{id=50bafb55-a0e7-41b0-8d7c-7b0f4ca9446d}

), which cannot be mapped to a

STRING

type. Why This Happens When Airbyte writes Parquet to GCS, it first converts the JSON stream to an Avro schema, then converts each record to Avro, and finally outputs Parquet. [GCS Parquet docs] The Avro schema is derived from the connector's declared JSON schema. If the connector declares

group

as a

string

type but the source actually returns an object (e.g.,

{"id": "50bafb55-..."}

) at runtime, the conversion fails. [JSON-Avro conversion] This is a known pattern across multiple connectors (MongoDB, MSSQL, Postgres, Pinterest, etc.) where the declared schema type doesn't match the actual data type. [Forum: MSSQL CDC] [GitHub: source-mssql] How to Fix It Since this is a custom connector, you have direct control over the schema definition. Here are your options: 1. Fix the JSON schema in your connector — Update the

group

field's type in your connector's catalog/schema from

string

to

object

(with the appropriate properties, e.g.,

id

as a string). This is the root cause fix. 2. Use `oneOf`/`anyOf` in the schema — If

group

can be either a string or an object, declare it as a combined type:"group": { "oneOf": [ { "type": "string" }, { "type": "object", "properties": { "id": { "type": "string" } } } ] }Note: Combined restrictions are supported in Avro conversion, but some union edge cases can cause unexpected behavior. [Combined restrictions] 3. Serialize the object as a string — In your connector's record transformation logic, serialize the

group

object to a JSON string before emitting the record, so it matches the declared

STRING

type. 4. Use JSON or CSV output format temporarily — As a workaround, switching the GCS destination output format to JSON or CSV will bypass the Avro conversion entirely, as these formats don't require strict type mapping. The cleanest long-term fix is option 1 or 3 — ensuring your connector's declared schema matches the actual data it emits. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

my code is generic so cant define schema and I cant hard code anything and I cant serialize anything in generic code... @kapa.ai

The knowledge sources do not contain enough information to provide a confident solution for a fully generic custom connector scenario where schema cannot be predefined. However, based on what the knowledge sources do tell us, here are the relevant facts: What Airbyte Does With Untyped/Unknown Fields • Untyped fields (no type specification in schema) default to

string

, and their values are serialized to string. [Untyped field] • Untyped objects (an

object

field with no

properties

) get their entire value serialized to a string representation automatically by the destination. [Untyped object] • Additional properties of unknown types are silently dropped. [Additional properties] Practical Implication for Your Case If your generic connector declares the

group

field as:

"group": {
  "type": "object"
}

(i.e., an object without a

properties

specification), Airbyte will automatically serialize whatever object value arrives into its string representation — no hardcoding required on your end. This means you only need to declare the field type as a generic

object

in your schema, and the destination handles the rest generically. This may be the least invasive change compatible with a generic connector design. That said, I'd recommend checking the latest connector and destination-gcs versions, as some of these conversion issues have been addressed in newer releases. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

whats group field? @kapa.ai is it json object ?

Based on the error message you shared:

Could not evaluate union, field group is expected to be one of these: NULL, STRING.
If this is a complex type, check if offending field (path: group) adheres to schema: {id=50bafb55-a0e7-41b0-8d7c-7b0f4ca9446d}

Yes — the

group

field is a JSON object in your actual data. The value

{id=50bafb55-a0e7-41b0-8d7c-7b0f4ca9446d}

is clearly an object with an

id

property. The problem is that your connector's schema declares

group

as a

STRING

, but the Cvent API is returning it as an object — so the Avro conversion fails because it can't fit an object into a

NULL | STRING

union. I'm not able to tell you more about the specific structure of the

group

field from the Cvent API, as the knowledge sources don't contain details about your custom Cvent connector's data model. You would need to inspect the raw API response from Cvent to understand the full structure of the

group

object (e.g., does it only have

id

, or other properties too?). 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

why its converting to avro then in parquet @kapa.ai

Great question! This is just how Airbyte's Parquet pipeline works internally. When writing to Parquet format on GCS, Airbyte uses a two-step conversion process: 1. JSON → Avro — The source JSON schema is first converted to an Avro schema, and each JSON record is converted to an Avro record. 2. Avro → Parquet — The Avro record is then outputted to the final Parquet format. As the docs explain: > "Under the hood, an Airbyte data stream in JSON schema is first converted to an Avro schema, then the JSON object is converted to an Avro record, and finally the Avro record is outputted to the Parquet format." [GCS Parquet docs] Why not go directly JSON → Parquet? Avro acts as an intermediate representation because: • Avro has a well-defined, strict type system that maps cleanly to Parquet's columnar format. • It provides a structured schema that Parquet needs to organize data into columns. • Airbyte reuses the same JSON-to-Avro conversion logic across multiple destinations (S3, GCS, R2, etc.). This is also why the error you're seeing is an Avro conversion error even though your output format is Parquet — the failure happens at step 1, before Parquet is even involved. [JSON-Avro conversion] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

for S3 also avro comes into picture @kapa.ai?

Yes, absolutely! For S3 as well, Avro comes into the picture — both for Avro format and Parquet format. The S3 docs explicitly state the same two-step process: > "Under the hood, an Airbyte data stream in JSON schema is first converted to an Avro schema, then the JSON object is converted to an Avro record, and finally the Avro record is outputted to the Parquet format." [S3 Parquet docs] So for S3: The same JSON-to-Avro conversion logic is shared across S3, GCS, and R2 destinations — which is why the same type of

AvroConversionException

error can appear regardless of whether your destination is S3 or GCS, and regardless of whether your output format is Avro or Parquet. [S3 output schema] [JSON-Avro conversion] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

but then why problem is coming only for GCS? only @kapa.ai

That's a great observation! The knowledge sources do not contain enough information to give a definitive answer on why the error is appearing only for GCS and not S3 in your case. However, here are some possible reasons worth investigating: Possible Explanations 1. You may only be using Parquet/Avro format on GCS — If your S3 destination is configured to use CSV or JSONL format, Avro conversion never happens there. The error only occurs when Avro/Parquet is the output format. 2. Different connector versions — Your GCS destination connector version may behave differently from your S3 destination connector version in how they handle type mismatches. 3. Different sync configurations — You may be syncing different streams or fields to S3 vs GCS, so the problematic

group

field may simply not be part of what's being synced to S3. What I'd Suggest Checking • What output format is configured on your S3 destination? (CSV/JSONL vs Avro/Parquet) • Are you syncing the same streams to both S3 and GCS? • What versions are your S3 and GCS destination connectors? I'm sorry I can't give a more definitive answer — the knowledge sources don't contain enough detail to explain the exact difference in behavior between your S3 and GCS setups. The answers to the questions above would help narrow it down significantly. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.