Hi! I’m trying to sync data from MSSQL (CDC) to S3. When I set the sync to be a full sync, it works fine, but as soon as I put it to incremental, it fails with the following message:

2022-10-12 09:53:41 - Additional Failure Information: tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field Location is expected to be one of these: NULL, DOUBLE. If this is a complex type, check if offending field (path: Location) adheres to schema: 9.447

@[DEPRECATED] Marcos Marx turned this thread into Zendesk ticket 2753 to ensure timely resolution!

Hey could you create a post over discourse so that team can help you

Also, check if your format is set to either of avro or parquet. If your answer is, change file format to json or csv. I got burn in the past read on SO.

thanks, posted on discource too. Switching to csv or json is a no-go for me, the table is too large for that

Hello Huib, there are some complex schemas the avro/parquet can’t handle today. Hope this get better in the future. For now the solution would be use a schemaless format like json.

Hmm, yeah, no. The schema we’re talking about is not complex at all, it’s a flat table with a handful of elementary fields (string, double). You can tell from the message that it’s struggling to convert the value

9.447

to DOUBLE. I’m 100% sure the schema itself is not the issue here, because loading the table with spark directly and writing to parquet works just fine.

Sorry the delay in reply Huib. Nice you found a workaround to the issue. You can always open a Github issue reporting the problem to future fixes or improvements.