Hello, I'm syncing a MySQL database and all but on...
# replication-troubleshooting
m
Hello, I'm syncing a MySQL database and all but one table is successful. Here is a snippet of the error:
Copy code
2023-02-02 06:51:40 [32mINFO[m i.a.w.g.DefaultReplicationWorker(getReplicationOutput):495 - failures: [ {
  "failureOrigin" : "destination",
  "failureType" : "system_error",
  "internalMessage" : "tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field p_count is expected to be one of these: NULL, INT. If this is a complex type, check if offending field (path: p_count) adheres to schema: 2840014740",
  "externalMessage" : "Something went wrong in the connector. See the logs for more details.",
  "metadata" : {
    "attemptNumber" : 2,
    "jobId" : 65,
    "from_trace_message" : true,
    "connector_command" : "write"
  },
p_count data type is int(10) unsigned, so 2840014740 is less than its max value of 4294967295... I am thinking that the connector is changing to a int which the max value is 2147483647
n
Hi! Could you please give me bit more info: 1. Airbyte version 2. Connector versions 3. Any sync logs
m
Airbyte version: v0.40.32 Connector versions: mysql beta I think I found the issue here: https://docs.airbyte.com/understanding-airbyte/json-avro-conversion/ If I read this correctly, it does change my int(10) to int. Any way of change this data type on the fly?
BTW: I'm doing a sync from mysql to aws s3 in parquet format with snappy compression
n
You can transform data after they have been moved to the destination, you can read more about that here: https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-dbt#transformations-with-dbt
m
I see... I need to do it on the fly or the json-avro conversion will fail
or is there another way to ignore this error during the sync?
n
I can't currently think of one, but let me sit on this for a bit and I'll get back to you!