http://starrocks.io logo
Join Slack
Powered by
Hello Team. While collecting JSON-formatted data u...
# questions-and-troubleshooting
u
Hello Team. While collecting JSON-formatted data using routine load, I encountered JSON data with a malformed format. The routine laod task keeps failing due to this event, and since it cannot process the data in the partition where that event is stored, the consumer lag for that specific partition keeps increasing. Is there any way to skip this invalid JSON event? Setting 
max_filter_ratio
had no effect.
Copy code
Error: Data quality error: parse error. Failed to iterate document stream as object. error: UNESCAPED_CHARS: Within strings, some characters must be escaped, we found unescaped characters.
r
Can you share the 
show create routine load <routine_name>;
? If possible can you share sample of the malformed data?
u
Incompleted JSON.
Copy code
{ 
  "k1": "v1",
  "k2": "v2",
  "k3
Copy code
CREATE ROUTINE LOAD ... on ...,
COLUMNS ...,
WHERE ...
PROPERTIES
(
"desired_concurrent_number"="6",
"max_error_number"="100",
"max_filter_ratio"="1.0",
"max_batch_interval"="10",
"max_batch_rows"="200000",
"task_consume_second"="15",
"task_timeout_second"="60",
"format"="json",
"jsonpaths"="[...]",
"strip_outer_array"="false",
"json_root"="",
"strict_mode"="true",
"pause_on_fatal_parse_error"="false",
"timezone"="UTC",
"partial_update"="false",
"trim_space"="false",
"enclose"="0",
"escape"="0",
"log_rejected_record_num"="0"
)
FROM KAFKA
(
"kafka_broker_list"="...",
"kafka_topic"="...",
"kafka_partitions"="",
"property.kafka_default_offsets"="OFFSET_END"
);
Open in Slack
PreviousNext
Powered by