When streaming data via Kafka to a realtime table, does it have to be 1 record per message or is there a way to put multiple records in a single message?
x
Xiang Fu
03/04/2021, 2:44 AM
one record per msg only
Xiang Fu
03/04/2021, 2:44 AM
Kafka is made for streaming ;)
k
Ken Krugler
03/04/2021, 3:30 AM
I guess you could use a JSON column to effectively store multiple records per message.
k
Kishore G
03/04/2021, 3:50 AM
You can also use flatten udf to convert one message into multiple records
j
Josh Highley
03/04/2021, 4:07 PM
The flattening "one record into many" documentation states this isn't natively supported yet -- is that still correct? I'd have to write my own decoder/reader?