We have a system that is sending JSON data from many devices to a single Kafka topic and the messages have many different schemas. Is there a good way to ingest this into multiple data sources efficiently? I looked at the filter function but that comes fairly late in the process, after transformations and flattening. Each of these different schemas will need very different flattening and/or transformations configured so filtering doesn't seem to be the way to go. Unfortunately, there is no Kafka header value to look at, just a single entry in the JSON payload which denotes the different message format. I thought about just pulling everything into a single data source, pulling the value in the JSON message up to its own dimension then having batch ingestions separate the data out to their own data sources by that dimension but the data is meant to be consumed in near real-time to look for issues that need to be alerted on; I do not know if batch ingestions can be set up to constantly pull data in from another data source (I have only worked with streaming ingestion to date) Any suggestions will be welcomed 🙂 Thanks, Sean

Filtering would be the easiest, providing real-time data although not the most efficient. It would require a different supervisor for each data source and would be consuming data n-times based on the number of data sources you have. The batch option is the most efficient, but we do not have any mechanism to automatically ingest data via batch. The only other option would be to load the data into JSON fields, and do the transformations and flattening at query runtime. Depending on your query complexity and volume, this may or may not be viable as it would require more calculation for each query. The "ideal" case would be to have the Kafka topics split upstream (possibly using something like Flink).

Thanks for this, it confirms what I was thinking. I wish the messages were split upstream but the team wants to have everything go into one topic and let the clients sort it out. I am doing the last option, reading everything into a single table (the data comes in as a single JSON object in one field of the message) but was hoping for something better. I may try the first option and see what kind of performance hit this causes. I just hate the thoughts of reading the same data over and over again, just to ignore it in most cases 😞