Hello, this is about ingestion json parsing. We ha...
# general
e
Hello, this is about ingestion json parsing. We have a large Kafka topic which is split to multiple data sources on the Druid side, using filter transformation. This means there's compute overhead of parsing the same events across all supervisors (of course, in addition to network overhead). Before I go and benchmark it, does anyone know how likely is it to be a large overhead?
s
Hi, depends a bit of your reserved resources and the definition of "large topic" here. I would guess you won't run in performance. We have used once this extension https://github.com/iunera/iu-code-ingestion-druid-extension running even python code during ingestion to unfold JSON object on a datastream which was ~1TBytes data per Day without any major issues.