https://linen.dev logo
Join Slack
Powered by
Hi All, In Kafka ingestion, when roll up is enable...
# general
a
Hi All, In Kafka ingestion, when roll up is enabled, does it consider only the dimensions mentioned in the schema or all the fields in the message? In some of our flows, the dimension list in the schema contains a subset of the fields in the Kafka message. With roll up enabled, it is not incrementing the "count" metrics field based on the specified list of dimensions. Instead it seems to be considering all the fields in the message and hence the "count" is incorrect. We archive all the messages from the topic on S3 and when we extract the data from S3 using a MSQL, the count is higher than the count in the table ingested directly from Kafka. Is this expected? Druid version: 24.0.2 Thanks, AR.
b
The count should be the number of input rows with those particular values for the druid dimensions. Are the other kafka fields being ingested? Could they be getting considered as dimensions too? Iiuc, total count of source rows from MSQL should equal sum(count) in druid.
g
the 
count
metrics should reflect the number of rows that came in, regardless of what your dimensions are defined as
for example- if you have defined a 
count
metric called 
cnt
, then if you run a query like 
SELECT SUM(cnt) FROM tbl
, the result would be the number of input rows, regardless of what dimensions are set to
a
@Ben Krug - That was our understanding and expectation too. But the sum(count) from the realtime table is less than the "count()" value from MSQL query executed on S3. In fact there is no record in the realtime table which has "count" value > 1. This would imply that other fields in the Kafka message that are not listed in the "dimensionsSpec" of the supervisor are being taken into account when calculating the "count" value for a row in the realtime table. We have disabled "rollup" in the spec and restarted the supervisor to confirm this. Will share an update after it runs for a day to two. Thanks, AR.
Actually, I checked it again and there are records in the realtime table with hits > 1. So it does seem to be rolling up but we are also seeing cases where the sum(count) from realtime table does not match the count() from MSQL on S3. We checked the task logs for intervals where we found a mismatch and did not find any errors or dropped records in the task stats. In any case, we have disabled rollup to verify that Druid is ingesting all the records from the topic and the count from realtime table matches the data archived on S3. Thanks, AR.
The "inputFormat" of "ioConfig" section of the supervisor spec has the below setting: "inputFormat: { "type": "json", ... "*keepNullColumns*": false } What does 
keepNullColumns
do? We have not specified it explicitly in the supervisor config and am unable to find anything in the documentation about it. How does it affect roll ups? Thanks, AR.
In Kafka ingestion, how does Druid handle JSON array fields during rollup? Does it ungroup them and then calculate the metrics? Or is the JSON array field value treated as an atomic value? Thanks, AR.
g
i believe 
keepNullColumns
is something used internally by the druid web console as part of its usage of the sampler. it would be 
false
for normal operation
JSON array fields are treated as atomic units for purposes of rollup
a
Thank you Gian. I believe that JSON based batch ingestion would treat the JSON array fields as atomic units as well. For MSQ based ingestion, we need to use MV_TO_ARRAY() to avoid ungrouping of the array fields during aggregation. Is the understanding correct?
An update on the original query: We disabled rollup on the kafka supervisor. Now every record has a "count" value of "1". And importantly, the sum(count) from the Druid table matches the sum(count) from the archived files S3. So it looks like there is some issue with the rollup in the kafka indexer. I checked the logs and didn't find any dropped records (parseerrors or otherwise) so am not sure where to check further. We noticed a similar issue with another table as well. So we have disabled roll up on all the kafka ingested tables. May be a Druid version upgrade will solve this but without knowing the root cause, it is difficult to arrive at the solution. šŸ˜ž Thanks, AR.
g
that seems odd, although druid 24 was the very first version that included nested columns, so i also wonder if this is something that was fixed in later versionsā€¦
is what you're seeing something that could be reproduced by someone else, by any chance?
a
Thanks Gian. We have 8-10 realtime flows currently. Roll up was enabled for 4 of them. Only 2 of them had this issue. Since we disabled roll up on all of them, the issue has gone. Not sure how it could be replicated. As you said, it may be data specific but which data.... We are in the process of upgrading to v27. Once it is done, may be we can enable roll up again and check. Unfortunately we don't have good data volume in our test env so we need to play around in our prod env.
g
thanks for the info. what you describe certainly sounds like a bug, i am definitely interested in understanding if it's still present in the more recent versions-- there certainly have been various fixes since nested data columns were first introduced, but i'm not sure if this was one of them. if you see it in 27 please let us know!
Open in Slack
PreviousNext
Powered by