I think I see an issue while running segment metadata queries against realtime segments containing an HLLSketch column. The

errorMessage

field in the query result gives:

cannot_merge_diff_types: [HLLSketch] and [HLLSketchBuild]

The complex type for HLL is

HLLSketchBuild

in IncrementalIndexStorageAdapter but for persisted segments, the complex type is

HLLSketch

and this causes a mismatch during the columnanalysis merge phase. Any ideas on how this can be fixed?

You can modify the merge strategy for SMQ https://druid.apache.org/docs/latest/querying/segmentmetadataquery/#aggregatormergestrategy

Hey @Laksh Singla sorry for the delay in getting back to this. Unfortunately, the merge strategy helps with aggregator merges, but not when the column types themselves have to be folded. This merge happens before the aggregator merge strategy is executed. A side effect of this behavior is that after starting a supervisor that ingests realtime data with HLLSketches, the segment metadata does not show the HLLSketch columns since the metadata cache skips columns with analysis errors. Consequently, users cannot see/query these ingested columns on the web console until the segments are handed off to the historicals. I'm not fully sure how this can be fixed at the moment. Choosing either of the types instead of setting

error_message

while merging column analyses might not be a good solution, as this may end up failing upstream at the coordinator while building the datasource row signature. If you have any thoughts on how we can fix this, let me know. I think we can at least document this behavior so that users aren't confused why they can't see their ingested HLLSketch columns.

we might need to special case the knowledge that

HLLSketch

and

HLLSketchBuild

are "the same" and it's ok to collapse to one or the other? Or provide a way for extensions like datasketches to drive this on their own AFAIK, there aren't issues querying these two types together, so this is a pure metadata issue, right?

Yes this is purely a temporary metadata discrepancy that exists while ingesting HLLSketch objects.