I was thinking of adding an emitter of sorts that would publish an array of datasource -> [columns used] for every Druid query executed. The idea ultimately is to figure out what all dimensions and metrics are not used so that we can tell users to remove them to ultimately reduce data size and improve query performance. I can see how this possibly could be extended to also include the query granularity, filter columns etc. What would be a good place to capture and publish this kind of information? We obviously want to do this after the query has been validated.

this can be emitted in the ClientQuerySegmentWalker, and the columns can be captured using the toolchest.getResultArray signature

Thanks for the ideas, @Laksh Singla.

@Laksh Singla Can’t we add a new method in QueryMetrics? Something like

columns(QueryType query)

. Then we can call query.getRequiredColumns() and then set that String Array returned in the builder.setDimension(…)

that would attach the column array as a dimension instead. Is that what you are trying to achieve?

Yes. The idea is that we may have a datasource with dimension a b c d. Using query metrics, we may find out that all the query are only reading/using column a b c and hence the column d is not needed. We can then reindex the datasource without column d to reduce segment size and increase rollup

lemme check the code once. I was thinking that you’ll be attaching the columns to the metrics.