https://pinot.apache.org/ logo
Join Slack
Powered by
# group-by-refactor
  • m
    @User @User I am looking at the GroupKeyGenerator code, and there are way too many variations (SV/MV/Dict/NoDict/ArrayBased/MapBased/etc). For SQL based order-by we need non-concatenated group key, so we can sort on individual group-by columns.
  • m
    From code perspective, it might be easier to see if we can modify the existing GroupKeyGenerator to not use single concatenated String key, but an array of objects. What do you guys think?
  • m
    Otherwise, we will have to implement all the variations for the new GroupKeyGenerator
  • k
    we need both right
  • k
    we need concatenated group key for fast look up
  • m
    We need both behaviors of group-by yes. But that is independent of whether group key is single string or individual objects
  • k
    no, i meant both concatenated and separate
  • m
    The only benefit of concatenated I see is performance
  • m
    Is there any other benefit?
  • k
    thats a big one right
  • m
    I am not sure, we can benchmark how much is the perf gain
  • m
    We pay the price of concatenate and split (cpu + garbage)
  • m
    But you are right, we can try to keep both
  • m
    Without having to duplicate all teh code
  • k
    whats the alternative
  • k
    right, thats my feeling as well.
  • m
    Yeah, let me investigate that further
  • k
    Think of a List<Comparable[]>
  • k
    this is the table
  • m
    Yes
  • m
    The table will be in IntermediateResult
  • k
    we need a hashmap which will map hashCode(group key) --> list of rows
  • k
    can you list out the various scenarios
  • k
    no order by
  • k
    order by group key
  • k
    order by aggregation function
  • m
    order by some expressions of projected columns (mix of group keys / aggr functions) -> most general form
  • k
    @User has left the channel
  • s
    @User has left the channel
  • x
    @User has left the channel