Do Ingest Level Aggregations happen across segments? Also, do the aggregations occur across partitions? I saw the requirement for the lowLevel kafka api, so I'm guessing aggregations only happen at the partition level.

Only within segment, which implies within partition

noob question how is this any different from

aggregateMetrics:true

(?)

just more types of aggregation; https://docs.pinot.apache.org/developers/advanced/ingestion-level-aggregations#i-already-use-the-aggregatemetrics-setting

ofc i didn’t read the FAQ 😄 thank you

not yet. you still have to do

sum(count_column)

. but there have been talks to get the broker to magically issue a

sum

on a pre-aggregated

count

column.

count

is actually a bit confusing as it’s one of the functions where further aggregating is not the same function as

count

itself. min/max/sum, you can keep min/maxing/summing results. but with

count

, once you’ve counted N things, then M things, the resulting count of those is N + M, not 2 (which is what you’ll get if you call

count(*)

with pre-aggregation)

oh wait what i mean is that for our purposes we use

aggregateMetrics:true

but in the records that we consume we have like count and we always send one, but since it’s for the same primary keys pinot keeps on aggregating, but with this new feature you wouldn’t need to add that count record on the producing topic all the time cause you can do it in pinot itself with this. But yea you will have to issue the sum when querying for it. as you said depending on the granularity that you are looking at to i suppose, like in the example it’s daily sales so if you are issuing a query then i suppose that the final representation of the table would be -> day ts, total sales per day, sales count per day

yes, if you use the feature, it will automatically append the

1

on every row then sum it across all rows with the same non-metric fields. in your example, you would need 2 ingestion aggregation configs: one for count and one for sum_sales