is there any way to aggreagete time series data on basis of tumbling time window while ingesting in pinot itself say on basis of time and update the records automatically without any need of tumbling window aggregation in my stream processing job(ex-samza job/spark streaming) .. means can inot keep updating the same countPerMin column automatically whenever new feeds for same time window comes .. or they have to be handled in samza jobs/spark streaming only .cc: @Mohamed Hussain

right now you need to do this in samza or flink job. Pinot consumes data from kafka as it as and only support row level transformation.

also my topic is not partitioned by key on which I am aggregating(It was partitioned while creation of the topic itself to say 20 partitions) .. thus multiple events on topic from where pinot consumes will be generated for same key for same time .. ie suppose no of page views for 9th june at 14:13 IST .. 5 records are ingested by pinot with no of page views for the same user id as 10,12,4,5,3 since topic was partitioned and multiple samza consumers of same samza consumer group were aggregating for the same topic from different set of partitions .. and thus emitting different msgs on output topic from where pinot consumes .. can I upsert the pinot record for same user id by adding the last value with the latest value ie for userid="123",time="09/06/2021 14:17 IST",countPerMin=5 was there already .. now same userid="123",time="09/06/2021 14:17 IST",countPerMin=10 record came on kafka topic where pinot is consuming from .. then will it update the existing record from userid="123",time="09/06/2021 14:17 IST",countPerMin=15 .. is such kind of upsert possible? incremental upsert kind of ufnctionality or maybe through some sort of user defined function?

for upsert, if you make user id as primary key and time, then you can do pre-agg on your streaming job to publish the second msg like

{*userid="123",time="09/06/2021 14:17 IST",countPerMin=10*}

and ``{userid="123",time="09/06/2021 14:17 IST",countPerMin=15}` pinot will upsert the field

countPerMin

so here.. will countPerMin become 15 or it will become 25 since I dont just want to be updated with the latest value .. I want it to add with the last existing value

latest value

any way to add with the existing value by some UDF or any means

then you don’t need upsert, you can just do it during runtime to sum up

u mean in my samza job itself

or you generate this 25 in your streaming job and use upsert

you can just do it during runtime to sum up > how to do that in pinot

query

ok so every event before i need to publish on output kafka topic from where pinot consumes .. i should make a query on pinot broker for existing value and then do the addition

or you just do aggregation on samza job

ok then i have to hold the entire state of my jobs in samza inbuilt state management database like leveldb or rocksdb or KV store

yeah, partition + rocksdb should work

sure will try that