Hello Team Hello i pushed message to offline table and reading from a abc source table. in ABC table retention period is 1 hour. Here we are recreating segment file every 15 minutes with nearest order15MINHOUR and make segmenttimestamp as order15MINHOUR.so all order hour data is replacing and have latest status of that order. now order ststus came after 2 hours in source table and we have only now one data..so when we recompute the data upload the data its has only 1 record . in this case we are planning to either append that single record when updatetimestamp is greter than 1 hour or replace that record some how.. how can I replace that segment and know the segment information for that order

Sounds like you are doing a lot of workaround to get around upsert?

yes..

Why?

we want latest record per row.For a order there will be multiple update status but we want one latest status.we can not use offsert because of huge RAM size. so we are replacing the file every 10 minutes for last 1 hour records

How many keys do you have? Can you not horizontally scale? This is exactly why upserts where built, you are doing a ton of workaround which may cost you a lot more than just scaling upserts

We did the horizontal scale of server increasing 2 to 5 but did not get an impressive result.and it add more cost in cloud for us. So we decided to file based load using jobspec

What’s the cost of 5 servers, vs the Eng cost you have spent finding a workaround ?

We took 3 week times to build it and it's very stable. But some use case like it retention period over in source table and records status update after retention period it's the issue for us because we are not recalculate the whole value. We are creating parquet file and 15 minutes segment and push to the pinot with a k8 job.k8 job runs every 10 minute. Server we have 24gb each of 4.one controller and 3 broker. The processing pod is just 1gb and 1 cpu.and limit to 3gb and 2cpu.