We are currently performing batch injection into a Pinot table using the provided example. In our first iteration, we have the following data:

event_time, source, country, event_type
epochmilli1, nokia, ind, event1
epochmilli1, nokia, ind, event2
epochmilli2, samsung, USA, event2

We uploaded this data to Pinot, resulting in the following records:

epochmilli1, nokia, ind, event2
epochmilli2, samsung, USA, event0

Subsequently, we received new data:

epochmilli1, nokia, ind, event3
epochmilli1, Apple, ind, event0

Our program processed this new data and prepared it as follows:

epochmilli1, nokia, ind, event3
epochmilli1, Apple, ind, event0

We aim to update the segment to reflect the final records, which should be:

epochmilli1, nokia, ind, event3
epochmilli2, samsung, USA, event0
epochmilli1, Apple, ind, event0

To clarify, we want to replace the record for

epochmilli1

where the device type is

nokia

and add any new records for device types that are not already in the database. Could you please provide guidance on how to achieve this update in the Pinot table? we will use offline table only no kafka topic use and no upserttable

seems a lot of customized logic, I would suggest to use spark connector to read data from pinot and do your replace logic then create pinot segments

we are able to build the logic and able to create the file. if we push this file ,will it remove existing non duplicate records or replace the whole records

you need to give the segment same name, so pinot will replace the segment and the data in batch