Hi team we are planning to implement a usecase for offline table with tracking event with a timestamp .. One of our requirements would be that the accounts will be added based on adoption ..and when they are added we will have to backfill their events.Wanted to know does pinot support deleting and creating a new offline segment to include the new account for back dated events ?

Yes, you can use the same segment name(s) and push the data that includes the newly adopted accounts, and pinot will replace the old segment(s) with new one(s).

Thanks @Subbu Subramaniam are there documents or examples of this ?

Does this help? https://docs.pinot.apache.org/basics/data-import/batch-ingestion/backfill-data

Thanks will poc this approach

yes, segments can have overlapping timestamps. Note that all records are counted for queries.

ok thanks so then might not need to edit the segment ok may be we can edit in future incase we feel the queries performance can improve

It depends what you mean by "overlapping" If all segments have all day data, then yes, what you say is true. If on the other hand, segments have data for days 15-16, 16-17, 17-18, etc. (as an example), then time based pruning can apply. If you have an APPEND table, then this is how I expect things will happen. If you have a REFRESH table, then there is no concept of time that pinot pays attention to, so there is no question of time based pruning.

Thanks @Subbu Subramaniam so basically in our use case we will be loading 3000 accounts for last 2 years of tracking data into an Offline table.It is possible that after a month we want to now load 100 more accounts.We will then once again load 2 years worth data in Pinot table for those 100 accounts along with recent tracking data for the existing 3000 accounts.Newer segments will then have time stamp which is less than 2 years old.But before the 100 accounts were added all segments where timesliced as only the current events were loaded.Thats why there will be segments will over lapping timestamp.Will this break time based pruning ? Our plan was to use APPEND

It seems that you should be using REFRESH, since you will be refreshing the data every month with new customers.

We also want to keep appending data to the 3000 accounts so REFRESH would have worked if we wanted to do a daily snapshot

You have two choices: (1) Push entire data about all customers everytime (maybe once a month, once a week, whatever). (2) You add new data every day/week/ about customers -- in which case the data you add will be date dependent (some activity happened with a customer on a certain date).. I am not familiar with your data, so you need to choose between these two models and what makes sense. My suggestio is not to get hung up with time based pruning but decide your data model first

Thanks @Subbu Subramaniam did not see the slack notification last week.So we are definitely planning to do a POC on different approaches.Currently if we import for all customers we will have 500 Billion events which will take a lot of memory and increase costs due to which we have to do this customer import based on adoption and signup.Our requirement though is that moment a new customer signs up we have to load last 2 years worth of data into the existing table