Hello Everyone, wondering if I can get some guidance on something I am working on. I am storing events in a pinot table and we have a modified retention manager to delete segments based on the min and max values of an expiry date column on this table that is populated at ingestion time. Each event row in the pinot table is also associated with some external objects stored in S3 and we use the pinot table as source of truth. When a pinot segment goes out of retention we would like to delete those related objects in S3. Are there patterns on how to accomplish this?

You mean you have modified the Pinot source code to change retention manager behavior?

That is correct

Pinot doesn't provide callbacks / hooks on data deletion today. But this sounds like you are customizing Pinot for something that is beyond OLAP requirements.

It's a valid issue because if the s3 object isn't deleted prior to segment expiration, the object can be re-ingested by the SegmentGenerationAndPushTask

Hmm, if re-ingested it will be removed again by retention manager right? One should not rely on retention manager for querying. Queries should always have explicit time filters. I agree it is probably an inefficiency, but should not cause functional issues, which can be addressed as you proposed.

It seems like what I wrote above is confusing. The objects that I am storing in S3 are associated to the individual events. I am not referring to the segments in the deep store. In my case, each event / row in the pinot table is associated with one or more files in S3. During ingestion, I receive events and associated files, I store the events in pinot, and I store the files in S3. The files in S3 have a naming convention that lets them be associated with a specific event in pinot (e.g. the pinot table has an event_id and the files have the event_id as a prefix which enables me to look up the files associated with a specific event).

Yeah that is how I had interpreted your message originally and responded. This doesn’t seem like an OLAP engine requirement