Hello Everyone,
I have set retentionTimeUnit and retentionTimeValue in Upsert table config but table contains all data since inception, i want to drop data after some time , is it possible?
I have set retentionTimeValue as 2 Days but data is still there for last 15 days
I understand this might be the case not sure : Segments are retained as long as they contain active records, regardless of their age. Retention in upsert tables is governed by both the segment time range and the upsert semantics (i.e. ensuring data correctness by keeping the latest version of each record).
but i have primary key : id, window_start_time, window_end_time
in that case this combination will always be active but i want to drop it after some time as i want to retain only last N days of data.