Hello Everyone, I have set retentionTimeUnit and ...
# troubleshooting
j
Hello Everyone, I have set retentionTimeUnit and retentionTimeValue in Upsert table config but table contains all data since inception, i want to drop data after some time , is it possible? I have set retentionTimeValue as 2 Days but data is still there for last 15 days I understand this might be the case not sure : Segments are retained as long as they contain active records, regardless of their age. Retention in upsert tables is governed by both the segment time range and the upsert semantics (i.e. ensuring data correctness by keeping the latest version of each record). but i have primary key : id, window_start_time, window_end_time in that case this combination will always be active but i want to drop it after some time as i want to retain only last N days of data.
any other solution apart from sending delete records?
j
@Mayank this property will help in reducing in-memory data but i was under assumption that upsert table will drop records like non-upsert table once record will pass the retention period but compaction task rewrites existing data into new segment and keeps the older records. Only way i can see is to send delete column as true value to remove from data from existing segments