Hello Team, I ingested around 10K rows into a rea...
# general
s
Hello Team, I ingested around 10K rows into a real-time table from Kafka. After that, I marked
is_deleted
(configured as
deleteRecordColumn
) as
true
for soft deletion of 5K rows. However, after the
deleteRecordTTL
period, the primary keys are not being deleted. I can see that the documents are unqueryable, but they are still present in the primary key map. Is there a scheduler that handles this cleanup? Does it run automatically, or do we need to trigger it manually? Can we configure the frequency at which it runs?
m
It should run automatically, but may not immediately happen. How long did you wait?
s
I waited for around 10–15 minutes, but the number of primary keys is still the same 10K. I'm actually doing remote debugging using a local setup. Is there a way to trigger it manually?
s
how and where you set this property
s
@San Kumar which property? deleteRecordTTL?
After debugging, I found that the default interval for running the scheduler was 1 hour. So, I added the property
controller.realtime.segment.validation.frequencyPeriod=2m
to the controller config. Now, the periodic task
RealtimeSegmentValidationManager
runs in 2 minutes and is responsible for removing the deleted primary keys from the metadata.
s
Good finding Will it apply to normal realtime table
✅ 1
m
I won’t recommend 2 minutes in production it is too frequent
s
I had reduced the time for local testing; I will time it properly for production