https://linen.dev logo
Join Slack
Powered by
Cleaning segment files in deep storage. Anyone has...
# general
y
Cleaning segment files in deep storage. Anyone has succeeded? I am using S3 as a deep storage. Kill task won’t delete unused segments in deepstorage. not sure some or all. I marked all segment as unused and issued kill task. I still can find data in deep storage.
a
What do your kill task stats and logs say?
y
task was completed. let me check log.
j
If this is a newer release, you may need to set 
druid.coordinator.kill.bufferPeriod
to a smaller value, or to zero, to eliminate the grace period allowed after the segment is marked as unused.
druid 1
j
For me it is mostly successful with some garbage left behind that has to be manually cleaned up.
This is the configuration that works for me:
Copy code
# Enable auto-delete from deep storage
druid.coordinator.kill.on=true
druid.coordinator.kill.period=PT10M
druid.coordinator.kill.durationToRetain=P1D
druid.coordinator.kill.bufferPeriod=PT6H
druid.coordinator.kill.maxSegments=10000
druid.coordinator.period.indexingPeriod=PT10M
a
@JRob which version of Druid do you use? The config that John has mentioned plays an important role in kill task operation as well
j
This config is for version 29.0.1 In a recent update, I had to set 
druid.coordinator.period.indexingPeriod
to match the 
druid.coordinator.kill.period
I should note that my configuration above is particularly aggressive. The default values for durationToRetain and bufferPeriod are much greater.
Also, I have made some special notes internally on this due to past confusion. The buffer period creates a sort of recycle bin; when a segment is dropped it isn't immediately available for kill; instead, it will stay in the recycle bin for 30 days (configured via 
druid.coordinator.kill.bufferPeriod
). In order for a segment to be candidate for deletion it must be marked unused and not fall within the configured bufferPeriod & durationToRetain 1. Druid only kills segments that have been marked unused 2. Druid will not issue kill commands for segments created in the past 90 days (can be configured via 
druid.coordinator.kill.durationToRetain
) 3. Druid will not issue kill commands for segments marked unused in the past 30 days (configured via 
druid.coordinator.kill.bufferPeriod
) 4. If a segment is within the kill interval but was marked unused in the past 30 days (configured via 
druid.coordinator.kill.bufferPeriod
), it will be skipped. Effect of Retention Policy: 1. When a segment is dropped due to a drop rule; it is marked unused. 2. The effective retention period for segments in deep storage is extended by 30 days (configured via 
druid.coordinator.kill.bufferPeriod
). Effect of Compaction: 1. Compaction tasks create new versions of segments. The old versions of segments are marked unused. 2. The old versions of segments marked unused by compaction will be retained for at least 30 days after compaction (configured via 
druid.coordinator.kill.bufferPeriod
). 3. The old versions of segments could be retained even longer if they were originally created within 90 days (configured via 
druid.coordinator.kill.durationToRetain
).
🙌 1
💯 1
y
I see 
forceTimeChunkLock
is true but can I set it false? since it is unused segment,
j
That makes sense to me, but setting this to false is still experimental. I've not personally tested this though I've never had a kill task conflict with another task.
y
we have a lot of late arrival data and it is frequent. so I have to run compaction very often. This cause size of S3 increase drastically. Kill task wont start. it is stuck on waiting state. I believe it is because it could not acquire lock..
In metadata database, I see datasource existing but I could not list the datasource in web ui ( I checked show unused). The datasource was was kill by api. This is why I still sees data existing on S3. What is the issue here? Metadata db and druid seems out of sync..
I do not see any left overs in druid_segments table…
a
Sorry, your last two messages are a bit inconsistent. Do you mean druid_segments in the metadata store is empty?
y
Sorry for confusion, druid_dataSource table was not empty. but I see druid_segments was empty. but I see data left over in S3.
it suspect that when S3 api fails this incident happens.
a
Do the kill task logs have anything interesting?
y
task logs were lost I recently found the issue. I thought data was fully dropped but I got alert on s3.
a
Which version of Druid is this?
y
We recently updated to 30.0.1. I think that incident happened on 29+
We are going to test dropping large datasource. We will check if this problem is reproducible.
a
Please take a dump of the segment metadata before dropping if possible
y
Could you clarify?
a
Please backup the data in druid_segments before you launch the kill task
That way we can see the state of segments left in s3
y
ok.
a
I recall you were experimenting with concurrent locks. Is that still enabled btw?
y
We decided not to use concurrent locks.
a
I see
y
Copy code
select used,
       count(*)
from druid_segments where dataSource = dataSourceName
group by 1;
+------+------------+ | used | dataSource | +------+------------+ | 0 | 410 | | 1 | 7954 | +------+------------+ Count on s3 object, 16000+ checked log nothing special..
ignore the result above. since s3 has 
descriptor.json
file as well.
j
The way I have observed Kill tasks to work (pre-batching versions) is: 1. First, the job deletes the records from the druid_segments table 2. Next, the job proceeds to delete the S3 segment files, one by one. The above is not atomic transactional ... and the S3 file deletes can take a while ... so if the Kill job abnormally terminates in the middle of the S3 delete process it will leave the remaining S3 files orphaned, with no reference to them in the druid_segments table anymore. Possibly is this what has happened in your case? The counts from the druid_segments table -- used or unused -- will not tell you about the orphaned S3 files. The only "clean" way I know how to clean up the S3 orphaned files is to: (a) reindex your table into a new table with a different name (b) drop/kill the original table as best you can (c) manually delete the entire S3 subfolder containing the original table segment files. (d) reindex your table back to the original name Thanks. John
j
I have also noticed garbage is left behind when a job fails to complete after it has pushed some data to deep storage. Every time I have an incident related to deep storage this happens. There's always a ton of garbage too because the indexer will keep retrying. I have to manually clean up deep storage from time to time. I reported this here: https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1726018931517189
2 Views
Open in Slack
PreviousNext
Powered by