https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
g
I haven't noticed that: I have seen it legit delete files in the past
Wonder if you see anything interesting if you enable debug logging for the 
com.amazonaws
package?
It should show the req/resp to/from S3
k
Checking the task logs would be helpful to figure out whats happening. IMHO, we should log what we are deleting. If we are not logging , then maybe its a Github issue in the waiting
d
With INFO, I do get logs of what we are deleting but the problem is that the files are still there.
c
are you deleting a lot of segments at once? we have seen that when 
druid.coordinator.kill.maxSegments
is very high. this explains how that can happen: https://support.imply.io/hc/en-us/articles/360039970093
d
Thanks Caroline!
Example:
Copy code
2022-09-22T05:38:09,373 INFO [task-runner-0-priority-0] org.apache.druid.storage.s3.S3DataSegmentKiller - Removing index file[<s3://mybucket/druid/segments/mytable/2022-06-23T00:00:00.000Z_2022-06-24T00:00:00.000Z/2022-06-25T06:27:22.013Z/2910/index.zip>] from s3!
2022-09-22T05:38:09,467 INFO [task-runner-0-priority-0] org.apache.druid.storage.s3.S3DataSegmentKiller - Removing index file[<s3://mybucket/druid/segments/mytable/2022-06-23T00:00:00.000Z_2022-06-24T00:00:00.000Z/2022-06-25T06:27:22.013Z/2373/index.zip>] from s3!
2022-09-22T05:38:09,567 INFO [task-runner-0-priority-0] org.apache.druid.storage.s3.S3DataSegmentKiller - Removing index file[<s3://mybucket/druid/segments/mytable/2022-06-23T00:00:00.000Z_2022-06-24T00:00:00.000Z/2022-06-25T06:27:22.013Z/1836/index.zip>] from s3!
2022-09-22T05:38:09,612 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "coordinator-issued_kill_mytable_bpnkchlm_2022-06-23T00:00:00.000Z_2022-06-24T00:00:00.000Z_2022-09-22T05:30:09.902Z",
  "status" : "SUCCESS",
  "duration" : 475129,
  "errorMsg" : null,
  "location" : {
    "host" : null,
    "port" : -1,
    "tlsPort" : -1
  }
}
But when I checked the files, they are still there.
c
so 
2910/index.zip
, 
2373/index.zip
, and 
1836/index.zip
are still in s3?
that would be weird
d
yes, wild right?
💯 1
Apparently this drift has been going on for a long time, data in S3 is 10X more than actual size in metadata store. (sorry finance team 😛 🙇 )
😂 1
but seriously tho, this is why I am asking the question here, I need to fix this somehow.
c
does it delete any of the segments?
d
I need to look into that further later.
g
that's pretty weird. the call being made is a straightforward deleteObject call and I don't see logic to suppress errors or anything like that
I'm not aware of anything that causes deletes to get chomped on the s3 side either
There are versioned buckets, where storage will still be used by older versions, but the deleteObject call should still add a delete marker and cause the object to not show up in listings
d
I have a question about this scenario: 1. table A has been writing data for 1 year and I forgot to turn on the retention policy. 2. Then I turn on retention policy for latest 1 month. 3. Will Druid then starts a kill job from beginning of time to last 1 month + 1 day? 4. Or will Druid starts a kill job for 1 day at last 1 month + 1 day?
I also think that CloudWatch did not collect the correct metrics.
Ahh… yep, we actually did not waste unnecessary space. CloudWatch is the one that incorrectly count the size.
Open in Slack
PreviousNext
Powered by