This message was deleted.
# troubleshootings
Slackbot
03/23/2023, 6:29 PMThis message was deleted.
j
JRob
03/23/2023, 6:33 PMI should note this is version 0.23.0
JRob
03/31/2023, 12:55 AMAny ideas? Do I have a corrupt segment or something? I'm able to query the data for this time frame just fine...
b
Ben Krug
03/31/2023, 1:19 AMSeems to be something with the segment efoe-search-started-events_2023-03-06T000000.000Z_2023-03-07T000000.000Z_2023-03-23T180546.745Z_2. Assuming your replication factor is >1, you might try to find that segment on historicals, shut one down and delete it, start up, repeat on other, then try compaction again. That will reload it from deep storage. Maybe something wrong with that segment or a file missing, but I'm not sure.
Ben Krug
03/31/2023, 1:21 AMYou might be able to narrow down the search with
SELECT * FROM sys.server_segments WHERE segment_id LIKE 'search-started-events_2023-03-06'j
JRob
03/31/2023, 1:22 AMWhen I set the maxRowsPerSegment to 1000000 (instead of 5000000) then it fails at
efoe-search-started-events_2023-03-06T00:00:00.000Z_2023-03-07T00:00:00.000Z_2023-03-31T00:26:42.716Z_10JRob
03/31/2023, 1:27 AMI notice logs lines like:
Copy code
Unzipped 115925121 bytes from [/data/tripstack/druid/deep-storage/segments/efoe-search-started-events/2023-03-06T00:00:00.000Z_2023-03-07T00:00:00.000Z/2023-03-06T00:24:42.084Z/1/86753203-74fe-450c-a75f-142ac8d80c4e/index.zip] to [/data/tripstack/druid/apache-druid-0.23.0/var/druid/task/compact_efoe-search-started-events_jbjgpnkb_2023-03-31T01:11:10.617Z/work/indexing-tmp/efoe-search-started-events/2023-03-06T00:00:00.000Z_2023-03-07T00:00:00.000Z/2023-03-06T00:24:42.084Z/1]b
Ben Krug
03/31/2023, 1:30 AMdrop like clear from cache and reload?
j
JRob
03/31/2023, 1:30 AMI killed it
b
Ben Krug
03/31/2023, 1:33 AMYou don't need that segment?
j
JRob
03/31/2023, 1:34 AMNot if it's corrupt 😉
I'm willing to part with a few million rows if it means I get compaction working again
JRob
03/31/2023, 1:34 AMSo, unfortunately, this did not work. It fails at the same point, same offset ID
JRob
03/31/2023, 1:35 AM... So it's not failing working on that segment
JRob
03/31/2023, 1:35 AM Copy code
2023-03-31T01:31:50,739 ERROR [[compact_efoe-search-started-events_inkebabb_2023-03-31T01:23:40.174Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Incremental persist failed: {class=org.apache.druid.segment.realtime.appenderator.AppenderatorImpl, segment=efoe-search-started-events_2023-03-06T00:00:00.000Z_2023-03-07T00:00:00.000Z_2023-03-31T01:23:40.183Z_2, dataSource=efoe-search-started-events, count=4}
2023-03-31T01:31:50,743 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS.
net.jpountz.lz4.LZ4Exception: Error decoding offset 2154422 of input bufferJRob
03/31/2023, 1:36 AMI also noticed a little too late that the LocalDataSegmentPuller seems to unzip it just fine
JRob
03/31/2023, 1:40 AMAuto-compaction no longer works for this datasource because it keeps retrying the same day
JRob
03/31/2023, 1:50 AMSide note: I wonder why it unzips from deep storage twice
JRob
03/31/2023, 1:53 AMFirst by LocalDataSegmentPuller into
var/druid/task/{{datasource}}_{{task_id}}/work/var/druid/task/{{datasource}}_{{task_id}}/work/indexing-tmp/JRob
03/31/2023, 1:59 AMDropping segment 3 unblocked it.
JRob
03/31/2023, 2:00 AMWhen in doubt go nuclear
b
Ben Krug
03/31/2023, 2:00 AMOh! Well, good to hear. I was thinking just clear and reload, but if dropping it is OK and it worked, then gtk.
j
JRob
03/31/2023, 2:00 AMIt's loading from deep storage so I'm unsure what clear and reload would do
b
Ben Krug
03/31/2023, 2:11 AMOh, you're right, good point!
d
Didip Kerabat
04/03/2023, 5:00 PMWith strange issues, I usually re-ingest into a new datasource name. If an identical query works on the new datasource name, then I’ll just blow away the segment cache files.