This message was deleted.
# troubleshootings
Slackbot
12/16/2022, 3:49 PMThis message was deleted.
s
Sergio Ferragut
12/16/2022, 4:45 PMIf they are available on the historicals, are they queryable?, perhaps you could try reingesting them.
s
Satish N
12/16/2022, 5:04 PMyes they are available in historical and we can query them
Satish N
12/16/2022, 5:05 PMwe r doing the delta load and it's keeping the intermediate copies which metadata is aware but we are not able to delete them
Satish N
12/16/2022, 5:06 PMSatish N
12/16/2022, 5:06 PMkind of similar situation
Satish N
12/16/2022, 7:47 PMAny inputs/suggestion
v
Vadim
12/16/2022, 7:54 PMIn your question you said "Is it possible to move data from historical to deep storage"? In general any data in a historical should already be in deep storage - that is where historicals load their data from. If you find yourself in a situation where a historical has some data that is not in deep storage then you are in deep deep shit. The typical way to get into that situation is if an intern (who will not be named!) messes up an
aws s3Vadim
12/16/2022, 7:55 PMin the second part of your question you mentioned "We have stale segments in deep storage (metadata is not aware of them) and the kill command is not taking care of cleaning." which suggests that you want to clean your deep storage and kill tasks are not working for you. If this is actually where you are then lets all be grateful it is not the previous thing I posted. 😛
Vadim
12/16/2022, 7:56 PMSo which one is it?
s
Satish N
12/16/2022, 8:02 PMwe are in second part of situation, druid_segments where used='false' gives the positive number(which shows we have versions of the segments which came with delta load) so we are trying to clean them from deep storage. For example we a dataset with one segment but in deep storage we see 200plus files for it. we tried all possibilities of kill command but the kill command is not working
v
Vadim
12/16/2022, 8:03 PMHow are you launching the kill task?
Vadim
12/16/2022, 8:03 PMAlso do you have realtime ingestion in this datasource?
s
Satish N
12/16/2022, 8:05 PMwe have both batch and stream , this case batch injestion one, below is the jsaon the last segment is on 4-1 so we are killing until 3-31. We are submitting this to the coordinator through curl {
"type": "kill",
"dataSource": "abc",
"interval" : "2022-03-01/2022-03-31"
}
v
Vadim
12/16/2022, 8:23 PMCan you share the logs for that kill task?
s
Satish N
12/16/2022, 8:27 PMwe were able to run it successfully after initating it from coordinator dynamic config
Satish N
12/16/2022, 8:27 PMConfigure the coordinator/runtime.properties with the following
Copy code
druid.coordinator.kill.on=true
druid.coordinator.kill.durationToRetain=P30D
druid.coordinator.kill.maxSegments=100
druid.coordinator.kill.period=P1DSatish N
12/16/2022, 8:28 PMWhen we updated our parameters above, nothing was running
Satish N
12/16/2022, 8:28 PMso our question is druid.coordinator.kill.on=true only works when we do through coordinator
k
Karan Kumar
12/17/2022, 5:16 AMJust read through the thread . The property
druid.coordinator.kill.ons
Satish N
12/18/2022, 6:58 PMWe got that kiran, The point i want to mention is even though u have that property u have to have the job through coordinator otherwise these parameters are of no use.
Satish N
12/18/2022, 7:00 PManyone have idea, where we specify the no of parallel sessions to run with kill job?
k
Karan Kumar
12/19/2022, 3:58 PMthe job through coordinator otherwise these parameters are of no use.How are you launching the kill task job ?
v
Vadim
12/19/2022, 4:49 PMThey are sending a kill task via curl (see above in thread)
k
Karan Kumar
12/19/2022, 5:01 PMAh I see.
Yeah we would need to kill task job logs.
anyone have idea, where we specify the no of parallel sessions to run with kill job?I donot understand what you mean by this. Do you want to control how many kill tasks can run in parallel ?
s
Satish N
12/19/2022, 5:10 PMCorrect
k
Karan Kumar
12/20/2022, 5:20 AMI donot think we can control that. Is there a reason you would want to control it though. I guess after the initial pruning, where it might launch a lot of tasks, things would become stable. No ?