This message was deleted.
# generals
Slackbot
11/23/2022, 9:25 PMThis message was deleted.
s
Sergio Ferragut
11/23/2022, 9:37 PMI haven't, but just thinking through this. If you are just pruning older data should be a matter of setting up retention rules, and running a kill task. You have the data split into time intervals (as folders) in Deep Storage and you have the metadata rows for the sys.segments table. I think that's all you need to backup: copy the segment files into a different bucket/folder and backup the metadata rows. In order to be able to restore them later, add those rows back into the sys.segments table and put the segment files back into their respective Deep Storage location... it would be nice to automate this.
a
Arthur Sherman
11/23/2022, 9:40 PMInteresting @Sergio Ferragut let me look at this further, I could experiment with some of the older segments. Thanks!!
g
Gian Merlino
11/23/2022, 11:18 PMyou can certainly back up the files on deep storage; that's the easiest way
Gian Merlino
11/23/2022, 11:19 PMcheck out the
loadSpecpayloaddruid_segmentsGian Merlino
11/23/2022, 11:19 PMit has all the exact paths in deep storage to look for
a
Arthur Sherman
11/25/2022, 1:39 PMThanks @Gian Merlino I will do that. Looks like I need to take all that data and load it into a relational DB like Hive. I would imagine that I have to go through a Druid instance to do that...
g
Gian Merlino
11/25/2022, 5:21 PMyou can use the dump-segment tool: https://druid.apache.org/docs/latest/operations/dump-segment.html
however as that page points out:
> Note that the dump is not necessarily a full-fidelity translation of the segment. In particular, not all metadata is included, and complex metric values may not be complete.
Gian Merlino
11/25/2022, 5:22 PMso watch out for issues with complex fields, if you use any of those (like datasketches)
Gian Merlino
11/25/2022, 5:23 PMbtw I'm wondering what the purpose of such pruning is? is it mainly a cost thing?
we're planning some new features around being able to query data from deep storage without preloading it on Historicals; designed for use with older or less user data
Gian Merlino
11/25/2022, 5:24 PMif that sounds interesting i'd like to understand more about your needs here, as that will help us build the feature in a way that is useful to everyone
a
Arthur Sherman
11/28/2022, 1:27 PMHey @Gian Merlino thanks for the pointer to the tool, I don't have direct access to the file storage but I'll ask our team about this. We are doing this pruning to reduce the file system footprint of our Druid cluster, and this dataset was identified as one of the larger ones.
Arthur Sherman
11/28/2022, 4:38 PMHey @Gian Merlino I think I may go down the route of doing scan queries: https://druid.apache.org/docs/latest/querying/scan-query.html. While this means that I'll need to do more than 100,000 of these, I can rate limit them and keep track of offsets so I can restart the process. This can be fairly easily scripted and run off-hours, without impacting our application server that sits on top of Druid.
6 Views