This message was deleted.
# generals
Slackbot
09/27/2023, 3:23 PMThis message was deleted.
r
Renato Santos
09/27/2023, 4:21 PMI don't think you can decide which server it will be loaded, but you can use the compaction job to make sure all users going in the same segment file
Renato Santos
09/27/2023, 4:21 PMbut the servers are going to be distributed by other factors such tiers and free space, etc
Renato Santos
09/27/2023, 4:26 PMinstead of hash users, it may be best to use https://imply.io/blog/multi-dimensional-range-partitioning/
🙌 1
l
Laksh Singla
09/27/2023, 4:34 PMI don’t think you can decide which server it will be loaded, but you can use the compaction job to make sure all users going in the same segment fileDepending on the segment size, it might not hold. Assuming that the request-response cycle b/w any pair of nodes take similar time, I think optimizing the higher level parameters like primary, secondary partitioning, replication, and data server tiers rather than location of segments with same user_id would be better.
Laksh Singla
09/27/2023, 4:36 PMSince druid is not primary partitioned by user_id, it is not feasible to ensure that every user with same user_id are on same server, since in worst cases, it would mean every segment would be forced onto a single historical.
Laksh Singla
09/27/2023, 4:38 PMConversely, by tiering your data servers, you can ensure that this happens for segments with similar __time partitioning, though that’s not your use case, and that might not be helpful in optimizing the query times.
Laksh Singla
09/27/2023, 4:40 PMI’d suggest that you try to ingest the data with optimum parameters like correct primary and secondary partitioning, and check the query times then. I suspect it should be good enough.
t
Takaaki Nakama
09/28/2023, 10:50 AMThanks, got it.
I'm intrigued by the idea of placing the user_id nearby, but as you pointed out, I'm not sure yet if it's truly the necessary optimization. I'll try out tune partition key, tier, etc.
https://engineering.mixpanel.com/resharding-petabytes-of-data-to-improve-performance-for-our-largest-customers-1a3be4637778
👀 1