This message was deleted Apache Druid #general

Join Slack

This message was deleted.

# general

Slackbot

03/23/2023, 8:42 PM

This message was deleted.

churro

03/23/2023, 9:35 PM

https://druid.apache.org/docs/latest/operations/mixed-workloads.html#historical-tiering

JRob

03/27/2023, 1:36 PM

Hmm, how would this work, exactly? In the example provided in the documentation both tiers are hot: the nodes have to have all the data loaded locally, it can't just pull from deep storage when the query demands it. If I wanted to move data to S3 for example, can I have the different tiers using different deep storages? How would the cold tier pull data out of the hot tier? I can't just put deep storage on S3 for the hot tier because it wouldn't be fast enough for Druid's replication from broker to historical...

JRob

03/27/2023, 1:57 PM

Is this sort of thing possible?

Copy code

{
  "type" : "loadByPeriod",
  "period" : "P3M",
  "includeFuture" : true,
  "tieredReplicants": {
      "hot": 1
  }
},
{
  "type" : "loadForever",
  "tieredReplicants": {
      "cold": 1
  }
}

Will the hot tier above only load the latest 3 month and drop anything older than that (I noticed drop rules can't apply to specific tiers).... When Druid adds the default loadForever rule to the bottom of the list, is it only for the default tier?

JRob

03/27/2023, 2:15 PM

Or perhaps, since I'm adding a new cold tier, the hot tier is actually just my

_default_tier

👍 1

JRob

03/31/2023, 12:22 AM

To update; we tried this and it is working wonderfully. We put two new data nodes into a "cold" tier that are much smaller and have access to cheaper disk array. Thanks!

Open in Slack

Previous Next