This message was deleted.
# generals
Slackbot
09/18/2023, 6:53 PMThis message was deleted.
p
Peter Soderberg
09/18/2023, 7:08 PMWe could get philosophical but might have to pull in some MBAs for workplace hierarchies... https://en.wikipedia.org/wiki/Flat_organization
🤣 1
Peter Soderberg
09/18/2023, 7:13 PMI kid! Good question - in Druid terms, the supervisor is responsible for managing consuming from a stream. The middle manager is responsible for managing ingestion - Peons / tasks are spun up to do the work of ingestion, generally in "slots" on middle manager nodes. And the overlord oversees it all
Peter Soderberg
09/18/2023, 7:19 PMSo yeah, we run a few processes to make sure things are reliable - neither of the 2 you mentioned do the real work of consuming from batch or streaming sources and creating Druid segments, but they're crucial in making sure those peons are doing what you want them to.
Peter Soderberg
09/18/2023, 7:20 PMI will note that some folks are replacing the MM with k8s - is that what got you wondering?
a
Aggelos Karalias
09/18/2023, 7:39 PM@Peter Soderberg am I getting it right that the k8s integration is an overlord implementation spawning k8s jobs?
In a similar approach have you heard anyone experimenting with a Nomad MM-less implenation (most probably using Nomad's batch jobs)?
We're running Druid on Nomad and that'd be really interesting..
p
Peter Soderberg
09/18/2023, 7:50 PMpretty much - you as the CEO in our little workplace hierarchy can go around MMs since k8s can also manage resources pretty well https://druid.apache.org/docs/latest/development/extensions-contrib/k8s-jobs/
Peter Soderberg
09/18/2023, 8:01 PM@Aggelos Karalias The MM-less ingest is still a bit experimental. And I'm not familiar with MM-less Druid on Nomad, but could be plausible
👍 1
k
Krishna
09/18/2023, 9:35 PMcan we have just MM on k8 and rest of the druid components on EC2 ?
s
Sergio Ferragut
09/18/2023, 11:33 PM@Krishna In principle, I think yes. What makes all the nodes in Druid belong to a cluster is primarily that they use the same discovery mechanism. As long as they use the same zookeeper instance or the k8s version of that with the druid-kubernetes-extensions then it should be possible. I haven't heard of anyone trying this though. Let us know if you try and how it goes.
k
Krishna
09/18/2023, 11:46 PMsure
3 Views