https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
l
Copy code
How it works

The K8s extension takes the podSpec of your Overlord pod and creates a kubernetes job from this podSpec.
Does it work if Im using coordinator as overlord? Or do I need to explicitly have both defined so 
Overlord
pods can be ‘clone’ into ingestion tasks? The peons will have the same Xmx/MaxDirectMemory values as Overlord? The same memory limit/request?
j
• It currently does not work on K8s 1.25+ but there is an active fix being written. • It currently does not support TLSv1.3 • It seems to work well otherwise • I don't know about the overlord/coordinator combined pod • Yes it will take some specs from the overlord pod such as service account, resources most likely, etc.
👍 1
d
I've been experimenting with it, it works well but I will have to modify the resource requests / limit by forking druid itself. As it is, it uses the javaOpts array to guess at memory and hard codes cpu to 1. And I had a problem with the max task count once that I had to restart the coordinator for but I'm not sure it is related to mmless.
👍 1
l
Copy code
static long getContainerMemory(PeonCommandContext context)
  {
    List<String> javaOpts = context.getJavaOpts();
    Optional<Long> optionalXmx = getJavaOptValueBytes("-Xmx", javaOpts);
    long heapSize = HumanReadableBytes.parse("1g");
    if (optionalXmx.isPresent()) {
      heapSize = optionalXmx.get();
    }
    Optional<Long> optionalDbb = getJavaOptValueBytes("-XX:MaxDirectMemorySize=", javaOpts);
    long dbbSize = heapSize;
    if (optionalDbb.isPresent()) {
      dbbSize = optionalDbb.get();
    }
    return (long) ((dbbSize + heapSize) * 1.2);

  }

  protected Container setupMainContainer(PodSpec podSpec, PeonCommandContext context, long containerSize, String taskContents)
  {
(...)

    mainContainer.setName("main");
    ImmutableMap<String, Quantity> resources = ImmutableMap.of(
        "cpu",
        new Quantity("1000", "m"),
        "memory",
        new Quantity(String.valueOf(containerSize))
    );
    mainContainer.setResources(new ResourceRequirementsBuilder().withRequests(resources).withLimits(resources).build());
    return mainContainer;
  }
Interesting so for the pod requests and limits: • CPU: always 1 • Memory: (Xmx + MaxDirectMemorySize) + 20%
c
we use it everywhere, but we also wrote it 🙂
🎖️ 3
l
so you’re using for actual production jobs?
c
yeah i have a few patches on top which I haven’t upstreamed yet. I will do that this week in one big PR
@Luiz Augusto yeah we use it for 90% of our druid clusters, one really nice thing about the patch, is we are constantly rolling our druid to upgrade it and the tasks don’t die 🙂
like they do with the mm-version of druid, they just resume when the overlord starts back up because they are decoupled from druid
l
for deployments, druid-operator is the way or is there a better way?
c
we use the druid-operator, its not very well maintained right now, which is a problem.
I want to discuss maybe forking it or taking over responsibility
🎖️ 2
but yes we use the operator to deploy
d
It works sufficiently, I think the advent of mmless druid may accelerate it's use.
c
just exclude the mm in the spec for the operator and you are good
l
does it copy tolerations, affinities, topologySpreadConstraints etc from Overlord too?
c
yes, right now it tries to make it as easy as possible
one idea i had for a follow-up was the user could specify a template in a configmap or something, then the overlord could use that to launch peon tasks for the more power-user
but right now, the goal was to make it as easy as possible for the everyday druid user, but i do think i will need something more….
i don’t know what that is, also if you have an idea, please by all means make commits 🙂
l
one idea i had for a follow-up was the user could specify a template in a configmap or something, then the overlord could use that to launch peon tasks for the more power-user
yeah it would be great, eventually it’d be possible to configure peons to run in spot instances with ssd disks.
c
or PRs
so for us, our team has a way to specify spot instances based on annotations, so those can be overriden
but again, this patch was written with only my view point in mind, and I would love to hear what others want or think could be improved.
l
but that means your overlords are also running in spot, right?
c
nope, you can add additional annotations and labels for just your peon tasks
that is configurable right now
to add annotations and labels just to the peon k8s tasks
l
how?
c
check out
Copy code
druid.indexer.runner.labels
druid.indexer.runner.annotations
those let you add additional annotations and labels to just your peon k8s jobs
l
oh i see
c
but again, please let me know if you have other ideas, i just wrote it for usecases i encountered, i would love to hear what other people think / want.
also feel free to put up PRs the more people contributing will make this a better feature
l
dude, this is awesome, mm-less is game changer getting clusters cheaper, less things to manage
d
If you could specify objects in the config you could probably get pretty far by just allowing top level pod blocks in druid config, eg. affinity, env, envFrom, tolerations, limits, etc.
l
are the peons in mm-less capable to emit metrics?
c
yes, i have an internal patch for that, ill push that up this week!
before it was inheriting from the parent, i just added an internal patch to allow you to specify metrics you want to push
the 3 main things i have patched internally are, 1. metrics 2. upgrade the client to work with newer versions of k8s 3. better sidecar support
there might be more smaller things, but i will try to get a patch up this week with all my changes in a single PR so its easy for people top cherry-pick
a
@churro - what PRs are pending to be reviewed? You can also post the PR links on #C030CMF6B70 channel if they are not getting the attention from druid committers.
l
so we can expect these changes to a 25.0.1 or only to 26?
a
26 most likely.
l
When is Druid 26 is expected for? April?
a
We haven't started the process yet. April seems like the right time
j
Is there any way I could convince you to release a 
25.0.1
? Things are currently broken for a standard K8s release that has existed for 6 months
👍 1
c
i have one more patch for the k8s mm-less stuff
which i will put up a PR today
fyi: if you want all our internal patches here they are: https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1676401570977559
🥳 2
d
This looks so awesome. But I have a question if you don’t mind. With this setup, how do you get the task’s logs? Are the logs still viewable from the tasks UI?
j
I can confirm they are
❤️ 1
d
This is super! It will definitely lower the costs of running Druid for sure.
d
The UI just gives you k8s logs as if you were tailing the pod.
c
the only thing you don’t get streaming is the reports, you get those at the end if you set the 
enableTaskLogPush
flag….
i didn’t want to go into the pod and start copying out files 🙂 also reports are not important enough during task runs to deal with that
d
yeah, report is the least important thing 😂
c
one other thing @Jason Witkowski helped find a bug with mm-less and 1.25, I will put up a patch for this but long story short
at the end of a task, the overlord launches a task, waits for it to complete and then returns the status. Now in k8s < 1.25 it would just mean once the pod is not active, go grab the status. Now the contact has changed a bit, because in 1.25 they introduced finalizers: https://github.com/kubernetes/kubernetes/pull/110948 And we noticed that when we ran tasks on jason’s druid cluster they would complete fine but marked as failure. We outputted the job status and noticed that the job was not active, but both success and failure were not set, but there was this field that had
Copy code
uncountedTerminatedPods=UncountedTerminatedPods(failed=[], succeeded=[e916cbf9-467a-45f3-86a7-3767145d6384], additionalProperties={})
which from the docs:
Copy code
UncountedTerminatedPods holds the UIDs of Pods that have terminated but the job controller hasn't yet accounted for in the status counters. The job controller creates pods with a finalizer. When a pod terminates (succeeded or failed), the controller does three steps to account for it in the job status: (1) Add the pod UID to the arrays in this field. (2) Remove the pod finalizer. (3) Remove the pod UID from the arrays while increasing the corresponding counter. This field is beta-level. The job controller only makes use of this field when the feature gate JobTrackingWithFinalizers is enabled (enabled by default). Old jobs might not be tracked using this field, in which case the field remains null.
So now what happens is the job goes from a state where it is not active, to having uncountedTerminatedPods to then having a status with success or failure. I will push up a one-line fix to make this work, but for those of you working with 
1.25
version of k8s, I’m sure you will be affected as well. Basically add another check to wait on, Right now we wait for this:
Copy code
// block until
job.getStatus() != null && job.getActive() == null
// then return 
return job.getStatus().getSucceeded() != null
Now the change will be
Copy code
// block until
job.getStatus() != null && job.getActive() == null && (job.getStatus().getFailed() != null || job.getStatus().getSucceeded() !=null)
// then return 
return job.getStatus().getSucceeded() != null
This should keep things backwards compatible and working in all versions of k8s
Open in Slack
PreviousNext
Powered by