Hello there, the docs say that a shared volume is required for controllers if more than one is to be deployed. Can someone shed some light as why this is needed instead of each controller having its own storage? Will all the controllers be active? Will they all write to the volume simultaneously? Any considerations we should take into account in a multi-controller environment? My deployment is on k8s

The shared volume is used for storing golden copy of the data as it is pushed. Typically you want to configure a deepstore (eg S3) for this purpose

Cool, so in fact all the controllers will be active if I understand what you're saying, is this correct?

Yes all controllers will be active to provide fault tolerance. So in case one goes down you will not have unavailability

Thanks for confirming 👍

Hi, adding to this conversation. Currently, I see the controllers running in active/passive mode and just provide fault tolerance. Is there a way to run them in active/active mode so that controllers can share the load as well?

The tables should be distributed across controllers. Can you check the api

leader/tables

?

Here I can see the tables distributed amongst controllers.

cool

The RAM usage has a huge difference in both of my controllers. Hence, I wanted to clear this out.

In ZK check if this is the lead controller. Also check how much if it is heap (and obsolete references)

Hi @Mayank is there a way to define both the controllers in ZK? The load is not being shared amongst our controllers

How many tables do you have? If more than one, the periodic tasks for each should be spread across the controllers.