Are multiple controllers needed for fault tolerance? Or at what scale are multiple controllers needed?

Probably want at least 2 at any production scale. Otherwise losing one means you can’t use the API at all

Yes. At least 2 for all components, preferably 3. Also, multiple controllers mean you need a shared storage (Deepstore or NFS)

Don’t forget running zookeeper clustered as well