I am using livekit cloud but self hosting livekit agents. Would I benefit from livekit inference? What would be the benefit compared to plugins?

Yes, you would still get all the benefits mentioned in https://blog.livekit.io/introducing-livekit-inference/, except the paragraph about global co-location. Specifically for inference: • Simplified setup and management with a single API key to access supported STT, LLM, TTS • Consolidated billing • Ability to quickly switch between different models, e.g. to try out a newly released model. There are still reasons some customers choose to stick with plugins, e.g. support for custom voices (the question immediately above this) but we are working hard to close these gaps. It's not an all-or-nothing... you could use inference for your STT and LLM whilst still using a plugin for your TTS, if you chose to.

So there is no latency benefits with inference if agents are self-hosted, right?

Correct, we recommend hosting in LiveKit cloud to take advantage of co-location, though I understand that isn't always feasible/sensible/priority if you have a known-working self-hosted setup.

In my case, the agent will be calling my server's endpoint multiple times for each turn. For this reaason, I chose to co-locate livekit agent to my server. What is the recommendation for folks like me?

In general, the biggest bottleneck we see for latency is between the STT/LLM/TTS and the agent, as that's where the most traffic is exchanged, and it's been the recommendation to locate these as close as possible since even before inference: https://kb.livekit.io/articles/4490830410-how-can-i-reduce-latency-in-voice-pipeline-agents-using-stt-tts-and-llm. It obviously depends how frequently you are hitting your own server - it might be best to compare the performance between inference and your current setup side-by-side and run your own measurements in a realistic scenario: https://docs.livekit.io/deploy/observability/data/#metrics