Team, We are facing a latency issue where it takes ~4 secs for ai to speak post the user (human) ends speaking. We are seeing in the logs that the LLM (gemini 2.5 flash) is taking ~3 sec to give the first token. What's the way to manage this ? What's the latency everyone else is getting ?

Have you tried GPT 4O instead of gemini 2.5 flash ? you should aim for 800ms latency not more