### Answer Using a custom LLM with Retell AI requires you to set up and manage your own websocket server and handle the LLM integration yourself [(1) ]() . This means you're responsible for the entire response system, including streaming responses and managing latency . Based on a Discord discussion, one specific challenge users have encountered is function calling synchronization [(2) ]() . The model may speak confirmation phrases like "Got it, you're thinking about April 21st. Let me check our availability real quick. One moment, please." before actually invoking the function call, leading to awkward silence . This happens because the model outputs text instead of immediately calling the required function . The key consideration is latency - your custom LLM's time to first sentence latency (time to first token + time to generate a sentence) directly impacts the overall conversation experience [(1) ]() . You need a low-latency LLM inference to keep interactions smooth . You'll also need to handle all customization yourself, including RAG, internal states, dynamic prompts, and tool calling logic .

​ To learn more, see the sources I considered: [[1] Integrate LLM](https://docs.retellai.com/integrate-llm/integrate-llm) [[2] Custom LLM (GPT 4 Turbo) Function Calling Syncronization](https://discord.com/channels/1186424376300679209/1227360951683252244) ​

Feel free to tag @AI Evy with additional questions. I also appreciate feedback, it helps me improve.

@AI Evy do we also need to handle turn-taking when using a custom LLM?

### Answer No, you don't need to handle turn-taking when using a custom LLM [(1) ]() . Retell AI's proprietary turn-taking model handles this for you automatically [(2) ]() . When you integrate a custom LLM via WebSocket, Retell manages the turn-taking logic and only sends you events when it's appropriate for the agent to respond [(1) ]() . Specifically, Retell sends "response and reminder required events" to your server when it determines the agent should speak . Your server simply needs to respond to these events with the agent's message . Retell's turn-taking model uses advanced contextual understanding and predictive timing to determine when to transition between speakers [(2) ]() . It combines Voice Activity Detection (VAD) with predictive cues to handle interruptions and ensure smooth conversation flow . This all happens on Retell's side before your custom LLM even receives a request [(1) ]() . Your custom LLM integration focuses on generating the content of responses, not managing when those responses should occur . Retell handles the timing, interruption detection, and turn-taking decisions [(2) ]() .

​ To learn more, see the sources I considered: [[1] LLM WebSocket](https://docs.retellai.com/api-references/llm-websocket) [[2] Retell AI's Turn-Taking Model for Seamless Calls | Retell AI](https://www.retellai.com/blog/how-retell-ais-turn-taking-model-ensures-seamless-calls) ​

Feel free to tag @AI Evy with additional questions. I also appreciate feedback, it helps me improve.

Thanks for the feedback, glad I was helpful.

### Answer When using a custom LLM with Retell AI, you'll need to handle the following: [(1) ]() [(2) ]() [(3) ]() 1. Backend WebSocket Server Setup [(1) ]() [(2) ]() You must set up a backend server that handles WebSocket connections from Retell's servers [(1) ]() . Retell will connect to your server at

{your-server-websocket-endpoint}/{call_id}

[(2) ]() . 2. WebSocket Protocol Implementation You need to handle these event types from Retell: -

update_only

events: Real-time transcript updates -

response_required

events: When Retell needs a response from your LLM -

reminder_required

events: When the user hasn't spoken for a while -

ping_pong

events (optional) : For connection health checks 3. LLM Integration and Response Generation [(3) ]() You're responsible for connecting to your chosen LLM provider (OpenAI, Azure OpenAI, etc.) and generating responses . This includes managing the prompt engineering and conversation context . 4. Response Streaming [(2) ]() [(3) ]() You must send responses back to Retell in the correct format, including streaming partial responses for lower latency [(2) ]() . Each response needs these fields:

response_type

,

response_id

,

content

,

content_complete

, and

end_call

. 5. Latency Optimization [(4) ](

tegrate-llm/llm-best-practice) [(3) ]() Your LLM's time to first sentence latency (time to first token + time to generate a sentence) directly impacts overall experience [(3) ]() . You need to select a low-latency LLM and optimize inference [(4) ]() . *6. Function Calling (if needed) * [(2) ]() If you want your agent to perform actions like booking appointments or querying databases, you need to implement function calling logic in your LLM integration . You'll send tool call invocations and results back through the WebSocket . 7. Configuration Management You can optionally send a config event to control auto-reconnect behavior and whether Retell sends call details . Retell handles turn-taking, audio processing, and telephony for you [(1) ]() [(2) ]() . Your focus is on the LLM response generation and any custom business logic [(1) ]() [(3) ]() .

Feel free to tag @AI Evy with additional questions. I also appreciate feedback, it helps me improve.

​ To learn more, see the sources I considered: [[1] Custom LLM Overview](https://docs.retellai.com/integrate-llm/overview) [[2] LLM WebSocket](https://docs.retellai.com/api-references/llm-websocket) [[3] Integrate LLM](https://docs.retellai.com/integrate-llm/integrate-llm) [[4] Custom LLM Best Practices](https://docs.retellai.com/integrate-llm/llm-best-practice) ​

### Answer ### Can You Achieve the Same Latency as Retell with a Custom LLM? Achieving Retell's latency with a custom LLM is challenging. Retell AI's proprietary Retell LLM is specifically fine-tuned for voice interactions with sub-800ms end-to-end latency [(1) ]() . In independent testing, Retell AI achieved 620ms average end-to-end latency and 180ms time-to-first-token . Your custom LLM's time to first sentence latency (time to first token + time to generate a sentence) directly impacts overall experience [(2) ]() . Production voice AI agents typically aim for 800ms or lower latency to maintain conversational flow [(1) ]() . Beyond this threshold, users notice delays leading to conversation overlap, reduced trust, and abandoned interactions . To optimize a custom LLM, you should [(3) ]() : - Keep responses short and concise - Keep prompts concise (longer prompts can harm performance) - Set temperature lower for function calling to boost accuracy - Consider using RAG to filter only relevant information from large knowledge bases - Check LLM benchmarks for latency and throughput, focusing on time to first token + throughput of first sentence ### What Retell Provides Out of the Box with Their LLM When using Retell's built-in LLM frameworks (single prompt, conversation flow) , you get [(4) ]() : 1. Turn-Taking Management Retell's proprietary turn-taking model handles interruption detection and speaker transitions automatically [(1) ]() . 2. Streaming Optimization - Edge depl

oyment with distributed processing to reduce geographic latency - Chunked audio processing to minimize buffering delays - Predictive pre-loading for reduced response preparation time - Adaptive bitrate for performance under network stress 3. Built-in Tool Sets [(4) ]() More capabilities and built-in tool sets compared to custom LLM integration . 4. Low Latency Performance [(1) ]() - 620ms average end-to-end latency - 180ms average time-to-first-token - 140ms average barge-in response - 45ms jitter variance (standard deviation) - 99.7% stream continuity 5. Managed Infrastructure You don't need to set up websocket servers, handle streaming protocols, or manage LLM connections [(4) ]() [(2) ]() . Retell handles audio processing and telephony [(4) ]() . The recommendation is to use Retell's agent frameworks if possible, and only use custom LLM integration if you have specific compliance or use case requirements .

​ To learn more, see the sources I considered: [[1] ](https://www.retellai.com/resources/ai-voice-agent-latency-face-off-2025) [[2] Integrate LLM](https://docs.retellai.com/integrate-llm/integrate-llm) [[3] Custom LLM Best Practices](https://docs.retellai.com/integrate-llm/llm-best-practice) [[4] LLM WebSocket](https://docs.retellai.com/api-references/llm-websocket) ​

Feel free to tag @AI Evy with additional questions. I also appreciate feedback, it helps me improve.

### Answer Based on the available sources, post-call analysis pricing depends on which LLM model you select for the analysis [(1) ]() . **GPT-4o Mini**: Post-call analysis is free . **GPT-4o**: Post-call analysis costs $0.017 per session . This is separate from the per-minute call pricing. The cost is incurred once per call when the analysis is performed, not continuously during the call . Additionally, there's a workspace-level consideration: every workspace includes 10 free knowledge bases . If you need additional knowledge bases (which can be used with post-call analysis) , those are charged at $8/month each, billed at the end of the month .

​ To learn more, see the sources I considered: [[1] Platform Changelogs | Retell AI](https://www.retellai.com/changelog) ​

Feel free to tag @AI Evy with additional questions. I also appreciate feedback, it helps me improve.

### Answer ### Answer The documentation recommends keeping prompts concise to minimize latency [(1) ]() . Longer prompts can actually harm performance . While there isn't a specific character or token limit mentioned, the guidance emphasizes that shorter is better for latency optimization . The key principle is to include only essential information in your prompt. Best practices for prompt length: - Keep prompts concise and focused - If you have a large knowledge base, consider using RAG (Retrieval Augmented Generation) to filter out only the relevant information rather than including everything in the prompt - Make responses short and concise (filler words and some stammering can make the agent more human-like) Additionally, for token-based LLM pricing that went into effect June 1st, prompts up to 3,500 tokens follow the existing flat rate [(2) ]() . Prompts exceeding 3,500 tokens scale proportionally in cost . This pricing structure suggests that keeping prompts under 3,500 tokens is also cost-effective. For function calling specifically, setting the temperature lower can help boost accuracy [(1) ]() , which can indirectly help with latency by reducing errors and retries.

​ To learn more, see the sources I considered: [[1] Custom LLM Best Practices](https://docs.retellai.com/integrate-llm/llm-best-practice) [[2] Pricing Changes, Latency Improvements, One-Time SMS, and More](https://www.retellai.com/changelog/pricing-changes-latency-improvements-one-time-sms-and-more) ​

Feel free to tag @AI Evy with additional questions. I also appreciate feedback, it helps me improve.

### Answer Backchannel is not deprecated and is fully available via the API [(1) ]() [(2) ]() [(3) ]() . You can enable and configure it using the following parameters [(1) ]() [(2) ]() : **`enable_backchannel`**: Set to

true

to enable backchannel (defaults to

false

) [(1) ]() [(2) ]() [(3) ]() **`backchannel_frequency`**: Controls how often backchannel triggers when possible, ranging from 0 (never) to 1 (always) . Default is 0.8 **`backchannel_words`**: Customize which words the agent uses for backchannel (like "yeah", "uh-huh") . If not set, default words apply based on language and voice provider The feature works by having the agent interject with phrases during user speech to show interest and engagement . Backchannel tends to appear more in longer user utterances [(2) ]() [(3) ]() . If you're not seeing it in the dashboard UI, you can still configure it through the [Update Voice Agent]() API endpoint [(2) ]() .

​ To learn more, see the sources I considered: [[1] Add backchannel](https://docs.retellai.com/build/interaction-configuration) [[2] Update Voice Agent](https://docs.retellai.com/api-references/update-agent) [[3] Get Voice Agent](https://docs.retellai.com/api-references/get-agent) ​

Feel free to tag @AI Evy with additional questions. I also appreciate feedback, it helps me improve.

Thanks for the feedback, glad I was helpful.