https://livekit.io logo
Join Slack
Powered by
How can we use this prewarm function from LLM clas...
# ask-ai
b
How can we use this prewarm function from LLM class to prewarm LLM? The class is present in llm/llm.py file
Copy code
class LLM(
    ABC,
    rtc.EventEmitter[Union[Literal["metrics_collected", "error"], TEvent]],
    Generic[TEvent],
):
    def __init__(self) -> None:
        super().__init__()
        self._label = f"{type(self).__module__}.{type(self).__name__}"

    @property
    def label(self) -> str:
        return self._label

    @abstractmethod
    def chat(
        self,
        *,
        chat_ctx: ChatContext,
        tools: list[FunctionTool | RawFunctionTool] | None = None,
        conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS,
        parallel_tool_calls: NotGivenOr[bool] = NOT_GIVEN,
        tool_choice: NotGivenOr[ToolChoice] = NOT_GIVEN,
        extra_kwargs: NotGivenOr[dict[str, Any]] = NOT_GIVEN,
    ) -> LLMStream: ...

    def prewarm(self) -> None:
        """Pre-warm connection to the LLM service"""
        pass

    async def aclose(self) -> None: ...

    async def __aenter__(self) -> LLM:
        return self

    async def __aexit__(
        self,
        exc_type: type[BaseException] | None,
        exc: BaseException | None,
        exc_tb: TracebackType | None,
    ) -> None:
        await self.aclose()
t
Thanks for reaching out, Harsh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
The 
prewarm()
function in the 
LLM
class is intended to pre-warm the connection to the LLM service. In the base class, this method is defined but does nothing by default—it simply contains a 
pass
statement. To use it, you would call 
prewarm()
on your LLM instance:
Copy code
llm_instance = SomeLLMProvider()  # Your specific LLM implementation
llm_instance.prewarm()
However, for this to have any effect, the specific LLM provider implementation must override the 
prewarm()
method with actual logic to establish or warm up the connection. If you are using a custom or third-party LLM provider, check whether it provides its own implementation of 
prewarm()
. This method is distinct from process-level prewarm functions (such as those used to load models in worker processes), and is specifically for pre-warming LLM service connections at the class or instance level. Sources: examples/avatar_agents/bithuman/agent_worker.py | LLM Integrations | livekit/livekit_composite | DeepWiki | agent/main.py | agent_extensions/agent_extensions/handlers/wake_word_handler.py | agent-examples/agent_examples/pipeline_agent.py
b
Does google gemini models have any prewarming abilities?
t
Hey there! Let me break down something interesting about Google Gemini LLM models. So, they've got this 
prewarm()
method that comes from their base LLM class, but here's the thing - it doesn't really do much. It's like having a button that looks functional but doesn't actually press anything. 😊 Just to clarify, when the docs talk about prewarming, they're actually referring to process-level prewarming for agent workers, not anything specific to the LLM itself. Bottom line: right now, there's no real prewarming magic happening for Google Gemini LLM models at the class level. Hope this helps clear things up! Sources: livekit-plugins/livekit-plugins-google/livekit/plugins/google/llm.py | Anatomy of an Agent | LiveKit Docs
2 Views
Open in Slack
PreviousNext
Powered by