https://livekit.io logo
Join Slack
Powered by
In the agents framework, is there any way that I c...
# ask-ai
p
In the agents framework, is there any way that I can observe the input and output of the LLM node? If I want to log to an external system the entire LLM request that was sent and the LLM response, what is the best practice to do so?
t
Thanks for reaching out, Marcel. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
You can capture and export the full LLM request and response in the Agents framework. Recommended approaches: 1) Override the 
llm_node
in your Agent • Purpose: Gives full control over the exact chat context, tools, model settings sent to the LLM and allows you to intercept streaming response chunks. • Pattern (conceptual): ◦ Log the outgoing request: chat messages, tools, and model settings. ◦ Call the default 
llm_node
to get a response stream. ◦ Buffer or forward response chunks as they arrive, logging them (or streaming them) to your external system. • Benefit: Complete visibility (request + streamed response) and preserves streaming behavior for downstream pipeline nodes. • Example reference: Pipeline nodes / 
llm_node
docs: https://docs.livekit.io/agents/build/nodes/ 2) Use the 
metrics_collected
event for telemetry • Purpose: Get token counts, latency, and other LLM metrics for monitoring and cost analysis (not full text). • Use case: Send metrics to Datadog, New Relic, etc., for dashboards and alerts. • Docs: Observability data hooks: https://docs.livekit.io/agents/observability/data 3) Forward logs to external observability platforms • Purpose: Send structured logs (from your 
llm_node
override or other logging) to Datadog, Sentry, New Relic, CloudWatch. • How: Configure provider credentials/secrets so runtime logs are forwarded automatically. • Docs: Log collection / forwarding: https://docs.livekit.io/agents/ops/deployment/logs/ 4) (Optional) Add OpenTelemetry tracing • Purpose: Trace the LLM request lifecycle across async operations and correlate traces with logs/metrics. • Use case: End-to-end debugging and performance analysis. • Docs: Observability data hooks (OpenTelemetry section): https://docs.livekit.io/agents/observability/data Best-practice notes • For full request/response capture, override 
llm_node
— this is the canonical method. • Log structured data (JSON) and avoid logging secrets or PII unless you have consent and proper controls. • Stream logs out incrementally if you need near-real-time visibility; buffer only if you must assemble the complete output first. • Combine 
llm_node
logging with metrics and tracing for cost/latency monitoring and end-to-end observability. Would you like a short code example in your preferred language (Python or TypeScript) showing the 
llm_node
override that logs request + streamed response to an external HTTP endpoint? Sources: Pipeline nodes and hooks | LiveKit docs | Observability data hooks | LiveKit docs | Log collection | LiveKit Docs
p
How can I get the current llm being used from the llm_node? I have a fallback adapter I would like to know the current provider and model name
t
You're asking how to read the active LLM provider and model name from inside 
llm_node
(for a 
FallbackAdapter
). You can inspect the agent's 
self.llm
and, if it's a 
FallbackAdapter
, use 
self.llm.inner
to get the currently-active inner LLM. Then read common attributes like 
model
or 
provider
(names vary by adapter). Example (Python):
Copy code
async def llm_node(self, chat_ctx: llm.ChatContext, tools: list[llm.FunctionTool], model_settings: ModelSettings) -> AsyncIterable[llm.ChatChunk]:
    # Get active LLM (unwrap fallback adapter if present)
    active_llm = self.llm.inner if hasattr(self.llm, "inner") else self.llm

    # Extract model and provider (attribute names may vary by adapter)
    model_name = getattr(active_llm, "model", None) or getattr(active_llm, "_model", None)
    provider = getattr(active_llm, "provider", None) or type(active_llm).__name__

    # Use or log the values
    print(f"LLM provider: {provider}, model: {model_name}")

    # Continue with normal behavior, streaming response through
    async for chunk in Agent.default.llm_node(self, chat_ctx, tools, model_settings):
        yield chunk
If your adapters use different attribute names, tell me which adapters you use (e.g., OpenAI, Gemini, DeepSeek) and I can show the exact attribute to read. Sources: Events and error handling | LiveKit docs | Google Gemini LLM | LiveKit Docs | DeepSeek LLM | LiveKit Docs | OpenAI LLM models | LiveKit Docs
p
I'm getting NotGiven
t
Thanks — you're seeing the sentinel value "NotGiven" when trying to read provider/model. That usually means the attribute exists but was set to a placeholder rather than a concrete string. Let's troubleshoot so you can reliably detect the active provider and model. Quick checks to run inside your 
llm_node
(or share these outputs if you want me to inspect): 1. Inspect the active LLM object
Copy code
active_llm = self.llm.inner if hasattr(self.llm, "inner") else self.llm
print("active_llm type:", type(active_llm))
1. Dump available attributes (helps find where real values live)
Copy code
print("attrs:", {k: getattr(active_llm, k) for k in dir(active_llm) if not k.startswith("_")})
1. Check common fields and handle the NotGiven sentinel
Copy code
from livekit.agents import NotGiven  # if that symbol is available in your env

model_name = getattr(active_llm, "model", None) or getattr(active_llm, "_model", None)
provider = getattr(active_llm, "provider", None) or type(active_llm).__name__

# handle sentinel
if model_name == "NotGiven" or model_name is NotGiven:
    model_name = None
if provider == "NotGiven" or provider is NotGiven:
    provider = None

print(f"provider={provider!r}, model={model_name!r}")
What to look for and next steps • If the dump shows attributes with "NotGiven", those are placeholders; the concrete values may be stored under different names (e.g., 
name
, 
model_name
, 
_model
, 
engine
) or only populated when a request is actually dispatched. • If attributes are missing or still placeholders at request time, try inspecting the object right before calling the default 
llm_node
(some adapters assign the active inner LLM only when a call starts). • If you tell me which adapter(s) you use (e.g., OpenAI adapter, Google adapter, a specific fallback adapter name) and paste the small output of the type and attrs dump above, I’ll point to the exact attribute to read and give a minimal code snippet to robustly detect provider+model.
p
These are the attrs I'm getting:
Copy code
attrs: {'aclose': <bound method FallbackAdapter.aclose of <livekit.agents.llm.fallback_adapter.FallbackAdapter object at 0x11fee5400>>, 'chat': <bound method FallbackAdapter.chat of <livekit.agents.llm.fallback_adapter.FallbackAdapter object at 0x11fee5400>>, 'emit': <bound method EventEmitter.emit of <livekit.agents.llm.fallback_adapter.FallbackAdapter object at 0x11fee5400>>, 'label': 'livekit.agents.llm.fallback_adapter.FallbackAdapter', 'model': 'FallbackAdapter', 'off': <bound method EventEmitter.off of <livekit.agents.llm.fallback_adapter.FallbackAdapter object at 0x11fee5400>>, 'on': <bound method EventEmitter.on of <livekit.agents.llm.fallback_adapter.FallbackAdapter object at 0x11fee5400>>, 'once': <bound method EventEmitter.once of <livekit.agents.llm.fallback_adapter.FallbackAdapter object at 0x11fee5400>>, 'prewarm': <bound method LLM.prewarm of <livekit.agents.llm.fallback_adapter.FallbackAdapter object at 0x11fee5400>>, 'provider': 'livekit'}
t
Thanks — with that attrs dump we can see you're inspecting the FallbackAdapter itself (it reports 
model: "FallbackAdapter"
and 
provider: "livekit"
). To get the actual active provider and model used for the current request, read the adapter's active inner adapter instead of the adapter object. Use this robust pattern inside `llm_node`:
Copy code
from livekit.agents.llm.fallback_adapter import FallbackAdapter
from livekit.agents import NotGiven

async def llm_node(self, chat_ctx, tools, model_settings):
    # start with the agent LLM
    effective = self.llm

    # if it's a FallbackAdapter, prefer the currently-active adapter (or fall back to the first configured adapter)
    if isinstance(self.llm, FallbackAdapter):
        effective = getattr(self.llm, "current_adapter", None) or getattr(self.llm, "_adapter_list", [None])[0] or self.llm

    # read model and provider safely, handling NotGiven sentinel
    raw_model = getattr(effective, "model", None) or getattr(effective, "_model", None)
    raw_provider = getattr(effective, "provider", None)

    model_name = None if raw_model in (None, NotGiven, "NotGiven") else str(raw_model)
    provider = None if raw_provider in (None, NotGiven, "NotGiven") else str(raw_provider)

    print(f"Active LLM provider={provider!r}, model={model_name!r}")

    # continue streaming response
    async for chunk in Agent.default.llm_node(self, chat_ctx, tools, model_settings):
        yield chunk
Notes: • 
current_adapter
points to the active inner LLM when a fallback has been chosen; 
_adapter_list[0]
gives the primary configured adapter if none is active yet. • 
NotGiven
and the string "NotGiven" are sentinel placeholders — treat them as missing values. • If you still see placeholders, inspect 
type(effective)
and 
dir(effective)
to find adapter-specific attribute names (
name
, 
engine
, etc.) and read those instead. If you tell me which concrete adapters you configured (e.g., OpenAI, Google, etc.), I can give the exact attribute names to read for provider and model. Sources: Events and error handling | LiveKit docs
p
I configured OpenAI , Antrophic and Google
t
Thanks — since you have OpenAI, Anthropic, and Google adapters behind a FallbackAdapter, read the adapter's active inner adapter (
current_adapter
) and then check adapter-specific fields (handling the 
NotGiven
sentinel). Below is a compact, robust 
llm_node
snippet that covers those three adapters and falls back safely.
Copy code
from livekit.agents.llm.fallback_adapter import FallbackAdapter
from livekit.agents import NotGiven

async def llm_node(self, chat_ctx, tools, model_settings):
    # Resolve effective adapter: current_adapter if fallback chose one, otherwise first configured adapter
    effective = self.llm
    if isinstance(self.llm, FallbackAdapter):
        effective = getattr(self.llm, "current_adapter", None) or getattr(self.llm, "_adapter_list", [None])[0] or self.llm

    # Helper to treat sentinels as missing
    def clean(val):
        return None if val in (None, NotGiven, "NotGiven") else str(val)

    provider = None
    model_name = None

    # Adapter-specific attribute checks
    t = type(effective).__name__.lower()
    if "openai" in t or hasattr(effective, "model") and getattr(effective, "provider", "").lower().startswith("openai"):
        # OpenAI adapters typically expose `model` (or `_model`)
        model_name = clean(getattr(effective, "model", None) or getattr(effective, "_model", None))
        provider = provider or "openai"
    elif "anthropic" in t or getattr(effective, "provider", "").lower().startswith("anthropic"):
        # Anthropic adapters may use `model` or `model_name`
        model_name = clean(getattr(effective, "model", None) or getattr(effective, "model_name", None) or getattr(effective, "_model", None))
        provider = provider or "anthropic"
    elif "google" in t or getattr(effective, "provider", "").lower().startswith("google"):
        # Google adapters may expose `model` or `engine`
        model_name = clean(getattr(effective, "model", None) or getattr(effective, "engine", None) or getattr(effective, "_model", None))
        provider = provider or "google"
    else:
        # Generic checks
        model_name = clean(getattr(effective, "model", None) or getattr(effective, "_model", None) or getattr(effective, "name", None))
        provider = clean(getattr(effective, "provider", None)) or (type(effective).__module__.split(".")[0] if type(effective).__module__ else None)

    print(f"Active LLM provider={provider!r}, model={model_name!r}")

    # Continue streaming response
    async for chunk in Agent.default.llm_node(self, chat_ctx, tools, model_settings):
        yield chunk
Notes: • 
current_adapter
is the active inner adapter once fallback selection occurs; 
_adapter_list[0]
is the primary configured adapter. • Handle 
NotGiven
(and the string "NotGiven") as missing values. • Attribute names vary by adapter; the snippet checks common names used by OpenAI, Anthropic, and Google adapters and falls back to generic 
model
, 
_model
, 
name
, or 
engine
. • If you still see placeholders, run 
print(type(effective), dir(effective))
and paste the output and I’ll point to the exact attribute to read.
2 Views
Open in Slack
PreviousNext
Powered by