https://livekit.io logo
Join Slack
Powered by
what's the best place to emit a metric on whether ...
# ask-ai
p
what's the best place to emit a metric on whether the agent succeeded/failed?
t
Thanks for reaching out, Mario. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — great question. You're asking where to emit a metric that indicates agent success or failure. Here are the best options and when to use each: • Per-turn success/failure: emit in 
on_user_turn_completed()
. Use this when each agent response should be evaluated (e.g., whether the turn achieved its goal or captured required info). • Session-level success/failure: emit in 
on_exit()
. Use this when you evaluate overall task completion at the end of a session (e.g., order placed, issue resolved). • Component or runtime failures: emit from error event listeners (LLM, TTS, STT) or from catch/except blocks around tool calls. Use this to capture exceptions, tool failures, or downstream errors. Emit metrics via the session event system (e.g., 
session.emit("metrics_collected", CustomMetrics(...))
) so they can be handled by a listener that forwards them to your monitoring/analytics backend. If you want, I can: • Show a short snippet for your preferred hook (turn vs session vs error), or • Suggest metric fields (labels) to include for better analysis. Which would you like? Sources: Creating and Publishing Custom Metrics
p
Yeah, show me on_exit and session.emit in Python
t
Here's a concise Python example showing 
on_exit
emitting a success/failure metric via `session.emit`:
Copy code
from typing import Any

# Example custom metric/event classes (adjust to your app's event types)
class CustomMetrics:
    def __init__(self, label: str, success: bool, reason: str | None = None):
        self.label = label
        self.success = success
        self.reason = reason

class CustomMetricsCollectedEvent:
    def __init__(self, metrics: CustomMetrics):
        self.metrics = metrics

class MyAgent:
    def __init__(self, session: Any):
        self.session = session
        self._task_completed = False  # set this during the session when task succeeds

    async def on_exit(self):
        # Decide success/failure based on your agent state
        if self._task_completed:
            metrics = CustomMetrics(label="agent_success", success=True)
        else:
            metrics = CustomMetrics(label="agent_success", success=False, reason="task_incomplete")

        # Emit the metric for downstream listeners/monitoring
        self.session.emit("metrics_collected", CustomMetricsCollectedEvent(metrics=metrics))
If you want, I can: • Add an example listener that receives 
metrics_collected
and forwards to logging/Prometheus/DataDog, or • Show how to set 
self._task_completed
in typical workflows. Which would you prefer?
p
Is there a way to listen on session.on_exit?
t
Short answer: No — there is no 
session.on_exit
event to listen to. 
on_exit()
is an Agent lifecycle method you override in your Agent subclass. What to use instead: • Override 
on_exit(self)
in your Agent to run cleanup or emit metrics when the agent relinquishes control. • If you need session-wide shutdown handling, use 
ctx.add_shutdown_callback()
in the job context or listen for the session 
close
event. Example: overriding 
on_exit
and emitting a metric via 
session.emit
(Python)
Copy code
class MyAgent(Agent):
    def __init__(self, session):
        super().__init__(session)
        self._task_completed = False

    async def on_exit(self):
        # determine success/failure
        metrics = CustomMetrics(label="agent_success", success=self._task_completed,
                                reason=None if self._task_completed else "task_incomplete")
        # emit to session listeners or monitoring pipeline
        self.session.emit("metrics_collected", CustomMetricsCollectedEvent(metrics=metrics))
If you want an example listener that receives 
metrics_collected
and forwards to logging or a monitoring backend, tell me which backend (e.g., Prometheus, DataDog, simple logging) and I’ll provide it. Sources: Pipeline nodes and hooks | LiveKit docs | Job lifecycle | LiveKit docs | Agent session | LiveKit docs
2 Views
Open in Slack
PreviousNext
Powered by