https://livekit.io logo
Join Slack
Powered by
is there a way to make the agent pass a JSON to `o...
# ask-ai
m
is there a way to make the agent pass a JSON to 
on_user_turn_completed
? the idea is that I want the LLM to always return a JSON, not text. And I want the agent to be able to access the JSON response. tis there a way to make
t
Thanks for reaching out, Anas. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Yes, the agent can pass a JSON object to the 
on_user_turn_completed
method. This method allows you to modify the conversation context before the agent generates a response. Here's how you can achieve this: 1. Modify the 
on_user_turn_completed
Method: You can override the method to include JSON data in the conversation context. For example:
Copy code
async def on_user_turn_completed(self, turn_ctx: ChatContext, new_message: ChatMessage) -> None:
    # Create a JSON payload
    json_data = {"key": "value"}
    # Add the JSON data as a message in the context
    turn_ctx.add_message(
        role="assistant",
        content=[json.dumps(json_data)]
    )
    await super().on_user_turn_completed(turn_ctx, new_message)
2. Accessing the JSON Data: The JSON data can be accessed from the 
ChatContext
in subsequent parts of the conversation:
Copy code
# In the LLM's response handling
for message in turn_ctx.items:
    if message.role == "assistant" and message.type == "message":
        try:
            json_content = json.loads(message.content[0])
            # Use the JSON data as needed
        except json.JSONDecodeError:
            pass
3. Ensuring the LLM Always Returns JSON: To ensure the LLM always returns JSON, structure its prompts to request JSON responses. For example:
Copy code
# Example system message
system_message = """
You are a helpful assistant that always returns responses in JSON format.
Always format your responses as JSON objects.
"""

# Add this message to the ChatContext
turn_ctx.add_message(role="system", content=[system_message])
Notes: • Ensure proper serialization and deserialization of JSON data when adding or retrieving it from the context. • Handle potential JSON decoding errors when parsing responses from the LLM. By following these steps, you can pass JSON data through 
on_user_turn_completed
and ensure the LLM processes and returns JSON responses as needed. Sources: tests/test_agent_session.py | External data and RAG | LiveKit Docs
m
is there a way to force the OpeanAI 
RealtimeModel
to generate JSON instead of text?
t
Hey there! Just a heads-up: the OpenAI 
RealtimeModel
in LiveKit doesn’t directly generate JSON. It’s mainly built to give you text or audio outputs. But no worries—you can set it up to use the text mode and then easily turn that text into JSON if you need to. Here’s a simple way to use text mode and convert the output to JSON: 1. First, set the model to use the text modality: 2. Then, take the text output and parse it into JSON like this: Quick tip: Make sure the text the model generates is valid JSON so you don’t run into parsing errors. You can help with this by giving clear instructions in your prompt to encourage JSON-like responses.
m
how do I intercept the LLM output before it goes into the TTS in order for me to strip away the json and only pass the spoken text to the TTS model?
t
Hey there! If you want to catch the LLM output before it heads into the TTS and clean out any JSON, you can use the 
before_tts_cb
(that’s the before TTS callback) in the 
VoicePipelineAgent
. This gives you a chance to tweak the text before it gets sent to the TTS engine. Here’s how you can do it: 1. Define the Callback Function: 2. Set up the `VoicePipelineAgent`: This way, you make sure the JSON is stripped out from the LLM output, so only the spoken text gets passed along to the TTS engine. Quick heads-up: In newer versions of LiveKit, the 
before_tts_cb
has been swapped out for the 
tts_node
method. If you’re working with a newer version, just let me know—I’m happy to help you with the updated info! Sources: chain-of-thought-tts/agent.py | Agents v0.x migration guide | LiveKit Docs
Open in Slack
PreviousNext
Powered by