how do I force openai LLM to always return a json LiveKit Community #ask-ai

Join Slack

how do I force openai.LLM to always return a json?

# ask-ai

mysterious-agent-76276

09/19/2025, 9:58 AM

how do I force openai.LLM to always return a json?

tall-airline-16756

09/19/2025, 9:59 AM

Thanks for reaching out, Anas. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

tall-airline-16756

09/19/2025, 9:59 AM

To help you get a consistent JSON response every time from

openai.LLM

, you can use the

response_format

parameter in your API call. Here’s a friendly guide on how to do that: • Set

response_format={"type": "json_object"}

in your request. This gently nudges the model to return valid JSON. • If you want, you can also define a JSON schema or use a Pydantic model to shape the response even more precisely. • It’s a great idea to include clear instructions in the system message asking the model to return only valid JSON. Here’s a simple example in Python to get you started:

Copy code

response = client.chat.completions.create(
    model="llama3.1-70b",
    messages=[{"role": "system", "content": prompt}],
    response_format={"type": "json_object"}
)

This method helps ensure you receive reliable JSON output from the LLM. If you have any questions or want to explore this further, I’m here to help! Sources: tools/server/README.md | meditation_assistant/assistant_functions.py | livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/llm.py

mysterious-agent-76276

09/19/2025, 10:01 AM

I'm using AgentSession. It seems like openai.LLM() doesn't have a response_format

tall-airline-16756

09/19/2025, 10:02 AM

Hi there! Just to let you know, the

openai.LLM

AgentSession

does support the

response_format

parameter. This is a great way to specify how you'd like the response to come back, like making sure it’s in JSON format. If it’s not showing up for you, it might be worth double-checking that it’s being passed correctly in your API call. Here’s a quick example to guide you:

Copy code

response = agent_session.openai.LLM.chat(
    messages=[{"role": "system", "content": prompt}],
    response_format={"type": "json_object"}
)

If you’re still running into any trouble, please don’t hesitate to reach out—I’m here to help you through it! Sources: Tool definition and use | LiveKit Docs | livekit-agents/livekit/agents/llm/utils.py | Tool definition and use | LiveKit Docs

mysterious-agent-76276

09/19/2025, 10:03 AM

Copy code

llm=openai.LLM(
        ~~~~~~~~~~^
        api_key=self.config.llm.openai_api_key,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        model=self.config.llm.openai_model,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        response_format={"type": "json_object"},
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ),
    ^
TypeError: LLM.__init__() got an unexpected keyword argument 'response_form

tall-airline-16756

09/19/2025, 10:04 AM

It looks like the error is happening because the

response_format

parameter isn’t valid for the

openai.LLM.__init__()

method. No worries, though! You actually want to pass

response_format

as an argument to the

chat()

method instead. Here’s a little example to help you fix it:

Copy code

llm = openai.LLM(
    api_key=self.config.llm.openai_api_key,
    model=self.config.llm.openai_model
)

response = llm.chat(
    messages=[{"role": "system", "content": prompt}],
    response_format={"type": "json_object"}
)

If anything’s unclear or you’d like me to walk you through it further, just let me know—I’m here to help! Sources: livekit-plugins/livekit-plugins-google/livekit/plugins/google/utils.py | Tool definition and use | LiveKit Docs | examples/json_schema_pydantic_example.py

mysterious-agent-76276

09/19/2025, 10:04 AM

doesn't the agent pipeline automatically call llm.chat() ?

tall-airline-16756

09/19/2025, 10:05 AM

Thanks for reaching out! I wanted to share that the agent pipeline in LiveKit doesn’t automatically call

llm.chat()

. Instead, the agent needs to explicitly invoke it when generating a response. For example, in implementations like

listen-and-respond.ts

uninterruptable.ts

, you’ll see that

llm.chat()

is called manually to create a response stream, which is then processed further. If you’re working with a custom pipeline, just make sure to call

llm.chat()

at the right point in your flow. And please don’t hesitate to ask if you’d like some help figuring out exactly where to include it in your setup—I’m here to help! Sources: packages/basics/src/uninterruptable.ts | pipeline-llm/replacing_llm_output.py | livekit-agents/livekit/agents/llm/llm.py

Open in Slack

Previous Next