i have a working client launching the voice assistant via a LiveKit Community #ask-ai

i have a working client launching the voice assist...

ripe-school-55881

02/13/2025, 2:09 AM

i have a working client launching the voice assistant via a webpage. the agent connects and allows voice assistant interaction through the browser window. When I launch the agent and begin the session, how do i pass the backend information submitted by the user on the we frontend? I currently have working postMessages relaying data to the backend, but I need to be able to pass it to the agent from page.tsx before the voice assistant is launched so it has the correct instructions. can you help me with the calls or api needed on page.tsx to send the instructions to the agent?

gray-monkey-83525

02/13/2025, 2:09 AM

To pass backend information from your frontend to the agent before launching the voice assistant, you can use LiveKit's data messages feature. This allows you to send arbitrary data to the agent, which can include instructions or any other necessary information. 1. Send Data from Frontend: Use the

publishData

method to send data from your frontend to the agent. This can be done before the agent is launched to ensure it has the correct instructions.

Copy code

room.localParticipant.publishData(
  new TextEncoder().encode(JSON.stringify({
    type: 'instructions',
    data: yourData,
    timestamp: new Date().toISOString()
  })),
  {
    reliable: true,
    topic: 'agent_instructions'
  }
);

2. Receive Data on Agent: On the agent side, listen for data messages using the

@chat.on("message_received")

event to process the instructions. For more details, refer to the LiveKit documentation on data messages. Would you like more detailed guidance on implementing this? Sources: - https://livekit-users.slack.com/archives/C07FY8WHGPM/p1730640211.906019 - https://docs.livekit.io/agents/voice-agent/function-calling/ 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

ripe-school-55881

02/13/2025, 2:14 AM

@millions-hair-24956 yes, please send more instructions, and please be aware that the frontend only takes data once per session and is not a chat. I would like the page.tsx code to be able to take the postMessage data from the form submit (since this is already setup and working) and be able to pass it to the agent when starting it up.

gray-monkey-83525

02/13/2025, 2:14 AM

It seems likely that you can achieve this by integrating the data submission process with the LiveKit agent startup sequence. Here's a step-by-step breakdown of how you might implement this: 1. Capture Form Data: Ensure your form submission captures the necessary data and sends it via

postMessage

. This data should be structured in a way that the agent can interpret, such as JSON. 2. Handle
postMessage
in
page.tsx
: Set up an event listener in your

page.tsx

to handle the

postMessage

event. This will allow you to capture the data sent from the form.

Copy code

useEffect(() => {
  window.addEventListener('message', (event) => {
    if (event.origin !== 'your-frontend-origin') return; // Ensure the message is from a trusted source
    const { data } = event;
    if (data.type === 'formData') {
      // Store the data for later use
      setFormData(data.payload);
    }
  });

  return () => {
    window.removeEventListener('message', handleMessage);
  };
}, []);

3. Pass Data to Agent on Startup: When you start the agent, use the stored form data to send instructions to the agent. This can be done using the

publishData

method as part of the agent's initialization process.

Copy code

const startAgent = () => {
  if (formData) {
    room.localParticipant.publishData(
      new TextEncoder().encode(JSON.stringify({
        type: 'instructions',
        data: formData,
        timestamp: new Date().toISOString()
      })),
      {
        reliable: true,
        topic: 'agent_instructions'
      }
    );
  }
  // Code to start the agent
};

4. Initialize Agent: Ensure that the agent is initialized after the data is sent. This ensures that the agent has the necessary context before starting the session. Would you like further clarification on any of these steps? Sources: - https://docs.livekit.io/agents/voice-agent/function-calling/ 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

ripe-school-55881

02/13/2025, 2:25 AM

@millions-hair-24956 can you provide more details about step 3? where does this go and is there a link to the API where I can see more about publishData()?

gray-monkey-83525

02/13/2025, 2:26 AM

Certainly! In step 3, the

publishData

method is used to send data from the frontend to the agent. This method is part of the LiveKit SDK and is used to publish arbitrary data messages to other participants in the room, such as your agent. Where to Place the Code The

publishData

call should be placed in the part of your code where you handle the initialization of the agent. This is typically after you have established a connection to the LiveKit room and before you start the agent session. Here's a more detailed example:

Copy code

// Assuming you have a function to connect to the room
async function connectToRoom() {
  const room = await connectToLiveKitRoom(); // Your function to connect to the room

  // Publish data to the agent
  if (formData) {
    room.localParticipant.publishData(
      new TextEncoder().encode(JSON.stringify({
        type: 'instructions',
        data: formData,
        timestamp: new Date().toISOString()
      })),
      {
        reliable: true,
        topic: 'agent_instructions'
      }
    );
  }

  // Start the agent session
  startAgentSession(room);
}

API Reference For more details on the

publishData

method, you can refer to the LiveKit documentation on data messages. This documentation provides an overview of how to use data messages, including delivery options and usage examples. Would you like more information on any specific part of this process? Sources: - https://docs.livekit.io/home/client/data/messages/ - https://docs.livekit.io/client-sdk-js/classes/LocalParticipant.html#publishData.publishData-1 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

ripe-school-55881

02/13/2025, 2:35 AM

@millions-hair-24956 if I am using the voice-assistant-frontend sample at https://github.com/livekit-examples/voice-assistant-frontend/blob/main/app/page.tsx where is the entrypoint?

gray-monkey-83525

02/13/2025, 2:35 AM

It might be possible that the entry point for the

voice-assistant-frontend

sample is the

Page

component defined in the

page.tsx

file. This component is responsible for setting up the connection to the LiveKit room and initializing the voice assistant. Key Components in
page.tsx
1. Page
Component: This is the main component that sets up the LiveKit room and handles the connection details. It uses hooks like

useVoiceAssistant

to manage the state and audio track of the voice assistant. 2. onConnectButtonClicked
Function: This function is likely responsible for generating room connection details and initiating the connection to the LiveKit room. It is triggered when the user clicks the connect button. 3. LiveKit Components: The component imports and uses several LiveKit components such as

LiveKitRoom

VoiceAssistantControlBar

, and

RoomAudioRenderer

to manage the room and audio interactions. For more details, you can refer to the voice-assistant-frontend GitHub repository. Would you like more detailed guidance on how to modify or extend this setup? Sources: - voice-assistant-frontend GitHub repository 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

ripe-school-55881

02/13/2025, 2:38 AM

@millions-hair-24956 yes, please give more details about these key components and where I would 'initialize' the agent with the data from the user (note I already have that part setup and am using a modified page.tsx, but would like to know how and where to send the potMessage data to the agent)

gray-monkey-83525

02/13/2025, 2:38 AM

It seems likely that you can integrate the data submission process with the LiveKit agent startup sequence in your modified

page.tsx

. Here's a more detailed breakdown of the key components and where you might initialize the agent with the data from the user: Key Components in
page.tsx
1. Page
Component: This is the main component that manages the connection to the LiveKit room. It uses hooks to manage state and handle the connection lifecycle. 2. onConnectButtonClicked
Function: This function is responsible for initiating the connection to the LiveKit room. It typically fetches a token and connects to the room when the user clicks the connect button. 3. LiveKit Components: Components like

LiveKitRoom

VoiceAssistantControlBar

, and

RoomAudioRenderer

are used to manage the room and audio interactions. Initializing the Agent with User Data To send the

postMessage

data to the agent, you can follow these steps: 1. Capture
postMessage
Data: Ensure you have an event listener set up to capture the

postMessage

data. This data should be stored in a state variable for later use.

Copy code

useEffect(() => {
  const handleMessage = (event) => {
    if (event.origin !== 'your-frontend-origin') return; // Ensure the message is from a trusted source
    const { data } = event;
    if (data.type === 'formData') {
      setFormData(data.payload); // Store the data for later use
    }
  };

  window.addEventListener('message', handleMessage);

  return () => {
    window.removeEventListener('message', handleMessage);
  };
}, []);

2. Send Data to Agent on Connection: Modify the

onConnectButtonClicked

function to include the logic for sending the stored form data to the agent using the

publishData

method.

Copy code

const onConnectButtonClicked = async () => {
  try {
    const resp = await fetch(`/api/token?room=${room}&username=${name}`);
    const data = await resp.json();
    setToken(data.token);

    // Connect to the room
    const room = await connectToLiveKitRoom(data.token);

    // Publish data to the agent
    if (formData) {
      room.localParticipant.publishData(
        new TextEncoder().encode(JSON.stringify({
          type: 'instructions',
          data: formData,
          timestamp: new Date().toISOString()
        })),
        {
          reliable: true,
          topic: 'agent_instructions'
        }
      );
    }

    // Start the agent session
    startAgentSession(room);
  } catch (e) {
    console.error(e);
  }
};

gray-monkey-83525

02/13/2025, 2:39 AM

3. Initialize the Agent: Ensure that the agent is initialized after the data is sent. This ensures that the agent has the necessary context before starting the session. Would you like more information on any specific part of this process? Sources: - https://livekit-users.slack.com/archives/C07FY8WHGPM/p1729541882.418149 - https://livekit-users.slack.com/archives/C07FVFGAUKX/p1729110526.781199 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

ripe-school-55881

02/14/2025, 1:37 AM

@millions-hair-24956 can you explain the usage of useLiveKitRoom()

gray-monkey-83525

02/14/2025, 1:37 AM

It seems likely that the

useLiveKitRoom

hook is used to implement the

LiveKitRoom

component or your custom implementation of it. This hook returns a

Room

instance and HTML props that should be applied to the root element of the component. Usage 1. Import the Hook: First, you need to import the

useLiveKitRoom

hook from the LiveKit React components package.

Copy code

import { useLiveKitRoom } from "@livekit/components-react";

2. Initialize the Hook: Use the hook in your component to get the

Room

instance and HTML props.

Copy code

const { room, htmlProps } = useLiveKitRoom();

3. Apply HTML Props: Use the

htmlProps

in your component's root element to ensure proper integration with LiveKit.

Copy code

return <div {...htmlProps}>Your content here</div>;

Properties - serverUrl: The URL to the LiveKit server. It can be a string or undefined. A valid URL is required to establish a connection. - token: A user-specific access token for authenticating to the room. It can be a string or undefined. A valid token is necessary for connection. - audio: Optional. Determines if audio should be published immediately after connecting. - connect: Optional. If set to true, a connection to the LiveKit room is initiated. For more details, you can refer to the LiveKit documentation on `useLiveKitRoom`. Would you like more detailed guidance on implementing this? Sources: - LiveKit documentation on `useLiveKitRoom` 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

ripe-school-55881

02/14/2025, 2:00 AM

@millions-hair-24956 can i use the context returned from @livekit/components-react when using

<LiveKitRoom>

gray-monkey-83525

02/14/2025, 2:00 AM

Yes, you can use the context returned from

@livekit/components-react

when using the

<LiveKitRoom>

component. The

LiveKitRoom

component provides the

RoomContext

to all its child components, allowing them to access the room state and interact with the LiveKit room without needing to pass props down the component tree. How to Use the Context 1. Wrap Components with
<LiveKitRoom>
: Ensure that your components that need access to the room context are wrapped within the

<LiveKitRoom>

component.

Copy code

import { LiveKitRoom, ConnectionState } from '@livekit/components-react';

   function MyApp() {
     return (
       <LiveKitRoom serverUrl="your-server-url" token="your-access-token" connect={true}>
         <ConnectionState />
         {/* Other components that need room context */}
       </LiveKitRoom>
     );
   }

2. Access Context in Child Components: Use hooks like

useRoomContext

to access the room context within child components.

Copy code

import { useRoomContext } from '@livekit/components-react';

   function MyComponent() {
     const room = useRoomContext();

     // Use the room object to interact with the LiveKit room
     console.log(room.state);

     return <div>Room State: {room.state}</div>;
   }

Important Considerations - Ensure Context Availability: Make sure that any component using the room context is a descendant of the

<LiveKitRoom>

component. Otherwise, it will not have access to the context and may throw an error. - Use Provided Hooks: LiveKit provides several hooks like

useRoomContext

useParticipantContext

, etc., to access different parts of the context easily. For more details, refer to the LiveKit documentation on contexts. Would you like more information on using contexts with LiveKit? Sources: - LiveKit documentation on contexts 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

Open in Slack

Previous Next