how can I detect the agents transcription and the ...
# ask-aic
chilly-soccer-432
07/24/2025, 7:17 PMhow can I detect the agents transcription and the users transcriptions properly and attribute them to the correct owner in the react frontend
t
tall-airline-16756
07/24/2025, 7:17 PMThanks for reaching out, Aaron. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
tall-airline-16756
07/24/2025, 7:18 PMTo detect and attribute agent and user transcriptions correctly in a React frontend using LiveKit, you can use the following approaches:
1. Using
useVoiceAssistantuseVoiceAssistantuseTrackTranscription Copy code
import { useTrackTranscription, useVoiceAssistant } from "@livekit/components-react";
import { useMemo } from "react";
import useLocalMicTrack from "./useLocalMicTrack";
export default function useCombinedTranscriptions() {
const { agentTranscriptions } = useVoiceAssistant();
const micTrackRef = useLocalMicTrack();
const { segments: userTranscriptions } = useTrackTranscription(micTrackRef);
const combinedTranscriptions = useMemo(() => {
return [
...agentTranscriptions.map((val) => {
return { ...val, role: "assistant" };
}),
...userTranscriptions.map((val) => {
return { ...val, role: "user" };
}),
].sort((a, b) => a.firstReceivedTime - b.firstReceivedTime);
}, [agentTranscriptions, userTranscriptions]);
return combinedTranscriptions;
} Copy code
export function TranscriptionTile({
agentAudioTrack,
accentColor,
}: {
agentAudioTrack?: TrackReferenceOrPlaceholder;
accentColor: string;
}) {
const agentMessages = useTrackTranscription(agentAudioTrack || undefined);
const localParticipant = useLocalParticipant();
const localMessages = useTrackTranscription({
publication: localParticipant.microphoneTrack,
source: Track.Source.Microphone,
participant: localParticipant.localParticipant,
});
// Attribution logic
function segmentToChatMessage(
s: TranscriptionSegment,
existingMessage: ChatMessageType | undefined,
participant: Participant,
): ChatMessageType {
const msg: ChatMessageType = {
message: s.final ? s.text : `${s.text} ...`,
name: participant instanceof LocalParticipant ? "You" : "Agent",
isSelf: participant instanceof LocalParticipant,
timestamp: existingMessage?.timestamp ?? Date.now(),
};
return msg;
}
} Copy code
useEffect(() => {
if (!room) {
return;
}
const updateRawSegments = (
segments: TranscriptionSegment[],
participant?: Participant,
publication?: TrackPublication,
) => {
setRawSegments((prev) => {
const newSegments = { ...prev };
for (const segment of segments) {
newSegments[segment.id] = { segment, participant, publication };
}
return newSegments;
});
};
room.on(RoomEvent.TranscriptionReceived, updateRawSegments);
return () => {
room.off(RoomEvent.TranscriptionReceived, updateRawSegments);
};
}, [room]);ParticipantTrackuseVoiceAssistantuseTrackTranscriptionfirstReceivedTimec
chilly-soccer-432
07/24/2025, 8:01 PMtranscripts dont properly attribute to the correct speaker. How can I ensure this happens?
t
tall-airline-16756
07/24/2025, 8:02 PMTo make sure your transcripts are correctly linked to the right speaker in LiveKit, I've got some friendly tips to help you out! 1. Enable Speaker Diarization If you're working with an awesome STT provider like Speechmatics, you can easily distinguish between speakers. Here's a quick way to do it:
Copy code
stt = speechmatics.STT(
transcription_config=speechmatics.types.TranscriptionConfig(
language="en",
operating_point="enhanced",
enable_partials=True,
max_delay=0.7,
diarization="speaker",
speaker_diarization_config={"max_speakers": 2}, # Adjust as needed
)
) Copy code
@function_tool
async def set_speaker_name(self, context: RunContext[SpeechmaticsUserData], speaker_id: str, name: str):
context.userdata.speaker_names[speaker_id] = name
return f"I'll now refer to speaker {speaker_id} as {name}." Copy code
async def process_stream():
current_speaker_id = None
async for event in parent_stream:
if hasattr(event, 'type') and str(event.type) == "SpeechEventType.FINAL_TRANSCRIPT" and event.alternatives:
transcript = event.alternatives[0].text
speaker_id = event.alternatives[0].speaker_id
if current_speaker_id != speaker_id:
current_speaker_id = speaker_id
speaker_name = context.userdata.speaker_names.get(speaker_id, speaker_id)
modified_transcript = f"{speaker_name}: {transcript}"
event.alternatives[0].text = modified_transcript Copy code
session = AgentSession(llm=openai.realtime.RealtimeModel())
room_io = RoomIO(session, room=ctx.room, participant="participant-identity")
await room_io.start()
await session.start(agent=MyAgent())c
chilly-soccer-432
07/24/2025, 8:03 PMI'm using deepgram, do you have examples with deepgram
t
tall-airline-16756
07/24/2025, 8:06 PMHey there! š Let me walk you through how to make sure your transcripts are perfectly attributed to the right speakers using Deepgram in LiveKit. It's actually pretty straightforward! 1. Enable Diarization in Deepgram Deepgram has this awesome feature called speaker diarization that makes tracking who's saying what super easy. Just set the diarize parameter to true. Check out this simple example:
Copy code
const deepgram = new Deepgram(DEEPGRAM_API_KEY);
const options = {
punctuate: true,
diarize: true, // Enable speaker diarization
model: 'general',
};
const response = await deepgram.transcription.preRecorded({
url: 'audio-file-url',
}, options);
response.results.channels[0].alternatives.forEach((alternative) => {
alternative.words.forEach((word) => {
console.log(`Speaker ${word.speaker}: ${word.word}`);
});
}); Copy code
response.results.channels[0].alternatives.forEach((alternative) => {
alternative.words.forEach((word) => {
console.log(`Speaker ${word.speaker}: ${word.word}`);
});
}); Copy code
const speakerNames = {
0: 'Agent',
1: 'User',
};
response.results.channels[0].alternatives.forEach((alternative) => {
alternative.words.forEach((word) => {
const speakerName = speakerNames[word.speaker] || `Speaker ${word.speaker}`;
console.log(`${speakerName}: ${word.word}`);
});
});3 Views