explain how wordtokenizer and sentence tokenizer works in transcription in voicepipeline

Thanks for reaching out, Harshita. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

so is it use before llm

Yes, the word tokenizer and sentence tokenizer are used before the content is passed to a large language model (LLM) in the transcription pipeline. Their roles are as follows: • Sentence Tokenizer: Determines when a transcript is finalized by identifying complete sentences. • Word Tokenizer: Breaks speech into individual words to simulate interim transcription results before a complete sentence is formed. These processes help structure the transcription data effectively before any further processing, such as by an LLM. Sources: livekit.agents.multimodal API documentation

agent = VoicePipelineAgent( vad=ctx.proc.userdata["vad"], stt=ctx.proc.userdata["stt"], llm=ctx.proc.userdata["llm"], tts=ctx.proc.userdata["tts"], # Using the prewarmed OpenAI TTS min_endpointing_delay=0.0001, interrupt_min_words=0, # noise_cancellation=noise_cancellation.NC(), # max_endpointing_delay=3, # allow_interruptions=True, # interrupt_speech_duration=0.5, chat_ctx=initial_ctx, preemptive_synthesis=True, # before_llm_cb=truncate_context, transcription=AgentTranscriptionOptions( user_transcription=True, # Enable user STT transcriptions agent_transcription=True, # Enable AI-generated transcriptions agent_transcription_speed=1.0, # Control transcription processing speed sentence_tokenizer=ctx.proc.userdata["sentence_tokenizer"], word_tokenizer=ctx.proc.userdata["word_tokenizer"], hyphenate_word=lambda word: word.replace("-", " "), # Optional: Modify word tokenization rules ), # before_tts_cb=_before_tts_cb, fnc_ctx=CallActions(api=ctx.api, participant=participant, room=ctx.room), ) is this the right way im using it