how does this plan sound? This plan describes our ideas on how to provide conversation history context to the voice agent, allowing for natural continuity between text chat and voice interactions. The implementation uses participant metadata to pass conversation history from the TypeScript token generator to the Python voice agent. ### Key Benefits - Voice agent can reference previous text conversations naturally - No additional API calls or authentication needed in Python - Simple, reliable implementation following LiveKit patterns - Minimal changes to existing code ### Success Metrics - Agent references previous conversation topics - Context available immediately when voice call starts - No performance degradation - Graceful fallback if history unavailable ## Architecture Overview ### Current Flow

1. User starts voice call
2. React Native app requests token from Medplum bot
3. Token generator creates JWT with basic metadata (patientId, threadId, patientName)
4. App connects to LiveKit with token
5. Python agent receives participant metadata
6. Agent starts conversation with no prior context

### Enhanced Flow with Conversation History

1. User starts voice call
2. React Native app requests token from Medplum bot
3. Token generator:
   - Fetches recent conversation history from thread
   - Formats messages as role/content pairs
   - Includes history in participant metadata
4. App connects to LiveKit with enhanced token
5. Python agent:
   - Parses conversation history from metadata
   - Creates ChatContext with history messages
   - Prepends messages as actual conversation
6. Agent starts with full conversation awareness

## Implementation Details ### Part A: Enhance Token Generator Bot (TypeScript) File:

/Users/ajwwong/v1/progress/src/bots/livekit-token-generator.ts

Add conversation history retrieval:

typescript
import { BotEvent, MedplumClient } from '@medplum/core';
import { AccessToken } from 'livekit-server-sdk';
import { Communication } from '@medplum/fhirtypes';

_// Add this function to retrieve conversation history_
async function getConversationHistory(
  _medplum_: MedplumClient,
  _threadId_: string,
  _limit_: number = 15
): Promise<Array<{ role: string; content: string }>> {
  try {
    _// Search for recent messages in the thread_
    const messages = await medplum.searchResources('Communication', {
      'part-of': `Communication/${threadId}`,
      _sort: '-sent',
      _count: limit
    });

    _// Filter and format messages_
    return messages
      .filter(_msg_ => {
        _// Skip voice-related metadata messages_
        const hasVoiceExtension = msg.extension?.some(_ext_ =>
          ext.url?.includes('voice-call') ||
          ext.url?.includes('voice-transcript')
        );
        _// Only include messages with text content_
        return !hasVoiceExtension && msg.payload?.[0]?.contentString;
      })
      .map(_msg_ => ({
        role: msg.sender?.reference?.startsWith('Patient/') ? 'user' : 'assistant',
        content: msg.payload[0].contentString.trim().substring(0, 500) _// Limit length_
      }))
      .reverse(); _// Return in chronological order (oldest first)_
  } catch (error) {
    console.error('Failed to get conversation history:', error);
    return []; _// Return empty array on error_
  }
}

export async function handler(
  _medplum_: MedplumClient,
  _event_: BotEvent
): Promise<any> {
  const input = event.input as { threadId?: string; patientId?: string };
  const { threadId, patientId } = input;

  if (!threadId || !patientId) {
    return {
      success: false,
      error: 'Missing threadId or patientId',
    };
  }

  try {
    _// Get LiveKit credentials from bot secrets_
    const apiKey = event.secrets?.['LIVEKIT_API_KEY']?.valueString;
    const apiSecret = event.secrets?.['LIVEKIT_API_SECRET']?.valueString;

    if (!apiKey || !apiSecret) {
      throw new Error('Missing LiveKit credentials in bot configuration');
    }

    _// Get patient name for better context_
    let patientName = 'User';
    try {
      const patient = await medplum.readResource('Patient', patientId);
      patientName = patient.name?.[0]?.given?.[0] || 'User';
    } catch (error) {
      console.error('Failed to get patient name:', error);
    }

    _// Get conversation history_
    const conversationHistory = await getConversationHistory(medplum, threadId);
    console.log(`Retrieved ${conversationHistory.length} messages for context`);

    _// Get LiveKit URL_
    const livekitUrl = event.secrets?.['LIVEKIT_WS_URL']?.valueString || '<wss://progress-notes-axnubc06.livekit.cloud>';

    _// Create token with participant metadata including conversation history_
    const token = new AccessToken(apiKey, apiSecret, {
      identity: patientId,
      metadata: JSON.stringify({
        patientId,
        threadId,
        patientName,
        conversationHistory _// Add conversation history here_
      }),
    });

    _// Generate unique room name for each session to ensure new agent spawns_
    const roomName = `${threadId}-${Date.now()}`;

    _// Grant permissions for voice room_
    token.addGrant({
      room: roomName,
      roomJoin: true,
      roomCreate: true,
      canPublish: true,
      canSubscribe: true,
      canPublishData: true,
    });

    _// Token expires in 6 hours_
    const jwt = await token.toJwt();

    return {
      success: true,
      token: jwt,
      url: livekitUrl,
    };
  } catch (error) {
    console.error('Error generating LiveKit token:', error);
    return {
      success: false,
      error: error instanceof Error ? error.message : 'Unknown error occurred',
    };
  }
}

### Part B: Update Python Voice Agent File:

/Users/ajwwong/v1/cora-voice-agent/agent.py

Modify the agent to use conversation history with ChatContext:

python
_import_ asyncio
_import_ os
_import_ json
_import_ logging
_import_ re
_from_ datetime _import_ datetime
_from_ dotenv _import_ load_dotenv
_from_ typing _import_ AsyncIterable

_from_ livekit _import_ agents
_from_ livekit.agents _import_ Agent, AgentSession, JobContext, WorkerOptions, cli, ModelSettings, tokenize, llm
_from_ livekit.plugins _import_ deepgram, anthropic, silero
_from_ livekit.plugins.turn_detector.multilingual _import_ MultilingualModel

_# ... existing imports and DeepgramTTSWrapper ..._

class CoraAssistant(Agent):
    def __init__(_self_, _chat_ctx_: llm.ChatContext = None):
        _# Keep original instructions, pass chat context_
        super().__init__(
            _chat_ctx_=chat_ctx _or_ llm.ChatContext(),
            _instructions_="""You are Cora, a supportive AI companion."""
        )

    _# ... existing tts_node method ..._

async def entrypoint(_ctx_: JobContext):
    """Main entry point for the agent"""
    <http://logger.info|logger.info>(f"\n=== AGENT ENTRYPOINT CALLED ===")
    <http://logger.info|logger.info>(f"Room: {ctx.room.name}")
    <http://logger.info|logger.info>(f"Job ID: {ctx.job.id _if_ hasattr(ctx, 'job') _else_ 'N/A'}")
    <http://logger.info|logger.info>(f"Agent PID: {os.getpid()}")
    <http://logger.info|logger.info>("===============================\n")

    _# Parse room metadata if provided_
    room_metadata = {}
    thread_id = None
    patient_id = None
    patient_name = "User"
    conversation_history = []

    _if_ hasattr(ctx.room, 'metadata') and ctx.room.metadata:
        _try_:
            room_metadata = json.loads(ctx.room.metadata)
            thread_id = room_metadata.get('threadId')
            patient_id = room_metadata.get('patientId')
            patient_name = room_metadata.get('patientName', 'User')
            conversation_history = room_metadata.get('conversationHistory', [])
            <http://logger.info|logger.info>(f"Parsed room metadata: threadId={thread_id}, patientId={patient_id}, patientName={patient_name}")
            <http://logger.info|logger.info>(f"Conversation history: {len(conversation_history)} messages")
        _except_ json.JSONDecodeError _as_ e:
            logger.error(f"Failed to parse room metadata: {e}")
            logger.error(f"Raw metadata: {ctx.room.metadata}")
    _else_:
        <http://logger.info|logger.info>("No room metadata provided")

    _# Try to get metadata from participants as fallback_
    _if_ not thread_id or not patient_id:
        <http://logger.info|logger.info>("Checking participant metadata as fallback...")
        participants = ctx.room.remote_participants
        <http://logger.info|logger.info>(f"Found {len(participants)} remote participants")
        _for_ participant_id, participant _in_ participants.items():
            <http://logger.info|logger.info>(f"Checking participant {participant_id} (identity: {participant.identity})")
            _if_ participant.metadata:
                _try_:
                    participant_metadata = json.loads(participant.metadata)
                    thread_id = thread_id or participant_metadata.get('threadId')
                    patient_id = patient_id or participant_metadata.get('patientId')
                    patient_name = patient_name _if_ patient_name != "User" _else_ participant_metadata.get('patientName', 'User')
                    conversation_history = conversation_history or participant_metadata.get('conversationHistory', [])
                    <http://logger.info|logger.info>(f"Found participant metadata: threadId={thread_id}, patientId={patient_id}, patientName={patient_name}")
                    <http://logger.info|logger.info>(f"Conversation history from participant: {len(conversation_history)} messages")
                    _break_
                _except_ json.JSONDecodeError:
                    logger.error(f"Failed to parse participant metadata: {participant.metadata}")
            _else_:
                <http://logger.info|logger.info>(f"Participant {participant_id} has no metadata")

    _# Build ChatContext with conversation history_
    initial_ctx = llm.ChatContext()

    _if_ conversation_history:
        <http://logger.info|logger.info>(f"Loading {len(conversation_history)} messages into ChatContext")
        _for_ msg _in_ conversation_history:
            _try_:
                initial_ctx.add_message(
                    _role_=msg.get('role', 'user'),
                    _content_=msg.get('content', '')
                )
            _except_ Exception _as_ e:
                logger.error(f"Failed to add message to context: {e}")
        <http://logger.info|logger.info>("ChatContext populated with conversation history")

    _# Create the assistant with ChatContext_
    assistant = CoraAssistant(_chat_ctx_=initial_ctx)

    _# ... rest of existing entrypoint code ..._

Thanks for reaching out, Albert. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.