https://livekit.io logo
Join Slack
Powered by
How to add custom knowledge base for livekit and d...
# ask-ai
f
How to add custom knowledge base for livekit and don't send sensitive information to llm like user mobile, email
t
Thanks for reaching out, Nikhil. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’ll answer both parts: how to add custom knowledge to LiveKit agents, and how to prevent sensitive fields (mobile, email) from being sent to the LLM. What “custom knowledge base” means here • By custom knowledge, LiveKit agents use agent instructions, prompt templates, and job metadata (not model fine-tuning) to inject company-specific facts, policies, or docs at runtime. How to add custom knowledge (high level) • Put static/company rules or FAQs into agent instructions or prompt templates. Use placeholders for runtime values (e.g., 
{{user_name}}
). • Load dynamic context via job metadata when dispatching the job—include only the fields needed to answer the question (e.g., product name, plan tier, recent activity). • If you have larger documents, add a lightweight retrieval step that returns only short, relevant excerpts and pass those excerpts into the prompt. Practical steps 1. Create an agent instruction that describes behavior and knowledge sources (policies, FAQs, product facts). 2. Create prompt templates with placeholders for contextual metadata. 3. When creating a job, pass a metadata object with only non-sensitive fields required for the task. 4. If you must include document excerpts, trim them to the minimum necessary and pass them as context rather than whole documents. Preventing mobile numbers, emails, and other PII from reaching the LLM • Data minimization: do not include phone/email fields in metadata or prompts unless strictly necessary. • Redaction/masking: replace PII with tokens before sending (e.g., 
user_email -> user_123@email
, 
+1-555-0100 -> phone_***0100
). • Use anonymized identifiers: pass internal IDs or hashed values (e.g., 
user_id: "u_ae34"
) instead of raw PII. • Guardrails in instructions: include explicit agent rules to refuse to request or echo PII (for example: "Never include or return user emails or phone numbers in responses."). • Tool/connector hygiene: ensure external tool inputs and headers don’t contain PII; store secrets (API keys) in secret stores, not in prompts or logs. • Logging and observability: disable or filter transcript/log capture of PII. If transcripts are stored, redact PII at ingestion. • Review and test: run sample prompts containing masked PII to confirm the agent does not reveal or reconstruct the original values. When PII is required (only do this if absolutely necessary) • Mask as much as possible; send only the minimal piece needed (e.g., last 4 digits of phone). • Add explicit instructions to the agent to treat that masked value as non-sensitive and not to attempt to reverse it. • Consider an out-of-band flow for sensitive operations (e.g., verified portal or secure API) so the model never sees raw PII. Quick checklist to implement now • [ ] Move company facts into agent instructions/prompts. • [ ] Strip phone/email fields from metadata payloads. • [ ] Replace PII with tokens or hashed IDs. • [ ] Add a “no PII” rule to agent instructions. • [ ] Confirm connectors/tools do not forward PII in headers/payloads. • [ ] Disable or redact PII in logs/transcripts. If you want, tell me: • Which fields you currently pass as metadata and one example metadata object (sanitized) — I’ll suggest exactly what to remove or mask and a sample prompt/template you can use. Sources: Prompting guide | LiveKit docs | LiveKit | Data Processing Addendum | LiveKit | Agent Builder | LiveKit docs
f
In livekit can I build RAG application
t
Yes — you can build a RAG application with LiveKit. What I’m responding to: your question about using LiveKit to implement Retrieval-Augmented Generation (RAG). How to do it (options you can use) • Use the 
on_user_turn_completed
hook to run a retrieval query on the user’s latest message, then inject the retrieved excerpt into the conversation before the model responds. • Connect external data sources (databases, vector stores, knowledge bases) and pass relevant context via job metadata or participant attributes. • Expose a custom tool the model can call to search your knowledge base or run queries at response time. • Use the provided LlamaIndex example to wire a vector store + retriever into an agent for QA-style RAG. Tips and best practices • Return short, relevant excerpts (not whole docs) to limit prompt size and reduce leakage of sensitive data. • Combine RAG with the PII protections we discussed: redact/mask emails and phones in retrieved text, and add agent instructions to never disclose raw PII. • Test retrieval accuracy and prompt templates to avoid hallucinations from ambiguous documents. If you want, tell me: • which data source (S3, DB, vector store, LlamaIndex) you plan to use and whether this is for voice or text — I’ll give a short example snippet and a recommended prompt/template for that setup. Sources: External data and RAG | LiveKit docs
f
Can we use Amazon bedrock for llm
t
Yes — you can use Amazon Bedrock as the LLM provider with LiveKit Agents. What I’m responding to: whether Bedrock can be used as the LLM for LiveKit agents. How to get started (concise) • Install the AWS plugin: 
uv add "livekit-agents[aws]~=1.3"
. • Provide AWS credentials in your environment: 
AWS_ACCESS_KEY_ID
and 
AWS_SECRET_ACCESS_KEY
. • Configure Bedrock in your 
AgentSession
using the plugin, for example:
Copy code
from livekit.plugins import aws

session = AgentSession(
    llm=aws.LLM(
        model="anthropic.claude-3-5-sonnet-20240620-v1:0",
        temperature=0.8,
    ),
    # other components (tts, stt, etc.)
)
• You can choose any foundation model available in Bedrock by changing the 
model
parameter. Helpful links • Amazon Bedrock LLM plugin guide: https://docs.livekit.io/agents/models/llm/plugins/aws/ • AWS + LiveKit integrations overview: https://docs.livekit.io/agents/integrations/aws If you’d like, tell me which Bedrock model you plan to use and whether this is for voice or text — I’ll provide a short example config (STT/TTS + RAG workflow) and notes on PII/redaction for that setup. Sources: Amazon Bedrock LLM plugin guide | LiveKit docs | AWS AI and LiveKit | LiveKit Docs
4 Views
Open in Slack
PreviousNext
Powered by