@numerous-tailor-79021 You also mention you're using one index per coach. It may be better to segregate by namespace as multitenancy is actually one of the purposes of that concept. This is likely not related to the problem you're facing though! Read more here: https://docs.pinecone.io/guides/index-data/implement-multitenancy

This is perfect. I need to learn a lot more about this.

ok this is what I am currently doing for upserting into the index on Pinecone: Here’s how the ingestion project (video_knowledge_base/complete_video_system.py) takes raw video files and feeds Pinecone. 1. Video → transcript/vision features process_video (video_knowledge_base/complete_video_system.py:701-758) drives the pipeline. It registers the video in the interaction linker, then: Transcribes audio with OpenAI Whisper “base” (_transcribe_optimized, 828-844). If enhanced mode is on, it first aligns speakers via pyannote diarization (_transcribe_with_speakers_optimized, 804-827). Extracts frames every few seconds (_extract_frames_optimized, 845-855) and asks GPT‑4o to describe them (analyze_frames_original, 955-990) or a richer variant in the enhanced path.

2. Chunking strategy The default path is the “enhanced” speaker-aware chunker (create_speaker_aware_chunks, 1363-1515). It scans the diarized transcript for semantic boundaries (teaching moments, Q&A, speaker changes) via _detect_speaker_aware_boundaries (1517-1601). Each chunk blends: Speaker-organized transcript (primary/secondary speakers) Visual context and OCR text when available Heuristics like concept type, teaching emphasis, keywords, etc. (metadata block 1448-1505) If you disable the enhanced mode it falls back to simple time slicing (create_chunks_original, 1071-1148), using configurable min duration / overlap.

3. Embedding model Regardless of chunker, embeddings come from Pinecone’s hosted inference API. _initialize_embedding_model (402-407) sets self.inference_model = "multilingual-e5-large" (1024 dims). In the enhanced path, each chunk calls: embeddings_response = self.pc.inference.embed( model=self.inference_model, inputs=[combined_text], parameters={"input_type": "passage"} ) embedding = embeddings_response[0]["values"] (video_knowledge_base/complete_video_system.py:1441-1446). Those 1024‑dim vectors get attached as "values" in the chunk dict (1509-1513). The non-enhanced path would need a similar embedding call before

4. Upserting into Pinecone setup_coach_indexes (474-497) ensures a serverless cosine index per coach (jared-dating-advice-v2, albert-destrade-semantic-enhanced-kb, etc.) with dimension 1024 in AWS us-east-1. store_chunks (1152-1164) batches up the chunk dictionaries and calls index.upsert(vectors=batch), so Pinecone receives {id, values, metadata} for each chunk. Metadata includes timestamps, speaker info, visual flags, teaching scores, etc., which later power filtering and display.

5. Retrieval side (for completeness) When the server runs, search_pinecone (dating-advice-fastmcp/server.py:360-426) embeds user queries with the same multilingual-e5-large inference endpoint and queries the coach’s index. It unwraps the Pinecone response into the normalized chunk structure the widget consumes, with your metadata carried over.

In short: Whisper (audio) + GPT‑4o (vision) generate rich chunk text/metadata; Pinecone Inference (multilingual-e5-large, 1024-dim) embeds it; index.upsert stores the vectors plus metadata in per-coach serverless indexes.