How does third party LLM + pinecone work?
I just ...
# 🧶︱generalc
colossal-actor-72431
09/09/2025, 11:23 PMHow does third party LLM + pinecone work?
I just spent $7k on a powerful PC that can run large local models to avoid API costs
m
mammoth-lighter-94680
09/09/2025, 11:24 PMYou could use Pinecone and a local LLM too! Run inference locally to avoid API costs, then just hit Pinecone for the context you need
mammoth-lighter-94680
09/09/2025, 11:24 PMWhoa, that's a beefy machine. Are you planning to serve a lot of requests locally from your machine?
mammoth-lighter-94680
09/09/2025, 11:25 PMThird Party LLM + Pinecone means whenever you have a query you:
1. query against Pinecone to find relevant context
2. pass context + system prompt + query to Open AI or Cohere or Anthropic to generate a response conditioned on that context
mammoth-lighter-94680
09/09/2025, 11:26 PMmammoth-lighter-94680
09/09/2025, 11:26 PMIf you wanted to do local, I guess you could run ollama or mlx or whatever, then just have an application hit your LLM after retrieving context from Pinecone. People do this with embedding models when testing; just run embedding models locally, then deploy once you are raedy
c
colossal-actor-72431
09/10/2025, 10:53 AMWill be using the local model for development of tools and for me to run them for free locally.
Once they are built and ready for monetisation, we will set everything up properly
m
mammoth-lighter-94680
09/10/2025, 3:55 PMNeat! That must be pretty fun to setup and get going! What model are you running locally? And yeah, using Pinecone will get super helpful once you are ready.
I'm not actually sure what people use locally to train; I've heard good things about huggingface libraries that makes setting up trainers easy, otherwise it really depends on what you are working with hardware-wise and what training objective you have.
Inference, i've used ollama and mlx both are quite nice!