Is it possible to bring your own API key to use with the generation model for Pinecone Assistants, or is it on the roadmap? Our company has started using LiteLLM as a gateway to track all AI usage, and we'd like to route the assistant's calls through it.

hey, just pinged the assistant team to hop in here

Hey @great-addition-84402 - at the moment, no. But as we release Nexus to the wider public, there are definitely plans to allow for bringing your own model. In this specific case though, why not route your application's calls to Assistant through LiteLLM in order to track?

LiteLLM tracks the token usage and cost by using BYO-API key. I could route the traffic through it and keep using Pinecone's API key, but then LiteLLM wouldn't know the token usage or cost, and that's the whole reason we are using it. The alternative I'm considering, is rebuilding the assistant using the /context endpoint for RAG only, and then use OpenAI for the text generation so that I can get the token usage into LiteLLM.

understood, and that's one option. Nexus will really be the longer-term approach here so we'd be interested in taking you through that and seeing where the integration point with LiteLLM really makes sense

I'd be open to migrating it to Nexus. I'll look over the docs and see how the integration with LiteLLM could work.

It’s still probably a bit early for us to open that up to you, but would be great to engage you with our product and Eng team during this early access.

hey, just wanted to follow up and let you know that I was able to get Pinecone's Assistant API routed through LiteLLM by switching over to the OpenAI-compatable endpoint and creating a custom LLM in LiteLLM with the Pinecone host address. So the BYO-API Key is not required to use Pinecone with LiteLLM if you use the OpenAI -compatible endpoint. This means you loose the citations in the response that Pinecone adds, but everything else works.

hey thanks @great-addition-84402 . Is there anything that would make sense as a pull request against LiteLLM directly to make this better/easier for the future?

Not sure about a PR, but they have this form to request to add new providers. https://models.litellm.ai/providers I got it to work using the 'OpenAI-like' provider, once I refactored the code to use Pinecone's OpenAI-compatible endpoint. I think if I had started with that instead of the Pinecone default, I would have gotten it working in LiteLLM a lot faster/easier. At first I was trying to add Pinecone as a new proivder but keeping the default Pinecone API format and nothing would work.

@mammoth-lighter-94680 / @brainy-leather-20096 is this something we can look into from our side as well?