https://promptfoo.dev/ logo
Question: If we're using Pinecone to get relevant ...
# ❓|questions
a
Specifically for the RAG metrics, the context variable is compulsory for the metric to run. If we're taking the context from Pinecone, how do we pass that in the context variable?
@ian
i
right now you have 2 options: 1) Include the context directly. In other words, choose some examples that reflect the data in pinecone 2) Use a dynamic variable that calls your code to fetch from pinecone. For example
Copy code
py
tests:
  - vars:
      context: file://context_fetcher.py
a
Hey @ian , it seems that the file code is being taken as context directly into the output. We are also providing a python file as a the provider in our yaml. How can we address this?
i
hmm, I can look into this. Semi-related question: What context do you expect to be passed into the python script?
a
Essentially the python file gets these Document objects from Pinecone and the file would return the page content of these document objects as context to the context variable
i
what information does the python file need in order to fetch the documents from pinecone? or does it fetch the same objects every time?
a
The file only needs the query essentially Rest of the code is just using the Pinecone library from Langchain to do a similarity search on the documents in the vector store with the query
i
by query do you mean the full prompt, or just the other vars? I'm thinking about how to support this and it would be easy to pass the other
vars
to the file - the full prompt is more complicated because it's not yet constructed when the tests are read
a
So, by query here, I don't mean the full prompt but the question that is passed to extract relevant context. It's the query variable here. For example, if the document passed is the constitution, then relevant queries could be - Who's the 30th president? And then the right answer should be within the context extracted from the constitution. That context extraction part is what the python file would do. If the context variable can somehow trigger the python file like in here
Copy code
tests:
  - vars:
      context: file://context_fetcher.py
That would actually solve the issue
The python file would just be returning text from the context extracted
@ian