https://promptfoo.dev/ logo
How to hook context in YAML?
# ❓|questions
a
Hi, everyone. I spent around 3-4 hours to understand how dynamic context works, but whatever I did, every time I got an error. I connected to my own RAG using custom call_api
Copy code
def call_api(prompt, options=None, context=None):
.................. some logic......
    data = response.json()
    contexts = [source.get('content', '')
                for source in data.get('sources', [])]

    return {
        "output": data.get('content', ''),
        "context": context_text
    }
and part of YAML for this metric
Copy code
assert:
      - type: context-relevance
        contextTransform: context
        value: ''
But when I tried to catch this context field from the RAG response, I got an error below Whatever I did, I tried to use a string or array, just context or output.context, every time I had an error
Copy code
Error: Failed to transform context using expression 'context': Invariant failed: contextTransform must return a string or array of strings. Got object. Check your transform expression: context

Error: Failed to transform context using expression 'context': Invariant failed: contextTransform must return a string or array of strings. Got object. Check your transform expression: context
    at resolveContext (/Users/aleksandrmeskov/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/assertions/contextUtils.js:60:19)
    at async handleContextRelevance (/Users/aleksandrmeskov/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/assertions/contextRelevance.js:23:21)
    at async runAssertion (/Users/aleksandrmeskov/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/assertions/index.js:353:24)
    at async /Users/aleksandrmeskov/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/assertions/index.js:400:24
In documentation, it looks pretty simple, but look like it doesn't work correctly https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/context-relevance/ Any suggestions, how I can handle that? https://cdn.discordapp.com/attachments/1432757147405651968/1432757147648786584/image.png?ex=69023693&is=6900e513&hm=e3561b5fac664cff41e9131fc0c4327ce0fa1634c74a9240f06dff1d91c6ffb1&
w
How is
context_text
defined?
contextTransform
would expect a value of
output.context
, not
context
.
2 Views