https://promptfoo.dev/ logo
retrieval asserts while using standalone test file...
# ❓|questions
d
Hey, love the work you guys are doing at promptfoo and the way you are so responsive, I had a query regarding if there is a way to include dynamic or static context for standalone test files as I would like to test my rag-app with these metrics based on a large number of test cases and I don't seem to understand if there is a way to provide context for the Context adherence, Context recall and Context relevance metrics
If this is not possible do you have any suggestions to integrate these metrics for a large number of test cases.
m
Hey @dev_nerd_41284, Thanks for your kind words! We're glad you're enjoying Promptfoo. Regarding your question about including dynamic or static context for standalone test files to test your RAG app with metrics like Context Adherence, Context Recall, and Context Relevance: Context Recall: This metric specifically takes a context variable. You can find more details in our documentation [here](https://www.promptfoo.dev/docs/guides/evaluate-rag/#context-recall). Many of these metrics are copied from RAGAS: RAGAS Metrics. If you notice any errors or have suggestions for improvements, we'd love your contributions! Thanks again, and feel free to reach out if you have any more questions.
d
Yeah I understood about it needing the context variable, but how can i provide one while using csv files for test cases I'm unable to figure out how the config yaml file will look like for this use case
e
You can pass static context for RAG evaluation like this:
Copy code
yaml
tests:
  - vars:
      query: How did the student calculate the volume of the mold?
      context: file://mold_volume.md
    assert:
      - type: context-relevance
        threshold: 0.9
        provider: openai:completion:explodinggradients/Ragas-critic-llm-Qwen1.5-GPTQ
      - type: context-faithfulness
        threshold: 0.9
        provider: openai:completion:explodinggradients/Ragas-critic-llm-Qwen1.5-GPTQ
      - type: answer-relevance
        threshold: 0.9
        provider: 
          text: 
            id: openai:completion:explodinggradients/Ragas-critic-llm-Qwen1.5-GPTQ
          embedding:
            id: ollama:embeddings:nomic-embed-text:latest
If you want to use the same provider for all tests, you can try using the defaultTest option like this:
Copy code
yaml
evaluateOptions:
  maxConcurrency: 1

prompts:
  - file://rag_prompt.txt

providers:
  - id: ollama:chat:command-r:latest

defaultTest:
  options:
    provider: 
      text: 
        id: ollama:chat:command-r:latest
      embedding:
        id: ollama:embeddings:nomic-embed-text:latest

tests:
  - vars:
      query: What types of silicone did the student use for molding?
      context: file://silicone_types.md
    assert:
      - type: context-relevance
        threshold: 0.9
      - type: context-faithfulness
        threshold: 0.9
      - type: answer-relevance
        threshold: 0.9
m
Thank you @elsatch! @dev_nerd_41284 we don't currently support providing context via CSV but would happily accept a PR to do so. One other workaround is: write a script to map your schema into the yaml.
e
Claude Sonnet suggests this code to extract the static context files and populate the yaml test:
Copy code
python
import pandas as pd
import yaml
import os

# Read the CSV file
df = pd.read_csv('your_file.csv')

# Create a directory for context files if it doesn't exist
os.makedirs('context_files', exist_ok=True)

# Initialize the YAML structure
yaml_data = {'tests': []}

# Process each row in the DataFrame
for _, row in df.iterrows():
    # Extract the id, query, and context
    id = row['id']
    query = row['query']
    context = row['context']
    
    # Create the context file
    context_filename = f"{id:03d}_context.txt"
    with open(os.path.join('context_files', context_filename), 'w') as f:
        f.write(context)
    
    # Create the test entry for YAML
    test_entry = {
        'vars': {
            'query': query,
            'context': f"file://{context_filename}"
        },
        'assert': [
            {'type': 'context-relevance', 'threshold': 0.9},
            {'type': 'context-faithfulness', 'threshold': 0.9},
            {'type': 'answer-relevance', 'threshold': 0.9}
        ]
    }
    
    # Add the test entry to the YAML data
    yaml_data['tests'].append(test_entry)

# Write the YAML file
with open('output.yaml', 'w') as f:
    yaml.dump(yaml_data, f, default_flow_style=False)

print("Processing complete. Context files and YAML file have been created.")
Note 1: I haven't tested the code yet, but looks mostly ok to me. Note 2: Iterrows is the slowest way to iterate over a large csv file, but you could improve the performance working with vectorial functions if needed.