https://promptfoo.dev/ logo
Multiple scores in a single test
# ❓|questions
c
Hello everyone, I am in the process of testing my retriever for accuracy. I get five results for one query. The metrics I am testing are "recall at one" and "recall at five". I use the contains for "Recall at five", because I get five results. I test "recall at one" using a python script. So far, everything works fine. The only thing I wish is that I would receive both evaluations separately for the score. That means one score for "recall at one" and one for "recall at five". How this is possible seems unclear to me at the moment. Below is a test case of how I use it. Many thanks and best regards.
Copy code
yml
prompts:
- '{{query}}'
providers:
- exec:python retrieve.py --collection squad_train-v2.0_100
tests:
- assert:
  - type: contains-all
    value:
    - context_1.txt
  - type: python
    value: "import sys\nexpected_output = 'context_1.txt'\noutput_lines = [line.strip()\
      \ for line in output.split('\\n') if line.strip()]\nresult_recall_at_one = output_lines[0].startswith(expected_output)\
      \ if output_lines else False\nprint(json.dumps({\n    'pass': result_recall_at_one,\n\
      \    'score': 1.0 if result_recall_at_one else 0.0,\n    'metric': 'Recall at\
      \ One'\n}))"
i
Definitely understand the ask here - it's something I've wanted too. Right now, there is just a single aggregate score for each cell, and you can view the composite scores by clicking the 🔍 icon. You may also assign [weights](https://promptfoo.dev/docs/configuration/expected-outputs/#weighted-assertions) to individual assertions in order to shape the final score. But the short answer is separate scores aren't supported yet, although this is something that I'm planning to implement very soon.
I've created a PR here: It adds the ability to label assertions with metric names. For example:
Copy code
yaml
  - # ...
    assert:
      - type: icontains
        value: arrr
        metric: Pirateness

  - # ...
    assert:
      - type: is-json
        metric: Consistency
And the tags appear in the webview. Do you think this would help your use case?
c
Wow, that looks promising! I'm going to test it and will give you feedback. But I think it satisfies my use case perfectly.
i
This functionality is published in version 0.30.0 🙂
c
Great, it works. Thank you for implementing it. The only downside I noticed is, that the summarized score at the very top is not affected. I think it would be nice if we can calculate a score for each metric. For the moment, I do it manually with JavaScript.
Copy code
js
function calculateAndDisplayScores() {
    const metricContainers = document.querySelectorAll('.custom-metric-container');
    if (metricContainers.length === 0) {
        console.error('No metric containers found');
        return;
    }

    const metrics = {};

    metricContainers.forEach(container => {
        const metricElements = container.querySelectorAll('div');

        metricElements.forEach(element => {
            const parts = element.textContent.split(':');
            const metricName = parts[0].trim();
            const score = parseFloat(parts[1]);

            if (!metrics[metricName]) {
                metrics[metricName] = { trueCount: 0, totalCount: 0 };
            }

            metrics[metricName].totalCount++;
            if (!isNaN(score) && score > 0) {
                metrics[metricName].trueCount++;
            }
        });
    });

    // Calculate and display the results in the console
    console.log('Scores:');
    for (const [metric, data] of Object.entries(metrics)) {
        const score = (data.trueCount / data.totalCount * 100).toFixed(2);
        console.log(`${metric}: ${score}% (${data.trueCount}/${data.totalCount})`);
    }
}

calculateAndDisplayScores();
i
This PR will show the aggregated scores at the top: https://github.com/promptfoo/promptfoo/pull/322
y
very nice stuffs, is there any way to specify a metric within a CSV?
i
not currently, but here's a PR that adds support for `__metric`: https://github.com/promptfoo/promptfoo/pull/889