https://promptfoo.dev/ logo
Asserting on complex nested extracted entities
# ❓|questions
d
Hi everyone! I'm humbly looking for suggestions and opinions on how to assert on complex, sometimes deeply nested, extracted entities. My task extracts several entities from pdf documents and I am already using promptfoo to assert and build up metrics for performance. But it's getting ever so hard because of the complexity of the extracted entities. For example, this is one of my assertions:
Copy code
- type: python
      value: ('Outorgar Poderes' in output['pessoas_analisadas'][1]['restricoes_valor'] and '12' in output['pessoas_analisadas'][1]['restricoes_valor']['Outorgar Poderes']['valor_alcada'] and 'meses' in output['pessoas_analisadas'][1]['restricoes_valor']['Outorgar Poderes']['valor_alcada'])
      metric: alcada
And this is not even robust, becase it depends on the order of the
output['pessoas_analisadas']
list being consistent across different evals. I'd appreciate any sugestion. Meanwhile, I was even considering contributing a
transform
property to assert-sets, which would enable this kind of syntax:
Copy code
tests:
  - description: test for persona 1
    vars:
      - file://path/to/pdf
    assert:
      - type: assert-set
        transform: next(o for o in output['pessoas_analisadas'] if o['nome'] == 'NAME OF PERSON')
        assert:
          - type: python
            value: ('Outorgar Poderes' in output['restricoes_valor'] and '12' in output['restricoes_valor']['Outorgar Poderes']['valor_alcada'] ...
Opinions?
a
I don't know if the writing a test in a Python file works but you can use .js as a test file.
Copy code
defaultTest:
  assert:
    - type: javascript
      value: file://./image-classification-new.test.js
      metric: Image_Classification
Here's how I write test, here you can check the values for nested objects etc.
Copy code
js

const test = async (output, context) => {
  const { response } = output;

  const tests = [
    {
      fn: (_items, items) => items.every((item) => item.title?.trim().length),
      passReason: "The titles aren't empty",
      failedReason: "The titles are empty",
    },
    {
      fn: (_items, items) =>
        items.every((item) => item.paragraph?.trim().length),
      passReason: "The paragraphs aren't empty",
      failedReason: "The paragraphs are empty",
    },
  ];

  if (context.vars?._expected?.includes("null"))
    return {
      pass: true,
      score: tests.length,
      reason: "Expected value",
      componentResults: [],
    };

  const results = tests.map((test, index) => {
    const pass = test.fn(response);
    return {
      pass,
      score: pass ? 1 : 0,
      reason: pass
        ? `${index + 1}. ${test.passReason}`
        : `${index + 1}. ${test.failedReason}`,
    };
  });
  const failReasons = results
    .filter((result) => !result.pass)
    .map((result) => result.reason)
    .join("\n");
  const passReasons = results
    .filter((result) => result.pass)
    .map((result) => result.reason)
    .join("\n");
  const pass = results.every((result) => result.pass);
  return {
    pass,
    score: results.reduce((acc, result) => acc + result.score, 0),
    reason: pass ? passReasons : failReasons,
    componentResults: results,
  };
};

module.exports = test;