donatoazevedo_79335
06/05/2025, 1:52 PM- type: python
value: ('Outorgar Poderes' in output['pessoas_analisadas'][1]['restricoes_valor'] and '12' in output['pessoas_analisadas'][1]['restricoes_valor']['Outorgar Poderes']['valor_alcada'] and 'meses' in output['pessoas_analisadas'][1]['restricoes_valor']['Outorgar Poderes']['valor_alcada'])
metric: alcada
And this is not even robust, becase it depends on the order of the output['pessoas_analisadas'] list being consistent across different evals.
I'd appreciate any sugestion. Meanwhile, I was even considering contributing a transform property to assert-sets, which would enable this kind of syntax:
tests:
- description: test for persona 1
vars:
- file://path/to/pdf
assert:
- type: assert-set
transform: next(o for o in output['pessoas_analisadas'] if o['nome'] == 'NAME OF PERSON')
assert:
- type: python
value: ('Outorgar Poderes' in output['restricoes_valor'] and '12' in output['restricoes_valor']['Outorgar Poderes']['valor_alcada'] ...
Opinions?abhagsain
06/10/2025, 3:36 PMdefaultTest:
assert:
- type: javascript
value: file://./image-classification-new.test.js
metric: Image_Classification
Here's how I write test, here you can check the values for nested objects etc.
js
const test = async (output, context) => {
const { response } = output;
const tests = [
{
fn: (_items, items) => items.every((item) => item.title?.trim().length),
passReason: "The titles aren't empty",
failedReason: "The titles are empty",
},
{
fn: (_items, items) =>
items.every((item) => item.paragraph?.trim().length),
passReason: "The paragraphs aren't empty",
failedReason: "The paragraphs are empty",
},
];
if (context.vars?._expected?.includes("null"))
return {
pass: true,
score: tests.length,
reason: "Expected value",
componentResults: [],
};
const results = tests.map((test, index) => {
const pass = test.fn(response);
return {
pass,
score: pass ? 1 : 0,
reason: pass
? `${index + 1}. ${test.passReason}`
: `${index + 1}. ${test.failedReason}`,
};
});
const failReasons = results
.filter((result) => !result.pass)
.map((result) => result.reason)
.join("\n");
const passReasons = results
.filter((result) => result.pass)
.map((result) => result.reason)
.join("\n");
const pass = results.every((result) => result.pass);
return {
pass,
score: results.reduce((acc, result) => acc + result.score, 0),
reason: pass ? passReasons : failReasons,
componentResults: results,
};
};
module.exports = test;