https://promptfoo.dev/ logo
Allowing local files to define providerOutput valu...
# ❓|questions
a
Related to my previous PR: https://github.com/promptfoo/promptfoo/pull/671 I want to allow local files to leverage the new providerOutput field in TestCase. What should be the keyword that the local file readers use?
from reading the
readStandaloneTestsFile
method in testCases.ts seems like all of the files are being parsed into rows of `TestCase`s through the
testCaseFromCsvRow
method
Copy code
export function testCaseFromCsvRow(row: CsvRow): TestCase {
  const vars: Record<string, string> = {};
  const asserts: Assertion[] = [];
  const options: TestCase['options'] = {};
  let providerOutput: string | undefined; // Change made
  let description: string | undefined;
  for (const [key, value] of Object.entries(row)) {
    if (key.startsWith('__expected')) {
      if (value.trim() !== '') {
        asserts.push(assertionFromString(value));
      }
    } else if (key === '__prefix') {
      options.prefix = value;
    } else if (key === '__suffix') {
      options.suffix = value;
    } else if (key === '__description') {
      description = value;
    } else if (key === 'provider_output') { // Change made
      providerOutput = value; // Change made
    } else {
      vars[key] = value;
    }
  }
  return {
    vars,
    ...(providerOutput ? { providerOutput } : {}), // Change made
    assert: asserts,
    options,
    ...(description ? { description } : {}),
  };
}
I think doing the above change would work for this. @ian what do you think about this?
I added comment
// change made
to the lines I added to the code
cc: @pragyadaga regarding what we discussed before on allowing users to define output to evaluate directly from the local file
I am not sure the keyword to look for the
providerOutput
value in the input files
https://github.com/promptfoo/promptfoo/pull/675 I made a PR for this @User , let me know what you think about it
p
It would be nice to extend if the user could provide it's own output lambda, but I am not sure about that.
a
@pragyadaga I think lets start first by allowing user to define the output value to evaluate directly
I think that might require a different implementation pattern
p
for sure!! definitely not in this PR or issue.
a
we can discuss it separately
PR merged, now providerOutput is a field accessible through either from local files or when using node module
4 Views