https://promptfoo.dev/ logo
n
Unable to use extensions kindly someone help me import openai # Function to call the 'why' prompt when a test fails def call_why_prompt(failed_test_description, expected_value, actual_value): diagnostic_input = f"Why did the following test fail? Test: {failed_test_description}, Expected: {expected_value}, Got: {actual_value}" # Call OpenAI's GPT-4 API (you need to replace with your API logic) response = openai.Completion.create( engine="gpt-4", prompt=diagnostic_input, max_tokens=150 ) # Return the diagnostic response return response.choices[0].text.strip() # Main hook function def extension_hook(hook_name, context): if hook_name == 'beforeAll': print(f"Setting up test suite: {context['suite'].get('description', '')}") elif hook_name == 'afterAll': print(f"Test suite completed: {context['suite'].get('description', '')}") print(f"Total tests: {len(context['results'])}") elif hook_name == 'beforeEach': print(f"Running test: {context['test'].get('description', '')}") elif hook_name == 'afterEach': test_description = context['test'].get('description', '') success = context['result'].get('success', False) expected = context['test'].get('expected') actual = context['result'].get('actual') if not success: print(f"Test failed: {test_description}") diagnostic_response = call_why_prompt(test_description, expected, actual) print(f"Diagnostic response: {diagnostic_response}") else: print(f"Test passed: {test_description}") python file extensions: - file://path/to/your/extension.py:extension_hook
i
what error or problem are you seeing?
n
some : error
undefined:3 return extension.py:extension_hook ^ SyntaxError: Unexpected token ':' at new Function () at getInlineTransformFunction (/usr/local/lib/node_modules/promptfoo/dist/src/util/transform.js:81:12) at getTransformFunction (/usr/local/lib/node_modules/promptfoo/dist/src/util/transform.js:92:12) at transform (/usr/local/lib/node_modules/promptfoo/dist/src/util/transform.js:110:33) at runExtensionHook (/usr/local/lib/node_modules/promptfoo/dist/src/evaluatorHelpers.js:188:41) at Evaluator.evaluate (/usr/local/lib/node_modules/promptfoo/dist/src/evaluator.js:300:55) at evaluate (/usr/local/lib/node_modules/promptfoo/dist/src/evaluator.js:803:15) at runEvaluation (/usr/local/lib/node_modules/promptfoo/dist/src/commands/eval.js:133:56) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async doEval (/usr/local/lib/node_modules/promptfoo/dist/src/commands/eval.js:273:5) Node.js v20.18.0
i
try just
Copy code
yaml
extensions:
  - file://path/to/your/extension.py
without
:extension_hook
n
undefined:3 return extension.py ^ ReferenceError: extension is not defined at eval (eval at getInlineTransformFunction (/usr/local/lib/node_modules/promptfoo/dist/src/util/transform.js:81:12), :3:1) at transform (/usr/local/lib/node_modules/promptfoo/dist/src/util/transform.js:111:39) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async runExtensionHook (/usr/local/lib/node_modules/promptfoo/dist/src/evaluatorHelpers.js:188:9) at async Evaluator.evaluate (/usr/local/lib/node_modules/promptfoo/dist/src/evaluator.js:300:9) at async runEvaluation (/usr/local/lib/node_modules/promptfoo/dist/src/commands/eval.js:133:25) at async doEval (/usr/local/lib/node_modules/promptfoo/dist/src/commands/eval.js:273:5)
i
seems like it might be a bug on our side. cc @michaelmichaelmichael
n
oooh fine
i wanted to run a different prompt for only those which are getting failed any other way
@ian
m
@NoAimNoBrain I'm not able to replicate your bug locally. Are you on the latest promptfoo version, and can you throw some time on my calendar today: https://cal.com/michael-dangelo/15min ?