https://promptfoo.dev/ logo
Custom Grader not working
# ❓|questions
m
I’m trying to configure promptfoo to use my organization’s custom LLM wrapper for both generating responses and analyzing the results. The organization has a mandatory requirement to use a custom API wrapper that encapsulates models like OpenAI and Claude. I’ve successfully set up the response generation with the custom provider, but for the evaluation, I need to use a local JavaScript file to make an HTTP call to the custom LLM wrapper API for analyzing the generated vs expected results. with the --grader flag i am not able to use my local llm wraper JS file this is how i try to call it promptfoo eval --grader ./providers/custom-provider.js but it is just using regular openai
m
@abhi127 is working on this! We'll have a fix out for you soon
In the mean time you can set the grader in your yaml file as
Copy code
defaultTest:
  options:
    provider:
      id: file://path/to/file.js
      config:
        optional_key: value
instead of using the --grader flag
m
thx for ur hint. I have tried it but for me it doesnt have any effect trying it that way Hmmm. Do you have an idea?
2 Views