https://promptfoo.dev/ logo
Anyone able to use Fireworks AI as a provider usin...
# ❓|questions
w
I'm trying to use Fireworks.ai, but I'm unable to pass their models through promptfoo using the openai compatible method. Fireworks is openai sdk compatible, but I think they way they handle their model naming messes things up.
Copy code
providers:
  - id: openai:chat:accounts/fireworks/models/llama-v3-70b-instruct
    config:
      apiBaseUrl: https://api.fireworks.ai/inference/v1
      apiKeyEnvar: FIREWORKS_API_KEY
This will result in the following error in promptfoo:
Copy code
API error: [{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 0}], Type: invalid_request_error

{
  "error": {
    "object": "error",
    "type": "invalid_request_error",
    "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 0}]"
  }
}
I suspect it has to do with the slashes in the fireworks model naming conventions. They use the full
accounts/fireworks/models/llama-v3-70b-instruct
as the model value.
i
it actually looks like it's getting upset because we're also sending a
seed
param
unfortunately that's hardcoded into the OpenAI provider, so this will require a fix in promptfoo itself (or you'd have to write a custom provider for fireworks)
w
oh gotcha - I’ll just try a custom provider for now. Thank you!