https://promptfoo.dev/ logo
Claude 3.7 "streaming is strongly encouraged" err...
# ❓|questions
c
I'm trying to run tests using Claude 3.7, and I get this error for each run: " API call error: Streaming is strongly recommended for operations that may take longer than 10 minutes. See https://github.com/anthropics/anthropic-sdk-python#streaming-responses for more details" This doesn't happen with 3.5 Here's my setup -- maybe I'm doing it wrong?
Copy code
- id: anthropic:messages:claude-3-7-sonnet-20250219
    label: anthropic-3.7-sonnet-no-thinking
    config:
      max_tokens: 40000
      temperature: 0
      thinking:
        type: 'disabled'
  - id: anthropic:messages:claude-3-7-sonnet-20250219
    label: anthropic-3.7-sonnet-thinking
    config:
      temperature: 0
      max_tokens: 40000
      thinking:
        type: 'enabled'
        budget_tokens: 32000
OK I think I figured it out -- I was asking for more than 21,333 max_tokens, and over that number you need to have streaming enabled. Bringing down the max_tokens and the budget_tokens resolved this error. Moreover, you cannot set temperature in the thinking model. This was all in the docs, sorry I missed it
One thing I don't know how to do is enable streaming in PromptFoo (there's a bit about it in the docs, but it seems like you need to put something together yourself as opposed to it working out-of-the-box
m
We do not currently support streaming in the anthropic provider but would welcome a PR if this is something you are interested in taking on. I can walk you through the code on a quick call and send you some promptfoo swag when the PR lands if you are interested. If not, I'll add this to our backlog!