Hi, happy promptfoo user for about a year but a recent upgrade has slowed my evals down significantly. I just updated from 0.113.2 to 0.117.6 and noticed all evals with my local HTTP Provider test configs are run serially even if I specify a

--max-concurrency

. The command line reports it as running concurrently when it hasn't >

Duration: 1m 46s (concurrency: 4)
> Successes: 9
> Failures: 0
> Errors: 0
> Pass Rate: 100.00%

I noticed a comment in this issue https://github.com/promptfoo/promptfoo/issues/1280#issuecomment-2251765379 - "We recently refactored evaluations to do providers 1 at a time" and hoping this isn't a permanent loss of functionality. EDIT - (Just noticed the date on that from last year. Probably not related but I couldn't find any other relavant mention). Is there a way I can re-enable concurrent evals? I'm running against my own local server for testing my multi-agent service, the testing configuration allowed me to validate more complex agentic tasks. Maybe HTTP Provider is no longer the best way to handle that?

Hey, can you share your provider config? (Can redact url/headers/etc).

Sure, here's the general structure

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: "Agentic Workflow #1"
prompts: '{{ input }}'
  
providers:
  - id: http://127.0.0.1:8000/blah
    config:
      method: POST
      headers:
        Content-Type: 'application/json'
      body: 
        businessId: '{{businessId}}'
        actorId: '{{actorId}}'
        threadId: '{{threadId}}'
        message: 
          body: '{{prompt}}'      


tests:
  - description: "Test 1"
    vars:
      businessId: file://../test_data/businessId.txt
      actorId: file://../test_data/actorId.txt
      threadId: file://../test_data/gen_thread_id.py
      input: 
        - blah blah
        - bleh bleh
        - blu blu
    assert:
      - type: llm-rubric
        value: "test does the thing"
      - type: is-json

  - description: "Test 2"
    vars:
      businessId: file://../test_data/businessId.txt
      actorId: file://../test_data/actorId.txt
      threadId: file://../test_data/gen_thread_id.py
      input: 
        - blah blah 2
        - bleh bleh 2
        - blu blu 2
    assert:
      - type: llm-rubric
        value: "test does the thing 2"
      - type: is-json

The progress bar UI was recently updated to just be 1 progress bar, but the threads are running concurrently (underlying concurrency has not changed). Maybe the progress bar change is what you've noticed?

Ahh yeah, that's it. Sorry I should have validated more before posting. The single progress bar and some recent api latency threw me off.