Hi All,
# questionsu
umut.console
10/07/2025, 11:55 AMI would like to test my agents which I created in Azure AI Foundry. The agents don't have any deployment name, instead they have Agent ID and Agent Name.
The endpoint is generated like this: https://promptfoo-testing-resource.services.ai.azure.com/api/projects/promptfoo-testing
What would you recommend to configure my "provider" agents in YAML file? I would like to stay away from implementing a self HTTP provider, if there is an easier way.
Thanks a lot for your recommendations.
Kind Regards,
Umut
u
[Promptfoo] Faizan
10/09/2025, 8:43 PMHey @umut.console , I actually have a PR for adding support for that feature here (https://github.com/promptfoo/promptfoo/pull/5181). I'll get this merged and released in the next promptfoo release
u
umut.console
10/10/2025, 3:52 PMThanks a lot, Faizan. I am looking at the PR now.
umut.console
10/10/2025, 3:55 PMI believe assistants and agents are different, but I will try by setting the endpoint URL and use "Agent ID" as "Deployment Name". Thanks a lot.
u
[Promptfoo] Faizan
10/10/2025, 3:57 PMAh, you're right- I suspect adding support for the agents wouldn't be much different than this, though. I'm a bit busy but I can try to implement support for the agents & test it sometime next week.
u
umut.console
10/10/2025, 3:58 PMThanks, i will also try if I have time this weekend and let you know. All the best.
umut.console
10/12/2025, 1:03 PMHello Faizan, I tried several combinations but they all failed with an error.
1. I tried treating an Azure AI Foundry agent as an "assistant", but I got the error that says "API error: 400 Bad Request" by expecting an assistant ID starting with asst.
2. My agent name is "deployment-pf-gpt-5-nano" and setting provider ID in YAML as "azure:chat:deployment-pf-gpt-5-nano" created the correct URL endpoint, however I got the following response error about max_tokens:
"statusText": "Bad Request",
"response": "{\"error\":{\"message\":\"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\",\"type\":\"invalid_request_error\",\"param\":\"max_tokens\",\"code\":\"unsupported_parameter\"}}"
u
[Promptfoo] Faizan
10/13/2025, 5:41 PM@umut.console Hey, I just checked in azure and it turns out I just mislabled it, and it was the ai foundry agent sdk I implemented. I tested it with some agents I wrote in azure ai foundry and it looks like it works. I need to release a new promptfoo version, but once that's out, I think you can follow the example here: https://github.com/promptfoo/promptfoo/blob/main/examples/azure-foundry-agent/promptfooconfig.yaml
[Promptfoo] Faizan
10/13/2025, 6:48 PMI just released promptfoo 0.118.15, could you give it a try?
u
umut.console
10/14/2025, 4:03 PMHello Faizan, yes it works 🙂 Thanks a lot.