https://pact.io logo
Join Slack
Powered by
Hi team!, I am trying to use Pactflow-ai to genera...
# pactflow
r
Hi team!, I am trying to use Pactflow-ai to generate the consumer contract test from our openapi spec file but every time the getting the 
ERROR: Server error: 500 Internal Server Error
, maybe can you help ? it is generating for happy path but failing 400, 404 etc
m
Can you please show what you’re doing (i.e. share the command you’re using)? There is an API bug where it should probably be a 
400
.
From memory, the bug was related to: 1. Specifying an invalid 
--endpoint
argument when passing an OAD 2. The endpoint requested doesn’t match any routes in your OAD
r
this is the command which I use 
pactflow-ai generate   --openapi  ./dist/openapi.v1.json   --endpoint '{"path": "/v2/rules/{ruleId}", "methods": ["Get"], "status": ["400"]}'  --output   ./src/test/java/com/outsystems/dp/rules/api/RulesPactTestForRuleId400.java   --language java
I have the api and responses mapped in the open api spec
m
Any chance you could please add a 
-vvv
to your request, and DM me the output?
also please the output of 
pactflow-ai status
r
-vvv
in the end of the command?
I get this , looks like need to update the cli
Copy code
0.000359s DEBUG pactflow_ai pactflow-ai 1.9.1 (b2c43b4 2025-06-12)
    0.004895s DEBUG pactflow_ai::config Reading configuration from environment variables
 pactflow_ai::commands::generate::generate
    0.005458s   0ms DEBUG pactflow_ai::commands::generate Running generate command
   pactflow_ai::client::Building API client
      0.005559s   0ms DEBUG pactflow_ai::client Building API client
      0.005576s   0ms DEBUG pactflow_ai::client User-Agent: pactflow-ai/1.9.1 (b2c43b4 2025-06-12)
      0.005640s   0ms DEBUG pactflow_ai::client Using HTTP timeout: 30s
      0.005686s   0ms DEBUG pactflow_ai::client Using WebPKI certs
      0.005992s   0ms DEBUG pactflow_ai::client Using token
      0.006036s   0ms DEBUG pactflow_ai::client Using base URL: <https://outsystems.pactflow.io>
    0.017656s  12ms  INFO pactflow_ai::client::models Unable to parse endpoint matcher JSON. Assuming it is a path.
    0.020393s  15ms DEBUG pactflow_ai::client POST generate
   14.267627s  14s  DEBUG pactflow_ai::client Checking API version
   14.267746s  14s  DEBUG pactflow_ai::client No API version returned
   14.267951s  14s  DEBUG pactflow_ai::client HEAD <https://outsystems.pactflow.io/api/ai/status/741c313d-8ffe-4a65-9092-ff4701c6067c>
   25.795664s  25s  DEBUG pactflow_ai::client Checking API version
   25.795774s  25s  DEBUG pactflow_ai::client API version: 1.9.3
   25.795789s  25s  DEBUG pactflow_ai::client CLI version: 1.9.1 (b2c43b4 2025-06-12)
   25.795796s  25s   WARN pactflow_ai::client A newer version of the CLI is available. Please update your CLI from <https://pactflow.io/ai/>   
   25.795817s  25s  DEBUG pactflow_ai::client GET <https://outsystems.pactflow.io/api/ai/result/741c313d-8ffe-4a65-9092-ff4701c6067c>
ERROR: Server error: 500 Internal Server Error
logs of 
pactflow-ai status
command
Copy code
Organization: outsystems
AI enabled (plan): true
AI enabled (system preferences): true
AI credits: 8 / 1000 (992 remaining)
User AI permissions:
  - ai:*
👍 1
what might be the problem ? I see the api and responses on open api spec, shall I upgrade the cli ?
m
OK it is the problem I referred to before - “No OpenAPI interactions matched the provided endpoint”
can you please share the OAD (again, DM is fine if you prefer)
r
let me check
m
Unable to parse endpoint matcher JSON. Assuming it is a path.
this line seems relevant
r
yes, we are specifying like this
Copy code
"paths": {
    "/v2/rules": {
      "get": {
        "tags": [
          "Rules API"
        ],
        "summary": "Get all details of all Rules",
        "operationId": "getAll",
the endpoints
m
That wouldn’t match the endpoint you provided in 
--endpoint
- is there one that matches 
/v2/rules/{ruleId}
?
r
yes we have that also
I tried for both the endpoints and it failed
with same 500
m
Can you please share the OAD in a DM?
r
okay
m
I’m just seeing if I can reproduce myself
OK I think the issue is that 
status
needs to be 
status_codes
. Something looks to have changed, I’ll raise an internal bug. Once I update that, things are working again
r
oh okay let me try
👍 1
now it worked
thanks
but tests are not accurate
m
great (first part), what do you mean by “not accurate”?
Without extra context, PactFlow AI can only produce high-level output. For best results 1. Give it access to your API client code (and domain models) via the 
--code
flag (you can use this multiple times) 2. Use a test template 3. Create a set of prompts for re-use across all tests in the repo See https://docs.pactflow.io/docs/ai/using#using-test-templates for more on this
May I ask, do you have any other AI tools - e.g. Copilot, Claude Code, Windsurf etc.?
r
by "not accurate" I meant we don't pass a body for the response for 400 requests, if we do the tests will fail, but genAi added the response body for 400 by reading the open api, I will try to use with the api-client and experiment more, for 200 it is accurate
Yes we use AI tools like copilot
👍 1
can we pass multiple files for client code too to the command ? like controller, model and api client code together together ?
m
> by “not accurate” I meant we don’t pass a body for the response for 400 requests, if we do the tests will fail, but genAi added the response body for 400 by reading the open api, I will try to use with the api-client and experiment more, for 200 it is accurate (edited) is there a body in the OAD? Or did it just make that up entirely? UPDATE: I see you said it is in the OAD. As you’ve only given AI the OAD, it doesn’t know what your code is doing so it can’t infer what to do. I’d have a read of the guide I shared above…and the video below for what’s next 🙂
can we pass multiple files for client code too to the command ? like controller, model and api client code together together ?
yep. As per above, you can pass in 
--code
multiple times
(my upload seems to be hanging 😞 ) But the accompanying text to it:
The reason I ask about copilot, is that we’re looking to release an MCP server in the next month or so.
Here’s an example of what that might look like when integrated into VS Code with Copilot (via MCP)
I need to run, but I’ll check back in the morning. Good luck!
r
nope, it worked right I think, we have a body, but the test failed, I will experiment and play more with this gen ai one, btw it helped us great to increase our test creation process, thanks for the tool, good luck
🙏 1
m
The reason I ask about copilot, is that we’re looking to release an MCP server in the next month or so.
generate-tests-mcp.mov
r
looks cool, can't wait to try this feature
🙏 1
Hi Matt!, can we generate tests for negative cases(401,404, etc) providing client code to pactflow ai ? I don't see any example for this, I know we can generate using openapi spec, but what about client code ?
m
The simplest way is to add it to the 
--instructions
flag. e.g. 
--instructions "You must add tests for negative scenarios including for 400, 401 and 404 related errors
When passing the OAD + Code, it can infer it from the OAD (although it’s good practice to encourage the scenarios you need also). The tutorial covers this, I think. The docs should also have an example of a template/instructions to use
r
can you pls share the doc and tutorial link, that would be helpful, thanks
m
No worries - also thanks for participating in the research with Ying. I really appreciate it!
Open in Slack
PreviousNext
Powered by