Hey everyone, Does anyone have any experience gui...
# general
r
Hey everyone, Does anyone have any experience guidelines or tips for measuring the coverage of our pact tests? As part of our transition to pact we have added tests for most of our consumers, and we believe we are pretty far. However we are looking for a way to measure the coverage from the consumer perspective. Since we'd like to know how many integrations are still missing tests. How do other deal with this?
b
This is a cool idea, but I don't know how to automatically instrument it. The main starting point for me would be: which kind of coverage are you most interested in, and do different kinds build on each other?
E.g. • proportion of endpoints that have any tests, vs • branches in individual fields in request/response body/params, vs • line/branch coverage in an API client (framework-dependent), • etc
r
I would probably look at integration coverage from the consumers perspective. So the percentage of endpoint that a consumer integrates with, which has at least 1 pact test. Individual field variations in body/params would add way too much complexity to this I did find https://docs.pact.io/faq/question_archive#how-can-i-tell-if-i-have-good-contract-test-coverage-of-my-api which suggests using code coverage tooling. It this could be a way to check this, but it would only work if you measure coverage when running consumer tests only, and then you need to check the coverage of the client code only. So while this can work, it it hard to give a good report in a simple way
👍 1
m
Hey Ruud! Apologies for the delay. You’re right in wanting to answer the question “How do I know I’ve got sufficient coverage of my _consumer_’s use of an API”. This is often requested under the guise of “coverage”. Yes, the simplest / crudest form is to use coverage checks, and ensuring your Pact tests run in a separate stage from your other tests (then you know any coverage comes from those tests). However they are code level, and not use-case driven (or otherwise). Another option which I’ve been toying with (and I hope to bring to PactFlow soon) is using AI to help us. It’s actually a decent candidate for this case because it can be hard programatically to understand a code base and then determine it’s behaviour. AI is reasonably good at it, and can at least give you some insights.
Here is an example “coverage analyzer” prompt: https://github.com/mefellows/example-consumer/blob/feat/ai-pr-review/scripts/pact-coverage-analyzer.sh It has some hard coded files in there, but that should be generalised to add any relevant files. This prompt gets pretty good results, and is the basis (using the above script) for what is printed in this example GitHub PR (analysed by claude code):
Copy code
I need a COMPLETE, DETAILED analysis. Do NOT summarize. Output EXACTLY this format:

## Coverage Summary
| Metric | Value |
|--------|-------|
| Coverage Percentage | [calculate]% |
| Total API Scenarios | [count] |
| Total Pact Interactions | [count] |
| Pact Test File Status | ✅ Has Tests / ❌ Empty |

## Coverage Details
| API Resource     | HTTP Method | Path          | Scenario | Status |
|------------------|-------------|---------------|----------|--------|
| getAllProducts   | GET         | /products     | 200      | ✅/❌   |
| getAllProducts   | GET         | /products     | 401      | ✅/❌   |
| someOperation    | POST        | /op/{id}      | 404      | ✅/❌   |

## 🚨 Coverage Gaps & Recommendations
- List each missing test case
- Provide specific implementation recommendations

ANALYZE THESE FILES:

=== API CLIENT (xyz.ts) ===
...

=== OPENAPI SPEC (xyz.yml) ===
...

=== PACT FILE (abc-def.json)===
...

=== PACT TESTS (xyz.pact.spec.ts)===
...

REQUIREMENTS:
1. Fill in ALL table rows with actual data
2. Calculate coverage percentage as: (covered scenarios / total API scenarios) * 100
3. List every missing test scenario
4. Do NOT summarize - provide complete detailed output
5. Follow the exact table format shown above

**Analysis Rules:**
- Only consider methods that exist in the API client - ignore OpenAPI-only endpoints
- Check for coverage of common (or as documented in the OpenAPI spec) HTTP status codes: 200 (success), 400 (bad request), 401 (unauthorized), 404 (not found)
- Mark as covered if ANY Pact test exists for that method/endpoint and status code combination
- Calculate coverage percentage as: (covered scenarios / total API scenarios) * 100
- **Total API Scenarios = sum of all unique combinations of (HTTP method + path + status code) that should be tested based on the API client methods and OpenAPI spec**
- **Each row in the Coverage Details table represents ONE scenario (one method + path + status code combination)**

**CRITICAL VALIDATION STEPS:**
1. **Before creating the Coverage Details table**: List out all scenarios you identified above
2. **After creating the Coverage Details table**: Count the actual rows in your table
3. **Final Check**: Verify the row count matches your "Total API Scenarios" in the Coverage Summary
4. **If they don't match**: Re-examine your analysis and fix the discrepancy before finalizing

START YOUR RESPONSE WITH "## Coverage Summary"
I’ve just (now) tested that locally with Ollama and gpt-oss and it gives me this:
Copy code
## Coverage Summary
| Metric | Value |
|--------|-------|
| Coverage Percentage | 10% |
| Total API Scenarios | 10 |
| Total Pact Interactions | 1 |
| Pact Test File Status | ✅ Has Tests |

## Coverage Details
| API Resource     | HTTP Method | Path          | Scenario | Status |
|------------------|-------------|---------------|----------|--------|
| getAllProducts   | GET         | /products     | 200      | ✅ |
| getAllProducts   | GET         | /products     | 401      | ❌ |
| getProduct       | GET         | /product/{id} | 200      | ❌ |
| getProduct       | GET         | /product/{id} | 400      | ❌ |
| getProduct       | GET         | /product/{id} | 401      | ❌ |
| getProduct       | GET         | /product/{id} | 404      | ❌ |
| deleteProduct    | DELETE      | /product/{id} | 200      | ❌ |
| deleteProduct    | DELETE      | /product/{id} | 400      | ❌ |
| deleteProduct    | DELETE      | /product/{id} | 401      | ❌ |
| deleteProduct    | DELETE      | /product/{id} | 404      | ❌ |

## 🚨 Coverage Gaps & Recommendations
- **Missing Test Case:** `GET /products` – 401 Unauthorized
  **Recommendation:** Add a Pact interaction with status `401` and validate that the client throws or handles the error appropriately.

- **Missing Test Case:** `GET /product/{id}` – 200 OK
  **Recommendation:** Create a Pact test that returns a single product and ensure `API.getProduct(id)` maps the response to a `Product` instance correctly.

- **Missing Test Case:** `GET /product/{id}` – 400 Bad Request
  **Recommendation:** Write a Pact test where an invalid ID is supplied, expecting a `400` response, and confirm that the client handles it gracefully.

- **Missing Test Case:** `GET /product/{id}` – 401 Unauthorized
  **Recommendation:** Add a Pact test for `401` status and verify that authentication failures are surfaced properly.

- **Missing Test Case:** `GET /product/{id}` – 404 Not Found
  **Recommendation:** Include a Pact test that simulates a non‑existent product, ensuring the client throws a `404` error or returns `null`.

- **Missing Test Case:** `DELETE /product/{id}` – 200 OK
  **Recommendation:** Implement a Pact interaction that deletes a product and returns `200`; confirm the client's `deleteProduct` method completes without error.

- **Missing Test Case:** `DELETE /product/{id}` – 400 Bad Request
  **Recommendation:** Test how the client reacts to a malformed request with a `400` response.

- **Missing Test Case:** `DELETE /product/{id}` – 401 Unauthorized
  **Recommendation:** Add a Pact test for `401` to ensure proper handling of authentication failures during deletion.

- **Missing Test Case:** `DELETE /product/{id}` – 404 Not Found
  **Recommendation:** Create a Pact scenario where the product to delete does not exist, verifying that the client handles the `404` status correctly.
So you could potentially get away with free tools in GH Actions. If you have claude, openapi or copilot you could use those too (e.g. opencode + copilot)
💯 1