Hello Pact Team,
I was wondering if there is any ...
# generalb
Brett Knapik
03/13/2023, 5:34 PMHello Pact Team,
I was wondering if there is any guidance on how to measure contract test coverage?
I have stakeholder buy-in on contract testing, however we need a way to measure that contracts are covered and what are not. Our concern comes from the fact the can-i-deploy tool is only as good as the data we give to the pact broker. if we are missing contracts, can-i-deploy will misrepesent that the safety of the deployment. Long term we would like to block PRs if they are missing pacts for new endpoints/consumers.
(I apologize if this is the wrong channel for this question, please direct me to the correct one if needed.)
🦻 1
t
Timothy Jones
03/13/2023, 10:39 PMI’m not sure what you’re asking for exactly- this post seems to imply a few different meanings of “contracts”. But, one of the advantages of contract testing is that it only covers what you’re using. So, if you have an endpoint that isn’t used, it’s not covered by the tests (because you could make arbitrary changes to it without consequences)
Timothy Jones
03/13/2023, 10:40 PMOn the client side, you can get some of the same advantage by tying the test fixtures that you use for your Pact tests to your other tests - so that your contract must be covering exactly what your functional tests cover
b
Brett Knapik
03/13/2023, 10:49 PMSorry for the confusion.
We know that these endpoints are being used however, they do not have existing pact tests. We have 2 scenarios we are looking to address:
1.) we are rolling out contract testing using Pact. How can we measure what endpoints/producers/consumers are missing tests with Pact. We have 60+ microservices, so onboarding pact is quite an effort.
2.) A developer added a new endpoint/producer/consumer and they forgot to add a pact test for it.
Is there any approach/tooling to help us with these scenarios?
t
Timothy Jones
03/13/2023, 10:52 PMI’m not aware of any tooling, although it would be possible to build something that looked at schemas and gave an answer. I think the reason that general purpose tooling doesn’t exist is because Pact isn’t supposed to cover the whole API surface, it’s supposed to cover the parts that your consumer actually needs. Removing a field from a response isn’t a breaking change if your consumer isn’t expecting that field.
Timothy Jones
03/13/2023, 10:52 PM2.) A developer added a new endpoint/producer/consumer and they forgot to add a pact test for it.How would you solve this for other test types like e2e or unit tests? This doesn’t feel like a problem that should be solved with tooling
Timothy Jones
03/13/2023, 10:54 PM1.) we are rolling out contract testing using Pact. How can we measure what endpoints/producers/consumers are missing tests with Pact. We have 60+ microservices, so onboarding pact is quite an effort.I think this is a great question
Timothy Jones
03/13/2023, 10:55 PMThe answer probably depends a lot on what your existing ecosystem is. Do you have schemas for those APIs?
Timothy Jones
03/13/2023, 10:59 PMbtw - for forgetting to add a test, I think consistency of patterns is important. If your test fixtures are tied to the Pact tests, then people will see the pact stuff when they try to mock the API for the unit tests.
Timothy Jones
03/13/2023, 11:00 PMAlso, I would expect unit test coverage tools to have full coverage of some parts of the client code. For example, this code should have full coverage during a pact test:
Copy code
const api = (baseurl: string): Api => {
const server = makeAxiosConnector(baseurl);
return {
getAllProducts: () => server.authedGet<string[]>('/products'),
getProduct: (id) => server.authedGet(`/products/${id}`),
getUser: (id: string) =>
server.authedGet<User>(`/users/${id}`).catch((e) => {
if (e.code === API_NOT_FOUND) {
throw new UserNotFoundConsumerError(`Unable to find user '${id}'`);
}
throw e;
}),
....
};
};Timothy Jones
03/13/2023, 11:01 PMIt’s tricky though, because not all API client code will be 100% covered during a pact test. For example, your API client might handle unexpected error scenarios that the server never produces
Timothy Jones
03/13/2023, 11:02 PM(eg, if the client has code for “what do I do if the sever doesn’t conform to the contract” it’s hard to put that in the contract)
m
Matt (pactflow.io / pact-js / pact-go)
03/14/2023, 5:29 AM
Thanks Tim. Yes, the answer to that question is hard.
The usual API coverage tools are the best starting point. I would recommend you run pact tests separate from the rest of your tests - you can then at least scope the coverage to your API client packages and know what was covered was from the pact tests and not your other testing types, and assess from there (automated-ly or otherwise)
b
Brett Knapik
03/14/2023, 1:06 PMThanks for your replies; they are appreciated.
In our case we use SonarCloud quality gates to require 80%+ unit test coverage. If the developer does not meet that standard their PR is blocked from being checked in using Azure DevOps quality gates. It is how we control our unit testing standards.
For E2E tracking, we use Zephyr Squad with Jira where each ticket would have their E2E tests tracked.
I do agree on running pact tests seperately.
đź‘Ť 1
11 Views