QUESTION(S): If you would draw the "scope" of respectively "functional testing" (owned by the provider) and "contract testing" (owned by the consumers) as a Venn diagram, (a) would you fully include the "contract testing" circle/set within the "functional testing" circle/set, or (b) would you have a part of the "contract testing" circle/set not included within the "functional testing" circle/set? If you would go with option (b), what would be a representative example of a "contract test" not being as well a good "functional test"? I do have my own answers to these questions - being more "dynamic" than just replying (a) or (b) - but, I am wondering what are your thoughts... P.S. This question has been triggered by the great video "Best practices for writing consumer tests" with @Beth (pactflow.io/Pact Broker/pact-ruby) |

Great discussion topic @Patrice Krakow I think my answer would be it depends on my current client situation and problem they are trying to solve. Without contract testing I would expect to have functional testing at both the provider and consumer side with unit tests and component integration tests. These would use mocks/doubles and could go out of sync from the providing service. We would normally have system integration tests that run through some smoke/sanity tests of core user flows when its deployed into a shared environment. This is where we would see our biggest problems, where regressions were introduced not due to changes in the functional flow, but due to the messaging exchange breaking down due to a schema change in an API. Contract testing plugged that gap, and the way the pacts are generated in the consumer side code, means the pacts are generated as a side effect of the functional testing taking place on the consumer side. (so I would use a unit testing framework such as jest to assert that my client does whatever it should with a valid response, as I assert against the clients output) so getting my functional testing covered, but with the benefit of generating that pact for provider verification, ensuring that when we deploy we don't have a regression because of a schema change. We might still find that a system integration test may still fail but just to application logic and not message schema. On the provider side, the contract verification isn't about functionally testing the providers business logic.

I was writing my question in the context of https://docs.pact.io/consumer#use-pact-for-contract-testing-not-functional-testing-of-the-provider, avoiding the pitfall of having the consumer testing the provider API. The goal of the consumer is not to discover bugs in the provider (though this might come up as a by-product). So, it is clear that the functional testing of the provider API goes beyond a specific Pact written by a specific customer. So, from a Venn diagram perspective the "provider functional testing scope" is certainly not included in the "consumers contract testing scope". The fact that discovering bugs in the provider [...] might come up as a by-product, makes it clear that there is a non-empty intersection between the "consumers contract testing scope" and the "provider functional testing scope". Then, the last question is if the "consumers contract testing scope" is fully included in the "provider functional testing scope", or if there are tests in the "consumers contract testing scope" that are not in the "provider functional testing scope". My answer is "eventually no". Let me explain... The goal of the "provider functional testing" is to ensure that the provider does the right thing with a request, coming for any/all consumers! So, it's a "no". However, there is a dynamic. When the provider first defines its functional tests, you cannot expect him to cover all possible requests, and the contract tests from the consumers can enrich its functional tests. So, a contract test will be either already included within the "provider functional testing scope", or the provider will extend its functional tests in order to have it eventually included. So, as shown in the diagram below, I would see the "provider functional testing scope" successively expanding with new contract tests to eventually cover the "consumers contract testing scope". Note that, ideally, at each "iteration" the intersection is bigger making the needed expansion smaller/easier and smaller/easier.

Well thought out @Patrice Krakow