Probabably the biggest limitation with Pact's consumer-driven contract tests (for us) has been ensuring exhaustive enum (or object literal) compatibility. That is to say - if the provider response looks like

{
  animalType: "Cat" | "Dog" | "Mouse"
}

then we need it to be a breaking change if the provider might start returning an

animalType

of "Giraffe". As far as I know, there's no real solution for that with consumer-driven contract testing. The consumer contract can say

animalType: regex('Cat|Dog|Mouse', 'Dog'),

That will result in a contract-by-example having "Dog". But when the provider evaluates itself against that contract, the evaluation will succeeed so long as the test doesn't set the

animalType

to "Giraffe" (even though it could in production). We're wondering - does switching to bi-directional contract testing solve this problem? If the provider publishes an OAS schema that says (under version 2.0.1 of the provider)

animalType

can now take on a value of "Giraffe", that version of the provider is incompatible with the consumer?

So yes, this is a negative test scenario and that is very hard to test. Think about it, the only way to ensure it doesn’t return a value is to …do what exactly? With BDCT, all we do is compare the consumer requests (ignoring matching rules) against the Provider OAS (including the schemas). If the consumer is a valid subset, then it passes

Yeah - I think we're going to need to look into Json Schema comparision on both sides.

Yes, that was sloppy language on my part. Pact will fail if it’s not in the constraints, but the hard problem is proving that the provider won’t ever send a different value. If you can guarantee the schema is always correct (e.g. by generating it from types or something) then perhaps you could give some additional confidence. What problem are you actually trying to solve, though? What if the OAS said one thing but your provider could return an additional value anyway? How do you know for sure it can’t? I’m generally not a fan of enforcing such constraints from the consumer perspective, and dealing with the range of values that could come back. Easier said than done without context

If you can guarantee the schema is always correct (e.g. by generating it from types or something) then perhaps you could give some additional confidence

Yeah I think this is the approach we'll probably take • Start with a JSON schema • Using SourceGenerators (C#) on the provider side, create C# types from the schema ◦ C# codebase will use the generated types • Use some tool to generate TypeScript types from the schema on the client side ◦ TS codebase will use the generated types • Do something (possibly as part of the contract) to ensure the schema versions are the same ❓

What problem are you actually trying to solve, though

Our consumer is uploading a document to be processed/transformed. The provider returns a response based on the uploaded document; the consumer needs to fully understand the response it got back (transformation result). If the consumer can't support a given transformation / aspect of the result, the provider shouldn't generate it until the consumer has been updated to handle it.

Would the provider behaviour still be complying to their own spec / expectations if they’d return

Giraffe

as an

animalType

? Because to me, this doesn’t sound like a case that should be covered through a contract test but rather by provider-side testing of the implementation. But of course, that also means you’ll have to trust the provider to do that type of testing.

This is a divisive topic in the OAS world. Zalando’s (popular) guidelines suggest enums should be extensible/open by default: https://opensource.zalando.com/restful-api-guidelines/#112. I can understand why they go this way, but strictly speaking I would also say it’s a breaking change. It’s tricky to statically check though, at least for something like Pact, because we can’t know what we haven’t tested. And by definition, we can’t test things we don’t know about! (i.e. we can’t test all possible values of the enum that aren’t described). In BDCT, we could consider including the Pact matchers in the comparison. This sounds simply, but is in fact quite complicated because there aren’t straightforward projections/transformations of all matchers to JSON schema types. So one shortcut could be to only consider some matchers. Regexes aren’t directly comparable to an enum though, so we might need to consider adding an “enum”-like matcher to support it