https://pact.io logo
Join Slack
Powered by
Does anybody know if the open source Pact solution...
# general
a
Does anybody know if the open source Pact solution (not pactflow) supports provider driven contract testing? In all the results i am getting back this is not directly answered, but I am leaning to a no.
m
If you're referring to what's called bidirectional contract testing then no. That's PactFlow only
a
No just provider driven contract testing
I felt like birectional 'solves' both approaches
m
can you elaborate on what you mean by that? Pact requires you to first write a consumer test, and then verify it with the provider. You can certainly have the provider kind of drive the process, but Pact is naturally consumer-driven.
If that’s all you mean by it, then yes you can certainly have the provider guide the consumer on writing tests etc.
a
i mean the opposite flow 1: provider writes a test which produces a pact json contract/uploads the openapi spec 2: the consumer somehow verifies if this works from their side
m
no, that can’t be done with Pact
a
Yeah I thought so, just wanted to check
thanks 🙂
m
no probs
b
Hi Matt, is it a restriction to the open source version? as in, does provider driven work in pactflow ?
I kind of fail to understand why it would not be possible. The provider creates the contract. The consumer now knows from the contract the shape of the request and the response, and can verify if that request and response matches. Cant the contract be used as a stub for the consumer? It sounds really easy, so i feel like i am missing something why it can't work.
y
you could as a provider create a pact file and upload it, but to what benefit? this might be an additional overlay, such that its an example of expected usage for consumers, which could be used as documentation of sorts. however i think you will end up with duplication of effort a) consumers and the provider need to write pact contracts b) where does provider create contract tests from? their own codebase, or an example consumer that resides in another codebase but generated by the team. how are they going to update it? what happens if their considered use cases are far removed from their actual consumers. ie how can you make that assumption of the fields a consumer may or may not use, so would the provider pact just always expect all items in the resp, or a subset? what about the request permutations? lots of potential scenarioes to test of limited value, as you don’t have directed scope so as a provider developer, may actually find it hard to find truely required endpoints / fields from consumers, because they now have an effective consumer - themselves, that consume all the things you can use pact to diff two pact contracts. as with life, and open source especially, you could theoretically do all the things, however that thing needs to be a considered part of the application. The Pact broker hasn’t been built to cater for your scenario so work would need to be done to make it a first class thing. Essentially a Pact Broker plus aka PactFlow
tldr; no, not natively with a pb, it hasn’t been designed that way. however. there is a potential for a yes, with additional glue either on top / outside of the pb, which also requires you document that bespoke thing, which may impact adoption with Pact and its standard workflow. I’ve thought about the contracts advertised as example by providers, which could be used for diffing against incoming consumer pacts, as a potential supplement to onboarding docs, but know it involves some thought and additional coding
b
Okay, i think you explanation is indeed more towards bi-directional, but i still firstly fail to understand why the consumer can't run a pact verification. This would remove the need to create a consumer contract The reason why we are asking about this, is because we have a provider with ~5 consumers, what is the threshold for the amount of consumer for consumer driven to still be maintainable and valuable?
y
if the consumer doesn’t create the contracts, surely a provider developer has to assume all the surface area of that api is in use, until proven otherwise. would your scenario include a way of the consumer reporting what it will send and expect to receive so the provider developer can get a overlay of the exposed api, vs the consumed api? the threshold will probably differ on a case by case basis. I don’t see it as huge additional burden as these things may not change often, and if they do change offen, having that early feedback as a provider is invaluable as they can make more informed decisions, prior to having conversations with consumers, as they have the eagle eye view of exactly who, is using exactly what. we preferred cdct for that reason and then used the pact stub server to help drive our higher level tests and demo environments against pr’s to help showcase the change to PO’s. there was/is a cost to Pact, but that time otherwise spent later in the cycle, with less visibility. btw you can post verification results via api calls to the pact broker yourself or via automation, so there is no reason you couldn’t potentially implement what you are suggesting, but I can’t see the value add, and which is why I don’t really get the OpenAPI as the single oracle for contract testing that some other vendors approach are directed towards and prefer the granular insight from the contracts being generated from a consumers unit tests, closest to the actual code that matters. as a provider, i should be doing everything in my power to meet their ongoing obligations, otherwise what is the point of providing a service?
b
I'd add: it's not really about the threshold, it's an intentional philosophical opposition. Every consumer can (and usually does) have a different contract footprint (i.e. they rarely use every endpoint the same). An alternative approach might be to build an SDK lib for consumers to use, then it could have a single consumer-driven contract (with a new set of testing problems).
m
This point is probably the key one, and the reason why Pact (as a tool) doesn’t support it. Pact intentionally takes quite a strict approach to contracts - specifically shying away from general schema validation (there are plenty of ways to do that already).
But, that doesn’t mean we haven’t thought about provider-driven approach. In theory, yes, it could be done in reverse. There are some limitations in the Pact model that inherently make it more complicated to implement, but putting those aside, the big ones are preserving the “known consumer surface area” element of usage and of course, preventing breaking changes. The other big barrier is more practical - rolling it out ecosystem wide (all languages need to adopt it, the Pact Broker has assumptions about order and role etc. etc.) Let’s humour the idea, though:
The consumer now knows from the contract the shape of the request and the response, and can verify if that request and response matches.
Cant the contract be used as a stub for the consumer?
It sounds really easy, so i feel like i am missing something why it can’t work.
Let’s assume the consumer gets its hands on a contract (a pact file). It would be relatively trivial (in fact I think it’s possible it already exists in the core engine) to give a pact file to a mock server and ensure the consumer doesn’t make any calls to it that don’t align with it (actually, the current FFI method that supports this would fail unless all interactions are called, but this again should be straightforward to address). So far so good. The problem is we don’t know what field values are used (just the endpoints and variations on them, assuming the provider states are used). So that’s a downside. We would need a way to send results back up the chain, but that’s relatively easy. What I would want from this, is to possibly regenerate a Pact file that is specific to this consumer, so we now have a snapshot of what it needs (or perhaps that’s encoded in the consumer verification results, somehow). You could make this version support the next level of detail, by also making the consumer write Pact tests so we get the field-by-field fidelity (but this would obviously complicate the process, and probably reduce the benefit you’re looking for)
The provider creates the contract.
how would it go about doing this? Are you expecting them to effectively write their own consumer tests against themselves and publish it?
a
Interesting discussion:
how would it go about doing this? Are you expecting them to effectively write their own consumer tests against themselves and publish it?
I suppose the provider could generate an openAPI spec from their source code (every time all their unit tests pass) Then it is a matter of converting that spec to the DSL Pact understands and publishing it
m
Going from pact to OAS is possible but not the other way around, at least not without taking certain assumptions. In particular, anyOf and related keywords make it impossible to know statically what combinations of in/out payloads are allowed (polymorphisms).
b
Thank you all for the replies, concepts and idea's. I thought i remembered somewhere that when you have an API with a lot of consumers, eg a public api, then consumer driven is not the right approach. From the responses i gather that consumer driven can still be achieved with multiple consumers, and the alternative is generally is bi-directional testing, to ensure the consumers needs are clear. I will read the replies a few more times to get a good grasp on the topic.
Open in Slack
PreviousNext
Powered by