https://pact.io logo
Join Slack
Powered by
Hi all. I've been working extensively with Pact, M...
# general
s
Hi all. I've been working extensively with Pact, MSW, and the rust implementation of pact mock server to run Jest tests against our generated pact files. I've looked at the msw adapter and I'm hoping to get some understanding on two things: 1. Doesn't generating pact files from UI test interactions violate Pact's warnings against writing contracts based on UI test cases? 2. Pact provides an API for clearly identifying exact values vs. types of values that may change via matchers - I would expect generated test cases to either be too exact or too loose. Can we really generate quality contracts this way? I guess I'm trying to understand if we're losing significant value by using this adapter. I'm interested in the efficiency of it, though. Any insights are appreciated!
t
What adapter?
Anyway, generally:
Doesn’t generating pact files from UI test interactions violate Pact’s warnings against writing contracts based on UI test cases?
Yes, try not to do this
What is under test in a Pact test is the API layer. You could drive that test from the UI layer, but I would expect it to be a pain to maintain and set up.
It’s much better to put the test “callipers” of Pact around what is actually under test - ie, the API client
generated test cases to either be too exact or too loose.
A subtlety that you might not have thought of is that the generic matchers aren’t for describing the schema, they’re for making it easier to write tests. They’re a convenience, so you don’t have to have the same exact test data on both sides. If you say 
{ name: like("Steve")}
, you’re saying “this particular test case covers all possible responses where the name field is a string”
Not “the name field could be any string”
(well, you are saying that too, but that’s not the main point)
This is why there aren’t optional matchers for “it could be this type or that type”
s
@Timothy Jones thanks for the response here! Replying to a couple of your comments: 1. The adapter I'm referring to is `pact-msw-adapter`: https://github.com/pactflow/pact-msw-adapter. MSW (mock service worker) is used here to capture requests and generate pact files on the fly without any actual pact test authoring by the engineer. Which, if this is truly a viable approach to contract testing, may be a game changer. Pretty much no one is using it, but it's official from Pact. So I'm at least intrigued. 2. I did not realize 
"Steve"
and 
Matchers.like("Steve")
produce identical output in the pact file, wow. Good point on the role of matchers, that is indeed subtle. One of the biggest challenges for contract testing is the paradigm shift, I'm still wrapping my head around that. So maybe using the adapter is a fine approach, and it's up to us to target tests specifically against the API layer. The thing that stands out to me, though, is the experience based feedback from Pact folks is that using UI tests with Pact should be avoided because the written tests are brittle and maintaining them is overly cumbersome while providing limited value. The adapter arguably changes this - we're not writing tests, pact files are just generated based on the tests, so they may maintain themselves to an extent. I still don't have a lot of real world experience with Pact yet, so wanted to see if this rang true at all for folks who do.
t
but it’s official from Pact
From Pactflow. A different thing
👍 1
Pactflow != Pact
But, thanks for the link, I’ll take a look
The adapter arguably changes this - we’re not writing tests, pact files are just generated based on the tests, so they may maintain themselves to an extent.
Hmmm… At first glance, I would say that it feels like it would have the same drawbacks long term, but be much faster to get going. Provider states would be the hard part
In http testing (which you’re probably doing if you’re using this mock service worker thing) you’ll have situations where the same request would generate different responses in different provider states
and it’s up to us to target tests specifically against the API layer
You’re 100% right about this. This, I think, is the main part of your question - if, in your situation, it’s possible to write UI tests that target the API layer, then you’ll be ok. If not (as is usually the case, at least long term), then you might be in for a bad time
m
This adapter (and others like it - e.g. wiremock, cypress) are really not designed to work with Pact (the tool) but to produce a pact file (contract). The pact file is an interoperable format that can be used in other contexts - in this case, it’s really designed for use with the Pactflow BDCT feature: https://docs.pactflow.io/docs/workshops/bi-directional-contract-testing. Doing so whilst still verifying pacts using the Pact tool still has the potential for several drawbacks as per other articles. In particular (and as mentioned), matching rules, provider states and the proliferation of tests are still causes for concern. I’m going to have a standard warning/note added to these adapters Github readme’s to make this much clearer.
It might be possible to add matchers and states which alleviate part of this, but that will also have its own set of tradeoffs.
3 Views
Open in Slack
PreviousNext
Powered by