https://pact.io logo
#general
Title
# general
d

David Van Couvering (DVC) - eBay

04/05/2022, 9:30 PM
Hey, all. One of the complaints I am hearing from teams looking at Pact is the amount of work it takes to both create and maintain all the interaction specifications that are included in writing the consumer tests, as well as the work to create and maintain the mocks used by the provider verification tests. We had one team who just didn't have the time to fix the contract tests due to failures when the contract changes, and instead just stopped running the contract tests. I'm curious, have others encountered this challenge? How have you addressed this? Thanks!
b

Boris

04/05/2022, 11:23 PM
I tend to take an aggressive stance on that attitude, I rarely see the point in good faith . . . like, people are saying they're too busy to make sure their software works properly? From a more constructive perspective, I'd want to know where they get their confidence from 🙂 Are there other tests that cover things enough that they don't see the contracts as high-value? Or are there few tests, and things have been manually tested in a rush? Or something else? Also, where is the time pressure coming from? E.g. if the devs had more time would they be happy to write the tests, or is it just an excuse to avoid (perceived) boring/unproductive work?
If the refactoring burden is genuinely huge, I also wonder if the balance of coverage is wrong. Conversely, if there are many many edge cases to cover, the tests are surely necessary, because people won't remember to manually test them all.
Also interested in which team(s) didn't want to maintain stuff: consumer-side or provider-side (or the same team is doing both sides?)?
b

Beth (pactflow.io/Pact Broker/pact-ruby)

04/06/2022, 6:04 AM
Andras makes good points (as always). We do hear that provider state maintenance is one of the biggest challenges, and it's one of the reasons we recently released a feature called bi-directional contract testing which does help mitigate the provider state issue https://pactflow.io/blog/introducing-bi-directional-contract-testing/
☝️ 1
d

David Van Couvering (DVC) - eBay

04/07/2022, 5:49 PM
Thanks. In my case it's maintaining the expectations defined in the consumer tests that is a huge burden. And as I have been thinking about this more, if you have a huge contract that is difficult to maintain, the underlying problem is actually that the API (and likely the underlying service) is poorly designed - a big surface area indicates a service that has poor cohesion and likely high coupling. I am now asking myself, how much time do I want to spend easing the burden of you having to work with such an API, when the right way to solve this problem is to refactor the API and/or the service.
That said, it is a lot of boilerplate work to define your expected interactions in the consumer tests. I am working on a tool that can listen to live traffic, record the interactions, and then generate a starting set of expected interactions code. You can then go in and tweak the code, rather than having to write it all by hand. Does that sound useful/interesting?
👍 1
The bi-directional contracts are great for provider state. We need something for the consumer tests
This code could also be used to publish an initial contract that is likely way too strict and brittle, and then over time could be made more reasonable as you address unnecessary failures in the contract (like an exact match failing that does not need to be an exact match)
This could also bootstrap some stubs for you - you publish this as an initial contract, and now your UI tests and/or your verification tests could use these stubs when interacting with downstream dependencies.
🤔 1
These are all just ideas, would love your thoughts
@Boris I wasn't sure what you meant by this can you explain? What is "balance of coverage" in this context?
If the refactoring burden is genuinely huge, I also wonder if the balance of coverage is wrong.
👀 1
b

Beth (pactflow.io/Pact Broker/pact-ruby)

04/07/2022, 10:20 PM
And as I have been thinking about this more, if you have a huge contract that is difficult to maintain, the underlying problem is actually that the API (and likely the underlying service) is poorly designed - a big surface area indicates a service that has poor cohesion and likely high coupling.
That is such a valid point. When tests are hard it's sending you a message that using it is likely also hard.
This code could also be used to publish an initial contract that is likely way too strict and brittle,
Yes, it's very easy to make overly specific pacts. I rant about this at great length in some of my "how to do the consumer side" materials. eg. https://docs.pact.io/consumer#only-make-assertions-about-things-that-will-affect-the-consumer-if-they-change https://docs.pact.io/consumer#choose-the-right-type-of-matching-for-the-situation and the video in https://docs.pact.io/consumer#watch-a-video-writing-good-consumer-tests
I am working on a tool that can listen to live traffic, record the interactions, and then generate a starting set of expected interactions code. You can then go in and tweak the code, rather than having to write it all by hand.
Does that sound useful/interesting?
Yes - we've long talked about being able to generate pact tests from a set of interactions, but we've always had too many other things to do first. The key is that we want to generate test code, not the contract itself. I'm not 100% certain how much "wheat" will come out of such a generation, and how much "chaff" though. It's absolutely worth the experiment though, and would be such a big help introducing contract testing to large existing systems if it ended up working as we'd hope.
d

David Van Couvering (DVC) - eBay

04/07/2022, 10:48 PM
Thanks, Beth. I am working on an experiment. I was doing it in Python using Jinja2 as a templating language to create templates for various target languages. If you feel another language would be a better fit, let me know. I have just gotten started so no great loss if I go to another language (although I don't know Rust so that could take me some time to pick up). I am using WireMock as v1, but that is generally limited to environments where you can configure your client to send http to Wiremock. Setting up Wiremock to be an HTTPS person-in-the-middle is tricky. You generally need to use an actual HTTP/HTTPS proxy recorder. I know this could be of general use to the community, so if there is a process you like to use to make sure it gets visibility, input and maybe cross-contribution, please let me know how I might do that. On my side I need to see what eBay requires of me before I contribute work to open source.
b

Beth (pactflow.io/Pact Broker/pact-ruby)

04/07/2022, 11:01 PM
Sounds great!
d

David Van Couvering (DVC) - eBay

04/07/2022, 11:01 PM
👍
Python ok?
b

Beth (pactflow.io/Pact Broker/pact-ruby)

04/07/2022, 11:02 PM
Yeah, I reckon.
d

David Van Couvering (DVC) - eBay

04/07/2022, 11:02 PM
k
b

Beth (pactflow.io/Pact Broker/pact-ruby)

04/07/2022, 11:03 PM
We don't have any formal process yet, but getting some shape around how to contribute pact related tools is something that is right in the job description for @Yousaf Nabi (pactflow.io).
d

David Van Couvering (DVC) - eBay

04/07/2022, 11:04 PM
OK. I'll work on it in my internal repo for now, and will coordinate with @Yousaf Nabi (pactflow.io)
b

Beth (pactflow.io/Pact Broker/pact-ruby)

04/07/2022, 11:04 PM
I reckon, get a POC going, see how you find it, then if you think it's worth keeping on going, write up something and/or present to an audience who might like to be involved with contributing or testing, and then see where we go from there.
If we reckon it's a goer, we can work out a plan to make it an "official pact foundation" resource.
d

David Van Couvering (DVC) - eBay

04/07/2022, 11:06 PM
ok sounds good. BTW, the only place I've heard "I reckon" is from farmers in the American West and now, apparently, Auzzies 🙂
b

Beth (pactflow.io/Pact Broker/pact-ruby)

04/07/2022, 11:06 PM
HA!
it is very american isn't it.
d

David Van Couvering (DVC) - eBay

04/07/2022, 11:06 PM
Yes ma'am, I reckon it is
b

Beth (pactflow.io/Pact Broker/pact-ruby)

04/07/2022, 11:06 PM
I don't think anyone would think it out of place in australian slang though. we've picked up so much from the US.
except when I hear it in my head in a US accent, it's more like "ah reahcahn"
in australia it's more "oi reckn"
d

David Van Couvering (DVC) - eBay

04/07/2022, 11:08 PM
Ah I see, I thought maybe it had a shared ancestor from English that went to both American and Australia. But I guess it could have just come from movies
☝🏼 1
Nope, here it's also "reckn"
😆 1
b

Beth (pactflow.io/Pact Broker/pact-ruby)

04/07/2022, 11:25 PM
Australia has so much american content, we're very familiar with any lingo that makes it to a screen. Not so much the other way though! Apparently we can be quite hard to understand 😆
Yousaf is just being introduced to our Australian passion for abbreviating things and whacking an "o" on the end. eg. arvo (afternoon)
b

Boris

04/08/2022, 1:33 AM
Not sure how relevant it is anymore, but re:
I wasn't sure what you meant by this can you explain? What is "balance of coverage" in this context?
> If the refactoring burden is genuinely huge, I also wonder if the balance of coverage is wrong.
You've both kinda covered it with "contract that is likely way too strict and brittle" and "Yes, it's very easy to make overly specific pacts". Sometimes people push too much stuff into Pact (so some testing should move to other areas of the pyramid), or have too many specific interactions that could be generalised a bit.
d

David Van Couvering (DVC) - eBay

04/08/2022, 2:07 AM
Got it thanks
👍 1
2 Views