Those things aren’t in conflict, although I can un...
# general
m
Those things aren’t in conflict, although I can understand the confusion. Ideally, you just test your code as close to production as you can - that means testing as many layers of your provider that is reasonable to do. The objective, however, is not to perform a functional test. A side effect of this setup, is that you probably will incorporate some functional testing. i.e. it’s an accidental part of the test itself, not the objective. If testing many layers creates complexity in the setup, then you can stub more layers as necessary. All of these decisions need to come from you, because only you know your own test setup and the details of what’s covered. If you have really good separation of concerns, and those concerns are well tested, then you can go shallower in your contract tests because you have the other tests in place to give you confidence
d
Hi @Matt (pactflow.io / pact-js / pact-go) I’d like to clarify some aspects of provider testing in the context of contract tests: 1. Provider Test Environment: Do provider tests typically run against real service endpoints? We’re aiming to validate contracts against an environment that closely mirrors production, rather than a heavily mocked setup. 2. Data Availability: Given that consumer tests use mocks, how do we ensure the necessary data is available in the provider’s test environment? This seems crucial for meaningful contract validation. 3. Handling Dependencies: Our provider service often depends on multiple upstream services. What’s the recommended approach for setting up test data when the provider has a complex dependency chain? 4. Best Practices for Data Setup: Could you outline best practices for preparing the test environment and data for provider tests? We’re particularly interested in strategies that balance realism with test isolation and repeatability. 5. Mocking vs. Real Services: Is there a general guideline on which components to mock and which to use real instances of during provider verification? How do we strike a balance between comprehensive testing and manageable test environments?
m
1. Provider Test Environment: Do provider tests typically run against real service endpoints? We’re aiming to validate contracts against an environment that closely mirrors production, rather than a heavily mocked setup.
real service, yes, usually with stubbed downstream dependecies (because otherwise you’re potentially straying back into functional e2e testing with all of the challenges. It’s also hard to manage provider states this way.
👍 1
1. Data Availability: Given that consumer tests use mocks, how do we ensure the necessary data is available in the provider’s test environment? This seems crucial for meaningful contract validation.
see provider states
> 1. Handling Dependencies: Our provider service often depends on multiple upstream services. What’s the recommended approach for setting up test data when the provider has a complex dependency chain? > stubs + provider states, and contract tests with the stubbed downstream systems
1. Best Practices for Data Setup: Could you outline best practices for preparing the test environment and data for provider tests? We’re particularly interested in strategies that balance realism with test isolation and repeatability.
Similar to the above point.
1. Mocking vs. Real Services: Is there a general guideline on which components to mock and which to use real instances of during provider verification? How do we strike a balance between comprehensive testing and manageable test environments?
This article talks about the “scope” which might help: https://docs.pact.io/getting_started/testing-scope
🙌 1
Overall comment - the more you think about contract testing as a type of unit testing the better you will be, and the more obvious/easier many of these answers start to become. It’s strictly not a unit test, but the mindset and approach should look like it. If you go back through your above questions with the assumption that you’re writing unit tests, the pathway might be a bit clearier
👌 1
How do we strike a balance between comprehensive testing and manageable test environments?
To reiterate this point: running pact tests against live environments is discouraged, because of the above point about unit tests, but also it really takes us a lot closer to e2e tests again - which is usually what we’ve been avoiding by setting up contract tests in the first place
d
Thank you for the detail response!
🙌 1