Hi all - I'm investigating using Pact on a project and have read through all the docs, but could use help better understanding what sort of test cases to use pact for. A few questions: - Is contract testing mainly about catching breaking structural API changes? If it goes beyond that, how do you determine where to draw the line between functional tests? - If there are several ways we can populate an update (PUT) request which lead to different values in the response, would we want to test several of those, or just one overall test for an update? - We have some large requests that encompass several smaller components. Do we want to have tests that cover the entire large object in addition to tests for the smaller components?

Is contract testing mainly about catching breaking structural API changes?

It depends. I would say contract testing is about confirming whether two systems are able to communicate - in some system designs, this will mainly be about catching breaking structural API changes.

would we want to test several of those, or just one overall test for an update?

Test each communication scenario - each time you would be exercising a different meaning or a different code path, add a test for that. Think of it as contract-by-example.

large requests that encompass several smaller components.

I don't think I understand this question. Could you elaborate on what you mean by component?

Thanks for the responses!

Test each communication scenario - each time you would be exercising a different meaning or a different code path, add a test for that. Think of it as contract-by-example.

Hmm, that sounds a bit like functional testing to me. Maybe a bit more about my situation would be helpful. My domain involves managing clustered "deployments" of services. We have one request/resource representing a deployment that you can update in different ways, which somewhat represents different use cases. For example, updating a deployment's version causes it to be upgraded. Upgrading a deployment's node count causes it to be scaled out. These are different use cases, but are accomplished via the same "update deployment" request/API. Would it be appropriate to contract test each of these since the structure of the request is the same, and only the contents of the requests and responses differ?

Ah, I see what you're asking. I would say that the contract test covers the serialisers and the API calling / receiving code. In the case of multiple different responses but with similar structure, it can be helpful to add a case for different responses - for example, say you could have a response that is:

{
  "status": "deployed"
}

and

{
  "status": "updating"
}

then it's useful to explicitly cover those: "given some service exists, then a request to update that service, responds with the status: updating"

Thanks. Another example that might help clarify things for me, say I have a request that looks like this:

{
  "applicationCount": 3,
  "applicationConfig": ...
}

and leads to a response like this:

{
  "applications": [
    {
      "endpoint": ...
      "credentials": ...
    },
    {
      "endpoint": ...
      "credentials": ...
    },
    {
      "endpoint": ...
      "credentials": ...
    },
  ]
}

I imagine it would make sense to assert that some value for applicationCount and applicationConfig leads to the appropriate response, but not necessarily that various different values do. Does that seem right?

Yes, that’s right. To determine where I stop enumerating scenarios like that, I would think about what risk the test is reducing, if that makes sense?

Thanks, yea, there are use cases we certainly want to test, I'm just trying to figure out how we draw the line between making something an end-to-end or integration test vs a contract test. For example, the difference between an "upgrade" vs an "upscale" use case for us is just changing one field vs another in the same request document. How that impacts the API response is minimal, but how that impacts the rest of our system is big.

Right, yes. Contract testing is about understanding - can service A understand service B. This is important because it’s cheaper to test that with a contract test than an integration test, and that can give you deployment confidence

Cool, yea so is it fair to say that contract tests can replace e2e or integration tests to the extent that they're asserting API compatibilities, but that asserting correct functionality is still needed separately?

To get the same confidence with an integration test you would need to deploy exactly the same version of all of your dependencies as you have in prod. For most ecosystems this is not practical

Re: one of the other questions above:

Do we want to have tests that cover the entire large object in addition to tests for the smaller components?

I don't think I understand this question. Could you elaborate on what you mean by component?

Sure, for example, say we have a "create deployment" request that has a few different sections in it:

{
  "databaseConfig": {
    "nodeCount": 3,
    "nodeSize": "4G",
    "partitions": 24
  },
  "proxyConfig": {
    "nodeCount": 3,
    "nodeSize": "4G",
    "loadBalancing": true
  }
}

Would we want to create separate test cases to assert that the

databaseConfig

or

proxyConfig

are individually valid, or a single test case for the entire request?

Ah, I understand. To answer that it might depend a bit on how many combinations you have. I had a similar case with a document description that had several different possible section types in arbitrary combinations- so I had contract tests for each type of section, and then one “here’s an example complex doc”

I think @Timothy Jones deserves a whole meal, and not just a 🌮 for this thread! 👏

Yes, thanks Timothy!

You're so welcome! Let us know how you go

Should a consumer test assert that the contents of a response correspond to the contents of a request?

I appreciate your effort in all your thoughtful answers @Jonathan Halterman (this one, and many others) I've learned a lot from them!

Should a consumer test assert that the contents of a response correspond to the contents of a request?

I'm not 100% on what you're asking here - I think there are two interpretations, so I'll answer both: If you're asking about what to put in the contract (whether to be specific

"status": "deployed"

, or general with a matcher

"status": any string

), then it depends. If it's part of the expectation of the consumer, then yes. For example, if you say "I'd like to change the status of this object to 'deployed'" and the resulting object is returned in the response, then yes, it should be in the contract. Another yes scenario might be where the consumer relies on it being a specific value (eg, if you're switching display logic on

deployed

vs

not-deployed

or something). If you're just displaying it, then no, it doesn't matter. A good guide for "should I use a matcher here" is to realise that using a matcher isn't a description of the response schema, it's a promise that all responses that pass the matcher are covered by the test that you've written.

Thanks - yea I noticed some of the example consumer tests basically just assert that a response deserializes successfully, but others assert the contents of the response either using a matcher or an exact match. Here's an example consumer test where an exact match is being performed: https://github.com/pact-foundation/pact-go/blob/master/examples/messages/consumer/message_pact_consumer_test.go#L36 It sounds like that kind of thing is ok if the value being exactly matched is meaningful for the test.

That’s right. The matchers really are a convenience thing for the provider team (so they don’t have to have identical test data)

The downside of that test above is that the provider team has to ensure that the StateHandler they use for that test populates "name": "Baz" in the provided object?

Like “get a user has a username field, which I promise I will understand if it is a string”