A quick question for the experts 1. If we already have schema and integration tests in our CI pipelines and are tested as a part of each PR to master, do I still benefit from using Pact? 2. We are following the following approach: Acceptance Criterias created by BA-> Team refines the story -> Squad teams pick up the story fort implementation(please bear that the contracts are not ready here, before our domian services needs to call the third psrty vendor’s APIs) -> Implementation completed (domain services end points created after understanding how the third party vendors apis behave)-> Swagger docs are created-> QEs create their integrations tests and schema tests and are ready to merged into the main PR-> PR is now raised against master -> all checks are made before it gets deployed to master.. Same happens at the consumer side as well.. Can somebody let me know if we would still benefit out of Pact and if yes how? Highly appreciate your thoughts on it. Thanks

1. You can benefit when making changes to the "interface", i.e. when you are wanting to add or change the producer/consumer. I'm currently adding into some existing services to capture how they currently worked, before then going onto make a few hefty structure changes and it's been helpful so far picking out when my integration tests were all fine but e.g. one end had a timestamp with a T while the other had a space. By themselves they worked fine, together they did not

integration tests

can you please elaborate what you mean by this? The definition has widely ranging meanings, so it’s important to make this clear

integration tests - testing the integration between my domain services & third party vendors and when I hit my application endpoint it actually hits the third party api endpoint and gets the real time response and not mock data.

Well, only the usual benefits of speed, reliability, visibility, precision etc.. If you test all of the scenarios you need against the real thing, then Pact tests are just going give you faster feedback loops, but no more guarantees than testing against the real thing.

Vs integration tests: Pact gives you deployment safety that most people aren't getting with integration tests. With integration tests, when you're about to deploy something, you need to run an integration test against exactly the versions it's about to deploy against. This is expensive in time and resources (and very hard to do without a test environment that needs booking exclusively for you). It's .... possible, but most of the time it's so much work that people aren't doing it, or aren't doing it strictly. Even if you are, the test setup time is so long that it substantially increases the friction for small fixes Vs Schema tests: Schema tests tell you whether something matches the grammar of the request/response contract. A best-practice Pact test will tell you whether something matches the semantics of the contract. This is substantially more valuable. Pact is by-example, so you're actually testing whether your provider can really understand the requests that your consumer is really sending. Also, depending on your contract design some things can't be expressed (or are difficult to express so people take shortcuts) in schema tests. For example, when you have dependent properties:

{
   "type": "foo"
   "fooField": {....}
}

and 

{
   "type": "bar"
   "barField": {....}
}

typically this is expressed in schema as:

{
   type: "bar" | "foo"
   barField: Optional<BarField>
   fooField: Optional<FooField>
}

which means the following two examples are schema-compliant, but probably wrong:

{
   type: "bar"
}

and

{
   type: "bar"
   fooField: { ... }
}

Spot on @Timothy Jones - will get that on the site either in the FAQ https://docs.pact.io/faq or under Pact concepts https://docs.pact.io/getting_started/conceptual_overview will check messaging on here too https://docs.pact.io/faq/convinceme

@Timothy Jones Thanks for your detailed response. I am running my integration tests at CI level , well before CD. In my CI, a docker build agent will spin up and I am creating an environment to have all the domain services up and running and connected to different third party vendors and hence I am able to run the tests before it gets deployed. Thats helping me to identify issues even before it gets deployed. AT the same time , you are also right--->setting up this environment in the build agent might take few minutes , and will eventually add up to the whole CI execution time. If possible can you give me an example of difference between grammar vs semantics testing of a contract. I want to understand what my current schema tests lack?

The example in my post above is the main one I like to illustrate this problem. Another problem is that the schema can't tell you what's valid- eg a delete for a resource gives a different response if the response doesn't exist. This information is implied in a schema, or encoded in comments. A good contract test will include this information directly.

@Timothy Jones I am using AJV to do my schema testing. I am using an http client to call my endpoint with -verb, headers, payload, and matching the response with the schema.. And I am doing this for variety of combinations (e.g what req will give back different response codes/response body). So am I wrong in assuming that I am also checking the HTTP level semantics using AJV?

It's been a while since I looked at AJV in detail. My understanding is that v doesn't know about HTTP semantics like paths, verbs etc.

Interesting. Would you be comfortable sharing an example from your tests?

@Matt (pactflow.io / pact-js / pact-go) @Timothy Jones AJV doesn’t have to know the paths and semantics, we write our tests using an http client(superagent in our case) to hit an endpoint with respective verbs, headers, endpoint , query params, etc etc…and capture the response and then validate them against the schemas we generated through a script( this script basically takes the response which we think is ok once the endpoint is implemented, and then convert it into a schema). 2. Not only we verify the schema but we also check response code, values within the response(e.g, the currency type should be £ and not $)- this way we also test E2E integration and contract test in one single scenario. Dont need two tests for it. 2. We write tests to check all possible scenarios - negative tests, and simulate different scenarios which sends different response code. I am very keen to run past an example with both of you and really wanted to understand and also give some insights to you on how I am doing my contract and integration testing — I went through the complete documentation of Pact and BDCT , but just have few things which would be good for all of us to discuss and get a different perspective.. Any time slot you both are comfortable and I can run past the things in less than 10 minutes - which covers my Ci-CD process, Contract and Integration testing (all happening before even the code gets deployed — basically how I implemented Pacts’s ‘CAN-I-DEPLOY’..