Hello team, I have some questions about matching rules in bidirectional. it seems like all matches are using the pact examples, while ignoring the matchingRules Given this sample swagger:

{
  "openapi": "3.0.1",
  "info": {
    "title": "IDEServices",
    "description": "IDEServices REST API",
    "version": "v1"
  },
  "paths": {
    "/asset-model/v1/features": {
      "get": {
        "summary": "Gets the stored Feature Toggles.",
        "description": "Gets the stored Feature Toggles.",
        "operationId": "GetAllFeatureTogglesValues",
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "type": "array",
                  "items": {
                    "required": [
                      "key"
                    ],
                    "type": "object",
                    "properties": {
                      "key": {
                        "minLength": 10,
                        "type": "string"
                      }
                    },
                    "additionalProperties": false
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

and this sample pact:

{
  "consumer": {
    "name": "web_ide"
  },
  "interactions": [
    {
      "description": "Get all features",
      "pending": false,
      "request": {
        "method": "GET",
        "path": "/asset-model/v1/features"
      },
      "response": {
        "body": {
          "content": [
            {
              "key": "bigexample"
            }
          ],
          "contentType": "application/json",
          "encoded": false
        },
        "headers": {
          "Content-Type": [
            "application/json"
          ]
        },
        "matchingRules": {
          "body": {
            "$[*].key": {
              "combine": "AND",
              "matchers": [
                {
                  "match": "what I put here doesnt seem to matter"
                }
              ]
            }
          },
          "header": {},
          "status": {}
        },
        "status": 200
      },
      "type": "Synchronous/HTTP"
    }
  ],
  "metadata": {
    "pact-js": {
      "version": "14.0.0"
    },
    "pactRust": {
      "ffi": "0.4.22",
      "models": "1.2.3"
    },
    "pactSpecification": {
      "version": "4.0"
    }
  },
  "provider": {
    "name": "IDE-Services"
  }
}

I get no errors no matter how I change the matchingRules. but if I change the example to something else, I get

location: '[root].paths./asset-model/v1/features.get.responses.200.content.application/json.schema.items.properties.key.minLength'

are we doing something wrong? aren't the matchingRules suppose to be more important than the example in the request/response?

Hey, Matchers are unsupported for bi-directional https://docs.pactflow.io/docs/bi-directional-contract-testing/contracts/pact#converting-mocks-into-a-pact-compatible-format it does schema based validation based on rules in the provider contract, in your case an openapi document

so bidirectional only validates types? and can't even do regex?

it will validate regex constraints set in your openapi document

yes, but the consumers won't be able to have any inputs on that, right? the providers can set in their swaggers their rules, but there's nothing similar from the pact side. I was under the assumption that the rules of the pact had to match the rules of the swagger, not that the example of the pact had to match those rules

I was under the assumption that the rules of the pact had to match the rules of the swagger, not that the example of the pact had to match those rules

did you see that in the documentation anywhere? ie the following block in an openapi

bank_account_currency:
          maxLength: 3
          minLength: 3
          pattern: '^[A-Z]{3,3}$'
          type: string
          description: ''

the consumer is bound by the pattern in the openapi, if they need something different then they would need to discuss that with their provider. you can use matchers in your tests on the consumer side to make them more flexible for matching there, but as our documentation states, they will be ignored on the provider verification side. if you need verification with matchers, just switch to using full pact verification

yes, it looks like our assumptions were wrong.

you can use matchers in your tests on the consumer side to make them more flexible for matching there, but as our documentation states, they will be ignored on the provider verification side.

so this means that the consumer validation takes it in consideration? isn't it always a validation via

swagger-mock-validator

? so they would be ignored regardless, right?

your consumer tests that generate the pact in the first place

yeah, what I meant is that the true validation happens during that comparison, right?

so bidirectional only validates types? and can’t even do regex?

For clarity. it validates against the schema in the OpenAPI document

For clarity. it validates against the schema in the OpenAPI document

@Matt (pactflow.io / pact-js / pact-go) it is possible to have matchingRules in the request too, correct? we've been investing in bidirectional since we already generate our openAPIs for backstage so it had good synergy, but this is kind of a problem. our fear is that it creates a gap: 1. Consumer has pact with an example in it's body and some regex matchingRules 2. That example matches the regex from the provider swagger a. the test will pass, as expected 3. The provider changes their regex into something that still matches that example but breaks the consumer matching rules a. the test will pass when it breaks the consumer matchingRules is this a correct assumption?

it is possible to have matchingRules in the request too, correct?

yes, but they are only evaluated at contract generation time - they aren’t checked to match the OAS

I see. Well, thanks for the info, we'll have to ponder a bit on this, perhaps open a feature request. In your opinion, would it be possible to have a 2-way validation? The pact adhering to swagger rules and vice-versa?

we considered that, but it gets very hairy very quickly

I agree, but I do believe that the value brought by this additional protection would be meaningful, making bidirectional strictly superior to consumer-driven, even if with some constraints. We plan to make a feature request about this soon, since bidirectional is important to us, so please consider it.

Thanks. You could give it a try here to see how it works: https://github.com/pactflow/openapi-pact-comparator/ I think as you go through you’ll see some of the inherent complexity in doing so, but I think it would be an interesting exercise.

making bidirectional strictly superior to consumer-driven

I don’t think it would ever be superior to consumer-driven with Pact, because Pact actually replays the requests against the running provider - whereas BDCT only checks the schema. But I agree it would be stronger

Yeah. Sorry, I meant from our perspective, not as a generalized rule. The way we looked at bidirectional (considering validation from both sides) was considered superior to traditional pact, even with the caveat of not being able to have consumers dictate provider testing.

One of the big difficulties, is there isn’t suitable tooling, imo, that aids teams in ensuring that the provider api description, is actually in sync with the provider implementation. So replaying requests against a running provider, will always give you the most confidence. You can use the schema based validation in your workflow as well, as using Pact, whether you roll it your own way, or you use BDCT & CDCT for the same integration.