👋 I need to establish a conformance certification program around an OAS that will have three conformance profiles related to three subsets of the overall OAS (forecast, real-time, and peering). I need to be able to certify conformance of multiple implementations of the OAS, and I'd like to be able to provide the conformance test suite to implementors for their use in their own CI/CD pipelines. (Part of the conformance suite will be a data file that represents that provider state at the beginning of the test.) I don't have a mandate to build clients or a provider, but I think I will need to build at least the latter of those to develop the conformance profile tests. Each of the profiles will need to ensure the successful execution of a handful of scenarios. A primary consideration is the DevEx of the provider implementors. In particular, I want it to be easy for them to understand why a particular scenario is not passing; some scenarios may involve a series of API calls so being able grok the overall journey is important. I am also concerned about sustainment: HTTP, JSON, and OAS were safe, accessible choices with long-term ecosystems. Relatedly, I need something that is "free-as-in-beer", even if the project is ultimately able to fund some quality of life improvements in the toolchain. There's more than one way to do it, as they say. I've looked at Karate, Hoppscotch, Postman/Newman, and Insomnia/Inso. They all seem plausible. My gut tells me that I want the provider contracts to live as independent artifacts of equal importance alongside the OAS itself. More to the point, since the OAS was developed spec-first instead of TDD/BDD-style, I'm quite certain developing the provider contract will identify gaps and other failings in the design, so I'd like to iterate the OAS and the contracts together until 1.0. Would you have any advice to give?

First up, sorry for the delay in response - lots of us have been/are on annual leave right now