https://pact.io logo
Join Slack
Powered by
Hey all! Is there anyone doing code coverage with ...
# pact-js
a
Hey all! Is there anyone doing code coverage with the consumer tests? I'm trying to get 100% coverage on all of our API calls to providers making sure there is an interaction in a test per API call in our code. If it's possible, what does that setup look like a test runner Jest for reporting coverage? We're running our consumer tests separately from our unit tests. Thanks again!
t
You would do this the same way you normally would with your test runner
in jest, I think it does this by default?
I just checked one of my projects, and the last commit to 
jest.config.js
is “chore: Collect test coverage”, so I guess it doesn’t. Anyway, you set 
collectCoverage: true,
or 
--coverage
a
Thanks for responding! I know Jest can do coverage on unit tests, but the coverage report for specifically our consumer tests is low because it's covering the entire codebase instead of just the API calls that are being made. I'm wondering if there is a strategy to just report coverage against just the code that calls any provider's API? If it's not possible, is there an alternative to this?
t
I think this isn’t really a pact question, sorry. I think you can tell jest to not collect coverage from everything.
Frankly, I’m not really sure what value 100% coverage would have in a contract test. I don’t even know if you could. Or if you could, I think it would be kind of artificial.
For example, you won’t cover timeout related handlers in a contract test, because that’s not part of the contract
Nor will you cover branches that are for when the server returns an error response that doesn’t have the payload you’re expecting
a
I see, I think the meaning of what 100% is for my ask, is it possible to guarantee every interaction between a consumer and provider in a codebase has a contract test through code coverage (or some other way)? From the docs and what I've read in the slack archive, doesn't seem possible. I do appreciate the help!
Copy code
While the coverage metric can be helpful, it unfortunately won't be able to tell you whether or not you've covered every semantic variation of an endpoint. Determining that is currently beyond the scope of Pact, but is something that we would love to be able to solve in the future.
t
I haven’t thought of a way to do this. Depending on your code, you might be able to do a best guess using coverage. For example, my clients tend to have an entry layer where the each call comes through- which I could look at to ensure the code is at least run during the test
I can post an example here later
🙌 1
Copy code
{
    getAllProducts: () => server.authedGet<string[]>('/products'),
    getProduct: (id) => server.authedGet(`/products/${id}`),
    health: () =>
      server.get<WireServerHealth>('/health').then(({ status }) => status),
  };
^ I usually separate the layers like this. The first layer translates the path and the type of request
The second layer knows about the API client that I’m using - for example:
Copy code
authedGet: (path) =>
    axios
      .get(`${baseurl}${path}`, {
             headers: { Authorization: `Bearer ${authToken}` },
      })
      .then(unmarshallSuccess, unmarshallFailure),
  get: (path) =>
    axios
      .get(`${baseurl}${path}`, {})
      .then(unmarshallSuccess, unmarshallFailure),
Copy code
export const unmarshallSuccess = <T>(response: AxiosResponse<T>): T =>
  response.data;
Copy code
const isWireErrorResponse = (data: unknown): data is WireErrorResponse => {
  const maybeResponse = data as WireErrorResponse;
  return 'message' in maybeResponse && typeof maybeResponse === 'string';
};

export const unmarshallFailure = (error: Error): never => {
  if (axios.isAxiosError(error)) {
    if (error.response) {
      if (error.response.status === 401) {
        throw new ApiError(
          "The server says that you're not authorised.",
          API_NOT_AUTHORISED
        );
      }

      throw new ApiError(
        error.response.data && isWireErrorResponse(error.response.data)
          ? error.response.data.message
          : `The server returned an error code (${error.response.status})`,
        API_ERROR
      );
    }
    if (error.request) {
      throw new ApiError("The server didn't respond", API_NO_RESPONSE);
    }
  }
  throw new Error(`[API Failed] ${error.message}`);
};
The final layer is the unmarshallers
layering your client this way has nothing to do with pact, I just like the design. It means the entry point doesn’t need to know what kind of client you’re using, and can be easily skim read
What I might do is check the coverage in the first layer - this would ensure I have at least one test covering each endpoint. However, it wouldn’t check that I have a success and failure in each.
You could move the unmarshallers to the first layer, which would give you that coverage
In a real app, you might want handlers there anyway, so that you could re-marshall not found errors as “product not found” error or something. However, this also illustrates why Pact doesn’t do this as a feature - even if we moved the error handlers to the first layer, and only looked at whether both branches of say:
Copy code
getProduct(id: string): () => server.authedGet<Product>(`/products/${id}`)
   .then(unmarshallSuccess, unmarshallFailure),
were hit, we don’t know if this tests all types of product that we might get.
You can also see that a typical contract test (“hey server, I will send this request, do you send a response that I understand?“) would not involve covering some of the failure handling lines:
Copy code
if (error.request) {
      throw new ApiError("The server didn't respond", API_NO_RESPONSE);
    }
Also, in a real app, I usually would have specific errors for the HTTP codes I care about, which might be remarshalled to business-logic errors that don’t know anything about HTTP.
Again, none of the ways that I write my clients are specific to pact, or even necessary to use pact. I’m just using this particular client as an example to show some ways you could try to use coverage to get what you want
If you’re stuck with this and are able to share the code (even just the structure), I’d be happy to talk through some ideas together
🙏 1
a
Really appreciate you taking the time to think through an alternative, I think the entry layer approach makes it much easier to group together API requests that are being made throughout the codebase so we visually see if there is a test for it. With how our services are currently built, I think it would require some significant refactoring which I don't think would be feasible at this time (I might take your idea for some new projects though!) The end result of this as you note for automated coverage would be not as meaningful, since we're not covering errors or all variations of the interaction.
I think I wanted to see if the ask was possible at this time, but if it is asked again, I'd love to talk through it with you in the future if possible! Thank you again for all the thoughtful help!!
🙌 1
t
You’re welcome!
🌮 1
6 Views
Open in Slack
PreviousNext
Powered by