Having embarked upon looking at consumer-driven contract testing against a GraphQL server, I've come to the point where I need to step back and ask folks here about the appropriate strategies for this compared to testing REST contracts. With a REST provider, contracts tend to be based on quite specific endpoints with focused responses, and (tricks with Accept headers and query parameters aside) there isn't a whole lot of flexibility in what the consumer can expect out of e.g. the response to a GET. This allows provider tests (state change handlers) to be less consumer-focused (ideally completely decoupled from a specific consumer). With GraphQL, the entry points to the graph that a particular consumer app uses may be relatively small in number to choose from, but the number of different navigation paths from that starting query grow exponentially as the graph becomes larger and more connected. So what should such contracts be about? It seems to me that at its simplest the contract could be verifying the schema-correctness of the provider query; schema-correctness is relatively simple to test in most GraphQL servers, but what about the details of the response matching, and how far down the GraphQL server stack should be mocked in the state change handler? Is this an appropriate use of pact? With the chances that any two consumer app's queries being alike very small, it feels like a full-stack state change handler that needs to provide the right returned data shape is going to either become extremely messy to maintain, or will need to do a lot of introspection and mock relatively high up the stack, or there will be effectively one state change handler per consumer contract. If anyone has any opinions or experience on the benefits of different approaches to GraphQL consumer-driven contract testing, I'd be very interested - TIA.

As a start, I put my general thoughts here previously: https://pactflow.io/blog/contract-testing-a-graphql-api/

Hey @Alan Boshier, Yep, graphql pacts can be tricky! One of the things we did was to have interactions use a print of the GQL (parsed GraphQL) as the

withQuery

and formatted variables format GQL definition:

import { gql } from '@apollo/client'

export const GQL_QUALIFIED_NAME = gql`
  mutation mutationName($type: TokenType!, $token: String!) {
    mutationName(type: $type, token: $token) {
      transactionId
      id
      orderNumber
    }
  }
`

Consumer test file

// These must be static for Pact versioning
const exampleRequestToken= '8b06e062-32e3-41b3-b3ae-fbddb637eca0' // generator: generateFakeGuid(),
const graphQLResult = {
  transactionId: '2c8ac019dfbe430d087b1d23', // generator: generateFakeMongoDBId(),
  id: 'ac1a5c72-73e8-4d36-bd79-6c1fbfcfb27f', // generator: generateFakeGuid(),
  orderNumber: '320403409' // generator: generateFakeExternalOrderId()
}

const requestFormat= {
  token: Matchers.regex({ generate: exampleRequestToken, matcher: Matchers.UUID_V4_FORMAT }),
  type: REQUEST_TOKEN_TYPE
}
const resultFormat = {
  transactionId: Matchers.regex({ generate: graphQLResult.exampleResultId, matcher: '^[0-9a-f]{24}$' }),
  id: Matchers.regex({ generate: graphQLResult.exampleResultGUID, matcher: Matchers.UUID_V4_FORMAT }),
  orderNumber: Matchers.regex({ generate: graphQLResult.exampleOrderNumber, matcher: '^[0-9]{9}$' })
}

// Define Interaction on client
new ApolloGraphQLInteraction()
  .uponReceiving('internal test name')
  .given('pact scenario name must be identical in provider and consumer')
  .withOperation('mutationName')
  .withQuery(print(GQL_QUALIFIED_NAME))
  .withRequest({
    path: '/graphql',
    method: 'POST'
  })
  .withVariables(requestFormat)
  .willRespondWith({
    status: 200,
    headers: {
      'Content-Type': 'application/json; charset=utf-8'
    },
    body: {
      data: {
        mutationName: Matchers.like(resultFormat)
      }
    }
  })
)

// Execute interaction on client
await expect(
  client.mutate({
    mutation: GQL_QUALIFIED_NAME ,
    variables: {
      token: exampleRequestToken,
      type: REQUEST_TOKEN_TYPE
    }
  })
).resolves.toEqual({
  data: {
    mutationName: graphQLResult
  }
})

Thanks for sharing @Shaun Reid 🌮

The other tricky bit we had was getting Pact to understand how to host these queries. We ended up spinning up an Express server to handle the request from Pact on the Provider side our pact server:

note: we didn't resolve getting typedefs dynamically built at test time, so they are manually added here

import express from 'express'
import { Express } from 'express-serve-static-core'

import { ApolloServer, gql } from 'apollo-server-express'
import { Server } from 'http'

// TODO really needs to use scheama.gql, and not be manually updated
const typeDefs = gql`
  enum TokenType {
    TOKEN_TYPE_A
    TOKEN_TYPE_B
  }
`

export default class GraphQLTestServerFactory {
  apolloServer: ApolloServer
  expressServer: Express
  // eslint-disable-next-line @typescript-eslint/explicit-module-boundary-types
  constructor(resolvers) {
    // Provide resolver functions for your schema fields

    this.apolloServer = new ApolloServer({ typeDefs, resolvers })
    this.expressServer = express()
    this.apolloServer.applyMiddleware({ app: this.expressServer })
  }

  listen({ port }: { port: number }): Server {
    return this.expressServer.listen({ port })
  }
}

Then jest problems, but it provided enough flexibility to test multiple scenarios Provider Test file:

let testFactory: GraphQLTestServerFactory
const mutationNameMock = jest.fn()  

beforeAll(() => {
  jest.setTimeout(30000) // IO is slow sometimes

  const resolvers = {
    Mutation: {
      mutationName: (_, params) => mutationNameMock(params),
    },
  }

  testFactory = new GraphQLTestServerFactory(resolvers)
})

// Test
describe('Pact Verification', () => {
    it('validates charge Paydock with token', async done => {
      const opts: VerifierOptions = {
        stateHandlers: {
          'pact scenario name must be identical in provider and consumer': async () => {
            mocked(serviceA.internalFunctionA).mockResolvedValueOnce(valueA)
            mocked(serviceA.internalFunctionA).mockResolvedValueOnce(valueB)
            mocked(mutationNameMock).mockImplementationOnce(({ token, type }) =>
              mutationsClass.mutationNameMethod(type, token),
            )
            return Promise.resolve('this worked')
          },
          'a different pact scenario': async () => {
            mocked(serviceA.internalFunctionA).mockResolvedValueOnce(valueA)
            mocked(serviceA.internalFunctionA).mockResolvedValueOnce(valueC)
            mocked(mutationNameMock).mockImplementationOnce(({ token, type }) =>
              mutationsClass.mutationNameMethod(type, token),
            )
            return Promise.resolve('this worked')
          },
          'a failing pact scenario': async () => {
            mocked(serviceA.internalFunctionA).mockResolvedValueOnce(valueA)
            mocked(serviceA.internalFunctionA).mockResolvedValueOnce(undefined)
            mocked(mutationNameMock).mockImplementationOnce(({ token, type }) =>
              mutationsClass.mutationNameMethod(type, token),
            )
            return Promise.resolve('this worked')
          },
        },
        provider: 'provider-service-name',
        consumerVersionSelectors: [{ tag: 'master', latest: true }],
        pactBrokerUrl: process.env.PACT_BROKER_BASE_URL,
        enablePending: true,
        includeWipPactsSince: '2020-01-01',
        providerVersion: process.env.BUILDKITE_COMMIT,
        providerVersionTags: process.env.BUILDKITE_BRANCH,
        providerBaseUrl: '<http://localhost:4002>',
        publishVerificationResult: <http://process.env.CI|process.env.CI> === 'true',
        logLevel: 'info',
      }
      const output = await new Verifier(opts).verifyProvider()
      console.log(output)
      done()
    })
  })

Many thanks @Shaun Reid - that's very useful in helping to understand just how the provider verification might proceed. I guess my questions about GraphQL-based contract testing are as much about the what and why as the how. If you have a growing number of client apps (consumers) all with very specific interactions with a very rich connected graph, then it feels to me that the contracts become extremely specific to the consumer and possibly quite complex to verify; the entry points (supported queries and mutations) may be smaller in number and more analogous to a set of REST entry points - but its the flexibility in specifying the response shape that feels to me where the contract approach struggles. I wonder whether a simpler level of contract that is based on schema validity (i.e. is this GraphQL request schema-valid?) might strike the appropriate balance between affording protection to consumers and not introducing an unbounded set of highly specific contracts to be maintained in the provider.

@Alan Boshier I guess my advice here has some facets. • Making a service too generalized in possible use cases may result in something that by nature is hard to develop edge cases for • Consumers should be able to validate their scenario is supported by the provider ◦ Consumer driven contract tests give assurances to the consumer that what they expect to happen, will happen, by the design of contract testing ◦ Consumers should be providing these Pacts to you for their scenario's, especially if they're quite complex • Graphql has inbuilt schema validation • Graphql queries define result shapes that could be a subset of data. ◦ I think this is a limitation around Pact's query and response definitions that they can't be exhaustively tested for each possible combination ◦ Contracts allow expectations of valid combinations to be set, and new Consumer Contracts can be easily added for new combinations, if the Consumer wanted a guarantee of the interface compatibility. It sounds like contract testing is exactly what you need. If their is a concern that a single provider is possibly doing too much work, it might be time to refactor into smaller responsibility interfaces.

The main point of graphql, as I understand, is that consumers choose what data to return. But the Provider still must be able to return the full dataset. So whilst each consumer may have wildly different shaped queries, it shouldn’t actually change the provider implementation. What you do get however, is complete visibility into the types of queries you need to actually support. Where it might get tricky is dealing with all of the various provider states. There are strategies here, but they are no different to regular HTTP Pact.

Thanks both for your thoughts on this - @Matt (pactflow.io / pact-js / pact-go) when you say "the Provider must still be able to return the full dataset", what would that practically entail in a large highly connected graph - the nature of graphql servers tending to be mostly gateways to their own set of external "providers", this seems to imply mocking all of those backend services in order to ensure that consumer contracts are covered; it would be very complex to introspect the query issued by a particular contract and figure out dynamically what mocks are required, and I feel that writing a dedicated state handler for every contract seems excessive in terms of coupling and brittleness. I'm wondering what all the effort of such mocking actually achieves; in GraphQL the schema is the language of the contract - if my query is schema-valid, what extra assurances does a consumer gain from the provider filling in the response, especially if most of the provider stack exercised by doing so tends to be boilerplate server framework code? The implied contract of a GraphQL server is "if your query is schema-valid, I will respond with data in the shape you ask for". Anyway thank you for taking the time to respond to my existential angst :-) I think my current feeling is that contract testing as a pathway to checking schema validity of the consumer's request seems the most useful approach, but detailed response checking isn't worth the investment of effort.