https://pact.io logo
Join SlackCommunities
Powered by
Question about debugging a provider verification i...
# pactflow
h
Question about debugging a provider verification in the past. We've been evaluating pact and pactflow for the past month (noob here, 👋🏽), and yesterday we inadvertently made a breaking change in a provider service that went undetected since the provider verification oddly passed. So my question, is there a way to replicate a verification for a provider commit based on the same moment in time the test ran? I'm at a loss to how the provider verification passed. In order to quickly fix, we fixed the consumer to meet the new specifications of the provider api. Next, I wanted to debug more and understand what happened. So then I essentially reverted the initial breaking change on a provider branch to try and replicate breaking the api again and this time it failed as I would have expected.
Copy code
_, err := pact.VerifyProvider(t, types.VerifyRequest{
		ProviderBaseURL: fmt.Sprintf("<http://127.0.0.1>:%d", port),
		PactLogDir:      brokerConfig.LogsDir(),
		ConsumerVersionSelectors: []types.ConsumerVersionSelector{
			types.ConsumerVersionSelector{
				MainBranch: true,
			},
			types.ConsumerVersionSelector{
				MatchingBranch: true,
			},
			types.ConsumerVersionSelector{
				DeployedOrReleased: true,
			},
		},
		BrokerURL:                  brokerConfig.BrokerURL,
		BrokerToken:                brokerConfig.BrokerAccessToken,
		EnablePending:              true,
		IncludeWIPPactsSince:       &includeWipPactsSince,
		PublishVerificationResults: allowPublishVerificationResults(),
		ProviderBranch:             gitInfo.Branch,
		ProviderTags:               []string{gitInfo.Branch},
		ProviderVersion:            gitInfo.Version,
		StateHandlers:              stateHandlers,
	})
I did look into see if the consumer had a matching branch, and it did not.
b
Good question.
Can you identify the build where it passed and shouldn’t have?
There should be output in the test logs that tells you exactly which versions of the consumer are being verified.
Copy code
DEBUG: The pact at <https://test.pactflow.io/pacts/provider/example-provider/consumer/example-consumer/pact-version/011de3a974a121cbf474b931e4de355393b52220> is being verified because the pact content belongs to the consumer versions matching the following criteria:

    * latest version from branch 'master' (acda70158a6d338e677a99a19d3fbc9c8371ae90)

    * consumer version(s) currently deployed to dev (acda70158a6d338e677a99a19d3fbc9c8371ae90) and prod (acda70158a6d338e677a99a19d3fbc9c8371ae90)

    * latest version from the main branch 'master' (acda70158a6d338e677a99a19d3fbc9c8371ae90)

DEBUG: This pact has previously been successfully verified by example-provider. If this verification fails, it will fail the build. Read more at <https://docs.pact.io/go/pending>
h
Thanks for the follow-up @Beth (pactflow.io/Pact Broker/pact-ruby). I did a bit more digging into the CI logs and I'm not able to find the message example from above. However, in the case where I revert the provider code and rerun, then I do see example message from above when it fails as expected. We're using pact-go module for running our verify tests so not sure if that somehow doesn't emit the ruby logs in non-failure cases. Interestingly, we are running can-i-deploy on our PR branches per another slack thread 😂 . And this does show our Provider PR Commit has a verification against the Consumer main branch commit. So presumably this the suspect verification I'm trying to debug. Ultimately the breakage was on a matcher in body response verification. The field is an enum and I was expecting a regex of a list of values that actually changed. I also looked into overriding the consumer version in ConsumerVersionSelector, but looks like setting the version is ignored now.
b
Yeah, you can’t set a consumer version in the selector. But you can provide the URL of a specific pact to be verified (as is done for webhook triggered verifications).
What version of pact go are you using?
We need to make sure that the output I mentioned is being shown.
h
<http://github.com/pact-foundation/pact-go|github.com/pact-foundation/pact-go> v1.7.0
b
ta
m
Thanks Hazeem. I believe because we use the 
t.Logf
function here, they are only printed on test failure or if you pass the 
-v
flag to the tests. I could force this to be printed to stdout, but is not a very idiomatic. I’ll have a think about the best way to tackle this
👍 1
h
Thanks for the tip/reminder about debugging with PactURLs. In case others search for regex matching with enums, my mysterious pact verification was the result of a bad regex: 1. Consumer defined matching a json response field for expected enums of: 
(OBJECT|STRING|INTEGER|FLOAT|BOOLEAN|TIME)
2. Provider verified the pact. 3. Next provider broke the api by changing the enums and added a suffix of 
_RESULT
and this unintentionally passed the not so great enum matcher, and allowed verification of the pact for the broken api. Outcome: The consumer's match regex should have been: 
^(OBJECT|STRING|INTEGER|FLOAT|BOOLEAN|TIME)$
b
Ah! Yes, I’ve seen that before.
h
Yeah, I wont forget this one. 😂
😆 1
b
@Hazem Borham how would you feel about adding that into the docs? https://docs.pact.io/getting_started/matching/gotchas
👍🏽 1
h
8 Views
Open in Slack
PreviousNext
Powered by