Always ask does my client care about this type of request I am encoding into a pact file, in your test.
If your consumer code makes no distinguish between a 4xx/5xx or just bails on a non 2xx you would probably only want to test for a generic error.
If you care a specific response code in your client code, then you can encode it.
Otherwise you can just test that your api client doesn’t explode for consumers when it does get a 500, in your regular unit tests.
It would maybe be nice if you could choose to exclude pacts from being serialised but still run selective tests, so you could do that functional testing of your consumer using pact as your mock, but without serialising all those cases into the pact and functionally testing the provider