The intent with Pact (as contract by example, and inline verification) is to prove that things work by exercising real implementations.
You write unit-style tests on the consumer side (to make sure it works as you expect), and verify those realistic recorded interactions against the real provider code (to make sure the provider also works as you expect).
You can do that via a separate repo (just as you can program whatever you want, given infinite time & resources), but you lose significant reasoning power.
To put it simply: if you're not testing your actual provider code, how do you know it's working?