What you described is the best approach when working with a test data set, this is best practice for it. I do the same, one config per environment and write tests to point to config variables so the same test can pass on different data sets.
We don't have the ability where I currently work to seed all DB data, otherwise we'd ideally seed test data at the start of tests, and unseed after the test finishes to remove it again.