Prompt optimization is a recurring idea. On practice costs tend to be too high (both on developing an evaluation harness and on actual runs).
I'm using that successfully to create classifying prompts. Costs make sense there since it's single LLM request, so I can run the prompt on 1000 examples for under 1 usd. Eval harness is also super simple.
Doing that on a benchmark, for instance, would be 2 orders of magnitude more effort.
Notice that is for a priori optimization. If the skill is run 100s of times a day, you can improve it a posteriori without incurring higher costs but you do have to deal with sporadic regressions.