Is there an out-of-the-box way to create a promotion that can only be applied via the Admin UI? If not, how conceptually hard would it be to create such a thing? (Context - we want our admins to be able to offer free shipping easily using a promotion, but we don't want users to be able to guess it themselves)

You could just use one off codes that random and generate them only when you need them.

Correct me if I'm wrong, but then you have to manually create the promotion each time, right? So that would make it hard to answer the question "how many orders were given free shipping", right?

No, I'm saying to use multiple codes on one promotion.

Still not totally sure I follow - would you basically have to go create a new batch of codes (ie, 1 code) each time you want to use the promotion?

That's why I'm suggesting you automate it, since doing that manually would be annoying. There's nothing built in that I know of to let you do what you're proposing. I'm just suggesting something that would be pretty easy to add to solve this use case.

Gotcha - and very much appreciated! I'm trying to wrap my brain around a reasonable way to "automate" it, though. I guess maybe we can add an action to that screen to generate a promo code or some such.

Yeah, that's what I'm thinking. A button that's like "add free shipping promo" that generates a new code and applies it to the order.