I know a lot of stores that sell a mix of digital and physical goods just ask for shipping anyway, I'm not sure how much work it would be to allow skipping that.

Yeah, at least I can tell you about Steam where it asks for "billing" address every time I use my card even though I buy digital goods there.

That's excessive though. Many stores have no actual need to capture billing addresses (beyond the zipcode for authorizing the card) and don't actually need to, if they want to reduce checkout friction.