I've had exact same problem. I had a certificate for storefront load balancing vip with 4098 key size. When I first used it, got the "Certificate with key size greater than RSA512 or DSA512 bits not supported" error. I've generated 2048 size key, but still the same.
As per
@Jan Tytgat suggestions on previous posts, I've installed the latest firmware first, then used 29.65 firmware file and upgraded the instance. 2048 key size cert file was imported successfully.
Check this link to my post in this channel:
https://worldofeuc.slack.com/archives/CKHRXATV2/p1725385086387549