are you affected if you had an IDP configured at one time but removed that aaa server?

No, the attack is targeting the SAML IdP Profile. When there's no AAA with a binding of that profile, you're fine, because that URL isn't accessible, anymore