Sweet, I managed to return a 403 when it is an htmx request with

HX-Refresh: true

and now everything works