Indeed I was rumbling. I've narrow down the issue, it should be easier to articulate.
If I access ADFS via the load balancer (internal VIP), the cert authentication as a second factor on port 49443 end up with the too many redirects. My LB is set with SSL_Bridge, same issue with SSL. Very basic conf. Without the cert as second factor, this conf works, SSL Bridge or not.
If I bypass my LB to point straight to the ADFS VM, it works so ADFS conf is good, LB conf is not. Also the external VIP (WAP) is not used in this flow, I shut it down to narrow down the issue and I could still authenticate without the cert as a second factor.
I have captured the traffic when issue happens, nothing obvious except than the form submitted by client end up with a redirect from the ADFS, client submit form, redirect, etc.
In ADFS log I can see eventID 1000 "An error occured during processing of a token request" for my test user.
Hopefully I make more sense. thanks for the help.