If you configure the LDAP server IP or the DNS, the traffic likely originates from the NSIP (management IP) and if you run the connetion via LB, then the traffic originates from SNIP (subnet IP).
I would capture network trace of both of the cases and see what happens. I'd suspect you might have a routing / firewall -related issue?