Challenge would be a waste, it does change with the initial target. But if u become the new target and are out of LoS, it will change to the closest Oathsworn in LoS and then to the closest regardless of LoS.