Right now it is impossible to allow traffic from certain IP's while restricting others. One approach could be to have a firewall in front of the load balancer.