Well, that sounds a bit odd. But the only thing that I could figure is the VPN firing up. If you have a VPN, proxy, firewall, and/or web filter between you and the ES server, then that could be the point of slowness. So it sounds like when the first connection being established to the ES server is happening you are having the slowness. So the first place I would look is the aformentioned items. Next, or potentially first, whichever you feel may be the culprit, I would look at the ES server. If the web service hasn't been contacted in a while, then the first time may be a little slower (10 seconds sounds like a lot for this though).
You might create a quick test app that does nothing but attempt to perform a query from the ES server and then you could get a better idea of where the issue may be.
Last, you could load a profiler to see where the timings are slow. Without actually having my hands on the client and the server it is hard for me to give a straight answer, so I am just throwing out some ideas of things to check.