Ok, that should not be the problem.
Maybe network/connectivity problems?
Did you check ping times, DNS resolution times, etc?
Also check if your delays are constant or variable. Constant delays are typically programmed timeouts (such as your primary DNS timing out, and then a request goes to the secondary).