Without getting into the details of the server, most WP sites show this inconsistent behavior:
I'm logged in so fast_cgi doesn't cache and I see something like:
This page loaded in 24.369 seconds with 40 database queries.
Then I refresh and get:
This page loaded in 0.307 seconds with 40 database queries.
The point is its random. Sometimes I get really high times consecutively, sometimes consecutively low load times, what could be the cause?
MYSQL query cache is huge and has plenty of available space as does the APC cache (in case it matters).