NGINX using 100% of the server CPU when testing with Blitz.io


#1

I’m facing an issue here and after four days of looking around I decided to ask for some help here, after all a million heads can think a lot better than one.

I have a Ubuntu 14.04 server setup with NGINX, HHVM, PHP5-FPM (as a backup), Percona MySQL, Memcached (which will be replaced by Redis). I have fastcgi_cache setup for WordPress and object caching done over memcached. All cool and dandy in theory, but not in practice.

This is a RamNode OpenVZ SSD VPS with 2GB of RAM and an Intel Xeon E5 with two cores for my VPS.

Running Blitz.io on it the server is getting absolutely murdered by the two NGINX worker processes, which each one using 100% CPU according to top and htop. I usually run with the following pattern:

--pattern 999-1000:60 https://www.geeksune.com/blog/hello-world/

That makes makes CPU go to the roof and according to Blitz.io this is the result of that:

135 HITS WITH 57,734 ERRORS & 234 TIMEOUTS

Obviously that isn’t good. RAM usage stay under 250MB all the time and it seems that all those requests from Blitz.io are hitting the cache, as seen here:

54.232.204.19 - HIT [23/Nov/2014:19:06:32 -0200] “GET / HTTP/1.1” 200 7632 “-” “blitz.io; 20bed746e60d4e53401c0c04d245c4eb@189.101.124.196

Notice the HIT at the start. I set a new log format and added $upstream_cache_status to it.

A similar setup on the same machine works just fine with Blitz.io, so there is definitely something wrong with my NGINX setup and it seems related to fastcgi_cache. I have the same results every time, even with just PHP5-FPM with Zend.

Does anyone have a clue about what is happening? My configuration files look like this:

Thanks in advance.

:slight_smile:


#2

Have you tweaked sysctl?

Link: https://rtcamp.com/tutorials/linux/sysctl-conf/

Also, if possible try to run same blitz.io test on exactly same setup on DigitalOcean or Linode $20 plan. (both charge you by hour so you won’t end up paying for complete month)


#3

Thank you for the suggestions :slight_smile:

Haven’t tweaked it yet, not sure if it’s possible on OpenVZ but I will take a look (EDIT: permission denied when running sysctl -p as root) . As for Linode they aren’t accepting my credit card for some reason, which is why I went with RamNode.

I’m sure there is something wrong with my NGINX setup because a similar setup run without any issues on the same server, and RamNode is known for being crazy fast (faster than Linode on some benchmarks).


#4

Replicated this setup on another server, this time with a quad-core CPU and using KVM. Same issue, all cores end up using 100% of the server CPU.


#5

Tested it with HTTP instead of HTTPS and while it did reach 100% of CPU usage from time to time, the average load was 50% and the Blitz.io test was a lot better:

  • 33,086 HITS WITH 29 ERRORS & 2,344 TIMEOUTS

Why would SSL perform so badly? I generated the key with 4096 instead of 2048, but I don’t believe that would affect CPU usage this much.


#6

Sorry. I missed SSL part.

Try removing lines 10-14 from http://paste.ubuntu.com/9236298/

Also try our SSL config https://rtcamp.com/tutorials/nginx/ssl-pci-compliance-performance/

If possible please share results of above 2 tests. Thanks.


#7

@rahul286 Turns out it was the key in the end, as 4096 was using too many CPU cycles. I will keep trying to optimize the settings as the CPU usage is still too high for my liking, but the actual problem here was my key.


#8

Thanks for details. :smile:

I think blitz like tool may not make use of SSL session cache very well. Difference between SSL v/s non-SSL could be because of SSL session cache.


#9

@rahul286 I do believe that is true. With the new server everything is running smooth with my new settings, but lets see once real traffic gets in how things will go.

Blitz.io hit the server with only one IP address, might as well be an issue as well. Do you have other benchmarks tools to recommend?


#10

@julianfernandes I really don’t get into benchmarks as they are hard to trust. I try to speed up a single request in debug mode and go after each error/warning/notice I see.

This is our checklist - https://rtcamp.com/tutorials/wordpress/performance-optimization/

Still blitz.io and other has chrome extensions to emulate transaction processing. You may try them if SSL is used for some e-commerce section.


#11

@rahul286 Gonna make sure I read that, thanks for all the help =)


#12