Hot on the heels of hivemind devops alert: nginx sucks at ssl
, I issue another hivemind devops alert: nginx does not
suck at ssl!aftermath
After circulating my previous alert through the 'tubes, it became abundantly clear: nobody has any idea how SSL performance works. A few people suggested I was running out of entropy (nice guess, but wrong), many people mentioned ssl session caching (nice try, but not relevant when testing all new connections), and a few people chimed in about keepalive (nice try, but then results get skewed depending on how many assets each client requests). Everybody seemed to care about the absolute numbers and not the relative performance differences.
A few people went with a very tactful "dude, that's just wrong. I know it works" response which is perfectly valid and appreciated. I knew something was wrong, but couldn't put my finger on it.
chimed in, and it clicked. I wasn't verifying equal cipher negotiation against all servers and the benchmark utility. Even so, why would nginx pick a more computationally intensive cipher than stunnel or stud? Let's find out.the what
The problem is that annoying line in your nginx config you copied from somewhere else and you're not entirely sure what it does. It's a lengthy security incantation. Maybe you picked a PCI compliant list? Maybe you filtered it yourself? It looks something like [a]:
ssl_ciphers ALL:!aNULL:!ADH:!eNULL:!MEDIUM:!LOW:!EXP:RC4 RSA: HIGH;
So what does it mean? The line above gives you a nice and secure encryption setup for nginx. Unfortunately, it also includes a very computationally intensive cipher using an ephemeral Diffie-Hellman exchange for PFS
. Sounds scary already, doesn't it?the how
The problem cipher is DHE-RSA-AES256-SHA [b]. It uses DH on each new connection to ensure PFS (DHE = "Diffie-Hellman Ephemeral"). The DH portion requires extra code in software using OpenSSL to enable DHE negotiation. nginx has the extra code built in. stud doesn't have at all. stunnel has it as a compile time/certificate configurable option.
nginx configures OpenSSL so completely, it enables the very-secure-yet-very-slow cipher by default! stunnel does not
enable that cipher by default because it doesn't configure DH itself (you can configure it by hand though). stud does not enable DH at all.
The next-most-secure cipher is AES256-SHA, which is what stunnel and stud were using to out-perform nginx on my thundering herd connection tests.the fix
You can force nginx to not
enable the expensive cipher by excluding all DHE ciphers. Add "!kEDH" to your cipher list. It disables (the ! disables) any cipher using Ephemeral Diffie-Hellman.the new nginx benchmarks
Now that we found the problem, let's look at nginx SSL using a sane performance cipher.
nginx (AES256-SHA) -> haproxy: 1300 requests per second
nginx (AES256-SHA with keepalive 5 5;) -> haproxy: 4300 requests per second
There is a slight speed boost from also disabling iptables. As always, do not
trust these numbers. Performance depends on: your firewall config, your ciphers, your backend latency, keepalive, session caching, and how many faeries currently live in your system fans [c].final answer
To get more performance out of nginx SSL, remove its ability to negotiate slow encryption ciphers. Add "!kEDH" to your list of allowed ciphers (unless you are passing around government secrets about aliens or are an internationally wanted arms dealer). Do it now.
Curious about what cipher your install is negotiating? Test it with a quick:
openssl s_client -host HOSTNAME -port 443
Look at the Cipher: line. If it says DHE-RSA-AES256-SHA, your site could be going much faster over SSL by disabling DHE.
: If you want to understand the format (! versus versus -), read http://www.openssl.org/docs/apps/ciphers.html
: "problem" from a website speed point of view.[c]
: Disable iptables on all web-facing boxes if you want to maximize performance. Use SSL session caching and (sane) keepalive values, but those settings aren't panaceas. Everything depends on your page composition, user habits, and individual system issues.bonus insight about SSL benchmarks
Don't trust anybody's SSL benchmarks unless they include at a minimum
details about: Is the OS firewall enabled? Is the benchmark being run over localhost on the same machine? Which cipher is being negotiated between the benchmark tool and the SSL server? Is keep alive on? Is the benchmark tool using keep alive? Is session resumption on? Is the benchmark tool using session resumption? Which benchmark program is being used (they all have different inherent performance problems)?
(note: I didn't mention any of those things. Do not trust my numbers. Benchmark your own systems.)
Over-reliance on becnhmarking keepalive and session resumption can yield false results unless you ever only have one client to your website and they use one keepalive connection and one SSL session constantly.
If you care about absolute numbers, require details about: How many cores? How fast? Do you have an OpenSSL-recognized hardware accelerator engine being used? What else is running on the box? What's the latency among all components?bonus insight about social diarrhea
I launched my original post over the HN fence and to my twitter account at the same time. It quickly fell off the new page of HN. It immediately started getting re-twatted on the twitters.
Every @reply I received from twitter was supportive, helpful, understanding, or very politely confused/questioning.
Later in the day, to stop me from whining, a friend re-submitted my post to HN. This time the article shot to the #1 spot. Uh oh. If I hadn't developed such think Internet Defense skin over the years, I would have been terribly offended by half of the HN comments.
Remember: The Internet is a big place. If you get upset when somebody isn't as perfect as you are, you'll spend your life being miserable. Be nice. Be understanding.
Final feeling: Twitter is better than HN in all social dimensions of engagement, kindness, and authenticity.