//
you're reading...
Camping IO, Ruby Scripts

Benchmarked

Just finished doing a bunch of benchmark tests for various ruby http servers on JRuby and MRI.

Posted the code on my github page.

The end goal wasn’t to find the fastest server per say, but to see how the various JRuby servers act and compare them against MRI versions.

Mainly I wanted to see how Reel is, I’m a fan of Erlang’s actor model and Celluloid and Reel-Rack are pretty interesting to me.

While it wasn’t the fastest at serving simple content, it felt very robust to test with, it threw the least amount errors, and recovered when crashed.

Here’s some of the results I got:
(The way to read the graphs is look at the number of requests done at a given time interval. A high number of requests done at a low time interval is a very fast server, but don’t be mislead, there’s a lot of factors at play and there’s also a cutoff on the time scale to keep the graph readable)


JRuby Simple Hello

jruby.hello

Thick is very fast at simple content, so it blasted the other servers in terms of speed.

Reel on Webmachine did well also.


MRI Simple Hello

mri.hello

By comparison, MRI is pretty fast, Webmachine here is screaming fast. much more than Thin is, which was surprising, but not shocking, since Webmachine is more raw HTTP engine.


JRuby Simple DB Queries

jruby.db

Here Thick and Camping win out.


MRI Simple DB Queries

mri.db

Reel and Webmachine pop up here also, while it took more time to serve on MRI than JRuby, it’s completing nearly 3x to 4x the amount of requests. Might have to do some head to head tests with Reel and Webmachine on MRI vs JRuby.


JRuby 20 number Fibonacci

jruby.fibonacci20

On this one Thick server wins out, again these numbers can be really misleading, the graphs don’t show the full time scale or give you clear picture of the total request times, but just to show that looking at the spikes aren’t super off target, here’s the Apache benchmark output for Thick server:


This is ApacheBench, Version 2.3
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.0.200 (be patient)

Server Software:
Server Hostname: 192.168.0.200
Server Port: 8000

Document Path: /
Document Length: 169 bytes

Concurrency Level: 100
Time taken for tests: 18.023 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 2340000 bytes
HTML transferred: 1690000 bytes
Requests per second: 554.85 [#/sec] (mean)
Time per request: 180.229 [ms] (mean)
Time per request: 1.802 [ms] (mean, across all concurrent requests)
Transfer rate: 126.79 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 1.9 0 34
Processing: 1 179 286.5 53 1898
Waiting: 1 178 286.3 52 1898
Total: 1 179 286.6 54 1898

Percentage of the requests served within a certain time (ms)
50% 54
66% 114
75% 192
80% 269
90% 562
95% 835
98% 1132
99% 1330
100% 1898 (longest request)


Compare that to the next spike on the graph, reel-rack camping:


This is ApacheBench, Version 2.3
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.0.200 (be patient)

Server Software:
Server Hostname: 192.168.0.200
Server Port: 8000

Document Path: /
Document Length: 0 bytes

Concurrency Level: 100
Time taken for tests: 36.440 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1120000 bytes
HTML transferred: 0 bytes
Requests per second: 274.42 [#/sec] (mean)
Time per request: 364.398 [ms] (mean)
Time per request: 3.644 [ms] (mean, across all concurrent requests)
Transfer rate: 30.02 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 79 535.1 0 15000
Processing: 9 276 102.3 242 1019
Waiting: 9 276 102.2 242 1019
Total: 14 355 547.1 244 15246

Percentage of the requests served within a certain time (ms)
50% 244
66% 282
75% 320
80% 340
90% 446
95% 871
98% 1343
99% 3206
100% 15246 (longest request)

And you can indeed see Thick is doing well. To read the percentage values listed, it’s saying (for Thick for instance) that within 54ms 50% of the requests have been completed, whereas for reel-rack, by 244ms 50% of the requests have been completed.

You can see this on the graphs, the huge spike for Thick is below 50ms, and the one for reel-rack is closer to 210ms.


MRI 20 number Fibonacci

mri.fibonacci20

Once again Reel Webmachine blasts through.


So what does all this mean? Really, nothing, I mean it gives me some idea how the servers act on my hardware, more info on how hard they are to work with or how stable they are, but I wouldn’t base a production choice on the data above.

Though I would probably narrow my focus on what I would test on production hardware.

Since my end goal was around JRuby, it’s nice to see how the various servers act, and some gave me more trouble than others, puma for instance gave me a few issues with the config.ru files, some others weren’t very graceful on shutdown.

I will probably try to come up with a much more complex test to really see how Reel does, the actor model can make concurrent tasks very robust.

Hopefully someone finds some of the info above useful for next time they need to look for a web server for their Ruby app.

Realistically, I would offload the static content handling to something like Nginx, though this could be done with some servers like Thin.

Advertisements

Discussion

No comments yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s