SimonHF's Blog

Just another WordPress.com site

G-WAN versus SXE “Hello World” April 26, 2012

Recently I’ve been very impressed reading about the performance figures for G-WAN:
http://gwan.ch/benchmark

G-WAN has quite the licensing model with the G-WAN binary being freeware and support costing very much money:
http://gwan.ch/buy

So I decided to do a simple libsxe versus G-WAN performance test like I did for libsxe versus NGINX and libsxe versus node.js. However, for this test I decided to use G-WAN’s very own multi-threaded load tool called weighttp:
http://redmine.lighttpd.net/projects/weighttp/wiki

I modified the simple libsxe HTTP server to make it take advantage of multiple CPUs.

These tests were run on a Ubuntu 11.04 instance running on a dual quad core i7 processor.

First the G-WAN figures:

I don’t know why G-WAN is talking about 16 cores upon starting because my i7 only has 8!

simon@ubuntu:~/gwan_linux64-bit$ sudo ./gwan

allowed Cores: 8 (‘sudo ./gwan’ to let G-WAN use your 16 Core(s))

loading
> ‘all.java’: to use Java (*.java) scripts, install ‘javac’ (sudo apt-get install javac)
> ‘hello.mm’: to use Objective-C++ (*.mm) scripts, install ‘gobjc++’ (sudo apt-get install gobjc++)
> ‘loan.java’: to use Java (*.java) scripts, install ‘javac’ (sudo apt-get install javac)..
> ‘argv.java’: to use Java (*.java) scripts, install ‘javac’ (sudo apt-get install javac).
> ‘hello.java’: to use Java (*.java) scripts, install ‘javac’ (sudo apt-get install javac).
> ‘hello.m’: to use Objective-C (*.m) scripts, install ‘gobjc’ (sudo apt-get install gobjc)
> ‘report.java’: to use Java (*.java) scripts, install ‘javac’ (sudo apt-get install javac)..

G-WAN 3.3.28 (pid:3110)

simon@ubuntu:~/weighttp$ ./build/default/weighttp -n 10000000 -c 1000 -t 4 -k “http://127.0.0.1:8080/100.html”
weighttp – a lightweight and simple webserver benchmarking tool

host: ‘127.0.0.1’, port: 8080
starting benchmark…
spawning thread #1: 250 concurrent requests, 2500000 total requests
spawning thread #2: 250 concurrent requests, 2500000 total requests
spawning thread #3: 250 concurrent requests, 2500000 total requests
spawning thread #4: 250 concurrent requests, 2500000 total requests
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 61 sec, 501 millisec and 457 microsec, 162597 req/s, 59862 kbyte/s
requests: 10000000 total, 10000000 started, 10000000 done, 10000000 succeeded, 0 failed, 0 errored
status codes: 10000000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 3770000000 bytes total, 2770000000 bytes http, 1000000000 bytes data

Now the libsxe figures:

simon@ubuntu:~/sxe-httpd/sxe-httpd$ ./build-linux-64-release/sxe-httpd 127.0.0.1 8080 10000
20120426 211759.525 T 10198 —— 1 – sxe-httpd starting // detected cpus: 8
20120426 211759.525 T 10198 —— 1 – sxe-httpd parent forking 7 times
20120426 211759.525 T 10199 —— 1 – sxe-httpd child created
20120426 211759.525 T 10200 —— 1 – sxe-httpd child created
20120426 211759.525 T 10201 —— 1 – sxe-httpd child created
20120426 211759.526 T 10202 —— 1 – sxe-httpd child created
20120426 211759.526 T 10203 —— 1 – sxe-httpd child created
20120426 211759.526 T 10204 —— 1 – sxe-httpd child created
20120426 211759.526 T 10205 —— 1 – sxe-httpd child created

simon@ubuntu:~/weighttp$ ./build/default/weighttp -n 10000000 -c 1000 -t 4 -k “http://127.0.0.1:8080/100.html”
weighttp – a lightweight and simple webserver benchmarking tool

host: ‘127.0.0.1’, port: 8080
starting benchmark…
spawning thread #1: 250 concurrent requests, 2500000 total requests
spawning thread #2: 250 concurrent requests, 2500000 total requests
spawning thread #3: 250 concurrent requests, 2500000 total requests
spawning thread #4: 250 concurrent requests, 2500000 total requests
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 34 sec, 79 millisec and 878 microsec, 293428 req/s, 108316 kbyte/s
requests: 10000000 total, 10000000 started, 10000000 done, 10000000 succeeded, 0 failed, 0 errored
status codes: 10000000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 3780000000 bytes total, 2780000000 bytes http, 1000000000 bytes data

Conclusion:

At 162597 versus 293428 requests per second, libsxe is significantly — or 1.8 times — faster than G-WAN for this simple performance test using 8 cores. Although G-WAN calls itself the fastest web server available — and admittedly is very fast — it obviously suffers internally from quite a bit of overhead even for such a trivial performance test such as this one. And with libsxe the CPU bottleneck is really the networking layer in the kernel… so what is G-WAN doing with all those spare CPU cycles? Looks like G-WAN might have room for optimization yet? Or maybe it’s partly due to libsxe’s fixed memory model which does away with the unnecessary and repetitive malloc() / free() cycle? I guess we’ll never know since G-WAN is closed source.

EDIT: Since running this test we have found two potential problems with G-WAN which mean that these figures are unreliable (see thread below): (a) G-WAN’s performance seems highly tuned to particular processors but it’s supplied as a single binary executable meaning that performance tests may vary wildly, and (b) G-WAN doesn’t scale linearly as the number of cores increase even with the simplest of performance tests.

Advertisements
 

Nginx versus SXE “Hello World” October 2, 2010

After my last post then a colleague offered the criticism that comparing C to a script language is a bit like shooting fish in a barrel 🙂 I think the colleague missed the point which is that often the main reason for choosing to use a scripting language in the first place is to achieve rapid application development at the expense of run-time performance and memory usage. The purpose of the post was to try to dispel this myth and show how few lines of C source code can be necessary to achieve ultimate performance. However, in order to keep the colleague happy, here is a similar head to head between nginx and SXE. What is nginx? Here’s what Wikipedia says about nginx: “Nginx quickly delivers static content with efficient use of system resources.” Now on with the “Hello World” comparison…

Here is the nginx.conf:

# cat /etc/nginx/nginx.conf
worker_processes  1;
events {
    worker_connections  10240;
}
http {
    server {
        listen 8000;
        access_log off;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
    }
}

 And here is the index.html file:

# cat /usr/html/index.html
Hello World

I use the same http.c from the previous post in order to load test nginx. Here are the results:

# ./http -i 127.0.0.1 -p 8000 -n 50 -c 10000
20101002 181142.250 P00006a5f ------ 1 - connecting via ramp 10000 sockets to peer 127.0.0.1:8000
20101002 181142.290 P00006a5f    999 1 - connected: 1000
20101002 181142.328 P00006a5f   1999 1 - connected: 2000
20101002 181142.367 P00006a5f   2999 1 - connected: 3000
20101002 181142.406 P00006a5f   3999 1 - connected: 4000
20101002 181142.445 P00006a5f   4999 1 - connected: 5000
20101002 181142.484 P00006a5f   5999 1 - connected: 6000
20101002 181142.523 P00006a5f   6999 1 - connected: 7000
20101002 181142.562 P00006a5f   7999 1 - connected: 8000
20101002 181142.602 P00006a5f   8999 1 - connected: 9000
20101002 181142.641 P00006a5f   9999 1 - connected: 10000
20101002 181142.641 P00006a5f ------ 1 - starting writes: 500000 (= 10000 sockets * 50 queries/socket) queries
20101002 181142.641 P00006a5f ------ 1 - using query of 199 bytes:
20101002 181142.641 P00006a5f ------ 1 - 080552a0 47 45 54 20 2f 31 32 33 34 35 36 37 38 39 2f 31 GET /123456789/1
20101002 181142.641 P00006a5f ------ 1 - 080552b0 32 33 34 35 36 37 38 39 2f 31 32 33 34 35 36 37 23456789/1234567
20101002 181142.641 P00006a5f ------ 1 - 080552c0 38 39 2f 31 32 33 34 35 36 37 38 39 2f 31 32 33 89/123456789/123
20101002 181142.641 P00006a5f ------ 1 - 080552d0 34 35 36 37 38 39 2f 31 32 33 34 35 36 37 38 39 456789/123456789
20101002 181142.641 P00006a5f ------ 1 - 080552e0 2f 31 32 33 34 35 36 37 38 39 2f 31 32 33 34 35 /123456789/12345
20101002 181142.641 P00006a5f ------ 1 - 080552f0 36 37 2e 68 74 6d 20 48 54 54 50 2f 31 2e 31 0d 67.htm HTTP/1.1.
20101002 181142.641 P00006a5f ------ 1 - 08055300 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 4b 65 65 .Connection: Kee
20101002 181142.641 P00006a5f ------ 1 - 08055310 70 2d 41 6c 69 76 65 0d 0a 48 6f 73 74 3a 20 31 p-Alive..Host: 1
20101002 181142.641 P00006a5f ------ 1 - 08055320 32 37 2e 30 2e 30 2e 31 3a 38 30 30 30 0d 0a 55 27.0.0.1:8000..U
20101002 181142.641 P00006a5f ------ 1 - 08055330 73 65 72 2d 41 67 65 6e 74 3a 20 53 58 45 2d 68 ser-Agent: SXE-h
20101002 181142.641 P00006a5f ------ 1 - 08055340 74 74 70 2d 6c 6f 61 64 2d 6b 65 65 70 61 6c 69 ttp-load-keepali
20101002 181142.641 P00006a5f ------ 1 - 08055350 76 65 2f 31 2e 30 0d 0a 41 63 63 65 70 74 3a 20 ve/1.0..Accept:
20101002 181142.641 P00006a5f ------ 1 - 08055360 2a 2f 2a 0d 0a 0d 0a                            */*....
20101002 181202.794 P00006a5f   9128 1 - read all expected http responses
20101002 181202.794 P00006a5f   9128 1 - time for all connections: 0.391057 seconds or 25571.718778 per second
20101002 181202.794 P00006a5f   9128 1 - time for all queries    : 20.152358 seconds or 24810.992567 per second
20101002 181202.794 P00006a5f   9128 1 - time for all            : 20.543415 seconds or 24338.699486 per second

Where nginx manages 25,571 connections per second, the SXE implementation manages 25,009 connections per second; a performance tie. Further, where nginx manages 24,810 queries per second, the SXE implementation manages 59,171 queries per second; a 2.4 fold increase. This is an especially great result for SXE because there is still scope for optimizing its code further.

During the test I also monitored memory usage of both the client and server processes:

# top -b -d1 | egrep "(nginx|http)"
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 44712 5160  692 S    0  0.1   0:18.46 nginx
27231 root      15   0 18064  16m  516 R   79  0.4   0:00.79 http
27216 nobody    16   0 57468  17m  692 R   67  0.4   0:19.13 nginx
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    17   0 60064  20m  692 R   98  0.5   0:20.12 nginx
27231 root      15   0 18064  16m  516 S   58  0.4   0:01.37 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    19   0 60064  20m  692 R   96  0.5   0:21.09 nginx
27231 root      15   0 18064  16m  516 R   68  0.4   0:02.05 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    20   0 60064  20m  692 R   97  0.5   0:22.07 nginx
27231 root      15   0 18064  16m  516 R   64  0.4   0:02.69 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    24   0 60064  20m  692 R   97  0.5   0:23.05 nginx
27231 root      15   0 18064  16m  516 R   66  0.4   0:03.35 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 60064  20m  692 R   86  0.5   0:23.91 nginx
27231 root      15   0 18064  16m  516 R   42  0.4   0:03.77 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 60064  20m  692 R   99  0.5   0:24.90 nginx
27231 root      15   0 18064  16m  516 R   50  0.4   0:04.27 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 60064  20m  692 R   99  0.5   0:25.90 nginx
27231 root      15   0 18064  16m  516 R   50  0.4   0:04.77 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 60064  20m  692 R  100  0.5   0:26.91 nginx
27231 root      15   0 18064  16m  516 R   50  0.4   0:05.27 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 60064  20m  692 R   99  0.5   0:27.91 nginx
27231 root      15   0 18064  16m  516 R   54  0.4   0:05.81 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 60064  20m  692 R   99  0.5   0:28.91 nginx
27231 root      15   0 18064  16m  516 R   53  0.4   0:06.34 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 60064  20m  692 R   99  0.5   0:29.91 nginx
27231 root      15   0 18064  16m  516 R   50  0.4   0:06.84 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 60064  20m  692 R   99  0.5   0:30.90 nginx
27231 root      15   0 18064  16m  516 R   51  0.4   0:07.35 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 60064  20m  692 R  100  0.5   0:31.91 nginx
27231 root      15   0 18064  16m  516 R   50  0.4   0:07.85 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 60064  20m  692 D   54  0.5   0:32.45 nginx
27231 root      15   0 18064  16m  516 S   28  0.4   0:08.13 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    19   0 60064  20m  692 R   89  0.5   0:33.35 nginx
27231 root      15   0 18064  16m  516 S   61  0.4   0:08.74 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    21   0 60064  20m  692 R   97  0.5   0:34.33 nginx
27231 root      15   0 18064  16m  516 R   66  0.4   0:09.40 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    22   0 60064  20m  692 R   68  0.5   0:35.01 nginx
27231 root      15   0 18064  16m  516 R   34  0.4   0:09.74 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 60064  20m  692 R   98  0.5   0:36.00 nginx
27231 root      15   0 18064  16m  516 S   66  0.4   0:10.40 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 60064  20m  692 R   97  0.5   0:36.98 nginx
27231 root      15   0 18064  16m  516 R   52  0.4   0:10.92 http
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27216 nobody    25   0 44712 5160  692 S   63  0.1   0:37.61 nginx
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx
27215 root      25   0 40788  836  344 S    0  0.0   0:00.00 nginx

Unlike SXE, nginx uses a dynamic memory model and top shows me that peak memory usage is only 20MB which is very similar to the peak memory usage of SXE of 16MB; another tie.

In conclusion, if you’re planning to serve static content and CPU is your bottleneck then using nginx could cause you to employ up to 2.4 times as many servers as if you had implemented with SXE. It would be interesting to create a real static content delivery system using SXE and post a more realistic head to head comparison. If anybody has ideas on what the more realistic head to head comparison might look like then please comment below.

 

 
%d bloggers like this: