django-evserver - Asynchronous server for Django

I mentioned the idea in previous post about asychronous wsgi.

Now we have working example of simple chat application. The chat is nothing special, but it's made in Django! To make this possible I created special server, django-evserver. It's quite light wrapper to libevent http layer (several hundred of lines in python/ctypes).

Using this technology it's now possible to create asynchronous WSGI applications. Currently I implemented the Comet javascript library and some wrappers for Django views, to make easier writing comet applications in Django.

Here is how example chat application evserver-chat looks like:

The code is commited to code.google.com, maybe in future I'll write some more code and documentation. For example it could be interesting how I used named fifos (mkfifo) to pass events.

32 or 64 bits?

Just a note, that I posted an article about advantages of 32 bit architecture over 64 bit and why we chose 32 bit at Grono.net.

It's in Polish, but the main point is that in our case Django at 32 bits is consuming about 30% less memory than on 64 bits.

To show the differece, here is output from top command at 64 bits:

VIRT  RES  SHR S %CPU   TIME+  COMMAND
212m  96m 3832 S    0  1:49.61 django '/gallery/3329236/0/'
203m  96m 3120 S    0  1:37.78 django '/gallery/4871149/6/0/'
211m  95m 3832 S    0  1:38.74 django '/mailbox/box/1/'
210m  95m 3824 S    0  1:45.45 django '/gallery/photo/40328359/'
210m  95m 3844 S    0  1:48.43 django '/users/index/'
210m  94m 3848 S    0  1:33.94 django '/users/'

And here from 32 bit:

VIRT  RES  SHR S %CPU    TIME+  COMMAND
68396  62m 3484 S    0  2:06.94 django '/users/433317/'
67716  61m 3488 S    0  1:50.23 django '/gallery/photo/42009962/'
66264  60m 3488 S    0  2:01.94 django '/pub/join/'
65312  59m 3484 S    0  2:08.75 django '/users/'
65148  59m 3508 S    0  1:48.02 django '/users/1533943/friendlist/'
64884  59m 3508 S    0  1:51.48 django '/'

Nmap NSE-pcap example

At nmap-dev mailing list Lionel asked me about NSE-pcap. He wanted to send raw UDP packet and capture the response.

I created an example script that does this.
The patch against nmap.

Or if you prefer already patched files:

• packet.lua
• pcap-example.lua

Is 8GB RAM faster on 64bit or on 32bit processor?

A friend asked me if 8 gigabytes of ram is much slower on 32bit than on 64bit linux. The standard 32 bit can handle at most 4GB of ram. To access more ram you have to enable the CONFIG_HIGHMEM64G option in kernel. The question is if this feature make access to memory much slower.

I created basic test to measure ram speed. The program sequentially writes to rather big memory chunk. After first benchmark, as a single process, the program forks to check if the memory access would be much slower for more concurrent processes. The goal is also to make use of more than 4GBytes of memory, so that we know that HIGHMEM is used.

The results showed that, uh, the memory access is as fast on 64 bits as on 32 bits for big memory usage.

My test program is here.

Some of the tests:
1. 4 processes, every uses 1500MB memory

32-bit-machine$ ./test-mem 4 10 1500
For single process:
In 10680.104ms done 13 tests. It means 478.629 mln (read int)/sec

For 4 processes:
In 34514.712ms done 13 tests. It means 148.105 mln (read int)/sec
In 34584.651ms done 13 tests. It means 147.806 mln (read int)/sec
In 34607.142ms done 13 tests. It means 147.710 mln (read int)/sec
In 34653.662ms done 13 tests. It means 147.511 mln (read int)/sec

64-bit-machine$ ./test-mem 4 10 1500
For single process:
In 10817.252ms done 13 tests. It means 472.561 mln (read int)/sec

For 4 processes:
In 35059.194ms done 13 tests. It means 145.805 mln (read int)/sec
In 35102.285ms done 13 tests. It means 145.626 mln (read int)/sec
In 35102.671ms done 13 tests. It means 145.624 mln (read int)/sec
In 35175.010ms done 13 tests. It means 145.325 mln (read int)/sec

2. 10 processes, every one uses 512MB of memory (this is more like context-switching test)

32-bit-machine$ ./test-mem 10 10 512
For single process:
In 10251.710ms done 37 tests. It means 484.412 mln (read int)/sec

For 10 processes:
In 82147.174ms done 37 tests. It means  60.453 mln (read int)/sec
In 82751.646ms done 37 tests. It means  60.012 mln (read int)/sec
In 83004.705ms done 37 tests. It means  59.829 mln (read int)/sec
In 82591.951ms done 37 tests. It means  60.128 mln (read int)/sec
In 82678.590ms done 37 tests. It means  60.065 mln (read int)/sec
In 82652.342ms done 37 tests. It means  60.084 mln (read int)/sec
In 82636.134ms done 37 tests. It means  60.095 mln (read int)/sec
In 82535.299ms done 37 tests. It means  60.169 mln (read int)/sec
In 82535.830ms done 37 tests. It means  60.168 mln (read int)/sec
In 82784.449ms done 37 tests. It means  59.988 mln (read int)/sec

64-bit-machine$ ./test-mem 10 10 512
For single process:
In 10273.896ms done 37 tests. It means 483.366 mln (read int)/sec

For 10 processes:
In 82660.408ms done 37 tests. It means  60.078 mln (read int)/sec
In 82557.546ms done 37 tests. It means  60.153 mln (read int)/sec
In 83497.783ms done 37 tests. It means  59.475 mln (read int)/sec
In 82660.669ms done 37 tests. It means  60.078 mln (read int)/sec
In 82708.567ms done 37 tests. It means  60.043 mln (read int)/sec
In 83360.999ms done 37 tests. It means  59.573 mln (read int)/sec
In 83044.817ms done 37 tests. It means  59.800 mln (read int)/sec
In 83117.576ms done 37 tests. It means  59.747 mln (read int)/sec
In 83245.083ms done 37 tests. It means  59.656 mln (read int)/sec
In 83203.710ms done 37 tests. It means  59.686 mln (read int)/sec

3. 3 processes, every one uses 2048MB of ram

32-bit-machine$ ./test-mem 3 10 2048
For single process:
In 10258.171ms done 9 tests. It means 471.023 mln (read int)/sec

For 3 processes:
In 20643.669ms done 9 tests. It means 234.059 mln (read int)/sec
In 21914.211ms done 9 tests. It means 220.489 mln (read int)/sec
In 24316.862ms done 9 tests. It means 198.703 mln (read int)/sec

64-bit-machine$ ./test-mem 3 10 2048
For single process:
In 10213.609ms done 9 tests. It means 473.078 mln (read int)/sec

For 3 processes:
In 20694.933ms done 9 tests. It means 233.479 mln (read int)/sec
In 23680.936ms done 9 tests. It means 204.039 mln (read int)/sec
In 24266.013ms done 9 tests. It means 199.120 mln (read int)/sec