NSE loop bug

Nmap Scripting Engine is an engine for running Lua scripts inside Nmap. It's based on Nmap asynchronous library NSock.

During work on NSE script I found that sometimes Lua threads were frozen for too long. For example socket_object:receive() should take few milliseconds, but it took more than a hundred. In normal usage I wouldn't even notice that events are delayed, but this time I needed exact timings.

Like every asynchronous library NSock has special event loop. The main loop in NSE looks similar:

int process_mainloop(){
    ...
    while(unfinished_lua_threads > 0){
        ...
        nsock_loop(50ms); // block for at most 50ms, or till the first event
        ...
        if(running_scripts.begin() == running_scripts.end())
            continue;

        current = *(running_scripts.begin());
        // execute thread. It should be yielded back to waiting_scripts or end.
        lua_resume(current); 
        ...
    }
}

It's not obvious to spot the bug. The problem is that we handle only one Lua thread for one loop iteration. But it's possible that many events have occurred during nsock_loop.

This kind of bugs is quite common. For example PGBouncer recently had(still has?) similar bug while accepting connections. It allowed only one new connection per one event loop iteration.

The fixed loop will look like this:

int process_mainloop(){
    ...
    while(unfinished_lua_threads > 0){
        ...
        nsock_loop(50ms); // block for at most 50ms, or till the first event
        ...
        while(running_scripts.begin() != running_scripts.end()){
            ...
            current = *(running_scripts.begin());
            // execute thread. It should be yielded back to waiting_scripts or end.
            lua_resume(current); 
            ...
        }
    }
}

UPDATE #1 Fix is commited. It is in nmap svn version newer than r6857.

django-evserver - Asynchronous server for Django

I mentioned the idea in previous post about asychronous wsgi.

Now we have working example of simple chat application. The chat is nothing special, but it's made in Django! To make this possible I created special server, django-evserver. It's quite light wrapper to libevent http layer (several hundred of lines in python/ctypes).

Using this technology it's now possible to create asynchronous WSGI applications. Currently I implemented the Comet javascript library and some wrappers for Django views, to make easier writing comet applications in Django.

Here is how example chat application evserver-chat looks like:

The code is commited to code.google.com, maybe in future I'll write some more code and documentation. For example it could be interesting how I used named fifos (mkfifo) to pass events.

32 or 64 bits?

Just a note, that I posted an article about advantages of 32 bit architecture over 64 bit and why we chose 32 bit at Grono.net.

It's in Polish, but the main point is that in our case Django at 32 bits is consuming about 30% less memory than on 64 bits.

To show the differece, here is output from top command at 64 bits:

VIRT  RES  SHR S %CPU   TIME+  COMMAND
212m  96m 3832 S    0  1:49.61 django '/gallery/3329236/0/'
203m  96m 3120 S    0  1:37.78 django '/gallery/4871149/6/0/'
211m  95m 3832 S    0  1:38.74 django '/mailbox/box/1/'
210m  95m 3824 S    0  1:45.45 django '/gallery/photo/40328359/'
210m  95m 3844 S    0  1:48.43 django '/users/index/'
210m  94m 3848 S    0  1:33.94 django '/users/'

And here from 32 bit:

VIRT  RES  SHR S %CPU    TIME+  COMMAND
68396  62m 3484 S    0  2:06.94 django '/users/433317/'
67716  61m 3488 S    0  1:50.23 django '/gallery/photo/42009962/'
66264  60m 3488 S    0  2:01.94 django '/pub/join/'
65312  59m 3484 S    0  2:08.75 django '/users/'
65148  59m 3508 S    0  1:48.02 django '/users/1533943/friendlist/'
64884  59m 3508 S    0  1:51.48 django '/'