Trinity choice: nsock over libevent

Everybody knows that Trinity is using nmap.
But what she has to nsock or libevent, well nothing.

Recently I wrote some software using libevent and I'm disgusted. Libevent is poorly documented (except the self-explanatory function names which are suggestive), there are not many examples on the web. Even though I took an effort and wrote few programs.

About a year ago I put my hands on nmap project, I wrote something for nmap's asynchronous networking library nsock.

There are some really interesting differences between these two libraries:

• In libevent you must allocate memory and set event structures by hand, from manual: "It is the responsibility of the caller to provide these functions with pre-allocated event structures.". In nsock event objects are managed by library.
• I thought that EV_PERSIST events should be persistent. I don't get exactly the idea of EV_PERSIST event type in libevent. It's not working for me when I think it should (for example for EV_WRITE). On the other hand nsock doesn't provide any kind of persistent events.
• Libevent provides the api for queuing signals. I haven't got an idea what application could theoretically take a use of such feature. I can't think of any use case for queuing signals in real world applications.
• Nsock executes callback function with the information about a state of an event like if it has succeeded, failed or timeouted. Libevent doesn't. User have to check the socket status by hand.
• In Nsock user can get current time (the time after last select() in main event loop) without syscall using nsock_gettimeofday(), which can speedup program. In libevent you have to use gettimefday().
• Libevent supports multiple polling methods, most notably epoll(), while nsock is using only slow select().
• Libevent could be used for both client or server side. Nsock is build especially for client side, it's possible to abuse it and use it for server side, but it's more like hacking.

The thing I like most in nsock is the easiness of using it. I don't have to worry about event structures. I have the guarantee that sooner or later my callback function will be called. To show the difference I have an example. It's the simplest program I could think of. It registers timer callback with null timeout, million times one after another sequentially.

First libevent example:

#include <stdlib.h>
#include <event.h>

 /* 
 gcc -Wall event_test1.c -levent && time ./a.out
 */

int counter = 0;
struct timeval tv = {0,0};
struct event ev;

void timer_handler(int fd, short event, void *ud){
    event_add(&ev, &tv);
    counter++;
    if(counter > 1000000)
        exit(0);
}

int main(){
    event_init();
    
    event_set(&ev, -1, EV_TIMEOUT, timer_handler, NULL);
    event_add(&ev, &tv);
    event_loop(0);
    return(0);
}

And very similar nsock code:

#include <nsock.h>
#include <stdlib.h>
 
 /* Sorry, some magic with compilation. Nsock isn't standalone library yet! 
 gcc -Wall nsock_test.c -I/home/majek/nmap/nsock/include /home/majek/nmap/nsock/src/libnsock.a -ldnet /home/majek/nmap/nbase/libnbase.a /home/majek/nmap/libpcap/libpcap.a && time ./a.out
 */

nsock_pool nsp; /* global */
int counter = 0;

void timer_handler (nsock_pool nsp, nsock_event nse, void *ud){
    nsock_timer_create(nsp, timer_handler, 0 /*ms*/, NULL);
    counter++;
    if(counter > 1000000)
        exit(0);
}

int main(){
    nsp = nsp_new(NULL);
    nsock_timer_create(nsp, timer_handler, 0 /*ms*/, NULL);
    nsock_loop(nsp, 1000000);
    return(0);
}

Times of execution this programs are very interesting for me. Unbelievably it seems that nsock is more than two times faster!
Libevent averate timings:

real    0m2.271s
user    0m0.640s
sys     0m1.624s

Nsock average time:

real    0m0.941s
user    0m0.392s
sys     0m0.548s

The "user time" shows that libevent implementation of internal structures is much heavier than nsock. Even that nsock should have more work because it's allocating event structures alone, without users help. The "sys time" should be similar for both libraries, but it's not. Strace is going to reveal the issue:

# The loop is reduced to 100 iterations.

# Data for NSOCK
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
   nan    0.000000           0       203           gettimeofday

# Data for LIBEVENT
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
   nan    0.000000           0       406           gettimeofday
   nan    0.000000           0       101           epoll_wait
   nan    0.000000           0         1           epoll_create
   nan    0.000000           0         1           epoll_ctl

Well, it seems clear that two things are broken in libevent. First, the gettimeofday() is called four times for every event loop iteration, it's unacceptable waste of resources. Second, the epoll() is executed for every event loop, while for nsock even a single select() isn't called. In this particular case, when timer events are registered with Null timeout there isn't a need of executing epoll().

Summary:
From programmers point of view, I like nsock programming style. Unfortunately nsock doesn't have many important features, like epoll or server side sockets handling.

It seems that libevent ain't no perfect either.

Maybe there is a need of async library with easy api like nsock and full-featured like libevent?