2007-11-02

Trinity choice: nsock over libevent


Everybody knows that Trinity is using nmap.
But what she has to nsock or libevent, well nothing.

Recently I wrote some software using libevent and I'm disgusted. Libevent is poorly documented (except the self-explanatory function names which are suggestive), there are not many examples on the web. Even though I took an effort and wrote few programs.

About a year ago I put my hands on nmap project, I wrote something for nmap's asynchronous networking library nsock.

There are some really interesting differences between these two libraries:

  • In libevent you must allocate memory and set event structures by hand, from manual: "It is the responsibility of the caller to provide these functions with pre-allocated event structures.". In nsock event objects are managed by library.
  • I thought that EV_PERSIST events should be persistent. I don't get exactly the idea of EV_PERSIST event type in libevent. It's not working for me when I think it should (for example for EV_WRITE). On the other hand nsock doesn't provide any kind of persistent events.
  • Libevent provides the api for queuing signals. I haven't got an idea what application could theoretically take a use of such feature. I can't think of any use case for queuing signals in real world applications.
  • Nsock executes callback function with the information about a state of an event like if it has succeeded, failed or timeouted. Libevent doesn't. User have to check the socket status by hand.
  • In Nsock user can get current time (the time after last select() in main event loop) without syscall using nsock_gettimeofday(), which can speedup program. In libevent you have to use gettimefday().
  • Libevent supports multiple polling methods, most notably epoll(), while nsock is using only slow select().
  • Libevent could be used for both client or server side. Nsock is build especially for client side, it's possible to abuse it and use it for server side, but it's more like hacking.
The thing I like most in nsock is the easiness of using it. I don't have to worry about event structures. I have the guarantee that sooner or later my callback function will be called. To show the difference I have an example. It's the simplest program I could think of. It registers timer callback with null timeout, million times one after another sequentially.

First libevent example:
#include <stdlib.h>
#include <event.h>

/*
gcc -Wall event_test1.c -levent && time ./a.out
*/

int counter = 0;
struct timeval tv = {0,0};
struct event ev;

void timer_handler(int fd, short event, void *ud){
event_add(&ev, &tv);
counter++;
if(counter > 1000000)
exit(0);
}

int main(){
event_init();

event_set(&ev, -1, EV_TIMEOUT, timer_handler, NULL);
event_add(&ev, &tv);
event_loop(0);
return(0);
}
And very similar nsock code:
#include <nsock.h>
#include <stdlib.h>

/* Sorry, some magic with compilation. Nsock isn't standalone library yet!
gcc -Wall nsock_test.c -I/home/majek/nmap/nsock/include /home/majek/nmap/nsock/src/libnsock.a -ldnet /home/majek/nmap/nbase/libnbase.a /home/majek/nmap/libpcap/libpcap.a && time ./a.out
*/

nsock_pool nsp; /* global */
int counter = 0;

void timer_handler (nsock_pool nsp, nsock_event nse, void *ud){
nsock_timer_create(nsp, timer_handler, 0 /*ms*/, NULL);
counter++;
if(counter > 1000000)
exit(0);
}

int main(){
nsp = nsp_new(NULL);
nsock_timer_create(nsp, timer_handler, 0 /*ms*/, NULL);
nsock_loop(nsp, 1000000);
return(0);
}
Times of execution this programs are very interesting for me. Unbelievably it seems that nsock is more than two times faster!
Libevent averate timings:
real    0m2.271s
user 0m0.640s
sys 0m1.624s
Nsock average time:
real    0m0.941s
user 0m0.392s
sys 0m0.548s
The "user time" shows that libevent implementation of internal structures is much heavier than nsock. Even that nsock should have more work because it's allocating event structures alone, without users help. The "sys time" should be similar for both libraries, but it's not. Strace is going to reveal the issue:
# The loop is reduced to 100 iterations.

# Data for NSOCK
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
nan 0.000000 0 203 gettimeofday

# Data for LIBEVENT
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
nan 0.000000 0 406 gettimeofday
nan 0.000000 0 101 epoll_wait
nan 0.000000 0 1 epoll_create
nan 0.000000 0 1 epoll_ctl
Well, it seems clear that two things are broken in libevent. First, the gettimeofday() is called four times for every event loop iteration, it's unacceptable waste of resources. Second, the epoll() is executed for every event loop, while for nsock even a single select() isn't called. In this particular case, when timer events are registered with Null timeout there isn't a need of executing epoll().

Summary:
From programmers point of view, I like nsock programming style. Unfortunately nsock doesn't have many important features, like epoll or server side sockets handling.

It seems that libevent ain't no perfect either.

Maybe there is a need of async library with easy api like nsock and full-featured like libevent?


3 comments:

Dalibor (dast_at_panelnet.cz) said...

EV_PERSIST works.
1. according to documentation EV_TIMEOUT results in an automatic event_del()

2. timeout set to zero is a bad idea

Try this code:

int counter = 0;
struct timeval tv = {1,1};
struct event ev;

void handler(int fd, short event, void *ud){
// event_add(&ev, &tv);
counter++;
printf("%x, x\n", event);
if(counter > 10)
exit(0);
}

int main(){
event_init();

event_set(&ev, 1, EV_TIMEOUT | EV_WRITE | EV_PERSIST, handler, NULL);
event_add(&ev, &tv);
event_dispatch();
printf("%d\n", counter);
return 0;
}

rudolpho said...

dalibor - PERSIST is not impl for timers. in your example you are writing to an FD, so the dispatcher calls your handler with ready state immediately.

Anonymous said...

hi,

hope you had a look at libev too.
http://libev.schmorp.de/bench.html

It is quite a bit faster than libevent, has a compatibility libevent interface and should fix the problems you had with libevent.

e.g: "Another reason for the higher performance, especially in the set-up phase, might be that libevent calls the epoll_ctl syscall on each change (twice per fd for del/add), while libev only sends changes to the kernel before the next poll (EPOLL_MOD)."