2007-11-02

Libevent under python

So I want to write asynchronous tcp/ip server in python.

I really hate overblown twisted. The thing I like most in Python is simplicity and easiness to read. Well, twisted in my opinion doesn't have this attributes. It's of course my personal feeling, mostly because I don't know twisted. But such statements are making me sick, at first sight I really don't understand what are they for (from core twisted documentation):


internet.TCPServer(7779, IFingerFactory(f)).setServiceParent(
service.IServiceCollection(application))

I remember such cascades from Java rather than Python and I dislike Java especially for this way of programming.

Getting back to point. If not twisted than what? Lets try libevent. Stable C library, which is a production standard now. Memcached is using it, Tor is using it, IO also. It must be really professional asynchronous library.

There are also some other async linux libraries, like liboop. Wait a moment, libevent 500k google hits, liboop 25k. Sorry, the only asynchronous library for linux is libevent for now.
(btw. Liboop's api is in my opinion totally broken)

Are there any python wrappers for libevent? Google says about two: libevent-python and pyevent. Both seem abandoned.
I don't like exceptions inside libraries, so it didn't took long to eliminate pyevent:
    Traceback (most recent call last):
File "./tights.py", line 37, in
main()
File "./tights.py", line 34, in main
event.dispatch()
File "event.pyx", line 262, in event.dispatch
TypeError: exceptions must be strings, classes, or instances, not type

Update: Michael Carter writes in comments that this bug is the fault of pyrex rather than pyevent and that it shouldn't happen in python2.4(I use 2.5). I'll have to take a closer look at pyevent again.

Libevent-python seems to be working. Unfortunately this wrapper removed the concept of 'userdata' passed to callback in favor of class-based approach. The difference is that callback is not called with 'userdata' but it's called on specific class instance, so callback has 'self' object. I personally prefer the 'userdata' approach, but it's quite easy to switch.

Here's an example of simple libevent-python server, which is just printing every data it receives (something like standard echo_server.py example from libevent-python, but simpler)
#!/usr/bin/python
# -*- coding: utf-8 -*-
import socket, signal, libevent

def callback_interrupt(signum, events, event_obj):
libevent.loopExit(0)

def callback_onconnect(fd, events, event):
# hack to avoid passing sock_srv from main()
sock_srv = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)
sock, (host, port) = sock_srv.accept()
conn = Connection(sock, (host, port))

class Connection:
def __init__(self, sock, addr):
self.sock = sock
self.addr = addr
self.sock.setblocking(False)
libevent.createEvent(sock, libevent.EV_READ, self.callback_onread).addToLoop()

def callback_onread(self, fd, events, event_obj):
buf = self.sock.recv(4096)
if not buf: # just disconnect
self.sock.close()
return
# Yeah, print!
print "%r %r" % (self.addr, buf)
# reuse current event
event_obj.addToLoop()

if __name__ == '__main__':
# bind.
sock_srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock_srv.setblocking(False)
sock_srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock_srv.bind(('0.0.0.0', 1111))
sock_srv.listen(5)

libevent.createSignalHandler(signal.SIGINT, callback_interrupt).addToLoop()
libevent.createEvent(sock_srv, libevent.EV_READ|libevent.EV_PERSIST, callback_onconnect).addToLoop()
libevent.dispatch()


UPDATE #1:
Here seems to be another way of connecting libevent with python, using raw ctypes: http://william-os4y.livejournal.com/3718.html.


15 comments:

Unknown said...

I think the error you got with pyevent is due to a bug in pyrex not the pyevent source. The error doesn't happen in python 2.4, only python 2.5 + pyevent 0.3.

Gary van der Merwe said...

Two other python libraries for libevent, but that also provide async io:
http://eventlet.net/
http://www.gevent.org/

markson said...

As of late, Data science has developed after some time with Big Data, a great deal of crisp innovations and systems have emerged. ExcelR Data Science Courses

ss said...

Cool stuff you have and you keep overhaul every one of us

data scientist course

Click here for more info

data analytics

priyash said...

wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
Data science Interview Questions
Data Science Course

datasciencecourse said...

I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.

data science course

Data Science Course said...

I am looking for and I love to post a comment that "The content of your post is awesome" Great work!

Data Science Course

Data Science Training said...

This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.

Data Science Training

priyash said...

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
Correlation vs Covariance
Simple linear regression

Gaurav said...

Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!
data science course in indore

Data Science Bangalore said...

Very interesting blog. Many blogs I see these days do not really provide anything that attracts others, but believe me the way you interact is literally awesome.You can also check my articles as well.

Data Science In Banglore With Placements
Data Science Course In Bangalore
Data Science Training In Bangalore
Best Data Science Courses In Bangalore
Data Science Institute In Bangalore

Thank you..

Priyanka said...

Attend The Data Analyst Course From ExcelR. Practical Data Analyst Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analyst Course.
Data Analyst Course

EXCELR said...

This post is great. I reallly admire your post. Your post was awesome.
data science course in Hyderabad

360digitmgas said...

This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck. ethical hacking course training in coimbatore

priyash said...

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
Correlation vs Covariance
Simple linear regression
data science interview questions