Asynchronous Django responses (comet), yes, for WSGI

Comet on python?

I was looking for comet server implementation in python. There are many projects that claim to provide this, but none of them is working for me. Most of the projects are just proofs of concepts that never been deployed for real usage.

Here is the list of projects that my friend found:

• fapws (based on libevent): http://william-os4y.livejournal.com/
• athena: http://divmod.org/trac/wiki/DivmodNevow/Athena
• allegra (based on asyncore)
  
• http://svn.berlios.de/svnroot/repos/allegra/
• http://laurentszyster.be/blog/my-twisted-apologies/
• asynwsgi (on asyncore)
• http://svn.wiretooth.com/svn/open/asynwsgi/
• http://trac.wiretooth.com/public/browser/asynwsgi
• twisted
• http://svn.xantus.org/shortbus/trunk/cometd-twisted/
• http://groups.google.com/group/cometd-users/topics
• orbited: http://orbited.org/
  

The Orbited project looks like the thing I was looking for. The problem is that even the examples from official documentation aren't working. Sorry, I don't want to work with undocumented projects.

Next candidate is Twisted, but I'm not the fan of it for religious reasons.

Fapws project also is looking good, but it supports only basics, no possibility of deferring (preeptimg) request.

The problem

I want to do an ajax request from browser that will be finished only when some event occured at the server side. The connection should stall and wait for the server. (this idea, of pushing events from server to browser is called comet)

Next thing is to allow django applications to use this technology. As far as I know no one has ever done this before, probably because django is based on WSGI stack which is, unfortunetally, fully synchronous.

Even though I think it's possible to allow WSGI stack to support asynchronous responses without modifying the WSGI stack at all.

The solution

Author of FAPWS suggested that it's possible to use libevent directly from python using ctypes. Libevent proved to be very reliable and fast enough in most cases, so it should be good idea to use it.

Using libevent glued with ctypes I created simple WSGI server (William examples helped me much).

As I mentioned the idea was to allow Django to do async responses. For the proof-of-concept I created website that produces output in five chunks, separated by delays of one second. Sounds easy, but it's not so simple to do without freezing server process for the whole time.

Going straight to the example, here's the view.py file from sample django app:

# Url handler of the basic url.
#   Headers only from this response are used.
#   The same with status code, this function
#   response status code is served to the client.
def comet_main(request):
    response = HttpResponse('First content<br /> ') # return something
    response.stage = 0
    return defer_in_time(response,
            1, '/stage/')       # after one second run the '/stage/' url handler
    
@deferred_view()
def comet_stage(request, response_prev=None):
    response = HttpResponse('stage%i<br /> ' % (response_prev.stage + 1) )
    response.stage = response_prev.stage + 1
    if response.stage < 3: # loop here three times
        url = '/stage/'
    else:
        url = '/stage_last/' # than end the connection
    return defer_in_time(response,
            1, url)             # after one second run the url 

@deferred_view()
def comet_stage_last(request, response_prev=None):
    # no magic here. just regular response
    return HttpResponse('last_stage%i<br /> ' % (response_prev.stage +1))
        

Here is how it looks from the browser:


And what really happens on the wire (notice that Transfer-Encoding: chunked is used):


The sources are here.

What's next
The next thing to do is to create the example of full comet application.