How to deliver an event to multiple processes? - scaling django-evserver

The hardest problem I had to solve during my work on django-evserver, was how to efficiently propagate an event to multiple listener processes.

Python, because of GIL, doesn't scale across multiple processors. One process can only use one processor.

In django-evserver we have multiple python processes and some of them could wait for some specific event. The question is how to deliver an event only to the processes that wait for it.

Signals?
The simplest thing we could do when event occurs, is to send Unix signal (for example SIGUSR1) to every process. But that doesn't scale at all, not every process must be waiting for this event. We could use some kind of central server to manage events and send it only to proper listeners. But this needs quite complicated statefull server.

Multicast?
Next idea is to bind to specific udp port and listen for multicast packets. If a process waits for event 1, it will bind to port 10001. Theoretically multiple threads could listen on one multicast udp port. The problem with tcp/udp sockets is that we're limited to 64k unique event types for one machine. Which is not too much.

Can it be done easier?
Well. Reading from named fifo file (mkfifo) is definetelly blocking. The point is that multiple processes can read from one fifo. When some process will write to this fifo, every listener would be noticed. One reader will get the written message, all others will get information that they can't listen any more becaouse writer closed pipe. But that is good enough, every waiting process got an event.

Good points of that approach:

• you could have millions of fifo's
• as many processes could 'bind' to one fifo as you would like to
• writer knows if someone is reading from the fifo - you could tell if the event was delivered to at least one reader
• very easy to implement
• very simple to understand

Bad points:

• doesn't scale across multiple machines (but a daemon that would forward events to other machines is quite simple to implement)
• it's based on specific fifo implementation, probably won't work on windows