Caching at grono

I posted article (in polish) about caching at Grono, with description of the caching decorator called memoize.

It's a very simple decorator, but it's working perfectly. Without it the site wouldn't be working :)

How to deliver an event to multiple processes? - scaling django-evserver

The hardest problem I had to solve during my work on django-evserver, was how to efficiently propagate an event to multiple listener processes.

Python, because of GIL, doesn't scale across multiple processors. One process can only use one processor.

In django-evserver we have multiple python processes and some of them could wait for some specific event. The question is how to deliver an event only to the processes that wait for it.

Signals?
The simplest thing we could do when event occurs, is to send Unix signal (for example SIGUSR1) to every process. But that doesn't scale at all, not every process must be waiting for this event. We could use some kind of central server to manage events and send it only to proper listeners. But this needs quite complicated statefull server.

Multicast?
Next idea is to bind to specific udp port and listen for multicast packets. If a process waits for event 1, it will bind to port 10001. Theoretically multiple threads could listen on one multicast udp port. The problem with tcp/udp sockets is that we're limited to 64k unique event types for one machine. Which is not too much.

Can it be done easier?
Well. Reading from named fifo file (mkfifo) is definetelly blocking. The point is that multiple processes can read from one fifo. When some process will write to this fifo, every listener would be noticed. One reader will get the written message, all others will get information that they can't listen any more becaouse writer closed pipe. But that is good enough, every waiting process got an event.

Good points of that approach:

• you could have millions of fifo's
• as many processes could 'bind' to one fifo as you would like to
• writer knows if someone is reading from the fifo - you could tell if the event was delivered to at least one reader
• very easy to implement
• very simple to understand

Bad points:

• doesn't scale across multiple machines (but a daemon that would forward events to other machines is quite simple to implement)
• it's based on specific fifo implementation, probably won't work on windows

Nmap - os detection for every open tcp port

While ago I created tool, that shows p0f-like signatures for every open port. This signature is based on information taken from single SYN-ACK packet from target port. Sample output:

$ sudo ./nmap -n -sT -PS80 -p21,22,53,80,443 --script=p0f.nse www.cisco.com

Starting Nmap 4.22SOC1 ( http://insecure.org ) at 2007-07-12 00:04 CEST
Interesting ports on 198.133.219.25:
PORT    STATE    SERVICE
21/tcp  open     ftp
|_ p0f signature: UNKNOWN [65500:61:1:64:M1436,N,N,S,N,W0,N,N,T:AT:?:?] (link: IPSec/GRE, up: 1446 hrs, ipid:54702)
22/tcp  filtered ssh
53/tcp  filtered domain
80/tcp  open     http
|_ p0f signature: UNKNOWN [8192:238:0:44:M1460:A:?:?] (link: ethernet/modem, up: disabled, fill:4008, ipid:1)
443/tcp open     https
|_ p0f signature: UNKNOWN [8192:238:0:44:M1460:A:?:?] (link: ethernet/modem, up: disabled, fill:f8ef, ipid:10)

Now I created similar test, but it does full os detection for open port. It uses nmap os detection algorithms to determine what os is on specific port. This is very useful if target uses destination nat and forwards incoming connections to other hosts. Let's look again at the cisco.com:

$ export NMAPDIR=.
$ sudo ./nmap -sS -p25,53,80,113,443,8080 -n www.cisco.com -PS80 --script=os.nse -O

Starting Nmap 4.53 ( http://nmap.org ) at 2008-02-29 12:51 CET
Interesting ports on 198.133.219.25:
PORT     STATE    SERVICE
25/tcp   open     smtp
|_ os: NCR 5676 or 5688 automated teller machine (94%), Rockwell Automation 1761-NET-ENI Ethernet-to-RS-232-C interface module (94%), Billion 7404VGO-M DSL router (93%)
53/tcp   filtered domain
80/tcp   open     http
|_ os: D-Link DWL-624+ or TRENDnet TEW-432BRP wireless broadband router (93%), Linksys BEFSR41 EtherFast broadband router or D-Link DCS-6620G webcam (93%), HP 9100c Digital Sender scanner (93%)
113/tcp  filtered auth
443/tcp  open     https
|_ os: D-Link DWL-624+ or TRENDnet TEW-432BRP wireless broadband router (93%), Linksys BEFSR41 EtherFast broadband router or D-Link DCS-6620G webcam (93%), HP 9100c Digital Sender scanner (93%)
8080/tcp closed   http-proxy
No OS matches for host
Uptime: 0.007 days (since Fri Feb 29 12:42:05 2008)

I don't suggest that cisco uses D-Link DWL-624. But it's obvious that different host answers to the port 25 and different to ports 80 and 443.

My os detection script is much less reliable than standard nmap -O, because it sends packets to single open port. Normal scan uses also icmp packets, udp probes and tcp closed ports tests.

But I hope that my script can give a brief info of forwarded ports.

If you know how to interpret nmap-os signatures, adding -d1 can give you more detailed information:

$ export NMAPDIR=.
$ sudo ./nmap -sS -p21,22,25,53,80,113,443,8080 -n www.cisco.com -PS80 --script=os.nse -O -d1

Interesting ports on 198.133.219.25:
PORT     STATE    SERVICE    REASON
21/tcp   closed   ftp        reset
22/tcp   filtered ssh        no-response
25/tcp   open     smtp       syn-ack
|  os: Rockwell Automation 1761-NET-ENI Ethernet-to-RS-232-C interface module (94%), Billion 7404VGO-M DSL router (93%), Vodavi XTS-IP PBX (93%)
|      seq info: got packets=6 try=1/1 variance=5.431390
|      ECN(R=Y%DF=Y%TG=40%W=8052%O=NW3NNSM5B4%CC=N%Q=)
|      T2(R=N)
|      T3(R=N)
|      T4(R=N)
|      SEQ(SP=100%GCD=1%ISR=10E%TI=RD%TS=14%uptime=9 minutes)
|      OPS(O1=NNT11NW3NNSM5B4%O2=NNT11NW3NNSM5B4%O3=NNT11NW3M5B4%O4=NNT11NW3NNSM5B4%O5=NNT11NW3NNSM5B4%O6=NNT11NNSM5B4)
|      WIN(W1=80AE%W2=8017%W3=802D%W4=80AE%W5=802F%W6=FFF7)
|_     T1(R=Y%DF=Y%TG=40%S=O%A=S+%F=AS%RD=0%Q=)
53/tcp   filtered domain     no-response
80/tcp   open     http       syn-ack
|  os: D-Link DWL-624+ or TRENDnet TEW-432BRP wireless broadband router (93%), Linksys BEFSR41 EtherFast broadband router or D-Link DCS-6620G webcam (93%), HP 9100c Digital Sender scanner (93%)
|      seq info: got packets=6 try=1/1 variance=2.828427
|      ECN(R=Y%DF=N%TG=FF%W=2000%O=M5B4%CC=N%Q=)
|      T2(R=N)
|      T3(R=N)
|      T4(R=N)
|      SEQ(SP=108%GCD=1%ISR=10E%TI=RD%TS=U)
|      OPS(O1=M5B4%O2=M5B4%O3=M5B4%O4=M5B4%O5=M5B4%O6=M5B4)
|      WIN(W1=2000%W2=2000%W3=2000%W4=2000%W5=2000%W6=2000)
|_     T1(R=Y%DF=N%TG=FF%S=O%A=S+%F=AS%RD=0%Q=)
113/tcp  filtered auth       no-response
443/tcp  open     https      syn-ack
|  os: D-Link DWL-624+ or TRENDnet TEW-432BRP wireless broadband router (93%), Linksys BEFSR41 EtherFast broadband router or D-Link DCS-6620G webcam (93%), HP 9100c Digital Sender scanner (93%)
|      seq info: got packets=6 try=1/1 variance=1.154701
|      ECN(R=Y%DF=N%TG=FF%W=2000%O=M5B4%CC=N%Q=)
|      T2(R=N)
|      T3(R=N)
|      T4(R=N)
|      SEQ(SP=105%GCD=1%ISR=106%TI=RD%TS=U)
|      OPS(O1=M5B4%O2=M5B4%O3=M5B4%O4=M5B4%O5=M5B4%O6=M5B4)
|      WIN(W1=2000%W2=2000%W3=2000%W4=2000%W5=2000%W6=2000)
|_     T1(R=Y%DF=N%TG=FF%S=O%A=S+%F=AS%RD=0%Q=)

I should also make it clear that p0f.nse don't sends any crafted packets (it only openes connection using standard connect()). But os.nse sends at least 10 crafted packet to every open port. That's not very stealthy method.


The sources are in my svn:

svn co svn://svn.insecure.org/nmap-exp/majek04/nmap-6861 nmap-majek