Nmap - os detection for every open tcp port

While ago I created tool, that shows p0f-like signatures for every open port. This signature is based on information taken from single SYN-ACK packet from target port. Sample output:

$ sudo ./nmap -n -sT -PS80 -p21,22,53,80,443 --script=p0f.nse www.cisco.com

Starting Nmap 4.22SOC1 ( http://insecure.org ) at 2007-07-12 00:04 CEST
Interesting ports on 198.133.219.25:
PORT    STATE    SERVICE
21/tcp  open     ftp
|_ p0f signature: UNKNOWN [65500:61:1:64:M1436,N,N,S,N,W0,N,N,T:AT:?:?] (link: IPSec/GRE, up: 1446 hrs, ipid:54702)
22/tcp  filtered ssh
53/tcp  filtered domain
80/tcp  open     http
|_ p0f signature: UNKNOWN [8192:238:0:44:M1460:A:?:?] (link: ethernet/modem, up: disabled, fill:4008, ipid:1)
443/tcp open     https
|_ p0f signature: UNKNOWN [8192:238:0:44:M1460:A:?:?] (link: ethernet/modem, up: disabled, fill:f8ef, ipid:10)

Now I created similar test, but it does full os detection for open port. It uses nmap os detection algorithms to determine what os is on specific port. This is very useful if target uses destination nat and forwards incoming connections to other hosts. Let's look again at the cisco.com:

$ export NMAPDIR=.
$ sudo ./nmap -sS -p25,53,80,113,443,8080 -n www.cisco.com -PS80 --script=os.nse -O

Starting Nmap 4.53 ( http://nmap.org ) at 2008-02-29 12:51 CET
Interesting ports on 198.133.219.25:
PORT     STATE    SERVICE
25/tcp   open     smtp
|_ os: NCR 5676 or 5688 automated teller machine (94%), Rockwell Automation 1761-NET-ENI Ethernet-to-RS-232-C interface module (94%), Billion 7404VGO-M DSL router (93%)
53/tcp   filtered domain
80/tcp   open     http
|_ os: D-Link DWL-624+ or TRENDnet TEW-432BRP wireless broadband router (93%), Linksys BEFSR41 EtherFast broadband router or D-Link DCS-6620G webcam (93%), HP 9100c Digital Sender scanner (93%)
113/tcp  filtered auth
443/tcp  open     https
|_ os: D-Link DWL-624+ or TRENDnet TEW-432BRP wireless broadband router (93%), Linksys BEFSR41 EtherFast broadband router or D-Link DCS-6620G webcam (93%), HP 9100c Digital Sender scanner (93%)
8080/tcp closed   http-proxy
No OS matches for host
Uptime: 0.007 days (since Fri Feb 29 12:42:05 2008)

I don't suggest that cisco uses D-Link DWL-624. But it's obvious that different host answers to the port 25 and different to ports 80 and 443.

My os detection script is much less reliable than standard nmap -O, because it sends packets to single open port. Normal scan uses also icmp packets, udp probes and tcp closed ports tests.

But I hope that my script can give a brief info of forwarded ports.

If you know how to interpret nmap-os signatures, adding -d1 can give you more detailed information:

$ export NMAPDIR=.
$ sudo ./nmap -sS -p21,22,25,53,80,113,443,8080 -n www.cisco.com -PS80 --script=os.nse -O -d1

Interesting ports on 198.133.219.25:
PORT     STATE    SERVICE    REASON
21/tcp   closed   ftp        reset
22/tcp   filtered ssh        no-response
25/tcp   open     smtp       syn-ack
|  os: Rockwell Automation 1761-NET-ENI Ethernet-to-RS-232-C interface module (94%), Billion 7404VGO-M DSL router (93%), Vodavi XTS-IP PBX (93%)
|      seq info: got packets=6 try=1/1 variance=5.431390
|      ECN(R=Y%DF=Y%TG=40%W=8052%O=NW3NNSM5B4%CC=N%Q=)
|      T2(R=N)
|      T3(R=N)
|      T4(R=N)
|      SEQ(SP=100%GCD=1%ISR=10E%TI=RD%TS=14%uptime=9 minutes)
|      OPS(O1=NNT11NW3NNSM5B4%O2=NNT11NW3NNSM5B4%O3=NNT11NW3M5B4%O4=NNT11NW3NNSM5B4%O5=NNT11NW3NNSM5B4%O6=NNT11NNSM5B4)
|      WIN(W1=80AE%W2=8017%W3=802D%W4=80AE%W5=802F%W6=FFF7)
|_     T1(R=Y%DF=Y%TG=40%S=O%A=S+%F=AS%RD=0%Q=)
53/tcp   filtered domain     no-response
80/tcp   open     http       syn-ack
|  os: D-Link DWL-624+ or TRENDnet TEW-432BRP wireless broadband router (93%), Linksys BEFSR41 EtherFast broadband router or D-Link DCS-6620G webcam (93%), HP 9100c Digital Sender scanner (93%)
|      seq info: got packets=6 try=1/1 variance=2.828427
|      ECN(R=Y%DF=N%TG=FF%W=2000%O=M5B4%CC=N%Q=)
|      T2(R=N)
|      T3(R=N)
|      T4(R=N)
|      SEQ(SP=108%GCD=1%ISR=10E%TI=RD%TS=U)
|      OPS(O1=M5B4%O2=M5B4%O3=M5B4%O4=M5B4%O5=M5B4%O6=M5B4)
|      WIN(W1=2000%W2=2000%W3=2000%W4=2000%W5=2000%W6=2000)
|_     T1(R=Y%DF=N%TG=FF%S=O%A=S+%F=AS%RD=0%Q=)
113/tcp  filtered auth       no-response
443/tcp  open     https      syn-ack
|  os: D-Link DWL-624+ or TRENDnet TEW-432BRP wireless broadband router (93%), Linksys BEFSR41 EtherFast broadband router or D-Link DCS-6620G webcam (93%), HP 9100c Digital Sender scanner (93%)
|      seq info: got packets=6 try=1/1 variance=1.154701
|      ECN(R=Y%DF=N%TG=FF%W=2000%O=M5B4%CC=N%Q=)
|      T2(R=N)
|      T3(R=N)
|      T4(R=N)
|      SEQ(SP=105%GCD=1%ISR=106%TI=RD%TS=U)
|      OPS(O1=M5B4%O2=M5B4%O3=M5B4%O4=M5B4%O5=M5B4%O6=M5B4)
|      WIN(W1=2000%W2=2000%W3=2000%W4=2000%W5=2000%W6=2000)
|_     T1(R=Y%DF=N%TG=FF%S=O%A=S+%F=AS%RD=0%Q=)

I should also make it clear that p0f.nse don't sends any crafted packets (it only openes connection using standard connect()). But os.nse sends at least 10 crafted packet to every open port. That's not very stealthy method.


The sources are in my svn:

svn co svn://svn.insecure.org/nmap-exp/majek04/nmap-6861 nmap-majek

Trinity choice: nsock over libevent

Everybody knows that Trinity is using nmap.
But what she has to nsock or libevent, well nothing.

Recently I wrote some software using libevent and I'm disgusted. Libevent is poorly documented (except the self-explanatory function names which are suggestive), there are not many examples on the web. Even though I took an effort and wrote few programs.

About a year ago I put my hands on nmap project, I wrote something for nmap's asynchronous networking library nsock.

There are some really interesting differences between these two libraries:

• In libevent you must allocate memory and set event structures by hand, from manual: "It is the responsibility of the caller to provide these functions with pre-allocated event structures.". In nsock event objects are managed by library.
• I thought that EV_PERSIST events should be persistent. I don't get exactly the idea of EV_PERSIST event type in libevent. It's not working for me when I think it should (for example for EV_WRITE). On the other hand nsock doesn't provide any kind of persistent events.
• Libevent provides the api for queuing signals. I haven't got an idea what application could theoretically take a use of such feature. I can't think of any use case for queuing signals in real world applications.
• Nsock executes callback function with the information about a state of an event like if it has succeeded, failed or timeouted. Libevent doesn't. User have to check the socket status by hand.
• In Nsock user can get current time (the time after last select() in main event loop) without syscall using nsock_gettimeofday(), which can speedup program. In libevent you have to use gettimefday().
• Libevent supports multiple polling methods, most notably epoll(), while nsock is using only slow select().
• Libevent could be used for both client or server side. Nsock is build especially for client side, it's possible to abuse it and use it for server side, but it's more like hacking.

The thing I like most in nsock is the easiness of using it. I don't have to worry about event structures. I have the guarantee that sooner or later my callback function will be called. To show the difference I have an example. It's the simplest program I could think of. It registers timer callback with null timeout, million times one after another sequentially.

First libevent example:

#include <stdlib.h>
#include <event.h>

 /* 
 gcc -Wall event_test1.c -levent && time ./a.out
 */

int counter = 0;
struct timeval tv = {0,0};
struct event ev;

void timer_handler(int fd, short event, void *ud){
    event_add(&ev, &tv);
    counter++;
    if(counter > 1000000)
        exit(0);
}

int main(){
    event_init();
    
    event_set(&ev, -1, EV_TIMEOUT, timer_handler, NULL);
    event_add(&ev, &tv);
    event_loop(0);
    return(0);
}

And very similar nsock code:

#include <nsock.h>
#include <stdlib.h>
 
 /* Sorry, some magic with compilation. Nsock isn't standalone library yet! 
 gcc -Wall nsock_test.c -I/home/majek/nmap/nsock/include /home/majek/nmap/nsock/src/libnsock.a -ldnet /home/majek/nmap/nbase/libnbase.a /home/majek/nmap/libpcap/libpcap.a && time ./a.out
 */

nsock_pool nsp; /* global */
int counter = 0;

void timer_handler (nsock_pool nsp, nsock_event nse, void *ud){
    nsock_timer_create(nsp, timer_handler, 0 /*ms*/, NULL);
    counter++;
    if(counter > 1000000)
        exit(0);
}

int main(){
    nsp = nsp_new(NULL);
    nsock_timer_create(nsp, timer_handler, 0 /*ms*/, NULL);
    nsock_loop(nsp, 1000000);
    return(0);
}

Times of execution this programs are very interesting for me. Unbelievably it seems that nsock is more than two times faster!
Libevent averate timings:

real    0m2.271s
user    0m0.640s
sys     0m1.624s

Nsock average time:

real    0m0.941s
user    0m0.392s
sys     0m0.548s

The "user time" shows that libevent implementation of internal structures is much heavier than nsock. Even that nsock should have more work because it's allocating event structures alone, without users help. The "sys time" should be similar for both libraries, but it's not. Strace is going to reveal the issue:

# The loop is reduced to 100 iterations.

# Data for NSOCK
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
   nan    0.000000           0       203           gettimeofday

# Data for LIBEVENT
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
   nan    0.000000           0       406           gettimeofday
   nan    0.000000           0       101           epoll_wait
   nan    0.000000           0         1           epoll_create
   nan    0.000000           0         1           epoll_ctl

Well, it seems clear that two things are broken in libevent. First, the gettimeofday() is called four times for every event loop iteration, it's unacceptable waste of resources. Second, the epoll() is executed for every event loop, while for nsock even a single select() isn't called. In this particular case, when timer events are registered with Null timeout there isn't a need of executing epoll().

Summary:
From programmers point of view, I like nsock programming style. Unfortunately nsock doesn't have many important features, like epoll or server side sockets handling.

It seems that libevent ain't no perfect either.

Maybe there is a need of async library with easy api like nsock and full-featured like libevent?

Libevent under python

So I want to write asynchronous tcp/ip server in python.

I really hate overblown twisted. The thing I like most in Python is simplicity and easiness to read. Well, twisted in my opinion doesn't have this attributes. It's of course my personal feeling, mostly because I don't know twisted. But such statements are making me sick, at first sight I really don't understand what are they for (from core twisted documentation):

internet.TCPServer(7779, IFingerFactory(f)).setServiceParent(
    service.IServiceCollection(application))

I remember such cascades from Java rather than Python and I dislike Java especially for this way of programming.

Getting back to point. If not twisted than what? Lets try libevent. Stable C library, which is a production standard now. Memcached is using it, Tor is using it, IO also. It must be really professional asynchronous library.

There are also some other async linux libraries, like liboop. Wait a moment, libevent 500k google hits, liboop 25k. Sorry, the only asynchronous library for linux is libevent for now.
(btw. Liboop's api is in my opinion totally broken)

Are there any python wrappers for libevent? Google says about two: libevent-python and pyevent. Both seem abandoned.
I don't like exceptions inside libraries, so it didn't took long to eliminate pyevent:

    Traceback (most recent call last):
    File "./tights.py", line 37, in 
        main()
    File "./tights.py", line 34, in main
        event.dispatch()
    File "event.pyx", line 262, in event.dispatch
    TypeError: exceptions must be strings, classes, or instances, not type

Update: Michael Carter writes in comments that this bug is the fault of pyrex rather than pyevent and that it shouldn't happen in python2.4(I use 2.5). I'll have to take a closer look at pyevent again.

Libevent-python seems to be working. Unfortunately this wrapper removed the concept of 'userdata' passed to callback in favor of class-based approach. The difference is that callback is not called with 'userdata' but it's called on specific class instance, so callback has 'self' object. I personally prefer the 'userdata' approach, but it's quite easy to switch.

Here's an example of simple libevent-python server, which is just printing every data it receives (something like standard echo_server.py example from libevent-python, but simpler)

#!/usr/bin/python
# -*- coding: utf-8 -*-
import socket, signal, libevent

def callback_interrupt(signum, events, event_obj):
    libevent.loopExit(0)

def callback_onconnect(fd, events, event):
    # hack to avoid passing sock_srv from main()
    sock_srv = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)
    sock, (host, port) = sock_srv.accept()
    conn = Connection(sock, (host, port))

class Connection:
    def __init__(self, sock, addr):
        self.sock = sock
        self.addr = addr
        self.sock.setblocking(False)
        libevent.createEvent(sock, libevent.EV_READ, self.callback_onread).addToLoop()
    
    def callback_onread(self, fd, events, event_obj):
        buf = self.sock.recv(4096)
        if not buf: # just disconnect
            self.sock.close()
            return
        # Yeah, print!
        print "%r %r" % (self.addr, buf)
        # reuse current event
        event_obj.addToLoop()

if __name__ == '__main__':
    # bind.
    sock_srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock_srv.setblocking(False)
    sock_srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock_srv.bind(('0.0.0.0', 1111))
    sock_srv.listen(5)
    
    libevent.createSignalHandler(signal.SIGINT, callback_interrupt).addToLoop()
    libevent.createEvent(sock_srv, libevent.EV_READ|libevent.EV_PERSIST, callback_onconnect).addToLoop()
    libevent.dispatch()

UPDATE #1:
Here seems to be another way of connecting libevent with python, using raw ctypes: http://william-os4y.livejournal.com/3718.html.

Website loading time debugging (what happens on the wire when you browse?)

One of my responsibilities is optimizing page loading time. There are some very useful tools on the web that help understand where are bottlenecks.

• Pigdom Full page test
• www.websiteoptimization.com

Firefox addons like Firebug with Yslow and Tamperdata are the most important and easy to use tools for every web developer.

On the other hand I see a lack of a tool that is more related to networking. It's fine that yslow can suggest me that caching headers are broken, but how can I test if a content is really cached.

I would like to see a tool that looks more or less like Firebug networking tab, but shows what is really going on on the wire. I would like to know all "If-not-modified" requests or if "Etags" are used. There is also tcp/ip layer that is sometimes forgotten. Are "Keep-alives" used, is establishing tcp/ip connection fast?

That's why I'm thinking of a new tool, nothing revolutionary, just useful tool for me. I'm currently writing the proof of concept. Sample screenshot from my work:

What happened to tcp flag URGENT, MSG_OOB and SIGURG?

Nobody today uses tcp urgent mode, so it's good topic to make some research on.

Usually when socket receives tcp packet with URG flag it treats it as normal tcp data. recv() is going to read urgent data as it was normal tcp stream. The only difference is that the last byte of data is discarded. The last byte in urgent data was always a problem due to incoherent rfc.

Pseudocode for this case:

server: send("ab", MSG_OOB)
client: recv() -> "a"

Tcp urgent packets whould trigger SIGURG for process that's listening on socket. It doesn't work until we set process pid of socket owner (thanks Michał for this tip).

if (fcntl(sd, F_SETOWN, getpid()) < 0) {
    perror("fcntl()");
    exit(-1);
}

The interesting thing is that after we enabled SIGURG it's possible to send a signal to remote process without sending any valid data. Such situation occures when process receives tcp URG packet with one byte payload. The only byte is discarded, no data is waiting on a socket, but SIGURG is sent to a process.

client: signal(SIGURG, handler);
client: fcntl(sd, F_SETOWN, getpid());
client: poll([sd], 1, 1000);
server: send("a", MSG_OOB);
client: signal SIGURG is received by a handler
client: poll ends, because of received signal
client: but there aren't any data waiting on a socket

That's all. Feel free to use draft source code of client and server I used for testing.

Is it possible to abuse icmp?

I wonder what's going to happen when malicious user will inject crafted icmp packet with some error information (like Port Unreachable) to tcp connection. Will the connection be closed? You may think that to inject something like this it's needed to guess sequential numbers. That's not exactly correct. Icmp payload can be relatively small, specification says it has to carry at least 8 bytes from OSI 3th layer, so only tcp source and destination port must be included.

I can assume that you're going to use google. To block your connection I have to only send 65K packets with payload: from google, to you, from 80port to, and sequentially select your local port. I should guess after while. That's even less than 65K possibilities, on standard linux there are less than 32K possible local ports (see proc parameter ip_local_port_range).

The question is how common tcp stacks are interpreting such crafted Icmp message.

There are also other possibly vulnerable applications. For example most dns servers, to make recursive queries use source port 53 (see bind 'query-source address' option). How dns servers react for icmp errors with relatively small payload? Is it possible to perform Dos attack using this method?

UPDATE #1:
Well, of course port number in tcp header uses only 16 bits. So to 8 bytes in icmp payload should fit: source, destination port, seq number.

I change my question. How common tcp stacks interpret icmp packets with too short payload?

UPDATE #2:
Linux seems to be immune for this kind of attack. The function that parses icmp errors for tcp transmission is in linux/net/tcp_ipv4.c:tcp_v4_err(). The code says, that payload must be at least 8 bytes long. Source and destination port in the payload must be correct and sequence number must be in correct range. In other case icmp packet is simply dropped.

Lsi Logic site sql procedure

Anonone's interested in sql procedure used on Lsi Logic main website?

Magia w tcp/ip linuxa. Bug czy ficzer?

Z tcp/ip dzieje się coś dziwnego, po odpaleniu skryptu, pierwsze dwa połączenia są poprawnie nawiązane. Natomiast system nie widzi trzeciego połączenia. Syn-ack po sieci śmiga, a jądro z jakichś powodów go nie zauważa.
Netstat pisze:

tcp        0      0 127.0.0.1:60885         127.0.0.1:8081          ESTABLISHED
tcp        0      0 127.0.0.1:8081          127.0.0.1:60885         ESTABLISHED

tcp        0      0 127.0.0.1:60886         127.0.0.1:8081          ESTABLISHED
tcp        0      0 127.0.0.1:8081          127.0.0.1:60886         ESTABLISHED

tcp        0      0 127.0.0.1:60887         127.0.0.1:8081          ESTABLISHED
tcp        0      0 127.0.0.1:8081          127.0.0.1:60887         SYN_RECV   

Taka magia, że jest SYN_RECV w jedną stronę a ESTABLISHED w drugą. Kiedyś posiedzę nad tym dłużej i wyjaśnię :)

Oto jak zreprodukować to dziwne zachowanie:
a) Odpal skrypt.
b) Na innej konsoli uruchom

tcpdump -np -i any port 8081

Jak już powiedziałem, z trzecim połączeniem dzieje się magia.

A może ktoś zna racjonalne wyjaśnienie? Jakiś ficzer protokołu tcp/ip czy bug kernela?

POPRAWKA:
Wyjaśniło się. To jest ficzer jajka, wymyślony przez Van Jacobsena. Chodzi o to, żeby w przypadku gdy kolejka się przepełni nie odrzucać połączenia. Host będzie się dobijał dłużej, i możliwe że za którymś razem będzie miejsce dla niego w kolejce.

Ja bym to zaimplementował po prostu dropując pierwszy pakiet SYN, lecz widać z jakichś powodów dropują dopiero ACK.

Aby wyłączyć takie zachowanie można użyć:

 echo 1 > /proc/sys/net/ipv4/tcp_abort_on_overflow