Majek's technical blog

Ipad thoughts

2010-05-29T12:43:00.004+02:00

A quick bunch of fresh thoughts on Ipad (I had no experience with Iphone):That the machine you'd like to take with you to the toilet or to the kitchen. Laptops are just too big.Yes, one doesn't know where to tap/click. I didn't know where to tap to buy app on Appstore!I'm missing the back button very much. Maybe that will go away when multitasking comes in.The back button in Safari is very far

Memory matters - even in Erlang

2010-03-01T12:41:00.001+01:00

I played with pretty interesting bug, or maybe a feature, in Erlang VM. Read more on LShift blog.

Search in the versioned world

2010-01-07T19:15:00.002+01:00

Whatever you are using: a blog, a portal, a forum or microblogging service, search works on current set of documents.If I remove my last blog post, I expect a search to omit it in the search results. Search ought to include only the most recent state of documents. But what if you can navigate through the history of document? What should appear in the search? We are used to see only the most

Undusting the blog

2009-12-05T17:55:00.003+01:00

It's time to undust this blog a bit. Here is the list of my recent posts on LShift blog, with a word of comment:Introducing RabbitMQ-Status plugin. Well, this plugin is intended to help sysadmins understand what's happening inside Rabbit. Stupid, simple, robust: the way sysadmin stuff should be.Python Quirks. My rant about Python language. I like Python in general, but it has some dark corners

Paypal/Ebay are broken

2009-08-27T01:16:00.004+02:00

I just spent more than an hour trying to buy a thing from Ebay. I'm really pissed off. In short words: Fuck you ebay.So the story goes like that:I found an itemI clicked "buy it now". So far, so good.They want me to register on ebay. I don't want yet-another-password and yet-another-stupid-user-name. Why the hell do I need that for? I'm trying to buy a thing, not an insurance. Why do you need my

Memcached protocol is not enough

2009-06-01T16:38:00.002+02:00

For some time I'm trying to hack together a prototype of real-time-full-text-search-engine (RTFTSE!). I used a memcached-binary protocol as a communication protocol between backends and storage-nodes.This is a pretty nice protocol and in fact there are strong reasons not to use ascii memcache protocol.

The ministry of strange syscalls

2009-05-19T13:57:00.003+02:00

My favorite syscall today:$ man 2 readahead"readahead() populates the page cache with data from a file so that subsequent reads from that file will not block on disk I/O."I don't really know when should I use that, but it sounds cool. Just an implementation of prefetching on yet another layer. Wait a moment..."readahead() blocks until the specified data has been read. "I'm lost. If it blocks,

GIT is ahead of SVN!

2009-05-18T16:13:00.005+02:00

Git is the most popular DVCS right now. Google trends confirms that:But the interesting fact is that Git is for the first time ahead of its grandpa Svn:Though it's worrying that CVS is still alive, it should be dead ten years ago. Hopefully we can see a process of slow death, the end is inevitable:

QCon mini InThe Brain session

2009-03-10T15:54:00.002+01:00

I'll be speaking about simplified Etherpad clone at QCon London on Wednesday, March 11th.The presentation will take place at Skillsmatter stand (booth number 10), at 16:45.

Evserver, part3: Simplified Etherpad clone

2009-03-04T14:52:00.005+01:00

This time I hacked together few open source projects. The result of my work, except from few minor bugs, seems to be working.I don't know what shall I do with this project next. There are few possible options:throw it away and forget about itadd support for markups other than reStructuredTextimplement new features, like private documents or downloading the rendered markup in different

EvServer, part2: Rabbit and Comet

2009-02-18T00:25:00.003+01:00

sourceI just published the next part of the EvServer story.I'm working at the third part. Though, my major concern is that I'm running out of proofreaders.

EvServer, Introduction: The tale of a forgotten feature

2009-02-06T19:42:00.003+01:00

This article about EvServer doesn't really show it as an innovative piece of software. I hope that future articles will.It seems that the most important thing in this post is a blinking exclamation mark.BTW, ItBlog seems to be dead. For historical reasons I placed my old posts from that blog here, here and here.

I was wondering how Twitter search works

2009-01-22T01:02:00.003+01:00

Few days ago I committed a blog post about Twitter search.As a matter of fact, I'm not sure what is the solution to the question I asked there. On the other hand I presented some interesting (at least for me) numbers.While writing this stuff I learned a lot and maybe this blog post will be a motivation for someone to actually create such a perfect system.I'm still thinking about the persistence

Asynchronous libraries performance

2008-12-15T18:06:00.004+01:00

I committed next post on LShift's blog.BTW. This is my 100th post in this blog!

Nmap Book finally released!

2008-12-06T15:03:00.003+01:00

I'm really excited. After years of work Fyodor finally managed to finish The Book "Nmap Network Scanning". It should be a great gift for every security geek. You can preorder it on Amazon for 34$.

AMQP? Maybe not yet. At least not with Python.

2008-11-21T15:43:00.005+01:00

sourceIn my last Python project I thought about using AMQP messaging. A quick look at Python AMQP libraries shows two possibilities:Apache Qpid python clientBarry’s py-amqplibBoth libraries are developed tightly with their brokers: Apache uses Qpid, py-amqplib uses RabbitMQ.Apache QpidI started my adventure by downloading the source. They don’t have binary packages so this is what’s needed:$

How to escape from blocked unactivated Vista

2008-11-18T19:16:00.009+01:00

Few times I was pissed off by this screen:It means that I haven't activated Vista on time and Microsoft stopped liking me. The problem is that at least two times I was caught by this screen when I haven't got access to the net. I had password to WiFi, but haven't entered it yet. When Vista is blocked you don't have access to any networking settings so the password is useless. Yet another time,

Tracing Python memory leaks

2008-11-17T06:23:00.001+01:00

Tracing Python memory leaks.

Simple inter-process locks

2008-11-11T01:40:00.003+01:00

At LShift blog I wrote about simple inter process locks in Python.

Youtube - a team of highly trained monkeys...

2008-11-10T13:53:00.002+01:00

In translation "A team of highly trained monkeys has been dispatched to deal with this situation."I'm not the only one to see this message. I think it's even better than "you broke reddit" message.

Jukebox XSRF

2008-11-04T02:29:00.005+01:00

sourceIn Lshift we use Tony’s erlang jukebox, it's great, anyone can play music at the office. I found XSRF there and exploited it maliciously. Every time someone from inside the office opens this blog, Britney is being played.The exploit is not especially complicated:

Blog success story

2008-11-02T04:00:00.010+01:00

sourceThe beginningsI wrote first post in this blog 22 months* ago. My intentions weren’t clear. I wanted to document my ideas (I had a lot of them), have a place to store code snippets and maybe promote myself.First posts I wrote in Polish and didn’t really know if I will be able to write in English. When look at my first posts, I start to understand that they were just testing the territory. I

Innovation is illegal

2008-09-09T21:48:00.006+02:00

New, interesting technologies on the web are often criticized, sued and sometimes even ruined by the court rulings. There are a lot of lawsuits on new technologies.Let's bring a few important technologies that were(are?) considered illegal by some people:Google (and other search engines) - deep linking. I remember the days when people said that deep linking is not fair. There were even court

Presentation: Nmap and asynchronous programming

2008-08-28T13:00:00.000+02:00

In this presentation I would like to talk about my adventure with extending Nmap and what I learned from it regarding asynchronous programming.Nmap is a port scanner with many other features.On this slide you can see an example of Nmap output.It shows open tcp ports on some target machine. It also shows service type (if it is ssh or http or something else) and version of a daemon running behind

Presentation: hacking Industrial Robots

2008-08-26T12:51:00.001+02:00

I'll talk about my work on industrial robots. The robots belong to my university - PJIIT.Here you can see our two Motoman SK6 robots. They were bought by the Japanese government from a fund to support developing countries. The robots were produced in Japan, assembled in Sweden, serviced by Germany and located in Poland.They are created to do welding and painting in the industry. They have no

Learning Erlang

2008-08-24T16:09:00.010+02:00

I mentioned before that I'd love to learn Erlang. Just before my holidays in mountains the Erlang book has arrived. I had to chose between the holidays and learning Erlang. I chose both.It was very nice to read it on the meadows at the top of the hills. But the book is rather heavy when you need to carry it uphill.I think it's one of the best written computer books I've ever read. I've learned a

Don't you need an employee?

2008-07-11T21:30:00.006+02:00

(image stolen from that guy, he's also looking for a job)I’m currently looking for a job.What job I’m looking for? I have no idea. I did some things in my life, but that doesn’t mean I want to do them again. Most people think: “well, you’re a python hacker, so here is a python job for you!”. That’s just wrong. I want new challenges, I want to learn.There are probably a lot of things that I could

Ideas are free

2008-07-03T12:43:00.007+02:00

Ideas are free, so here’s the list of mine (previous ideas here).I still think that the next big thing is an open social network, so that anyone can write an extension to it. Something like Facebook api, but hosted on a provider side. That’s why I believe in GAE. Open and easily extendable social network is the web3.0 for me.The new idea is to create a Reddit tracker. When you submit something to

Motoman robot controlled by two Wiimotes

2008-06-15T16:53:00.008+02:00

Some time ago I controlled a robot with a Haptic.Inspired by Johnny Chung Lee I started to work on controlling the Motoman industrial robot using two Wiimote infrared cameras. It's a bit less accurate method than using Haptic, but it's much cheaper.What you need:a robot (or whatever you'd like to control)two Wiimote devicesa software (okay, here's the source, but beware, it's beta, work in

LinkedIn graph - I don’t get it

2008-06-06T11:25:00.004+02:00

At Grono.net, the network graph had more than 80M edges and about 1.2M nodes. You know how much RAM it used on the machines? 320MB. It’s a bit more than 4 bytes for an edge, because of alignment, plus few megabytes of index.LinkedIn says they need 12GB RAM for keeping their graph with 120M edges. Wow. It’s 107 bytes for an edge! I think that the graph shouldn't consume more than 0.5GB RAM.Their

Concurrent programming: It’s not about the language, it’s the framework

2008-05-29T11:54:00.004+02:00

There’s a huge discussion on the web about concurrent programming. Now we have 4-core processors and that number will double every few years. The problem is that programmers don’t know how to use multiple cpus.There are several approaches that address this issue:Intel is developing a compiler that’s going to automatically parallelize softwarepeople from Python are developing extension that’s

Finding Iris on the image

2008-05-22T20:30:00.009+02:00

During one of the classes we're supposed to find Iris on given images. I created hackish script that does it. The script isn't finished and to be frank it barely works. But I think the result images look cool.

Google App Engine tips&tricks

2008-05-12T22:11:00.004+02:00

sourceA while ago I was writing some sample applications (source) for Google App Engine. I noted the things that can be useful for other GAE programmers.I used Google's webapp framework, my code here is using it.Please take a look at the shell application, it can help you test simple code.How to dynamically get application name and version?This question was asked before. You can use os.getcwd()

Shared Fridge Magnets - simple collaboration for GAE

2008-05-11T00:08:00.003+02:00

Recently I read The ELC Community Blog with their example of sharing objects using red5.The obvious task is to do the same without red5 and any flash. So I created yet-another-example of the comet daemon service.Just open the fridge example site in two browsers. You can move, resize (with shift) and upload images.

Google App Engine: Ytalk like multiuser chat

2008-05-07T22:58:00.003+02:00

This is a follow up on my last post describing missing services for AppEngine.The idea is to help developers writing apps for AppEngine by providing them some common functionality missing and impossible to have on AppEngine and offer them as external services.These services are accessible through a simple API over HTTP, and it is easy to call them from AppEngine applications using urlfetch

Missing services for Google App Engine (comet as a service!)

2008-05-01T01:01:00.007+02:00

Google App Engine is a great product, but it lacks several features. I created few simple services to help GAE developers. Of course services aren't GAE specific, you can use them from any site.The services are:Image resizingCron serviceComet service Image resizingOne can't easily resize image on Google architecture - they blocked PIL. This service just resizes images and uploads them back to

Python is the winner.

2008-04-08T11:33:00.004+02:00

Usually I don't comment the news on the web, but this one is in my opinion really important.Google made App Engine. Now you can write web applications in Python and run them on Google infrastructore. The main advantages?ultra scalability for renderer serversultra scalability for databaseeasy deployrather cheapauto load-balancingWhat they miss?memcached / advanced cachecomet supportview layer

[pl] Bootstrap 8.4 - Tomasz Lis

2008-04-06T19:21:00.004+02:00

Oto prezentacja Tomka Lisa z sobotniego Bootstrapa "Od startupu do fakapu, czyli co z tym gronem…". Sorry za jakość, ale to wszystko co ten sprzęt potrafi. Chrząknięcia i szepty należą do Sebastiana. Update 1Jesli google video nie działa, to tu jest źródłowy wmv.Update 2Ostatnie pięć minut okazało się być czarną planszą. Teraz mam nadzieję już wszystko widać.

Motoman controlled by Haptic

2008-04-06T18:20:00.003+02:00

It's a lot of fun to control a robot with the Haptic. I like the way the robot moves.

How many web browsers do you have on your computer?

2008-03-28T22:00:00.008+01:00

(no, it's not my desktop)Simple question: how many browsers do I have?I use Opera for reading reddit and browsing, Webkit for gmail and google docs (it's really fast), Firefox for web developing (FireBug rules).I use virtual machines, so it's not so easy to count :)main operating system: opera, firefox, safari, webkitsecondary os: opera, firefox, safari(comes with itunes), webkit, internet

Why I don't buy on ebay.

2008-03-27T22:41:00.003+01:00

>> Hi, I want to ask if you provide deliver to Poland (Europe)?Only after I am certain that good funds have completely cleared my account, cannot be reversed, you agree to accept all shipping charges, you understand you accept the item "as is" and have no right to return it. Reason for this harsh approach to international shipping? Too many scams coming from your location!!!

Useful C extensions (gcc specific)

2008-03-26T22:09:00.007+01:00

You think that knowing ANSI C is enough? Well, okay, it's enough. But I found that some gcc extensions are very useful.All gcc C extensions are listed in the manual.Here are the extensions I use:Labels as valuesI remember the old assembler days, when you could jmp to to an address from an array. Nowadays in C, you can create an array of pointers to functions and call pointer from array. But jmp

Emulating exit(3) and moving stack on the fly.

2008-03-25T23:56:00.003+01:00

Recently, during coding embedded stuff for industrial robots I had very interesting problem.I needed to implement exit(3) function. But the system on the robot is very simple. There aren't any kinds of syscalls.So, the functions exit(3) must emulate the only proper way of exiting the program. It should emulate returning/exiting from main() function.My friend, Michal Luczaj, found clever way of

Web applications? Think again.

2008-03-21T11:01:00.003+01:00

Some time ago I proved that writing comet applications can be very easy.Now I think, I can put almost anything online. I created chat, I have working proof of concept of ytalk-like web application. I'm thinking about putting Python interpreter online. These are rather standard web applications, someone did it before. What's the next step of "evolution" after comet?Ideas are free, so I can speak

Bidirectional search example

2008-03-11T11:59:00.003+01:00

Dorian asked for implementation of bidirectional search algorithm. So I created very simple implementation of bidirectional search, not optimized at all. But I hope this is enough to understand the algorithm.>>> from bidirectionalsearch import *>>> bidirectional_search(graph,1,3)[1, 2, 3]>>> bidirectional_search(graph,1,8)[1, 2, 3, 4, 7, 8]>>> bidirectional_search(graph,1,9)[]>>>

Why you should care about synchronous blocking I/O syscalls?

2008-03-08T11:42:00.003+01:00

In previous post I noticed that open(2) and stat(2) aren't asynchronous. But why you should care?Try to view a html file from CD using your favorite browser. I don't know how about you, but I switch to other browser tab while a site is loaded. But wait, when the drive starts to spin the CD the browser is blocked?That's because the browser did an open(2) on a file from CD and system have to load

Caching at grono

2008-03-08T11:19:00.002+01:00

I posted article (in polish) about caching at Grono, with description of the caching decorator called memoize.It's a very simple decorator, but it's working perfectly. Without it the site wouldn't be working :)

How to deliver an event to multiple processes? - scaling django-evserver

2008-03-08T10:57:00.002+01:00

The hardest problem I had to solve during my work on django-evserver, was how to efficiently propagate an event to multiple listener processes.Python, because of GIL, doesn't scale across multiple processors. One process can only use one processor.In django-evserver we have multiple python processes and some of them could wait for some specific event. The question is how to deliver an event

Nmap - os detection for every open tcp port

2008-02-29T12:22:00.003+01:00

While ago I created tool, that shows p0f-like signatures for every open port. This signature is based on information taken from single SYN-ACK packet from target port. Sample output:$ sudo ./nmap -n -sT -PS80 -p21,22,53,80,443 --script=p0f.nse www.cisco.comStarting Nmap 4.22SOC1 ( http://insecure.org ) at 2007-07-12 00:04 CESTInteresting ports on 198.133.219.25:PORT STATE SERVICE21/tcp

NSE loop bug

2008-02-27T00:23:00.006+01:00

Nmap Scripting Engine is an engine for running Lua scripts inside Nmap. It's based on Nmap asynchronous library NSock. During work on NSE script I found that sometimes Lua threads were frozen for too long. For example socket_object:receive() should take few milliseconds, but it took more than a hundred. In normal usage I wouldn't even notice that events are delayed, but this time I needed exact

django-evserver - Asynchronous server for Django

2008-01-18T22:45:00.000+01:00

I mentioned the idea in previous post about asychronous wsgi. Now we have working example of simple chat application. The chat is nothing special, but it's made in Django! To make this possible I created special server, django-evserver. It's quite light wrapper to libevent http layer (several hundred of lines in python/ctypes).Using this technology it's now possible to create asynchronous WSGI

32 or 64 bits?

2008-01-17T20:53:00.000+01:00

Just a note, that I posted an article about advantages of 32 bit architecture over 64 bit and why we chose 32 bit at Grono.net. It's in Polish, but the main point is that in our case Django at 32 bits is consuming about 30% less memory than on 64 bits.To show the differece, here is output from top command at 64 bits:VIRT RES SHR S %CPU TIME+ COMMAND212m 96m 3832 S 0 1:49.61 django '/

Nmap NSE-pcap example

2008-01-14T12:01:00.000+01:00

At nmap-dev mailing list Lionel asked me about NSE-pcap. He wanted to send raw UDP packet and capture the response.I created an example script that does this.The patch against nmap.Or if you prefer already patched files:packet.luapcap-example.lua

Is 8GB RAM faster on 64bit or on 32bit processor?

2008-01-12T11:08:00.000+01:00

A friend asked me if 8 gigabytes of ram is much slower on 32bit than on 64bit linux. The standard 32 bit can handle at most 4GB of ram. To access more ram you have to enable the CONFIG_HIGHMEM64G option in kernel. The question is if this feature make access to memory much slower.I created basic test to measure ram speed. The program sequentially writes to rather big memory chunk. After first

Asynchronous Django responses (comet), yes, for WSGI

2007-12-16T23:46:00.000+01:00

Comet on python?I was looking for comet server implementation in python. There are many projects that claim to provide this, but none of them is working for me. Most of the projects are just proofs of concepts that never been deployed for real usage.Here is the list of projects that my friend found:fapws (based on libevent): http://william-os4y.livejournal.com/athena: http://divmod.org/trac/wiki/

PLUG: Warsaw python meetings

2007-11-28T13:32:00.001+01:00

My friends are going to have python presentations in Warsaw. It's going to be a part of PLUG meetings.Thursday, 29th November 2007, 19:00 at Politechnika Warszawska, sala AL.More information:http://warszawa.linux.org.pl/nowe.php?date=1196359200http://zgodowie.org/2007/11/26/war-py/

Motoman weekly news #1

2007-11-15T23:49:00.000+01:00

If you're interested in my work with Motoman industrial robots this post is for you.The first thing is that this photo isn't connected to this post. It was taken by Marianek when the left Motoman got, out of my controll for a while. But getting back to point...We have suffered from serious errors on right robot, mostly with the R axis. I suggested that the fault could be because of a broken

Trinity choice: nsock over libevent

2007-11-02T22:07:00.000+01:00

Everybody knows that Trinity is using nmap.But what she has to nsock or libevent, well nothing.Recently I wrote some software using libevent and I'm disgusted. Libevent is poorly documented (except the self-explanatory function names which are suggestive), there are not many examples on the web. Even though I took an effort and wrote few programs.About a year ago I put my hands on nmap project, I

Libevent under python

2007-11-02T20:19:00.000+01:00

So I want to write asynchronous tcp/ip server in python.I really hate overblown twisted. The thing I like most in Python is simplicity and easiness to read. Well, twisted in my opinion doesn't have this attributes. It's of course my personal feeling, mostly because I don't know twisted. But such statements are making me sick, at first sight I really don't understand what are they for (from core

Website loading time debugging (what happens on the wire when you browse?)

2007-10-29T23:49:00.000+01:00

One of my responsibilities is optimizing page loading time. There are some very useful tools on the web that help understand where are bottlenecks.Pigdom Full page testwww.websiteoptimization.comFirefox addons like Firebug with Yslow and Tamperdata are the most important and easy to use tools for every web developer.On the other hand I see a lack of a tool that is more related to networking. It's

Set process name in python

2007-10-26T00:00:00.000+02:00

How to set process name from inside it? I know that nmap can fake name, but the question is how to do this from python.The code that is doing this magic in nmap:# from nmap.cc if (quashargv) { size_t fakeargvlen = strlen(FAKE_ARGV), argvlen = strlen(argv[0]); if (argvlen < fakeargvlen) fatal("If you want me to fake your argv, you need to call the program with a longer name. Try the

The mystery of extremly long "grep -i"

2007-10-22T21:36:00.000+02:00

It's well known fact, that grep is the fastest way of searching data. The mystery was why "grep -i" took so much longer than standard, case-sensitive grep. Finally depesz found the reason.

GNU pth instead of pthread: hardcore python tuning

2007-10-22T16:26:00.000+02:00

ProblemI've been working on speed of an application written in python. While doing strace I found out that the most of syscalls were futex(). This futexes came from python's internal synchronization code. But the application never used any kind of threading!Most of the users never actually realize that python is doing synchronization between threads even when threads aren't used. It's not so bad,

Interesting gettimeofday() on x86_64

2007-10-20T21:20:00.000+02:00

During doing strace() on application I found that there are no gettimeofday() syscalls. That's nothing new, for x86_64 experts, but it's interesting for me.Here is an example of the code:#include int main(){ printf("start\n"); gettimeofday(NULL,NULL); printf("stop\n"); return(0);}Sample strace from i386:...write(1, "start\n", 6) =

Finding paths between users in social network graph.

2007-10-16T18:06:00.000+02:00

For some time I'm working at Grono.net the biggest Polish social networking website (over 1.3mln registered users). Like in other networking sites, our users have the list of friends which they know. We have interesting feature of finding the shortest path between two users, measured in "handshakes" (by "handshake" I mean connection between two users).The problem with that "paths" service was

Die Google! - future search engines

2007-09-19T20:53:00.000+02:00

While talking with a friend, I realized that Google isn't the last word in the web search engines.From technical point of view google search engine isn't magic, the topic of indexing data is nothing revolutionary.The only interesting thing about Google success is their method of promoting good results and punishing bad in search results. In Google nomenclature it's called the (mistified)

Graphical representation of connections between users

2007-09-13T16:17:00.000+02:00

Recently, I've been looking for a way to graphically show connections between users in social network. The basic idea is to show it in 3d, in the simplest possible manner.I tried to code this, but unfortunately I didn't knew any good-enough algorithm for counting the scene.Other social networks tried to visualize connections. For example this graph was created at 2003 Club Nexus at Stanford

Magic popcount (popcnt) command

2007-09-08T01:16:00.000+02:00

From Frank de Groot blog:Every serious hacker sooner or later needs the popcount instruction.This "population count" instruction counts the set bits in a register, and is so useful that the NSA demands that all computers they purchase implement it in hardware.But this command is not present at x86 architecture. AMD64 documentation (p. 188) supports popcount extension, but I never heard of any

Python memory hack

2007-09-07T23:07:00.000+02:00

Django and most of python frameworks eat a lot of RAM. To serve multiple pages in the same time there must be many running python processes (this is not like light www daemons which serve all requests from only one process, like lighttpd or nginx).So every python process eats RAM. During it's live it imports many libraries and files. This imports are never freed by garbage collector. This imports

Crisis with asynchronous programming

2007-09-04T23:42:00.001+02:00

I'm currently designing an application. It's going to be quite simple linux network daemon which will serve some data from disk.Looks simple. I can easily predict that the bottleneck can be either disk IO or network bandwidth. Nowadays it's obvious that high performance network daemon must be done 100% asynchronous. The problem is that there isn't yet any technology that allows full asynchronous

Nmap nse descriptors overflow patch

2007-08-30T21:26:00.001+02:00

I just released quick patch for Brandon bug. It's interesting that this flaw wasn't found earlier.My patch is a bit hackish, but it should work. I wonder if anyone finds better way of fixing this issue. The descriptor overflow patch lies here.Update 1: Hurray! Patch is included in svn version of nmap (r5739).

Well, we need Google architecture, but how should we implement such a huge project?

2007-08-28T21:56:00.000+02:00

Every fast-growing company will have to answer this issues sooner or later:storage disks are fullit's hard to add more disks to existing systemsfiles are served too slow because disks seeks are becoming limitationsimple database is growing to millions rows, cost and complexity of partitioning is reaching the level of cost-effectivenessIt seems that Google architecture is solving this issues. By

Some thoughts on hashing

2007-08-26T18:51:00.000+02:00

I was looking for fast hash algorithms and I found Paul Hsieh SuperFastHash implementation. I also found that hash functions in Bloom filter algorithm are used to test if an element is member of a set.I found interesting fact, that it's possible to count any number of independent hashes using only two hash functions using this formula (f1 and f2 are hash functions, i is any number that is

How a bad password generator can ruin security

2007-08-13T22:56:00.000+02:00

This is a true story about really broken password generator that was used at my college. The good idea and a bad implementation.On 30th of December 2005 all students at PJIIT received email which sounded like this:From today (30.12.2005) new password policy is going to be used:Password must contain eight or more charactersPassword must not contain username or any part of itPassword should contain

What happened to tcp flag URGENT, MSG_OOB and SIGURG?

2007-07-21T14:50:00.000+02:00

Nobody today uses tcp urgent mode, so it's good topic to make some research on.Usually when socket receives tcp packet with URG flag it treats it as normal tcp data. recv() is going to read urgent data as it was normal tcp stream. The only difference is that the last byte of data is discarded. The last byte in urgent data was always a problem due to incoherent rfc.Pseudocode for this case:server

Peek at new nmap-nse scripts

2007-07-18T03:07:00.000+02:00

Well. I'll give you a peek at my new scripts for nmap. This scripts aren't public yet, I hope this post will give mi motivation to finish them.This time we're going to focus on traceroute. New traceroute function in nmap looks like this:# ./nmap -n -sS -p80 scanme.insecure.org --tracerouteTRACEROUTE (using port 80/tcp)HOP RTT ADDRESS1 0.33 (censor)2 7.93 (censor)3 ...4 7.84

Is it possible to abuse icmp?

2007-07-15T01:41:00.000+02:00

I wonder what's going to happen when malicious user will inject crafted icmp packet with some error information (like Port Unreachable) to tcp connection. Will the connection be closed? You may think that to inject something like this it's needed to guess sequential numbers. That's not exactly correct. Icmp payload can be relatively small, specification says it has to carry at least 8 bytes from

Some random Nmap ideas

2007-07-12T00:42:00.000+02:00

There were some interesting projects that were using nmap. For example Doug Hoyte Qscan idea. I hope his work won't be forgotten.But there are also some interesting projects on my desk.I think there is a possibility to implement Lcamtuf 0trace as nse script. This could be an add on to my p0f script. In 0trace we're of course interested in last hops only because standard traceroute was already

Pcap support for Nmap Script Engine

2007-07-11T23:22:00.000+02:00

Some time ago I wrote nse-pcap patch for Nmap that adds some libpcap features to Diman's NSE. Today I ported changes to current Nmap. It's definitely time to check if the code is worth time I spent on it!I think that pcap support one of the most promising features in NSE. It gives "new experience" to advanced programmers. It also introduces easy to use and powerful way of distributing received

Filesystems in 21st century

2007-07-10T01:50:00.000+02:00

At studies I was taught that there are three levels in theoretical filesystems:File name is tightly bound to the volume where file is stored (eg: on Microsoft Windows the path contains drive name, like c:\windows).File name that doesn't say anything about place. But specified directory can be mounted as disk (in unixes; you don't know on which disk file lies, but you know that every file in some

Lsi Logic site sql procedure

2007-07-03T10:21:00.000+02:00

Anonone's interested in sql procedure used on Lsi Logic main website?

My theory about the future of the web

2007-06-28T16:13:00.000+02:00

The shape of today's network is rapidily changing. First there was dot-com bubble, now we have Web 2.0 and „user generated content" era. It's time to wonder what's going to be next.I'm not the first one to mention that, but I think that in the nearest future we'd see User Generated Portal. I can imagine users that are programming features they need in portal, and share this code with others.

Social networking: shortest path between users

2007-06-28T00:57:00.000+02:00

Description of my algorithm for finding shortest paths in social network portal. It reduced search times to several milliseconds.

Introduction to Motoman industrial robots

2007-06-25T14:36:00.001+02:00

At robotics in PJIIT we have two Motoman SK6 industrial robots. Here are movies with robots doing sample jobs.Left robot with job called "bottle". The robot was intended to get bottle from table and shake it several times. It almost works...The same job, but view is from camera on robot arm. Right robot with my favorite fast-speed test

Motomany - podnoszenie kostki

2007-06-25T13:59:00.000+02:00

Projekt ma już dwa lata, ale może czas go przypomnieć.Robot miałby automatycznie znajdować Kostki w okolicy, podnosić je, i coś z nimi dalej robić. Na przykład układać w piramidkę :)Największą częścią projektu jest rozróżnienie kostki od otoczenia. Aby to uprościć oznaczyłem kostki specjalnymi markerami. Oto wczesny etap prac nad markerami. W oknie po lewej części widać obraz z kamery wraz z

Jak ustawić tablicę integerów na zadaną wartość?

2007-06-02T13:11:00.000+02:00

Mam proste zadanie. Ustawić tablicę kilku milionów intów na podaną wartość. (kompilaor gcc, procek p4/xeon)Brzmi banalnie. Pierwszy kod:void fill_standard(int bit_field, int *buf, int nodes_length){ int uid; for(uid=0; uid < nodes_length; uid++) *buf++ = bit_field;}Problem taki, że to strasznie długo trwa. W przypadku 16 milionów integerów zajmuje to 87ms.No to może by to

Ile to jest -2147483648 * -1?

2007-05-31T23:23:00.000+02:00

Ile to jest -2147483648 * -1 według Twojego 32 bitowego komputera?int main (){ int x; x = INT_MIN; /* czyli -2147483648 */ printf("%i\n",x); x = x / -1; printf("%i\n",x);}Mój komputer uważa, że to jest -2147483648. I kto mówi że komputery się nie mylą...

Nmap na OLPC?

2007-05-23T09:40:00.000+02:00

Wygląda na to że Fyodor chce odpalić nmapa na laptopach za $100. Ciekawe czy to się uda.Trzeba będzie pewnie trochę poprawić obsługę wifi, zmniejszyć zużycie ramu...

w Sejmie o społeczeństwie informacyjnym

2007-05-22T19:36:00.000+02:00

Ze stronki vagli:"Nazywam się Piotr Waglowski, mam 33 lata, jestem internautą. System prawa w moim kraju uznaje mnie za przestępcę... Ale nie chcę stąd wyjechać, bo to mój kraj. Co zatem możemy zrobić, by nieco poprawić tę sytuację? Zdaję sobie sprawę, że naturalną reakcją po wysłuchaniu tego typu wystąpienia jest milczenie..."Trafny jest jeszcze komentarz Jedynej:"Nieprzejmowanie się weszło mi w

Kolejna zagadka

2007-05-15T10:01:00.001+02:00

Can you find the next number in the sequence: 1, 11, 21, 1211, 111221, 212211 312211., …

Magia w tcp/ip linuxa. Bug czy ficzer?

2007-04-26T16:22:00.000+02:00

Z tcp/ip dzieje się coś dziwnego, po odpaleniu skryptu, pierwsze dwa połączenia są poprawnie nawiązane. Natomiast system nie widzi trzeciego połączenia. Syn-ack po sieci śmiga, a jądro z jakichś powodów go nie zauważa.Netstat pisze:tcp 0 0 127.0.0.1:60885 127.0.0.1:8081 ESTABLISHEDtcp 0 0 127.0.0.1:8081 127.0.0.1:60885 ESTABLISHEDtcp

Web 2.0

2007-04-19T23:43:00.000+02:00

Czym właściwie jest web 2.0? Spotkałem się z kilkoma definicjami.Web 2.0 to serwisy społecznościowe, gdzie każdy użytkownik może wpływać na prezentowaną treść. Taka definicja jest chyba najbardziej popularna.Podejście techniczne, gdzie aby utrzymać serwery portali społecznościowych potrzebne są zupełnie nowe rozwiązania dotyczące skalowania serwerowni. Web 2.0 w tym kontekście to nie ludzie i

Nmap nse-pcap, próba #2

2007-04-19T01:02:00.000+02:00

Właśnie skończyłem testować patch do nmapa, o roboczej nazwie nse-pcap.Zamiast opisywać jego działanie powiem tylko, że jest po prostu zajebisty.Wkrótce postaram się opisać jego możliwości.

Python wrappers for memcache

2007-04-18T09:56:00.000+02:00

Jeśli szukasz modułu do pythona, którym połączysz się do memcached, to najprostszym będzie python-memcached. Jest on prosty, stabilny, i po prostu działa.Problem zaczyna się później w memcache.py:_Host.readline() def readline(self): buffers = '' recv = self.socket.recv while 1: data = recv(1) [...]Tak, tak. python-memcache czyta z socketa po

Keep It Simple, Stupid

2007-04-11T22:06:00.000+02:00

Cytat z bloga toomuchcode:For anyone that might be struggling with this right now, some humble advice:Follow Alan Kay's axiom: "Simple things should be simple, complex things should be possible." -- If something complicated must be in the system, it still should not affect the simple things.View the project as a constant battle against complexity -- A single complex module may seem unimportant,

Praktyki w Mountain View? Nie tym razem.

2007-04-10T21:59:00.000+02:00

Złożyłem papiery na praktyki w Google. A co mi tam, przecież każdy chce tam pracować. Po jakimś czasie (ok 1,5 miesiąca) mejlem odezwała się grzeczna pani, i ustaliliśmy termin na dwie 45-cio minutowe rozmowy.Obie rozmowy odbyły się z regularnymi pracownikami Google - w pierwszej rozmawiałem z gościem z ładnym angielskim akcentem, a podczas drugiej z facetem, którego ledwo mogłem zrozumieć, bo

Zagadka o piratach i łupie

2007-04-08T09:00:00.000+02:00

Dziesięciu piratów zdobyło skarb złożony ze stu sztuk złota। Chcą podzielić się łupem. Są, na swój własny i szczególny sposób, piratami demokratycznymi - mają zwyczaj dzielenia łupów w następujący sposób. Najgwałtowniejszy pirat proponuje metodę podziału i wszyscy głosują; każdy ma jeden głos, wnioskodawca też. Jeśli 50% lub więcej głosów jest za, propozycja zostaje zatwierdzona i wciela się ją w

Idea silniejsza niż Microsoft

2007-04-08T00:58:00.000+02:00

Dziś przeczytałem dwa ważne artykuły, polecam do poduszki:Richard Hillesley: Be Realistic. Demand the ImpossiblePaul Graham: Microsoft is dead

Zagadka "najtrudniejsza ze wszystkich"

2007-04-07T14:19:00.000+02:00

Poniższa zagadka, znana jako "Historia Smitha, Johnesa i Robinsona" to już klasyka tego typu łamigłówek.Panowie Smith, Robinson i Jones stanowią załogę pociągu. Jeden jest palaczem, drugi hamulcowym, trzeci maszynistą; kolejność tych funkcji jest przypadkowa. Pociągiem jadą również trzej biznesmeni o tych samych nazwiskach.1. Pan Robinson mieszka w Detroit.2. Hamulcowy mieszka dokładnie w połowie

Zagadka "Einsteina"

2007-04-07T14:17:00.000+02:00

Tak na rozruszanie głowy, oto pewna zagadka:5 ludzi zamieszkuje 5 domów w 5 różnych kolorach. Wszyscy palą papierosy 5 różnych marek i piją 5 różnych napojów. Hodują zwierzęta 5 różnych gatunków.- Anglik mieszka w czerwonym domu- Duńczyk pija herbatkę- Norweg zamieszkuje pierwszy dom- Palacz Rothmansów mieszka obok hodowcy kotów- Mieszkaniec żółtego domu pali Dunhille- Niemiec pali Marlboro-