Concurrent programming: It’s not about the language, it’s the framework

There’s a huge discussion on the web about concurrent programming. Now we have 4-core processors and that number will double every few years. The problem is that programmers don’t know how to use multiple cpus.

There are several approaches that address this issue:

• Intel is developing a compiler that’s going to automatically parallelize software
• people from Python are developing extension that’s enabling Python to use multiple cpus using threads-like api (normal python threads use only one cpu - see GIL)
• there are many extensions to C that enable easy writing of parallel software
• Java has built-in threading support
• everyone is admiring Haskell support for multiple cpus
• some people believe that the Software Transactional Memory is the parallel processing silver bullet
  

I wonder if we need parallelization at this level at all. Maybe the next level over “one processor” is not “multiple processors” but rather “multiple machines”. Here are the strategies that are popular nowadays:

• Google map/reduce
  
• Erlang

Erlang as a language is horrible. It’s the language for really determined programmers, because the learning curve is so steep. But the Erlang’s framework is excellent. You can easily scale over many machines, using the Erlang message passing you can accomplish more than in thousand lines in other languages.

I believe that the Erlang framework ideas aren’t tied to the Erlang language. I’d love to have so powerful framework for other languages.

Maybe we should skip the “multiple processors” phase and learn to use “multiple machines” technologies right now.