The Bazaar & the Mosque
All open source people (should) know the “The Cathedral & The Bazaar” book by Eric S. Raymond published many many years ago. In this paper he presented the difference between closed source (proprietary) “cathedral” development model and the Linux-like “bazaar” one. In the bazaar model you are supposed to have freedom to pick any “free / Open Source Software (FOSS)” and use it. Sounds good but does this have anything to do with reality? No.
Note that I’m talking about “ordinary” users who don’t know how to compile, install and hack software packages downloaded from the net. Power users have the required skills to do whatever they want. However how many typical computer users have ever heard aboutany computer programming languages? Typical end users can only use the services that are already available in their computer. In the best case they can install some precompiled applications if the installation procedure is easy enough.
In the Linux bazaar you can visit any of the shops and pick the Linux distribution provided by that vendor. However this is the last moment you can make any choises. After you have installed the Linux distribution you have got your freedom is gone.
If all the devices in your system are not supported by the distribution then you will be more or less out of luck. If the Linux version doesn’t support your chipset, network adapter, IDE/SATA chip and other critical hardware components they you will be out of luck. There is no way to get the Linux distribution you have selected to work in your system. You cannot install any additional drivers from the driver CD that comes with the system because there are no Linux drivers. You can only hope that some other Linux distribution supports your hardware or that the support will be included in the next update.
Even if you manage to get Linux installed you will be limited to the applications supported by it. You cannot install any 3rd party applications because you would have to compile them yourself which is beyond your skills. If you go and buy the very latest scanner (or whatever) device then it will not work before your Linux vendor releases a kernel update that supports it. You can possibly download the source code for your hardware from it’s manufacturer’s web page but will you be able to compile it? It depends on your skills and also on the freshness of the driver code (Linux kernel may have been changed between the driver and the kernel version you have). In any case you will have to install the driver development environment and tools for your distro (which may mean 100’s of megabytes of disk space.
How does this work in Windows?
When you install Windows you have an opportunity to install vendor provided drivers. You can install the drivers from the install CD that comes with your motherboard. You can also do the same with the install CD’s of your network and SCSI cards. Then you proceed to install Windows itself and the result will be a fully functional system.
Can you do this with Linux?
The answer is no. When you install Linux (any distribution) you are not given any chance to install 3rd party drivers when you install Linux or even after that. You can install 3rd party drivers to Linux if you are a real hacker with years of Linux kernel experience. However for any “ordinary” Linux users this option is out of reach.
Is the bazaar really a bazaar?
After you have installed a Linux distribution (if you have managed to do so) you will become dependent of the support provided by the vendor of the distribution. You can install important updates released by the vendor or to upgrade to the later versions of the distribution. However if the vendor doesn’t see it necessary to support hardware or software you would need then you will be out of luck.
Experienced Linux hackers can always download whatever software they need and try to compile it themselves. However often they will end in never ending battle with differences between various kernel and library versions. The device driver interface of Linux seems to change between versions. This means that 3rd party drivers may require modifications before they compile, install and work with different kernel version. Even user land applications often depend on newer/older versions of various libraries than the ones provided by the distribution. You will have to download and recompile different versions of different libraries and to recompile them. Sooner or later you will be in a situation where some other applications stop to work because of wrong libraries. So to stay on solid ground you have to stick with whatever software/versions your Linux distribution vendor sees necessary to support.
Do you think this is the Bazaar? Are you sure you have not visited a Mosque instead of a Bazaar?
Advocates of “free software” often talk about “free beer” and “freedom of speech”. Have you ever heard anybody talking about “freedom of choice”. In the current situation there is very limited freedom of choice in Linux. You can (if you can) do this and that with Linux and the software installed under it. However this appears to be very difficult when compared to so callec “closed” operating systems like Windows or MacOS.
What suxx?
Once upon a time Linux had a relatively good device driver module interface (that was in the 1.2.x days). A driver compiled for some kernel version worked perfectly OK with several kernel versions after that. However things changed after Linux 2.0.0. It become more and more difficult to ship just one binary driver for all 2.0.x kernels. It appeared to be necessary to provide a wrapper module that was compiled for each kernel version. Each distribution introduced slightly incompatible kernels and soon the only way was to compile the wrapper in the customer system. That time it was easy because the driver development environment was installed by default by every distribution. Today even this is practically impossible because the kernel headers and the required tools (gcc, binutils, make, etc) are not installed in most systems.
Earlier our mistake was having OSS as a closed source product. Now we have open sourced OSS but nothing has improved. We still cannot provide precompiled OSS binaries that the customer could install out of box. The customer will have to install the kernel development environment (which consumes gigabytes of disk space) before he/she can install OSS. This requires significant amount of knowledge that most ordinary users don’t have. So only more or less experienced hackers will be able to install OSS.
Sooner or later OSS4 will become part of most Linux distributions. This will solve the problem. However this will take years. Even after that ordinary users will not be able to download the latest version of OSS from our site and install it out of box.
Is this the Bazaar or the Mosque? You decide…
The Dream World (TM)
Today you cannot take the very latest motherboard, network adapter, sound card or whatever and just install Linux to your system. The chances that you will be able to do this are minimal. In the best case you may be able to install some of the Linux distributions by using the noapic and noacpi options. However you will probably have to try several different Linux distributions just to find out that none of them work yet.
In (my) dream world you could take any Linux distribution and boot from it’s install CD/DVD. It will then ask you to insert the install CD’s that came with your motherboard and the add-on devices. After installing the vendor drivers the Linux installer continues. Few moments later you will have a Linux system that supports all the hardware you have got.
However this is not reality at this moment. None of the Linux distributions let you to install any 3rd party drivers. In fact developing 3rd party drivers that work in this way is completely impossible. There is no concept of precompiled binary drivers in Linux. Everything that is not included in the distribution itself would have to be recompiled on-site which requires gigabytes of disk space that is not available during the installation.
The problem is that Linux lacks a sane device driver (binary) interface like DDI in Solaris. It is totally impossible to distribute any precompiled kernel code (drivers) because Linux drivers depend on the deepest details of the given kernel image. I have proposed this kind of interface several times on the linux-kernel mailing list over past 12 years. However these guys don’t seem to like the idea. The current vendor-centric business model of Linux is not compatible with this idea. The current model is perfect for the current Linux players because they can make money by doing paid support for the customers who like to use new hardware. However for 3rd party players like 4Front Technologies this is an unbearable situation. Also the customers will suffer because their Linux vendor has pure monopoly to give any support to them.
May 29th, 2008 at 8:02 am
Especially MIDI-support is an area where FreeBSD should improve to be useful for musicians.
May 29th, 2008 at 3:26 pm
I hear what you’re saying - the Linux kernel doesn’t have a a stable binary API - http://www.mjmwired.net/kernel/Documentation/stable_api_nonsense.txt (let alone a stable binary ABI) and this impairs the ability for anyone to ship binary drivers outside the distro. I wish you hadn’t used the scanner example though. Scanners these days tend to be USB based and yes - support for new scanners is ALWAYS problematic on Linux but it’s a userland problem rather than a a kernel one ( http://www.kroah.com/log/linux/linux_driver_project_status-2008-04.html?seemore=y ) - (notice how most support is via SANE these days) so it’s quite possible to add support for your scanner without touching the kernel.
I don’t think that claiming using up 100s of megs to have a compiler is really a fair comment on modern desktops. For embedded devices this is massive but for a typical machine with an 80Gbyte drive you are using less than 1% of the disk space (assuming the compiler suite for compiling the kernel takes up 500Mbytes (which I know it doesn’t)).
Most enterprise Linux distros do support driver disks during the install but those are special cases.
It is quite possible for things to be automatically recompiled when installed - see Dell’s DKMS, so the user need not be aware of how to compile drivers so long as they are designed to match the distribution.
My take on it is that the lack of binary drivers is both a weakness and a strength in Linux. It means the kernel can change quickly without the need for someone to maintain a glue layer. I think this makes Linux unique and without advantages like this you might as well use other another OS (such as Windows, Solaris and OSX). Certainly Windows and OSX don’t have “new hardware” driver issues and in fact don’t have many other issues that Linux does. If Linux just plows the same route as them it will stagnate in under a year.
June 1st, 2008 at 9:54 am
Hey. I really don’t know what’s going on with other distributions, but my distro is PCLinuxOS, and it was very easy for me to install gcc and make (or were they installed by default?)
I really don’t know. But I do know that the Linux world is changing very fast. I don’t think it’ll take years for OSS 4 to become part of distributions.
Another thing. You really can’t blame Linux for your mistakes. If you want OSS to be part of all distributions you can’t just finish your job and “wait for the guys to arrive”. The real truth about the cathedral and the bazaar is that changes in Linux MUST be bottom-up, not top-down.
By closing the source of OSS, your little project became a cathedral. Everything went top-down, from the source (you) to the people. True bazaar projects are built by COMMUNITIES. You never formed a community. So don’t complain that a relatively new package takes a while to become the standard. The guys at pulseaudio did understand, and it’s now being used by default in a lot of distros.
What’s the matter… jealous?
Also, I don’t see any problem of how some hardware manufacturers (i.e. nVidia) can provide DKMS binaries for distros, and you can’t.
Finally, to address the freedom of choice. Yes, there are lots of Linux elitists who make life miserable to everyone of us. But the lack of Choice in linux very often comes from another evil: Greed.
Let me tell you a tale. Once upon a time, there was a guy who wrote a wonderful package called wxWidgets. It was very good and cross-platform, so anyone could write software using his toolkit. But designing the interface was a pain. So he started writing a GUI designer.
Unfortunately, he got the bright idea of gaining money, and his designer was distributed only for a fee. The result: The free world remained in the darkness.
It was when a friend told me that he had this kind of experience with other packages, that I understood the beauty of Open Source and why St. Ignutius was really an enlightened man. Without the GPL, Linux would be another Windows, not for its popularity, but because commercial vendors would screw up everything.
Here’s another tale: There was a software package called MainActor, which was a very good video editor. The project got discontinued, and even when they had a Linux version, it wasn’t truly free. Now the Linux version can’t be found anywhere. There are some very important application categories that make people stay in Windows. And programmers have one choice: To make money out of people’s misery, or to free the people. The MainConcept guys chose the former.
All advances in Linux are because people got brave enough to invest much of their spare time to aid the community. If you don’t like how kernel development packages are missing from many distros, then complain TO THEM and stop bitching about it in your blog. But right now, it sounds more like you’re angry at Linux because people can’t easily install *your* project in their machines, and not because you really care about them. If your project can’t be easily installed, there’s only one person at fault: YOU. You can’t blame others for not being fast enough when you left the Linux audio world in the dark ages for what seemed to be centuries. Bad karma? You decide.
P.S. Sorry if I seemed in a very bad mood, it’s just that I really get annoyed when I notice that someone “doesn’t get it” about Open Source and Freedom. In the Open Source world, there’s no such thing as “earning money from software”. Software must be free, and you need to search for income in other places. Like, you time - you know, like RedHat does by giving support and all that. It’s not that hard. In any case, I really appreciate that you released the source code for OSS. It’s a step in the right direction. Now I can choose whether to install OSS or pulseaudio.
June 1st, 2008 at 2:07 pm
There are two main reasons why your dream world does not happen.
First of all, in GNU/Linux we usually want free drivers, and the drivers on the CDs that came with my motherboard and/or other devices rarely are. Sure, we can install proprietary 3rd party drivers if we want to (see for example nVidia - which disproves your assertion that we become dependent on our Linux vendor), but only if the user knows exactly what they’re doing. Of course, the possibility to install such drivers is not (or should not be) immediatly accessible by default.
Second, the CDs that came with my motherboard and other devices almost always contain Windows drivers, sometimes contain also Macintosh drivers, but never EVER have I found one with Linux drivers. If the hardware vendors do not provide us with drivers for the Linux kernel, what do you want the GNU/Linux vendors to do? Use the Windows drivers, or perhaps make their own? That’s the hardware manufacturers’ fault, not ours. And that is why your “dream world” cannot happen, and will not yet happen.
June 9th, 2008 at 4:16 pm
“All open source people (should) know the “The Cathedral & The Bazaar” book by Eric S. Raymond published many many years ago. In this paper he presented the difference between closed source (proprietary) “cathedral” development model and the Linux-like “bazaar” one. In the bazaar model you are supposed to have freedom to pick any “free / Open Source Software (FOSS)” and use it.”
Sorry, but you’ve got it wrong. ESR wasn’t referring to *using* software. He criticized the closed, cathedral-style *development model* of GNU software (Emacs et al.), where RMS was teh boss and development took place behind closed doors, even if the final software was released as open source. In contrast, Linux was developed completely out in the open and everyone was welcome to offer their contributions so ESR called it “bazaar”.
A lot of open source software written as corporate projects are developed cathedral-style, with centralized planning and behind-closed-doors development. OpenSolaris, for example.
June 9th, 2008 at 5:02 pm
“The current model is perfect for the current Linux players because they can make money by doing paid support for the customers who like to use new hardware.”
True. It’s also perfect for people like RMS, who want 100% free software system. And it’s perfect for people like Linus, who want to develop software as effeciently as possible. Not to forget Debian developers, who are unpaid volunteers but produce an integrated operating system nonetheless.
A lot of people are happy with the current model. Why would they want to change it? You’d need to present a very compelling argument.
(Although Linux isn’t there 100% yet - many people have to install binary drivers to make their hardware work.)
“However for 3rd party players like 4Front Technologies this is an unbearable situation. Also the customers will suffer because their Linux vendor has pure monopoly to give any support to them.”
But you just said that most people are pretty clueless when it comes to computers (and I agree). Is it realistic to expect them to buy support from multiple vendors? Would they want to?
June 14th, 2008 at 3:33 pm
You’ve missed the point completely, its not about free drivers or a problem with open source. He is assuming that linux drivers are included, the point of this post is to show the difficulty of that. There is no stable platform for the development of a linux driver, if you want your device supported in has to be included in the kernel and there are patches which have been waiting many years just for that.
Thanks for the blog post it finally made the stable abi argument clear to me, it seems the fsf heads are in the way of progress again, oh well
June 20th, 2008 at 1:07 am
Isn’t that part of the dream world description though? A scenario where kernel module support was improved to be less version-specific, where manufacturers released Linux(/Solaris/random other UNIX-base) drivers along with their Windows ones on the cd, and where Linux live cds supported fetching and installing these from said cd before loading the live environment.
After all, for hardware manufacturers to *be able* to supply such drivers that work across a range of kernels, there must be support for such. Boiling it down to a catch 22 where x can’t happen because y doesn’t exist because x isn’t a reality because y doesn’t exist gets us (users) nowhere.
July 16th, 2008 at 8:52 am
Hello
I discovered your blog via reddit, via this news:
http://insanecoding.blogspot.com/2007/05/sorry-state-of-sound-in-linux.html
I love this quote:
“The reasonable man adapts himself to the world. The unreasonable man persist in trying to adapt the world to himselft.
Therefore, all progress depends on the unreasonable man.” George Bernald Shaw.
Being a programmer myself I understand what it feels to try to make things better my way and the world not following you. I have read all your blog and if I were you I would do this:
-Include OSS in every distro. To install drivers is so last century in Linux today.
-Use your unique knowledge for unique work only you can do.
-If OSS dies on linux, you became no one, if not you will always be the one that created it.
I’m a ubuntu user, I have a laptop I bought knowing it would work on linux. All hardware works ok out of the box, including my sennheiser USB headset on ALSA.
My experience: I wanted to make the machine to talk and understand real-time speech years ago. I looked for APIs to do it, OSS was out of the question because it was not included in the kernel. I started using high level libs, changed to ALSA as I needed low level stuff, things I could not do on standard Windows either.
ALSA is POOR, POOR documented, I will had given a lot for good documentation. I created some programs for understanding what speech is through, using a very simple common low level structure that finally works, everything I do is on top of that.
blender 3d program is GPL, they sell documentation, not only t-shirts, every one could sell t-shirts, but only one person could make a documentation book of OSS like the creator of it. There are hundreds of sound studios out there that are willing to pay for quality support, and a lot of debelopers willing to pay for 1:saving their time that is valuable, 2:helping them take the most of their hardware. Ubuntu founder Mark Shuttherworth knows there is a problem with sound in Linux. Have you talked to him?
There is a lot of people that loves linux because of freedom, I can do whatever I want if I know how to do that, not because of the “gratis”. If they are millions of Linux users, there will be thousands willing to help(pay) if a good reason(if they know you first and were their money is spend).
You need people knowing you(slashdot, reddit, digg…).
October 14th, 2008 at 7:48 am
Hannu, I just wanted to take a moment to state that I fully agree with everything you’ve said here; it’s like looking into a mirror. As someone who’s been part of the open-source world (Linux from 1993 to 1997, and FreeBSD from 1997 to present), I’m fairly well-versed in the realities of it all.
The points you make are spot on, and are very legitimate. Open source works very well for a very specific kind of environment: an environment in which the user has the capability and knowledge to fix bugs by themselves, reverse-engineer someone else’s code and improve it (on their own time) — and that assumes (with regards to kernel-level devices) they understand the kernel ABI.
The percentage of end-users who have this skill set is minuscule — I’d say maybe 10% of the user base (and that’s being lax). On FreeBSD, it’s even smaller; maybe 4%. Those of us who develop said software cannot *expect* our users to improve it when they do not have the skills or knowledge needed to do so.
What we *should* expect are users to report bugs or features to us, and it is our responsibility to fix/add them. It does not matter if the software is free or commercial — when you create something (think: child), that entity is your responsibility from that day forward.
The sooner we embrace that reality, the sooner we’ll have better results in the open-source world. And remember: this is coming from someone who *does* write software. We just need to keep an open and realistic mind about how to properly support software. Lack of responsibility is one of OSS’s biggest fault points — and that comment is coming from someone who does develop software, and does release it into the public realm.