Friday, January 28, 2005
Moving to the darcs side
A while ago, I went and asked the question about why Gnu Arch was so popular among Lispers. I got some replies from people that suggested that Arch was clearly superior to CVS/Subversion in a number of ways, and so I decided to give it a try. After a few months of relatively light use, here's my report. I won't rehash a lot of the general Arch vs. revision-control-X sort of stuff. You can find a lot of that on the Arch Wiki website and the websites of all the revision-control-X's out there. Note that these are my observations. If you're using Arch successfully and think it's the tip-top, fine; carry on.
On the pros side:
- I like that Arch supports distributed development and doesn't require a special server for publishing an archive. A simple HTTP or FTP server will work just great.
- I like that Arch supports branching relatively easily. Once a branch is created, you can stay in sync with upstream changes or publish your changes back to the upstream repository.
- The Arch Emacs interface, xtla, is quite nice, and seems to be evolving quite rapidly now. Xtla simplifies a number of Arch processes, which as I'll point out below is very helpful.
On the cons side:
- Arch is way more complicated than it needs to be. I printed up a little cheat-sheet for myself with all the various commands and a brief description. My cheat sheet was two pages of small print. The semi-official cheat-sheet that you can get from the Arch Wiki is six pages, but it includes options with explanations for each command.
- There are many commands that seem to be the same, but slightly different (
update
vs.replay
, which I still haven't figured out). Adding to the confusion, in other cases there are aliases in the command set where the commands really are the same (add
vs.add-id
, for instance). This is one area where xtla helps out. By following a standard xtla flow, you don't have to remember too many of the commands; they just become emacs keychords. - The Arch command names are, in many common cases, non-intuitive. For instance, why
star-merge
and not justmerge
? - Arch uses very strange file names for everything, composed of some characters that require shell escaping. Further, Arch seems to love long names (files, repositories, everything), making some commands painful to type. In some cases, these actually are file names and you can use shell tab-completion to help out; in other cases they are internal Arch names which may map to file names in a directory structure, but shell completion can't be used (full repository names, for instance). In that case, it's very difficult to type out a 40-character string of semi-gibberish and get it right the first time. This is one area where xlta is a huge help. By simply selecting names from buffers, it's relatively easy to avoid making mistakes.
- The Arch repository structure is painful. It's sort of hierarchical, with archives, categories, branches, versions, and revisions as the various components. It was never really clear to me how to use categories to good effect. Some of this seems like a throwback to a semi-centralized repository structure. That is, you can put all your projects into a single repository if you want, then have categories under the main repository for each project. At least that's the only thing I could figure out. But if that's the case, why call them categories?
- Arch processes are more complex than they need to be. Setting up a repository and importing code takes several more steps than it should. Branching and merging is more complex than it needs to be.
I spent some time reading comp.version-control.arch.user last night to get a feel of how other people are using the system. It turns out that many of these problems are known, though some not recognized (internalized?) as problems. I saw a lot of Arch defenders dismissing things simply as "user interface problems," as if we were simply talking about a pixel being out of place somewhere. Frankly, whenever I see that, I get scared. What is the purpose of a program if not to interface to a user such that the user can accomplish tasks with it. If the program makes it so confusing that users can't get done what they need to, quickly and easily, then what good is it? Arch strikes me as a powerful program that has evolved without a lot of UI direction.
Tom Lord, Arch's creator, is probably a brilliant guy, but he needs to take a class on user interface design. No, I'm not talking about GUIs with pretty colors and dialog boxes, but just your basic "users need a simple mental model of the system and if you don't provide that, everything will be confusing and complex, even when it doesn't need to be" sort of thing.
Others will say that I'm confusing complexity for power. "When a tool is powerful," they'll say, "it's bound to be complex." And Arch is certainly powerful. Well, there is a case to be made that tools that do a lot of things naturally end up with a lot of commands. Yes, that's true. But those commands can be organized in simple ways. And simple processes can be given simple commands to execute them. As Einstein is reported to have said, "Everything should be made as simple as possible, but no simpler." In this case, I believe Arch could be much more simple.
So, when I first asked my question about Arch, one kind reader suggested I check out darcs. After getting irritated at Arch last night, I did. In short, darcs seems amazingly streamlined and intuitive when compared to Arch, but provides basically all the same advantages of distributed version control. After quickly skimming the manual, I was able to set up repositories, create patches, branch, and merge. The command set is small and rationally organized. The basic processes take only a couple of commands to execute. There are no funky names of repositories, etc. Further, darcs seems to work on Windows quite well, which has been a limitation of Arch before this. While I use Linux mostly, there are times that I have to deal with Windows and it's nice to know that I could use the same systems on both environments.
In reading comp.version-control.darcs.user, there are a few limitations to darcs currently. Darcs is at version 1.0.1 as I write this, with a 1.0.2 imminent. Arch has had more time than darcs to become feature rich and optimized. Some current limitations of darcs seem to be:
- It's sometimes slower than it should be. Darcs is written in Haskell. If you're a functional programming junkie, you should see this as a win. What it means is, however, like most programs written in a high level language, there is always some performance tuning to be done. Darcs' author, David Roundy, is starting to work on some of the bigger problem areas as we speak. Given that I'm programming in Lisp now, I understand this completely. Over time, optimization will come, but Arch is probably faster for larger projects; small projects won't see much of a difference and darcs seems to be fine for everything I have used it on so far. Some operations are a bit sluggish, but nothing too problemmatic. As a test, David has the Linux source tree available via darcs.
- Darcs doesn't have too much support for signing of patches. If you're looking for a very secure version control system, darcs is not (yet) for you. There is a capability to send patches to a repository or a maintainer via email, and those emails can be signed with GPG. But Arch has the capabilities to sign every patch stored in the repository, not just in transit. Arch supporters argue that if your repository is broken into, this provides better protection against tampering. Whether this is a huge concern or not probably depends on your deployment model. If your repository will be outside a firewall environment, the Arch signature methods may make more sense.
As a result of all this, I decided to "go over to the darcs side" for a while and give it a try. I'll let you know what I find out for my own personal projects.
Wednesday, January 26, 2005
Microsoft is so clueless
Just found this on C|net. Microsoft is going to be forcing people to verify that their copy of Windows is legit before they can download security patches. This is a great example of the short-sighted corporate wonks exerting themselves without understanding anything about marketplace dynamics. See, here's the deal...
The corporate wonks look out into the world and see piracy, particularly in communist and former communist countries like China and eastern Europe. "Horror!" they cry. "We're losing millions and maybe even billions of dollars. Why, if everybody in China is pirating a copy of Windows, that's over $200B right there. It's our duty to our shareholders to crack down on all that."
When I look out at the world, I see lots of Microsoft advertising. The reality is, Microsoft is the only company that is able to extract a "tax" from the PC industry as a whole. Just about every computer is shipped with a copy of Windows, and most computers also have a version of Office, whether pre-installed or as a post-buy add-on. These installation rates are the same, whether the PC is sold in the USA where things are mostly legit, or whether it's sold in China where the copies are probably pirated. In my mind, the pirated copies are just advertising for Microsoft and help ensure they remain dominant. Kill off the pirates and you actually may create a problem for yourself.
See, when you're the default for a whole industry, the last thing you want to do is rock the boat and make people actually think about choosing your product. If you force them to a choice, you're re-opening the buying decision that was otherwise a done deal. Nevermind that in China they weren't paying you for your product. The usage of Windows and Office in China helps proliferate the number of programs and files in Office format that help the world-wide monopoly to stay in power. And as long as you have a world-wide monopoly, you get to charge the mostly-honest guys in North America and western Europe the "PC tax."
If Microsoft rocks the boat, as it looks like they are doing, one of two things will happen:
- People who previously would have chosen Windows by default will now look for other cheaper alternatives such as Linux. This will happen even if Microsoft creates a special "Windows 3rd World," priced at obscene discounts to regular Windows. Anything more than free is going to force the choice because most of the pirates in the third world can't afford to pay for all of it anyway (sure, a few can, but not nearly in proportion to the copying). Once people start to choose, it's a snowball that keeps rolling, and it isn't going in your favor. Rather than more programs written for Windows and documents in Office file formats, you'll see a growing base of Linux and OpenOffice. At some point the industry reaches a tipping point and Microsoft's hold is broken.
- The other alternative is that people will still pirate Windows, but they'll just ignore the security updates. This is also bad for Microsoft. All those Chinese PCs are going to be chock-full of viruses and worms and that's only going to perpetuate the image of Windows being a security problem. Yes, yes, those are unlicensed copies, but they're still Windows and they still highlight the Windows security problem, no matter what Microsoft says. Further, when those PCs get so burdened with those viruses, they'll get slow and crash all the time. Even the Chinese pirates want PCs that work and they'll start looking for stable alternatives.
So, either way Microsoft is just shooting themselves in the foot. The lesson here: when you have a good thing going, you'd best shut up and take the money. Trying to maximize your gain, thinking that people actually like your software is a poor bet, and you'll eventually be the worse for it.
That said, I use Linux almost completely now, so I'd actually like to see the world tip more in that direction. The Linux solutions are quite competitive to anything in the Windows world, run far faster, and are more stable. That wasn't the case as recently as five years ago, but things have come a long way of late.
SBCL 0.8.19 Released
SBCL 0.8.19 was released yesterday. The RPM package can be found at the SBCL files page on Sourceforge.
The RPM is compiled with thread and futex support.
Friday, January 14, 2005
XML + ASN.1 = faster? (or Stones Don't Float)
Do you ever have an idea, then somebody else has the same idea, but they screw it up? Well, I just had that happen to me. I stumbled on this article on Fast Infoset today. Fast Infoset is a specification for a binary version of XML. I had this same idea a couple of years ago, and the execution is amazingly similar, but, of course, they screwed it up.
Essentially, everybody is finally realizing that while XML is the first widely accepted data markup format, it's a pig. It's verbose and redundant, which makes it store poorly, transmit slowly, and parse, well, like a pig. Don't get me wrong, the original idea for XML was actually fine, but everybody has taken it way over the top and things like SOAP are just an abomination.
Well, one way to help XML while still retaining XML semantics is to make a binary version of it. While a compression program like gzip can reduce the overall transfer and storage size of an XML document, an XML parser still has to deal with the XML textual format on either end of a transfer. The textual format forces the decoder to examine each and every byte to determine its significance in the document, even when a decoder doesn't understand vast expanses of the XML schema that is being parsed. And that textual XML format actually expands binary data carried in an XML document by forcing it into BASE64 format which does a 3-goes-to-4 encoding.
So, my idea, and that of Fast Infoset, is to notice that most XML documents are extremely redundant with tag and attribute information. First, you have all those < and > symbols surrounding every tag. That's four bytes of redundant info for every opening and closing tag pair. Then you have the tag name itself, which gets repeated at least twice, once in the opening tag and once in the closing tag. Then, you have the fact that many XML documents are recursive tree structures where nodes at any given level of the hierarchy share a lot of the same type of info. For instance, an XML document storing a list of books would use TITLE, AUTHOR, and ISBN tags for each book. Add in namespace prefixes and attributes and you have a lot of redundancy, even before you start talking about SOAP. In short, the information density of your average XML document has Claude Shannon spinning in his grave.
Luckily, it's pretty easy to compress this information right out of an XML document. You simply create a tag/attribute hash table and assign each unique tag or attribute in the document a unique number. Rather than writing TITLE everywhere, you would instead use the number 0; 1 for AUTHOR; 2 for ISBN; and so on. So, rather than "<AUTHOR>" (8 bytes), we would simply have 0x0001 or something (2 - 4 bytes). By eliminating the end tag and encoding the content inside the AUTHOR tag with a 4-byte length, we have a net savings of 8 + 9 - 2 - 4 = 11 bytes every time we would otherwise use an AUTHOR tag. Do the same thing for all the other tags in your document and it adds up pretty quickly. Finally, by encoding binary data as binary octets, rather than BASE64, we eliminate the "ASCII tax" imposed by a textual format on binary data.
The interesting thing is that we still fundamentally have an XML document. This means that you can run this new format through a SAX-like or DOM-like decoder and produce exactly the same data that an XML-based application expects, and the application is none-the-wiser. Everything is just smaller and faster. Because the binary format parser doesn't have to actually look at the characters that make up all these tags and try to match them with other strings, it can whip through a document at light speed.
So, that was my theory. I even had a catchy name for it, BCX: Binary Coded XML.
Along comes Fast Infoset. Great minds think alike. Obviously, the need is there. Only they screwed it up. Rather than just keeping things simple, they decided to encode the whole thing in ASN.1 with packed encoding rules (ASN.1/PER). Now, for those of you who don't know, ASN.1 is about the most complex, worst, ugly data format ever designed. And it's no wonder, it was created by an international committee (ISO) as part of the Open Systems Interconnect protocols (OSI; anybody remember FTAM?). The various encoding rules used to actually serialize ASN.1 (and there are several, which is part of the problem) typically do a lot of bit-twiddling, which slows down encoder/decoders. It also makes them buggy. Anybody remember some of the recent buffer overruns attributed to ASN.1 endecs?
Well, it looks like Fast Infoset is being standardized in ISO (in fact, jointly with ISO/IEC JTC 1 and ITU-T, which is about as ugly as it gets in the international standards committee world), so they probably had to use ASN.1 or people would wonder why they weren't supporting their own standards. ASN.1 is used by a couple of protocols in common use today, including SNMP, X.509, and LDAP. That said, most IETF-originated protocols (you know, the ones that move all your web, ftp, email, etc., traffic around) use either straightforward text encodings, or far more simple binary encodings.
Fast Infoset does manage to get pretty good compression (20% - 80% reduction, depending on document size and content). Throughput for some documents is 2 - 3 times what standard XML delivers. Overall, these are good numbers.
But, it could have been even faster. Frankly, I'm partial to Sun's old XDR format, used by ONC-RPC and NFS, which was very fast, if not quite as tightly packed as some other formats. I was also recently reading about Joe Armstrong's Universal Binary Format (UBF), created to give Erlang a streamlined wire protocol.
In short, marrying XML and ASN.1/PER is like tying two stones together and hoping they will float.
SBCL RPMs now available on Sourceforge
William Newman graciously gave me access to Sourceforge to be able to upload the SBCL RPMs I recently built. The latest version is sbcl-0.8.18-3.i386.rpm. The latest build enables threads and futexes.
Enjoy!
Wednesday, January 12, 2005
My first Lisp anniversary
Well, it has been a full year since I took the plunge and decided to learn Common Lisp. While I had flirted with Common Lisp and Scheme back in 1988 and 1990, after reading some of Paul Graham's writings and going over Pascal Costanza's Highly Opinionated Guide to Lisp in 2003, I decided that it was time to go back and learn a Lisp dialect for good. This posting to comp.lang.scheme shows my initial struggle to decided whether Scheme or Common Lisp was the right dialect. I really do have a soft spot in my heart for Scheme. It's just so nice and simple. But as Nikodemus Siivola's email signature reads,
Schemer: "Buddha is small, clean, and serious."
Lispnik: "Buddha is big, has hairy armpits, and laughs."
I decided that while Scheme was lean and mean, Common Lisp showed the signs of a more industrial-strength, battle-hardened language. Does CL have rough spots where Scheme has clean edges? Yes, indeed. But it also shows some foresight for how to build a language that will work on a larger project and scale further than will Scheme. Is CL perfect? Nope, there are many things that I would improve if a quick decision were all it took. But CL works well. There are great implementations available for many platforms, both commercial and freeware, and those implementations have gotten better and better through all of 2004.
How's the progress after one year? Well, I don't get to hack as much as I'd like. But when I do, it has been consistently enjoyable. Frankly, I can't believe I never committed myself to this 40+ year old language before this. It isn't like it wasn't available all that time, and over the years I seem to have learned everything around it (BASIC, assembly language, FORTH, C, Fortran, Pascal, C++, Java, etc.).
If you're a newbie reading this and thinking about taking the plunge, all I can say is just do it. You will never stop thanking yourself. Yes, I know the parentheses are intimidating. Use it for a week and watch them disappear. Even John McCarthy thought they were a problem. You aren't the first.
"One can even conjecture that Lisp owes its survival specifically to the fact that its programs are lists, which everyone, including me, has regarded as a disadvantage."
- John McCarthy, "Early History of Lisp"
And, Mr. Newbie, when you decide to take the plunge, and have played with Lisp for a little while, tell us about your own personal Road to Lisp. We're happy to have you.
Sunday, January 09, 2005
Keyword parameters in macro expansions
I was hacking a macro last night (some work on state machines again) and discovered a subtle point that I hadn't realized before. It seems that the default values of keyword arguments are evaluated at the time a macro is expanded, rather than remaining unevaluated like keyword parameters in the macro call. Let's give an example here:
First, let's look at a simple macro:
(defmacro foo1 (x) `(+ ,x 1))
From the REPL, we can see what this expands to with various arguments using MACROEXPAND.
CL-USER> (macroexpand '(foo1 10)) (+ 10 1) T CL-USER> (macroexpand '(foo1 (+ 1 9))) (+ (+ 1 9) 1) T
Notice that when we call the macro with a complex form as the argument, the macro doesn't evaluate the argument. Rather, it simply sets X to "(+ 1 9)" and that is then inserted into the expansion at the right place. The fact that forms are not evaluated before the macro runs allows us to create some complex macros with interesting behavior. See AND and OR, for example, which evaluate their arguments one-by-one, as needed to calculate the value of the entire expression. Once AND has detected a single false value or OR has detected a single true value, the rest of the parameters are left unevaluated, resulting in the "short-circuit evaluation" for these expressions that we're all used to (in both Lisp and C).
Now, things get interesting when we have a keyword parameter in a macro:
(defmacro foo2 (&key (x (+ 1 9))) `(+ ,x 1))
In this case, we're telling the macro expander that x is an optional parameter, but if it's not there, we want it's value to be "(+ 1 9)". Lets see how this expands with a couple of cases:
CL-USER> (macroexpand '(foo2)) (+ 10 1) T CL-USER> (macroexpand '(foo2 :x (+ 1 9))) (+ (+ 1 9) 1) T
Okay, so what is going on here?? In the first case, the macro expands with the default value for X, but in this case it looks like the expansion has already evaluated the "(+ 1 9)" that we specified for the default. In the second case, we provide the value and the macro does not evaluate X the argument but binds X to the form "(+ 1 9)".
Now, in this simple case, it really isn't a big deal. The compiler and optimizer should crunch all this down to the same value. The case that bit me last night is when I wanted to specify a parameter with a default functional value. For instance:
(defmacro test-it (x y &key (test #'eql)) `(funcall ,test ,x ,y))
So this simple macro allows you to compare two values. If the user doesn't specify a value with the TEST keyword parameter, it defaults to EQL. Let's see how this expands:
CL-USER> (macroexpand '(test-it 'a 'a :test #'eql)) (FUNCALL #'EQL 'A 'A) T CL-USER> (macroexpand '(test-it "a" "a" :test #'string=)) (FUNCALL #'STRING= "a" "a") T
So far, so good. Just what we expect. Now, let's just go with the default value:
CL-USER> (macroexpand '(test-it 'a 'a)) (FUNCALL #<FUNCTION "top level local call EQL" {1076EAD}> 'A 'A) T
WHOA! What's going on here. Well, as we saw previously, the default value of TEST is being evaluated before the macro expansion takes place. In this case, the expression "#'EQL", which is shorthand for "(FUNCTION EQL)", is being evaluated. This produces the actual function object associated with EQL, which is unprintable. Thus, when the macro expands, this gets stuffed into the middle of the resulting expression. Now, it turns out that in the SBCL REPL, this actually works as you might expect:
CL-USER> (test-it 'a 'a) T CL-USER> (test-it 'a 'b) NIL
We'll run into problems, however, when we try to compile a file that uses this macro, with COMPILE-FILE, for instance. In this case, SBCL doesn't know how to serialize the raw function reference into the resulting FASL file.
So, how do we fix this? Well, the lesson is that we want to prevent evaluation of default values in this case. We can do that with a quick QUOTing of the value:
(defmacro test-it (x y &key (test '#'eql)) `(funcall ,test ,x ,y))
Notice the extra single-quote in the "'#'EQL" form. This results in:
CL-USER> (macroexpand '(test-it 'a 'a :test #'eql)) (FUNCALL #'EQL 'A 'A) T CL-USER> (macroexpand '(test-it "a" "a" :test #'string=)) (FUNCALL #'STRING= "a" "a") T CL-USER> (macroexpand '(test-it 'a 'a)) (FUNCALL #'EQL 'A 'A) T
Now, everything works as expected. So, note to self, remember to quote default optional and keyword parameters in macro definitions when we don't want them to be evaluated.
Thursday, January 06, 2005
SBCL 0.8.18 RPM: Testers wanted
I created an RPM package for SBCL 0.8.18 the other day and I'm looking for some testers. You can grab it at the moment at http://www.findinglisp.com/tmp/sbcl-0.8.18-2.i386.rpm.
This is not yet a permanent location for this file. I'll move this to someplace better once I get confirmation that things seem to be working out fine. I have done limited testing so far and it seems stable.
Update: I have uploaded these to Sourceforge in the SBCL area. Please retrieve RPMs from there.
This is stock 0.8.18 using default build options (no threads, for instance) with the exception of patching it such that the binary lands in /usr/bin, SBCL_HOME defaults to /usr/lib/sbcl, and documentation lands in /usr/share/doc/sbcl-0.8.18. These locations were done per standard Fedora packaging guidelines, with the long-term goal that this eventually ends up as part of Fedora Extras for easy installs.
I'll also probably turn on threading in subsequent builds, but just wanted to minimize changes here in case somebody finds something broken.
Thanks also to Miles Egan for the 0.8.10 RPM on sf.net, from which I snarfed some rpm spec ideas.
Wednesday, January 05, 2005
Some thoughts on Free Software and GNU
I was reading this interview with Richard Stallman on KernelTrap today. It's always interesting how a Stallman/GNU article brings out all the zealots, both for and against free software. The comments about the interview were of course even more voluminous than the actual interview itself. I have watched the FSF develop for decades now and this has always been the case from the very start. I remember reading some of the first Stallman writings in the 1980s and thinking, "Hmmm... this guy is probably crazy, but if he pulls it off, wouldn't that be interesting?" Here are some other thoughts on GNU, Open Source, and Free Software, formed over a very long period of time (in other words, if you write to me to debate these, you're unlikely to change my mind unless you happen to come up with something very incisive that I have somehow not considered ;-):
First, let me say that the one thing I disagree with very strongly is RMS's statements that non-free software is immoral and "antisocial." Proprietary software is what it is. If you don't want to buy it because it doesn't meet your need for having the source available and being able to hack on it, fine, but don't go around suggesting that other people are stupid for doing so. As long as nobody is forcing people to use non-free software, all software licenses are just options in the market place and will be adopted by various people depending on what they offer. The freedom to hack source code is just one more "feature" of the product, nothing more.
Similarly, don't suggest that corporations that produce non-free software are somehow evil and corrupt. Again, as long as they aren't somehow forcing people to buy their products, they are just one option in the market place. If they can make money selling non-free software, then obviously "freedom" isn't all that it's cracked up to be. Whenever anybody writes the phrase "corporation producing non-free software," most people think specifically of Microsoft. Note that I don't begrudge Microsoft their ability to field a product in the market and charge whatever they think their customers will pay for the product. However, some of Microsoft's past business practices were definitely oriented toward limiting competition and choice of consumers and I think they have gotten off pretty light in return for decades of anti-competitive behavior. The point is, when you hear the phrase "corporation producing non-free software," it's much better to think of somebody like Adobe, Macromedia, etc., than Microsoft.
Whenever one starts discussing free software, GNU, or Stallman, it's important to remind everyone that "free" in this context means "as in speech," not "free as in beer." Stallman is not saying that you can't charge for bits, just that once you have sold those bits, you can't prohibit somebody from taking the same bits, possibly with modifications, and propagating them downstream for whatever price they want, possibly for zero cost, and so on... In other words, RMS isn't suggesting a software economy where no money ever changes hands; rather, he's advocating an economy where source code always changes hands and people have the freedom to make alterations to a product and then redistribute the resulting hybrid product.
Whenever RMS gets interviewed, somebody always starts throwing around the "communist" label and suggesting that such subversive ideas will lead to the destruction of the software industry. I think such fears are overblown. However, I also think that the FSF's economic theories have a couple problems. Here's the way I see it:
- Freedom to innovate on top of other people's work is a good thing. It's certainly the case that a group of talented people can do more than a single person and that you can't ever predict where the next talented person will come from. By allowing information to flow more freely, you allow it to reach those with the talents and dispositions to use it to build something better than what has gone on before.
Now, that's great for raw ideas, but ideas are not software. Ideas are abstract things that, while they take talent to conceive of, take far more effort to reduce to practice. For instance, I have "designed" many, many computer programs in my head that I have never reduced to running code. Producing running, debugged code is hard. To be awarded a patent for anything (ignore the sensitive issues surrounding software patents here and focus on even hardware patents), the patent office requires an inventor to reduce it to practice. In other words, raw ideas are great, but an embodiment of those ideas is actually something of value because it required lots of effort to perform that reduction to practice. Simply thinking, "Wouldn't it be great if there was a piece of hardware/software that performed function X?" isn't the same as actually building such a thing.
So the question here is how people working to reduce great ideas to practice are going to eat in the mean time. Every economic theory has to answer this question. All philosophy about the exchange of knowledge and ideas will lead absolutely nowhere if you can't answer the question of how people eat.
- The thing I like about the GPL and FSF is that it does promote the ability of people to build onto the work of others. It says, "Hey, it's a great thing to have the source code/schematics/plans/internal details for a widget available such that if it doesn't work quite the way I want as a consumer, I can modify it to do what I want." That's a nice feature of a product and promotes innovation on top of the product. I don't believe that the "has source code available" feature is required in every product, however, or is somehow a right of a consumer. I do believe that it is a right to be able to choose such, however, as long as there is a producer willing to give it to me.
The main problem that I see with the GPL economically is that it then allows somebody to redistribute the original work without giving any economic benefit to the original creator of the work. The way to look at this is with the concept of a value chain. In the world of physical products, producer A creates widget A. Producer B buys widget A from producer A and transforms it (modifies it, incorporates it, etc.) into widget B. Producer B then sells to C, and so on, until the good is finally consumed by somebody at the end of the chain. The important concept here is that in the world of physical goods there is money flowing back from the consumer to everybody along that chain.
With the GPL, producer A can make software and even sell that software to producer B. Producer B can take that software and create a modified product, but then producer B can sell that code to producer C without any obligation to pass on any of that value to producer A.
The FSF theory states that this is okay, since producer A is entitled to get any modifications of its software from producer B and sell them for any price it wants. In other words, you get my stuff, but I get yours, so it all works out. I can't limit your freedom and you can't limit mine. I view this simply as a software version of mutually assured destruction. In practice, bits become free (as in beer) as a result. This is not necessarily a problem as long as we can still find a way for the developers who wrote the original code (or even the modifications by B) to eat.
FSF advocates respond to the charge that bits become free by suggesting that people will then derive compensation for their work by doing other things like support, training, selling documentation, etc. This is problematic for a number of reasons:
- Development is difficult and support is a necessary evil. As a business, I'd rather spend all my time developing my product and as little time supporting it as possible. If I do the reverse, it means I have lots of customers with issues and my product is stagnating in the market. If I'm deriving my revenue from support, I'm actually incentivized to spend as much time on support as possible and as little on development as possible, leading to exactly the reverse of what we'd like as consumers.
- It is difficult to differentiate on support. There are many organizations that can support all sorts of different products. This is particularly the case with open source where the source code is available. As the developer of that product my revenue stream is being undercut by competitors that are not burdened with my development costs. Put another way, in this model, another business is actually incentivized to let everybody else do the development and then come in and skim off the support revenue (say business C comes in and supports the system created by A and B and doesn't give anything back to either one).
- At this point, people often speak up about "community" and giving something back. Business C is bad, they would contend, if it simply reaps the rewards without doing some innovation itself. While I'm all for developing a good community and fostering community spirit, let's de-cloak that word and call it what it really means in this context: charity. While business C might have some sort of tacit moral obligation to not simply ride somebody else's coat tails, it is under no legal obligation to do so. In practice, there will always be at least one business like C, which will then gain competitive advantage by not shouldering the development costs. Whether C is ultimately successful depends on how much the "community" values "community spirit" (do they buy from C or avoid C because it is riding coat tails and therefore C is populated by bad people).
- In summary, if you're going to create a sustainable system, you need to find a way to tie revenue to effort as directly as possible. Any model which says "give this away and make it up over here" is going to create stresses on the system which can eventually cause it to fail. Can sustainable "loss leader" systems be produced? Yes, they can and there are many examples, but they need to be very carefully thought out and I don't think most FSF advocates have done much more than a hand wave here. In other words, certain product categories will do better here than others. Examples supporting the case include complex systems like JBoss where people inherently need training, etc. Examples against include anything that will be used by consumers who don't have time for training and don't want to be calling a 1-900 number for support.
Whew! So what does all this mean? Am I down on either free software, the FSF, the GPL, or RMS? No, not at all. I think free software is a great thing, but I think there is definitely a place in the world for non-free software. I don't buy into the ideology that non-free software is morally wrong and "anti-social." I believe that the amount of free software will reach equilibrium in the market according to the revenue that people are able to derive off of other sources associated with it. In short, if everybody spent all their time writing software for which they didn't derive compensation, they would shortly not be able to eat, they would die off, and would be replaced by developers with a better understanding of economics. Insofar as people want to contribute their work to the market place under various open source terms and conditions (the GPL, Apache, BSD, and even public domain), I think that's a great thing. But we need to recognize that this is a gift from such developers and not expect it as a fundamental consumer right.
As a last point, let me suggest that free software works best for more fundamental pieces of technology that are commoditizing and have an inherent need to be low cost in order to achieve widespread adoption. Things like web servers and operating systems are good examples. More specialized applications which generate lots of downstream revenue dollars for their users are more willing to be paid for. Whether source code comes along with those applications as a feature of the product sale is just that, a feature to be negotiated between the producer and buyer. An example here might be a specialized piece of video editing software used by a movie production company where the company derives millions of dollars of revenue from such a product.
Finally, let me note that it's interesting that some of the free Lisp implementations out there, notably CMUCL and SBCL, are public domain, with no license whatsoever (use the code in whatever way you wish).