A couple years ago, I exchanged email with Dan Weinreb about Lisp machines. Dan was one of the founders of Symbolics and wrote the Emacs variant on the Symbolics machines. During our email exchange, I asked Dan about some of the statements and writings of Richard Stallman with respect to Symbolics and Richard's characterization of Symbolics as one of the bad guys in the Lisp machine competition with LMI. This was interesting to me because the Symbolics vs. LMI competition figures prominently in RMS's reasoning to create GNU and the whole concept of Free Software.

Of course, there is always another side to the story, and Dan definitely disagreed with Richard's characterization of the situation. Having interacted with both Dan and Richard, I tend to believe Dan's version more than Richard's. But that's just my opinion. I told Dan that I thought he should document his version of events because the only side of the tale that was being told up until that point was Richard's, and without any evidence to the contrary, his side was being accepted as gospel.

At the time, Dan basically said that he just didn't feel like it. He was tired of the whole thing and just wanted to let Symbolics rest in peace. Having founded an unsuccessful startup myself, I know exactly where he is coming from. The last thing you want to do is relive things by going through the act of documenting history. Dan told me that he would probably document things at some point, but not at that moment.

Well, I'm pleased to report that Dan has started a new blog and one of his first postings was his version of the Symbolics vs. LMI competition. It directly refutes some of RMS's assertions and provides a good counterbalance to this controversy. I'm glad that Dan published his side of the story. Who you believe, well that's up to you. If you're an RMS fanboy, Dan's account of things is unlikely to change your mind. But at least history will record that there were two sides to the story.

As an aside, I'm glad Dan has started blogging in general. He struck me as a very smart cookie and I'll definitely be adding his RSS feed to my list. Whether he blogs often or not, I'm sure I'll be the better for reading his words.

Hat tip to Xach for noticing Dan's blog.

There was a day when I didn't know what a bignum was or why I would possibly want one. I was fat, dumb, and happy with basic fixed-precision integers in C and Java land. When I first started playing with Common Lisp, I thought, "Wow, that's a neat trick that I can actually evaluate (factorial 100) and it works correctly, but how often am I going to be doing that??"

Boy, was I wrong. I'm working on a project in Emacs Lisp in my spare time and I suddenly have a need to deal with integers that are greater than the anemic 29-bit fixnums that Emacs Lisp supports. While 29 bits might be okay for working with character positions in a buffer, it falls flat on its face whenever you want to manipulate anything from the outside world. In this case, 29 bits is too small to handle any 32-bit quantity, let alone some of the 64-bit stuff I want to deal with. I'm kind of amazed that RMS didn't just implement 32-bit boxed integers if he wasn't going to do full bignums.

I found that calc.el includes support for its own version of bignums and I might be forced to use those. Painful, though. I guess the old saying is true--you don't miss something until it's gone.

Does any Emacs guru know if bignums are being considered as a standard Emacs Lisp datatype for a future Emacs version? I'm currently working in Emacs 22.1 (Fedora 7). Given the release schedule of Emacs versions, any future support won't help me, but it would at least be comforting to know that other people see the same need.

I spent a while last night reading about Tim Bray's adventures with Erlang. Tim started investigating Erlang as a part of his "Wide Finder Project" in which he's looking for programming languages that will help accelerate common tasks on the soon-to-be-very-popular CPUs with many cores but slower clock rates.

Tim works at Sun, and so this question and project makes perfect sense in light of Sun's Niagra and T2 processors with many cores and CMT. It also makes perfect sense in light of Intel's Tera-scale computing initiative where they have demonstrated chips with 80 cores. In short, the future is going to be very, very parallel, and we had better come to terms with that.

Unfortunately, the modern multi-threaded programming paradigms are ill-equipped to take advantage of these modern processors. Most popular programming languages have the same simple threads+locks paradigm that was popularized with pthreads and Java. While this works, it doesn't work well. As with so many things in programming, the threads+locks paradigm forces programmers to remember a whole lot of crufty details, otherwise they will produce code with very subtle bugs that is very, very difficult to debug. Put another way, threads+locks is the parallel programming equivalent of manual memory management. In the same way that GC manages memory in many modern programming languages, we need an equivalent to help programmers manage parallel programming.

Erlang has many features that help it work well on multi-core systems. The language is inherently multi-threaded and concurrent and relies on threading almost down to the core (you can write non-threaded Erlang programs, but the language makes it so easy to use threading that you'd hardly want to). I have talked about Erlang before a few months ago (here and here).

So, here we have Tim Bray asking a perfectly sensible question, "What programming language is going to help us programmers exploit the soon-to-be-commonplace multi-core CPU?" Erlang is certainly a potential answer to that problem. Unfortunately, Bray decided to pick a problem for which Erlang is particularly unsuited and then compares Erlang to another programming language that optimized for the problem, albeit without any parallel programming support--Bray picked a simple web log analysis problem.

The web log analysis problem is one that is well suited to Perl, Python, or Ruby. One might even say that these languages were virtually created to solve problems in this exact domain. If Perl does anything really well, it processes text with regular expressions. Python is a bit more clunky than Perl or Ruby in terms of regex syntax, but still does quite well. Ruby was created to solve many of the same problems as Perl, but with a better object model and saner syntax (IHMO).

Erlang, on the other hand, was created to develop 24x7x365, long-running telecom software. This is stuff that aims to have downtimes measured in a handful of minutes per year and that must be able to be upgraded on the fly and recover from any faults or failures. In short, Erlang aims to help programmers write code that can take a bullet to the head, recover, and keep on doing its thing, later allowing the programmer to find the fault, fix it, and upgrade the system, all while staying up and doing its thing. This is a hugely complex task, I can assure you. And Erlang conquers it in fairly good fashion.

So, when Bray decides to try Erlang on this problem, he naturally finds that it blows chunks. His first attempt doesn't even attempt to use any Erlang threading features, which of course defeats the whole reason for the investigation in the first place. Knowing enough about Erlang to be dangerous, I found myself saying, "Well, duh! Why did you expect that to get great performance?"

Overall, I'm getting pretty tired about these simplistic comparisons that people do between programming languages. It always feels like they're an elaborate sort of "gotcha." Step 1, pick a task that runs particularly well on the evaluator's most familiar language (Bray picks web log analysis, which runs quite well in Ruby, his choice language). Step 2, pick a victim language. When the victim language doesn't measure up, yell "WTF?!?! [insert language] sucks." Now, in truth, Bray didn't do that last part, but you have seen the pattern other times, I'm sure (witness the number of people that list the Computer Language Shootout as justification for almost anything).

What would have made a better comparison is writing a multi-threaded web server in both Erlang and Ruby and see which server is able to deliver the best performance to 10,000 active clients with widely varying download speeds. I'd be willing to bet that Erlang does a better job. No, I wouldn't even suggest writing a 24x7x365 telecom switch in Ruby; as fine as Ruby is, Erlang would win that hands down.

So, rather than making languages do stupid tricks as the basis of comparison, let's acknowledge that there is something that we can learn from just about every language. The fact is, all languages optimize for particular problem domains and I don't think that a universal programming language exists that would perform well on all tasks. Bray rapidly found out that Erlang isn't optimized for doing line-oriented I/O and it's regex library sucks. So what? While those problems could be eliminated from Erlang, the fact that Ericsson has deployed large telecom gear without having to fix those issues means that Erlang is ideally suited to its original programming domain.

Brad Delp of Boston, one of the all-time greatest singers in rock-and-roll took his own life on March 9, 2007. He was iconic. His falsetto was incredible. The debut Boston album was the first record I ever bought with my own money as a kid and remains my all-time favorite. I'm glad I actually got to see him perform live.

It's hard to imagine taking your own life but rock stars seem to do it far too often.

At the end of my previous discussion of my recent language binge, looking at Factor and Erlang, I mentioned that a perfect language would use Lisp as a base and then add some interesting features from Erlang. I mentioned Erlisp as being one step towards something like that but noted some of the limitations of Erlisp because it builds on top of Common Lisp. I noted that a ground-up rethink and true fusion would be more interesting in trying to solve the fundamental problem that Erlang is trying to solve (non-stop operation for years) but gaining some of the advantages of Lisp. I also mentioned that there was a Scheme-based experiment like Erlisp but I couldn't remember the name. Several commenters reminded me that it's called Termite.

So, I went back and read the paper about Termite again. I had done so a while ago but had only really retained the fact that a "Scheme-based Erlang" existed. I had forgotten its name and everything about its implementation.

The fact is, Termite is very close to what I would be looking for. I think it may do a better job than Erlisp because the Gambit Scheme implementation on which it is based has some nice thread capabilities (the ability to create "millions of threads on ordinary hardware"). Still, from some comments made at the end of the paper, it's clear that even the implementors think there are some limitations in Termite driven by the fact that it's implemented on top of Gambit rather than from the ground up. I don't know enough yet to understand where the limitations might be, and the authors only hint at them, never stating them outright.

Still, I think Termite is close.

I have been on a bit of a language binge lately. I have been playing around both with Factor and Erlang. I'm greatly impressed with both languages, though for completely different reasons.

If you haven't spent any time with Factor, the best way to describe it is "high-level Forth." Forth is a stack-based language that is great for embedded work because you can do a whole lot in a very small footprint. Forth, like Lisp, is interactive. Unfortunately (or perhaps fortunately, if you're a big Forth fan), Forth is fairly low-level in terms of its operation. Forth likes to think in terms of machine words. A lot of things like string handling are done with pointers. When data is stored on the stack, it's untyped and if you put parameters in the wrong order, it's easy to blow things up. In general, if you're an embedded programmer, Forth rocks. If you're an application programmer, I think Forth is the wrong tool. (Please, Forth people, don't write to me and tell me that Charles Moore, the creator of forth wrote his own CAD system all in Forth to design Forth machine chips. I know all that. Moore is a genius and there are few people in the world that could do what he has done.)

But what would happen if you took some of the ideas that Forth has: interactivity, high-level compiler that travels with the application, implicit stack-oriented parameters in function calls, and married that with some high level data types? What if those high-level data types had embedded typing, like Lisp, and so the system could determine when you're trying to add a number and a string or other type errors? Well, you'd end up with Factor.

Factor is the creation of Slava Pestov, another genius. Factor runs on just about any OS that runs on x86 and on several different processors under Linux. Factor has a great GUI development environment that takes some cues from Lisp Machines and CLIM (think lots of hyper-linked documentation and help, along with presentations). Factor has lots of example code, including such things as a web server on which the Factor web site runs. Check out the Factor web site for more info. There is a lot of goodness here.

As good as Factor is, however, I'm not sure it's my cup of tea for general application programming. I generally like Reverse Polish Notation (RPN) and have always used HP calculators all my life. That said, I just find it difficult to keep track of stack parameters through a long set of Forth or Factor function calls. It isn't that working this way is wrong in any sense of the word, but I find that I prefer named parameters where I can attach a description to a value, as in something like Lisp.

So, I decided to look at Erlang. Erlang was developed by Joe Armstrong at Ericsson to address problems in the telecom field. Luke Gorrie (another programming genius) has been telling me that I should get some Erlang experience for years now. Luke and I both work in the telecom/datacom field. Luke was at Bluetail with some of the key Erlang folks and later wound up at Nortel through a series of acquisitions. I had just left Nortel a while earlier.

Erlang's big claim to fame is concurrency. An Erlang program is composed of multiple "processes," similar in function to OS-level processes, but running in one or more virtual machines. Processes communicate by message passing. When the processes are all located in the same VM, this happens very quickly, but the nice thing is that the communication semantics are identical if the processes are running on multiple VMs, possibly on multiple computers. This makes it very easy to write Erlang programs that run in a distributed fashion.

Erlang is functional and keeps the memory of each process completely separate from the others ("shared nothing"). This improves the reliability of programs for several reasons. First, you don't have to worry about mutable data structures messing you up, violating an assumption without you knowing about it. Second, processes can't interact with each other except through message passing. If a process crashes for whatever reason, the other processes around it can generally continue to function until the crashed process is restarted.

And Erlang processes can crash a lot. This isn't because the programs are necessarily buggy, though that's one reason, but rather because Erlang actually encourages you to program only for the common case and to crash the moment your program detects any violation of its assumptions. The theory here is that rather than trying to continue a failed computation, it's often better for the overall system reliability for a process to give up and let other processes outside the failed process clean up the resulting mess. This is an interesting philosophy but it has a lot of merit.

Think about your typical PC. If you're a Linux user and you encounter a buggy program, how many of you will restart it in order to try to clean up the mess? If you have a buggy Windows system, how many of you reboot it? And generally, this works. Erlang simply takes the same idea and applies what we all know intuitively to be true , rebooting often fixes problems, to processes within a larger program.

Now, one of the neat things about Erlang crashes is that they produce a stack trace, much like you'd have in a Lisp system, such that a programmer isn't left with no data, scratching his head, wondering why the process crashed. You at least have some data to go on when you start debugging. Erlang aims to have systems that operate non-stop for years. To support this, Erlang supports hot code replacement.

So imagine you have the scenario where a customer reports a bug. Your support person asks the customer to send the log file, in which is the stack trace. A programmer examines the data and determines a fix. You recompile the program and send the customer the new version. The customer loads the new version while the old one is still running and the new version takes over seamlessly, with no downtime. Yes, Lisp has had many of these same ideas, and Erlang incorporates them.

Erlang is not all a bed of roses. I don't like the syntax. It's scary to say this, but I really do like and appreciate Lisp sexprs. Erlang has primitive macros, ala C, but nothing approaching the power of Lisp.

Pet peeve: Modern software reliability is horrible. I think Erlang at least gets its philosophy right. It's built to make highly reliable systems and it has the features to support that goal. From the get-go, it says, "Okay, we're going to be building systems that will operate non-stop for years. Of course we'll find bugs, but we need ways to be able to debug the system and then introduce changes to it without taking the system down. Further, bugs should only result in partial failures if at all possible. Where there are other tasks in the system unaffected by the bugs, they should remain available through the whole problem period." There are very few languages that could rise to that challenge. Lisp and Smalltalk come the closest, I think, but even with those there are issues of corrupted data structures hanging around and causing problems.

I know it's all in fashion right now, but I'm starting to contemplate my dream language. It looks a lot like CL or Scheme, but with some rather nice ideas borrowed from Erlang (and possibly Smalltalk and maybe even Ruby). In particular, the ability to have large numbers of concurrent processes, with message passing semantics. Keep the same sexpr syntax as Lisp and leave in all the introspection and meta-programming facilities. While there have been attempts to merge Erlang and Lisp concepts before, notably with Erlisp (and another one on the Scheme side whose name escapes me right now), I think what's needed is a ground-up rethink. Erlisp attempts to capture some of Erlang's process and message passing ideas in standard Common Lisp. Unfortunately, most Common Lisps don't have great multiprocessing capabilities, and none have thought through the "shared none" semantics that Erlang uses to increase reliability of the overall system. Semantically, in an Erlang system, sending a message to another process always creates a copy (of course, the copy may be optimized away by the compiler if the semantics are preserved). SETF has all sorts of abilities to trip you up in standard Common Lisp if you aren't careful.

Maybe to spare myself the embarrassment of trying to implement this (I'm not a Slava Pestov, a Charles Moore, or a Luke Gorrie), I'll base it on Arc. As soon as Paul is done, I'll get cracking on this new thing...

(Oh, and I'm not a Paul Graham either.)

Ever since I read the Acknowledgments section of the scsh manual, I have gotten a chuckle out of Olin Shivers. I particularly liked the 9mm Sig-Sauer comment. That started a phase where I read through just about everything I could find written by Olin. His Scheme-related complier articles are definitely interesting reading.

After seeing Olin present at Daniel P. Friedman's 60th birthday celebration, I gained a new respect for the man. Simply, Olin is smart, funny, and very articulate. I'll think of him the next time I'm out shooting my 9mm Sig-Sauer.

I had the nice opportunity to speak at the Pacific Free and Open Source Software Convention (PFOSSCON 2007) in Honolulu, Hawai'i, last weekend. This was done as part of my job at Vyatta. While I managed to acquire a cold after coming home, earlier this week, the conference was great. Kudos to Scott Belford, Jim Thompson, and Julian Yap for putting everything together.

Along with me speaking, the infamous Richard Stallman, Free Software Foundation, and Barton George, Sun Microsystems, spoke as well.

The summary is that after meeting Stallman, I wasn't very impressed. I have been reading his writings, such as the GNU Manifesto, since the mid-1980s. I'm not sure what I was expecting, but I didn't get it. My reaction to both the man and the message fall into a few broad categories. I think I'll blog in more detail about these things in the future sometime, but here are the broad reactions:

• Richard is not the best spokesman for the FSF. Sure, he founded the organization. And I wasn't expecting him to wear a shirt and tie. But to be honest, he's just a shabby guy. If you ever wonder why generally Free Software went nowhere inside corporate circles until Linux came along, this is one reason. Just at a personal level, Linus is a better spokes-model than Richard. He's just as geeky, without looking shabby.
• I react very negatively to Richard's use of "GNU/Linux" versus "Linux." Richard's contention is that all the userland for Linux came from GNU and thus Linux is really just the kernel while the whole OS should be called "GNU/Linux." Whether that is true or not, the fact is, Linus built his own operating system and he should have the right to name it. If Free Software is truly as Free as Richard wants to claim it is, naming should be one of the things where the Freedom shows. Put another way, I have no issue if the FSF and GNU want to put out their own Linux distribution and name it "GNU/Linux," or even just "The GNU System," that's fine. I have big problems with Richard trying to convince me that I should tack "GNU" onto the front of "Linux" for some reason. I would worry that at any time Richard might claim that the use of GPL code might warrant changing the name of your application to "GNU/Foo." This is all the more annoying because Hurd still sucks and without Linux the overall goals of the original GNU project would still be unmet. Simply, this whole naming thing is a big case of sour grapes that Linux has been successful and has eclipsed GNU in terms of relevancy.
• My other big takeaway from the conference is that I don't agree with Richard on the fundamental philosophy of free software. There, I said it. HORRORS!. And here I am at an open source company. Sorry, while I like open source and believe there are certain advantages to it, I specifically reject Richard's moral basis for Free Software. Richard tries to portray access to source code and redistribution for no charge as abstract moral rights that every person should have, something akin to "life, liberty, and the pursuit of happiness." Using proprietary software, Richard says, is to make an immoral choice. I don't buy it. I like using Linux better than Windows for a variety of reasons, but I don't believe the people at Microsoft are immoral because they choose to keep their source code to themselves (I do think some of Microsoft's monopoly business practices are immoral, but those are another matter). In my opinion, this is why the FSF has been largely ineffectual in getting people to think about "Free Software" as opposed to "Open Source" (a term which Richard rejects as missing the point). Simply, I don't think most of the developers or users of software see a philosophical, moral argument to made with respect to closed software. Again, I think Linus here wins hands-down as the leader of a large software movement because he focuses on the real issue: the open-source development model allows users to have more control and harnesses the innovation of a larger number of creative developers.

Well, that's all for now. In spite of my disagreement with Richard, I bought a copy of Free Software, Free Society from him and had him autograph it.

When I first saw C++ back in the mid-1980s, it looked pretty interesting. Then it started to grow into a horrible monster. I was just reading about the D programming language today. All I can say is, "Wow!" I wish they had this way back when. This would seem to correct most of what I find distasteful about C, C++, and Java, while keeping most of the things I like--to the point where I would actually consider using it for certain projects.

The things I hate most about those programming languages are:

C

• No objects
• Manual memory management only
• Lame macros
• Having to define header files

C++

• Complex, obscure syntax that I can't ever remember completely without a copy of The C++ Programming Language sitting next to me
• Complex resource management schemes to avoid leading memory/etc. in the presence of exceptions
• Did I mention that I hate the complex syntax?

Java

• (Almost) Everything is an object (but not quite)
• Having to run in a VM with bytecodes even when you're writing something for a single platform and don't need or want to
• No ability for structures to overlay memory for efficient access to C and network data wire protocol data structures, forcing the programmer to do bit-twiddling with ByteArray and shifts and masks

In short, D looks interesting. Is it Lisp and would I go back to using D as my main programming language? No. It still doesn't have macros the way Lisp does and it doesn't have sexprs (yes, I really do like the parenthesis), but it is a better follow-on to C than C++ and still hits the system-programming domain that Java missed so bady. It seems like something to keep in one's hip pocket.

Welcome to a new year. I have been swamped the past six months with work and family. I'm still getting a bit of time to work on Lisp-related stuff, but not nearly as much as I'd like.

The biggest casualty has been blogging. I just haven't had much time to sit down and compose something cogent. Back in October, Xach added the sparklines display to Planet Lisp. My first thought regarding my own sparkline was "Yikes! I've seen people at the morgue with better EEGs than that!" Hopefully, Xach won't de-list me.

So, to kick off a new year, here's a post to make my sparkline wiggle a bit.