The other day, I realized that there wasn't a great place to discuss Lisp that was using a post-Y2K sort of interface. Sure, comp.lang.lisp has been around forever, and through Google Groups it has an HTTP interface, but when you post to Usenet you become an instant spam magnet. And new users think an all text-mode interface with no markup is just so 1985. And yes, there's always #lisp on Freenode.net IRC, which is great for realtime, but bad for searching for coherent answers.

So, I created Lisp Forum: www.lispforum.com

The content is a bit sparse right now. That's where you come in. Please register, hang out, and participate. My goal is to make this a friendly place with a bit less Smug Lisp Weenie-ness than comp.lang.lisp, suitable for both newcomers and old pros.

The other day, I discussed languages which I thought were timeless. Among them, I listed Lisp, C, and Forth. After writing that posting, I spent some time playing with Forth again. Today, while browsing Reddit, I stumbled on this interview with Charles Moore, the creator of Forth.

Forth is one of those languages, like Lisp, that I'd recommend that everybody study at least for a time. Even if you don't walk away believing it's the Language to End All Languages, you'll be better off for the experience. In fact, Forth shares a lot of fundamental attributes with Lisp while at the same time appearing almost completely different to a programmer writing code.

Some of the shared attributes include:

1. Forth and Lisp both erupt from a very small nucleus of fundamental constructs. As Alan Kay has described Lisp as being "Maxwell's equations of sofware," similar statements would also apply to Forth. Both Forth and Lisp are fundamentally simple as a result. Sure, the libraries could be huge, but learning the actual language itself, the core rules, requires no more than a few minutes for each language. With both languages, there is a set of advanced rules (things like macros for Lisp or compiling words for Forth), but the basics are trivial.
2. Forth and Lisp are duals of each other when it comes to their syntax. In both cases, the programmer is essentially handing the system a direct representation of the parse tree. The parsers for each language are trivial. Lisp uses prefix notation, whereas Forth uses postfix: "(+ 1 2)" vs. "1 2 +" for example. The use of prefix notation semi-requires delimiters to be inserted, giving us Lisp's beloved/hated parenthesis. With Forth, all computation revolves around the stack. Because the operation always occurs after the parameters are pushed, you don't need the delimiters. Words simply consume whatever parameters from the stack that they want to. One implication of this, however, is that you can do things like "(+ 1 2 3 4)" in Lisp. In Forth, this ends up being either "1 2 + 3 + 4 +" or "4 3 1 2 + + +".
3. Forth and Lisp are both extensible. When you create a new function or macro in Lisp, you're extending the language itself. Your new function or macro is a peer with everything else in the language, not a red-headed stepchild. Similarly, with Forth, a word is a word is a word. You can create Forth words that interact with the compiler and do all sorts of crazy stuff. Most Forth systems also include an assembler so you can create high-performance, primitive Forth words as well.
4. Forth and Lisp are both interactive. They both use a REPL. Forth doesn't call it that, but that's what it is. The benefits of this are similar in both. You tend to code a little, then test a little, then code a little, then test a little. In the interview with Moore linked above, you can see where he talks about the speed at which things got developed as a result of the interactivity of his Forth system.
5. Forth and Lisp both include a compiler. I guess this really isn't a fundamental attribute of Lisp itself (you could be fully interpreted), but most Lisp systems do have a compiler. In some cases, that compiler can be pretty simple and primitive. In other cases, it could be very sophisticated (CMUCL/SBCL). With Forth, the compiler for threaded code is both fundamental and at the same time trivial. More sophisticated Forth systems can create more complex compilers (subroutine threading, superoperations, etc.), but those are not required.

All that said, Forth and Lisp are also very different:

1. Typically, Forth operates at the machine level, with very direct exposed representations for objects. Forth programmers think in terms of machine words, bits and bytes. In some parts of a given program, a given bit pattern will represent a character or pointer or whatever, but from Forth's point of view they're all just bit patterns in a machine word. In contrast, Lisp programmers operate a higher levels of representation with first-class status for things like symbols, numbers, characters, etc.
2. As a result of this "level difference," Forth programs are more memory efficient than Lisp programs, but they're also more dependent on the underlying machine fundamentals. For instance, if you changed the word size of the machine, a Lisp programmer probably wouldn't be aware of it. A Forth programmer might have to scramble to rewrite a good portion of the code. But very useful Forth programs are measured in KB of size, not MB.
3. A big difference that typifies the issue of operating at the machine level vs. higher levels of abstraction is that Forth doesn't include any GC capabilities. All memory management must be done manually by the programmer.
4. Forth isn't very tolerant of program bugs. Because you're operating at the machine level, when things go wrong, you might end up with a crashed machine. The Forth response to that is to just push the reset button and reload the system. Because the compiler is so fast, you'll be back to where you were before the crash in no-time. In contrast, Lisp makes a lot of effort to land the programmer in an interactive debugging shell when it detects an error condition.

Note that people have proposed systems which bridge between the two worlds. Factor is basically a Forth stack machine and syntax, augmented with high-level, Lisp-like data types and a GC. The result is a system which delivers Forth-like syntax with a Lisp-like debugging and development environment. Depending on your point of view, you'll either think this is the best of both worlds or the worst.

For me personally, I like both Forth and Lisp, but I'd use them in completely separate domains. If I was working on a deeply embedded project, where I'd want to be close to the machine architecture and where I had only a few KB into which to implement the program, I'd choose Forth. If I was writing a large application that would be running on a server with GBs of memory, I'd choose Lisp. Each works well within its target domain and the advantages of each are nearly the same: a small, extensible language with an iterative, interactive development environment.

As for systems like Factor, for me it's a "tweener" that doesn't fit my needs. By getting away from the machine details, adding high-level data types, GC, etc., Factor necessarily pushes itself out of the embedded world. You simply won't have microcontrollers running Factor. And if I'm going to be running on a system with an underlying operating system, a large graphical display, and GB of memory, I'd rather do my development in Lisp. While I like Forth, I find that Lisp's sexpr notation more closely matches my thinking model. With Forth's implicit stack, I have to be thinking all the time about what data values are at what positions on the stack. I'd choose to deal with that for an embedded design to get all the other attributes of Forth in that environment, but with fewer constraints, I'd choose Lisp over a "Forth with high-level data types and GC" like Factor.

Now, that's just me. For you, Factor may be the ticket. Slava Pestov is not an idiot (and you can quote me on that, Slava). As Factor's creator, he has obviously built a system that works well for him. Other people who seem to have far better programming skills than I do are working with Factor, too. The development environment they have put together seems to have borrowed a lot of ideas from Lisp machines, and I could see the Factor environment being really productive.

Whatever you choose, realize that all of these languages have some things they share. And fundamentally, both Lisp and Forth are timeless.

I started thinking about it today. What if I was stuck on the proverbial desert island? What programming language would I want to have to hammer out that critical calculation that would somehow return me back to civilization?

I know this is heresy, but I think I'd choose C.

"WTF!?!?!" I hear you cry. "This is a Lisp blog, dammit. You can't choose C." Well, you're right. But let me ramble for a second.

In my thinking this afternoon, I started to consider which programming languages are really timeless? Not just popular, mind you, but timeless. There aren't that many. COBOL was popular once. It wasn't timeless. PERL is certainly popular, but given the changes happening in PERL 6, it's not timeless either. Python? A very fun language, but not timeless. Ruby? Very hip and cool, but not timeless. How about Java? I once read something from the Long Now folks saying that they would be using Java for any programming because it was a language that could stand the test of time because of WORA. (If somebody can point me to a reference for this, please do. I was Googling madly for it this afternoon, but I can't seem to find it. I believe it was something Bill Joy wrote.) I didn't believe it then, and I certainly don't believe it now.

Nope, there is a huge difference between popularity and timelessness. I think C is timeless. C has been called "high level assembler." There are few other languages that do such a great job of raising you just slightly out of the world of bits, bytes, and machine registers, just enough to give you high level conditionals and loops, but not far enough that you can't get back to machine words.

Lisp is also timeless. Alan Kay has said that Lisp is "Maxwell's Equations of software." He's right. There is something that is so fundamental about Lisp that it simply cannot go away. It's far more than the parenthesis or the lists or the macros or any of that stuff that Lispers love. It's more than Common Lisp or Scheme or MacLisp or Emacs Lisp. It's all that stuff and more. At its core, Lisp is so compact and powerful as to be timeless. From McCarthy's original paper springs forth the whole world of every program that can be written.

In fairness to Kay, Smalltalk is pretty close to being timeless, too. In a certain sense, Smalltalk lives on in Ruby, but it's really the idea of message passing rather than Smalltalk specifically. Insofar as Smalltalk is synonymous with message passing (which it really isn't), it's timeless.

FORTH is timeless. The basics of a stack machine are pretty fundamental. The various FORTH words and implementation techniques may change (c.f. Factor), but the fundamentals are still recognizable as FORTH.

And that's why I'd choose C as my desert island language. Because in less than 10 pages of C, I can bootstrap a basic Lisp interpreter (or FORTH interpreter, for that matter). And with Lisp, I can write everything else. When PERL and Python are as dated as COBOL and Pascal, we'll still have Lisp. And it'll still be very recognizable as Lisp.

A couple years ago, I exchanged email with Dan Weinreb about Lisp machines. Dan was one of the founders of Symbolics and wrote the Emacs variant on the Symbolics machines. During our email exchange, I asked Dan about some of the statements and writings of Richard Stallman with respect to Symbolics and Richard's characterization of Symbolics as one of the bad guys in the Lisp machine competition with LMI. This was interesting to me because the Symbolics vs. LMI competition figures prominently in RMS's reasoning to create GNU and the whole concept of Free Software.

Of course, there is always another side to the story, and Dan definitely disagreed with Richard's characterization of the situation. Having interacted with both Dan and Richard, I tend to believe Dan's version more than Richard's. But that's just my opinion. I told Dan that I thought he should document his version of events because the only side of the tale that was being told up until that point was Richard's, and without any evidence to the contrary, his side was being accepted as gospel.

At the time, Dan basically said that he just didn't feel like it. He was tired of the whole thing and just wanted to let Symbolics rest in peace. Having founded an unsuccessful startup myself, I know exactly where he is coming from. The last thing you want to do is relive things by going through the act of documenting history. Dan told me that he would probably document things at some point, but not at that moment.

Well, I'm pleased to report that Dan has started a new blog and one of his first postings was his version of the Symbolics vs. LMI competition. It directly refutes some of RMS's assertions and provides a good counterbalance to this controversy. I'm glad that Dan published his side of the story. Who you believe, well that's up to you. If you're an RMS fanboy, Dan's account of things is unlikely to change your mind. But at least history will record that there were two sides to the story.

As an aside, I'm glad Dan has started blogging in general. He struck me as a very smart cookie and I'll definitely be adding his RSS feed to my list. Whether he blogs often or not, I'm sure I'll be the better for reading his words.

Hat tip to Xach for noticing Dan's blog.

There was a day when I didn't know what a bignum was or why I would possibly want one. I was fat, dumb, and happy with basic fixed-precision integers in C and Java land. When I first started playing with Common Lisp, I thought, "Wow, that's a neat trick that I can actually evaluate (factorial 100) and it works correctly, but how often am I going to be doing that??"

Boy, was I wrong. I'm working on a project in Emacs Lisp in my spare time and I suddenly have a need to deal with integers that are greater than the anemic 29-bit fixnums that Emacs Lisp supports. While 29 bits might be okay for working with character positions in a buffer, it falls flat on its face whenever you want to manipulate anything from the outside world. In this case, 29 bits is too small to handle any 32-bit quantity, let alone some of the 64-bit stuff I want to deal with. I'm kind of amazed that RMS didn't just implement 32-bit boxed integers if he wasn't going to do full bignums.

I found that calc.el includes support for its own version of bignums and I might be forced to use those. Painful, though. I guess the old saying is true--you don't miss something until it's gone.

Does any Emacs guru know if bignums are being considered as a standard Emacs Lisp datatype for a future Emacs version? I'm currently working in Emacs 22.1 (Fedora 7). Given the release schedule of Emacs versions, any future support won't help me, but it would at least be comforting to know that other people see the same need.

I spent a while last night reading about Tim Bray's adventures with Erlang. Tim started investigating Erlang as a part of his "Wide Finder Project" in which he's looking for programming languages that will help accelerate common tasks on the soon-to-be-very-popular CPUs with many cores but slower clock rates.

Tim works at Sun, and so this question and project makes perfect sense in light of Sun's Niagra and T2 processors with many cores and CMT. It also makes perfect sense in light of Intel's Tera-scale computing initiative where they have demonstrated chips with 80 cores. In short, the future is going to be very, very parallel, and we had better come to terms with that.

Unfortunately, the modern multi-threaded programming paradigms are ill-equipped to take advantage of these modern processors. Most popular programming languages have the same simple threads+locks paradigm that was popularized with pthreads and Java. While this works, it doesn't work well. As with so many things in programming, the threads+locks paradigm forces programmers to remember a whole lot of crufty details, otherwise they will produce code with very subtle bugs that is very, very difficult to debug. Put another way, threads+locks is the parallel programming equivalent of manual memory management. In the same way that GC manages memory in many modern programming languages, we need an equivalent to help programmers manage parallel programming.

Erlang has many features that help it work well on multi-core systems. The language is inherently multi-threaded and concurrent and relies on threading almost down to the core (you can write non-threaded Erlang programs, but the language makes it so easy to use threading that you'd hardly want to). I have talked about Erlang before a few months ago (here and here).

So, here we have Tim Bray asking a perfectly sensible question, "What programming language is going to help us programmers exploit the soon-to-be-commonplace multi-core CPU?" Erlang is certainly a potential answer to that problem. Unfortunately, Bray decided to pick a problem for which Erlang is particularly unsuited and then compares Erlang to another programming language that optimized for the problem, albeit without any parallel programming support--Bray picked a simple web log analysis problem.

The web log analysis problem is one that is well suited to Perl, Python, or Ruby. One might even say that these languages were virtually created to solve problems in this exact domain. If Perl does anything really well, it processes text with regular expressions. Python is a bit more clunky than Perl or Ruby in terms of regex syntax, but still does quite well. Ruby was created to solve many of the same problems as Perl, but with a better object model and saner syntax (IHMO).

Erlang, on the other hand, was created to develop 24x7x365, long-running telecom software. This is stuff that aims to have downtimes measured in a handful of minutes per year and that must be able to be upgraded on the fly and recover from any faults or failures. In short, Erlang aims to help programmers write code that can take a bullet to the head, recover, and keep on doing its thing, later allowing the programmer to find the fault, fix it, and upgrade the system, all while staying up and doing its thing. This is a hugely complex task, I can assure you. And Erlang conquers it in fairly good fashion.

So, when Bray decides to try Erlang on this problem, he naturally finds that it blows chunks. His first attempt doesn't even attempt to use any Erlang threading features, which of course defeats the whole reason for the investigation in the first place. Knowing enough about Erlang to be dangerous, I found myself saying, "Well, duh! Why did you expect that to get great performance?"

Overall, I'm getting pretty tired about these simplistic comparisons that people do between programming languages. It always feels like they're an elaborate sort of "gotcha." Step 1, pick a task that runs particularly well on the evaluator's most familiar language (Bray picks web log analysis, which runs quite well in Ruby, his choice language). Step 2, pick a victim language. When the victim language doesn't measure up, yell "WTF?!?! [insert language] sucks." Now, in truth, Bray didn't do that last part, but you have seen the pattern other times, I'm sure (witness the number of people that list the Computer Language Shootout as justification for almost anything).

What would have made a better comparison is writing a multi-threaded web server in both Erlang and Ruby and see which server is able to deliver the best performance to 10,000 active clients with widely varying download speeds. I'd be willing to bet that Erlang does a better job. No, I wouldn't even suggest writing a 24x7x365 telecom switch in Ruby; as fine as Ruby is, Erlang would win that hands down.

So, rather than making languages do stupid tricks as the basis of comparison, let's acknowledge that there is something that we can learn from just about every language. The fact is, all languages optimize for particular problem domains and I don't think that a universal programming language exists that would perform well on all tasks. Bray rapidly found out that Erlang isn't optimized for doing line-oriented I/O and it's regex library sucks. So what? While those problems could be eliminated from Erlang, the fact that Ericsson has deployed large telecom gear without having to fix those issues means that Erlang is ideally suited to its original programming domain.

Brad Delp of Boston, one of the all-time greatest singers in rock-and-roll took his own life on March 9, 2007. He was iconic. His falsetto was incredible. The debut Boston album was the first record I ever bought with my own money as a kid and remains my all-time favorite. I'm glad I actually got to see him perform live.

It's hard to imagine taking your own life but rock stars seem to do it far too often.

At the end of my previous discussion of my recent language binge, looking at Factor and Erlang, I mentioned that a perfect language would use Lisp as a base and then add some interesting features from Erlang. I mentioned Erlisp as being one step towards something like that but noted some of the limitations of Erlisp because it builds on top of Common Lisp. I noted that a ground-up rethink and true fusion would be more interesting in trying to solve the fundamental problem that Erlang is trying to solve (non-stop operation for years) but gaining some of the advantages of Lisp. I also mentioned that there was a Scheme-based experiment like Erlisp but I couldn't remember the name. Several commenters reminded me that it's called Termite.

So, I went back and read the paper about Termite again. I had done so a while ago but had only really retained the fact that a "Scheme-based Erlang" existed. I had forgotten its name and everything about its implementation.

The fact is, Termite is very close to what I would be looking for. I think it may do a better job than Erlisp because the Gambit Scheme implementation on which it is based has some nice thread capabilities (the ability to create "millions of threads on ordinary hardware"). Still, from some comments made at the end of the paper, it's clear that even the implementors think there are some limitations in Termite driven by the fact that it's implemented on top of Gambit rather than from the ground up. I don't know enough yet to understand where the limitations might be, and the authors only hint at them, never stating them outright.

Still, I think Termite is close.

I have been on a bit of a language binge lately. I have been playing around both with Factor and Erlang. I'm greatly impressed with both languages, though for completely different reasons.

If you haven't spent any time with Factor, the best way to describe it is "high-level Forth." Forth is a stack-based language that is great for embedded work because you can do a whole lot in a very small footprint. Forth, like Lisp, is interactive. Unfortunately (or perhaps fortunately, if you're a big Forth fan), Forth is fairly low-level in terms of its operation. Forth likes to think in terms of machine words. A lot of things like string handling are done with pointers. When data is stored on the stack, it's untyped and if you put parameters in the wrong order, it's easy to blow things up. In general, if you're an embedded programmer, Forth rocks. If you're an application programmer, I think Forth is the wrong tool. (Please, Forth people, don't write to me and tell me that Charles Moore, the creator of forth wrote his own CAD system all in Forth to design Forth machine chips. I know all that. Moore is a genius and there are few people in the world that could do what he has done.)

But what would happen if you took some of the ideas that Forth has: interactivity, high-level compiler that travels with the application, implicit stack-oriented parameters in function calls, and married that with some high level data types? What if those high-level data types had embedded typing, like Lisp, and so the system could determine when you're trying to add a number and a string or other type errors? Well, you'd end up with Factor.

Factor is the creation of Slava Pestov, another genius. Factor runs on just about any OS that runs on x86 and on several different processors under Linux. Factor has a great GUI development environment that takes some cues from Lisp Machines and CLIM (think lots of hyper-linked documentation and help, along with presentations). Factor has lots of example code, including such things as a web server on which the Factor web site runs. Check out the Factor web site for more info. There is a lot of goodness here.

As good as Factor is, however, I'm not sure it's my cup of tea for general application programming. I generally like Reverse Polish Notation (RPN) and have always used HP calculators all my life. That said, I just find it difficult to keep track of stack parameters through a long set of Forth or Factor function calls. It isn't that working this way is wrong in any sense of the word, but I find that I prefer named parameters where I can attach a description to a value, as in something like Lisp.

So, I decided to look at Erlang. Erlang was developed by Joe Armstrong at Ericsson to address problems in the telecom field. Luke Gorrie (another programming genius) has been telling me that I should get some Erlang experience for years now. Luke and I both work in the telecom/datacom field. Luke was at Bluetail with some of the key Erlang folks and later wound up at Nortel through a series of acquisitions. I had just left Nortel a while earlier.

Erlang's big claim to fame is concurrency. An Erlang program is composed of multiple "processes," similar in function to OS-level processes, but running in one or more virtual machines. Processes communicate by message passing. When the processes are all located in the same VM, this happens very quickly, but the nice thing is that the communication semantics are identical if the processes are running on multiple VMs, possibly on multiple computers. This makes it very easy to write Erlang programs that run in a distributed fashion.

Erlang is functional and keeps the memory of each process completely separate from the others ("shared nothing"). This improves the reliability of programs for several reasons. First, you don't have to worry about mutable data structures messing you up, violating an assumption without you knowing about it. Second, processes can't interact with each other except through message passing. If a process crashes for whatever reason, the other processes around it can generally continue to function until the crashed process is restarted.

And Erlang processes can crash a lot. This isn't because the programs are necessarily buggy, though that's one reason, but rather because Erlang actually encourages you to program only for the common case and to crash the moment your program detects any violation of its assumptions. The theory here is that rather than trying to continue a failed computation, it's often better for the overall system reliability for a process to give up and let other processes outside the failed process clean up the resulting mess. This is an interesting philosophy but it has a lot of merit.

Think about your typical PC. If you're a Linux user and you encounter a buggy program, how many of you will restart it in order to try to clean up the mess? If you have a buggy Windows system, how many of you reboot it? And generally, this works. Erlang simply takes the same idea and applies what we all know intuitively to be true , rebooting often fixes problems, to processes within a larger program.

Now, one of the neat things about Erlang crashes is that they produce a stack trace, much like you'd have in a Lisp system, such that a programmer isn't left with no data, scratching his head, wondering why the process crashed. You at least have some data to go on when you start debugging. Erlang aims to have systems that operate non-stop for years. To support this, Erlang supports hot code replacement.

So imagine you have the scenario where a customer reports a bug. Your support person asks the customer to send the log file, in which is the stack trace. A programmer examines the data and determines a fix. You recompile the program and send the customer the new version. The customer loads the new version while the old one is still running and the new version takes over seamlessly, with no downtime. Yes, Lisp has had many of these same ideas, and Erlang incorporates them.

Erlang is not all a bed of roses. I don't like the syntax. It's scary to say this, but I really do like and appreciate Lisp sexprs. Erlang has primitive macros, ala C, but nothing approaching the power of Lisp.

Pet peeve: Modern software reliability is horrible. I think Erlang at least gets its philosophy right. It's built to make highly reliable systems and it has the features to support that goal. From the get-go, it says, "Okay, we're going to be building systems that will operate non-stop for years. Of course we'll find bugs, but we need ways to be able to debug the system and then introduce changes to it without taking the system down. Further, bugs should only result in partial failures if at all possible. Where there are other tasks in the system unaffected by the bugs, they should remain available through the whole problem period." There are very few languages that could rise to that challenge. Lisp and Smalltalk come the closest, I think, but even with those there are issues of corrupted data structures hanging around and causing problems.

I know it's all in fashion right now, but I'm starting to contemplate my dream language. It looks a lot like CL or Scheme, but with some rather nice ideas borrowed from Erlang (and possibly Smalltalk and maybe even Ruby). In particular, the ability to have large numbers of concurrent processes, with message passing semantics. Keep the same sexpr syntax as Lisp and leave in all the introspection and meta-programming facilities. While there have been attempts to merge Erlang and Lisp concepts before, notably with Erlisp (and another one on the Scheme side whose name escapes me right now), I think what's needed is a ground-up rethink. Erlisp attempts to capture some of Erlang's process and message passing ideas in standard Common Lisp. Unfortunately, most Common Lisps don't have great multiprocessing capabilities, and none have thought through the "shared none" semantics that Erlang uses to increase reliability of the overall system. Semantically, in an Erlang system, sending a message to another process always creates a copy (of course, the copy may be optimized away by the compiler if the semantics are preserved). SETF has all sorts of abilities to trip you up in standard Common Lisp if you aren't careful.

Maybe to spare myself the embarrassment of trying to implement this (I'm not a Slava Pestov, a Charles Moore, or a Luke Gorrie), I'll base it on Arc. As soon as Paul is done, I'll get cracking on this new thing...

(Oh, and I'm not a Paul Graham either.)

Ever since I read the Acknowledgments section of the scsh manual, I have gotten a chuckle out of Olin Shivers. I particularly liked the 9mm Sig-Sauer comment. That started a phase where I read through just about everything I could find written by Olin. His Scheme-related complier articles are definitely interesting reading.

After seeing Olin present at Daniel P. Friedman's 60th birthday celebration, I gained a new respect for the man. Simply, Olin is smart, funny, and very articulate. I'll think of him the next time I'm out shooting my 9mm Sig-Sauer.