The Road to Enlightenment Is Littered with Irritating, Superfluous Parentheses

Originally published at http://bendiken.net/2007/01/01/the-road-to-lisp and the ALU Wiki.

I, Arto Bendiken, do solemnly offer these my responses to The Road to Lisp Survey:

When did you first try Lisp seriously, and which Lisp family member was it?

I read Peter Seibel’s Practical Common Lisp and Paul Graham’s On Lisp sometime in mid-2005 or thereabouts. Both proved to be useful introductions to Common Lisp and gave me an initial appreciation of the unique characteristics of Lisp-derived programming languages.

After being sufficiently enlightened by those two books, I moved onward to Scheme. This came about primarily as a result of struggling my way through the Wizard Book and watching the 6.001 video lectures, as well as perusing various online tutorials and papers. I’ve been programming in Scheme ever since, with the occasional foray into Common Lisp.

Where did your road originate?

I first started programming at the tender age of 11, with the unholy combination of BASIC and C. (Yes, I’ve since worked hard to overcome the brain damage that caused, thank you very much for asking.)

Having always been fascinated by the concept of artificial intelligence, I gophered all the AI programs I could find as soon as the Internet hit the scene (which was just a couple of years later, in my case). To my dismay, these programs were pretty much all written in an obscure, half-forgotten language called LISP (the name was still usually uppercased back then, as I recall).

At that point in time, this was immensely frustrating. I wanted to write my ELIZA clone and my first neural net in C, dammit, and I was pretty sure that example programs written in all uppercase text and containing lots and lots of irritating, obviously superfluous parentheses wouldn’t help me learn how to do so. Oh, the impatience of youth.

I proceeded to ignore Lisp for the next decade, wondering aimlessly through the vast wasteland of mainstream programming technologies.

What led you to try Lisp?

It took half a dozen other programming languages, and seven years of full-time software development, before I came full circle. The key ingredients were Graham’s writings, Seibel’s book, and familiarity with Ruby. But not to get ahead of myself; the story resumes at the turn of the millennium.

At that time, I was employed as a J2EE developer at a largish company, working on various horribly overengineered, enterprisey systems. It was bad enough, at times, that I had almost come to regret my choice of a career. Writing software was apparently a joyless and dull activity, with little room for individual creativity. Half the time was wasted fighting the bloated toolchain, and the other half was lost to red tape, useless meetings or just the constant interruptions in the cubicles we developers worked in. Thank goodness for overtime. (But of course, it all paid very well; selling your soul tends to.)

In my copious free time, I worked on a 3D physics engine and did some Linux kernel hacking in C and assembler; just enough to keep the old passion for programming alive. Then one day, though I didn’t know it at the time, I happened to take a first small step in the right direction at a crucial crossroads: I needed to write a specialized web crawler and had read somewhere that Google was using a programming language called Python, so out of pure curiosity I decided to try and implement my crawler in Python.

After embarking on this project, it didn’t take me long to understand how much vastly more productive Python coding was compared to Java’s endless boilerplate drudgery. Python truly sold me on the benefits of dynamically-typed languages and rapid prototyping. I began to see that many of the sacred GoF design patterns were not, in actuality, grand universal truths of software engineering, but simply collateral damage caused by incidental limitations in the abstractive power and object model of certain manifestly-typed programming languages.

Fast forward a couple more years. I had, naturally, ditched Java (and the old job as a code monkey) for good, and was hacking away with Python, mostly happily. However, I had another date with destiny coming up: a superficially similar scripting language called Ruby was on the rise, and to find out what all the fuss was about, I went ahead and learned that, too.

The similarity between Python and Ruby turned out to be skin-deep. After grokking the initially strange new idioms in Ruby, I recognized that it was a yet more powerful language than Python — the abstraction gap wasn’t anything as wide as it had been from Java to Python, but it was definitely there. The Ruby community also offered a (for me) more attractive philosophy than Python’s “our way or the highway”, so I promptly chose the highway — good riddance, Python.

By this time I had already stumbled across Graham’s eloquent writings regarding the Blub Paradox and the power continuum of programming languages, and I’d also read his book Hackers and Painters. From my own first-hand experience, I was thoroughly comfortable with the (at times surprisingly controversial) notion that programming languages indeed do vary in expressive power.

It was also fast becoming obvious to me that I was something of a power junkie: though I liked Ruby just fine, the grass is always greener on the other side, and soon enough I was curious to find out if there really was something yet more powerful out there. Researching the matter extensively brought me, of course, squarely back to dear Old Faithful, per Graham’s very words. Oh, the horror.

It took some serious mental juggling to overcome the deeply-rooted trauma of my unfortunate first encounter with the language, and to understand that despite the widespread meme that Lisp was dead, it wasn’t really any deader than usual that in fact, being not an invention but a discovery, Lisp is essentially unkillable and is destined to rise time and again to the murmured incantations of Greenspun’s Tenth in the pallid, ghostly glow of a display monitor in a dark room, forever and ever.

More seriously, from my experience with Python and Ruby I had already learned not to judge a book by its cover. After all, I had overcome the weird significant whitespace to hack in Python, as well as Ruby’s equally weird block syntax to think in closures. I could give a shot at tackling lots of irritating, possibly superfluous parentheses, right? (Besides, I like horror stories, so I knew Lisp would haunt me if I didn’t.)

Ruby proved to be the perfect penultimate stepping stone, priming me for Lisp: not only did the language familiarize me with symbols, closures and the basics of metaprogramming, but many eminent Ruby hackers were also erstwhile Lispers. Ruby’s Lisp heritage often came up in discussions, and was mostly presented in a favorable light. As Matz, Ruby’s creator, explained: “Some may say Ruby is a bad rip-off of Lisp or Smalltalk, and I admit that. But it is nicer to ordinary people.”

With uncanny timing, Seibel’s Practical Common Lisp was published online at just the right moment to provide instant gratification for my curiosity. I finished the book in a couple of days, followed it up with a good dose of On Lisp, and as they say, that’s all she wrote. I had been lost, and now I was found again.

What other languages have you been using most?

For systems programming, I still use C — but only as a glorified, somewhat portable assembly language. While I think it fulfills this limited role adequately, I have lately been looking into replacing it with a static subset of Lisp (perhaps something like Pre-Scheme).

My daily work, though, mostly involves web development, where my tools of the trade are Ruby and PHP. I use Ruby because it’s an enjoyable and thorougly dynamic language, essentially Smalltalk for the mainstream. It’s nowhere near as “pure” as Smalltalk, but still manages to swing some pretty powerful metaprogramming facilities (as demonstrated by the Rails web framework).

I use PHP simply because it is ubiquitous and, despite the language’s obvious failings, there’re some very useful web applications written in PHP. It’s a dynamic language (which makes it tolerable), but not a particularly good one; if I had to describe PHP in one adjective, it would be “impoverished”. That only applies to its abstractive capabilities, though, and not to the wealth of available libraries and software, or to the massive commercial and open-source ecosystem thriving around the language. Still, as Java could be described as the present-day COBOL, so PHP is essentially BASIC or Pascal for the web; if used in isolation, it will surely cripple the mind.

Other languages I use on a semi-regular basis are Python and JavaScript.

Something I’ve noticed in the past year is that Lisp idioms have begun permeating through to my code in other programming languages. This has really thrown a spotlight on the specific limitations of the various languages I use, because so few of them can keep up (Ruby and JavaScript perhaps being the only real contestants, on a good day).

Most recently, I’ve been dabbling in Haskell and Erlang to improve my comprehension of best-of-breed functional programming and distributed computing techniques, respectively; moreover, I’m keeping close tabs on Factor and other modern Forth-derivatives, and of course, I can’t help but admire what the Smalltalkers have built with Squeak. (I suppose once you’ve reached the Source, being a polyglot becomes second nature.)

How far have you gotten in your study of Lisp?

Right now, I know just enough to realize how much I have yet to learn. Lisp and the lambda calculus served as an eye-opening introduction to the more theoretical aspects of computation and information theory, and have forced me to accept that despite over a decade of practical experience, I didn’t really have a clue what programming was fundamentally all about. I may still not have a crystal-clear grasp of the Big Picture, but at least I’m on the right path.

I’ve become fairly experienced in Scheme, but am yet somewhat less so in Common Lisp. Despite the early BASIC-induced brain damage, I did manage to successfully rewire my neural pathways to handle all the central new concepts, from higher-order functions and functional programming to closures, multiple dispatch in CLOS, continuations, and, of course, macros and syntactic abstraction. I’ve learned first-hand what Dijkstra meant when he claimed that Lisp has assisted programmers in thinking previously impossible thoughts.

Currently, I’m busy implementing several small-scale, experimental metacircular interpreters and am pushing further into mastering the wealth of compilation techniques developed over the past four decades. I’m on my second reading of Christian Quiennec’s Lisp in Small Pieces, and am looking forward to soon tackling Peter Norvig’s Paradigms of Artificial Intelligence Programming, Gregor Kiczales’s The Art of the Metaobject Protocol, and Chris Okasaki’s Purely Functional Data Structures.

I’ve also been reading a lot of the relevant academic literature published since the 1970s; for instance, the Lambda Papers and the myriad works of Henry Baker have been particularly inspiring. I continue to be struck by the harsh reality of the often-asserted, but seldom-accepted, truth that most of the great work in software was indeed done very early on, with virtually no further fundamental progress having been made during the past couple of decades.

It’s understandable how this state of affairs leads some old-timers to cynically grumble how those who ignore history are doomed to only reinvent it. However, it also implies that there may surely still be a whole untapped frontier before our very noses, with only a comparatively few people seriously attempting to explore it. In Alan Kay’s words: “The real romance is out ahead and yet to come. The computer revolution hasn’t started yet.”

If so, then opportunity truly abounds for an ambitious new generation of programmers, unafraid of paradigm shifts and looking to stake their claim — if they can first find their way to the frontier along the very long, windy road littered with the roadblocks of commonly accepted wisdom and conventional industry practices.

What do you think of Lisp so far?

Realizing the equivalence between code and data, and understanding how Lisp’s uniform syntax and macros conspire to bridge the two in practice, was an epiphany something akin to encountering E=mc² for the first time.

I fully understand how many Lispers feel something like prophets who have ventured to the Other Side, catching a brief glimpse of a higher realm and the transcendent fabric of reality. It’s no wonder at all that the meme regarding the “smug Lisp weenies” persists, and not infrequently hits home, too. How could it be otherwise?

I’m thrilled at how Lisp allows me to solve problems in the most direct conceivable representation, meaning that I spend more time thinking about the solution than actually typing it in on the keyboard. I like that any new control structure or language extension I can imagine, I can also implement; I’m restricted only by my own abilities, not by some arbitrary constraints imposed by the programming language.

I’m cautiously optimistic about the longevity of sufficiently well-written, abstract Lisp code (that is, code that doesn’t explicitly rely on incidental present facilities such as IPv4). Writing in some other language, I certainly wouldn’t expect (or want) my code to survive a couple of decades unchanged; whereas with Lisp it seems that might be a real possibility, since you can work at a higher abstraction level, almost directly in touch with the actual problem domain.

In terms of expressive power, Lisp lacks any true competition due to its unique ability to assimilate any new programming paradigm when one crops up. (Indeed, “embrace and extend” would be fitting words to describe the manner in which Common Lisp assimilated object-oriented programming in the 1980s, given that CLOS goes far and beyond the OOP facilities found in most object-oriented mainstream programming languages.)

What’s more, as has occasionally been remarked, other programming languages may not be able to catch up to Lisp without actually becoming Lisp — which would have to be the ultimate expression of Greenspun’s Tenth: the alien creature bursting out from within the rib cage to take over the host.

In view of all the above, I definitely buy Graham’s argument that Lisp is a competitive advantage, a supremely malleable medium for exploring the gap between what is currently possible and what is ultimately possible. (Which is the very reason, of course, why Lisp became the lingua franca for that hardest of problems: artificial intelligence.)

I think this also ties into the perceived smugness of Lispers: the lots of irritating, supposedly superfluous parentheses serve to actually hide the secret weapon in plain sight. It’s right there, up for grabs, but so very few ever approach it with the right mindset needed to grok it. So, if you are one of those very few, how could you not be a little smug about having been smart enough to self-select into that elite?

This attitude has a serious flaw, though: a superiority complex of this magnitude inevitably induces complacency. Giddy with the omnipotent abstract power afforded to us by Lisp in its platonic form, it’s all too easy to ignore the very real problems with the presently-existing, concrete dialects of Lisp.

So, while I’m sufficiently convinced that Lisp is the most powerful programming language and paradigm yet invented (or discovered, if you will), I find neither of the major dialects, Common Lisp or Scheme, particularly satisfactory in their present form; Common Lisp is just plain anachronistic, and Scheme is too minimalistic to be all that useful in itself.

Birch’s corollary to Greenspun’s Tenth rings dangerously true: “Even an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp is better than most other languages.”

Given that any useful new concept introduced in some lesser language can be copied into Lisp at will, what incentive is there to truly push the envelope in Lisp itself? Apparently not much, if recent history is an indication. The Lisp machines seem to have been the apex of Lisp, a brilliant achievement still unequalled (except by the Smalltalkers, as always). These wonderous machines have been followed by two decades of standstill, or even retrograde, progress.

Richard P. Gabriel wrote in his famous essay Lisp: Good News, Bad News, How to Win Big: “On the other hand, there should be a strong effort towards the next generation of Lisp. The worst thing we can do is to stand still as a community, and that is what is happening.” The thing is that Gabriel wrote that in 1991, for crying out loud.

Not only is Common Lisp anachronistic, it seems stagnant, a victim of its own very success. This is why personally I’ve focused on Scheme instead of Common Lisp: I don’t want to get trapped into accepting that Common Lisp is already so damned good that it doesn’t need to change or to be reinvented from time to time. I don’t want to get comfortable with Common Lisp, and then, gradually, complacent. I want to keep asking: is this all there is?

The Scheme community is a much more chaotic, fragmented space, but at least the lights are still on. I think Scheme, with its minimalistic, elegant core, provides a decent basis on top of which something new, something truly great, might again be built without having to throw out the baby with the bath water.

After all, isn’t perhaps the single most significant quality of Lisp its ability to, without all that much effort, bootstrap a whole new disparate language implementation on top of an existing one, taking full advantage of all the compilation facilities and built-in libraries of the existing implementation?

Lisp is literally the perfect language for reinventing itself, and I think it’s about time it did so.