[a long entry, this, on why a using better languages makes all kinds of technical and economic sense. Much talk herein of Lisp, Java, software economics, risk, processes, outsourcing, drawing as programming, luddites, and perpetual motion too. It ends happily.] Always start with a fallacy: analysis as argument In software, we're living in an interesting time, when the outlook is more uncertain than usual and the industry is somewhat humbled following the dot-com bubble and a tough recession. I'm speculating that major change is coming to mainstream programming, and the change will be in the kinds of programming languages we use. I think this change will be more significant in impact than Object Orientation, and will result in evidently better productivity and software. The drivers are an increasing need for expressive power and a requirement to continuously rework and adapt the code. For this, we want a better language, whether we know it or not. If I had to pick one thing as most important to programming today it's the ability to change your mind, to adapt the code to new circumstances. Existing popular languages and software processes do not support these meta-requirements very well, if at all. Everything indicates that we need to be able to change code quickly to keep up with our customers and express their and our ideas with as few subsidies as possible paid to the compiler. Oddly enough, we're trying to do this without using languages we already know to be a good fit. I speculate this approach has lead to a continuous cycle of reinvention, and the expression of language design as creative amnesia. Better is better Sometimes what you need a better saw. Movements like test-first, agile and extreme programming are all very well, but they're for the most part focused on controlling existing pathologies in how we manage programming today. Some people are going to object to my calling some languages better than others. Yes, it's a poor basis for argument. We all do it though. Bjarne Stroustrup is still telling us how good C++ is. There is a spectacular community advocacy around Java. VB and Perl programmers love to talk about getting things done quickly. And so on. Take your pick, but what I do believe is that there's not an iota of merit to the arguments that language choice doesn't matter. But better to me means better at change....
How I learned to program
I'd suggest, only half jokingly, that Java and Python have come up with something that we'd love to have: the adulation of the masses. Some elitist grouch will no doubt rejoin that he doesn't want Lisp to be popular, but sentiment here says otherwise. Anyway, I don't know how to bottle the certain je ne sais quoi that Java and Python have, or had. - Cameron McKinnon
I love reading c.l.lisp. When I started out in software it was one of those places where I hung out and augmented my learning - along with c.l.object, arsdigita, xml-dev, extremeprogramming, the wiki (especially the wiki).
I came to programming despite myself. In art college training as an industrial designer I had to be taught CS101 via Pascal and it was such a horrible unfun experience that confirmed my then-held suspicion that with computers and technology came misery. After that I didn't look at a program for nearly seven years. But I naturally feel back into it - as artificial intelligence which has always been my favourite todo with a computer - while many people suspect that AI is a useless endeavour it's both fun and a challenge trying to turn Pinocchio into a Real Boy. Though like any fable, you wouldn't want to take it too literally. With an AI degree you learn computing differently to many others as the emphasis is very different. The first language we learned was Prolog. Then C. I learned some Lisp on the side, even though it was That American AI Language.
Why am I telling you all this? Well I speculate that my views on what makes for a good programming language do not accord with the industry I work in, and I think that has had something to do with my education which was somewhat impractical in terms of the marketplace.
I have never been happy using Java to generate web pages. Or later on, with scriplets. Or C++ for neural networks and machine learning. I can't read more than a few screenfuls of Perl or XSLT without getting motion sickness. I constantly bemoan the fact the you can't pass methods as arguments in Java. I keep screwing up Groovy closures with that inane bracket line convention. Make a note of that last one, it's important if we're going to talk about a better language for the job.
Syntax matters in a programming language. Groovy is inspired by features in other languages such as Dylan (and I suspect Python and Ruby), but because it also aspires to keep Java programmers comfortable it inherits some Java syntax. It has less syntax than Java but it still has a lot of syntax. Syntax seems helpful at first but it gets in the way when it comes to doing more interesting and flexible things you might to do with a program - such as closures or function factories. So to support closures, the Groovy parser needs you to help it by using the whitespace to give it clues.
One thing you'll notice about some languages is that they have less syntax. I think this quality is one dimension which indicates how good a language is. Lisp has almost no supporting syntax - it's like working with the parse tree directly - which is why one person has called it a programmable programming language. My favourite language Prolog has more syntax than Lisp but it looks pretty barren compared to Java. As does ML.
If you don't believe that your programming language has a lot of syntax,
Write the lexer/parser for it.
Write the lexer/parser for Lisp, or ML.
The less syntax you have the more uniform the language is and the less special case logic you need to manipulate or extend it - for example you can alter it yourself as you program instead of requiring a supporting language/compiler/paradigm.
I suspect that if your automagical "find all callers" refactoring Java IDE was sentient, Java would look not look like Java in its minds eye. It would like a Lisp syntax tree with some extra bits. Imagine having that sort of introspective power as part of the language itself.
Along with syntax are the evaluation rules for language. In Lisp and Scheme these are quite brief yet are extraordinarily flexible. The processing models for most popular languages are much longer while being much less flexible. Partially this a reflection of the complexity of the syntax, partially the syntax reflect the complexity or incoherence of evaluation rules, but mainly it's a reflection of the internal consistency and formal model of the language (if it has one).
Once they go down this route, languages that want to extend have to keep piling on new special case syntax to work around the existing extra syntax, to the point where they go off the deep end and the signal-noise ratio is just too much. Consider the progression of Perl through versions 4,5 and 6. Or witness languages piled on top of JSP. Java has done a good job of managing syntax creep over time. C++ has so much syntax and semantics it took years to get the compilers right. So you need consider both syntax and semantics together when considering a programming language.
One man's paradigm is another man's practice
This is one reason why Lisp and functional programming folks are a bit cool on stuff like Object and Aspect orientation. They can extend to Aspects or Objects using the language constructs directly. To most of us working in popular languages these are paradigm shifting ways of thinking about programming. But we can't use Java directly to support something like aspects or generics. There's whole communities working on language and compiler extensions, open source libraries, as well as that all important vendor support - J2EE app servers are being renegineered just to support aspects. To a Lisp person these are just constructs expressed as macros (which are written in Lisp itself), something you do on a rainy weekend. What's the fuss about?
Of course one of persons driving AOP in Java is Gregor Kiczales, who is an ex-Lisp hacker. And before that so was Guy Steele, who has contributed a lot to the Java's programming model. Richard Gabriel, yet another Lisp luminary, works for Sun. So you have to wonder.
It turns out that people of done a lot of work on making languages powerful, evolvable and flexible. In fact all of the things we are evidently craving for today. They tend to end up aiming toward Lisp. It's something a running joke in the Lisp community that all language efforts are doomed to reinvent Lisp. Lisp by the way, is not my favourite language, but in my very limited objectivity, it is by design as good a language available to a professional application programmer as any, and the best place to start if you are intending to design a new language. I'm wary of universals or absolutes but Lisp seems to represent a pinnacle of achievement in programming languages that is as yet, unmatched.
The tensile strength of language
One assumption made with Java/J2EE was to conflate the language you'd use to build the managed runtimes with the language you'd use to build the managed applications - that assumption is only sound if the language is sufficiently flexible. Java is a good language to build an application container and runtime. I will happily build message queuing software or a web server with it. But it is a weak language for application logic. Too often, expressing yourself in Java is tedious. You cannot easily change the code that exists already. We rely overly on frameworks to provide the flexibility the language does not give us. This is not a sustainable approach to building applications - over time the cost to change the system and the risk of breakage through change grows. The application is gradually strangled in the grip of its own logic. It comes to be replaced, ported to new language, or let die. This is the way with any system that cannot be easily adapted.
Clearly lots of people agree with me, since the main innovation in the middleware/enterprise space since the adoption of VMs and managed runtimes has been a steady stream of languages whose interpreters are written in Java and an attempt to shift data, contracts and configurations out of software objects into XML documents. Once upon a time that was a controversial thing to say - now it's just obvious that there is plenty of work we do not want to use Java for. JSP. Velocity. JSTL. EL. Ant. Groovy. Jython. There are multiple Lisp like languages available for the JVM, but they don't get much use commercially, if at all.
I pick on Java as an example but what I'm saying is equally applicable to C++, COBOL and VB. And C# will possibly go through a phase like Java's current one.
If it's so great, why aren't we all using Lisp? Lisp programmers seem unable to figure this out. After twenty years of introspection, the best anyone has come with is Richard Gabriel's infamous as "Worse is Better" maxim. There are some reasons we have resisted Lisp-or-something-like-it adoption, it's complicated, there is no single overarching cause, but we can split it into two areas. First issues with Lisp itself. Second a lack of understanding of why popularity is important in the software business.
To understand why we don't use Lisp, but keep reinventing parts of it every few years, I think you have to have run across a few Lisp people and used a few other languages to put food on the table.
How to love Lisp
Lisp in case you don't know is one the oldest programming languages, coming up to fifty years by the end of the decade. It's positively antique compared to C++, Perl or Java. But it has evolved with the times, because at it's core, it's evolvable. It's believed by some that popular programming languages are evolving toward Lisp, just not fast enough. I believe the meta-requirements of change and adaptation I started with , are accelerating this trend.
[I'd love to be able to compare Lisp to a shark, while comparing popular languages to primates, that would be very quotable - but of course the shark hasn't evolved for millions of years, and there are no signs of humans and chimps growing fins and extra cartilage.]
The non-syntax is distinctly weird initially, especially the ordering of arguments. The term I would use for how this feels is 'disorienting'. If you've been trained to write x = 1+2, then writing (setf x (+ 1 2)) is going to take some getting used to. The former version is called infix and the latter is called prefix. Prefix seems to suck, but it is one of the best things about Lisp since it allows one to manipulate expressions much more easily than the popular infix notion does while eliminating redundancy - adding three numbers is simply (+ 1 2 3). Most popular languages use a mix of both - infix for expressions and prefix for functions. Not everyone likes all those brackets, but with Lisp you use indentation not brackets to layout code - as you do with most languages, whether you care to admit it or not.
How to hate Lisp
We're not complete idiots though, in ignoring Lisp. Two of the main reasons we don't use Lisp are the Lisp community themselves and the lack of portable libraries that are relevant to modern programming work.
It's corny to say the worst thing about Lisp is a Lisp programmer. But there's some truth in it. Work long enough in this industry and you'll run into a Lisp bigot. Every language has bigots but Lisp ones are the worst. They're annoying, not just because they have a point (they do), and not just because they are better students of programming history (they are) but because they're annoying in the way they make a point. These people however right they may be are not doing themselves or you any favours with their arrogance. Not at a time when no-one knows where we're going or what the software landscape will look like in a year. With big change comes big opportunity. Lisp missed the boat once already in large part through misplaced arrogance, missing it again through more misplaced arrogance would be careless.
The issue of good support libraries is the one of big lessons learned by Java, VB and C#. A programming language is one thing, but a libraries is another. I think Lisp never traditionally emphasized libraries because the language is so flexible you can build one on your own, but then you have the twin nightmares of integration and reuse lying in wait for you. "On your own" is not how modern systems are built, especially distributed ones. Indeed the lack of interopability and repeated effort was one of the drivers to standardize Lisp and libraries, now called ANSI Common Lisp. Unfortunately it standardized functions a la the C++ STL - so Lisp looks like a language with a kitchen sink of a java.util.* package, but domain packages a la java.jdbc.* you have to hack together yourself. And javax.swing.* - well there is none. Which is so very Ninety Eighties. Thus, while the book, ANSI Common Lisp, does a great job telling you how to on generate HTML documents, it doesn't tell you how to serve them up or read them from disk or render them.
Why wish for popularity in a programming language at all? It depends on what you're doing - sometimes you might not want to be working with a popular language, just a better one that gives you an edge. If you're in the city courier business and all your competition are using the postal service while you use crazy people on bicycles, that's an advantage right there. You might not let on you're using crazy people on bicycles lest the competition get wind of it and imitate you. In the area I work in today, middleware and business systems integration, areas where Java, C++ and .NET dominate, working in an unpopular but better language is not the advantage it should be. This is because we have something more like a game instead of an industry. Most paid programmers are doing something like this.
Why are supposedly better languages unpopular? Why are we using more suitable ones? Paul Graham, the Lisp luminary, has laid the blame squarely at the feet of mediocre management and ignorant developers. It's a compelling idea that, to seek out an idiot, but it's much too simplistic to reflect how our industry actually works to be true or useful. Languages are of course popular because they are popular - it's a virtuous circle. It's perfectly sensible for grads and developers to gravitate to a language they can get work in, and the popular languages you can get work in by definition. But to understand why popular languages matter, we have to stop talking about them and talk a bit about the economic system most developers are working under.
The code game
In the game of building business systems, we have demonized programming. What has mattered is not so much the technical ability and power the language gives a programmer, but the purchasing ability the language gives the supplier and customer over the whole project and lifespan of the system. The reality is that Java C# and VB are well supported, well funded languages. It's easy to find material and training to get skilled up on. By extension, it's easy to find a Java or .NET or VB programmer. And it doesn't matter so much that they are good but that they are easily found and easily replaced; that there is a pool of people who can work the language and critically the associated platforms the language works under. In paid work, it's often more important to understand J2EE that Java. Therefore at a business level we can talk about managing risk with popular languages, yet at the technology level it makes no sense to use an inferior language. It's like using wood instead of steel to build a railway bridge, because the world is shy on blacksmiths but teeming with carpenters. Fitness for purpose is often a secondary concern.
The idea that a system could be built in one language in a fraction of the time and cost it took to build in another, more popular language doesn't matter as much as the fact that the world is awash in people who can help build and support a system built with a popular language. Over the time of the system the ad-populum approach is perceived to be less expensive and less risky than the alternative. And to paraphrase the old saying, no-one ever got fired for buying what's popular.
Longer than you can stay solvent
The economics of building with inferior languages are self serving. They require larger and larger people pools which require increasingly popular languages. They also encourage high-ceremony processes that attempt to coordinate large teams building in large systems. We're building large systems because we need to express as many of our requirements upfront. We express our requirements upfront because we can't easily change the system later on. Of course these processes don't do that to our satisfaction, but what they do allow is to isolate almost everything that is not programming into a repeatable well-understood phases of a process. All the messy parts are left over to the phase called "build" or "construction" or "integration". The result is that we have software processes that make us feel less exposed to the risk of programming with popular languages than we actually are. I speculate that we are in fact over-exposed as result of demonizing programming and making it a process shibboleth, but that's not what it feels like.
We don't like to talk much about programmer meat-markers and bodyshopping - we'd rather talk about skills shortages and democratizing programmer. Outsourcing is about as close as we'll get to acknowledging that the business is a harsh mistress. But outsourcing is a tactical response as we'll see soon.
On being smart
People who advocate better languages I suspect do not always appreciate the consequences of this macro-economic model of the software business. And I think there is also a more direct, interpersonal, problem here. Programming something even half decent requires considerable intellectual ability. That's not all there is to it, but if you're not smart, it will be much harder for you to be a decent programmer. We're mostly in denial about this and would like to believe that programming can be done by anyone or can be made somehow unnecessary. It's the opposite view we take to managerial or executive ability, where we pay through the node for talent. Yet it's been known since we started building software that some programmers are much better than others. Maybe all disciplines are like this, but it is very obvious in programming. Naturally this intelligence makes people who don't have it uncomfortable, particularly if that person is supposed to lead or organize very smart people to do something that they don't fully understand, even though that person is probably better paid, more socially equipped and has nothing to worry about.
As with meat-markets this is not something we like to talk about a lot. It's not egalitarian and intelligence is something we're sensitive about culturally. But consider that it's not controversial to say being a good musician or athlete requires a level of ability, some of which is innate. Or that our educational systems are based around being ranked smarter than the next person anyway. So we should get over ourselves a bit and accept the fact the good software requires more than stringent process; it requires well-above average intelligence to create it. One problem with acknowledging this is that is does not fit with the aforementioned economic doctrine. There, we'd like to democratize programming and so commoditize that ability. But, there are simply not enough above average intelligence people to go around - or more accurately there are not enough above average people to get things done using popular but inferior languages - so even though it seems to be hard to get work for many people, the providers and buyers of software are prone to talk about rising wage costs and skills shortages. And given that any organizations consider software to be a capital cost rather than investment, it's irritating to have to compete for developer talent. In truth, under this doctrine, there will tend to be a skills shortage more often than not.
Historically this march to commoditization is nothing new. When the commercial world depended on the guild halls, entire technologies were invented to help work around their members - most famously in the period of the industrial revolution. In the last century industries and governments did their utmost to work around, and then break, the trade unions. Some people believe we're seeing the same thing happen to programming. As had been said to me from time to time - surely you don't want to be a programmer all your life? But the analog doesn't hold up so well when you consider that programming is not like any activity ever encapsulated by a guild or a trade union. You might as well be automating mathematics, or law.
Despite this, the companies that seem to succeed in pure play software are the ones that have organized themselves around finding very smart people and servicing them so then have everything they need to do their jobs as they would with their managers. Microsoft, Google and the pre-merger HP are good examples of this culture. The service and solutions organizations have tended to prefer an emphasis on repeatable process and outward professionalism. There are exceptions in both cases.
Outsourcing your head into the sand
So we come to nub of the problem. Unpopular languages do not fit the business and risk models that most developers work under. But perhaps there's an alternative. Adjust the model to support programmer productivity.
I think at some point this will have to happen - the current mass-customization economics for software do not do enough to support business' unslakable thirst to change, integrate and adapt software. There has to be a better way to negotiate than change control. I described the current model earlier as a self-serving, but it is so by way of being something of a bubble.
Outsourcing development is a tactical option, one that offsets the inevitable sea-change for a few more years perhaps. It does not itself change the underlying model or address problems in how we build systems. Those in the East will run into the same problems as those in the West have. We might understand outsourcing as a form of creative accounting. Remember also that within the current model, an enormous supporting services infrastructure has grown up around weak languages used by masses of developers; from vendor platforms to service integrators to tool and IDE builders to recruitment agencies. In the same way open source is eating away at the margins dictated by software vendors, the pressing need for adaptation will put pressure on the margins we can justofy as suppliers of solutions that are not adaptable enough.
Base metal, Perpetual motion
Perhaps the most widely cited strategic answer to reducing programming costs is not to improve existing languages or even to make unpopular, potentially high-productivity languages popular. It's to draw programs instead of write them. That way maybe anyone can program and we can get rid of those unpredictable programmers and awkward phases in the software process. I and many others find this strategy to be flawed as a the next step in programming productivity - we've seen CASE come and go and come and go, and UML remain on the whiteboard for anything that remotely looks like a working system. Today it's the turn of XML based business process modelling languages and from the people that brought you the UML, the MDA. But listen, if you can't or won't understand how to program in Java, you will find no solace in drawing pictures for the benefit of a business process engine. Wanting this to be so is the same sort of irrationality that is held when trying to turn base metal into gold, or crate a perpetual motion machine. It's a belief that is largely supported through the ignorance resulting from demonizing programming. Consider that years ago, people said that 4GLs would allow business people to work with their data without the need for programming. The reality is that the single widely used 4GL, SQL has generated an enormous market for software and services if not an entire sector - which seems to be the exact opposite of no programming.
A rainbow, with no crock
What are our options? Honestly I don't see us rushing to embrace Lisp this decade. I think the candidates today are clearly Ruby, Python, and perhaps on Java, Groovy. All of these have enough of lessons learned from Lisp to represent a big leaps forward in productivity. All of the languages are capable of running on the JVM or the CLR - we're not talking about ripping out billions of euros worth of infrastructure.
The signs are positive. The Java community are considering adding Groovy to their toolkit and Ruby is a regular topic of conversation. Sun have expressing interest in scripting on the JVM. Python usage in the enterprise is growing - and Jim Hugunin has recently demonstrated that Python can run well on .NET. Enterprise types don't snigger so much at scripting languages anymore, not even Perl and PHP. More and more the debt we owe to the Lisp community is recognized.
Where things will be tricky is in software process. We will need alternative processes to support the kind of work allowed by these languages and the needs for change demanded by customers. A lot of what do now is not going to be applicable. This is why I say the business models will need to be adjusted. Software process are even more a reflection of commercial practice then they are engineering.
To bootstrap a language into the mainstream requires marketing and selling it - it's very expensive to do this. You could build a community around it, as Python and Ruby have done but that will only take you so far, about as far as Python and Ruby are. What's really needed is the industry to consider the economic reality that what we're giving customers isn't exactly what they want, or perhaps watch a new breed of service oriented entrepreneurs fill the demand.
I'm not speculating that we will see another downturn. Quite the opposite. I think the move to better languages, and letting developers use the best tools for the job will herald an economic boom, one that is sustainable and wealth-creating because it represents genuine increases in productivity. It will also represent an offset to sunk costs in software expenditure by resulting in more flexible systems.