The recent proposal to add closures to Java brought up some questions for me. How much more productive would I be with closures in Java? How much more fun would I have?
Last week Gilad Bracha, Neal Gafter, James Gosling, and Peter von der Ahé floated a proposal on the internet for adding Closures in Java [PDF]. I was excited to see that adding closures to Java was being considered by such influential people, however, reading through the proposal I was reminded of the difficulties of adding such features to a language as mature as Java.
The recent rise in popularity of Ruby, driven by the success of Ruby on Rails, has perhaps raised the profile of closures in the general development community. Just fifteen days ago, for example, Bruce Tate put closures at the top of his list of 10 Things Java Should Steal from Ruby. Others have lobbied for closures for much longer. Gilad Bracha, for example, tells us in Achieving Closure that he has "personally argued for adding closures since 1997/98."
And in The Long Strange Trip to Java, Patrick Naughton, one of the original members of the "green team" at Sun from which Java emerged, described in his personal history of Java a scene in which Bill Joy was insisting that closures be added to Java, back when it was called Oak:
[Bill Joy] was often comparing Oak to more complicated and elegant languages like Python and Beta. He would often go on at length about how great Oak would be if he could only add closures and continuations and parameterized types. While we all agreed these were very cool language features, we were all kind of hoping to finish this language in our lifetimes and get on to creating cool applications with it. The more we argued with Bill about making those changes the more strongly he would fight us. After a while it became a choice between not having Bill involved at all or losing control of the language. James [Gosling] and I got into a rather epic battle with Bill in his office in Aspen one evening about this issue. He started out by insulting both of us about how poorly Oak compared to better languages and then he volunteered to resign from being the Live Oak architect if we wanted him to. James and I agreed that would be best and walked out to go across the street to watch "Speed". What a rush.
The next day, Bill was pretty much his normal old nice guy self again, a little relieved, I think, to be out of the role of being directly responsible for our destiny. Bill is annoyingly smart, and I wouldn't put it past him to have planned that whole scenario the night before to force us to defend our positions and take ownership of our project. The interesting thing is that we were right about needing to finish the language even though it had missing features. It was a timing issue, there was only about a three-month window in which the whole Java phenomenon could have happened. We barely made it. It is also interesting that Bill was absolutely right about what Java needs long term. When I go look at the list of things he wanted to add back then, I want them all. He was right, he usually is.
According to Naughton, that exchange with Bill Joy took place in the summer of 1994, twelve years before the Bracha, et. al., proposal appeared. Is it better late than never? Maybe, but maybe not. As Gilad Bracha put it in Achieving Closure:
Since the late 90s, I've brought the topic [of adding closures to Java] up now and again. At times, even I have reluctantly been convinced that it is too late, because we've done so many things that would have been easy with closures in different ways.
A Natural Law of Language Design
There's a natural law in programming language and API design: as backwards compatibility increases, elegance decreases. Backwards compatibility is very important. There's a cost to breaking code, but there's also a cost to not breaking it—complexity in the developer's face.
For example, Joe Walker mentioned in Closures: Cheers, Library Issues and Solutions that while closures are nice, what would be really nice is if Java Collections could have .each() methods like, for instance, Ruby. This would be tough to do because adding more methods to the Collection interface, for example, would cause all existing implementations to not compile. Instead, the methods would likely end up as static methods in the Collections, which require that you pass in the collection as a parameter. It would work, but is arguably not as elegant.
Moreover, even if that were done, there would now be two idiomatic ways to do iteration in Java collections. The new way using passed closures, and the old way using Iterators. Actually there would also be the old, old way: Enumerations.
For several years I have been jealous of the joy people seem to feel when they program in Python or Ruby. I believe the sense of satisfaction stems primarily from feeling, and actually being, more productive in these languages. Although I have no doubt that my choice to use Java for Artima's architecture was the right one, for me Java was a practical choice, not a heart choice. When I work in Java, I wouldn't say I feel joy. I don't feel like I'm riding a fast stallion through beautiful wooded hills, the wind in my long, flowing hair. I feel more like I'm struggling to convince a workhorse to pull a plow.
The closures proposal seems primarily aimed at improving the Java programmer experience, smoothing out some of the rough edges and verbosity of anonymous inner classes, and I think that is a worthy goal. We should try to find ways to improve the usability of the Java language. But because of the natural law of language design, there will be a usability cost of increased surface area that accompanies the benefit of adding closures. Some things done the old way will become obsolete, perhaps deprecated, and officially ignorable. But they will still take up space in the documentation. They will still be in the developer's face, who has to remember they can be ignored. And they can't always be ignored, because important legacy APIs will still do things the old way. For example, the much-used Servlet API still returns Enumeration, not Iterator.
Breaking Source, But Not Binary Compatibility
I've long wondered how I could get the joy of using a language I love, while still getting all the benefits of the Java ecosystem. Earlier this year, I noticed Scala, and realized that perhaps a path may someday open in front of me that will allow me to gain not just the practical benefits of the Java ecosystem, but more joy of programming. I have not yet found an excuse or time to program in Scala, and I can't say whether I would even like it, much less love it. But reading its documentation made me realize that perhaps the best way to beat the natural law problem is to break source compatibility with the Java language, but not binary compatibility with Java class files. This way I can program in a language that is more elegant, fun, and productive, but still call into any Java API, including all the Java APIs we've written for Artima, use tools that perform static analysis on class files, and run my application on highly optimized JVMs.
Looking at Scala, I see it does seem to do all the things Bill Joy purportedly wanted to see in Java back in 1994, including, notably, closures. What Scala and every other alternate language for the JVM currently lacks, however, is anywhere near tool support that the Java programming language enjoys. The Java language has great refactoring IDEs, static analyzers that look for bugs, code complexity, and code style problems. Some of these tools are available to compiled Scala code, because they work off of class files, and there is a Scala plug-in for Eclipse. Nevertheless, the tool support for the Java language is so much greater, I judge it more appropriate to leverage that than to chase fun and productivity by experimenting with a new language. For this path of using a different language for the JVM to actually become practical, some other language besides Java will need to become mainstream enough to motivate the tool support. (And of course, good tool support would help the language become mainstream: a classic chicken and egg problem.)
I asked Martin Odersky, the creator of Scala, for his opinion on the closures proposal and the difficulties of adding features to a language as mature as Java:
I think [Closures in Java] is a very reasonable proposal, done by people who are the experts in the field. It had to resolve
some thorny problems related to fitting in with an existing language which is already quite complicated and which was not designed to accommodate the extension. The proposal manages this well, but at the price of some additional complexity. Besides the syntactic complications, there are also the issues of name resolution (when is something a function, when a value?) which look quite tricky and whose ramifications I have not yet fully understood.
Like you, I think there's a law of diminishing returns at work. As languages evolve, each new addition becomes more complicated, because it has to interact with more conflicting constructs. More complicated extensions cause proportionally more difficulties for future extensions, so it might even be a quadratic increase in complexity. At some point, the additions are no longer worth their price. That's the point when sometimes a new language has a chance of replacing an established one.
I will be programming in Java for the forseeable future, because I have a big investment in the Java code I've written for Artima. I do hope to see the Java language continue to evolve and improve, but I also wish for an alternative that makes me even more productive. If I could use the same IDE to go back and forth between the languages, perform refactorings on both languages at the same time, that would give me a more practical transition path. I could start programming some new things in the new language, while continuing to program the existing things in Java. Over time I would expect I'd be doing more and more in the new language.
Do you believe that another language for the JVM could someday become as mainstream as Java? Do you know of any language that might be capable of doing it?
> Do you believe that another language for the JVM could > someday become as mainstream as Java? Do you know of any > language that might be capable of doing it?
It seems to me that what a lot of people seem to miss is that the reason Java is so popular is that it doesn't contain all the complex features that advanced developers love yes I;m talking about Scala.) It's a language that an advanced developer can be fairly productive with while a average developer can still make sense of it. The syntax used to be very small. This made Java code easy to understand. You couldn't find Java code where you'd look at it and not understand what it was doing in terms of the syntax. The complexity explosion of 1.5 along with the sloppy way a lot of the features were shoe-horned (I think) is the beginning of the end for Java. Perhaps a cleanup and a true Java 2.o would fix it. I don't know.
But Scala? I think it's really cool but there's no way it's going to become mainstream. Sorry. It's a language for language designers not Joe Shmoe down the cube hall. You should see the looks I get when I mention implicit parameters. Some poeple are horrified, others are completely lost. No one has said that they liked the idea.
I'd like to first say that there seems to be three schools of thought regarding the Java platform:
(1) Don't touch Java. It's fine the way it is. "We" don't need anything else. Alternative languages on the JVM just cause confusion. And so on...
(2) Extend Java to keep up with the features in other languages, probably most notably the features that will be coming in C# 3.0, as that is the biggest competition right now.
(3) Stop "kludging" new features into Java and come up with a new JVM language that all the "cool kids" can enjoy, while remaining class compatibility.
I probably fall into camp 3 more than any of the other ones, even though I don't have a problem with adding new features (such as closures), as long as its a well thought out process. Syntax is important.
As an outsider just speculating, it seems to me that there are politics inside Sun that have overemphasized the importance of Java (the language) in relation to the importance of the JVM platform.
Microsoft, on the other hand, doesn't seem to really care (even though C# is the flagship), as long as the language is running on the .NET platform. They hired the IronPython developer which is scheduled for a 1.0 release soon. And there is already four or so languages that run out of the box on a VS install.
You also bring up the important point of tools support. It doesn't matter if the language is the greatest functional-style thing since sliced bread. If it doesn't have proper IDE support in 2006 and beyond, it loses much of its supposed benefits. It surprises me the number of people that don't seem to grok that.
I must agree with you regarding point 3. There is indeed a great need of reviewing the JVM. Since with the release of Mustang, it is expected to have Dynamic Language Support as well in Java. All this is being handled in OLD JVM BYTE CODE syntax. Sun should revise to make it more efficient if they wish JAVA to be at competing edge with C#.
Excellent analysis, Bill. You've brought up a lot of the issues that I've been thinking about lately, especially the backwards-compatibility problem which I saw while on the C++ standards committee, and which caused the pseudo-generics debacle in Java.
I've heard the "we had to rush Java out the door" argument way too many times as a justification for not doing it right, throwing in poorly-written libraries, etc. And I saw "backwards compatibility" invoked on the C++ committee when there was a relatively small amount of existing code out there -- and most of that code has been retired.
Bill Joy was right, it's worth the extra time to do things well rather than rush them out because it seems expedient at the time. And the argument that Java is successful weakens over time, as the initial flaws are added to and you end up with complexity accretion increasing over time.
This is not to say that Java hasn't contributed a great deal to the programming world. C++ was supposed to make library use easy, but it didn't make library creation easy. Java moved that a big step forward, as we've seen with the proliferation of libraries. Also, it brought OOP into the mainstream, and showed that garbage collection was not an impractical and esoteric idea (C++ is even adding hooks for garbage collection in the next version of the standard).
But the inability to fix the language, and the staunch declarations that it was all done right in the first place so it doesn't need fixing, is what will eventually cause Java to diminish. Not disappear -- C++ hasn't disappeared either, but both languages will move towards niches: C++ is a more powerful C, and Java has captured a great deal of knowledge in its libraries.
Whatever the justification, saying that you can't fix a language because of backwards compatibility is effectively the same as saying that you can know all the requirements for a project up front. Gosling and Naughton thought that Java would never need such esoteric features as parameterized types (what generics should have been), so they didn't have to be put in at the beginning. If they had, C#'s generics mechanism would have paled in comparison, but because they've had to be retrofitted, the so-called "generics" in Java are really just syntactic sugar for autocasting, and the really useful features of a true parameterized type mechanism are missing.
Now they're talking about adding closures, another feature that they were sure they didn't need when the language was originally being designed. Based on the generics experience, I shudder to imagine what new Frankenstein will come out of this effort. If it's anything like generics, it will end up being something that vaguely seems like a closure mechanism, but will be loudly declared to be closures. And perhaps enough people will be fooled by this (again) that it will get by (although I wonder about that, since there will be enough Ruby programmers who know what they really are to raise a ruckus about it).
If they're not going to fix the language, I have to agree with you that a new JVM language should be created rather than these exhausting attempts to add such fundamental features while retaining backwards compatibility.
One of the most powerful ideas that I've seen in both Python and Ruby is the willingness to fix things in the language even if it breaks backwards compatibility. Python has done it when fundamental problems were discovered, and has improved the language and kept it growing in good directions. Apparently Matz is really planning on removing many of the Perlisms in Ruby -- the aesthetics of those Perlisms are primarily what has given me a bad taste about the language, so I'm really impressed by this.
Python and Ruby are dynamic languages in more than just the experience of programming with them. The languages themselves are responding to new knowledge and understanding . If Java can't do that at a fundamental level, perhaps it should be left alone rather than adding more and more complicated patches that could do more damage than benefits. (The classic tale of the Swedish ship "Vasa" comes to mind here).
> Python and Ruby are dynamic languages in more than just > the experience of programming with them. The languages > themselves are responding to new knowledge and > understanding . If Java can't do that at a fundamental > level, perhaps it should be left alone rather than adding > more and more complicated patches that could do more > damage than benefits.
I'm leaning more and more towards using Python (Jython) with Java. Some parts of an application are better off in a staticly typed, rigid style. Core pieces that don't change a lot and need to be very reliable and transparent (and fast).
Other parts of the app like constantly changing business logic and data transformations are probably better off in a dynamic language. I'm not sure about the best apporach to doing this would be but I have some vague ideas. I also think Groovy may be a waste of time and effort since we already have Jython.
> One of the most powerful ideas that I've seen in both > Python and Ruby is the willingness to fix things in the > language even if it breaks backwards compatibility.
Alternatively, include 'versioning' into the language like PHP. PHP3, PHP4, PHP5 simply are different versions of the language and run with different interpreters without breaking backwards compatibility.
The messiest code base I've worked with was written by MIT grads in a scripting language, and I couldn't believe how nightmarish *and* brilliant it was at the same time.
It's a fascinating paradox to work through because the solutions were tailored to their own thinking and the creativity that comes with simply superior intellect. Unforunately over time, briliance compromises the feature set simply because the language offers you shortcuts and a smart person exploits them when under extreme pressure. If you don't do it, you aren't scripting, you're emulating a more formal language in a loose environment, and what's the point of that?
Scripting works great for smaller code bases, for expirable code bases, for the necessary glue that comes between various components of technology, but not as a main business language. The code will always hit the barrier of scale, whether it's in maintainability, or performance, or both.
"Joy" in this new generation of Blog Driven Developers comes with the bliss of ignorance. Like I've said many times before; this generation will just have to relearn the lessons of the past.
> It's a fascinating paradox to work through because the > solutions were tailored to their own thinking and the > creativity that comes with simply superior intellect. > Unforunately over time, briliance compromises the feature > e set simply because the language offers you shortcuts and > a smart person exploits them when under extreme pressure.
I also think the availibilty of shortcuts can keep a developers from stepping back and looking at the big picture for a better overall approach.
I liken it to the fact that ancient inventors came very close to starting the industrial age thousands of years ago but didn't. The availability of cheap labor (slaves) kept them from considering these inventions as anything more than novelties and tricks.
Reading Bill's recount of Java history about the inclusion of closures in the language, I couldn't help but recall Gosling's 1995 white paper, which is where I first learned about Java.
The most attractive feature described in that paper was that programming in Java promised a lot more joy than programming in the alternative, which at that time was mainly C++. Having just gone through reams of C++ documentation, I, indeed, found Java to be more fun.
What happened in the intervening eleven years?
The observant may note that Java programs ca. 1995 tended to be either tinker-toy code examples, simple applets, or script-like one-offs, such as retrieving some data from a Web server.
No one today would write such programs in Java. Instead, most people using Java today develop sophisticated enterprise applications, with code bases comprising thousands of classes. When developing highly complex applications, language alone may offer but small productivity improvement in a project. Instead, we spend most of our time interacting with APIs to address specific application tasks, such as connecting to external resources, or authenticating users.
When Bill writes about feeling "like I'm struggling to convince a workhorse to pull a plow," it may partly be because there is a huge context, a large infrastructure, behind most real-world Java applications. But that infrastructure serves the purpose of helping to provide sophisticated functionality. To continue the analogy, pulling a plow may not feel as joyful as "riding a fast stallion through beautiful wooded hills," but it can lead to a harvest in time, whereas riding a horse through a mountain may still leave you with an empty stomach.
On a more serious note, I wonder about the experience of those maintaining Ruby or Perl or Python applications that equal the sophistication in functionality of complex Java applications. Looking at RoR, for instance, I think most of the productivity comes from Rails itself (Active Record and Action Pack), since for simple applications there is hardly any Ruby coding involved. I have not seen really feature-rich RoR apps yet, so I'm curious about the experience there.
I also think there is a psychological mindset now in the Java community that using simple tools for simple things is passe, and that even for simple things you need to use some kind of a framework. Using plain-ol' JSP pages for a simple Web-app might bring some of the fun back into coding, and might even approximate the productivity of, say, PHP.
> The messiest code base I've worked with was written by MIT > grads in a scripting language, and I couldn't believe how > nightmarish *and* brilliant it was at the same time.
The messiest code I have ever seen was written in Java. I have seen terrible code written by brilliant and highly-educated people. I have also seen very well-written, well-organized and clear code in VB. Although I think a language can strongly affect the style that people use, genetics != destiny.
Also, I think you need to distinguish between "shortcuts" and language features that are paradigm shifts. Java generics introduces a shortcut, Python list comprehensions are a different and clearer way of thinking about the problem. Ruby & Smalltalk continuations are a paradigm shift.
> I also think there is a psychological mindset now in the > Java community that using simple tools for simple things > is passe, and that even for simple things you need to use > some kind of a framework.
As another example, consider the camp that says that all classes should have interfaces, and all instances should be generated by factories. I think the complexity of the language encourages complexity in design and implementation.
Flat View: This topic has 53 replies
on 4 pages