Bill Venners: What problems do closures try to solve?
Neal Gafter: What I'm try to accomplish with closures is to allow you to write methods that are part of APIs that act like control statements in the language. Because a lot of APIs do have specialized needs. You have the concurrency framework where you'd really like something that acts like the
synchronized block, but for the concurrency locks. We did add a statement in JDK 5, the for-each loop, which is specially designed to work with collections. There are a lot of APIs that would benefit from having statement forms, and there are proposals to add even more statement forms in JDK 7. My hope is that by adding one language construct, we won't have to have a never-ending series of additional statement forms because we keep having more and more APIs that we care about. Instead, we can have one language extension and those APIs can include those statement forms.
Bill Venners: One of the examples you gave us in your talk was
withLock. Can you elaborate on that?
withLock is a method that I hope would be added to the concurrent framework. It basically looks just like a
synchronized statement. You say,
withLock, open-paren, then a lock variable, close paren, open curly brace, then a bunch of statements, and then close curly brace. It looks just a like a built-in statement form, but it's a method invocation, an invocation of this
withLock method. And the
withLock method takes two parameters, the lock, and a closure. It does the sort of boilerplate that you would have to do to synchronize a block of code using the concurrency framework yourself, but it's much more convenient to use.
withLock example is a poster-child for closures. If you don't have closures, and you're never going to have closures, you really want to put something like that in the language. But, again, it's a never-ending series of language additions.
Bill Venners: What makes a block of code a closure, instead of just a block of code?
Neal Gafter: The syntax I just described, where you have a function declaration that looks like it has a block of code after it, that's not legal syntax today. That is a new syntax in the language, and we would define that syntax to mean that you take the block at the end and wrap it in a closure and pass it as an extra parameter to the method.
There is also syntax where you can write closure literals, where you can explicitly pass two or three of them as parameters to a method, and that's a new syntax too. The way you write a closure is, open curly brace, the argument declarations, an arrow (equals-greater-than), a bunch of statements, and then a closed curly brace—that's a syntax that's not in Java right now. That would be the way you write a closure literal in the language. It would be wrapped by the compiler in an interface type when it's passed to the method to which you pass it in as a parameter.
Bill Venners: In your talk, you highlighted the differences between closures and anonymous inner classes, such as the meaning of
return. When you have a closure, you can return from...
Neal Gafter: ...the enclosing method. Actually, there are some places where there are no enclosing methods. If you wouldn't be allowed to write a
return anyway, then you won't be allowed to write a
return inside of a closure. But a
return from a closure simply returns from the enclosing method. The fact that it's inside of a closure doesn't change the meaning of the
Bill Venners: What happens under the covers? In your talk, you showed the interface that has an
invoke method in it. Could you describe what really happens?
Neal Gafter: The closure is turned at compile time into an instance of the interface. If you take the example of the
withLock method, that method is declared to take a function type as its last parameter. At compile time, the compiler looks at the closure and says, oh, I actually need something of this interface type. (Function type is an interface type.) So it builds an instance of that interface, just like an anonymous inner class, but it's not an anonymous inner class, because the scoping rules for a closure are different from an anonymous inner class's scoping rules. It creates an object that implements that interface where the body of the
invoke method does what the code inside the closure says it should do. And it passes that instance as an extra parameter to the
withLock method. And inside the
withLock method, in order to cause that code to be executed, you have some parameter, like
block—you just say
block.invoke(). That's all you need to do, and it simply calls the
invoke method on that interface from the object that was passed in.
Bill Venners: If that block executes a
return, how does it return from the enclosing method, which called
Neal Gafter: There are actually a few different answers to that question. The simplest answer is that a return from a closure will be implemented using exceptions. So the compiler will create and throw an exception, and that will pass out through the invocation of
withLock to the caller that was calling
withLock, where it will be caught. And the value that you're returning will be pulled out of the exception, and the code that catches it will actually do the returning from the enclosing method.
As a practical matter, if you use that pattern, where you put a return inside of a closure, it will get inlined by HotSpot, and it will turn into a jump, which is exactly what it would have done if you used the statement in the language instead of a closure. If the closure only executed once, it won't, but if the code is executed more than a couple of times, and the return is executed very frequently in the closure, then HotSpot will inline that. And it will turn into the same code at the machine code level that it would have, if you had done it via a control statement added to the language.
Bill Venners: One thing I really liked in your presentation was about methods that exist in existing APIs today that you could pass a closure to, even though they never imagined that would be possible.
Neal Gafter: Let's take a particular example. In the concurrency framework there is an interface called
Executor. It represents a thread pool, among other things. One of the things you typically do with a thread pool is you submit a
Runnable to it. It takes the
Runnable, and assigns it to a thread to run now, or as soon as a thread is available. That method is called
execute. If you want to use this method, but you want to use a closure, because of the convenience of the syntax, you can. Remember I said the closure conversion takes the closure and figures out what kind of interface type it needs and that's the kind of interface that's built at compile time to be passed it. So if you
say, executor.execute(), and then open-close curly braces, and write a block of code inside the curly braces, that block of code is turned into a closure. It's turned into an object that implements
Runnable, and the
run method does whatever that block says it should do. That object is passed to the
execute method of the executor framework.
No change needs to be made to the executor framework because a closure is always converted to whatever interface type you need statically in the context of the invocation of the code. You can use a closure in place of a
Runnable, you can use it in place of a
Callable—whatever interface you need—you can write a closure literal in that place and pass it to that API.
Bill Venners: So if make a new API after closures are in the language, and I say explicitly that I take a closure...
Neal Gafter: It's just a parameter, right? All you have to do is declare it as the last parameter of the interface.
Bill Venners: So all I have to do to be ready for a closure being passed to me is that the last parameter is an interface?
Neal Gafter: You can receive a closure in any parameter position. The only advantage of having it in the last parameter position is you get to use the special syntax that looks like a statement. That's the point. If it is the first parameter instead of the last parameter, it's no problem. But you would use the closure-literal syntax with the fat arrow instead of putting a block after the invocation. But you can do that. In fact, if you have an API where you need to pass in more than one closure, you have to use the closure literal syntax.
Bill Venners: Basically any existing method call at the end, if it's an interface... It has to be an interface, can't be a class?
Neal Gafter: It has to be an interface, and an interface with one method in it.
Bill Venners: So that narrows it down a little bit the existing APIs that will be able to be called with closures.
Neal Gafter: It does, but there a lot of those—all of the call-back registration stuff in Swing, the executor framework, and so on. There are
Runnables—a lot of places where you could take advantage of this, even
java.util.Comparators that you use for sorting. You can write a block of code that describes the
Comparator right in the
sort invocation. That works just fine. In many of the APIs where people currently use anonymous instances in the invocation, you can use closures instead—but not all of them. If you have something where you really need more than one method to be in there, you can't easily use closures for that. Actually, you can, but not easily. You have to write a method that makes an instance of a class and then invokes the closures for each one. It's not pretty. It's really mainly suitable for the ones that have a single method.
Bill Venners: Tell me about exception transparency.
Neal Gafter: The idea of exception transparency is that sometimes you have an API to which you pass in a closure, and if the closure throws an exception, you want your method to be known to throw the same exception so that exception checking works.
A really good example of that is the
withLock that works with the new locks. If the block of code in the
withLock statement throws some exceptions, you want the compiler to know that those exceptions are going to be propagated out from the
withLock call. In the closure specification, therefore, we've added some stuff to the machinery of generics. It's only a little bit; it's not very complicated. We added just enough to make it possible for you to express that the exception propagates out.
Bill Venners: How does this affect the way checked exceptions work?
Neal Gafter: If the block throws, say, a
NumberFormatException, then the compiler is going to complain that it's not caught. But if you can declare that in the enclosing method, then the compiler won't complain. You can't declare the
withLock method to throw
NumberFormatException. That would be silly, because not everyone who uses
withLock has to deal with
NumberFormatException. In fact, there is no particular exception that you can declare in
withLock. But using this new mechanism, you can say whatever type is thrown by the block is thrown by
withLock. If no checked exceptions are thrown by the block, then nothing is thrown by
withLock. So you have a type parameter for the exception in the API declaration (such as the
withLock method), and you say, "The closure throws this exception X, and so do I." That allows you to get the exception checking for APIs where the implementation will propagate the exceptions out.
The reason you need a special syntax for that, the reason you can't just use an ordinary type parameter, is if you use an ordinary type parameter, that will work as long as there is exactly one exception that's thrown. If there is no exceptions thrown, what are you going to fill in there? And what if there are two exceptions thrown? Then you're just out of luck.
The idea is that if you declare a type parameter with the
throws keyword, then you're allowed to put multiple types in there for the exceptions, and the way you list them all out, you can't use commas, those would look like extra type parameters, you use a vertical bar. But you don't normally have to specify those type parameters explicitly. You just pass a closure that happens to throw three things, and the caller has to catch those three things. That just works. But if you do want to explicitly pass a type parameter, which is the three different exceptions that can be thrown by a block, you just list the three in that type parameter position with a vertical bar between them.
Bill Venners: So the compiler will always be able to tell by looking at each individual invocation of a method that takes a closure what the exceptions thrown by the block will be, right?
Neal Gafter: The set of exceptions that are thrown by a closure are part of the type of the closure. The interface to which that's converted must declare those exception types. The closure that's received by
invoke must declare that in the
throws clause of the method of that interface.
Bill Venners: Can you give an example?
Neal Gafter: For example,
withLock is a method. It's second parameter is an interface that contains a single method called
invoke. Whatever exceptions the passed block throws must be in the throws clause of that method. And the closure conversion checks that.
withLock method, you don't know what that's going to throw. That's why you use a generic type parameter. But the compiler, when it's looking at the closure, knows exactly what is thrown. It can see that you're calling
Integer.parseInt(), for example, and it knows that it throws
NumberFormatException. The compiler knows what exceptions are thrown by the closure, and it matches those and makes sure that the interface that the closure is converted to declares all the exceptions that are actually thrown. If it's a generic, then the compiler knows what a generic is supposed to be, it's supposed to be whatever is actually thrown by the closure. It fills in the type parameter on that basis.
Bill Venners: The other thing you showed at the end was some functional programming styles. I had written a blog post about how it becomes increasingly difficult to add features to a language over time because you have to be backwards compatible with more and more existing features. If there is a better way to do it, like iterations, you leave in the old way. So we have an
Enumeration, and with closures we'd be able to pass a closure to an
Neal Gafter: Yes, but I don't think there is any point in doing it. I don't think there is a point in adding an
each method to
Collections, or anything like that. I don't think it does anything for you. And if you do it, you don't need to put it in the interface. If you want to put a static method in
each that takes a
Collection and a closure, and loops over the
Collection and calls the closure on each of them, you can do that just fine, there is no backward compatibility problem.
Bill Venners: So you say it we don't need it because we already have the for-each?
Neal Gafter: I'm saying that there is no problem leaving the for-each in, and as a practical matter I don't think you need it. It doesn't do anything for you that you can't already do.
Now, if you had closures before JDK 5, you probably wouldn't have wanted to add that statement to the language. But I don't think there is any harm in having it there.
Bill Venners: So basically your original justification for adding closures was adding control statements. So instead of adding a bunch of them, you can use closures. The for-each was one of those things. We already have that one, so we don't need to use closures to do that.
Neal Gafter: You don't, but the point is there are lots and lots of APIs in the world that would benefit from closures besides collections. It's not the only API in the world that needs to have API-specific control constructs. It's the only one that got them so far, but it's not the only one that deserves them.
Bill Venners: At the end of your talk, you were discussing some of the functional programming styles that can be done with closures. You talked about map. Can you talk about how useful the functional styles might be? When you add closures, can you program in a different style in Java?
Neal Gafter: There is a style of programming that you see in the functional programming community. Some people like it. I think the main benefit from adopting that style is when you start using that with the concurrency framework that takes advantage of that. For example, just having that map operation might be only useful in some cases, but if you can do it in such a way that you invoke the function concurrently, especially if the function is particularly expensive, that becomes a very concise way of writing something that is otherwise very difficult to do. That is what I was alluding to when I mentioned the fork-join concurrency framework that Doug Lea has waiting in the wings. There is no easy way for clients to use that fork-join concurrency framework because there is no way of expressing the API in Java now that makes it convenient to use. If you have closures, you could have this
map interface that takes one
T and this function that maps from
U, and returns a
U. That's mildly useful, but when you add a parameter that's an
Executor so that you can do that mapping in parallel, it becomes much more useful because you have a way of very conveniently doing fork-join parallelism. There are some cases where there is a huge benefit to do that kind of thing concurrently, if that function is expensive, for example.
I think the most important uses of stuff from the functional programming community will be in a concurrent setting. I don't really have in my mind all of what that looks like, but I hope to get Doug Lea involved early, before I actually put out a prototype or a compiler that does closures. I want to get Doug Lea involved so I can include some of his fork-join concurrent framework, have him express those using closures, so that other people would get a feel for what those actually would look like in JDK 7.
Bill Venners: Given you work at Google now, what would be the process for getting this proposal into JDK 7?
Neal Gafter: I'm working on this on my own time. The main reason I'm working on this on my own time is that Google's JCP representative has decided that Google does not support this work. So for that reason, I cannot propose a JSR on Google's behalf. But what I can do is provide a prototype that someone, for example, Sun Microsystems, perhaps, could use a basis for a JSR proposal. As a practical matter, I think Sun wants to roll all language changes that might go into JDK 7 into a single JSR anyway. If that's the case, it doesn't make sense to have me be the spec lead for the language change JSR for JDK 7. It will probably be someone at Sun Microsystems, and closures would be included in that, I hope, as a candidate for JDK 7. Doing the prototype, I hope, increases its chance of being considered, because people will have a chance to try it out and see what it actually buys for them.
Bill Venners: What has the response been from your viewpoint?
Neal Gafter: It's very mixed. People seem to have strong feelings about it, one way or another. I don't think there's really a consensus. There are a lot of people who have experience with languages like Ruby that feel that having closures will increase the joy of programming in Java and make it possible to do things where they currently have to fight with the compiler to coerce it to do what they want. There are some people that think that you can actually do with anonymous inner classes all of the things that you can do with closures. I, frankly, don't believe that. That was really the point of my presentation, trying to make that clear. And there are some people that understand the difference in power, but believe that Java programmers are not up the task of taking advantage of closures without shooting themselves in the foot.
Bill Venners: How can they shoot themselves in the foot?
Neal Gafter: For example, Josh Bloch's position is that if we have closures in the language, then we effectively have an extensible language, and if you have extensible languages, different groups will be programming in different dialects of the language, and that means the programmers will not be able to move from one project to the other and understand what the code does. I think that's true, however, programmers that use different API sets already program in different dialects of the language, and there is already a problem that when people start looking at code using an unfamiliar API, they don't understand the code until they learn the API. And closures are just APIs in that sense.
I don't think closures increase the level of difficulty of understanding code that uses an unfamiliar API at all compared to the situation we're in today. If you use one of the twelve XML APIs, and you use a different one when switching from one project to another, you simply have to learn it. If a project uses certain closure methods that you're unfamiliar with, you need to read the documentation and see what those methods do. If you use a good IDE, you can hover over the method name, and it will point you to the JavaDoc. The JavaDoc pops up on the screen and tells you. That's all there is to it. I think that's better than adding two or three language constructs every two or three years, because with those language constructs, you don't get the popups that tell you what the language construct does. You need to learn them whether you're going to use them or not, because they're pervasive. Whereas with APIs, you do have the JavaDoc to help you.
I think it is really an honest difference of opinion. My hope is that having a prototype out there for people to be able to use, and having people using it, will give some actual evidence about whether or not this is a problem, and to what extent it's a problem, and how much does it really solve people's problems to have closures in the language versus not having closures.
Closures for the Java Programming Language, a proposal by Gilad Bracha, Neal Gafter, James Gosling, and Peter von der Ahé:
Closures for Java, a webcast of Neal Gafter's JavaPolis talk that he gave just before this interview:
Neal Gafter's blog, in which he discusses the closures proposal:
Bill Venners is president of Artima Software, Inc. and editor-in-chief of Artima Developer. He is author of the book, Inside the Java Virtual Machine, a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Bill has been active in the Jini Community since its inception. He led the Jini Community's ServiceUI project, whose ServiceUI API became the de facto standard way to associate user interfaces to Jini services. Bill also serves as an elected member of the Jini Community's initial Technical Oversight Committee (TOC), and in this role helped to define the governance process for the community.