Sponsored Link •
Bill Venners: What problems do closures try to solve?
Neal Gafter: What I'm try to accomplish with closures is to allow you to
write methods that are part of APIs that act like control statements in the
language. Because a lot of APIs do have specialized needs. You have the
concurrency framework where you'd really like something that acts like the
synchronized block, but for the concurrency locks. We did add a statement in
JDK 5, the for-each loop, which is specially designed to work with collections.
There are a lot of APIs that would benefit from having statement forms, and
there are proposals to add even more statement forms in JDK 7. My hope is
that by adding one language construct, we won't have to have a never-ending
series of additional statement forms because we keep having more and more
APIs that we care about. Instead, we can have one language extension and
those APIs can include those statement forms.
Bill Venners: One of the examples you gave us in your talk was
withLock. Can you elaborate on that?
withLock is a method that I hope would be
added to the concurrent framework. It basically looks just like a
synchronized statement. You say,
open-paren, then a lock variable, close paren, open curly brace, then a bunch of
statements, and then close curly brace. It looks just a like a built-in statement
form, but it's a method invocation, an invocation of this
method. And the
withLock method takes two parameters, the
lock, and a closure. It does the sort of boilerplate that you would have to do to
synchronize a block of code using the concurrency framework yourself, but it's
much more convenient to use.
withLock example is a poster-child for closures. If you don't have closures,
and you're never going to have closures, you really want to put something like
that in the language. But, again, it's a never-ending series of language
Bill Venners: What makes a block of code a closure, instead of just a block of code?
Neal Gafter: The syntax I just described, where you have a function declaration that looks like it has a block of code after it, that's not legal syntax today. That is a new syntax in the language, and we would define that syntax to mean that you take the block at the end and wrap it in a closure and pass it as an extra parameter to the method.
There is also syntax where you can write closure literals, where you can explicitly pass two or three of them as parameters to a method, and that's a new syntax too. The way you write a closure is, open curly brace, the argument declarations, an arrow (equals-greater-than), a bunch of statements, and then a closed curly brace—that's a syntax that's not in Java right now. That would be the way you write a closure literal in the language. It would be wrapped by the compiler in an interface type when it's passed to the method to which you pass it in as a parameter.
Bill Venners: In your talk, you highlighted the differences between
closures and anonymous inner classes, such as the meaning of
return. When you have a closure, you can return from...
Neal Gafter: ...the enclosing method. Actually, there are some places where there are no
enclosing methods. If you wouldn't be allowed to write a
return anyway, then
you won't be allowed to write a
return inside of a closure. But a
return from a
closure simply returns from the enclosing method. The fact that it's inside of a
closure doesn't change the meaning of the
Bill Venners: What happens under the covers? In your talk, you showed
the interface that has an
invoke method in it. Could you describe what really
Neal Gafter: The closure is turned at compile time into an instance of
the interface. If you take the example of the
that method is declared to take a function type as its last parameter. At compile
time, the compiler looks at the closure and says, oh, I actually need something
of this interface type. (Function type is an interface type.) So it builds an
instance of that interface, just like an anonymous inner class, but it's not an
anonymous inner class, because the scoping rules for a closure are different
from an anonymous inner class's scoping rules. It creates an object that
implements that interface where the body of the
does what the code inside the closure says it should do. And it passes that
instance as an extra parameter to the
withLock method. And
withLock method, in order to cause that code to be
executed, you have some parameter, like
block.invoke(). That's all you need to do, and it simply calls
invoke method on that interface from the object that was
Bill Venners: If that block executes a
return, how does it return from the
enclosing method, which called
Neal Gafter: There are actually a few different answers to that question.
The simplest answer is that a return from a closure will be implemented using
exceptions. So the compiler will create and throw an exception, and that will
pass out through the invocation of
withLock to the caller that
withLock, where it will be caught. And the value that
you're returning will be pulled out of the exception, and the code
that catches it will actually do the returning from the enclosing method.
As a practical matter, if you use that pattern, where you put a return inside of a closure, it will get inlined by HotSpot, and it will turn into a jump, which is exactly what it would have done if you used the statement in the language instead of a closure. If the closure only executed once, it won't, but if the code is executed more than a couple of times, and the return is executed very frequently in the closure, then HotSpot will inline that. And it will turn into the same code at the machine code level that it would have, if you had done it via a control statement added to the language.
Bill Venners: One thing I really liked in your presentation was about methods that exist in existing APIs today that you could pass a closure to, even though they never imagined that would be possible.
Neal Gafter: Let's take a particular example. In the concurrency
framework there is an interface called
Executor. It represents a
thread pool, among other things. One of the things you typically do with a thread
pool is you submit a
Runnable to it. It takes the
Runnable, and assigns it to a thread to run now, or as soon as
a thread is available. That method is called
execute. If you want
to use this method, but you want to use a closure, because of the convenience
of the syntax, you can. Remember I said the closure conversion takes the
closure and figures out what kind of interface type it needs and that's the kind of
interface that's built at compile time to be passed it. So if you
executor.execute(), and then open-close curly braces, and write a block
of code inside the curly braces, that block of code is turned into a closure. It's
turned into an object that implements
Runnable, and the
run method does whatever that block says it should do. That
object is passed to the
execute method of the executor
No change needs to be made to the executor framework because a closure is
always converted to whatever interface type you need statically in the context of
the invocation of the code. You can use a closure in place of a
Runnable, you can use it in place of a
interface you need—you can write a closure literal in that place and
pass it to that API.
Bill Venners: So if make a new API after closures are in the language, and I say explicitly that I take a closure...
Neal Gafter: It's just a parameter, right? All you have to do is declare it as the last parameter of the interface.
Bill Venners: So all I have to do to be ready for a closure being passed to me is that the last parameter is an interface?
Neal Gafter: You can receive a closure in any parameter position. The only advantage of having it in the last parameter position is you get to use the special syntax that looks like a statement. That's the point. If it is the first parameter instead of the last parameter, it's no problem. But you would use the closure-literal syntax with the fat arrow instead of putting a block after the invocation. But you can do that. In fact, if you have an API where you need to pass in more than one closure, you have to use the closure literal syntax.
Bill Venners: Basically any existing method call at the end, if it's an interface... It has to be an interface, can't be a class?
Neal Gafter: It has to be an interface, and an interface with one method in it.
Bill Venners: So that narrows it down a little bit the existing APIs that will be able to be called with closures.
Neal Gafter: It does, but there a lot of those—all of the call-back
registration stuff in Swing, the executor framework, and so on. There are
Runnables—a lot of places
where you could take advantage of this, even
java.util.Comparators that you use for sorting. You can write a
block of code that describes the
Comparator right in the
sort invocation. That works just fine. In many of the APIs where
people currently use anonymous instances in the invocation, you can use
closures instead—but not all of them. If you have something where you
really need more than one method to be in there, you can't easily use closures
for that. Actually, you can, but not easily. You have to write a method that makes
an instance of a class and then invokes the closures for each one. It's not
pretty. It's really mainly suitable for the ones that have a single method.
Bill Venners: Tell me about exception transparency.
Neal Gafter: The idea of exception transparency is that sometimes you have an API to which you pass in a closure, and if the closure throws an exception, you want your method to be known to throw the same exception so that exception checking works.
A really good example of that is the
withLock that works with the
new locks. If the block of code in the
withLock statement throws
some exceptions, you want the compiler to know that those exceptions are
going to be propagated out from the
withLock call. In the
closure specification, therefore, we've added some stuff to the machinery of
generics. It's only a little bit; it's not very complicated. We added just enough to
make it possible for you to express that the exception propagates out.
Bill Venners: How does this affect the way checked exceptions work?
Neal Gafter: If the block throws, say, a
NumberFormatException, then the compiler is going to
complain that it's not caught. But if you can declare that in the enclosing
method, then the compiler won't complain. You can't declare the
withLock method to throw
NumberFormatException. That would be silly, because not
everyone who uses
withLock has to deal with
NumberFormatException. In fact, there is no particular
exception that you can declare in
withLock. But using this new
mechanism, you can say whatever type is thrown by the block is thrown by
withLock. If no checked exceptions are thrown by the block, then
nothing is thrown by
withLock. So you have a type parameter for
the exception in the API declaration (such as the
method), and you say, "The closure throws this exception X, and so do I." That
allows you to get the exception checking for APIs where the implementation will
propagate the exceptions out.
The reason you need a special syntax for that, the reason you can't just use an ordinary type parameter, is if you use an ordinary type parameter, that will work as long as there is exactly one exception that's thrown. If there is no exceptions thrown, what are you going to fill in there? And what if there are two exceptions thrown? Then you're just out of luck.
The idea is that if you declare a type parameter with the
keyword, then you're allowed to put multiple types in there for the exceptions,
and the way you list them all out, you can't use commas, those would look like
extra type parameters, you use a vertical bar. But you don't normally have to
specify those type parameters explicitly. You just pass a closure that happens
to throw three things, and the caller has to catch those three things. That just
works. But if you do want to explicitly pass a type parameter, which is the three
different exceptions that can be thrown by a block, you just list the three in that
type parameter position with a vertical bar between them.
Bill Venners: So the compiler will always be able to tell by looking at each individual invocation of a method that takes a closure what the exceptions thrown by the block will be, right?
Neal Gafter: The set of exceptions that are
thrown by a closure are part of the type of the closure. The interface to which
that's converted must declare those exception types. The closure that's
invoke must declare that in the
throws clause of the
method of that interface.
Bill Venners: Can you give an example?
Neal Gafter: For example,
withLock is a method. It's
second parameter is an interface that contains a single method called
invoke. Whatever exceptions the passed block throws must be
in the throws clause of that method. And the closure conversion checks that.
withLock method, you don't
know what that's going to throw. That's why you use a generic type parameter.
But the compiler, when it's looking at the closure, knows exactly what is thrown.
It can see that you're calling
Integer.parseInt(), for example,
and it knows that it throws
compiler knows what exceptions are thrown by the closure, and it matches
those and makes sure that the interface that the closure is converted to
declares all the exceptions that are actually thrown. If it's a generic, then the
compiler knows what a generic is supposed to be, it's supposed to be
whatever is actually thrown by the closure. It fills in the type parameter on that
Bill Venners: The other thing you showed at the end was some
functional programming styles. I had written a blog post about how it becomes
increasingly difficult to add features to a language over time because you have
to be backwards compatible with more and more existing features. If there is a
better way to do it, like iterations, you leave in the old way. So we have an
Enumeration, and with closures we'd
be able to pass a closure to an
Neal Gafter: Yes, but I don't think there is any point in doing it. I don't
think there is a point in adding an
each method to
Collections, or anything like that. I don't think it does anything for
you. And if you do it, you don't need to put it in the interface. If you want to put
a static method in
each that takes a
Collection and a closure, and loops over the
Collection and calls the closure on each of them, you can do
that just fine, there is no backward compatibility problem.
Bill Venners: So you say it we don't need it because we already have the for-each?
Neal Gafter: I'm saying that there is no problem leaving the for-each in, and as a practical matter I don't think you need it. It doesn't do anything for you that you can't already do.
Now, if you had closures before JDK 5, you probably wouldn't have wanted to add that statement to the language. But I don't think there is any harm in having it there.
Bill Venners: So basically your original justification for adding closures was adding control statements. So instead of adding a bunch of them, you can use closures. The for-each was one of those things. We already have that one, so we don't need to use closures to do that.
Neal Gafter: You don't, but the point is there are lots and lots of APIs in the world that would benefit from closures besides collections. It's not the only API in the world that needs to have API-specific control constructs. It's the only one that got them so far, but it's not the only one that deserves them.
Bill Venners: At the end of your talk, you were discussing some of the functional programming styles that can be done with closures. You talked about map. Can you talk about how useful the functional styles might be? When you add closures, can you program in a different style in Java?
Neal Gafter: There is a style of programming that you see in the functional
programming community. Some people like it. I think the main benefit from
adopting that style is when you start using that with the concurrency framework
that takes advantage of that. For example, just having that map operation might
be only useful in some cases, but if you can do it in such a way that you invoke
the function concurrently, especially if the function is particularly expensive, that
becomes a very concise way of writing something that is otherwise very difficult
to do. That is what I was alluding to when I mentioned the fork-join concurrency
framework that Doug Lea has waiting in the wings. There is no easy way
for clients to use that fork-join concurrency framework because there is no way
of expressing the API in Java now that makes it convenient to use. If you have
closures, you could have this
map interface that takes one
and this function that maps from
and returns a
mildly useful, but when you add a parameter that's an
so that you can do that mapping in parallel, it becomes much more useful
because you have a way of very conveniently doing fork-join parallelism. There
are some cases where there is a huge benefit to do that kind of thing
concurrently, if that function is expensive, for example.
I think the most important uses of stuff from the functional programming community will be in a concurrent setting. I don't really have in my mind all of what that looks like, but I hope to get Doug Lea involved early, before I actually put out a prototype or a compiler that does closures. I want to get Doug Lea involved so I can include some of his fork-join concurrent framework, have him express those using closures, so that other people would get a feel for what those actually would look like in JDK 7.
Bill Venners: Given you work at Google now, what would be the process for getting this proposal into JDK 7?
Neal Gafter: I'm working on this on my own time. The main reason I'm working on this on my own time is that Google's JCP representative has decided that Google does not support this work. So for that reason, I cannot propose a JSR on Google's behalf. But what I can do is provide a prototype that someone, for example, Sun Microsystems, perhaps, could use a basis for a JSR proposal. As a practical matter, I think Sun wants to roll all language changes that might go into JDK 7 into a single JSR anyway. If that's the case, it doesn't make sense to have me be the spec lead for the language change JSR for JDK 7. It will probably be someone at Sun Microsystems, and closures would be included in that, I hope, as a candidate for JDK 7. Doing the prototype, I hope, increases its chance of being considered, because people will have a chance to try it out and see what it actually buys for them.
Bill Venners: What has the response been from your viewpoint?
Neal Gafter: It's very mixed. People seem to have strong feelings about it, one way or another. I don't think there's really a consensus. There are a lot of people who have experience with languages like Ruby that feel that having closures will increase the joy of programming in Java and make it possible to do things where they currently have to fight with the compiler to coerce it to do what they want. There are some people that think that you can actually do with anonymous inner classes all of the things that you can do with closures. I, frankly, don't believe that. That was really the point of my presentation, trying to make that clear. And there are some people that understand the difference in power, but believe that Java programmers are not up the task of taking advantage of closures without shooting themselves in the foot.
Bill Venners: How can they shoot themselves in the foot?
Neal Gafter: For example, Josh Bloch's position is that if we have closures in the language, then we effectively have an extensible language, and if you have extensible languages, different groups will be programming in different dialects of the language, and that means the programmers will not be able to move from one project to the other and understand what the code does. I think that's true, however, programmers that use different API sets already program in different dialects of the language, and there is already a problem that when people start looking at code using an unfamiliar API, they don't understand the code until they learn the API. And closures are just APIs in that sense.
I don't think closures increase the level of difficulty of understanding code that uses an unfamiliar API at all compared to the situation we're in today. If you use one of the twelve XML APIs, and you use a different one when switching from one project to another, you simply have to learn it. If a project uses certain closure methods that you're unfamiliar with, you need to read the documentation and see what those methods do. If you use a good IDE, you can hover over the method name, and it will point you to the JavaDoc. The JavaDoc pops up on the screen and tells you. That's all there is to it. I think that's better than adding two or three language constructs every two or three years, because with those language constructs, you don't get the popups that tell you what the language construct does. You need to learn them whether you're going to use them or not, because they're pervasive. Whereas with APIs, you do have the JavaDoc to help you.
I think it is really an honest difference of opinion. My hope is that having a prototype out there for people to be able to use, and having people using it, will give some actual evidence about whether or not this is a problem, and to what extent it's a problem, and how much does it really solve people's problems to have closures in the language versus not having closures.
Have an opinion about the closures proposal? Discuss this article in the Articles Forum topic, Considering Closures for Java.
Closures for the Java Programming Language, a proposal by
Gilad Bracha, Neal Gafter, James Gosling, and Peter von der Ahé:
Closures for Java, a webcast of Neal Gafter's JavaPolis talk that he gave just before this interview:
Neal Gafter's blog, in which he discusses the closures proposal:
Bill Venners is president of Artima, Inc. and editor-in-chief of Artima Developer. He is author of the book, Inside the Java Virtual Machine, a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Bill has been active in the Jini Community since its inception. He led the Jini Community's ServiceUI project, whose ServiceUI API became the de facto standard way to associate user interfaces to Jini services. Bill also serves as an elected member of the Jini Community's initial Technical Oversight Committee (TOC), and in this role helped to define the governance process for the community.