A Conversation with Rich Hickey

---

In his keynote at the JVM Languages Summit last week, Rich Hickey, the creator of the Clojure programming language, suggested that the problems people associate with shared state are problems of time. He compared the lack of time management facilities in mainstream object-oriented languages today to the lack of memory management in C++. In this interview, Rich Hickey explains what he means by "problems of time," and why he believes that future languages should support constructs for time management directly.

Bill Venners: What do you mean when you say the problem of mutable state is a time problem?

Rich Hickey: If somebody hands you something mutable—let's say it has methods to get this, get that, and get the other attribute—can you walk through those and know you've seen a consistent object? The answer is you can't, and that's a problem of time. Because if there were no other actors in the world, and if time wasn't passing between when you looked at the first, second, and third attribute, you would have no problems. But because nothing is captured of the aggregate value at a point in time, you have to spend time to look at the pieces. And while that time is elapsing, someone else could be changing it. So you won't necessarily see something consistent.

For example, take a mutable Date class that has year, month, and day. To me, changing a date is like trying to change 42 into 43. That's not something we should be doing, but we think we can, because the architecture of classes is such that we could make a Date object that has mutable year, month, and day. Say it was March 31, 2009, and somebody wanted to make it February 12, 2009. If they changed the month first there would be, at some point in time, February 31, 2009, which is not a valid date. That's not actually a problem of shared state as much as it is a problem of time. The problem is we've taken a date, which should be just as immutable as 42 is, and we've turned it into something with multiple independent pieces. And then we don't have a model for the differences in time of the person who wants to read that state and the person who wants to change it.

Bill Venners: In what way is that a time problem?

Rich Hickey: If there was a way to glance at a date, at one point in time, there would be no problem. So that's the problem. You can't look at a date all at once. You only can look at it over time. That's a time problem. Similarly imagine you could change it all at once. Then you wouldn't have the problem of changing it from March to February, somehow in-between having some observable bad thing. That is also a time problem, because your perception wasn't atomic and your changing wasn't atomic.

The time problem is not easy to see in today's mainstream languages because there are no constructs that make time explicit. It is implicit in the system. We don't even know that's what we're doing when we use locks to try to make this work. Because what we're trying to do is partition time up to say, I'm going to get a portion of time when I get to look at it, and you're going to get a separate portion of time when you'll get to write it. That time management we have to do manually. We have to use locks and come up with some kind of convention, because it's not automatic. So that's why I'm saying, the problem here is a lack of automatic time management. We have to do that manually, just like we had to call delete before we had garbage collection. Somebody allocated something, and we had to call delete. It was our problem. It was manual. Now when we want to change a date or look at a date coherently, we have this time management problem that we use locks to try to solve.

Bill Venners: When you said "consistent," you meant you don't want to see February 31.

Rich Hickey: Right, that's inconsistent. No one ever intended that to exist. It's a side effect of our language that it can happen.

Bill Venners: In your keynote, you said we need to make decisions based on "stable values." What did you mean by that?

Rich Hickey: Let's say I'm trying to send out a notice a certain amount of time after a particular date. So I need to calculate something with that date. As I'm starting to do this calculation, if the date is mutable, it could change into something that doesn't work.

Bill Venners: Because the calculation takes time.

Rich Hickey: And the calculation uses the state. It will have to look at the pieces of it. It needs to perform addition with dates. It needs to look at the pieces and could see February 31 somewhere in the middle. Because it is not actually working with a value. If it were working with a value, if that date could not be changed, there would be no problem. We can't see that date without stopping time.

Bill Venners: What would built-in time management look like in a language?

Rich Hickey: For something as simple as a date, it just means get rid of the setDay, setMonth, and setYear methods. Treat dates as values. Have all the pieces be final. And all functions of dates produce new dates. They don't change dates. And I'm saying that's the right way to approach all of your objects. They shouldn't change. They should only have functions that produce new objects. That should be true for 42. When you add 7 to 42, 42 doesn't change. You get this different number over here, which is 49. Same thing with a date. If I want to say, three months from this date, I shouldn't be changing its fields. I should be making another date. And those fields should be final, immutable.

And if it is something bigger, like a collection, a persistent collection tree is the way to do that even for something big. You can take a two million item collection and add something to the end without copying the whole tree. And you still have the property where the old collection can't be changed. The new one is really a new collection. And that will make your programming life much better.

Bill Venners: You said in your talk that we'd be able to sleep better at night.

Rich Hickey: Yes, how do you sleep at night if you build your system on things that can be changing? How do you know you got it right? If you inserted locks in your program to try and coordinate time, how do you know you did it right? It is the same thing that kept you up at night when you had to manage memory manually. Do I have a memory leak? If I do, where is it? Where am I allocating that I should be freeing? Maybe in this call path I always free it, but when I go to this other call path, I fail to do that. Because it is a complex thing to do, it's easy to get wrong, which means it should be something that worries you.

We are suffering the problems of the lack of time management in our languages, but don't connect this as the cause. But we will over time. I think concurrency will make it more evident to people.

Bill Venners: Is the pain felt only when you have concurrency?

Rich Hickey: No, I don't think so. We talked about the date changing while you were looking at it. That's indeed a concurrency problem. But there's a second problem that exists without concurrency: You gave me a date, and when I looked at it, it said March 31. And I said, OK I'm going to need this date later. I want to remember it. Now it's not a concurrency problem. I've just aliased the date you gave me. I return from that method, but the date is stored somewhere. Another part of the program continues and changes that date, and eventually calls some code that looks at the date that was saved. And now it's different. So that's the non-concurrent case. That still makes programs very hard to understand.

Can I understand a large system if I call a method here and it can change things? How can I know the effect it will have? How can I know all the people who remembered that? How can I know that all the people who based a decision on that are not going to be thrown by this mutation that's occurred? I don't know. Somebody could have looked at a list of logged in users and seen that someone was there, and remembered that. They still have the collection. Then you change that. They are counting on something that's no longer true, but they have the same data structure. They have to keep questioning, do I need to look at this again?

A large system that's a graph of mutable objects is just very hard to understand. A lot of times you won't see that in a small program that an individual writes. But in a larger program written by multiple people, you can be looking at a problem in the program. You look at an object and its internal state seems strange or wrong. How did that happen? Those are the real problems of large object-oriented systems. That's very hard to understand. Also, if I change something over here, what will be impacted? Because the program is a graph. A big program becomes a network.

Bill Venners: Besides immutable values, what else would language support for time management entail?

Rich Hickey: It is a bunch of things that work together. You want to program with values. That can be anything from 42 to that date that you can't change, to a large collection that's also immutable. That's all you're going to be manipulating. And then you have time constructs. It can be as simple as a little box, and in this box will always be a value, and there are rules about how that can change, which will basically ensure that it will change from one thing to the next. And the thing that it changes to is a function of the thing that it is. There's a collection there with 10 things in it, and you call a function on that and produce a collection with 11 things in it, and the time construct makes sure that two people aren't doing that at the same time.

Bill Venners: What do you mean by time construct?

Rich Hickey: It will be something in the language. This is what we're missing in, say, Java or Scala: something in the language that does this. Maybe Scala Actors are a form of a time construct, where there you get the serialization because you send messages to the actors. You can also have a compare-and-swap (CAS) time construct where you say swap what's in this time construct using this function and these arguments. That will atomically call the function on what's in there already, make a new version and replace that back. And it will make sure if multiple people are trying to do that, only one will succeed at a time. They will be ordered. That's a tiny one. A bigger one is something like software transactional memory (STM), where everything that has an identity will have an associated reference, and those references will also have some function to change what's in there. If you do that outside of a transaction it will throw an exception to indicate it can't do this now.

So it's managing time. CAS would be a way to do it for a single cell. A transaction is a bigger thing that also has management. The time constructs will coordinate the time parts of your program. The rest of the program will be functions. We'll make new values by calling functions from values to values. And the only thing that will look like it is changing is the contents of these time constructs, which will have rules. The rules will enforce that changes happen atomically and sequentially. But it is very important that what's inside this time construct isn't the good old mutable date, because then it doesn't work. It has to be a succession of values.

Bill Venners: So the time construct encapsulates the succession of values.

Rich Hickey: It encapsulates the entire process. It manages the succession of values by giving you two things. First, it needs to make sure what is in it, and what's going to be in it, are both immutable. Second, it needs to give you a way to process what's in there, so you can move it forward in time using a function. The time management construct makes that happen sequentially and atomically.

Basically, it's just one level of indirection. For example, in Clojure there are things called references, and they are special. There are different kinds. There are ones that do CAS-like time, and some that do STM-like time. For the Clojure programmer, those are the only things that can change. Everything else can't change. Unless they are using Java stuff. Obviously I can't control that. So Clojure programmers work with things that can't change by default, but can also work with things that have identity when they need to. Maybe they have a collection that represents the logged in users. They are going to refer to it over time and expect to see different values over time. If they want to see values that differ over time, associated with that identity, they will have to put that inside one of these reference types.

Now they have a model. Now they know. If they look at it, they'll always see a valid list. They won't see a list that's in the process of being manipulated and has dead leaves, things like that. And they know that if multiple parts of the program need to maintain that list, that will be coordinated. That's now automatic. Before, when we both wanted to see a good date, that was our problem. Let's say we still want to see dates change logically, like we want to model a due date. That now has an identity. It's not the actual, physical date. It's an identity, a logical construct. When is this due? Maybe somebody will decide it is due at a different point.

Bill Venners: Like renewing a library book.

Rich Hickey: Right, so we need to change that date. Well, they're not actually changing the date. They are associating that identity, when it is due, with a different value. And the system can help. So that's what's automatic about it. It's automatic time management.

Resources

The Clojure programming language website is at:
http://clojure.org/

Talk back!

Have an opinion? Readers have already posted 46 comments about this article. Why not add yours?

About the author

Bill Venners is president of Artima, Inc., publisher of Artima Developer (www.artima.com). He is author of the book, Inside the Java Virtual Machine, a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Active in the Jini Community since its inception, Bill led the Jini Community's ServiceUI project, whose ServiceUI API became the de facto standard way to associate user interfaces to Jini services. Bill is also the lead developer and designer of ScalaTest, an open source testing tool for Scala and Java developers, and coauthor with Martin Odersky and Lex Spoon of the book, Programming in Scala.