Re: Groovy Creator James Strachan on Scala
Posted: Jul 10, 2009 5:24 PM
All very reasonable. I'll insert a few comments below:
> > I've never heard anyone claim that "Scala is somehow
> > simple,"
> It's somewhat subject to interpretation. I think that
> James Strachan's post gave the impression.
> > but I have heard several people claiming that
> > Scala is "incredibly complex." For example, you just
> > that. I think both claims that "Scala is simple" and
> > "Scala is complex" are misleading. Those claims, both
> > them, are too simplistic.
> True. It's hard to properly qualify it.
> > What I've been trying to get across is that Scala is
> > complex like your car is complex. Car these days are
> > indeed very complex, but does that mean no one but PhDs
> > will be ever able to figure out how to drive cars? Are
> > cars doomed to be a niche idea because they are so
> > complex? Of course not.
> Did I say anything about Scala being doomed?
No, sorry. I didn't mean to imply you said that. Several other people have said things like they think Scala is too complicated or esoteric for "mainstream" developers.
> > The trend I have seen is that as
> > the mechanisms that make up a car have become more
> > sophisticated, and yes more complicated, over the
> > the user experience has become easier.
> Easier is subjective. To continue with your car analogy,
> there are some people who find driving a manual
> transmission easier than an automatic, because it gives
> them more control, and thus enables them to do things that
> they couldn't do with an automatic transmission.
> > You don't need to
> > understand everything about how a car works to drive it
> > and get benefit out of it.
> Yes, but occasionally not understanding certain parts will
> bite you very hard.
> > Certainly in any of these mechanisms, Scala or cars,
> > sometimes the complexity bubbles up into your face.
> > But it
> > happens when you're actually programming in Scala about
> > often as it happens in modern cars, which is to say
> > relatively rarely.
> Cars, yes, Scala, not so much...
This is also subjective. Nowadays I can make Scala do whatever I want. I can imagine some DSL-like syntax and know first if it can be done in Scala, and also immediately how to do implement it. But at one point I was a beginner and somehow I made it through. I remember not knowing how to do the simplest things in Scala, and it taking a while to figure out. That was before there were good books to learn from and look things up in. But even then my main experience did not feel like I was fighting complexity so much as simply fighting unfamiliarity.
> > It happens more often in library design
> > than library usage, but even in library design the
> > occasional puzzles aren't that difficult to solve.
> The perceived complexity of the puzzles is usually
> proportionate to how far you push the language, and how
> much experience you have pushing it in that particular
> I think we're disagreeing on two points:
> 1. The extent to which complexity in libraries bleeds into
> application code
> 2. The dichotomy between library and application
> In many circumstances #1 is the library developer's fault,
> not Scala's. In my experience, most of the accidental
> complexity in these cases has to do with type inference
> failing, especially when this failure causes an implicit
> to not be triggered. This is a pretty easy situation to
> create, especially in there implicits on commonly used
Can you think of some specific examples? This doesn't ring a bell. My main complaint about libraries I've seen is that they just aren't designed tastefully. They may overuse operators, according to my subjective design sense, for example.
> #2 is probably where most of our real disagreement is.
> With Java, much of the really advanced stuff is
> s accomplished through means that are very cumbersome,
> like reflection, or extrinsic to the Java language, like
> runtime bytecode manipulation. Not many people are going
> to dive into those waters and play sorcerer's apprentice.
> But Scala is different. Scala provides a lot of powerful
> features directly within the language. There is no great
> barrier to scale in order to use them. Thus people will,
> and quite rightly I might add. It doesn't make much sense
> to me to pitch a language based on its powerful
> expressivity, and then in the next breath tell the masses
> that these features are only the province of library
> developers. Especially when it's so easy to make those
> libraries fall by failing to import something, importing
> too much, or defining something yourself that leads to
> ambiguous implicits.
> I also think application developers should always keep the
> idea that they are writing a library in mind. That
> doesn't mean they should always try to make library-like
> code, it just means they should always be on the lookout
> for patterns that could be abstracted away. Also, if
> someone else if calling into your code, then your code is
> essentially a library to them.
Well I wouldn't tell people "these features are only the in the province of library designers." I do suggest people be very conservative about operators and implicit conversions, but that's in general. It goes for library design too.
What I have tried to explain is that in my experience, it was only during library design that I've felt I was fighting what I'd call complexity, and even then not that often. When using libraries I've had compiler errors, but they didn't seem complicated to figure out. Mostly I have mistyped a method name or have an overload problem or what not. But twice in designing ScalaTest I had to think hard about a problem.
One occasion was when I needed to figure out what the variance of Matcher's type parameter should be. That required me to go jogging up a hill and visualize an inheritance hierarchy, and think about how it would be used. I decided on that jog that Matcher seemed to be contravariant in its type parameter. When I got back to my laptop I put in a minus sign to declare the type parameter contravariant. I did a compile. It worked, and neither I nor any ScalaTest user has had to worry about it since. That's what I mean by users of the library don't need to face the complexity that was faced and solved by the library designer.
The other head scratcher was getting this syntax to look like I wanted:
Map("one" -> 1, "two" -> 2) should (not contain value (5) and not contain value (3))
This syntax gave me a complicated problem to figure out. The trouble was that for the line to compile, the expression on the right (not contain value (5) and not contain value (3)) needed to have an inferred type of Map[String, Int]. But if you look, you'll see there's no way the compiler can figure out the key type of String, because there's no information about that in (not contain value (5) and not contain value (3)). All that's talked about there is the value type, Int. So the Scala compiler infers Map[Nothing, Int].
Nothing is a bottom type, which is a subtype of every other type, but because Map's key type parameter is nonvariant, that means Map[Nothing, Int] is not a subtype of Map[String, Int]. Which in turn meant I got a compiler error. This one really was a head scratcher for me, but at one point I said to myself, self, all I know about the key type in the right hand expression is that it has some type, but I just don't know what the type is yet. That sounded to me like an existential type. That's what an existential type says, so I decided to try and put one in there, and voila, it all worked. (And I haven't had to think about it since, and neither have my users.)
Both of these two examples are what I'd call fighting complexity, but in all of the work I've done on ScalaTest, these were really the only two puzzles like this. And users of ScalaTest probably have no idea what the variance of Matcher is, or that there's an existential type hidden in there in one place to get the Scala compiler to delay inference until it has enough information to do it. They just type in the syntax and it works.
> > What's
> > really hard about Scala given its sophistication, is
> > writing a Scala compiler, just like what's hard about
> > these days is actually building one.
> No one will argue with you there.
> > And that's why saying Scala is simple is also a
> > statement.
> > Writing client code in Scala is usually pretty
> > simple, but I would not say that without the "usually
> > pretty" in there. Because sometimes things will bubble
> > But it isn't hardly what I'd call complicated either.
> > is generally quite straightforward.
> I think that the bubbling up occurs frequently enough that
> a good Scala developer needs to know the more advanced
> features of Scala. Either that or start memorizing
> patterns to avoid the pitfalls without actually
> understanding what causes the pitfalls. While I'm sure I
> do this myself occasionally, I this approach makes
> significant contributions to the sorry state of software
> engineering. And, as I said before, I don't think people
> should be chased away from language features.
> It's worth pointing out that I don't consider may
> statements about Scala's complexity to be a criticism of
> Scala. Scala merges two very different paradigms (OO and
> Functional) while maintaining an astounding degree of
> backwards compatibility with Java. This is no small feat.
> But the power Scala gives you isn't free, and I don't
> t think the price can be paid entirely by library
> designers. Even if it could, I think it would be a
> terrible shame for application developers to use such a
> powerful tool in such a rudimentary way.
> I suppose I should say that I think there is very little
> accidental complexity in Scala. It's almost all there for
> good reasons to accomplish goals that justify the
I agree that there's not much accidental complexity in Scala. The complexity I see comes mostly from a few areas. One is that there's some extra complexity in Scala due to its goal of being compatible with Java. There are a number things you could toss out if you were to toss out that design goal, and that would make the language simpler. But for me the Java compatibility is very much worth the extra complexity it adds.
Another bit of complexity comes from the union of OO and functional. A more exclusively OO language, or a more exclusively functional language, would probably be simpler. But the union in Scala is pretty elegant and consistent, and frankly I prefer the choices it gives me as a programmer. Even though the language may be a bit more complex, I think having more choices at my fingertips makes my life as a programmer simpler. I've seen this over and over again when designing ScalaTest.
But of course the 800 pound complexity gorilla would have to be the type system, which if it dropped some of its features would be simpler. But with every feature dropped, something would become harder and more complex when using Scala as a tradeoff for a simpler language to learn. There's a good reason for each piece of the type system, and while it represents an impressive bundle of stuff you have to learn to be a highly proficient Scala programmer, it is learnable. And once you learn it, it really makes your programming life simpler.
What I'm not sure about is what the newbie experience really is. My newbie experience was skewed because I got to talk to Martin Odersky and Lex Spoon on the phone each week for two years as we wrote the book, and I could get good answers to my questions. We worked really hard to make the book carefully lead the reader one step at a time. So that each new concept built on the old ones. And reader comments seem to indicate that has worked OK. So I think with good educational materials it is an approachable language by the mainstream, but there will almost certainly be times as people are learning that the compiler forces them to face something they don't quite understand yet. But I think most people will be able to figure it out, and then they'll learn a lesson about Scala that will make it quicker for them to figure out next time. I think to really get good at Scala, people will have to write code in it and face down a few of those obscure-for-them compiler errors along the way.