Artima Developer Spotlight Forum - Groovy Creator James Strachan on Scala

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Artima Developer Spotlight Forum
Groovy Creator James Strachan on Scala

69 replies on 5 pages. Most recent reply: Jun 14, 2011 3:47 PM by Wanderson Santos

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 69 replies on 5 pages [ « | 1 2 3 4 5 | » ]

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Groovy Creator James Strachan on Scala

Posted: Jul 14, 2009 1:41 PM

> The compiler wanted a Set[Fruit]. So don't ask questions.
> Just give it one. And move on. The programmer may decide
> to try this route, especially if they are under deadline
> pressure. The result is not dangerous code. It works fine,
> even if the programmer didn't fully understand all the
> intricacies of variance that caused the compiler error in
> the first place.

Sure in this one example that you crafted, the result was not dangerous. This is light-years away from saying that there is no way for a developer to write dangerous code based on not understanding or misunderstanding the language. Are you suggesting this is true and can you show that it is the case? For example, can you show that implicit defs cannot cause any unexpected issues when the developer doesn't understand how they work? How about pattern matching?

In my experience I have seen kind of development cause major problems in terms of production issues and maintenance. Often these issues take years to become apparent and can cause unlimited damage. I have no reason to believe that Scala mitigates this problem. At no time should code be accepted when it is not understood why it works the way it does. This requires that either the developer understand it or a reviewer understand it. Either way, someone must fully understand. Otherwise it's not programming. It's witchcraft.

Erik Engbrecht

Posts: 210
Nickname: eengbrec
Registered: Apr, 2006

Re: Groovy Creator James Strachan on Scala

Posted: Jul 14, 2009 2:37 PM

> This requires that either the
> developer understand it or a reviewer understand it.
> Either way, someone must fully understand. Otherwise
> it's not programming. It's witchcraft.

I agree fully but I don't think this is a universally held belief. It's very difficult to do, and I don't think it happens often in practice.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Groovy Creator James Strachan on Scala

Posted: Jul 14, 2009 3:13 PM

Hi James,
> > The compiler wanted a Set[Fruit]. So don't ask
> questions.
> > Just give it one. And move on. The programmer may
> decide
> > to try this route, especially if they are under
> deadline
> > pressure. The result is not dangerous code. It works
> fine,
> > even if the programmer didn't fully understand all the
> > intricacies of variance that caused the compiler error
> in
> > the first place.
>
> Sure in this one example that you crafted, the result was
> not dangerous. This is light-years away from saying that
> there is no way for a developer to write dangerous code
> based on not understanding or misunderstanding the
> language. Are you suggesting this is true and can you
> show that it is the case? For example, can you show that
> implicit defs cannot cause any unexpected issues when the
> developer doesn't understand how they work? How about
> pattern matching?
>
No. You can still shoot yourself in the foot I'm sure. I can't think of much risk in pattern matching offhand, but implicits are a sharp tool. I could probably come up with some scenarios that would lead people to think one thing is happening when something else is. I always suggest people be very conservative about implicit conversions for that reason. Though I can't *remember* having that problem as yet.

The main problem I've had with Scala is either not understanding why I get a compiler error, or when looking at hard-to-read code, not being able to easily see what the heck it is trying to do. That's different from thinking code is doing one thing when it is really doing something else. Note in my variance example what was not understood by the imagined programmer was why the compiler didn't let that programmer pass a Set[Apple] to a method that expects a Set[Fruit]. To fix the problem they passed a Set[Fruit]. So they understood everything about the code they eventually checked in. They just didn't understand why the previous version didn't compile. The reason the lack of knowledge could do no harm is that it can't do harm if it doesn't compile. And that kind of thing has definitely happened to me when working with Scala.

I've encountered Scala compiler errors for which I couldn't fully explain or understand why it didn't work, but I could plainly see I wouldn't be able to go that route. So I found another route that the compiler was happy with. I definitely understood the new route I took, and only later as I gained more experience came to understand with more depth why I got the original compiler error. An example of that that springs to mind is that you can only overload curried methods in Scala if the initial argument lists are unique. Subsequent argument lists are not considered when disambiguating which overloaded method you want. I never read that anywhere, but when I tried it in ScalaTest it didn't compile. So I found a different syntax that would compile even though I wasn't quite sure why the initial thing I tried was not allowed. Actually I still don't know *why* that isn't allowed. I'd have to ask Martin. But I understand how the mechanism works now such that I don't even consider trying to overload curried methods based on more than the first parameter list.

> In my experience I have seen kind of development cause
> major problems in terms of production issues and
> maintenance. Often these issues take years to become
> apparent and can cause unlimited damage. I have no reason
> to believe that Scala mitigates this problem. At no time
> should code be accepted when it is not understood why it
> works the way it does. This requires that either the
> developer understand it or a reviewer understand it.
> Either way, someone must fully understand. Otherwise
> e it's not programming. It's witchcraft.
>
Can you give a specific example or two? That may help me jog a Scala example like that out of my memory.

Erik Engbrecht

Posts: 210
Nickname: eengbrec
Registered: Apr, 2006

Re: Groovy Creator James Strachan on Scala

Posted: Jul 14, 2009 4:24 PM

> Can you give a specific example or two? That may help me jog a Scala example like that out of my memory.

I have an example that just happened this morning. It's no where near as dramatic as James' "unlimited damage," but I think it illustrates a point.

We're developing a Django application, but using SQLAlchemy + Elixir to access a multitude of legacy schemas. So this morning a junior developer stops by my desk...

Dev: I deployed the app to the server and it stopped working
Me: That's probably because mod_wsgi is multithreaded and the Django dev server isn't. You have something that should be in a thread local but isn't.
Dev: Ok, yeah, that makes sense. But how do I fix it?

So I go to his desk and we fix it, and right now I'm reviewing and refactoring some of his code to make sure there aren't more insidious threading issues lurking about.

Now I'll use this to illustrate both James' and your points.

If the developer had, say stumbled upon the fact that he could turn off multithreading in mod_wsgi instead of figuring out the root cause of the issue, or had simply randomly jiggled options until it worked, then we could have ended up with a major production issue when the app can only process one request at a time. Of course, such an issue probably wouldn't make it to production, but... I'm pretty sure the developer in question wouldn't do that, but I've worked with plenty who would.

I think that illustrates James' point.

Now the developer is smart, and he learns quickly. He has an innate aversion to magic he doesn't understand. He's not afraid to ask for help and I'm here to give it. So something that could have created severe latent issue was just a short education problem solving exercise.

Which brings me to your point. You make a mistake, the compiler yells at you, you try different things and find something that works, you know why your new thing works and eventually you figure out why the old things didn't.

This is a harmless problem solving/education exercise for you because you are a smart, senior developer and no doubt like to learn.

But more commonly people will flip bits until their problem goes away, without any clear knowledge of why the solution they find works. Their code fills up with cruft. Eventually even small code bases become completely unmaintainable because they flip a variable here and unflip it there and null it out in this other place (which, btw, is a much more compelling argument for immutability than parallelism). They can't explain what their code is doing. They just know that pretty much everything there was put there to solve some problem, and are afraid to delete it because then the problem might come back.

I think this is the type of developer who James' is afraid of. I know it's who I'm afraid of. It's the developer who spend years with a technology without ever learning how to do more than repeat patterns from simple how-to articles.

The more complex a technology is, even if that complexity is elegant and justified, the more likely a given developer is to fall into randomly poking code and repeating patterns without understanding. There is a cognitive load in Scala that is not present in Java or C#, and I think Scala could reduce some competent, disciplined Java/C# developers into a very frustrated random pokers.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Groovy Creator James Strachan on Scala

Posted: Jul 14, 2009 10:47 PM

Hi Erik,

> The more complex a technology is, even if that complexity
> is elegant and justified, the more likely a given
> developer is to fall into randomly poking code and
> repeating patterns without understanding. There is a
> cognitive load in Scala that is not present in Java or C#,
> and I think Scala could reduce some competent, disciplined
> Java/C# developers into a very frustrated random pokers.
>
That may very well be. I don't have any direct experience with folks who resorted to random poking in Scala, nor have I heard stories of such things as yet. One thing I can say for sure is that compared to Java and C#, Scala takes a much more trusting or empowering attitude, less of a restricting attitude. To the extent that Scala can unleash talented developers to do even better things, I think it may at the same time empower less talented developers to do even worse things.

Erik Engbrecht

Posts: 210
Nickname: eengbrec
Registered: Apr, 2006

Re: Groovy Creator James Strachan on Scala

Posted: Jul 15, 2009 4:39 AM

> That may very well be. I don't have any direct experience
> with folks who resorted to random poking in Scala, nor
> have I heard stories of such things as yet. One thing I
> can say for sure is that compared to Java and C#, Scala
> takes a much more trusting or empowering attitude, less of
> a restricting attitude. To the extent that Scala can
> unleash talented developers to do even better things, I
> think it may at the same time empower less talented
> developers to do even worse things.

Scala is still in the early adopter phase so I wouldn't expect it.

That being said...I think the strength of Scala's type system and the favoring of immutability has a lot of offer with regards to this problem if you have a senior guy who really knows Scala doing an lot of the higher-level work and encode some of the invariants so that they can't be so easily poked out. Of course, then he has to watch the commit logs like a hawk (val becomes var, case class becomes regular class, etc), but I think it could be worth it.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Groovy Creator James Strachan on Scala

Posted: Jul 15, 2009 9:25 AM

> No. You can still shoot yourself in the foot I'm sure. I
> can't think of much risk in pattern matching offhand, but > implicits are a sharp tool. I could probably come up with
> some scenarios that would lead people to think one thing
> is happening when something else is. I always suggest
> people be very conservative about implicit conversions
> for that reason. Though I can't *remember* having that
> problem as yet.

I think you are missing the larger point here. Someone who doesn't know what implicit defs are or how they work won't know that they should be used conservatively. Why would they? How can someone know how to use a tool properly if they don't understand what that tool does?

> Can you give a specific example or two? That may help me
> jog a Scala example like that out of my memory.

Just to clarify, I'm not saying it's invalid to work with a language and run into issues and do some research. This is how most people learn to use a given language. What I am saying is that the idea that it's OK for developers to write code, not understand how it works and have that moved into a production status is extremely dangerous. I have no problem with the idea that developers can learn to use Scala by using it. I have a problem with the idea that code appears to work but that no one understands is somehow acceptable.

And also to clarify, by "unlimited damage" I mean that there is no general limit to the damage a problem might cause. Most problems will create small issues but there is a potential for extremely costly errors. For example, I've dealt with errors that caused large amounts of database records to be improperly created in a way that could only be corrected manually.

Probably the most universal kind of issue has to do with data types. Many developers don't understand the difference between a floating point number and a decimal and think they are equivalent. This causes innumerable problems. No compiler will tell you that you are using the wrong type for your money fields. Similarly, if you don't know how large of a number can be stored in an int in Java, it might takes years for the numbers to wrap around and suddenly a system that worked yesterday starts churning out garbage or corrupting data or worse. Scala won't tell me that int isn't big enough for my data set. How could it?

I've recently seen a developer post a comment on this site about how implementing equals is easy because you just compare hashcodes. There's no way to even estimate the kinds of issues that might cause. No compiler will flag this as an error and there is no guarantee that testing would catch the flaw.

If use the XPath expression //Address, it will appear to work. The compiler won't complain. An address will be retrieved. The problem is that there might be many addresses in a document. All of the sudden, thousands of items are shipping to Amazon's headquarters instead of their customers' homes.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Groovy Creator James Strachan on Scala

Posted: Jul 15, 2009 10:18 PM

> Just to clarify, I'm not saying it's invalid to work with
> a language and run into issues and do some research. This
> is how most people learn to use a given language. What I
> am saying is that the idea that it's OK for developers to
> write code, not understand how it works and have that
> moved into a production status is extremely dangerous. I
> have no problem with the idea that developers can learn to
> use Scala by using it. I have a problem with the idea
> that code appears to work but that no one understands is
> somehow acceptable.
>
We probably agree, but maybe not. Let me ask yet another clarifying question.

I understand my Scala code, but only at a certain level of abstraction. I don't actually know what bytecodes the Scala compiler generates, nor what the JVM actually JIT-compiles that into, nor what the processor does with that machine code, nor in fact what the electrons are doing in the chip, nor do I know what quarks are made out of. So there are many lower levels of abstraction at which I don't have any idea how my program is working.

So my question is at what level do you think programmers need to understand their code, and I'll give you a specific example where I think it is OK that the developer not completely understand something about the Scala language, but yet they understand at a high level how the program works. In chapter 3 of Programming in Scala we introduce the Map literal syntax:


val romanNumeral = Map(1 -> "I", 2 -> "II", 3 -> "III")

We explain that the -> is a method call, that 2 -> "II" becomes 2.->("II"), and that you can call it on any object. And in a footnote we say: "The Scala mechanism that allows you to invoke -> on any object, implicit conversion, will be covered in Chapter 21." So they aren't going to really know how this works for many chapters to come. But I think they can already understand what it means, and I think it is probably fine that they go ahead and write production code using this Map literal syntax that takes advantage of implicit conversions, even though they don't yet understand all the ins and outs of implicit conversions.

So that's one thing, but the other is if you do agree that a newbie could probably use the Map literal syntax reasonably safely before they understand implicits, how different is it for a less capable programmer to use Map literals, someone who may struggle to understand how implicit conversions work in Scala even when they get to chapter 21s, if they ever did. That makes me more nervous, but primarily I think I should be nervous because this person may not know what they are doing in general, not because they are using a built-in implicit conversion for mapping keys to values before understanding the implicit conversion mechanism by which it works.

Daniel Jimenez

Posts: 40
Nickname: djimenez
Registered: Dec, 2004

Re: Groovy Creator James Strachan on Scala

Posted: Jul 16, 2009 8:26 AM

> So that's one thing, but the other is if you do agree that
> a newbie could probably use the Map literal syntax
> reasonably safely before they understand implicits, how
> different is it for a less capable programmer to use Map
> literals, someone who may struggle to understand how
> implicit conversions work in Scala even when they get to
> chapter 21s, if they ever did. That makes me more nervous,
> but primarily I think I should be nervous because this
> person may not know what they are doing in general, not
> because they are using a built-in implicit conversion for
> mapping keys to values before understanding the implicit
> conversion mechanism by which it works.

I think that's kind of James's point (in general, re Scala): at most places, most of the programmers should make most "good" developers nervous in the sense that they - the regular developers - really don't know what they're doing in general. That's why, and I apologize if I'm misinterpreting, James often claims that Scala can't/won't replace Java in the area Java is strong (ie, "enterprise" shops who favor quantity of developers over quality).

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Groovy Creator James Strachan on Scala

Posted: Jul 16, 2009 11:08 AM

> We explain that the -> is a method call, that 2 -> "II"
> becomes 2.->("II"), and that you can call it on any
> object. And in a footnote we say: "The Scala mechanism
> that allows you to invoke -> on any object, implicit
> conversion, will be covered in Chapter 21." So they aren't
> going to really know how this works for many chapters to
> come. But I think they can already understand what it
> means, and I think it is probably fine that they go ahead
> and write production code using this Map literal syntax
> that takes advantage of implicit conversions, even though
> they don't yet understand all the ins and outs of implicit
> conversions.

I guess the question is whether not knowing how it works could result in nasty surprises. My concern with implicit conversions isn't really about the kind of code here. The concern I have is with someone who doesn't understand implicit conversions making use of them. That person might not know that what they do with them could affect the code above. It really comes down to the old saw about how a little knowledge is a dangerous thing.

Bill Pyne

Posts: 165
Nickname: billpyne
Registered: Jan, 2007

Re: Groovy Creator James Strachan on Scala

Posted: Jul 16, 2009 12:53 PM

James, another concern is whether or not we stop moving forward as an industry, in terms of tools and abstractions, to accommodate the bottom 10% of developers.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Groovy Creator James Strachan on Scala

Posted: Jul 16, 2009 2:21 PM

> I think that's kind of James's point (in general, re
> Scala): at most places, most of the programmers should
> make most "good" developers nervous in the sense that they
> - the regular developers - really don't know what they're
> doing in general. That's why, and I apologize if I'm
> misinterpreting, James often claims that Scala can't/won't
> replace Java in the area Java is strong (ie, "enterprise"
> shops who favor quantity of developers over quality).

I don't fault you for thinking that but it's not exactly my argument.

Let me say that I do believe that the above is probably true for those shops that think having 10 weak developers is better than 2 really good ones. It's really irrelevant, however, because the language that is chosen by shops like this will be based more on what high priests (e.g. Gartner) says than the features of the language. Until you can cheaply outsource Scala development on a large scale, this kind of shop will not be using it.

The argument I've been trying to make for a while is that there are some features of Scala that are at best unlikely to be very useful on projects that must be maintained by many developers and at worst going to create major headaches for those projects.

Implicit conversions are one of those features. Maybe a team could restrict the use of these to a small set of approved cases but when I think of the next language I want to work with, bureaucratic processes about what code shall be allowed are not what I envision. in most cases they would either be disallowed entirely or no restrictions will be enforced. The former means they might as well no exist and the latter is unsustainable for long term development. If the benefits of implicit conversions could be attained in a way that more transparent then they could have great value but as they are implemented they make the language less appealing for team development.

Other features such as closures, first class methods, and pattern matching are arguably just as complex if not more complex. And I have no problem with these features. My concern is not that developers lack the ability to use these complex features but rather that some of these features are inherently disadvantageous for team development regardless of their complexity.

All of this is really not related to the argument I was making in this thread specifically. I'll address that in my response to Bill Pyne.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Groovy Creator James Strachan on Scala

Posted: Jul 16, 2009 2:33 PM

> James, another concern is whether or not we stop moving
> forward as an industry, in terms of tools and
> abstractions, to accommodate the bottom 10% of developers.

I'm all for moving forward. I just don't think that all paths are equally good.

As I mentioned above, however, this really is neither here nor there in terms of this thread. My argument hold whether you are talking about Scala, Java, COBOL or Assembler. To reiterate: the idea that it's alright for people to write code that they don't understand and have it be used (without competent review) in a production system is extremely dangerous.

I don't even see why I have to make this argument. How would you feel if airplane mechanics worked this way? I wouldn't want to fly anywhere if the appearance of possibly being correct was all that was required on a given repair. We expect competence in most skilled trades. Why shouldn't we expect it from developers? How this ties back to Scala is that Bill (Venners) made the following [paraphrased] argument: 'yes Scala is complex, but that's OK because you can write code in it without understanding it.' I just don't buy that argument.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Groovy Creator James Strachan on Scala

Posted: Jul 16, 2009 2:42 PM

Fred Garvin

Posts: 52
Nickname: fredgarvin
Registered: Jan, 2008

Re: Groovy Creator James Strachan on Scala

Posted: Jul 16, 2009 2:44 PM

> James, another concern is whether or not we stop moving
> forward as an industry, in terms of tools and
> abstractions, to accommodate the bottom 10% of developers.

I don't think anyone is suggesting we stop moving forward. The concern is over how we attempt to move forward. Indeed, a lot of folks here question whether or not the net effect of Scala is a step forward. And they present some rather convincing arguments. Not the least of which is the impact on the "average" developer. Pesonally, I think "average" includes just about everyone -- complexity whether it be in syntax, programming paradigm, or the type system, taxes us all directly and indirectly.

Flat View: This topic has 69 replies on 5 pages [ « | 1 2 3 4 5 | » ]

Previous Topic

Next Topic