The Artima Developer Community
Sponsored Link

Weblogs Forum
Getting Dynamic Productivity in a Static Language

78 replies on 6 pages. Most recent reply: Apr 5, 2009 6:19 PM by Cedric Beust

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 78 replies on 6 pages [ 1 2 3 4 5 6 | » ]
Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Getting Dynamic Productivity in a Static Language (View in Weblogs)
Posted: Mar 30, 2009 11:05 AM
Reply to this message Reply
Summary
In "The Feel of Scala," a talk I gave at Devoxx which is now available on Parleys.com, I show Scala solutions to real problems to help you get a feel for the language. Along the way I demonstrate how you can get many of the productivity benefits offered by dynamic languages using alternative static techniques in Scala.
Advertisement

In The Feel of Scala, a talk I gave at last year's Devoxx conference, I explain how I ended up choosing Scala instead of Ruby, Python, or Groovy for Artima's next JVM language. I show some ways in which you can reap benefits offered by these dynamic languages using both alternative static techniques offered by Scala as well as using dynamic techniques directly in the Scala language itself.

One of the most commonly cited productivity benefits of dynamic languages is that they enable much more concise programs compared to Java. While this is true, the productivity benefit here is actually conciseness, and that doesn't require dynamic typing. I show how type inference in Scala lets you get much of the conciseness of say, a Ruby or Python program, but in a statically typed language.

Another productivity benefit of dynamic languages is that, via a feature called open classes in Ruby or Python, you can add a method to a class and then call it. You can't do this in a static language. Nevertheless, I point out that the productivity benefit isn't actually in the dynamic adding of the method. The benefit, instead, is in being able to call the method. And that you can do in static languages using various other techniques. In C# 3.0, for example, you can make it possible to call new methods on classes via C#'s "extension methods" feature. In The Feel of Scala talk, I show how you can use Scala's implicit conversion feature to obtain that same productivity benefit.

Another oft touted productivity benefit of dynamic languages is that duck typing allows you to pass classes with common structures but no common supertypes to a method, which treats them uniformly. I point out once again that the productivity benefit isn't actually the duck typing per se, but the ability to pass the different types. I demonstrate three ways to achieve that in Scala. First, I show "view bounds," which is a completely static approach that involves implicit conversions. I also show structural typing in Scala, which is dynamic in the sense that reflection is used to invoke methods at runtime, but static in the sense that everything is checked at compile time. Lastly, I demonstrate using plain-old duck typing, exactly the same dynamic technique used in Ruby and Python, but this time in a static language. I show that as with duck typing in a dynamic language, spelling errors result in runtime errors, but then demonstrate that with a compiler plug-in, you can move that run-time duck typing error to a compile-time error, effectively extending Scala's static type system.

Lastly, another commonly cited benefit of at least some dynamic languages is that their syntax is more flexible, allowing you to make domain specific languages. While this is also true, the benefit comes from the flexible syntax, not the dynamic typing. During the talk I show that dynamic typing is not required for syntactic flexibility. I demonstrate the flexibility of the Scala language, showing how you can define new control constructs and make internal DSLs.

The talk suggests that although dynamic languages such as Ruby, Python, and Groovy are indeed in many ways more productive than Java, it isn't so much the dynamic typing that makes the productivity possible. Rather, it is the way these languages use dynamic typing to enable programmers work at a higher level compared to Java. I conclude the talk by explaining I choose Scala because I felt Scala gives me many (though not all) of the productivity benefits of the dynamic JVM language alternatives, while still letting me enjoy the productivity benefits of static typing as well.

You can view the complete one-hour presentation of The Feel of Scala on Parleys via this URL:

http://tinyurl.com/dcfm4c


Michele Simionato

Posts: 222
Nickname: micheles
Registered: Jun, 2008

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 30, 2009 11:32 AM
Reply to this message Reply
All you say is correct. However, I see the
advantage of static typing in other situations.
Suppose for instance you have a list of objects,
where the objects are of different types and can
be lists themselves (typical case a hyerarchical
data structure, say coming from a XML file).
A typical problem is to flatten such a nested list.
This is a solution in Python <=2.5:

def flatten(x):
res = []
for el in x:
if hasattr(el, '__iter__'):
res.extend(flatten(el))
else:
res.append(el)
return res

print(flatten([1, 2.0, ["abc", [3, ""]]]))

How would you solve a problem like this in Scala?

Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 30, 2009 12:03 PM
Reply to this message Reply
> All you say is correct. However, I see the
> advantage of static typing in other situations.
> Suppose for instance you have a list of objects,
> where the objects are of different types and can
> be lists themselves (typical case a hyerarchical
> data structure, say coming from a XML file).
> A typical problem is to flatten such a nested list.
> This is a solution in Python <=2.5:
>
> def flatten(x):
> res = []
> for el in x:
> if hasattr(el, '__iter__'):
> res.extend(flatten(el))
> else:
> res.append(el)
> return res
>
> print(flatten([1, 2.0, ["abc", [3, ""]]]))
>
> How would you solve a problem like this in Scala?
>
The idiomatic way would be to use a recursive pattern match like this:


scala> val michele = List(1, "hi", List(1, 2, 3), new java.util.Date, List("fee", "fie", "foe", List('f', 'u', 'm')), 99, "Luftballons")
michele: List[Any] = List(1, hi, List(1, 2, 3), Mon Mar 30 23:42:37 MYT 2009, List(fee, fie, foe, List(f, u, m)), 99, Luftballons)

scala> def flatten(list: List[Any]): List[Any] =
| list match {
| case (x: List[Any]) :: xs => flatten(x) ::: flatten(xs)
| case x :: xs => x :: flatten(xs)
| case Nil => Nil
| }
warning: there were unchecked warnings; re-run with -unchecked for details
flatten: (List[Any])List[Any]

scala> flatten(michele)
res0: List[Any] = List(1, hi, 1, 2, 3, Mon Mar 30 23:42:37 MYT 2009, fee, fie, foe, f, u, m, 99, Luftballons)


It may look a bit obscure if you aren't familiar with Scala, so I'll walk you through the method:

def flatten(list: List[Any]): List[Any] =

This is the method signature basically. def means method declaration as in Python. flatten is the name of the method, as in Python. In parens is the argument, which I called list, and in Scala you have to give its type, so I put List[Any]. The type goes after the variable name, separated by a colon. Then because this is a recursive method you're required to specify a result type, so that's the second List[Any]. That's the result type of the method. Then there's an equals sign, which starts the body of the function. The rest is the body of the function. Since it is one expression, I don't need curly braces for the body.

list match {

This means do a pattern match on the incoming list. It is like a switch statement on steroids. There's three cases it will look at, and the first one to match will win. The pattern is between case and the => (the "rocket"). That's the thing that will be matched against. The expression to the right of the rocket symbol is the result value if that case matches. So here's the first one:

case (x: List[Any]) :: xs => flatten(x) ::: flatten(xs)

The pattern here is (x: List[Any]) :: xs. This matches any list whose first element is also a list. When this matches, the first element (the nested list) is assigned to the variable x, and the remainder of the list is assigned to the variable name xs. Then the result is the right hand side of the rocket: flatten(x) ::: flatten(xs). This means call flatten recursively, passing in the first element (the nested list), then concatenate that result with the result of calling flatten on the rest of the list. (The ::: operator means list concatenation.) Here's the next case:

case x :: xs => x :: flatten(xs)

This one just matches any list that has at least one element. If this case matches, the first element will be assigned to the variable x and the rest of the list will be assigned to xs. Now one thing we know about x here is that it isn't a list, because if it were a list, the previous case would have matched. So the result, the expression to the right of the rocket, in this case is the element x prepended to the result of calling flatten on the rest of the list. (The :: operator prepends an element to the beginning of a list.) Here's the last case:

case Nil => Nil

This one matches the empty list. In this case, the result is just the empty list. This is the one that finishes the recursion. As the unrolling recursion starts popping frames, the result list gets built from back to front.

James Iry

Posts: 85
Nickname: jiry
Registered: Nov, 2007

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 30, 2009 1:04 PM
Reply to this message Reply
> you can add a method to a class and then call it. You can't do this in a static language

There is actually no barrier to a static form of open class in a static language. A hypothetical static language could allow

// in one file
class Foo {
def bar = ...
}


// in another file
class Foo {
def baz = ...
}

Then the compiler, linker, or runtime bytecode generator could synthesize a single class from it, generating errors when there were ambiguities or collisions.

class Foo {
def bar = ...
def baz = ...
}

This is of course, still different from the dynamic open classes found in e.g. Ruby. I.e. you wouldn't be able to add methods or not depending on user input.

Jeff Ratcliff

Posts: 242
Nickname: jr1
Registered: Feb, 2006

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 30, 2009 1:33 PM
Reply to this message Reply
> This is of course, still different from the dynamic open
> classes found in e.g. Ruby. I.e. you wouldn't be able to
> add methods or not depending on user input.

I don't know much about dynamic languages and although I find the idea of adding methods at runtime pretty cool, I wonder if that capability really provides a significant advantage over just defining the methods ahead of time and calling the one you want.

Enlighten me.

James Iry

Posts: 85
Nickname: jiry
Registered: Nov, 2007

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 30, 2009 1:36 PM
Reply to this message Reply
A slightly cleaner solution using the flatMap higher order function.


scala> val michele = List(1, "hi", List(1, 2, 3), new java.util.Date, List("fee", "fie", "foe", List('f', 'u', 'm')), 99, "Luftballons")
michele: List[Any] = List(1, hi, List(1, 2, 3), Mon Mar 30 10:20:04 PDT 2009, List(fee, fie, foe, List(f, u, m)), 99, Luftballons)

scala> def flatten(xs : List[Any]) : List[Any] = xs flatMap {
| case Nil => Nil
| case y :: ys => flatten(y :: ys)
| case x => List(x)
| }
flatten: (List[Any])List[Any]

scala> flatten(michele)
res0: List[Any] = List(1, hi, 1, 2, 3, Mon Mar 30 10:20:04 PDT 2009, fee, fie, foe, f, u, m, 99, Luftballons)



James Iry

Posts: 85
Nickname: jiry
Registered: Nov, 2007

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 30, 2009 1:42 PM
Reply to this message Reply
> I don't know much about dynamic languages and although I
> find the idea of adding methods at runtime pretty cool, I
> wonder if that capability really provides a significant
> advantage over just defining the methods ahead of time and
> calling the one you want.
>
> Enlighten me.

What if you are using a 3rd party class hierarchy to which you don't have source?

Note that Scala's runtime type based pattern matching gives you much of the semantic power of open classes while implicit conversions allow much of the syntactic power.

James Iry

Posts: 85
Nickname: jiry
Registered: Nov, 2007

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 30, 2009 2:25 PM
Reply to this message Reply
I wish I could edit. I realized my previous solution had a redundant case checking for Nil. Flatten gets by just fine without it.


scala> def flatten(xs : List[Any]) : List[Any] = xs flatMap {
| case y :: ys => flatten(y :: ys)
| case x => List(x)
| }
flatten: (List[Any])List[Any]

scala> flatten(michele)
res0: List[Any] = List(1, hi, 1, 2, 3, Mon Mar 30 11:24:36 PDT 2009, fee, fie, foe, f, u, m, 99, Luftballons)

Fred Garvin

Posts: 52
Nickname: fredgarvin
Registered: Jan, 2008

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 30, 2009 4:14 PM
Reply to this message Reply
>> What if you are using a 3rd party class hierarchy to
>> which you don't have source?

If you know the method's declaration, which is the typical case, you can still add the method statically. C# has had this feature for some time via "extension methods".

Fred Garvin

Posts: 52
Nickname: fredgarvin
Registered: Jan, 2008

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 30, 2009 4:57 PM
Reply to this message Reply
Outstanding article Bill! I especially appreciated your recognition that type inference is key.

On the face of it, however, the dynamic lang fan will say, "So what? Scala goes to all that trouble to approximate the flexibility I already have with my dynamic language."

And he would have a point. Why should I bother with Scala when it doesn't really offer that much more than Python or Ruby or what have you? But this attitude misses the big picture.

The overwhelming advantage inherent in a static language is the ability to perform static analysis on the code e.g., deterministic code completion, refactors, code navigation, etc. A solid IDE like IntelliJ IDEA is what makes a programmer productive esp. on large projects.

Imo, a developer's productivity is measured by his ability to:

- Discover -- what types, methods, properties, fields, or variables are available from a given use-site?

- Navigate -- can quickly jump to a type, method, property, field, or variable declarations from a use site.

- Understand -- how fast can code on the screen materialize in your mind?

- Express -- how fast can the model in your mind find it's way to the editor without errors?

- Change -- how fast can code be structurally changed, mapped, moved, extracted, folded, etc.

- Test -- given a change in code how fast can I determine whether or not the change broke something.

Dynamic languages simply cannot compete at this level. Hopefully, someday Scala will have tools on par with Java. It has the distinct advantage over competing dynamic languages in that it *can*.

Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 30, 2009 8:56 PM
Reply to this message Reply
> Outstanding article Bill! I especially appreciated your
> recognition that type inference is key.
>
> On the face of it, however, the dynamic lang fan will say,
> "So what? Scala goes to all that trouble to approximate
> the flexibility I already have with my dynamic language."
>
> And he would have a point. Why should I bother with Scala
> when it doesn't really offer that much more than Python or
> Ruby or what have you? But this attitude misses the big
> picture.
>
> The overwhelming advantage inherent in a static
> language is the ability to perform static analysis on the
> code e.g., deterministic code completion, refactors, code
> navigation, etc. A solid IDE like IntelliJ IDEA is what
> makes a programmer productive esp. on large projects.
>
Yes, in the video I talk also about the productivity advantages of static languages. But don't forget there are some other advantages of dynamic languages too that I didn't cover. The three main missing ones to me are:

1. Conceptual simplicity. The language itself is easier to understand when you have a model of objects send each other messages, and at runtime if the message is handled it works, else you get an exception. That's the case in dynamic languages. In static languages, you have more complexity in terms of a type system to understand.

2. Quick feedback loop. Given dynamic languages (at least Ruby, Python, and Groovy, which I was considering as alternatives to Scala) usually run (or can run) by interpreting right from source code, you don't have a compile cycle to wait for. Now most of the time in a static language compiling the bit of code you changed (assuming you're making small changes each iteration) should be pretty darn fast, and with something like JavaRebel you can get an app server to update without restarting it. So most of the time it should be imperceptible, but in practice sometimes it isn't, and you have to wait. Now and then you find yourself doing a full build, and then you really have to wait.

3. Method missing. This technique allows Rails to let you invoke findByNameAndAddress, etc., on database objects even though those methods are never implemented by you by hand. At runtime the method missing handler will get control, parse the name and dynamically provide an implementation. In a static language you can use code generation to get the same benefit, but code gen adds some time to your build cycle and is a bit combersome. (But nevertheless I use code gen quite a bit in Java to get this benefit.) You can also do the dynamic method missing functionality in Scala pretty well, given Scala's syntax flexibility. An example is in Scala's actors library.

Sending a message to an actor is done with plain old duck typing. If the actor is able to handle the message, it gets handled. Else it just sits there in the inbox of the actor, unhandled. But in an actor you can provide a default case that will handle any unrecognized message, and that's in effect method missing. So you can do that kind of thing on a database object if you wanted to emulate Rails finder methods. Usually in a static language, though, I think most people would go in a different, more static direction most of the time.

Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 30, 2009 9:01 PM
Reply to this message Reply
Hi James,

> scala> def flatten(xs : List[Any]) : List[Any] = xs
> flatMap {
> | case y :: ys => flatten(y :: ys)
> | case x => List(x)
> | }
> flatten: (List[Any])List[Any]
>
Thanks for this contribution. Much simpler.

Now, it's not like we're keeping score here, but I count 8 lines of code in the Python version and 4 lines of code in the Scala one!

Fred Garvin

Posts: 52
Nickname: fredgarvin
Registered: Jan, 2008

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 30, 2009 10:47 PM
Reply to this message Reply
>> Yes, in the video I talk also about the productivity
>> advantages of static languages. But don't forget there
>> are some other advantages of dynamic languages too that
>> I didn't cover. The three main missing ones to me are:


>> 1. Conceptual simplicity. The language itself is easier
>> to understand when you have a model of objects send
>> each other messages, and at runtime if the message is
>> handled it works, else you get an exception. That's the
>> case in dynamic languages. In static languages, you
>> have more complexity in terms of a type system to
>> understand.

I'm not sure there is any meat in that statement. If the static language makes intelligent use of type inference, the type system is not in the way. In fact the type system actually *helps*. Consider a method declaration. Without types on the parameters how do you know what's ok to pass without having knowledge of the implementation? A type can be the best form of documentation. And I won't get into the *huge* benefit of code completion, refactoring, etc. etc.

>> 2. Quick feedback loop. Given dynamic languages (at
>> least Ruby, Python, and Groovy, which I was considering
>> as alternatives to Scala) usually run (or can run) by
>> interpreting right from source code, you don't have a
>> compile cycle to wait for. Now most of the time in a
>> static language compiling the bit of code you changed
>> (assuming you're making small changes each iteration)
>> should be pretty darn fast, and with something like
>> JavaRebel you can get an app server to update without
>> restarting it. So most of the time it should be
>> imperceptible, but in practice sometimes it isn't, and
>> you have to wait. Now and then you find yourself doing
>> a full build, and then you really have to wait.

While it's true that an interpreted language has the inherent benefit of being able to provide a dynamic feedback loop, the feature not exclusive to interpreted languages. It's harder for compiled languages, but not impossible esp. if the language runs in a vm. Java's type system simply was not designed so that classes could be reloaded. It could be done though. My point is that dynamic language != fast feedback and static language != slow feedback.

As an aside interpreted languages don't perform well relative to compiled languages like Java. I wonder how many projects start out using, say, Ruby because of the faster turn around when the project is young and simple. And when project matures Ruby becomes a liability, not just in performance but also in complexity esp. for new programmers. Probably more than many are willing to admit.


>> 3. Method missing. This technique allows Rails to let
>> you invoke findByNameAndAddress, etc., on database
>> objects even though those methods are never implemented
>> by you by hand. At runtime the method missing handler
>> will get control, parse the name and dynamically
>> provide an implementation. In a static language you can
>> use code generation to get the same benefit, but code
>> gen adds some time to your build cycle and is a bit
>> combersome. (But nevertheless I use code gen quite a
>> bit in Java to get this benefit.) You can also do the
>> dynamic method missing functionality in Scala pretty
>> well, given Scala's syntax flexibility. An example is
>> in Scala's actors library.

Imo, the best way for a static type system to handle special cases like object/relational mappings is to be extendable. For instance, why should Java define such a rigid type system where Class is a closed, internal implementation detail? Wouldn't it be great if you could implement a Type interface and provide a loader for your EntityType? First class custom types, now that's what I'm talkin' about!

Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 30, 2009 11:11 PM
Reply to this message Reply
Hi Fred,

> >> Yes, in the video I talk also about the productivity
> >> advantages of static languages. But don't forget there
>
> >> are some other advantages of dynamic languages too that
>
> >> I didn't cover. The three main missing ones to me are:
>
>
> >> 1. Conceptual simplicity. The language itself is easier
>
> >> to understand when you have a model of objects send
> >> each other messages, and at runtime if the message is
> >> handled it works, else you get an exception. That's the
>
> >> case in dynamic languages. In static languages, you
> >> have more complexity in terms of a type system to
> >> understand.
>
> I'm not sure there is any meat in that statement. If the
> static language makes intelligent use of type inference,
> the type system is not in the way. In fact the type system
> actually *helps*. Consider a method declaration. Without
> types on the parameters how do you know what's ok to pass
> without having knowledge of the implementation? A type can
> be the best form of documentation. And I won't get into
> the *huge* benefit of code completion, refactoring, etc.
> etc.
>
While I agree with the specific examples you give, it doesn't change that the conceptual simplicity of learning the language itself is greater when you have the Smalltalk/Ruby/Python/Groovy etc. model of method invocations are messages sent between objects. Type inference doesn't eliminate the need for programmers learning Scala to learn about Scala's type system. I wasn't saying understanding dynamically typed *programs* are easier to understand in this point, just the languages themselves. So one way this comes into play is that I think a dynamic language can be a better fit for an accidental programmer, someone who isn't a programmer by profession but wants to do some programming on the side. My brother, for a specific example, is a smart guy. Has a PhD and works as a professor, but he's not a programmer by trade. He's a statistician/scientist in the field of international public health. When he asked me what would be a good language to learn to do some stuff on the side, I sent him a Python book. (This was of course many years ago, before there was a Scala book to send. Today I'd probably still send him a Python book, but might also send him a Scala book if for no other reason than it has his last name on the cover.)

Michele Simionato

Posts: 222
Nickname: micheles
Registered: Jun, 2008

Re: Getting Dynamic Productivity in a Static Language Posted: Mar 31, 2009 12:19 AM
Reply to this message Reply
I see. So basically you are saying that Scala has both static typing AND dynamic typing (the Any type there). Given that, it is not surprising that you can easily implement the flatten function.
But I see this as a concession to the dynamic
typing camp i.e. realizing that for certain classes of
problems where types are unknown in advance you must use
dynamic typing.
Personally I am not in the dynamic camp nor
in the static camp; I have much more experience with
dynamic languages, nevertheless I have always liked
static typing, at least from the time of Pascal. I also
have some experience with SML for what concerns type
inference. I like the idea. For me it is more of a matter
of practicality. If I can find a readable language with
small learning curve, clean design and decent libraries
I have nothing against it, no matter if it is statically
or dynamically typed.
Having said that, we do not use Java at work,
nor we have to interact with the Java platform,
so I will likely stay ignorant about Scala
for a while and just read one article or two of
yours ;)

Flat View: This topic has 78 replies on 6 pages [ 1  2  3  4  5  6 | » ]
Topic: AntiSocial Networking Previous Topic   Next Topic Topic: Mixins considered harmful/3


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us