Weblogs Forum - Embedded DSLs and Productivity

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Weblogs Forum
Embedded DSLs and Productivity

43 replies on 3 pages. Most recent reply: Apr 22, 2006 4:13 AM by Andy Dent

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 43 replies on 3 pages [ « | 1 2 3 | » ]

Isaac Gouy

Posts: 527
Nickname: igouy
Registered: Jul, 2003

Re: Embedded DSLs and Productivity

Posted: Apr 11, 2006 2:36 PM

(Apologies for this piecemeal reply.)

Bill Venners For example, if I have a method that takes three Strings in a row, sometimes I might get them out of order. That happens now and then too, and since it isn't checked at compile time, I find out at runtime (or unit test time).
Let's look at that: we seem to be saying that these 3 method parameters don't mean the same thing, and we haven't captured that difference. One response would be to eliminate the mistake. Let's define 3 wrapper types that each hold a String and now we can use the compiler to check we are using the parameters appropriately.

We might characterize the Smalltalk response as let's reduce how often you'll make that mistake by using the method name to label each of the parameters, like this copyFrom:to: (we can find the parameter naming approach in some other languages, Nice.)

I think the more useful way to ask this question is how many errors does static type checking help you find that you don't find via unit testing, or quickly via plain old testing, using a non-static language such as Python, Ruby, or Smalltalk. I find that static type checking catches all kinds of problems, but the question is, is it worth the added programmer pain compared to dynamic alternatives? My feeling about that is that the answer depends on the situation.
Yes, it depends on the context - and that renders the interminable context-free quarrels futile. (Can we say how much an acre of land is worth without knowing where it is?)

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Embedded DSLs and Productivity

Posted: Apr 11, 2006 3:01 PM

> (Apologies for this piecemeal reply.)
>
> Bill Venners For example, if I have a method that takes
> three Strings in a row, sometimes I might get them out of
> order. That happens now and then too, and since it isn't
> checked at compile time, I find out at runtime (or unit
> test time).
> Let's look at that: we seem to be saying that these 3
> method parameters don't mean the same thing, and we
> haven't captured that difference. One response would be to
> eliminate the mistake. Let's define 3 wrapper types
> that each hold a String and now we can use the compiler to
> check we are using the parameters appropriately.
>
Perhaps, though you should do a cost/benefit analysis. Having three wrapper types increases the surface area your API, and increases the code to call the method, which are both costs to get at the benefit of static type checking. And to an extent you have just moved the problem. I could still mix up the strings I pass to the constructors of the wrapper types. Whether or not this is worth it depends. One thing I try to do now is use enums as much as possible, which can solve many of these problems without added API clutter.

> We might characterize the Smalltalk response as let's
> reduce how often you'll make that mistake by using
> the method name to label each of the parameters, like this
> copyFrom:to: (we can find the parameter naming approach in
> some other languages, Nice.)
>
I don't understand how that works in Smalltalk or Nice, but I think trying to figure out how to eliminate the class of bug instead of just fixing the bug is a great strategy.

> I think the more useful way to ask this question is how
> many errors does static type checking help you find that
> you don't find via unit testing, or quickly via plain old
> testing, using a non-static language such as Python, Ruby,
> or Smalltalk. I find that static type checking catches all
> kinds of problems, but the question is, is it worth the
> added programmer pain compared to dynamic alternatives? My
> feeling about that is that the answer depends on the
> situation.
> Yes, it depends on the context - and that renders the
> interminable context-free quarrels futile. (Can we say how
> much an acre of land is worth without knowing where it is?)
>
I think it isn't futile if we move the debate away from which is better, to when each is better. I don't have any experience building big systems with dynamic languages, just minor scripts. When I hear people like Dave Thomas, Martin Fowler, David Heinemeier Hansson, and Bob Martin suggesting scripting languages scale much better than static typing enthusiasts believe, I take notice. They are smart guys, and I can't really afford to be slowed down by static typing for applications where it is overkill. I'd really like to hear about real-world experiences building big systems with dynamic languages. I've asked about that before in the forums here, but haven't gotten anything very compelling back.

Isaac Gouy

Posts: 527
Nickname: igouy
Registered: Jul, 2003

Re: Embedded DSLs and Productivity

Posted: Apr 11, 2006 3:36 PM

And to an extent you have just moved the problem.
Yes, I agree, and that will only have value if we can say that we've isolated the problem, if we've moved the problem out of most of the code base into a small part of the code base which we can test exhaustively.

I don't understand how that works in Smalltalk or Nice
The problem you described seemed to be a matter of confusing which parameters were which, when you were dealing with positional parameters. If you have named parameters then there's less chance that you will stick the to value in the slot labelled from.
(I'm not clear what it is you don't understand, say again.)

I think it isn't futile if we move the debate away from which is better, to when each is better.
Which is to say - a more specific context.

I'd really like to hear about real-world experiences building big systems with dynamic languages. I've asked about that before in the forums here, but haven't gotten anything very compelling back.
JPMorgan Kapital http://www.cincom.com/pdf/CS040819-1.pdf
Ericsson AXD 301 switch http://www.erlang.se/publications/Ulf_Wiger.ppt

Isaac Gouy

Posts: 527
Nickname: igouy
Registered: Jul, 2003

Re: Embedded DSLs and Productivity

Posted: Apr 11, 2006 4:23 PM

Ivan Lazarte

Posts: 91
Nickname: ilazarte
Registered: Mar, 2006

Re: Embedded DSLs and Productivity

Posted: Apr 12, 2006 1:59 AM

I'd really like to hear about real-world experiences building big systems with dynamic languages. I've asked about that before in the forums here, but haven't gotten anything very compelling back.

I don't think you'll really know how painful it is until you've managed a large scripting application first hand. One where the needs have grown, when the performance has been exhausted, and the original authors are no longer around.

Ramzi Ben Yahia

Posts: 23
Nickname: miccheck
Registered: Jul, 2002

Re: Embedded DSLs and Productivity

Posted: Apr 12, 2006 6:01 AM

I think that you'll feel the same if you're left with a large Java project poorly written and without a single unit test. Even if you're still able to write code while the compiler behind is saying "yes yes", the system entropy will continue to grow.
RoR is definitely not a small framework, however the entropy has been kept low due to some discipline: good test coverage etc..

IMHO, the purpose of DSL is to be able to write more Data(knowledge,input...) and less Code; ideally make the data runnable
the following is lisp Code that harms.
(a person (age 27))
and this is data that doesn't harm.
'(a person (age 27))
and a is the macro that transforms the data into running code (insertions in database ...)
same thing in Ruby on Rails


validates_uniqueness_of :login, :message => "login already exists"

adds some validation code into a chain of validators.
but writing things like the following in java is insane

//static import of validatesUniqueness
static{
  Map options = new HashMap();
  options.put("message", "login already exists");
  validatesUniqueness("login",options,getValidators()/* a list where to add validator */);
}

better put it in xml


<validators forClass="com.MyClass">
  <validator method="validatesUniqueness" 
             check="login"
             message="login.alread.exists"/>             
</validators>

+ some java code to parse (this plays the role of a macro)

void parseValidators(List classValidators,Iterator validators){
  while(validators.hasNext()){
    ValidatorParser parser = validatorParserFactory.create(validator.getMethod());
    validators.add(parser.parse(validator));
  }
}

in a ruby style

//insanity again
static{
  addValidator("<validator method=\"validatesUniqueness\" 
             + "check=\"login\""
             + "message=\"login.alread.exists\"/>"             );
}
//better
static{
addValidator("validatesUniqueness:check=login,message=login.alread.exists");
}
//smalltalking
static{
addValidator("validatesUniqueness:login message: login.alread.exists");
}
//again
validates("uniqueness: login message: login.already.exists");

Well right now, I'm sorry wasting your time :)

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Embedded DSLs and Productivity

Posted: Apr 12, 2006 6:43 AM

> Do you use an IDE? Do you ever select from a list of
> methods to call that make sense for the type of the
> variable?

Yes. types provide this. not static type checking.

> What do you mean by making casting implicit?

In Java 1.4:

map.put("test", "example")
 
String s = (String) map.get("test");

What I am suggesting is that in the above, the (String) is completely redundant and rarely helpful. I would like this instead:

map.put("test", "example")
 
String s = map.get("test"); // implicit cast from Object to String

> This is something I've wanted to do too, use Java to build
> the API bricks that I then script against with a less
> verbose language. But I've never been able to let go of
> the refactoring support I get from my IDE when I stick
> with Java everywhere. How have you handled that? In other
> words, if you say script your controllers in Jython, and
> maybe your tests, and you go to do some major refactors of
> the Java part, don't you have to change all that Jython
> stuff by hand?

Well, this is still in it's incubation period.

> The refactoring browser is one of those
> analyzers that can do more useful things because the type
> information is there. In the Jython portion, even if the
> IDE supported refactoring, I don't see how it can do as
> good a job.

That's why it would be the 'trivial' portions of the system. I don't work on web apps (now) so my needs are a little different than yours, I think.

The main things I want to look at using Jython with are:

Test Scripting
Internal Team Utilities
Per Entity Logic (detail) below

We have a lot of different partners that we must communicate with and a handful of internal systems that we interface on the other side. We have messages going in both directions.

For a while, we attempted to force all of our partners to adhere to a single standard. It's not feasible. We have many many partners and little or no leverage with most of them. What happens is that we have many non-reusable customizations. Java isn't really good for this. It's great for our reusable pieces but not for this. The other tools we use are also not really good for this when used in the way the vendor steers you.

Jython is 1. very fast and easy to write 2. Powerful 3. Integrates very well with our Java code 4. Can be stored in a DB and executed dynamically in a JVM without a lot of rigamarole.

> What I've taken to doing instead is create DSLs. Our
> little DSLs are very concise, but the generate the verbose
> Java code that has strong types. So in cases where I can
> justify a DSL, I get what I want. Trouble is I force us to
> write things many times over by hand before creating a DSL
> to make sure we know how to abstract. In the long term, I
> think we'll be quite fast because of our
> DSL-code-generators, but in the short term, I'm pretty
> much going at a typical Java pace.

DSLs are appealing but I have doubts about creating a language although one of my coworkers has experience in this area.

Dick Ford

Posts: 149
Nickname: roybatty
Registered: Sep, 2003

Re: Embedded DSLs and Productivity

Posted: Apr 12, 2006 8:43 AM

compile time error checking-

Am I the only one who codes "error free" thanks to the IDE static type checking facilities? it's the nature of the language that gives the Java IDE their power. That productivity I haven't seen elsewhere. Dumb syntax checking everyone has and is a convenience. Being able to completely invert a large application design in 5 minutes with no errors or bugs *without* even having been the original author, that's power.

Eclipse's (and I would presume other Java IDEs) incremental compilation is very useful to me. You don't really build anymore and all the code completion and refactoring and other parsing related stuff that IDEs do is very nice.

"Yeah, but because I have less LOC,..." just sorta falls flat for me.

For me, less LOC is important to me because I don't want to read a whole lot of code. Writing the code in a modern Java IDE is probably just as fast for me, as writing Ruby in a dumb text editor.

For me, the ideal is something like Boo (boo.codehaus.org) that does a lot of type inferencing with some type declarations.

You can actually get a lot of the flexibility that dynamic types have without the dynamic typing.

Isaac Gouy

Posts: 527
Nickname: igouy
Registered: Jul, 2003

Re: Embedded DSLs and Productivity

Posted: Apr 12, 2006 11:49 AM

> > Do you use an IDE? Do you ever select from a list of
> > methods to call that make sense for the type of the
> > variable?
> Yes. types provide this. not static type checking.

Cough, cough... static types?

> > What do you mean by making casting implicit?
> In Java 1.4:
> map.put("test", "example")
>
> String s = (String) map.get("test");
>
> What I am suggesting is that in the above, the (String) is
> completely redundant and rarely helpful. I would like
> this instead:
>
> map.put("test", "example")
>
> String s = map.get("test"); // implicit cast from Object
> to String

Do you have a specific objection to

HashMap<String,String> map = new HashMap<String,String>();
map.put("test", "example");
String s = map.get("test");

Isaac Gouy

Posts: 527
Nickname: igouy
Registered: Jul, 2003

Re: Embedded DSLs and Productivity

Posted: Apr 12, 2006 11:59 AM

Ivan Lazarte wrote:
I don't think you'll really know how painful it is until you've managed a large scripting application first hand. One where the needs have grown, when the performance has been exhausted, and the original authors are no longer around.
What languages are you talking about?
What was the application intended to do?

Isaac Gouy

Posts: 527
Nickname: igouy
Registered: Jul, 2003

Re: Embedded DSLs and Productivity

Posted: Apr 12, 2006 12:54 PM

Dave Thomas I rarely (if ever) write a code generator that generates Ruby code: there's just no need, as Ruby is dynamic enough to let me do what I want without leaving the language.

In Smalltalk a class is created with a method call and the compiler is part of the language runtime. So we can create classes on-the-fly by sending a message to an object; and then use them in the same application. Code can be generated dynamically, and then used.

"the distinguishing feature of interpreted languages is not that they are not compiled, but that any eventual compiler is part of the language runtime and that, therefore, it is possible (and easy) to execute code generated on the fly."
Programming in Lua, p57

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Embedded DSLs and Productivity

Posted: Apr 12, 2006 1:23 PM

> > > Do you use an IDE? Do you ever select from a list of
> > > methods to call that make sense for the type of the
> > > variable?
> > Yes. types provide this. not static type checking.
>
> Cough, cough... static types?

Yes. Static types do not imply static type checking. Are you confused about this or just being pedantic?

> Do you have a specific objection to
> HashMap<String,String> map = new
> HashMap<String,String>();
> map.put("test", "example");
> String s = map.get("test");

No. You can have implied casts and generic collections.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Embedded DSLs and Productivity

Posted: Apr 12, 2006 1:25 PM

> > > Do you use an IDE? Do you ever select from a list of
> > > methods to call that make sense for the type of the
> > > variable?
> > Yes. types provide this. not static type checking.
>
> Cough, cough... static types?
>
James pointed out correctly that I was saying "static type checking" when I was really talking about static typing, where what I mean by static typing is that every variable and expression in a program has a type that is known at compile time. It is "static" to me because those things can't change at runtime.

Python has types, but it doesn't do static typing in this way. Ruby's the same. It has types that you can point to in the code, but at runtime, these types can really morph and change very dynamically. That's made possible because you invoke methods by looking for a signature that matches (since your variable holding the reference to the object on which you're invoking the method doesn't itself have a type). And that means a method invocation can still work even if you added the matching method dynamically at runtime. That method didn't even need to exist explicitly in the source code.

To me the most fundamental tradeoff between these two approaches is that the static approach allows you to do all kinds of analysis, including but not limited to type checking at compile time, better IDE refactoring, better optimization at runtime. The dynamic approach gives you a lot of flexibility to do slippery things like dynamic meta-programming, which helps programmer productivity. You can't do that slippery stuff as much in a static language, because once the types have passed the compiler, you can't change them anymore. You can change them in source as much as you want, but after that, they are static--they stay the same.

Isaac Gouy

Posts: 527
Nickname: igouy
Registered: Jul, 2003

Re: Embedded DSLs and Productivity

Posted: Apr 12, 2006 2:06 PM

> > Cough, cough... static types?
> Yes. Static types do not imply static type checking. Are
> you confused about this or just being pedantic?

For the question "Do you ever select from a list of methods to call that make sense for the type of the variable?" it seems quite important that we're talking about static types rather than runtime types.

> > Do you have a specific objection to
> > HashMap<String,String> map = new
> > HashMap<String,String>();
> > map.put("test", "example");
> > String s = map.get("test");
>
> No. You can have implied casts and generic collections.

With generic collections there's no need for (String) so why do you want implied casts?

Vesa Karvonen

Posts: 116
Nickname: vkarvone
Registered: Jun, 2004

Re: Embedded DSLs and Productivity

Posted: Apr 12, 2006 2:46 PM

I have not had the experience of creating an embedded DSL, but I'm curious to what extent the ability to raise the level of abstraction while still working in the general purpose programming language is the source of productivity in languages such as Ruby and Smalltalk, as Keith Braithwaite suggests in his weblog.

Ok. This isn't specifically what you asked for, but if you really want to find examples of embedded DSLs that are both beautiful and useful then you ought to take a look at functional languages and combinator libraries (try googling for "combinator library").

Flat View: This topic has 43 replies on 3 pages [ « | 1 2 3 | » ]

Previous Topic

Next Topic