The Artima Developer Community
Sponsored Link

Weblogs Forum
JavaOne 2005, Day 2: Hitting the Jackpot!

2 replies on 1 page. Most recent reply: Jun 29, 2005 2:09 PM by Eric Armstrong

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 2 replies on 1 page
Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

JavaOne 2005, Day 2: Hitting the Jackpot! (View in Weblogs)
Posted: Jun 29, 2005 2:09 PM
Reply to this message Reply
Summary
You go to JavaOne in hopes of hearing about realy cool things. Every once in a while, you get something extremely cool--elegant, even--something like Project Jackpot. If you need to clean up your source files en masse, do it elegantly, and do it quickly, then Jackpot is the tool you've been waiting for.
Advertisement

Early in my career, I saw my mentor (a certain David Smith), become outraged when keypunch operators were spending hour after hour making changes to files that could just as easily have been programmed. "Sure," he said to the project manager, "You don't mind doing the job with manual labor, because you're not the one who's doing the work." His reaction reflected a deep-seated appreciation for the value of human effort that turned out to be contagious.

It has been many years since I first heard that, but it's a lesson that always stuck with me: Never do large amounts of mindless, repetitive work if you can possibly get the computer to do it for you.

When you think about it, it only stands to reason:

  • The computer will work all day and all night, never taking a break, without complaining.
  • The computer will do what you told it to do, with no mistakes, at high speed.
  • Once you've taught the computer how do to the task, it doesn't forgets. You can ask it to do the same thing next year, and every year thereafter.

When it comes to making a large number of changes to a massive collection of source files--or even just examining them to look for patterns--Jackpot was made to order. Better yet, Jackpot puts an expert at your elbow, so you can get their advice on changes you should make, but may never have thought of on your own.

Introducing JackPot

Jackpot is a tool for source code examination and transformation that lets you automate the process of inspecting, reporting on, and modifying source files. It was developed by James Gosling and Tom Ball, with the help of Neal Gafter, who made the javac compiler reentrant so Jackpot could use it.

To define the changes you want to make, you create a pattern-based series of production rules and save them in a file. (I think of that file as a filter.) The rules have the form: source-pattern => result-pattern, where => is the production operator. When Jackpot runs, code that matches the source pattern on the left is transformed into the result pattern on the right.

Note: You can also write your own transformation class, instead of a series of rules. It requires knowledge of the Abstract Syntax Tree produced by javac, and an understanding of the Visitor pattern--the most useful pattern there is for operating on tree structures.

Jackpot is based on javac, so anything the compiler can process, Jackpot can process. And in addition to using javac to read the source file coming in, Jackpot uses javac to verify that resulting source code, after all changes have been applied, will still compile. That helps rules-authors to avoid mistakes.

Why You Want It

There is tremendous power in the concept of a rule set. In essence, it captures the expertise of the author. When you run that rule set, you have the expert at your elbow, suggesting changes. The transformations might produce cleaner code, better performing code, more localizable code, or adhere to any other principles that might guide an expert when modifying a class.

In effect, Jackpot puts a variety of experts at your beck and call, so you can ask them:

"If you wanted to make this code (cleaner, faster, localizable, take your pick), how would you modify it?"

In addition to making changes, Jackpot can be useful for examining code, as well. In all, Tom Ball noted several general categories of use for Jackpot:

  • Source auditing: Examining code for adherence to conventions (like lint).
  • Source archeology: Finding patterns in existing code and figuring out how it works.
  • Refactoring: Making the code better.
  • Reengineering: Converting to new APIs or new technologies.
  • Pretty-printing: Making code more readable and maintainable.

Some of the rule sets that Tom Ball demonstrated were designed to:

  • Clean up code (40-some rules that eliminate extraneous code)
  • Make code that compiles faster
  • Convert code that uses StreamTokenizer to use the new Scanner class
  • Simplify loops so they run faster

Some other possible applications for transformation rules include:

  • Making code localizable
  • Optimizing code for performance
  • Removing debug logic before compiling the production version

There is a lot to know in many of those areas. It's difficult to acquire the knowledge necessary to make code that is maintainable and localizable and high-performance and well-documented and makes use of the latest APIs--and oh, by the way, it has to run properly, too. Jackpot, and the library of transformation rules that will surely grow up around it, will make it possible to get the expert advice you need to achieve all of those goals.

Writing Production Rules

When you write a production rules, you include meta-variables that define place holders in the current syntax tree. At the moment, meta-variables are defined as syymbols that have a trailing underscore (although that may change if a better suggestion comes in).

For example, in the expression, if (v == true), v is a literal that only matches the symbol "v" in the source code. But in the expression, if (v_ == true), v_ is a meta-variable that matches any expression in that position in the syntax tree.

Having defined v_, you can now use it the right-hand side of the production. For example: if (v_ == true) => if (v_)

That rule says to change an expression like if (succeeded==true) to if (succeeded) --an obvious simplification.

You can also qualify an expression with a "where" clause like you used to see in mathematics texts. In this case, you introduce the where clause with a double-colon (::) For example, you could rewrite the previous expression like this: if (v_ == true) => if (v_) :: v_ instanceof Boolean

(You can also check to see if v_ extends some class, or is a superclass of it, among other things.)

Fast Operation, with Simple, Safe Undo

Under the Hood, Jackpot stores, examines, and/or operates on three trees provided by javac:

  • Abstract Syntax Tree (AST) (syntax hierarchy--code nesting)
  • Type System (class hierarchy--inheritance)
  • Symbol Table (instantiation hierarchy--ownership)

Using those three hierarchies allows precise operation. That operation occurs at high speed, for a variety of reasons:

First, the javac compiler isn't generating byte codes and it's not doing optimizations. It only parses the class files and produces an AST (abstract syntax tree). So that part of the process takes seconds, even for a large number of files.

Second, Jackpot is optimized for the AST. Patterns are found as early in the tree in possible, before the exponential explosion drains performance.

Third, although Jackpot creates a new AST for the result, it shares copies of existing nodes. So a minimum number of changes made, and most of those involve swapping a few pointers around.

Note: Node-sharing posible, because no node knows what its parent is.

When Jackpot modifies a node, that node and its parents are copied to to the new tree. When copying parents, Jackpot follows a linear chain up to the root, without the exponential explosion that occurs when you follow the branches down towards the leaves, so it operates in essentially linear time.

Undo is accomplished by removing the node, along with its parents--again, a linear operation that is both fast and safe, because the original nodes remain untouched.

In short, Jackpot runs very quickly, in essentialy linear time (Order N, where N is the product of the number of rules to process and the amount of code you're operating on).

Format Control

You can use a rule set to reformat an entire file, although in many cases that's undesirable. You can also use them to format the results of source code modifications.

Reformatting an entire file is useful when you're the sole author, especially when you've inherited a project. But for a large project with multiple authors, where the files are under source control, massive reformatting generally isn't a good idea--unless everyone knows it's coming and all files are checked in first. Otherwise, format changes will swamp the real differences, making them virtually impossible to locate.

When making a small change to a source file, therefore, it's generally not a good idea for Jackpot to reformat the entire file. On the other hand, it's impractical for Jackpot to deduce the style that happens to exist in that file. So when Jackpot makes changes, it uses your specified style settings (of which there are many) to format the new code.

As a result, the new code may or may not adhere to the style exhibited in that file. But it will definitely conform to your coding standard--and you can always run a Jackpot pretty-print filter on the whole file, if you have that luxury.

What Jackpot Doesn't Do

Jackpot doesn't parse code comments the way that javadoc does, for example, so it won't help very much in that area. For that, you still need to use a tool like DocCheck and make changes manually. (DocCheck is available at http://java.sun.com/j2se/javadoc/doccheck/)

Note: As the author of DocCheck, I can tell you that it does a thorough job. Another tool I wrote that may be published by Sun's java tools group in the near future is CommentMerge--a heavily-tested program that merges API comments into existing source code. At some point, I would dearly love to see the javadoc syntax tree grafted onto the AST, and use Jackpot rules to implement a merge of DocCheck and CommentMerge for comment transformations!

The only other thing that Jackpot doesn't do, besides slicing your bread, is checking out files from the source control system. If you're making changes to a large number of files, that's obviously an issue. The solutions are to:

  1. Use Jackpot as part of an IDE that will check out the files automatically.
  2. Use Jackpot to generate a list of files that need to be changed, and funnel that list to a script that checks them out. (That would be an excellent job for Groovy, in fact. See JavaOne, Day 1: It's a Groovy Day!, at http://www.artima.com/weblogs/viewpost.jsp?thread=116723)

Availability

Although the Jackpot engine is working, it isn't currently available because the NetBeans integration isn't totally working, at the moment. (The IDE will provide a GUI, so you can examine suggested changes and choose whether or not to apply them.)

To find out when Jackpot is released as a NetBeans module, register at http://www.netbeans.org/.

On the other hand, Tom Ball also left open the possiblity of creating a command-line version that could be built into an ANT task and eventually integrated into other IDEs. The best way to check progress on that front is most likely to monitor his blog at http://weblogs.java.net/blog/tball/.

Resources


Joe Cheng

Posts: 65
Nickname: jcheng
Registered: Oct, 2002

Re: JavaOne Report, Day 2: Hitting the Jackpot! Posted: Jun 29, 2005 3:26 PM
Reply to this message Reply
Interesting stuff. Sounds like there are some similarities to Dylan's macros...?

Isaac Gouy

Posts: 527
Nickname: igouy
Registered: Jul, 2003

Re: JavaOne 2005, Day 2: Hitting the Jackpot! Posted: Jun 30, 2005 1:10 PM
Reply to this message Reply
"Currently only Smalltalk applications have the benefit of the tool. ...
The search and replace functions of the Brant Roberts Rewrite Rules use a traditional technique of algorithms based upon generating a compiler parse tree.
This technique could be applied back to other languages to produce a ‘Rewrite Compiler’ whose output is rewritten source code, not machine level code. The key point is that a parse tree is essential to achieving the correct transformation of complex expressions, and a compiler seems the appropriate place for meta knowledge about the target language and system."

Transformation of an Application Data Layer pdf
http://csc.noctrl.edu/f/opdyke/OOPSLA2002/Papers/TransformDataLayer.pdf



"The Smalltalk Refactoring Browser was originally developed to study refactorings. After building specific transformations, we added a general rewrite engine to make it easier to add future transformations. However, the syntax for this was too complex for normal users. Therefore, we simplified the syntax and added a userinterface so that programmers would have access to the rewrite engine. Since then, several people have used the rewrite engine to transform their programs."

Tools for Making Impossible Changes pdf
http://csserver.evansville.edu:8888/roberts/uploads/1/unanticipatedSoftwareEvolution.pdf

Flat View: This topic has 2 replies on 1 page
Topic: Decompiling Java Previous Topic   Next Topic Topic: Typesafe Enums in C++


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us