This post originated from an RSS feed registered with Ruby Buzz
by Obie Fernandez.
Original Post: Improving the ActiveRecord Experience
Feed Title: Obie On Rails (Has It Been 9 Years Already?)
Feed URL: http://jroller.com/obie/feed/entries/rss
Feed Description: Obie Fernandez talks about life as a technologist, mostly as ramblings about software development and consulting. Nowadays it's pretty much all about Ruby and Ruby on Rails.
One of the subjects I touched on in my last blog entry about my team's experience with Rails was our difficulty with settling on a data model, which delayed our development of features. A couple of days ago, I pulled Jeff Patton and Michael Granger, the other two senior ThoughtWorks developers, into a chatroom to discuss our feelings about working with ActiveRecord. Here is the condensed version, posted with permission. My brief commentary is in italics.
Obie: So guys, I made the following comment in my blog last night, "Don't ever underestimate how long it will take to come up with workable data model. Data modeling is difficult and in retrospect the team should have gotten this ready during the first week." To which Carlos, commented: "As I understand it, you're just as adverse to the idea of a big design up front than I am, so why do this? How difficult were database migration and refactoring during development?" I want to post a followup blog entry on this subject, but I want to include your perspective on the subject. If the chat is good enough maybe I can use some of your comments directly.
Michael: For me, it was mostly trivial. Mostly a matter of tracking method calls down whose names had changed. Because the model is reflected from the DB, there's very little to change. I've been doing Perl and Ruby stuff like that for years in libraries other than ActiveRecord, and AR isn't any more difficult.
Jeff: I still felt "pushed" by rails to get the data model right...
Obie: But with respect to trying to get a team "flow" going to actually pound out some features, I felt like we were stuck a lot until the data model was frozen
Jeff: Rather than letting it emerge. Emergent design started to feel "expensive".
Michael: Well, if we had had the time to do things test-first, we could have distilled the model from the data requirements needed to make tests pass, right?
Obie: So, was the friction and expense mostly a function of our inexperience with Rails?
Jeff: I'll admit that once the object model gels enough for me to tell a basic story about the app, my brain wants to move on towards building it - and letting details shake out during building. Test first might have changed things. More experience with rails - and the fact that the model actually lives in the database - might have helped too.
Obie: Yeah, but in a data-centric app, is the db schema really just a 'detail'?
Michael: No, I think it was the fact that we started while you were gone. I think we would have hashed things out really quickly with you there, and wouldn't have needed to redo it in the second week. Yeah, and some way to look at the model without referring to the schema would be cool. I've been thinking about how to do that. Considering adding some extension to RDoc that'd let you dump a schema as method documentation.
Obie: Okay, but regardless of the reasons for needing to change, there was definitely some friction around making the changes. What if AR had a mode where you weren't bound to the db?
Jeff: That's where my head was going - the not being bound to the db. I'd rather have information about my model objects live in one place: in the model objects not spread out between the schema sql and the model objects.
Michael: So you'd just write model classes, and then the DB would be generated from that? Ala the Hibernate stuff Jeff was talking about?
Obie: That's what Django does, and I think it's pretty good.
Michael: Yeah, and Rails already has the mechanism to do that (migration functions). It'd be pretty easy to add. I was actually writing a library like that before Rails was released.
Jeff: I'd rather not create a single table till I saw the app functioning on the screen with some in-memory database. I'd argue to not create the schema till absolutely necessary.
Michael: There's nothing in the AR/migration stuff that would necessitate a real DB backend.
Obie: So there are a couple of factors .. 1) we would love to be able to see model attributes in the code and 2) it would be nice to work without the DB until it was needed.
Jeff: If you weren't using a relational db backend, how could you start to build up your model objects?
Michael: Well, what I'm talking about is either hooking ::method_added or using the attr_* style Module functions to define the model, then reflect that into whichever AR connection you're using.
Obie: Just generate model in same way and add attr :property statements at the top along with the relationships
Michael: Could be a simple text file (PStore or the like), or BerkeleyDB, or whatever.
Obie: or it could be serialized object graphs. I think Prevayler is like that in the Java world.
Michael: Yeah, that's what PStore is and there's a Prevayler clone called Madeleine or something, too. I like having the model expressed in the classes. Makes for less action at a distance.
Jeff: You guys have heard me rant - my approach has been with Java and Prevayler in the past.
Obie: So I wonder what it would take to hack ActiveRecord into what is described above, cause plying a different persistence API into the mix is NOT the solution IMHO. I'd like to run script/generate model Foo and not yet have a dependence on a foos table, yet be able to go ahead and add relationships and properties to foo.rb. Then at some point later on when things calm down, transition to db-based persistence.
Jeff: That feel like a best approach to me. Thrashing over the relationships and names of things in the model is to be expected, we just need to decrease the "cost" of that sort of thrashing.
Obie: It certainly seems better than how it is now in the context of a team with more than 1 programmer.
Michael: Since the migration stuff is expressed in terms of the AR::Connection class anyway, it wouldn't matter whether things happened in MySQL or PStore or <foo>. The attrs at the top could use the migration method calls. Hook ActiveRecord::Base.inherited and have it call create_table(), then db_attr :foo, ... could call add_column, etc. Or you could just defer all of that until some trigger point at which all that stuff is run, and then just build the execution at interpretation time.
Obie: Yep, that's making a lot of sense to me.
Michael: The trick is figuring out when that time is, and I wouldn't want to override the base attr* methods, 'cause someone might want a normal attr.
Obie: We have the concepts of development, test and production modes in Rails already. Production would never try to set things up itself and dev might always use SQLlight or some in-memory solution.
Michael: Yeah, but you still wouldn't want to be dropping tables and re-creating them with every request and with the current development execution model, that's what would happen. All classes are reloaded every time, so those hooks would be getting called every time, etc... So you'd need some kind of trigger or something that'd cause the "schema" to be re-uploaded. Otherwise any data you insert would be lost after every request.
Jeff: I'm not completely with you guys, but couldn't that be a manual [rake] step?
Michael: Yeah, that's what I'm saying. You couldn't do it purely within the class-loading system. You'd have to have some trigger that says "okay, now (re)create the datastore behind the model". So that's an argument for the hooks just building some internal concept of the data model and then having some other methods do the actual creating. But that's a pretty fundamental change to AR. It's moving method creation from reflection on the tables into the classes themselves. I think it's a good change, but still a big one.
Obie: Even if those methods were called everytime, if it had a way to detect changes it could just ignore them otherwise.
Jeff: How expensive would that sort of call be every time?
Michael: Yeah. I guess you could write something that diffed the backend every time, probably not much harder than defining methods every time.
Obie: I'm sort of in a haze about how it would all work during one stateless process, but then again right now it does a table describe with every model access anyway.
Michael: It could use one of the diff libraries to compare Arrays of table/column definitions. Then it'd just do the re-definition in 'dev' mode, and error in 'prod' mode or something...
Obie: Sounds right, and sounds more agile too. Estimate? ;-)
Michael: Hmmm... 4 days, maybe? Haven't looked much at the grotty bits of AR and it would have to keep the current way of doing things as the default so as not to hork people using that way.
Jeff: One thing I'll bring up: don't know if it's because I've spent most of the last 10 years in an Oracle shop - but coupling the model objects too tightly to the db schema still makes me twitchy. I realize we might be talking about a mechanism that supports building/prototyping as we were last week.
Obie: Jeff, I think it actually makes a lot of people twitchy. It's one of the biggest issues people have with Rails.
Jeff: It does seem like rails need more than one persistance strategy... what we've talked about sounds like a good next one.
Michael: Yeah, agreed.
Obie: Should I hold off publicizing this conversation until you put something together or go ahead and summarize it and blog?
Michael: Nah, blog away. If even just to gauge people's reactions and get early feedback.
Jeff: I'd vote summarize and blog. One thing we need to not lose sight of is the problem this was a solution to: After this work is done, it otta be easy to change the model quickly.
Obie: And the other problem this solves is being able to see your properties on the model class itself instead of having to flip over to the DDL or whatever graphical mysql frontend you're using.
Michael: Yeah, which is a big one, IMHO.
Jeff: ditto
Obie: i think that's enough of a pain point that people would use this over normal AR practice
Michael: And you can make RDoc output API docs for 'em.
Obie: okay, let's make it happen
In discussion on #rubyonrails later that day, it was pointed out to me that there are at least three other efforts underway to address this issue, most notably ActiveSchema.
Previous Topic
