Registered: Mar, 2002
Re: Traps & Pitfalls of Agile Software Development - A Non-Contrarian View
Posted: Feb 19, 2009 8:59 AM
I agree with some of the points, whereas some of the comments are why I wrote the article in the first place. Let me explain...
> Underdesign underdesign underdesign.
> Agile sounds extremely rational, everything about it:
> iterations, emergent design, test driven etc. I was
> excited to try it. But what I found when I worked on an
> agile project was that the place that claimed to be agile
> was merely a hack shop. It was almost as bad as
> non-iterative methods.
Just because an organization claims to do agile development, it doesn't mean they are. This is the kind of stuff I see too much of. It failed because the team failed, not because Agile failed.
> Since Agile proponents like guidelines I'll give yo mine
> based on my experience:
> (1) Agile as a concept is great; where it fails is that it
> under estimates the demands on fallible nature of
> management and overestimates the autonomy of developers.
I don't buy arguments like, "Agile fails because Management..." or "Agile fails because Developers..."
I personally think Agile is quite robust in the face of human nature. But that's not what I think you are getting at because of the comments that follow.
> (2) As a developer, you'll find when pressed for time,
The precondition here is "pressed for time." This is an environmental issue that is being responded to with some kind of behavior. Agile methods have a lot to say about how to best manage and avoid time pressure.
> refactoring is the hardest thing to justify to management.
This is a challenge for any software team. The challenge with agile is that it is hard for a team to hide what it is doing. This is the point of my #1. Agile teams can be allured into excessively tactical behavior and loose site of strategic behaviors like refactoring or design. I think the Agile Principles are pretty clear on how to address this and most agile methodologies speak to it as well. That doesn't mean teams practice the recommendations though.
> It's not that you aren't an "adult". It's that you want
> t to stay employed. It's not that you aren't competent,
> its that refactoring time is not built in to the schedule.
Sounds like a fear-motive. Reflect on that as it pertains to you job and your personal happiness ;-)
If refactoring isn't built into the schedule then it isn't going to get done - unless the team has a shadowy corner to hide work in. I'd say "adult" and "competent" do apply well here although they are somewhat loaded terms. Teams should plan their work. Refactoring should be done all the time and so it should be built into story estimates (or whatever your favorite vocabulary) just like bug rework and testing are.
> (3) Agile means to management - just code and the design
> will emerge. Any upfront design is often frowned upon.
"Agile doesn't work because Management..."
There is a huge body of work describing organizational and cultural preconditions where agile teams do best. Most organizations that aren't already succeeding with agile don't look that way. It's not a secret: Agile will force an organization to change. In an Agile-hostile organization, I think there are three options: 1) Try Agile expecting it to cause change and doing your part to help make the change, 2) Do something more consistent with the organization and culture, or 3) go work somewhere that has a less hostile culture and is amenable to change.
> (4) Without zero upfront design and near zero refactoring
> what we end up with is a hack shop.
A hack shop is full of hacks. The trap was identified but it's not in the nature of Agile discourage good software development techniques. It is important to note that Agile is NOT a set of software development techniques. The team is responsible for choosing to do the right thing; "adult" and "competent" come to mind.
> And we are back to the proposition that another commenter
> mentioned, communism sounded great as a concept but in
> practice failed because ignored human nature.
> The rebuttal to that was that employees are somehow
> insulated from human nature, because they are
> "professional" as it were and want the best for the
As I said, I think that agile respects and embraces human nature. Let's be sure do differentiate human nature from destructive human behaviors though.
> I beg to differ. Professionalism has nothing to do with
> it. At the time of the decision not to refactor,
> management may be right. And the developer's
> professionalism has nothing to do with it, because
> management communicates whats best for the company to
This is a problem with a specific organizational and cultural situation that sounds quite Agile-hostile. If management is telling the team what to do, then the team is not being allowed to self-organize and do what it thinks is best. (By team I include the Product Owner). This is another case of "Agile doesn't work because Management..."
> Management is right because the expense of refactoring is
> not justified by greater demand or a greater price for the
> product when at that moment the new feature is so
As I mentioned, one way to deal with paying down technical debt is to build its cost into estimates just like bugs. Is this organization you describe also hostile to allowing the team fix bugs? Does the team even estimate?
Another approach is to do the ROI analysis for the technical debt that accrues and in those terms, demonstrate the business value. There are other approaches. I don't think this example has anything to do with failures of agile development per se.
> Management will choose the new feature every
> time, because that in fact is better for the company. Oh
> there is the hidden expense of growing numbers of bugs;
> but at that point, refactoring is not the issue, the bugs
> are, so we patch the bugs. Developers are forced to go
> with management, that is what it is to be professional.
There is always a tension between tactical and strategic investment. In any scheme, I think it is the responsibility of teams to frame the trade offs so that sound business decisions can be made. This is professional and it is also prudent to the business. I can imagine situations where certain technical debt is forgone for tactical gain. At the same time, it is irresponsible to not frame the cost of deferring such debt. I am talking about debt that shouldn't be built into estimates so refactoring isn't what I mean. An example might be upgrading an open source tool, or rewriting a problematic module. If the decision is always made to not take care of technical debt then I think option 3 is a good choice because the software will eventually fail (and probably the company too).
> Agile isn't as much a methodology as a message.
Hopefully I made that clear.
> Methodologies will vary from group to group. Agile's
> s message assumes too much; it assumes that developers are
> running the show or there is some mythical customer on
> site all the time who will agree that the code shouldn't
> be rickity. That is usually not the case.
Maybe Agile assumes to much of the organization you are working in and option 1 isn't feasible. A software team has to be able to survive within its organization.
The thing I like about Agile is that it is clear what those assumptions are. It is irresponsible to adopt anything that requires change without mapping out the scope of that change. Adopting agile will require organizational change at all levels. That may be too big a mountain for an organization to climb.
Sadly consultants and writers and evangelists often don't point this out and they set teams and organizations up for big failures.
> To get the message right, Agile needs to add back a
> discipline of some amount of upfront design prior to
I think there's a misconception here. Defining agile by how you've seen it done is like defining a religion by observing the behaviors of those who claim to believe. Go to the source and see if you find any indication that Agile discourages design. What it does discourage is a "Design Phase" where a team attempts to do all the design up front (BDUF) and then exit that phase with design finished. But that's not to say you don't design. In fact, nothing in Agile prevents you from doing BDUF if that works for your team. In most projects though, the reality is that you can't anticipate all design decisions up front so some design will evolve.
> At least as much time thinking about the problem as
Do you believe Agile advocates not thinking about problems?
> And by the way, this is true with almost every discipline,
> no matter how good you are at implementing, no matter how
> long you've been doing it. Some amount of thought time
> should be put in before you begin.
I'd put this under software development or engineering principles not methodologies. I see the two as orthogonal to the extent they don't undermine each other. Agile says nothing about which engineering principles a team chooses to use.
> For example, papers turn out better when you outline
> first. Buildings turn out better when you plan them first
> ala blueprints. Interior decoration turns out better when
> you plan it first (watch any interior design show).
> Driving to a destination you've never been to (hell with
> traffic even that you have been to) turns out better when
> you think about how to get there first.
It sounds like you have the answer to your design problem in an Agile team!
> You think before you speak, I hope. You think before you
> write, I also hope. Why not think before you write code?
> While this concept may be implicit in agile, it needs to
> be explicit, because today's agile managers are not
> getting that message.
Like I said, agile isn't an engineering technique. And I wouldn't call any manager agile if they don't understand Agile in the first place.