The Artima Developer Community
Sponsored Link

Weblogs Forum
Software Engineering and the Art of Design

27 replies on 2 pages. Most recent reply: Apr 9, 2009 5:20 PM by Mohammad Malkawi

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 27 replies on 2 pages [ 1 2 | » ]
Jim Waldo

Posts: 34
Nickname: waldo
Registered: Sep, 2002

Software Engineering and the Art of Design (View in Weblogs)
Posted: Jul 17, 2003 2:45 PM
Reply to this message Reply
Summary
Software engineering is a lot less like other kinds of engineering than most of us would like to think. There is an aspect of art to what we do, that is learned not in school but by finding a master and serving an apprenticeship.
Advertisement

It's been over a month since I last posted, a break explained by the intervention of JavaOne and the experience of being Slashdotted on my last installment. But July is here, so it is time to declare independence once again and post some thoughts.

During the last couple of weeks, I've been reading a number of articles by Paul Graham on the connection between programming and art and the similarities in different kinds of things that we call design. I like a lot of what Graham has to say (and agree with some of it as well), and while he sometimes takes a long time to say it (at least for a web publication), it is because his points are complex rather than because his writing is long-winded. I highly recommend the site, both for its look and its content, especially his articles on what it is to develop software.

Software development is an odd sort of activity. While we in the industry talk about being engineers, it is engineering of a rather odd sort. The body of established science that you need to know to become a software engineer is rather small, and there are a number of very good software developers who have never formally studied the field at all. I consider myself at least a competent software designer. But my own formal training was in philosophy (as it turns out is also the case with Paul Graham), and about a third of those with whom I went to graduate school are now designing software. And nearly all of us made the switch with little or no formal training in software engineering or computer science.

While it is less common now than it was when I started in the field (low those many years ago), there are a surprising number of excellent software developers and designers whose background is in subjects such as mathematics, or music, or anthropology, or a number of other disciplines that are not thought of as engineering. While I don't think that it is impossible, I would bet there are very few people who switched from, say, philosophy or music to, say, mechanical or electrical engineering without any additional formal education. This is certainly less true now than it was 20 or so years ago when I started; I doubt that I could find the positions now that I was able to find then that allowed me to make the switch. But the fact that it was possible until fairly recently says something, I think, about the nature of software production.

One possible explanation is that software engineering is a fairly new field. There isn't a large body of knowledge that one needs to master to become a software engineer; you can start doing reasonable (and useful) work while picking up that knowledge. The same may have been true in the past of other forms of engineering, but now those fields have matured to a point where there is a lot of accumulated knowledge that is best learned as part of a formal education. But I think that the real reason is somewhat different than the youth of the subject; I think the real reason is that software engineering isn't like other forms of engineering so much as it is like a craft or an art form, both activities that can be learned by apprenticeships rather than formal education.

The notion of an apprenticeship is rarely talked about by the human resources types (although the newest incarnation of the idea, that of mentoring, is becoming pretty respectable), but it is a pretty standard way of getting your real training in software development. Ask most any good software developer who he or she served an apprenticeship with, and they will start telling stories of the master craftsman from whom they learned their art. Most won't even find the question odd, or ask for an explanation. How else could we have learned what we do?

This is even (or perhaps especially) true of those who have had formal education in software engineering. I've often told university professors that people coming straight out of school are mostly useless for at least six months when they enter my groups, and that it takes them about a year to become really useful. This is true for those with master's degrees and with bachelors (don't even ask me about Ph.D.s). And it shouldn't be surprising...after all, the sort of software development that is done in a university setting is very different than the sort of development (and maintenance) done in an industrial setting. Learning the latter takes more than a course; it takes practice, and the accumulated practice of someone (or someones) who have done it long enough and successfully enough to have an idea of the craft involved.

Of course, another reason that there is no good way to teach a class that will train people in the aspects of design and software development that are learned during an apprenticeship is that there is no single method that is right. There are many ways of doing good software development and design, just as there are many ways of doing those things badly. Some good designers will work long and hard on a specification before any code is written, trying to figure out all of the structure of a system before changing that structure becomes difficult. Others will do lots of prototypes, iterating over the code until the structure of the overall design becomes clear. Others do some combination of these two approaches; I'm sure that there are others that do something entirely different. The designs that result from these different approaches are not better or worse than those that result from some other way of coming at the problem. But they are different.

All of this was studied by Fred Brooks, author of the classic Mythical Man Month and Turing Award winner; the results of his study led to his papers on the design of design. His conclusion, which surprises few good software engineers but dismays many in management, is that good design comes from good designers, not from some process. This may dismay many managers because it means that the engineers they employ are not interchangeable parts that can be insured to give good results by the application of the right process. But it is the truth, and they need to deal with it, as do those who want to take the individual out of design by having a more (supposedly) rigorous set of rules for software engineering.

A corollary of the Brooks conclusion that good design comes from good designers is that good designers receive their training from other good designers. This training happens during the apprenticeship, when attempts at design are reviewed and criticized, when code is read, or during conversations over the lunch table. It generally doesn't happen in a classroom (although the master craftsman may be a teacher, who does the training during reviews, readings, and lunches). And, much like working with a Zen master, it happens at different times and in different ways for different people. Someone who is a good master for one person learning the craft may be just wrong for another.

The point of all this, if there is one, is that those who want to reduce software engineering (by which I mean the designing and producing of high-quality software systems) to a set of rules and procedures that can be followed by anyone to get consistently good results simply don't understand the field. Yes, there are processes which can help in the production of good software (any group that doesn't have a good way of doing code reviews is fooling itself if it claims to be producing quality software). But while lack of such processes can insure the production of bad software, having such processes does not insure the production of good software. For that something extra is needed, which is the craft and aesthetics that result from training with a master in the art.

In this way, what we call software engineering is really more like architecture (real architecture that produces buildings) than it is like other kinds of engineering. There is an element of science (a building has to obey the laws of physics) but there is also a large element of art. Depending on what kind of software you produce, the mix of the two may differ some, but there is always a mix. Much of the science can be taught in schools, but the art can only be learned by practice and can be produced in many ways. Those who are looking for the one true way to do the art don't understand what it is we produce, or wish that what they understand can be replaced by something more "scientific." It can't, so they should just get over it.

And, I might add, it is a good thing that what we do requires both art and science. That is what makes it so much fun...


Chris Dailey

Posts: 56
Nickname: mouse
Registered: Dec, 2002

Re: Software engineering and the art of design Posted: Jul 17, 2003 9:15 PM
Reply to this message Reply
You seem to have noticed a similar trend as me, that is, more people remarking on software development being more of an art than a science. Though I hope to find justification in the other direction.

I find smaller scale programming is a lot like cooking. There is certainly room for art. But there is also a lot of science in cooking, dealing with mixture of liquids and application of heat and chemical reactions.

When you get to larger scale programming, organization of code and isolation and validation of functionality are key. But so far I think there are only rules of thumb to try to achieve organization, and not much in the way of science. (Perhaps the John Lakos book is one of the few sources that I've found that provide some concrete engineering information on one aspect of software development. Robert Martin's Principles and Patterns available on the objectmentor.com site is probably near the border between rules of thumb and concrete engineering principles. I hope there's a body of literature that I'm not aware about for verification/testing and cohesion.)

You seem to blur the distinction between software developer and software engineer. This is with good reason -- there is no standard accepted definition of software engineer, and no standard accepted body of knowledge and competencies to define software engineer. I hope some day the difference can be defined more clearly.

There are efforts to regulate and license software engineers. I believe the ACM considered participating in Texas' effort to license engineers. Also, David Parnas, PhD at McMaster University in Hamilton, Ontario (currently on leave at the university of Limerick in Ireland), has had lots to say on licensing engineers and defining a core body of knowledge.

Tiago Antao

Posts: 26
Nickname: tiago
Registered: Feb, 2003

Re: Software engineering and the art of design Posted: Jul 18, 2003 9:39 AM
Reply to this message Reply
I disagree with the idea that to be a good software developer/engineer you don't need good formal education. Moreover that education should better be exactly in Computer Science.

I think one of the reasons with formal education is not regarded as important is because lots of people with formal education have also a very low level of productivity/quality. The reason for this is obvious: lots of these people choosed CS because only of easy money. They really don't like the craft, better yet, they really hate it, but don't admit it. This blurs the difference between formal/non-formal people.

Also, of course, there are a few good people without a formal CS background.

Please note that, eg, being able to handle a certain computer language very well doesn't make, at least for me, a good soft eng profissional. OTOH somebody that easely adapts to different languages (because (s)he has the conceptual knowledge to see that most languages are really similar) is certainly better prepared.

The problem is that the gross ammount of unprepared/unmottivated people generates an industry of mediocrity:
1. Tools (a la VB) to facilitate life on the bells and whistles level
2. LACK of tools to facilitate like on the conceptual/productivity level (I remember seeing an article on artima saying that lisp type macros would never make Java, as the typical Java Programmer would not understand them)
3. Job interviews based on buzzword compliance and not on the ability to think and adapt
4. Software shops being afraid of using new tools because "our engineers will take a long time to understand that, we are affraid"
5. Software shops developing simple software with a test engineer/software developer ratio of 1/1 !!!

This recession could even be good to weed out people that don't belong to the industry, but in most companies, the people that hire/fire are, lots of times, of the clueless type. But, in some sense, I still hope that the recession will bring some rationality to the industry.

And then they complain that some developers are 10x more productive than others... of course.

Isaac Gouy

Posts: 527
Nickname: igouy
Registered: Jul, 2003

Re: Software engineering and the art of design Posted: Jul 19, 2003 1:36 AM
Reply to this message Reply
Thank you for motivating me to re-read the The Mythical Man Month, and watch the Turing Award lecture.

"The Design of Design" lecture seemed to about Engineering Design. One message was that engineering design is not the orderly process we sometimes like to think.

The comment wasn't about how to achieve good design.
Product process raises the floor from mediocre to average. To move from good design to great design we need great designers.

"The State and Future of Software Engineering" (in the last pages of the 1995 anniversary edition MMM) gives a capsule history of chemical engineering. Which leads to Fred Brooks belief "that software engineering at age 27 may be not hopeless but merely immature, as chemical engineering was in 1945".

Alex Peake

Posts: 13
Nickname: alexpeake
Registered: Jul, 2003

Re: Software engineering and the art of design Posted: Jul 19, 2003 11:42 AM
Reply to this message Reply
I must not only disagree, but indeed lay some of the blame for the miserable state of our profession on thinking like this post.

Perhaps the reason that there are so many "with little or no formal training" doing quite as well as so many with formal training leads to another conclusion. The overall level of training is so low that rank amateurs with an aptitude can do *relatively* well.

I refer to two other greats of the software world - Dr. Alan Kay and Richard Gabriel. At a recent lecture at Stanford University, Dr. Alan Kay chastised the Computer Science schools for having dropped all concept of training students in Computer Science, and merely providing "vocational training" (e.g. learn Java). From Dick Gabriel read http://dreamsongs.com/NewFiles/RoadNotTaken.pdf. Make special note of slide 30 - it shows the pitifully inadequate requirements of even a Masters in Computer Science.

The conclusion is that there is insufficient formal training in the science.

Why have we let ourselves get into this? Well one issue was touched upon - the "employer" is (mostly) clueless in their selection process. They do not know how to select and that leaves the level low.

A recent example is from my own search. I got more than 100 resumes for a programming position. I asked each person to submit a small program solving a simple defined solution. More than half the programmers refused to do this, many writing back telling us how insulted they were at being tested so! The message - nobody tests! This perpetuates the hiring of low competence programmers. Out of the solutions submitted only two worked! (I hired both.) Two out of one hundred tells you what the average level is *very* low!

Real programming is difficult, and I cannot think how many of the greats of our world have pointed this out, yet we mostly hire people who have "learnt a language". Predominantly we hire people who know little of theory, and who know only one way to solve a problem (or none).

There are aspects of software that are indeed like architecture - the development of User Interactions - requiring both art and good science. But the rest is Science, and the sooner we start training people in Computer Science the sooner we will have people fit to begin their apprenticeships.

In any profession there are the great and the very bad (in our world the 10x programmers down to about one third of the people who actually cost us more money than they contribute). Programmers, I challenge you to become great. Lifelong learning of Computer Science is the key. Employers, I challenge you to develop methods to distinguish the great. Universities we all challenge you to develop Computer *Scientists* that have a great foundation that they can build their thinking on.

[Wisdom is choosing the right people to listen to, and then *listening*.]

Tiago Antao

Posts: 26
Nickname: tiago
Registered: Feb, 2003

Re: Software engineering and the art of design Posted: Jul 19, 2003 2:02 PM
Reply to this message Reply
> In any profession there are the great and the very bad (in
> our world the 10x programmers down to about one third of
> the people who actually cost us more money than they
> contribute). Programmers, I challenge you to become great.
> Lifelong learning of Computer Science is the key.
> Employers, I challenge you to develop methods to
> distinguish the great. Universities we all challenge you
> to develop Computer *Scientists* that have a great
> foundation that they can build their thinking on.

I would like to start by saying that I generally agree with the terms of your post which is, IMHO, very insightful (with a few minor cavets that can be read in my previous post).

Not only that but, there is a strong trend towards stupidification, let me give a personal example:

I have decided to do a period of strong self-training (something that I do every now and then on my free time, after work). I see mainly 2 options:

1. Domain Specific (Embedded) Languages (DSELs): This is mainly the design and implementation of languages "a la carte" for a certain problem at hand. It so happens that this languages are better implemented in either a logical (via Prolog) or functional language (I was thinking via Haskell, but not sure). I believe that this can be a signigicant productivity boost (I have done some work in the past with it, I am fairly confident), specially if the DSELs were to be compiled say, to J2EE.
OR spend time in:
2. .Net! Interesting, but from a conceptual/personal productivity point of view I expect to gain nothing/very little.

For my work I am pretty sure that 1. would be the most helpful in making me deliver (if not by applying at least from gaining the insights). But from an employability perspective, to the typical hiring manager what does sound better: "DSELs with Haskell" or "J2EE/.Net drone"?

So, being "employable" seems to be very compatible with becoming stupid. With a few exceptions, this is an industry where insight/understanding is worth nothing compared with "buzzword compliance".

Isaac Gouy

Posts: 527
Nickname: igouy
Registered: Jul, 2003

Re: Software engineering and the art of design Posted: Jul 19, 2003 2:55 PM
Reply to this message Reply
The current practice of software development may be more "like a craft or an art form" than engineering - and that was true of other forms of engineering in the past.

There isn't a large body of knowledge that one needs to master to become a software engineer
There isn't a large body of knowledge for software development - wouldn't that suggest that this is an immature field?


'Is “Software Engineering” an Oxymoron?' Alan Kay
p72 http://glab.cs.uni-magdeburg.de/~croquet/downloads/Croquet0.1.pdf

"Real Software Engineering is still in the future... The real question is whether there exists a practice in between the two — stronger than just piling up messes — that can eventually lead us to real modern engineering processes for software."


"A corollary of the Brooks conclusion ... is that good designers receive their training from other good designers."
Again, it's important to note that Fred Brooks was talking about great designers. My understanding of the Turing lecture was that great designers just are - the important thing is to learn how to spot them, how to nurture them, and then let them loose.


learned by apprenticeships rather than formal education
Both are necessary! Experience matters in every endeavour.

"Subject matter expertise must be obtained. Being a good programmer is not enough to be a good software designer (necessary, not sufficient)."
http://www.csis.ul.ie/Research/ParnasLectures/Design%20of%20Component%20Interfaces%20(3).pdf


Dick Gabriel... Make special note of slide 30
We should also note slide 28 "Craft-oriented, critical writing on work by other writers" ;-)

Alex Peake

Posts: 13
Nickname: alexpeake
Registered: Jul, 2003

Re: Software engineering and the art of design Posted: Jul 19, 2003 5:52 PM
Reply to this message Reply
There isn't a large body of knowledge that one needs to master to become a software engineer
There isn't a large body of knowledge for software development - wouldn't that suggest that this is an immature field?

The immaturity, I think, lies in the fact that there is a body that exists (whatever the size), and it is not learned from. See below (Alan Kay).

'Is “Software Engineering” an Oxymoron?' Alan Kay
p72 http://glab.cs.uni-magdeburg.de/~croquet/downloads/Croquet0.1.pdf

And on reading this, it becomes apparent that there is still little learning/training going on! In fact the net idea stated a few pages into the appendix (p.75) is:
Most of current practice today was invented in the 60s"
So why are so few using the last 30 years of ideas? Knowledge is sadly lacking.

"Real Software Engineering is still in the future...
Again, we are not even using what is known today!

I cannot agree with
great designers just are
Noting:
(learned by apprenticeships rather than formal education)
Both are necessary! Experience matters in every endeavour.

Note: Both and
"Subject matter expertise must be obtained. Being a good programmer is not enough to be a good software designer (necessary, not sufficient)."
My point is the formal training in the Science is weak. It is very difficult to be great without the base theory.

Dick Gabriel... We should also note slide 28 "Craft-oriented, critical writing on work by other writers" ;-)
I agree! But keep the context of the depth of training on the theory before reaching this stage of learning.

Isaac Gouy

Posts: 527
Nickname: igouy
Registered: Jul, 2003

Re: Software engineering and the art of design Posted: Jul 19, 2003 11:43 PM
Reply to this message Reply
Hey Alex, we aren't even disagreeing, merely giving emphasis to different timescales. I chose to respond to what seemed like odd statements of Fred Brooks' ideas.

I cannot agree with "great designers just are"
Maybe my shorthand summary was too short ;-)

It isn't formal training that makes or differentiates great designers it's talent.

(In MMM Fred Brooks gave Mozart as his example of great!He's talking about a very small group of people.)

MikeD

Posts: 15
Nickname: mike
Registered: Apr, 2002

Re: Software Engineering and the Art of Design Posted: Jul 21, 2003 9:52 AM
Reply to this message Reply
I always wanted to land myself in an apprentice-type role with a true Master, from whom I could learn all the ins-and-outs of Software Engineering and System Development. For whatever reason, that never happened. Sure, I've worked with some really smart people and tried to tap them for all the knowledge I could. But I never quite found my self in the apprentice-like role that I sought. I've mostly learned what I have from books and articles.

But is this really so different? When I read Bertrand Meyer's Object-Oriented Software Construction, it feels like I'm really learning from Bertrand. It's the same with other great author's works, like the GoF Patterns. True, it would be better to collaborate with them face-to-face and get feedback but that just hasn't happened for me. For those who have coded day-in and day-out at the feet of a Great Master - I envy you!

David Kempster

Posts: 9
Nickname: dkempster
Registered: Apr, 2003

Re: Software Engineering and the Art of Design Posted: Jul 21, 2003 11:44 AM
Reply to this message Reply
This post from Wado is exactly in line with what we have been saying for quite sometime. My background is also Philosophy; Maybe that has something to do with it ; )

But anyone who is acquainted with philosophy knows that most of the claims of scientific methodology is only some much hype anyway. The problem is that software design is design. And design weather in traditional Engineering or Software Engineering or Architecture is essentially a creative endeavor. Even discoveries of mathematical proofs often come to a mathematician while looking at the pattern of a napkin at Seattle's best over a cup of coffee. It was always said in philosophy of science that you could give a logic of confirmation, a logic of scientific explanation but never a logic of discovery. Why? Because discovery doesn't come form exercising a series of methods. Discovery is a creative process as well.

The problem with using engineering as an analogy with what software designers do is partially because we don't understand what engineers do. Engineers have to have the formal background because the materials that they work with have complicated constraints and relations. The materials we work with have comparatively few constraints.

Our discipline is not made better by the fact that we (i) don't understand science (ii) don't understand engineering (iii) don't understand design and instead use poor metaphors to help us understand where we understand neither the analogue nor the analogy.

The factory metaphor that software development is like making a car is fundamentally flawed. Software development is more like the design of the car. This is not a simple process. The production of the car is done by the compiler. The compiler is the cheep labor management wants.

See http://www.bleading-edge.com/Publications/C++Journal/Cpjour2.htm

I think it is interesting that one attack of this post quotes Alan Kay and Richard Gabriel. This is a funny case of missing the point. Both of the individuals would agree with the post. Richard Gabriel in particular does not think of software engineering as engineering. Oh well.

David Kempster

Posts: 9
Nickname: dkempster
Registered: Apr, 2003

Re: Software Engineering and the Art of Design Posted: Jul 21, 2003 11:45 AM
Reply to this message Reply
Ooops Sorry about the typo Waldo

Isaac Gouy

Posts: 527
Nickname: igouy
Registered: Jul, 2003

Re: Software Engineering and the Art of Design Posted: Jul 21, 2003 2:35 PM
Reply to this message Reply
That we don't understand science, engineering, design - perhaps suggests we are lacking in fundamental formal training?

This is a funny case of missing the point
The Jack Reeves article has some interesting lines:

"We in the software business are close to being engineers, but we need a couple of perceptual changes."

"Software development is still more a craft than an engineering discipline. This is primarily because of a lack of rigor in the critical processes of validating and improving a design."

David Kempster

Posts: 9
Nickname: dkempster
Registered: Apr, 2003

Re: Software Engineering and the Art of Design Posted: Jul 21, 2003 5:10 PM
Reply to this message Reply
Actually my point was that Science and Engineering are no more rigorous on their design process than we are. The only part that is is after something is finished being designed and needs to be done over and over again. We don't do that in software. ALL of the programming work is design and the part that is done over and over is done by the compiler.

I find this funny all the time. I was a PhD student in philosophy and took graduate courses in other areas, Linguistics and Biology for example. The seminars were always filled with "Why can't we be more like physics". When you go to the physics graduate lectures they say "We can't understand any of this, and they would wax philosophical usually way off any possible rigorous way of looking at philosophical problems.

I say again. DESIGN is a craft and an Art in every discipline not just ours. When I say we don't understand engineering and science I mean that we are fooling ourselves when we think Science is a science. Nothing matches what we think of when we think of the rigors of science and engineering.

Alex Peake

Posts: 13
Nickname: alexpeake
Registered: Jul, 2003

Re: Software Engineering and the Art of Design Posted: Jul 21, 2003 5:40 PM
Reply to this message Reply
I think we are all saying similar things, and our terms (definitions) are different. I define mine.

Engineering is applied technology that requires a specific education.
Design is creative problem solving.
Creativity is a process of combining things to form something new and useful.

My point, and the elements from Alan Kay and Dick Gabriel I choose to quote, are about the lack of "specific education" amongst a great part of our industry, notwithstanding the fact that there is a huge pool of knowledge to be had.

If one does not have the underlying knowledge of the theory and practice (gained in school, by private study, under tutelage or however), you are unlikely to come upon a good design. Good design combines existing good designs, so if you do not have good designs to combine, what do you get?

Is there more than one way to solve a problem (create a design)? Of course! Is it possible to say one design is better than another? Mostly. Constraints on the problem are often complex, but it often boils down to:
Correctness - the program does the job
Maintainability - someone can evolve the program as needs change
Productivity - the design/solution was created in a reasonable time

I have interviewed many programmers over the years, and here is only one small example of lack of specific education. Three solutions -- one is O(n^2), one O(n) and the last O(constant amortized). More than half have no idea what I am talking about. How can they, then, come up with good designs if they have not been educated in computer science (other than by dumb luck)?


P. S. I am scientist (theoretical physics), engineer (telecommunications system testing) and "software engineer".

P. P. S. Science is not science? Only a philosopher ;-)

Flat View: This topic has 27 replies on 2 pages [ 1  2 | » ]
Topic: Mixins considered harmful/1 Previous Topic   Next Topic Topic: OSCON 2009 Registration Open


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us