Software engineering is a lot less like other kinds of engineering than most of us would like to think. There is an aspect of art to what we do, that is learned not in school but by finding a master and serving an apprenticeship.
It's been over a month since I last posted, a break explained by the
intervention of JavaOne and the experience of being Slashdotted on my
last installment. But July is here, so it is time to declare independence
once again and post some thoughts.
During the last couple of weeks, I've been reading a number of articles
by Paul Graham on the connection
between programming and art and the similarities in different kinds of
things that we call design. I like a lot of what Graham has to say
(and agree with some of it as well), and while he sometimes takes a long
time to say it (at least for a web publication), it is because his points
are complex rather than because his writing is long-winded. I highly
recommend the site, both for its look and its content, especially his
articles on what it is to develop software.
Software development is an odd sort of activity. While we in the industry
talk about being engineers, it is engineering of a rather odd sort. The body
of established science that you need to know to become a software engineer
is rather small, and there are a number of very good software developers who
have never formally studied the field at all. I consider myself at least a
competent software designer. But my own formal training was in philosophy
(as it turns out is also the case with Paul Graham), and about a third of
those with whom I went to graduate school are now designing software. And
nearly all of us made the switch with little or no formal training in
software engineering or computer science.
While it is less common now than it was when I started in the field (low
those many years ago), there are a surprising number of excellent software
developers and designers whose background is in subjects such as
mathematics, or music, or anthropology, or a number of other disciplines
that are not thought of as engineering. While I don't think that it is
impossible, I would bet there are very few people who switched from, say,
philosophy or music to, say, mechanical or electrical engineering
without any additional formal education. This is certainly less true
now than it was 20 or so years ago when I started; I doubt that I could
find the positions now that I was able to find then that allowed me to make
the switch. But the fact that it was possible until fairly recently says
something, I think, about the nature of software production.
One possible explanation is that software engineering is a fairly new
field. There isn't a large body of knowledge that one needs to master to
become a software engineer; you can start doing reasonable (and useful) work
while picking up that knowledge. The same may have been true in the past of
other forms of engineering, but now those fields have matured to a point
where there is a lot of accumulated knowledge that is best learned as part
of a formal education. But I think that the real reason is somewhat
different than the youth of the subject; I think the real reason is that
software engineering isn't like other forms of engineering so much as it is
like a craft or an art form, both activities that can be learned by
apprenticeships rather than formal education.
The notion of an apprenticeship is rarely talked about by the human
resources types (although the newest incarnation of the idea, that of
mentoring, is becoming pretty respectable), but it is a pretty standard way
of getting your real training in software development. Ask most any good
software developer who he or she served an apprenticeship with, and they
will start telling stories of the master craftsman from whom they learned
their art. Most won't even find the question odd, or ask for an
explanation. How else could we have learned what we do?
This is even (or perhaps especially) true of those who have had formal
education in software engineering. I've often told university professors
that people coming straight out of school are mostly useless for at least
six months when they enter my groups, and that it takes them about a year
to become really useful. This is true for those with master's degrees and
with bachelors (don't even ask me about Ph.D.s). And it shouldn't be
surprising...after all, the sort of software development that is done in a
university setting is very different than the sort of development (and
maintenance) done in an industrial setting. Learning the latter takes more
than a course; it takes practice, and the accumulated practice of someone
(or someones) who have done it long enough and successfully enough to have
an idea of the craft involved.
Of course, another reason that there is no good way to teach a class that
will train people in the aspects of design and software development that are
learned during an apprenticeship is that there is no single method that is
right. There are many ways of doing good software development and design,
just as there are many ways of doing those things badly. Some good
designers will work long and hard on a specification before any code is
written, trying to figure out all of the structure of a system before
changing that structure becomes difficult. Others will do lots of
prototypes, iterating over the code until the structure of the overall
design becomes clear. Others do some combination of these two approaches;
I'm sure that there are others that do something entirely different. The
designs that result from these different approaches are not better or worse
than those that result from some other way of coming at the problem. But
they are different.
All of this was studied by Fred
Brooks, author of the classic Mythical Man Month and Turing
Award winner; the results of his study led to his papers on the design of
design. His conclusion, which surprises few good software engineers but
dismays many in management, is that good design comes from good designers,
not from some process. This may dismay many managers because it means that
the engineers they employ are not interchangeable parts that can be insured
to give good results by the application of the right process. But it
is the truth, and they need to deal with it, as do those who want to take
the individual out of design by having a more (supposedly) rigorous set of
rules for software engineering.
A corollary of the Brooks conclusion that good design comes from good
designers is that good designers receive their training from other good
designers. This training happens during the apprenticeship, when attempts
at design are reviewed and criticized, when code is read, or during
conversations over the lunch table. It generally doesn't happen in a
classroom (although the master craftsman may be a teacher, who does the
training during reviews, readings, and lunches). And, much like working
with a Zen master, it happens at different times and in different ways for
different people. Someone who is a good master for one person learning the
craft may be just wrong for another.
The point of all this, if there is one, is that those who want to reduce
software engineering (by which I mean the designing and producing of
high-quality software systems) to a set of rules and procedures that can be
followed by anyone to get consistently good results simply don't understand
the field. Yes, there are processes which can help in the production of
good software (any group that doesn't have a good way of doing code reviews
is fooling itself if it claims to be producing quality software). But while
lack of such processes can insure the production of bad software, having
such processes does not insure the production of good software. For that
something extra is needed, which is the craft and aesthetics that result
from training with a master in the art.
In this way, what we call software engineering is really more like
architecture (real architecture that produces buildings) than it is like
other kinds of engineering. There is an element of science (a building has
to obey the laws of physics) but there is also a large element of
art. Depending on what kind of software you produce, the mix of the two may
differ some, but there is always a mix. Much of the science can be taught
in schools, but the art can only be learned by practice and can be produced
in many ways. Those who are looking for the one true way to do the art
don't understand what it is we produce, or wish that what they understand
can be replaced by something more "scientific." It can't, so they should
just get over it.
And, I might add, it is a good thing that what we do requires both art
and science. That is what makes it so much fun...
You seem to have noticed a similar trend as me, that is, more people remarking on software development being more of an art than a science. Though I hope to find justification in the other direction.
I find smaller scale programming is a lot like cooking. There is certainly room for art. But there is also a lot of science in cooking, dealing with mixture of liquids and application of heat and chemical reactions.
When you get to larger scale programming, organization of code and isolation and validation of functionality are key. But so far I think there are only rules of thumb to try to achieve organization, and not much in the way of science. (Perhaps the John Lakos book is one of the few sources that I've found that provide some concrete engineering information on one aspect of software development. Robert Martin's Principles and Patterns available on the objectmentor.com site is probably near the border between rules of thumb and concrete engineering principles. I hope there's a body of literature that I'm not aware about for verification/testing and cohesion.)
You seem to blur the distinction between software developer and software engineer. This is with good reason -- there is no standard accepted definition of software engineer, and no standard accepted body of knowledge and competencies to define software engineer. I hope some day the difference can be defined more clearly.
There are efforts to regulate and license software engineers. I believe the ACM considered participating in Texas' effort to license engineers. Also, David Parnas, PhD at McMaster University in Hamilton, Ontario (currently on leave at the university of Limerick in Ireland), has had lots to say on licensing engineers and defining a core body of knowledge.
I disagree with the idea that to be a good software developer/engineer you don't need good formal education. Moreover that education should better be exactly in Computer Science.
I think one of the reasons with formal education is not regarded as important is because lots of people with formal education have also a very low level of productivity/quality. The reason for this is obvious: lots of these people choosed CS because only of easy money. They really don't like the craft, better yet, they really hate it, but don't admit it. This blurs the difference between formal/non-formal people.
Also, of course, there are a few good people without a formal CS background.
Please note that, eg, being able to handle a certain computer language very well doesn't make, at least for me, a good soft eng profissional. OTOH somebody that easely adapts to different languages (because (s)he has the conceptual knowledge to see that most languages are really similar) is certainly better prepared.
The problem is that the gross ammount of unprepared/unmottivated people generates an industry of mediocrity: 1. Tools (a la VB) to facilitate life on the bells and whistles level 2. LACK of tools to facilitate like on the conceptual/productivity level (I remember seeing an article on artima saying that lisp type macros would never make Java, as the typical Java Programmer would not understand them) 3. Job interviews based on buzzword compliance and not on the ability to think and adapt 4. Software shops being afraid of using new tools because "our engineers will take a long time to understand that, we are affraid" 5. Software shops developing simple software with a test engineer/software developer ratio of 1/1 !!!
This recession could even be good to weed out people that don't belong to the industry, but in most companies, the people that hire/fire are, lots of times, of the clueless type. But, in some sense, I still hope that the recession will bring some rationality to the industry.
And then they complain that some developers are 10x more productive than others... of course.
Thank you for motivating me to re-read the The Mythical Man Month, and watch the Turing Award lecture.
"The Design of Design" lecture seemed to about Engineering Design. One message was that engineering design is not the orderly process we sometimes like to think.
The comment wasn't about how to achieve good design. Product process raises the floor from mediocre to average. To move from good design to great design we need great designers.
"The State and Future of Software Engineering" (in the last pages of the 1995 anniversary edition MMM) gives a capsule history of chemical engineering. Which leads to Fred Brooks belief "that software engineering at age 27 may be not hopeless but merely immature, as chemical engineering was in 1945".
I must not only disagree, but indeed lay some of the blame for the miserable state of our profession on thinking like this post.
Perhaps the reason that there are so many "with little or no formal training" doing quite as well as so many with formal training leads to another conclusion. The overall level of training is so low that rank amateurs with an aptitude can do *relatively* well.
I refer to two other greats of the software world - Dr. Alan Kay and Richard Gabriel. At a recent lecture at Stanford University, Dr. Alan Kay chastised the Computer Science schools for having dropped all concept of training students in Computer Science, and merely providing "vocational training" (e.g. learn Java). From Dick Gabriel read http://dreamsongs.com/NewFiles/RoadNotTaken.pdf. Make special note of slide 30 - it shows the pitifully inadequate requirements of even a Masters in Computer Science.
The conclusion is that there is insufficient formal training in the science.
Why have we let ourselves get into this? Well one issue was touched upon - the "employer" is (mostly) clueless in their selection process. They do not know how to select and that leaves the level low.
A recent example is from my own search. I got more than 100 resumes for a programming position. I asked each person to submit a small program solving a simple defined solution. More than half the programmers refused to do this, many writing back telling us how insulted they were at being tested so! The message - nobody tests! This perpetuates the hiring of low competence programmers. Out of the solutions submitted only two worked! (I hired both.) Two out of one hundred tells you what the average level is *very* low!
Real programming is difficult, and I cannot think how many of the greats of our world have pointed this out, yet we mostly hire people who have "learnt a language". Predominantly we hire people who know little of theory, and who know only one way to solve a problem (or none).
There are aspects of software that are indeed like architecture - the development of User Interactions - requiring both art and good science. But the rest is Science, and the sooner we start training people in Computer Science the sooner we will have people fit to begin their apprenticeships.
In any profession there are the great and the very bad (in our world the 10x programmers down to about one third of the people who actually cost us more money than they contribute). Programmers, I challenge you to become great. Lifelong learning of Computer Science is the key. Employers, I challenge you to develop methods to distinguish the great. Universities we all challenge you to develop Computer *Scientists* that have a great foundation that they can build their thinking on.
[Wisdom is choosing the right people to listen to, and then *listening*.]
> In any profession there are the great and the very bad (in > our world the 10x programmers down to about one third of > the people who actually cost us more money than they > contribute). Programmers, I challenge you to become great. > Lifelong learning of Computer Science is the key. > Employers, I challenge you to develop methods to > distinguish the great. Universities we all challenge you > to develop Computer *Scientists* that have a great > foundation that they can build their thinking on.
I would like to start by saying that I generally agree with the terms of your post which is, IMHO, very insightful (with a few minor cavets that can be read in my previous post).
Not only that but, there is a strong trend towards stupidification, let me give a personal example:
I have decided to do a period of strong self-training (something that I do every now and then on my free time, after work). I see mainly 2 options:
1. Domain Specific (Embedded) Languages (DSELs): This is mainly the design and implementation of languages "a la carte" for a certain problem at hand. It so happens that this languages are better implemented in either a logical (via Prolog) or functional language (I was thinking via Haskell, but not sure). I believe that this can be a signigicant productivity boost (I have done some work in the past with it, I am fairly confident), specially if the DSELs were to be compiled say, to J2EE. OR spend time in: 2. .Net! Interesting, but from a conceptual/personal productivity point of view I expect to gain nothing/very little.
For my work I am pretty sure that 1. would be the most helpful in making me deliver (if not by applying at least from gaining the insights). But from an employability perspective, to the typical hiring manager what does sound better: "DSELs with Haskell" or "J2EE/.Net drone"?
So, being "employable" seems to be very compatible with becoming stupid. With a few exceptions, this is an industry where insight/understanding is worth nothing compared with "buzzword compliance".
The current practice of software development may be more "like a craft or an art form" than engineering - and that was true of other forms of engineering in the past.
There isn't a large body of knowledge that one needs to master to become a software engineer There isn't a large body of knowledge for software development - wouldn't that suggest that this is an immature field?
"Real Software Engineering is still in the future... The real question is whether there exists a practice in between the two stronger than just piling up messes that can eventually lead us to real modern engineering processes for software."
"A corollary of the Brooks conclusion ... is that good designers receive their training from other good designers." Again, it's important to note that Fred Brooks was talking about great designers. My understanding of the Turing lecture was that great designers just are - the important thing is to learn how to spot them, how to nurture them, and then let them loose.
learned by apprenticeships rather than formal education Both are necessary! Experience matters in every endeavour.
There isn't a large body of knowledge that one needs to master to become a software engineer There isn't a large body of knowledge for software development - wouldn't that suggest that this is an immature field? The immaturity, I think, lies in the fact that there is a body that exists (whatever the size), and it is not learned from. See below (Alan Kay).
'Is Software Engineering an Oxymoron?' Alan Kay p72 http://glab.cs.uni-magdeburg.de/~croquet/downloads/Croquet0.1.pdf And on reading this, it becomes apparent that there is still little learning/training going on! In fact the net idea stated a few pages into the appendix (p.75) is: Most of current practice today was invented in the 60s" So why are so few using the last 30 years of ideas? Knowledge is sadly lacking.
"Real Software Engineering is still in the future... Again, we are not even using what is known today!
I cannot agree with great designers just are Noting: (learned by apprenticeships rather than formal education) Both are necessary! Experience matters in every endeavour. Note: Both and "Subject matter expertise must be obtained. Being a good programmer is not enough to be a good software designer (necessary, not sufficient)." My point is the formal training in the Science is weak. It is very difficult to be great without the base theory.
Dick Gabriel... We should also note slide 28 "Craft-oriented, critical writing on work by other writers" ;-) I agree! But keep the context of the depth of training on the theory before reaching this stage of learning.
I always wanted to land myself in an apprentice-type role with a true Master, from whom I could learn all the ins-and-outs of Software Engineering and System Development. For whatever reason, that never happened. Sure, I've worked with some really smart people and tried to tap them for all the knowledge I could. But I never quite found my self in the apprentice-like role that I sought. I've mostly learned what I have from books and articles.
But is this really so different? When I read Bertrand Meyer's Object-Oriented Software Construction, it feels like I'm really learning from Bertrand. It's the same with other great author's works, like the GoF Patterns. True, it would be better to collaborate with them face-to-face and get feedback but that just hasn't happened for me. For those who have coded day-in and day-out at the feet of a Great Master - I envy you!
This post from Wado is exactly in line with what we have been saying for quite sometime. My background is also Philosophy; Maybe that has something to do with it ; )
But anyone who is acquainted with philosophy knows that most of the claims of scientific methodology is only some much hype anyway. The problem is that software design is design. And design weather in traditional Engineering or Software Engineering or Architecture is essentially a creative endeavor. Even discoveries of mathematical proofs often come to a mathematician while looking at the pattern of a napkin at Seattle's best over a cup of coffee. It was always said in philosophy of science that you could give a logic of confirmation, a logic of scientific explanation but never a logic of discovery. Why? Because discovery doesn't come form exercising a series of methods. Discovery is a creative process as well.
The problem with using engineering as an analogy with what software designers do is partially because we don't understand what engineers do. Engineers have to have the formal background because the materials that they work with have complicated constraints and relations. The materials we work with have comparatively few constraints.
Our discipline is not made better by the fact that we (i) don't understand science (ii) don't understand engineering (iii) don't understand design and instead use poor metaphors to help us understand where we understand neither the analogue nor the analogy.
The factory metaphor that software development is like making a car is fundamentally flawed. Software development is more like the design of the car. This is not a simple process. The production of the car is done by the compiler. The compiler is the cheep labor management wants.
I think it is interesting that one attack of this post quotes Alan Kay and Richard Gabriel. This is a funny case of missing the point. Both of the individuals would agree with the post. Richard Gabriel in particular does not think of software engineering as engineering. Oh well.
Actually my point was that Science and Engineering are no more rigorous on their design process than we are. The only part that is is after something is finished being designed and needs to be done over and over again. We don't do that in software. ALL of the programming work is design and the part that is done over and over is done by the compiler.
I find this funny all the time. I was a PhD student in philosophy and took graduate courses in other areas, Linguistics and Biology for example. The seminars were always filled with "Why can't we be more like physics". When you go to the physics graduate lectures they say "We can't understand any of this, and they would wax philosophical usually way off any possible rigorous way of looking at philosophical problems.
I say again. DESIGN is a craft and an Art in every discipline not just ours. When I say we don't understand engineering and science I mean that we are fooling ourselves when we think Science is a science. Nothing matches what we think of when we think of the rigors of science and engineering.
I think we are all saying similar things, and our terms (definitions) are different. I define mine.
Engineering is applied technology that requires a specific education. Design is creative problem solving. Creativity is a process of combining things to form something new and useful.
My point, and the elements from Alan Kay and Dick Gabriel I choose to quote, are about the lack of "specific education" amongst a great part of our industry, notwithstanding the fact that there is a huge pool of knowledge to be had.
If one does not have the underlying knowledge of the theory and practice (gained in school, by private study, under tutelage or however), you are unlikely to come upon a good design. Good design combines existing good designs, so if you do not have good designs to combine, what do you get?
Is there more than one way to solve a problem (create a design)? Of course! Is it possible to say one design is better than another? Mostly. Constraints on the problem are often complex, but it often boils down to: Correctness - the program does the job Maintainability - someone can evolve the program as needs change Productivity - the design/solution was created in a reasonable time
I have interviewed many programmers over the years, and here is only one small example of lack of specific education. Three solutions -- one is O(n^2), one O(n) and the last O(constant amortized). More than half have no idea what I am talking about. How can they, then, come up with good designs if they have not been educated in computer science (other than by dumb luck)?
P. S. I am scientist (theoretical physics), engineer (telecommunications system testing) and "software engineer".
P. P. S. Science is not science? Only a philosopher ;-)
Flat View: This topic has 27 replies
on 2 pages