Weblogs Forum - Software Development Has Stalled

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Weblogs Forum
Software Development Has Stalled

164 replies on 11 pages. Most recent reply: Mar 28, 2010 10:20 AM by Florin Jurcovici

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 164 replies on 11 pages [ « | 1 ... 6 7 8 9 10 11 | » ]

Rinie Kervel

Posts: 26
Nickname: rinie
Registered: Oct, 2005

Re: Software Development Has Stalled

Posted: Feb 11, 2010 9:35 AM

> So what you mean is that a spreadsheet may contain hidden
> structure since data in one cell may depend to data in
> another cell? and you want this structure to be available?
Yup. I think you need multiple levels of abstractions and accept irregularities, existing programs and protocols.
I think a reason for stalling is that we try to use objects for everything: persistent data and I/O, and don't reuse abstractions that work but are not purely OOP.
So no Corba, Soap but REST and JSON, HTTP and SqLite.

Merriodoc Brandybuck

Posts: 225
Nickname: brandybuck
Registered: Mar, 2003

Re: Software Development Has Stalled

Posted: Feb 11, 2010 10:44 AM

> > Just before Dr. Dobb's and the other CMP publications
> all
> > had their print editions killed, there was a series of
> > articles talking about programming the cell processors
> > that power the Playstation. With the architecture you
> can
> > make some tried and true algorithms really sing, but
> > programming it was hard in a lot of cases. You need to
> > totally rethink how you break up jobs and traverse
> > structures with millions upon millions of nodes because
> of
> > the architecture. http://www.drdobbs.com/cpp/197801624
> The rethinking has already been done and is called map
> reduce. Trouble is, that you should not solve that in a
> single program, but in a scalable way: IO-protocol HTTP or
> protocol buffers, grid file system and say CouchDB or
> Hadoop...

Which is nice, in theory, but at this point in time there is no way that you are going to map the amount of data that the play station does using the types of technologies you describe to my display (tv, computer monitor, etc.) at a rate of greater than 30 frames per second. While allowing me to run around and interact with dozens of characters controlled by the processor's AI and blow a lot of shit up and have it be smooth and seamless. I can't even get a 5 minute flash cartoon to play for 5 minutes consistently without some kind of hiccup. The kinds of things the PlayStation does rely very heavily on the hardware in the machine. As nice as it would be to abstract that away into some generic set of libraries and data stores, you can't. Yet. And by the time you can, somebody will stumble upon some other set of problems where the big gains are going to heavily dependent on some thing that can't be automagically abstracted away and explained with multisyllabic words.

And again, you're talking about things that aren't new or innovative in some people's definition of the word. They are existing tools and technologies or, at best, very small increments on existing tools and technologies. The reality is that for a great many jobs that are time sensitive you need to have large amounts of memory and processing power available and in those cases, at least today, those are at least somewhat hardware dependent. The MapReduce you so haughtily trot out (I mention it in the paragraph following the one you quote, too, it's not new to me) had Google file the patent in 2004 and that itself refers to a patent document filed in 1975. Yes, MapReduce is, in some form at least, older than Yoda.

Lastly I will submit that scaling out isn't always the answer. Some people seem to believe very deeply that scaling out will, eventually, solve every problem. As if grid file systems and distributed databases are the solution to every problem. In some ways they seem to be solutions looking for a problem. I see them recommended a lot, but used in practice very seldom. Maybe that will change, for a short while, when somebody finds some relatively neat way to use this technology in an innovative and profitable way. The couch db in the wild page is a nice collection of things, but I don't see anything that makes me go WOW. If it was so great, why isn't that the datastore used for something like amazon or yahoo or google? Google is famous for writing their own custom datastore/file system because nothing available would do what they needed to do. I don't know how somebody with as much data as amazon or yahoo stores their data and has it quickly accessible. Does anybody here have a clue? I'd love to know.

And how is a distributed database and HTTP anything going to help with my on board guidance system and heads up display in my fighter jet? Or with the imaging on the medical equipment that will show me the insides of my heart or lungs in real time? Or any of probably hundreds of other problems that don't fit the model you propose?

Much like we made the big shift from mainframe computing to client server and are now moving back to the mainframe model. Except now the mainframes are called 'web servers' or 'the cloud'. But the concept is the same. I'm pointing my dumb terminal (web browser) at some piece of big iron (maybe we need a new term for all the virtual machines out there in the cloud? Hot air, maybe?) and all of the interesting work is done 'over there somewhere' and I see the results on my screen.

Initially I had disagreed with Bruce's opinion, but after reading this thread I think he's right. We have a long way to go to efficiently use what we already know, never mind come up with big new tools and techniques to do things.

Merriodoc Brandybuck

Posts: 225
Nickname: brandybuck
Registered: Mar, 2003

Re: Software Development Has Stalled

Posted: Feb 11, 2010 10:48 AM

> > So what you mean is that a spreadsheet may contain
> hidden
> > structure since data in one cell may depend to data in
> > another cell? and you want this structure to be
> available?
> Yup. I think you need multiple levels of abstractions and
> accept irregularities, existing programs and protocols.
> I think a reason for stalling is that we try to use
> objects for everything: persistent data and I/O, and don't
> reuse abstractions that work but are not purely OOP.
> So no Corba, Soap but REST and JSON, HTTP and SqLite.

Does anybody else find this contradictory? The more irregularities and existing programs and protocols you decide to work with, the more difficult it is to come up with clean abstractions. Eventually you may end up with so many abstractions that you have one unique abstraction for each problem, which is the same as having no abstractions at all.

Raise your hand if you've ever worked on something that is so overly abstracted that the abstractions can't be applied to any other project you have worked on, are working on or will ever work on. My hand is high in the air. Anybody else?

robert young

Posts: 361
Nickname: funbunny
Registered: Sep, 2003

Re: Software Development Has Stalled

Posted: Feb 11, 2010 11:09 AM

> You can still have transactions in undisciplined data.

No, you can't. I'll leave it that.

robert young

Posts: 361
Nickname: funbunny
Registered: Sep, 2003

Re: Software Development Has Stalled

Posted: Feb 11, 2010 11:15 AM

> (I'm
> surprised nobody has yet mentioned prolog and its ilk.
> Maybe they have and I've just glanced over it?)

Not sure about this thread, but, yes I have.

>
> That the discussion has gone on for 100+ posts and does
> little to address Bruce's initial thought I think proves
> his initial thought brilliantly. We developers, as a
> group, generally latch onto the interesting part of a
> bigger problem and dissect and analyze and deconstruct and
> reconstruct that particular piece and in many instances
> this piece does little to advance the solution to the
> bigger problem. Sure, we're making some progress, but not
> as much as we might otherwise make if we were to
> occasionally take a step back and look at the bigger
> picture.

Depends on how one defines that Bigger Problem/Picture, which is why the thread has grown like topsy. Some think the Big Picture lies in new & better high level languages, others concurrency, others data management, others tooling. Writing iPhone apps is a different problem from defining and managing data for a commercial enterprise, which is completely different from the problem of managing the development of weather or weapons simulations. And so on.

robert young

Posts: 361
Nickname: funbunny
Registered: Sep, 2003

Re: Software Development Has Stalled

Posted: Feb 11, 2010 11:21 AM

> > > So what you mean is that a spreadsheet may contain
> > hidden
> > > structure since data in one cell may depend to data
> in
> > > another cell? and you want this structure to be
> > available?
> > Yup. I think you need multiple levels of abstractions
> and
> > accept irregularities, existing programs and protocols.
> > I think a reason for stalling is that we try to use
> > objects for everything: persistent data and I/O, and
> don't
> > reuse abstractions that work but are not purely OOP.
> > So no Corba, Soap but REST and JSON, HTTP and SqLite.
>
> Does anybody else find this contradictory? The more
> irregularities and existing programs and protocols you
> decide to work with, the more difficult it is to come up
> with clean abstractions. Eventually you may end up with so
> many abstractions that you have one unique abstraction for
> each problem, which is the same as having no abstractions
> at all.
>
> Raise your hand if you've ever worked on something that is
> so overly abstracted that the abstractions can't be
> applied to any other project you have worked on, are
> working on or will ever work on. My hand is high in the
> air. Anybody else?

^^^^ ^
||
||

Achilleas Margaritis

Posts: 674
Nickname: achilleas
Registered: Feb, 2005

Re: Software Development Has Stalled

Posted: Feb 11, 2010 11:56 AM

> Bruce's thought is that development has stalled. That a
> lot of the big wins have been had and that we are not
> using the tools we already have well, never mind making
> BIG new advances in tools or implementation. Your 'X',
> from reviewing previous posts, involve decoupling, which I
> was taught in my introductory programming class in 1991
> and references to the many things lisp has done right. So
> you are proposing that we move forward by going back to
> things that are between 19 and 50 years old. I find that
> particularly humorous because you were the first person to
> take issue with the text book definition of innovate that
> I found and posted :-)

Perhaps I did not make myself clear.

My opinion is that we can have big new advances if we go back to some concepts that were abandoned but seem to worth revisiting.

One of these concepts, in my opinion, is the uniform representation of data. The concept of uniform representation originated in LISP. I do not propose to go back to actual LISP, but to revisit some of the concepts of LISP, perhaps in new way to implement them.

Regarding the decoupling of data and code, I do mean to abandon encapsulation. I mean to decouple information from programs. Right now, information is not readily available for anyone to manage. One has to go through a particular application to manage the information. This is very different from encapsulation, which I do not think it should be abandoned.

>
> And many have gone on tangents that have something to do
> with defining language constructs that might make things
> easier in some cases, but are clearly a loss in others. We
> already know a lot about typing and type systems and
> relational data stores and modeling and how the current
> implementations fall short of the theory proposed decades
> ago. And in defense of people's positions they point to
> material that has existed for many, many year. Move along,
> nothing NEW to see here.

And I say to that the next step may be to forget the theory, and go to something that has not been attempted so far: go to undisciplined data. Let the computer find the patterns! do not expose the programmer to the underlying technologies. Make information abstract, just like humans are used to deal with.

>
> > The question is: what do YOU propose for continuing the
> > software progress.
>
> There was no question. Bruce simply made a statement based
> on his opinion and observation.

I think it was. A statement of this magnitude has some implicit questions in it:

a) first of all, is it true that software development has stalled?

b) secondly, if so, then what can we do to progress?

In my opinion, that is the hot topic that should be discussed. Otherwise, we can simply reply 'yes' or 'no' to Bruce and go on with our business. But this is not worthy of a discussion.

>
> But since you asked, I think a lot of the big wins we have
> left to gain from tools involve figuring out how to enable
> the majority of programmers to write robust systems that
> can execute as many tasks in parallel as possible. It is
> mind boggling to me how much computing power is utterly
> wasted on the average corporate network. From what I've
> seen, we are still scratching the surface here.

Good. I agree with you - one of the biggest steps would be to enable parallel programming for the masses.

If you are interested to hear my ideas on this, I can tell you.

> Other posts have led me to believe that some people have
> an idea that there is one magical language somewhere that
> would be good for all problems. I don't think that will
> ever be the case.

I agree with you. I am not proposing abandoning existing programming languages, but to make new ones that fill the existing gaps.

> At this point I think my favorite
> example of this is D, which looks to be a solid systems
> language and has pulled elements from a variety of other
> languages to take care of a lot of mundane things that
> don't matter in most cases, but when they do matter you
> can pry the box open and tinker with them. Another is
> erlang. It is becoming the defacto language for people
> that want to write large stable parallel systems. I've run
> across a few good examples of online game engines and
> other things that people have put together in the past
> couple years. Very recent and things that a few years ago
> a single or group of 2 or 3 programmers couldn't even
> imagine putting together, never mind actually build and
> have something working.
>
> The other road is to have a language so flexible and open
> that you can construct your own DSL to do your specific
> task very well. I think, much to many people's chagrin
> here, that C++ was the start of that trend. I can simply
> type C++ and I can already tell a few people are
> reflexively convulsing. Many people, even C++ experts,
> have made it pretty clear that the language is LARGE and
> nobody, not even Stroustrup himself, knows all the nooks
> and crannies and everything it can do. I think that
> tradition has been carried on by Ruby and I would point to
> Rails and Rake as two very fine examples of DSL's that do
> a bang up job of making programing in their domain much
> simpler than in the general case. But knowing Rails
> doesn't help you with Rake very much and to take advantage
> of all of the features of either you need to have a pretty
> darn good understanding of Ruby to begin with.

I have some ideas that combine the performance of C++, the flexibility of LISP and the parallelism of Erlang. We can open a discussion about this, if anyone is interested.

Achilleas Margaritis

Posts: 674
Nickname: achilleas
Registered: Feb, 2005

Re: Software Development Has Stalled

Posted: Feb 11, 2010 11:57 AM

> > > That's not the question I asked but it doesn't really
> > > matter now. If you are really curious about what I
> > wrote
> > > you can always go back and look.
> >
> > I am trying to understand how do you see the problem of
> > software development having stalled, and what possible
> > solutions would you see.
>
> I don't think I said that I believe software development
> has stalled. If I did, I apologize for lying because I
> don't really think that.
>
> I think software development is maturing albeit very
> slowly. What has stalled is constant churn of recycled
> ideas and proposed silver bullets.
>
> I believe that the core skill of software programming is
> an intractable problem that cannot be solved by any
> technology. The shift from trying to find silver bullets
> to building tools that help people solve those problems is
> the reason the development tool bubble is popping.
>
> Frankly, I think this industry would be far better off
> without people who continue to push the magical thinking
> that a tool or language or framework will somehow solve
> all the problems of software development.

Thank you for the reply. For the record, I agree with you.

Achilleas Margaritis

Posts: 674
Nickname: achilleas
Registered: Feb, 2005

Re: Software Development Has Stalled

Posted: Feb 11, 2010 11:59 AM

Achilleas Margaritis

Posts: 674
Nickname: achilleas
Registered: Feb, 2005

Re: Software Development Has Stalled

Posted: Feb 11, 2010 12:01 PM

> > You can still have transactions in undisciplined data.
>
> No, you can't. I'll leave it that.

Yes, you can. The concept of transaction is about 'either all succeeds, or nothing succeeds'. This has nothing to do with how much the data are disciplined. You can do transactions on binary blobs, if you like.

Achilleas Margaritis

Posts: 674
Nickname: achilleas
Registered: Feb, 2005

Re: Software Development Has Stalled

Posted: Feb 11, 2010 12:03 PM

> Writing iPhone apps is a different problem from
> defining and managing data for a commercial enterprise,
> which is completely different from the problem of managing
> the development of weather or weapons simulations. And so
> on.

Not really. The similarities between those are way more than you think. I am confident about this, since I have been involved in military applications for the last decade, and in iPhone development the last six months, with the occasional mix of commercial enterprise apps thrown in from time to time.

Merriodoc Brandybuck

Posts: 225
Nickname: brandybuck
Registered: Mar, 2003

Re: Software Development Has Stalled

Posted: Feb 11, 2010 12:48 PM

In no particular order. These threads are why I love this site :-)

> > Writing iPhone apps is a different problem from
> > defining and managing data for a commercial enterprise,
> > which is completely different from the problem of
> managing
> > the development of weather or weapons simulations. And
> so
> > on.
>
> Not really. The similarities between those are way more
> than you think. I am confident about this, since I have
> been involved in military applications for the last
> decade, and in iPhone development the last six months,
> with the occasional mix of commercial enterprise apps
> thrown in from time to time.

Yeah, but it is the differences that can kill you. If you've done them all, I'm sure you've worked with people that were somewhere between mediocre and good with enterprise apps that would cry if they had to follow the rigor of something that was shipping for the military. Likewise I've known people that worked on military grade stuff that couldn't handle the pace and flexibility needed to do enterprise stuff. Some folks just aren't wired that way.

>> Frankly, I think this industry would be far better off
>> without people who continue to push the magical thinking
>> that a tool or language or framework will somehow solve
>> all the problems of software development.

> Thank you for the reply. For the record, I agree with
you.

I agree completely, but we can come up with tools and frameworks that make our lives easier. I just don't want that to get lost in the shuffle. I'm guilty of throwing the baby out with the bathwater on that one from time to time.

> I have some ideas that combine the performance of C++,
> the flexibility of LISP and the parallelism of Erlang. We
> can open a discussion about this, if anyone is
> interested.

Definitely interested. I'd be interested to see how your thoughts compare with what D has already done. While D certainly isn't as flexible as LISP, it was originally put together as a better C++ (it performs almost as well in most situations from the benchmarks I've seen and it keeps the one thing I love about C++ which is at best an afterthought in almost every other language and framework out there, and that is deterministic destruction) and it has some pretty solid primitives for parallel programming that take care of most of the issues with threads.

>My opinion is that we can have big new advances if we go
>back to some concepts that were abandoned but seem to
>worth revisiting.

So you're saying innovation isn't all it's cracked up to be? Because this is much more in line with my view/definition of innovation that I originally put forth, which you beat up. Revisiting existing, abandoned concepts is about as far from 'new' as you can get.

>One of these concepts, in my opinion, is the uniform
>representation of data. The concept of uniform
>representation originated in LISP. I do not propose to go
>back to actual LISP, but to revisit some of the concepts
>of LISP, perhaps in new way to implement them.

>Regarding the decoupling of data and code, I do mean to
>abandon encapsulation. I mean to decouple information from
>programs. Right now, information is not readily available
>for anyone to manage. One has to go through a particular
>application to manage the information. This is very
>different from encapsulation, which I do not think it
>should be abandoned.

At some point data needs context. badbread can be a string, a hex representation of an integer or a memory address, depending on the context and what level you are comfortable working at. I can understand why you want to do away with that sort of thing, but in some cases, it is important to know things like register addresses and memory locations. After all, somebody has to write all these interpreters and compilers and programming tools. In the average case you don't need, or even want, to know about these things, but when you need them, you need them.

The program still has to run and do something. Sometimes how fast and how much information it can process is part of the requirement and we don't have enough spare cycles available yet where you can just make the need to know this stuff disappear. We're getting there, slowly, but we're not there yet. I'll be surprised if we are there in my lifetime. Sure we should work toward the theoretical goal, but every year I get a little more saddened by how many people have absolutely no understanding of what they are doing when they type stuff into a text editor. It almost physically pains me when somebody hits a wall and they have no idea where to even begin to work around it. They just did what the manual/book/tutorial said and suddenly it doesn't work and I don't know why and can you please please please fix it.

Having a big pile of stuff without any structure or context is good in some cases and bad in others. If I am a herpatologist who specializes in studying constrictors and google for 'python', I'll find the results to be useless. Information, like sound, without structure and context is just noise.

robert young

Posts: 361
Nickname: funbunny
Registered: Sep, 2003

Re: Software Development Has Stalled

Posted: Feb 11, 2010 1:11 PM

> > > You can still have transactions in undisciplined
> data.
> >
> > No, you can't. I'll leave it that.
>
> Yes, you can. The concept of transaction is about 'either
> all succeeds, or nothing succeeds'.

There is a tad more to it than that, in practice. What matters is the implementation, and therein lies the discipline: the transaction manager. Again, if you want to read the algebra involved, get W&V; if just the concepts then G&R.

Merriodoc Brandybuck

Posts: 225
Nickname: brandybuck
Registered: Mar, 2003

Re: Software Development Has Stalled

Posted: Feb 11, 2010 1:16 PM

I've only breezed though this, but it seems particularly relevant given the turn this discussion has taken

http://www.info.ucl.ac.be/~pvr/VanRoyChapter.pdf

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Software Development Has Stalled

Posted: Feb 11, 2010 1:44 PM

>>> Frankly, I think this industry would be far better off
>>> without people who continue to push the magical
>>> thinking that a tool or language or framework will
>>> somehow solve all the problems of software development.
>
>> Thank you for the reply. For the record, I agree with
> you.
>
> I agree completely, but we can come up with tools and
> frameworks that make our lives easier. I just don't want
> that to get lost in the shuffle. I'm guilty of throwing
> the baby out with the bathwater on that one from time to
> time.

I'm all about better tools. I don't think we are anywhere near have the best possible toolset (and no, I don't think LISP is the answer either.)

What I am saying is that we need to leave behind the idea that if we can just get everyone to use <i>X</i> or follow the <i>Y</i> paradigm, all our issues would evaporate.

This would be a big move forward on two fronts. 1. On the tool user side, we would refocus on problem solving and what value our solutions provide instead of what technologies we are using. 2. Tool builders would have to accept that while what they create will have a lot of value, it won't, by itself, be enough. In other words, the ability to be composable with other solutions will be a major requirement.

I think we are already seeing this shift and there are already many people on the maker and user side (all makers are also users) who have internalized this concept. What remains is to bring everyone else along.

Flat View: This topic has 164 replies on 11 pages [ « | 6 7 8 9 10 11 | » ]

Previous Topic

Next Topic