The Artima Developer Community
Sponsored Link

Weblogs Forum
I Need Smart RSS

14 replies on 1 page. Most recent reply: Aug 11, 2007 11:50 AM by Helgi Helgason

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 14 replies on 1 page
Bruce Eckel

Posts: 875
Nickname: beckel
Registered: Jun, 2003

I Need Smart RSS (View in Weblogs)
Posted: Apr 6, 2007 2:55 AM
Reply to this message Reply
I've come to rely on other people rather than RSS feeds to point out the really important stuff to me.

I stopped watching TV news and reading newspapers years ago, and am much happier for it. And I don't feel terribly uninformed. If some non-infotanment thing happens, someone always seems to tell me about it, so I find out. Or I'll occasionally watch John Stewart and he brings me up to speed without being too awfully serious about it.

I tried RSS from both ends and found it a necessary idea but a bad implementation. Necessary because it's helpful to know that something useful has appeared on the internet. Notice I say "that" something has appeared, rather than "when" it has appeared, and the "when" is part of the reason I turned off the RSS feeds. It became another form of noise, interrupting my thought process and throwing me out of the flow. Yes, there are all kinds of adjustments you can make so your RSS reader isn't popping things up all the time. I know that, but the general problem is noise. Having something that goes out and collects everything is like subscribing to the New Yorker. They just pile up and give you yet another task to accomplish, most of which is wading through yet more accumulated cruft trying to figure out what's important so you can read that. And as much as we try to say "it's not that hard to skim," in my experience it takes a remarkeable amount of mental energy to do the wading, and I seem to have a lot more in reserve to focus on the real thing when my informal network of filter-people (usually folks who have come to conferences and workshops) do it for me.

The other reason RSS is a bad implementation -- but in the current world, the only type of implementation that is practical -- is that what we really want is a "publish-subscribe" mechanism. You only really need to be told once that a new article is posted. But you don't want to give me your identity, because I might abuse it. So instead of a simple publish-subscribe, RSS is a constant-checking mechanism, which is so inefficient that it might be second only to porn in sucking down bandwidth on the Internet. That's just conjecture, but my experience of writing my own RSS feeder was that it rapidly hammered my web site so much that my ISP started charging me significantly more money. This is why intermediate feed services appeared, but of course those feed services must then handle the traffic. If RSS had been given any serious thought when it was designed, the feed provider would have been given control over the number of visits the reader could have. Maybe I can't know your information, but I should be able to throttle the amount of hammering you're doing on my site.

It all comes down to noise. RSS gives me as a consumer way too much noise, and it inflicts the provider with too much noise.It might be considered a kind of agile solution because of the way that it was created -- quick and simple -- but that's not all there is to agility. Agility is also, and possibly most importantly, about improving the implementation as new information is acquired. To me, agility means you are always in the midst of an ongoing experiment.

Because of its static nature, RSS is really a Web 1.0 technology. There's no way for the community at large to add information. Because of this, RSS pours more noise into your world. What I need is something that only gives me the right information, for me. This is something I would be more than happy to pay for; enthusiastic, even. I'm not sure that I would need to pay for it, or if that would be the right solution -- especially because it might end up being payment per good article and whoever was being paid would then have incentive to push more articles at me in hopes of getting more money.

A better solution might be something that combines the approach of with a friend list. Pandora allows you to select music you like from general categories, and when they play songs you do or don't like, you give feedback and the mechanism theoretically gets better and better at predicting what you want to hear. Things like Digg have touched on this but they are producing a mass vote, not a customized preference. I don't really want to read what gets the highest votes. I want to read what is important to me, not to the largest number of people that happen to be interested in voting on articles.

So we need an inversion of control, where it's not about "most popular" but rather about "most appropriate." And it constantly improves the way it defines what this means.

That, of course, continually narrows the selection, which is desirable but can become too narrow. That's where the "friends" mechanism comes in. If I have a pool of friends and we can throw suggestions into that pool, this will help discover new potential interesting material. If one friend tends to throw in things that are too noisy, I can quietly block that person out without them knowing about it. They might still be making valuable contributions to others in the pool but if those contributions don't work for me then I can tune them out.

This design could be the next newspaper, the one that the futurists have been promising for awhile: the customized newspaper where you only have articles that are good for you. Of course, most of the futurists couldn't see how this was going to happen and so they assumed the current newspaper organizations would build these papers for you. In general I think this is still unlikely, mostly because newspapers are too stuck in the way they do things right now. However, I could still imagine a design like the one I describe above being the thing that saves newspapers, precisely because I would be willing to pay for it. It provides a valuable enough service for me. And if there were traditional newspaper columnists that I felt were valuable and informative, I think I'd be willing to pay for those as well -- of course, that would probably inevitably extend to "regular" bloggers, especially when micropayment systems finally get sorted out.

Dima Berastau

Posts: 3
Nickname: xored
Registered: Jul, 2006

Re: I Need Smart RSS Posted: Apr 6, 2007 5:14 AM
Reply to this message Reply
This is very timely and I hope is going to resonate with a lot of people. Whatever happens to RSS needs to be incremental though. RSS is a simple standard that's got widely adopted, so there's definely no need to mess with that. However we totally need to step away from "Pull" model towards a "Pull/Push" model.

Brian Panulla

Posts: 6
Nickname: bpanulla
Registered: Aug, 2005

Re: I Need Smart RSS Posted: Apr 6, 2007 6:25 AM
Reply to this message Reply
I see the same things. I love using RSS over email distribution lists, but there's no intelligence to filter the feeds unless you only subcribe to certain keywords. There are several bloggers that I read pretty much everything they write, but others talk about their pets, and nature photography, and other random hobbies that, while interesting, do get chalked up to "noise" in my pile of feeds.

But if I only subcribed to specifically tagged feeds (like "flex" or "coldfusion") I feel like I could be missing something interesting that just happens to be outside of a given blogger's normal territory.

One thing that does provide a sort of filtered RSS experience is A bunch of my colleagues and I use it for our bookmarks, and we subscribe to each other's bookmark feeds via RSS. So I can sort of look over people's shoulders as they go about their bookmarking business. If it was interesting enough for them to tag, it's probably worth reading.

Mine are here if you're curious:

Joe Cheng

Posts: 65
Nickname: jcheng
Registered: Oct, 2002

Re: I Need Smart RSS Posted: Apr 6, 2007 9:23 AM
Reply to this message Reply
> If RSS had been given any serious
> thought when it was designed, the feed provider would have
> been given control over the number of visits the reader
> could have. Maybe I can't know your information, but I
> should be able to throttle the amount of hammering you're
> doing on my site.

The Expires HTTP header is designed for this, and well-written feed readers do respect it. Some sites like Slashdot produce this header and if the reader doesn't respect it, it gets banned for 24 hours (unfortunately their implementation is IP based, so networks behind a single IP will often get blocked--at least this was true in 2005 or so).

There are also the Etag and Last-Modified HTTP headers that will greatly reduce the amount of bandwidth required if there is no change.

Though I agree that RSS in general seems to have had little thought put into it :)

Kevin Lawrence

Posts: 13
Nickname: kevlaw
Registered: Feb, 2005

Re: I Need Smart RSS Posted: Apr 6, 2007 9:49 AM
Reply to this message Reply
If there was ever an task for which web browsers are ideally suited, it's reading web pages - and blogs are just web pages. I recommend reading blogs in a browser rather than a standalone RSS reader - definitely don't let them pollute your email client as you'd never get anything done.

I use Google Reader for RSS and the experience is exactly that of reading a custom newspaper where I get to choose the authors. If and when support for 'tags' becomes mainstream, I'll be able to customize the content even more.

- For example - I don't care about Andrew Sullivan's dogs or the view from his window, so don't show me those.

- Another example, if you liked Bob's post on closures, you might also be interested in Fred's post which is rated 4 stars by people like you.

- You might also set filters like 'posts about TDD rated 5 stars by people like me'.

ISTM that RSS is more than adequate to support applications like this. We just need smart aggregators on top of RSS to make it more manageable. Google's reader is 80% of the way there already.

In the old days of the web, you used to read everything (admit it, you did, didn't you?) but you don't do that any more because it started to become a waste of time. That wasn't the fault of HTML or HTTP - it's just that we had not learned to use the new medium well.

You can avoid many of the downsides of RSS with disciplined reading habits. For instance:

- I check it twice a day so it never interrupts my flow (TIP - don't use the customized Google home page or your will be tempted to read your RSS every time you search).

- Don't subscribe to news that you would go read anyway (bbc, google news, cnn etc) - that's when you'll start to fall behind and it will become a chore.

- If someone starts posting about their cat, unsubscribe.

- If you get more than about 2 posts a day from someone, unsubscribe.

I am subscribed to about 150 blogs and I receive about 10, maybe 15, posts a day. Google makes it easy to scan them and skip over the ones that don't look interesting and when, as occasionally happens, there is a post by Bruce Eckel, I am delighted and save it for a comfortable read when I get home from work.

David Paul Robinson

Posts: 1
Nickname: davidpaul
Registered: Apr, 2007

Re: I Need Smart RSS Posted: Apr 7, 2007 3:11 PM
Reply to this message Reply
It's interesting how universal this problem is - both simple and complicated at the same time.

I don't want to subscribe to 50 different feeds - it's ridiculous. On the other hand I don't want to get all my information from only a few writers or sources.

What I really want is to read a truly excellent linkblog edited by someone I trust and who thinks exactly the same way I do...... Sort of a surfing clone!

Martin Stanhope

Posts: 3
Nickname: slithytove
Registered: Jun, 2006

Re: I Need Smart RSS Posted: Apr 8, 2007 7:41 AM
Reply to this message Reply
Hi Bruce,

it might be worth having a look at the new Pipes service from Yahoo which you can use to combine and filter RSS feeds to your heart's content.


David Beutel

Posts: 29
Nickname: jdb
Registered: May, 2003

Re: I Need Smart RSS Posted: Apr 10, 2007 4:12 PM
Reply to this message Reply
A P2P rating system for blogs would be nice, like Pandora or Netflix. It should be a web site, though, not a new RSS that would need to be built into a bunch of clients.

Florin Jurcovici

Posts: 66
Nickname: a0flj0
Registered: Feb, 2005

RSS is perfect! Posted: Apr 11, 2007 2:39 AM
Reply to this message Reply
A smart guy once said that when you design software, you should try to mimick the real world. The smart guy is Arthur J. Riehl, and he said this in his book "Object-oriented design heuristics", which was enlightening to me.

Suppose you don't have RSS. What you would do to collect news, in this situation, is to browse several web pages, and filter the interesting news.

This is what RSS does. Only, RSS isn't that smart to filter out the news you're not interested in - it cannot do so in an universal way, since each person has a different way of thinking, and different interests. It just can automate the process of browsing the web sites.

So IMO it is not a bad design of RSS that makes you unhappy, but the lack of an app sitting on top of RSS, thinking exactly the same way you do, and filtering out stuff you don't want to know about.

As for push vs. pull: this is where RSS acts more or less like a human - thus mimicking the real world. RSS mimicks a human browsing web sites looking for new news items. Why would then RSS do otherwise?

The push alternative, however, is also available: newsletters.

Jeff Manson

Posts: 2
Nickname: jeffman
Registered: Jan, 2003

Re: I Need Smart RSS Posted: Apr 11, 2007 3:06 AM
Reply to this message Reply
I think the StumbleUpon approach is fairly close to the sort of thing you're talking about. You register the topics of interest to you and you can explore all the new sites which other people with the same interests as you have found.

You can provide feedback on each site you visit so that your likes/dislikes can be fine tuned and in turn those sites can be presented to others as potential new sites to visit.

You get that network/community of people with similar interests to your own, and you can come and go as often as you wish without being overwhelmed with unnecessary feeds.

Jim Goss

Posts: 1
Nickname: jgoss1074
Registered: Apr, 2007

Re: I Need Smart RSS Posted: Apr 13, 2007 1:04 PM
Reply to this message Reply
I have actually built something like what you're looking for, though it's still essentially in beta mode at the moment (and the friends/social aspect is not yet fully implemented). Essentially, I've built a site that pulls content off of thousands of popular feeds (news & blogs primarily) and allows the user to mark articles liked/disliked, then makes recommendations as it learns topics of interest. It doesn't use a popularity algorithm nor does it assume that if Bob and Jeff like article A, and Bob likes article B that Jeff will like article B.

I won't post the url here, in the interest of not violating any potential no advertising rules (I'm new here--actually came across this article through the site I'm referring to above). But if you're interested (or if it's OK for me to post the url here) respond on this thread and I'll pass it along.


Shad ilac

Posts: 1
Nickname: shadilac
Registered: Apr, 2007

Re: I Need Smart RSS Posted: Apr 13, 2007 2:22 PM
Reply to this message Reply
You may want to take a look at, it solves a few of the concerns you raised.. Of course it will take care of updates for you, and it also has the concept of a "shared bundle" which essentially means that you create a topic then all your friends can subscribe to the topic, and everyone has the ability to add feeds that are relevant. I have actually discovered a few good feeds that way, when some guy signed up to the site, saved my bundle and added a feed.

Of course it also supports all the keyword filtering in the bundles, but even cooler is a complete separate filter product. Essentially you can find/replace anything you want in a feed and re-publish it. Strip out ads, whatever.

It also has a digg-like voting feature, and in your account it will tell you all the articles your friends have voted for.. Almost like the facebook news feed. That's a great way to share news with your friends.

anand kishore

Posts: 1
Nickname: anandkish
Registered: Apr, 2007

Re: I Need Smart RSS Posted: Apr 24, 2007 11:40 PM
Reply to this message Reply
What I really want is to read a truly excellent linkblog edited by someone I trust and who thinks exactly the same way I do...... Sort of a surfing clone!

I have already been working on this.

Related links:

Mike Demmitt

Posts: 1
Nickname: mikedemm
Registered: May, 2007

Re: I Need Smart RSS Posted: May 28, 2007 10:17 AM
Reply to this message Reply
Hey Jim,
Your rss aggregator idea sounds interesting. Can I check out the beta?

Helgi Helgason

Posts: 1
Nickname: hphelg
Registered: Aug, 2007

Re: I Need Smart RSS Posted: Aug 11, 2007 11:50 AM
Reply to this message Reply
Just wanted to point out that there is an intelligent RSS reader available from that learns your interests and recommends items for you. It's called Perseptio FreeAgent and is available for free download from

Flat View: This topic has 14 replies on 1 page
Topic: I Need Smart RSS Previous Topic   Next Topic Topic: Windows Vista

Sponsored Links


Copyright © 1996-2019 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use