The Artima Developer Community
Sponsored Link

Artima Developer Spotlight Forum
Seven Lessons on Scalability from Reddit

51 replies on 4 pages. Most recent reply: Jun 29, 2010 8:15 PM by John Zabroski

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 51 replies on 4 pages [ 1 2 3 4 | » ]
Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Seven Lessons on Scalability from Reddit Posted: May 17, 2010 7:52 PM
Reply to this message Reply
Advertisement

At a talk at the Future of Web Apps this year, Steve Huffman, co-founder of Reddit.com, summarized seven lessons learned while building and growing the Reddit.com social link and discussion site. The video is here:

http://vimeo.com/10506751

The slides are here:

http://www.slideshare.net/carsonified/steve-huffman-lessons-learned-while-at-redditcom

And a transcript is here:

http://carsonified.com/blog/dev/steve-huffman-on-lessons-learned-at-reddit/

In brief, the lessons are:

  1. Crash often - automatically detect problems like excessive memory or cpu consumption or slow response times, and automatically restart the process
  2. Separate services - put different things in different processes so you can put them on different machines easily
  3. Open schema - avoid problems upgrading and replicating different database schema versions by always using one schema with two tables, a "thing" table and a "data" table
  4. Keep the app servers stateless - you can easily add more app servers, cache memory in memcached not the app servers
  5. Memcache everything - store data in memory to take load off the on-disk data stores
  6. Store redundant data - store the same data in as many formats as you may need to display
  7. Work offline - minimize the amount of work needed to generate a response for the user

What are your thoughts on these "lessons learned." Does anything jump out at you as surprising or missing?


James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Seven Lessons on Scalability from Reddit Posted: May 18, 2010 11:04 AM
Reply to this message Reply
> Open schema - avoid problems upgrading and replicating
> different database schema versions by always using one
> schema with two tables, a "thing" table and a "data"
> table

[cue: Robert Young]

John Zabroski

Posts: 272
Nickname: zbo
Registered: Jan, 2007

Re: Seven Lessons on Scalability from Reddit Posted: May 19, 2010 9:56 AM
Reply to this message Reply
Why does Artima insist on posting really dumb stuff lately?

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Seven Lessons on Scalability from Reddit Posted: May 19, 2010 10:04 AM
Reply to this message Reply
> Why does Artima insist on posting really dumb stuff lately?

Why is this dumb exactly? I don't necessarily disagree but I'm curious.

John Zabroski

Posts: 272
Nickname: zbo
Registered: Jan, 2007

Re: Seven Lessons on Scalability from Reddit Posted: May 19, 2010 11:09 AM
Reply to this message Reply
James,

I have no idea how or if reddit makes money, but these seven lessons on scalability seem to have no numbers on how the architect's decisions affected reddit's bottom line.

Generally speaking, the right way to scale isn't through ad-hoc notions like a "thing" table. If you are using persistent storage with a RDBMS and you are using this sort of schema, then you don't understand how to design data invariants.

From the presentation:

> The database gets bigger and if your database is under
> a lot of load, you can’t just add another column or add
> another table and expect it to be totally fine.

It sounds like Steve had never previously worked on a database larger than 50 gigs before, with tables over 100 million rows. There are very simple upgrade patterns to handle this. There are simply no tools baked into RAD toolsets to do this for you (at least that I am aware of), but they are extremely easy to roll. The real problem is if your inserts exceeds the rate at which you can copy over to a new table and then drop the old one. This should be a planned upgrade and not something you just do in 5 seconds, like you might do on a database with under 50 gigs in size. What effect does this have on reddit's bottom line? What was reddit's release schedules like? From the presentation, it sounds like they had a hard time building test builds and probably even released code to live servers without testing.

There are other examples from this presentation that are shockingly bad.

Bill,

Stop posting this garbage and putting "What do you think?" questions at the end. Nobody will learn anything from Reddit, the only two people who believed Paul Graham's essays on Lisp on the web. It is not really surprising, given this presentation, why their Lisp-based engine failed; they are simply bad programmers.

John Zabroski

Posts: 272
Nickname: zbo
Registered: Jan, 2007

Re: Seven Lessons on Scalability from Reddit Posted: May 19, 2010 11:43 AM
Reply to this message Reply
Let me clarify for a second.

Just about ANY "how we pulled it off" presentation from ANY startup programmer is going to be shortsighted and filled with hackish advice.

You clearly cannot learn much from these presentations, for that very reason. Statements like the one I quoted above from the presentation just demonstrate that. Usually startups have very poor timetracking on how much time they spent on something, so all we get here is they spent a lot of time on database design, and that ultimately that time was misspent (in their eyes) because all they had to do instead was a set of hacks.

So the set of hacks really isn't a valuable lesson. Instead it shows you how much the programmers didn't know going into the venture, and just how lucky they were Conde Nast bought them out.

This presentation would've been better titled Luck - The Entoxicating Underbelly of Our Success.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Seven Lessons on Scalability from Reddit Posted: May 19, 2010 11:46 AM
Reply to this message Reply
> This
> should be a planned upgrade and not something you just do
> in 5 seconds, like you might do on a database with under
> 50 gigs in size. What effect does this have on reddit's
> bottom line? What was reddit's release schedules like?
> From the presentation, it sounds like they had a hard
> d time building test builds and probably even released
> code to live servers without testing.

Well, I think there are a couple of interesting things here. They are probably not what the presenter thinks is interesting:

1. One more example (out of many thousands) of how some green developers made a bunch of mistakes that became the backbone of system.

2. Another example of how technical acumen and success in software is not correlated (or only slightly so.)

I'm not trying to be mean with these but it's often assumed by the uninitiated that if software is successful, it's well made. There are two sides to this. Just being good at development isn't enough. You need some luck and a good idea. The other side is that just because someone squeaks out a win doesn't mean you should emulate their design.

> Stop posting this garbage and putting "What do you think?"
> questions at the end.

Maybe that's too harsh. I think it's a good topic for discussion. I'm apparently the only one though.

John Zabroski

Posts: 272
Nickname: zbo
Registered: Jan, 2007

Re: Seven Lessons on Scalability from Reddit Posted: May 19, 2010 12:03 PM
Reply to this message Reply
I'm not being mean.

I'm being demanding.

You and Bill should both do yourself a favor and read the logo for the site: "artima developer: BEST PRACTICES IN ENTERPRISE SOFTWARE DEVELOPMENT".

What does this post have anything at all to do with what this site is supposed to be about?

I think it is a waste of my time to be directed to links to dumb presentations. If it keeps up, I will vote with my feet, as I suspect others will.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Seven Lessons on Scalability from Reddit Posted: May 19, 2010 12:04 PM
Reply to this message Reply
One question I had was why you would stick with a relational database if you aren't using it's relational features. I found elsewhere that Reddit has dropped memcached and now uses the non-relational Casandra DB.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Seven Lessons on Scalability from Reddit Posted: May 19, 2010 12:12 PM
Reply to this message Reply
> I'm not being mean.

Sorry, I didn't mean to imply that you were. I was only talking about myself when I use the term. I will note that you seem a bit more angry than I would expect over an article on a free website, though.

Personally, I often prefer the discussions to the articles because of people such as yourself.

What artima has going for it is the community. So if posting stuff like this would result in losing you, I think Bill should be concerned. I had an experience recently that suggests to me that artima doesn't understand that the community is more important than the authors and articles. Not enough to make me leave but it's a start.

John Zabroski

Posts: 272
Nickname: zbo
Registered: Jan, 2007

Re: Seven Lessons on Scalability from Reddit Posted: May 19, 2010 1:00 PM
Reply to this message Reply
> One question I had was why you would stick with a
> relational database if you aren't using it's relational
> features. I found elsewhere that Reddit has dropped
> memcached and now uses the non-relational Casandra DB.

So WHY did they do that?

How do they model performance tradeoffs, or are they just like the NFL - copycating what the super bowl team appears to be doing for success.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Seven Lessons on Scalability from Reddit Posted: May 19, 2010 3:48 PM
Reply to this message Reply
> > One question I had was why you would stick with a
> > relational database if you aren't using it's relational
> > features. I found elsewhere that Reddit has dropped
> > memcached and now uses the non-relational Casandra DB.
>
> So WHY did they do that?

Sorry, which 'that'? Using a relation DB in the first place or switching to Casandra?

> How do they model performance tradeoffs, or are they just
> like the NFL - copycating what the super bowl team appears
> to be doing for success.

I can't really speak to it. I try not to assume I understand the requirements of others. I've experienced too many 'aha' moments when it gets down to details. Since I don't work in this kind of situation, I won't pretend to know what's best for them.

Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Re: Seven Lessons on Scalability from Reddit Posted: May 19, 2010 4:09 PM
Reply to this message Reply
Hi John,

> Bill,
>
> Stop posting this garbage and putting "What do you think?"
> questions at the end. Nobody will learn anything from
> Reddit, the only two people who believed Paul Graham's
> essays on Lisp on the web. It is not really surprising,
> given this presentation, why their Lisp-based engine
> failed; they are simply bad programmers.
>
Sorry you didn't like the presentation, but please keep in mind that the discussion here is a big part of the educational value we try to provide. The reason I posted it is that I was surprised by some of the things in the presentation, and wanted to hear what our audience had to say about it.

As far as the thing table goes, that sounded like the "no-sql" movement to me. It seems some high-profile, high-traffic websites have been going away from sql towards big-hash-table data stores. I think that's a good topic for conversation here. I myself was surprised that the Reddit folks felt they were spending too much time making data models, because data model design seems like it should be straightforward to me. But the other reason they went away from sql, which was interesting to me, was that they had trouble updating schemas on their database while it was in use. We have never had that problem, but we don't have such a high-volume site. But a lot of people who do have high volume seem to get along fine with sql databases. So I thought that would be a good issue to discuss here too.

I also thought it was interesting that they had deamon's automatically restart things when they grew too large. I've seen that before, and as I think it is a great technique. And he said to watch the logs, so that you find out when there probably is a memory leak or other problem so you can actually fix it. That all made sense to me.

His points on storing redundant data and doing as much work offline as possible made sense to me too, the goal being make the user wait as little as possible. I thought that was useful.

It sounded like they never thought about clustering until after they needed to cluster, so they had state in their app and no way to share it between app servers. That seemed short sighted to me, but on the other hand, since most ventures don't succeed anyway, maybe it's best to throw something out there quickly and worry about clustering it later if you have enough success. Still, I'd lean towards doing a two-app server cluster at the beginning.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Seven Lessons on Scalability from Reddit Posted: May 19, 2010 4:43 PM
Reply to this message Reply
> I also thought it was interesting that they had deamon's
> automatically restart things when they grew too large.
> I've seen that before, and as I think it is a great
> technique. And he said to watch the logs, so that you find
> out when there probably is a memory leak or other problem
> so you can actually fix it. That all made sense to me.

It seems like pretty obvious solution if your app server keeps crashing. Ideally, you'd resolve the root cause but until then ... Even better, if you have multiple servers then you do this an the users are hardly affected if at all.

> It sounded like they never thought about clustering until
> after they needed to cluster, so they had state in their
> app and no way to share it between app servers. That
> seemed short sighted to me, but on the other hand, since
> most ventures don't succeed anyway, maybe it's best to
> throw something out there quickly and worry about
> clustering it later if you have enough success. Still, I'd
> lean towards doing a two-app server cluster at the
> beginning.

I'd start with statelessness. Too many websites are unnecessarily stateful.

Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Re: Seven Lessons on Scalability from Reddit Posted: May 19, 2010 4:55 PM
Reply to this message Reply
Hi James,

> > It sounded like they never thought about clustering
> until
> > after they needed to cluster, so they had state in
> their
> > app and no way to share it between app servers. That
> > seemed short sighted to me, but on the other hand,
> since
> > most ventures don't succeed anyway, maybe it's best to
> > throw something out there quickly and worry about
> > clustering it later if you have enough success. Still,
> I'd
> > lean towards doing a two-app server cluster at the
> > beginning.
>
> I'd start with statelessness. Too many websites are
> unnecessarily stateful.
>
Well there's going to be state somewhere. Do you mean you'd keep all state in the database initially?

Flat View: This topic has 51 replies on 4 pages [ 1  2  3  4 | » ]
Topic: Hiring as a Core Competence Previous Topic   Next Topic Topic: Agile Hiring by Sean Landis PrePrint Released


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us