The Artima Developer Community
Sponsored Link

Artima Developer Spotlight Forum
Seven Lessons on Scalability from Reddit

51 replies on 4 pages. Most recent reply: Jun 29, 2010 8:15 PM by John Zabroski

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 51 replies on 4 pages [ « | 1 2 3 4 | » ]
nes

Posts: 137
Nickname: nn
Registered: Jul, 2004

Re: Seven Lessons on Scalability from Reddit Posted: May 21, 2010 11:34 AM
Reply to this message Reply
Advertisement
I take advice from the Reddit guys over academic assertions about things without any concrete evidence any day. The Reddit guys (pretty or not) have a site that is working today. Furthermore we have at least three accounts of social web sites that have switched from SQL to non-SQL databases. I will believe that SQL databases are up to task when somebody tells me how they are successfully running a popular social web site long-term using a SQL database.

Mike Swaim

Posts: 13
Nickname: swami
Registered: Apr, 2004

Re: Seven Lessons on Scalability from Reddit Posted: May 21, 2010 2:42 PM
Reply to this message Reply
Actually, the complaint seems to be against MySQL, rather than SQL databases in general. Does anyone know of a site that tried PostGres or one of the commercial venders (MS, Oracle, Sybase) and decided to go NoSQL?
In any case, I've always heard that heirarchical databases are faster, assuming that you've got heirarchical data. We have a heirarchical DB at work (dating back to the 60s), and nonparent foreign keys are stored as strings, which isn't very efficient.

John Zabroski

Posts: 272
Nickname: zbo
Registered: Jan, 2007

Re: Seven Lessons on Scalability from Reddit Posted: May 21, 2010 3:17 PM
Reply to this message Reply
Mike,

Keys stored as strings and hierarchical database from the 60s indicates to me that you work in medical services industry.

The reason hierarchical databases are "Faster" has to do with transportation of records. Storing everything inline in a single row has very good locality of reference characteristics. These systems tend to be very slow at certain end-of-month closing reports that pull data from many types of objects in the system. On the other hand, when you are trying to meet Health Leven Seven requirements, strong locality of reference can be important: you really want strong real-time guarantees on that data feed that can control whether doctors and nurses respond in time when some life support equipment is causing and/or reporting a patient's kidneys to fail. Typically, what I see in most hospitals we've worked with is that the corporate staff are the ones last thought about when selecting a software system to run the hospital. The doctors and nurses needs simply trump things like billing. Nobody who uses these systems even asks good data mining questions, because they can't.

The relational model aims to solve both those issues, by ideally allowing you to separate physical storage concerns such as locality of reference from logical storage concerns such as coming up with a data model that represents all the needs of your enterprise in such a way as to minimize queries that don't obey all the data invariants.

Object-oriented databases solve this problem by ignoring it; it assumes that all data invariants are covered by problem domain invariants and fulfillment of behavior responsibilities of every object in the system. Whether you actually persist in an OO database or a RDBMS database is largely irrelevant, since there is no actual "enterprise" to speak of. In this way, the OO DBMS is fairly different from a hierarchical DBMS such as one you might purchase from Epic or Meditech.

John Zabroski

Posts: 272
Nickname: zbo
Registered: Jan, 2007

Re: Seven Lessons on Scalability from Reddit Posted: May 21, 2010 3:32 PM
Reply to this message Reply
> I take advice from the Reddit guys over academic
> assertions about things without any concrete evidence any
> day. The Reddit guys (pretty or not) have a site that is
> working today. Furthermore we have at least three accounts
> of social web sites that have switched from SQL to non-SQL
> databases. I will believe that SQL databases are up to
> task when somebody tells me how they are successfully
> running a popular social web site long-term using a SQL
> database.

nes,

I've read your posts in the past and I understand where you are coming from. -- There are also way more than three social networking web sites using Cassandra. Like the NFL, dotCOM entrepreneurs are copycats and do what everyone else seems to be doing. There are very few innovative minds. There is a morsel of truth that: It is better to pursue a religion than to seek after truth for oneself and end up empty.

However, there are a lot of things to consider here, including what it will take for RDBMSes to compete in the social networking web services market as the data storage layer.

I personally don't think writing SQL directly, in 2010, makes much sense, if any. Tools should not force developers to deal with tools like CassandraDB directly. Some people realize this... there are even memcached storage engine projects for MySQL to allow for staged storage.

Regardless of tools, the techniques remain the same. People will add more and more layers of caching to squeeze out differences and overcome the cost of faster, volatile memory technologies over slower, non-volatile memory technologies. It's important to realize this is a performance model and you can predict how well it will scale. Reddit should've been able to predict this when they pursued a strategy of scaling their memcached implementation by adding tons of RAM.

Whether MySQL is up to the task for Reddit is an orthogonal question. The bigger issue here is that a site serving 7.5 million users has no real methodology for how they go about improving its QoS. Instead, it is exposed as a copycat.

Krisztian Sinka

Posts: 30
Nickname: skrisz
Registered: Mar, 2009

Re: Seven Lessons on Scalability from Reddit Posted: May 21, 2010 4:06 PM
Reply to this message Reply
> If you don't know what circumstances led to the rules,
> then it is just like magic - do this and be happy. Also it
> is dangerous - because someone might think he should not
> only read these rules but really apply them! If the
> conditions of his project differ this can lead to a
> disaster.

I agree. In the video one key sentence is:
“...it just works for us...”

It really resembles me to this Artima article (and the book) - The Management Myth - http://www.artima.com/weblogs/viewpost.jsp?thread=282344 but for now it is for software.

I think it is not so wise to suggest something to others that is not proven or thought out but just “found” for a particular case.

A better title would be How did you manage? not What had been learnt?.

Wolfgang Lipp

Posts: 17
Nickname: wolf2005
Registered: Sep, 2005

Re: Seven Lessons on Scalability from Reddit Posted: May 21, 2010 5:44 PM
Reply to this message Reply
i find it befuddling how much controversy is aired in the first few posts in this forum. i found the video great and watched it through. to me the majority of steve’s points are very reasonable, and from my own research i have come to much the same conclusions: like threads are bad, seek to cleanly separate stateless from stateful elements, largely abandon databases with complex relational schema; i independently also hit on the same tools like memcached, rabbitmq and so on.

@mr zabroski, you’ve come down so hard on both the poster and the reddit guy, so take this: you’re so boring. you’re so shareholder value. so corporate. so numbers. you maybe even right. but so numbers.

look this steve huffman guy is so proud of their work, and reddit just works as far as i can tell, and a lot of their decision make a lot of sense to me. y’know after being back from an environment where everyone was into those relational table forests, codebases partly written out as stored procedures/UDFs inside the db, huge xml schemas, all that stuff, all the redpills and koolaids of yesteryear. and yes: xml is bad. shudder. so bad. so corporate. stored procedures are not even faster than doing it in the app. everybody knows how to stop seconds of cpu time, but who has numbers for programmer happiness?

about adding columns on big tables you say """I really don't know how on earth that could be a hard problem to solve""", and i agree insofar as i do not see a problem at least as long as a database lives on one box, with you as the single person on the team, and your schema having up to five tables. go beyond that, and the process stops scaling all that swell, for a lot of soft reasons. what i find really difficult is the conceptual overhead of re-scheming. the more convoluted a relational schema gets, the more likely you are to blow everything up when you touch the schema. in an open-schema / flat database basically what you do is add more data, more facts into the database, which gives your application software more opportunities to report data and do stuff with it. at least that’s the theory. so it is really a different way to organize things.

in a later post, you say """What most programmers lack is insight, or data mining, on how they design systems""", which is probably true. not only in software, computers in general are very lacking in diagnostics, and that includes scientific statements about code performance, but doesn’t stop there. when i have e new box and a new broadband internet access and it doesn’t work, there are a million things that could all have gone wrong, but where are the tools to narrow down the point of failure? i know of none an inexperienced person could use.

i do not really doubt the other points you mention in that posting of yours, either. but i can tell you i am not going to move from python to java just because the figures say so. as programmers, we do not just have to build the fastest car in town, we also want to build machines we like with tools we like.

sure steve didn’t mention any figures but then maybe this was never intended to be another executive level shareholder meeting speech either. he did mention one managerial (in the best sense) thing, tho, and that was (paraphrased) ‘better to just kill a misbehaving process and look after reasons later; rather add another box and do some computation that happens never to be used; rather add some more memory than have customers wait and leave in disappointment’.

that is very valuable and pragmatic. i mean, reddit is still alive, right, and as long as the bottom line is alright, why not have a tool that works, that you like to use, and which is maybe a little on the sluggish side, as long as you can make up by shelling out the bucks to add a little memory or whatnot? within limits, this is a sound approach.

and sure they started out with lisp. i hate that language so i would’ve told them right from the start it would be a bad idea, but then i remember to have fallen for some hype countless times before in my life. long into the rails hype a friend and me opted to give an application a chance which was built with RoR; i never want to touch RoR again. others feel fine with it.

"""[...] spaghetti coding languages like PHP3, .e.g. look at Drupal and Joomla as two examples"""—thanks for mentioning all of php, drupal, joomla and spaghetti in one sentence. you are afraid that steve’s speech could infect unknown college brains, but i’d rather assume a majority of those is already infected with the spaghetti virus so almost any competitor for neural resources in that area is just welcome.

@bill venners: """they never thought about clustering until after they needed to cluster [which] seemed short sighted to me [...] maybe it's best to throw something out there quickly and worry about clustering it later if you have enough success. Still, I'd lean towards doing a two-app server cluster at the beginning."""—i couldn’t agree more.

i think one lesson from agile programming is that you just cannot plan for everything up front. personally, i like to keep concerns like scalability, clustering and so on ‘in mind’ and try not to build too many assumptions into my code that would cause conflicts the day you start to cluster. and i really really love the idea of the ‘minimal cluster’ which is why i am running several machines in dev, some of them real, some virtual.

traditional corporations are used to doing a lot of upfront planning and producing design-by-committee documents, but i wonder how many things got re-written in a place like google when they moved from one box to the untold numbers of data centers they’re running now.

John Zabroski

Posts: 272
Nickname: zbo
Registered: Jan, 2007

Re: Seven Lessons on Scalability from Reddit Posted: May 21, 2010 10:43 PM
Reply to this message Reply
Wolfgang,

- I don't like design by committee. There are very few good designers in the world, and putting many mediocre designers on the same team does not make up for it. More often than not, each designer comes from a different division in corporate and is seeking to protect their own feifdom, making them behave even dumber than they actually are. That's unfortunate, since mediocre designers are precisely the ones who cannot afford political wars. Great designers, in my experience, simply find ways to ignore the rules and deal with the consequences later.

- I am a statistician by training, and approach problems mathematically, so being "so numbers" is a good thing. Math wins.

I also don't like writing stored procedure code, user defined functions or other SQL caca. I am not sure what gives you the impression I think that is a good idea. I don't really know what kind of code you like to write. It's evidently not Lisp, not Java, and its likely not SQL or any XML language like XQuery or XSLT. It's maybe Python, without threads?

Most of our software is driven by a relational algebra model; SQL is merely the backend target for a model compiler. You should be able to sink and hoist SQL concepts like UDFs and SQL calculated columns, stored procedures, parameterized queries, etc. You should also have tools that automatically tell you how long a query is taking, from the perspective of the application server. You should then be able to orthogonally create optimizations using a copy-on-write peephole optimizer that rollback optimizations when the specific query has changed. None of this stuff is done today by most, primarily because how our software systems premature or "early binding". Having an open-schema is really just a hack around the fact that ORM-centric systems like LINQ require early binding or hard to maintain text strings. We don't require our developers to use either of those baroque tools, since they are both proven ineffective; every developer who can feel pain hates them, just as you seem to. Just as you want eventual consistency in your data, I want code to be eventually consistent with an online data dictionary, and to partially fail if the online data dictionary is not consistent with the code. Stuff like read-repair supported by CassandraDB is really just a form of partial failure; look at it that way! [Theory+practice note: In programming language theory terms, this means that your type system is more easily rooted in a Curry-style effect system (where the meaning of a term doesn't depend on type), rather than a Church-style effect system (where the effects depend on the type), and the overhead of keeping track of all those types would be a big headache.] If you TRIED building software our way using traditional ORM-centric toolsets, then you would die from typing too much. Do you have any idea how much memory an AppDomain in .NET takes up when you naively load 100,000 different DAO/DTO classes? In an unmodified JVM? (The same question comes up when you have many prototypes in a prototype-based language, and want a linear performance curve and not have to microcode leaf prototypes). ORM-centric tools just aren't built for extremely late-bound faceted materialization. I once watched a BOOST tutorial on the C++ BOOST database interface tool, and everyone on the BOOST mailing list basically agreed there was not much need for metaprogramming in an ORM. When everyone is in agreement on something, it is an opportune time to dissent. Who knows, dissenting might be right.

As for using the JVM vs. Python: Python is sort of stupid and ugly (I hate it), and a much nicer and funner to use system is Erlang. Erlang has a lot of cool features like on-the-fly compression for compact distributed communications, thereby minimizing linking overheads. Serialization in CLR and JVM is way more convoluted. Likewise, Erlang's notion of processes are way more lighter than, say, CLR's notion of AppDomain, which is in some ways lighter than the JVM comparable features.

Morgan,

You need a reliable build process. Without a reliable build process, you cannot increase quality. Your concerns over how to increase performance while the schema is constantly shifting is noted. In any RDBMS, constantly screwing up indexes will alter the statistics catalog and influence the Cost Based Optimizer, so you may not be getting true statistics, ever.

If you are limited to one prototype machine shared among many developers, and have no way to setup a real test environment, then that stinks. If you are dependent on a live copy of "valuable data", then that stinks too. I can't imagine why valuable, rare data would be stored in a database constantly undergoing schema changes. Something sounds horribly wrong. I would recommend treating the build process itself as a problem, and modeling the problem domain using OOA, and breaking up the problem so that invariants are encoded into class responsibilities. This might seem obtuse, but it will really help you to see where your build process is going wrong. For example, Ruby on Rails uses Active Record precisely because they treat every database table as an object. Logical changes to the schema are THEREFORE driven by logical changes to the object model, which will ONLY happen if the specification changes the OO Analysis, because it will require class collaborations to be redefined, thus changing all data invariants imposed in the RDBMS used strictly to serialize the OO app state. Do you see what I am saying? I am showing you how the problem here must be methodological, otherwise it makes little sense for the schema to constantly be changing among many developers. I could make the same argument for a Data Mapper where the object model is automagically refreshed when the schema changes.

Kay Schluehr

Posts: 302
Nickname: schluehk
Registered: Jan, 2005

Re: Seven Lessons on Scalability from Reddit Posted: May 22, 2010 1:06 AM
Reply to this message Reply
> > > You know, a REAL engineering
> > > discussion. Not a superfluous list.
> >
> > When you'd expose your companies database architecture
> on
> > the level of detail that measurements can be reproduced
> > you'd probably get fired but we surely had a better
> > discussion ;)
>
> Uh...
>
> I don't get it.
>
> These social networking sites are all using CassandraDB
> now anyway. How on earth can you possibly tell me that
> BEFORE vs. AFTER benchmarks are proprietary?

What? A benchmark is a normed test-sample used to create reproduceable performance measurements. Why on earth shall this poor reddit guy create a benchmark for CassandraDB, not to say MySQL? He just used his companies data and publishing measurements on those data without publishing those data is no better than story telling.

Of course having good benchmarks would have merits for those data storage developers just as their users but creating them would be a project on its own. Maybe you can grab the next CS prof who has diploma students who can work those things out.

Kay Schluehr

Posts: 302
Nickname: schluehk
Registered: Jan, 2005

Re: Seven Lessons on Scalability from Reddit Posted: May 22, 2010 2:17 AM
Reply to this message Reply
> I think it is not so wise to suggest something to others
> that is not proven or thought out but just “found” for a
> particular case.

A behaviourist just had sex with his girlfriend. He says: "For you it was beautiful, but how was it for me?"

Sometimes it makes sense to put assertions into proportion. Saying "it works for me" means that it could possibly not work for you. Not sure how this can be misread?

Wolfgang Lipp

Posts: 17
Nickname: wolf2005
Registered: Sep, 2005

Re: Seven Lessons on Scalability from Reddit Posted: May 22, 2010 9:12 AM
Reply to this message Reply
@john """SQL caca"""—well said. at the time of this writing, i am still fighting my way through your assertions, which seem both classy and informed to me. i actually gae up on ORMs and am back to writing SQL strings, manually, for the better or the worse. just because you see as for ORMs """every developer who can feel pain hates them""" and i sure do. might change in the future, i think DBs *should* be transparent but somehow ORMs are just too, well, OOP these days i feel.

"""Python is sort of stupid and ugly [...] a much nicer and funner to use system is Erlang. Erlang has [...] on-the-fly compression for compact distributed communications"""—i once met a guy from rigel 4 who had an anti-gravity dual multi-responder gizmo with him. omg. we had a lot of fun, but then, he was definitely from another planet.

"""It's maybe Python, without threads?"""—yeah that about hits the head on the nail. threads i have found to be so 70s, you can use a thread in a controlled fashion, but otherwise i dream up cooperative concurrency plus extra processes to be so much cooler, and no, i have no figures. iknowiknowiknow.

there are many things about computing, and i’m currently very much on the wording camp, meaning the ole confucian advice to 用正名 ‘use the right words’ seems all-important to me. performance is one thing, and paramount, but there is also an obligation to put things together in a way that allows others to disassemble and re-assemble the machinery. and read enjyoable code, too.

""" I can't imagine why valuable, rare data would be stored in a database constantly undergoing schema changes. Something sounds horribly wrong."""—my best guess is that the way that a gang of hip start-uppers work is like they have an idea, go thrash a webjob, gather a user base in the thousands, then realize shortcomings, mend that, win more users, realize defieciencies, mend that, count millions of users, rewrite, add features, collect more data, add features, mend the schema again, you know, that way of working. their valuable data is a work in progress, not a conscription set in stone. they may have a hazy grand vision of things, but sure no finalized schema of data lying around. which is fine. i mean it’s perhaps not scientifically sound, but then it’s great it’s possible at all. you sure can’t mass-produce cars and planes that way, but then hey this is the web. hope they have fun, nothing wrong with that.

Morgan Conrad

Posts: 307
Nickname: miata71
Registered: Mar, 2006

Re: Seven Lessons on Scalability from Reddit Posted: May 22, 2010 2:24 PM
Reply to this message Reply
John,

I did a poor job explaining our system. The "real, hard to obtain" raw data was huge (close to a terrabyte at the time, it's growing!) biological data stored in a large number of files. Generally text format, and more or less organized by a directory hierarchy. The analysis took place on these files, in multiple steps. The analysis algorithms took a long time and we worked hard to speed them up. The speed of DB access was, in general, trivial by comparison.

The RDB largely stored what some would call meta-data - information about workflow: which of the large text files belong together, the order in which they should be analyzed, what algorithm to use, and the settings for the analysis, the status of the analysis (e.g. not-started, failure, completed), and links to results files. Users could view the status and control many of the settings via a web based interface. Hence the Postgress DB talking to JSF (or maybe JSP, I always mix them up).

The rules for the order of analysis, the possible algorithms and settings, and general file organization were changing frequently (a few times a month). So the metadata schema changed frequently. Occasionally the raw file format changed, or more files/directories were added.

Since I was writing the workflow daemon that worked closely with the DB, core support classes for the web services, as well as helping with many of the analysis algorithms, the frequent DB schema changes were annoying. Even though Hibernate shielded me from the nitty-gritty details, as metadata changed, the ability to repeat a canned analysis or canned "unit test" (yes, many would not call a test that actually used the DB a unit test, hence the quotes) from the day before was problematic. IMO, using a more "flexible" DB like an OODB or big hashtably thingy would have been preferable early in the project. Once the schema was largely stable, then an RDB would be more appropriate.

John Zabroski

Posts: 272
Nickname: zbo
Registered: Jan, 2007

Re: Seven Lessons on Scalability from Reddit Posted: May 22, 2010 4:40 PM
Reply to this message Reply
Morgan,

Thanks for the extra details.

I still think the problem is methodological, and see no reason for many schema changes. I've used a relational database in the past for storing configuration information, and know many programmers who like to (ab)use RDBMSes in such ways. For example, Microsoft has an internal tool called SNAP (Shiny New Automation Process) that handles their build process for them by building on every checkin, and testing all code against all unit tests, before the testers get their hands on it. SNAP depends on SQL Server to work. (I say abuse, because I know message queues would probably make more sense, but I don't really dislike the solution, so long as it works.)

Let me clarify:

- I don't think the schema should change if the location of files change. General file organization changing means you need to model the invariants about file organization and store the volatile information in a data structure. You then model what you plan to DO with files in a particular file set. If necessary, make this recursively defined.

- If you have rules for ordering an analysis, then you need to model the Sequences and Stages of that analysis. If you don't explicitly model this, then your unit tests will get harder and harder to write. I recently went over this on comp.object with somebody, explaining why it is important. See: http://groups.google.com/group/comp.object/browse_thread/thread/6a16b14ee9ae4f6f#

- If possible algorithms change, then this is not an issue. If the structure of the outputs change as the algorithms change, then you have a problem.

- If settings are volatile, then you must make sure that your algorithms do not depend on several layers of static and dynamic settings information. The algorithms themselves should not "Reach up" and grab the settings, because that will lead to tight coupling; the settings will in effect control the selection of the algorithm, rather than the settings merely supplying data packets to the algorithm. The settings themselves should have no knowledge of which algorithms they pertain to.

So, in summary, I think you can make your build process more reliable while retaining your use of the RDBMS as-is, you don't need to gold plate the solution with a message queue, and you can probably do better simply by using conventions that everyone sticks to to keep everyone's life easier.

nes

Posts: 137
Nickname: nn
Registered: Jul, 2004

Re: Seven Lessons on Scalability from Reddit Posted: May 24, 2010 12:12 PM
Reply to this message Reply
I would also always start with a good SQL-database if I was starting anything new. That said the big advantage of SQL databases are also its disadvantages. It allows good structure and very flexible reporting. But the two things they are not:
1.Faster than specialized data storage: Memcache and other persistent key-value stores for example. And yes, premature optimization and all that, but when you have speed issues it is only natural to look in that direction.

2.Flexible when trying to store arbitrary data:
When you are working in a dynamic language like Python that doesn't require builds, rebuilding the database slows you down, especially if it is already heavily populated.

Ideally you would have a pretty good idea of what your data looks like but maybe not when you are two college students throwing something random on the web to see what sticks. There is something to be said about organically growing a system and thinking only about the now. You can lower business risks by increasing technical risks. Two young programmers would probably prefer challenges of the later type.

Vardan Akopian

Posts: 1
Nickname: vakopian
Registered: May, 2010

Re: Seven Lessons on Scalability from Reddit Posted: May 24, 2010 8:08 PM
Reply to this message Reply
> You need a reliable build process. Without a reliable build process, you cannot increase quality.

John, do you happen to have some references at hand for a "reliable build process" that you would recommend? I find this extremely important, yet information on it is very scarce. So I would really appreciate your advice (links, books, existing software, ...)

Thanks

John Zabroski

Posts: 272
Nickname: zbo
Registered: Jan, 2007

Re: Seven Lessons on Scalability from Reddit Posted: May 25, 2010 11:09 AM
Reply to this message Reply
I'll agree with you that information is either scarce or poorly organized. I've had several 'buildmasters' tell me that they learned their craft mostly by picking up wisps of wisdom via chinese whispers, usually by being subscribed to a mailing list like help-make, user@ant.apache.org, ivy-user@ant.apache.org, or users@maven.apache.org, etc.

I have a number of books on software configuration management, product configuration management (the SCM analagous idea for hardware systems), version control, revision control and automated build processes. I'll see if I can come up with an annotated bibliography in a few days -- but Memorial Day weekend is approaching and I want to get household chores done before, so I can dedicate the weekend to pure relaxation.

Flat View: This topic has 51 replies on 4 pages [ « | 1  2  3  4 | » ]
Topic: Hiring as a Core Competence Previous Topic   Next Topic Topic: Agile Hiring by Sean Landis PrePrint Released


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us