Scaling and traditional databases

By way of Greg Linden, a set of reading references from Werner Vogels.

I had the pleasure of sharing a table with Werner Vogels at lunch at the Google Seattle conference on scalability and got a chance to talk briefly with him. One of the things we talked about was how traditional dababase management systems (RDBMSs) had a hard time scaling to accomdate the date sizes that enterprises like Amazon, Google and eBay were dealing with. The point was made that traditional RDBMSs tended to lump everything together and achieved scale through ever bigger machines which was not sustainable in the long run.

I made a few points though: one was that it was probably not economical for traditional dababase vendors to focus on the really big market because it was pretty small and the costs clients of such system would be prohibitive; the other was that these clients were going to need fairly customized solutions, so a generic RBDMS was probably not going to cut it; finally there were some people (such as Wikipedia & YouTube) who were using RBDMSs (mysql in this case) and scaling pretty well provided they used the RDBMS as a table server and segmented their data.

On that last point, if you watch the video of the YouTube session at that conference, you will see a fairly habitual scaling story. Initially mysql works well but does not scale, requiring the application of all sorts of incremental tweaks to get 5%-10% more every month, until the solution of segmenting the data is implemented.

Updated July 10th, 2007 – I just finished reading the interview of Michael Stonebraker by Margo Seltzer mentioned by Werner Vogels, very readable assessment of the state of traditional dababase management systems, strongly recommended.


Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: