LiveJournal Architecture

Greg Linden has an interesting post about a presentation on the LiveJournal architecture.

He says:

In particular, there seems to be a heavier emphasis on caching layers and read-only databases than I would expect. I tend to prefer aggressive partitioning, enough partitioning to get each database working set small enough to easily fit in memory.

I know little about LiveJournal’s particular data characteristics, but I wonder if aggressive partitioning in the database layer might yield performance as high as a caching layer without the complexity of managing the cache consistency. Databases with data sets that fit in memory can be as fast as in-memory caching layers.

Likewise, I wonder if there would be benefit from dropping the read-only databases in favor of partitioned databases. With partitioned databases, the databases may be able to fit the data they do have entirely in memory; read-only replicas may still be hitting disk if the data is large.

Hitting disk is the ultimate performance killer. Developers often try to avoid database accesses because their database accesses hit disk and are dog slow. But, if you can make your database accesses not hit disk, then they can be blazingly fast, so fast that separate layers even might become unnecessary.

My experience with replication has been very mixed. The key is understanding that the slave will have to support the same write load as the master AND the read load. That really is key.

In you can afford it, it makes much more sense to partition the data across a number of machines and use replication as a backup should the primary fail. Creating lots of little piles of data is a much better way to scale than creating a few large piles.


One Response to LiveJournal Architecture

  1. Pingback: Talking About MySQL « François Schiettecatte’s Blog

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: