Scaling and Uptime

I just came across this updated post on scaling by Greg Linden.

First he has a reference to an old article “Don’t Scale”. The article makes the (flawed) argument that scaling does not matter. It also confuses uptime and scaling, the two are very different.

There is a hardware aspect and a software aspect to uptime. The hardware aspect relates to the redundancy in your hardware architecture and the software aspect relates to the redundancy in your software architecture. Getting perfect uptime is really hard, especially when many things out of your control can go wrong. Getting all those nines after the decimal point eventually mean that you can only be down for minutes per years, clearly difficult to achieve.

Scaling has to do with how your system responds to increased usage, again there is a hardware aspect and a software aspect to this. A system will scale well from a hardware aspect if all you need to do is bring more boxes online when usage increases. Similarly, a system will scale well from a software aspect if you don’t need to go through wrenching software changes as your usage increases. Scaling hardware is relatively easy to achieve because the bandwidth and latency of hardware is well understood and easy to measure. Scaling software is more difficult to achieve since it can be difficult to figure out ahead of time where bottlenecks will occur and software behavior is more difficult to predict than hardware behavior. This article by Dan Pritchett is a good primer in scaling software.

Secondly he made two comments on the article:

Stepping back for a second, a toned down version of David’s argument is clearly correct. A company should focus on users first and infrastructure second. The architecture, the software, the hardware cluster, these are just tools. They serve a purpose, to help users, and have little value on their own.

But this extreme argument that scaling and performance don’t matter is clearly wrong. People don’t like to wait and they don’t like outages. Getting what people need quickly and reliably is an important part of the user experience. Scaling does matter.

I agree that you initially need to focus on getting the application working and building up your user base, but scaling and performance really does matter and you need to think hard about those issues from the start. In my experience it is usually much easier to scale a system which was conceived with scaling in mind than a system which was just thrown together without real thought. I’ve been there, trust me, nothing makes your heart sink faster than code that has not been properly tiered, or a crappy data model, or a schema that was just thrown together, or seeing IP addresses in the code.

If you don’t build with scaling in mind from the start, you will find that you will have to throw everything away much sooner than you thought.

Greg then follows up with an except from an interview with a Twitter developer Alex Payne who says:

Twitter is the biggest Rails site on the net right now. Running on Rails has forced us to deal with scaling issues – issues that any growing site eventually contends with – far sooner than I think we would on another framework.

The common wisdom in the Rails community at this time is that scaling Rails is a matter of cost: just throw more CPUs at it. The problem is that more instances of Rails (running as part of a Mongrel cluster, in our case) means more requests to your database. At this point in time there’s no facility in Rails to talk to more than one database at a time.

The solutions to this are caching the hell out of everything and setting up multiple read-only slave databases, neither of which are quick fixes to implement. So it’s not just cost, it’s time, and time is that much more precious when people can[‘t] reach your site.

None of these scaling approaches are as fun and easy as developing for Rails. All the convenience methods and syntactical sugar that makes Rails such a pleasure for coders ends up being absolutely punishing, performance-wise. Once you hit a certain threshold of traffic, either you need to strip out all the costly neat stuff that Rails does for you (RJS, ActiveRecord, ActiveSupport, etc.) or move the slow parts of your application out of Rails, or both.

I am no expert in Ruby or Ruby on Rail, but a few things jumped out at me in this section, and this moves away from the higher level concepts and more into implementation issues.

The first was the comment about caching. Cache is king so the (corrupted) saying goes, and the more data you can cache, the less frequently you will have to go back the source for data. Data can be cached on the local file system, either in files or in a small database (Berkeleydb or SQLite being good options.) A good place for those on Linux is to use the shared memory partition (/dev/shm) which functions much like a RAM disc. Data there will usually be stored in RAM but will be swapped out if there is a memory shortage, so is only good for small amounts of data. Data on the shared memory partition will be wiped if the machine is rebooted.

There are also memory based caching systems, such as memcached, which allows for caching either on the local machine or a dedicated machine (or a batch of dedicated machines). I have used memcached before and the best configuration is to set up a batch of machines dedicated to caching, preferably with lots of RAM in each machine. The two caveats is that you much make sure that the memcached process never, ever, gets swapped out otherwise your cache performance will crash, and never rely on memcached to keep an object in cache indefinitely, objects will get aged out from the cache if space is needed for younger objects.

The second was about using multiple read-only slave databases. I used to a fan of that until Peter Zaitsev pointed out to me that this really does not work well in an environment where there is a lot of write activity because the slaves will have support both the write load and the read load, while the master will only have to support the write load (assuming you don’t want anyone reading from the master unless they have to.) This gets nastier because the temptation is make the “baddest” machine the master and to use less powerful machines for the slaves. This gets even nastier if you use MySQL (very likely) because there is only a single replication thread running on the slave which impacts the write performance. Once your slaves start to fall behind, they will never catch up.

What I am a fan of now is to dice data so that it can be spread across multiple machines. You can also slice your data into partitions too to make your individual databases and indices smaller, which will help performance. The impact of this is what you will need to move some amount of integrity checking out of your database (like triggers and constraints) into a higher layer. This is not all bad as it means that your database will be doing less work. Of course how this slices and dices depends on your application. This presentation on how eBay scales provides a very good illustration on how to do this.


5 Responses to Scaling and Uptime

  1. Greg Linden says:

    Hi, François. Thanks for the mention. Sounds like we agree on much of this.

    I also am a proponent of partitioning for scaling over only relying on replication. I do not know if you are interested, but I wrote up some thoughts on Wikipedia’s architecture a while back that suggested that they move to much heavier use of partitioning.

    Thanks again, François.

  2. Hi Greg, I remember reading the article when it came out, and it makes perfect sense, the more machines you have, the more throughput you can deliver before you become I/O bound.


  3. Pingback: Scaling Without A Database « François Schiettecatte’s Blog

  4. I’m glad you don’t buy into the ‘cache is king’ kool aide either. Caching is fine if you have a read only system and the means to support it (memory, slave DBs, etc.). I got no more than two sentences into this post and immediately thought about the Twitter issue and I was glad to see that you mentioned it later on.

    While I think it is bad to over architect an application and try to make it the ultimate scaling solution from the get go, you should have some idea of intended usage (is it going to be a public facing app like Twitter or a department level app?) and proceed such that you can scale it later or at least easily refactor it to perform or scale when the time arrives.

    The eBay architecture also came to mind, and which you mentioned as well. I highly doubt that they built it that way from day one. I wish I could work on an app that needed that kind of architecture. I think it would be great fun and good challenge.

  5. Cache it not everything and you need to know what to cache. Clearly there is no point in caching fast changing data, or having a cache so small that data barely has time to live in it before being aged out. A good example that comes to mind if the MySQL query cache which is worse than useless if you have fast changing data.

    I think we agree on the architecting issue, over-architecting is most likely counter productive at the start when your focus is to get the application out of the door. As the application matures, good architecture becomes more important, so that scaling is sustained and (relatively) painless.

    Slides 8 through 13 of the presentation I refer to has a timeline of the various architectures eBay had in place, very interesting reading.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: