Memcached, again…

The High Scalability blog has a good summary of a presentation given by Farhan Mashraqi of Fotolog.

As I have written before, I am really, really, really ambivalent about using memcached to cache data coming out of MySQL.

Fotolog has 51 instances of memcached on 21 servers with 175G in use and 254G available.

I can’t help but wonder how MySQL would perform if given 21 extra servers with all that memory.

They also mention MySQL’s cache, my advice on that is don’t even bother with it, it is worse than useless.

One place where I was interested in their use of memcached is in caching filesystems accessed via NFS. Again I have to ask whether they are solving the wrong issue. It is a bit like saying “my car is slow therefore I will buy a faster car to tow it with, and my slow car will now go faster.” The real solution is to see why your car is slow in the first place, and then trade it in for a faster one if need be.

Installers ahoy

I am wrapping up work on the installer I started working on over the weekend.

The goal of the installer is to have a single script I can run on a pristine machine (or image), tell it what I wanted installed (crawler, api, indexer, etc…) and boom! five minutes later I have a fully installed, fully configured instance. Or course the best thing to do is to create an instance image which I can run in the cloud, but I need the installer to create the image. The exercise of building the installer is very good at getting a handle on which files, libraries and what-have-yous go with each sub-system. As a side-bar we did not have that at Feedster for the longest time, which was a bad mistake.

By the end of the weekend I decided that the best way to approach this is to assume that I was installing on a pristine machine (say a CentOS installation) and assume nothing, well assume that there is at least a network and a few tools like svn, gcc and make. So everything needs to checked for and, if needed, checked out and installed, right down to things like java and ant. This caused me to look again at the structure of my svn repository and effectuate a reorganization which helped a lot both from a code organization point of view and an install process.