Some notes on MySQL and Wikipedia, sorta thrown in with the MySQL 12 days of Scale-Out.
These notes are very high level but they are very good nonetheless, the way you scale MySQL is to partition your data, Wikipedia slices by:
That is the only way to go, the approach I take is that if you can’t slice and dice your data, it will not scale.
At Feedster the data was very time dependent, old posts, new posts, etc… people want to see the new stuff and are not so keen on the old stuff, so we sliced up the posts into segments and delta segments, lower numbered segments contained older posts, higher numbered segments contained newer posts, and we could replicate those segments across servers.
It is nice to see that Froogle has morphed into Google Product Search.
It wasn’t the best name one could have thought up.
A little bit of trivia, Feedster used to be called Roogle, a sort of mashup between RSS and Google, but that was changed before Scott and I joined forces. Which was a good thing I think since it would have probably attracted attention from Google’s lawyers.
ps - Roogle now seems to be owned by a real estate company.
Early on in Feedster’s life, we had Michael Fagan doing an internship at the company. Among the various things he worked on, he added support for A9’s meta search format (basically RSS with some extensions).
I was looking around the A9 web site for information about it and found that it had been forked out to OpenSearch.org.
Turns out that Micheal continued to work on it.
It looks like it has evolved into a very functional spec for supporting meta searching.
While working at Feedster, I was originally responsible for the crawler, the indexer and the search engine. Since I knew nothing about crawlers, I still don’t but I know more about them now than I did then, I handed that over to people who were much more competent than me in that area.
So I was left looking after the indexer and the search engine. Initially, back in 2003, our traffic was very low but it kept increasing. As time went on, I would have to re-architect the indexer and the search engine to be able to deal with the increasing amount of data we were crawling and with the increasing number of searches we were getting. The trick to aim for the growth we were anticipating in six to twelve months time and not the growth we were anticipating in two to three months time. That way, the indexer and the search engine were able to deal with traffic growth very easily and without causing issues.
The other lesson learnt here was to aim for simplicity. You want to make the data administration easy when you are dealing with large amounts of data, and increasingly large amounts of data. You want to make the system robust so when machines crash, or networks go down, or power goes out, whatever is developed has to deal gracefully with system degradation and has to recover on its own when whatever when down comes back up.
All good things end, and last friday I left Feedster after spending four years there.
Feedster was founded in early 2003 during a particularly tenebrous Boston winter. Scott Johnson and myself had started to build feed search engines independently of each other. After about a month of doing this, we started an email conversation one friday evening and decided to meet for lunch to talk about what we were doing.
We talked on and off for a few weeks and it became clear to both of us that there was something there, that neither of us could got it alone and that we both had different strengths, so we decided to join forces and Feedster was truly born.
Now after four years of work, it is time for me to move on. Working on Feedster has been a lot of fun, I have been privileged to work with a lot of very smart people, and I have learnt a lot along the way.
Feedster has been through a lot of changes over the years, and the current team has been in place for just over a year. It is a very strong team and I will miss working with them a lot. I have a very deep respect for them and I have made many good friends along the way.
I will be remaining as an advisor to Feedster for the foreseeable future, so I will be in touch with them on a regular basis. And I know that Feedster will go onto greater and greater things.
For my part, I have no immediate plans, save taking some much needed vacations and batting around some ideas that have been running around my head.
In the continuing saga of lessons learnt at Feedster, one of the top ones for me would be to go for stability.
In the very beginning, way back in 2003, we had an internal debate over which linux distribution to use. We decided to use gentoo for a few reasons. The first reason was that the sys admin who was working with us was most familiar with that particular distribution. The second reason was that it afforded us a high degree of granularity in choosing which packages would be installed on specific machines. The third reason was that it was tweacked and so performed a little better than other distributions.
I have since done a u-turn (or 180) on this. All linux distributions share a common DNA if you will, so 95% of admin skill you have on one can be transfered to another. While you can pick and choose which packages you want to install on a specific machine if you have 5 or 10 machines, this quickly becomes pointless when you have 50 or a 100 machines. Most distributions performs well enough for most workload, and my experience is that tweacked distributions lose a some amount of stability.
So at this point, I feel that CentOS is a stable, no surprises distribution.
I read Matt Mullenweg’s post about his dealings with Sun, after just watching the same Scoble Show interview with Jonathan Schwartz that he watched.
And it all reminded me of the gyrations that Feedster went through with Sun early last year (2006).
But my story with Sun started longer ago than that. My first exposure to Unix was SunOS 4.1 on a SparcStation 1 on 1989, and I loved it. Until three years ago Sparc/Solaris (Sparclaris?) was my primary development platform, at which time I switched to Linux (though I had used Linux for about 5 years prior to that). And recently I gave away my trusty UltraSparc 1/200 which I had owned for 10 years. So I am a fan.
Anyway, back to our story. Sun approached Feedster with an offer to provide us with gear at a very advantageous price if we switched from “Intel/AMD/Linux” to “Sparc/Solaris”. The offer was very generous and we got schmoozed by both Jonathan Schwartz, and Scott McNealy who came to our offices to talk to us. Unfortunately I was out of town both times.
But when we took a hard look at the deal, a number of issues came up:
- There were non-negligible switching costs. Even if you run on a LAMP stack, you need to make sure that everything works as expected on the target platform. In addition to the LAMP stack, we had a number of custom applications which would need to be ported and tested. The opportunity cost of making the move also needed to be factored in.
- We would need to familiarize ourselves with a new platform, admittedly Solaris and Linux are both Unix variants, but I have spent enough time with both to know that there are big differences between the two.
- There were also the politics of such a move, the offer from Sun came with strings attached, they wanted to make sure people knew that Feedster was running on Sparc/Solaris.
- Looking at the longer term, say 12 to 18 months, there was nothing to indicate that we were going to get the same generous deal on future purchases, and Sun hardware is considerably more expensive than Intel/AMD based systems, even after deep discounts. Sun hardware also comes from a single source, which there are no end of sources for Intel/AMD based hardware.
In the end the deal was not consummated, for the reasons listed above and others too.
Hiring is always a subject of debate, and there are a lot of books/posts/opinions out there on how/who/what to hire.
For me it boils down to few simple rules:
- Hire because you want to, not because you need to. Hiring someone just because you need to fill a seat is a recipe for failure. I have hired people in the past because I needed someone, a programmer, a graphic artists, whatever, and the pain of hiring the wrong person is much higher than not hiring at all.
- Hire smarter than you. I have seen this one before, colleagues sometimes feel threatened when someone walks in the door who is smarter than they are and decides against them for some bogus reason.
- Hire multi-skilled people. It is always good to hire people because they have multiple skills, that stands to reason. But you need to look for disparate skills which will be useful to you in the future. For example, if you plan to expand into the Japanese market, hiring a developer who is also fluent in Japanese makes very good sense.
- Never hire based on phone interview alone. Always interview someone face to face. Human beings are social animals (sorry to be so base), and the phone is a very poor communication medium when you don’t know someone.
- Fire quickly. If the person is not a fit, or they bluffed their way through the screening process (you have a screening process right?) or is clearly out of their depth, it is much better to let them good sooner rather than later. Delay does damage to the project they are on and to the rest of the team because they see that there is an issue (usually before management), and keeping them on is bad for morale. People will not fault you because you fired them, but because you did not fire them fast enough.
One of the better articles I have read on hiring were written by Joel Spolsky and can be read here and here.
In my previous Lessons Learned post, I mentioned how it was important to get code right because, most than likely, it is going to be in use for far longer than you imagine.
The flip side of this is that you also need to know when enough tinkering is enough and move onto the next thing. Value will not be created if the same bit of code is being re-written ad nauseam.
