Lessons Learnt - Aim in Front of the Target

While working at Feedster, I was originally responsible for the crawler, the indexer and the search engine. Since I knew nothing about crawlers, I still don’t but I know more about them now than I did then, I handed that over to people who were much more competent than me in that area.

So I was left looking after the indexer and the search engine. Initially, back in 2003, our traffic was very low but it kept increasing. As time went on, I would have to re-architect the indexer and the search engine to be able to deal with the increasing amount of data we were crawling and with the increasing number of searches we were getting. The trick to aim for the growth we were anticipating in six to twelve months time and not the growth we were anticipating in two to three months time. That way, the indexer and the search engine were able to deal with traffic growth very easily and without causing issues.

The other lesson learnt here was to aim for simplicity. You want to make the data administration easy when you are dealing with large amounts of data, and increasingly large amounts of data. You want to make the system robust so when machines crash, or networks go down, or power goes out, whatever is developed has to deal gracefully with system degradation and has to recover on its own when whatever when down comes back up.