The “We’re sorry…” page

Google has an interesting way of dealing with people who launch too many searches from their computers.

While working at Feedster I routinely saw the same search coming in many times a second, usually from the same subnet or the same computer. In some cases these were due to crawlers gone wild, but in other cases the pattern looked malicious.

What we did to handle that was to put in place a way of measuring the search rate coming in and looking for hight numbers of searches from the same subnet or the same computer. When we could contact an admin to check into it we did so, but if there was no contact, we initially throttled the searches, effectively putting in a delay. If that did not help, we would just cut off the IP address.

I remember two patterns to the searches, one was that a lot of them came from China, and the other is that a lot of them were looking for kiddie porn.

I think Google handles this problem better than we did at Feedster, then again they have a lot more engineers.