May 31, 2007 Leave a comment
This is a very interesting article on relevance, by Elizabeth Van Couvering (Department of Media and Communications, London School of Economics.)
The abstract reads:
In the face of rising controversy about search engine results—that they are too restrictive, too comprehensive, lacking in certain areas, over-represented in others—this article presents the results of in-depth interviews with search engine producers, examining their conceptions of search engine quality and the implications of those conceptions. Structuration theory suggests that the cultural schemas that frame these discourses of quality will be central in mobilizing resources for technological development. The evidence presented here suggests that resources in search engine development are overwhelmingly allocated on the basis of market factors or scientific/technological concerns. Fairness and representativeness, core elements of the journalists’ definition of quality media content, are not key determiners of search engine quality in the minds of search engine producers. Rather, alternative standards of quality, such as customer satisfaction and relevance, mean that tactics to silence or promote certain websites or site owners (such as blacklisting, whitelisting, and index “cleaning”) are seen as unproblematic.
The article does a very good job of illustrating the issues with defining relevance and concludes that it is not as cut and dry as one would believe, but we all knew that:
The research questions that began this article were first, how do search engine producers conceive of quality? and second, what are the implications of these conceptions of quality for the future development of search engines? The evidence from the interviews examined in this article suggests that search engine producers conceive of quality in two separate but interrelated ways. First, a quality search engine, from the producer’s perspective, has high customer satisfaction. This definition of search quality is embedded in a larger cultural schema that I have called the “market” schema, in which search engines are primarily conceived of as businesses. Second, a quality search engine produces very relevant responses to queries. Again, this definition of quality is related to the cultural schema that I have characterized as “science/technology.” Search engines from the science-technology point of view are primarily pieces of engineering.
The implications of these conceptions of quality are far reaching precisely because they are embedded in larger cultural schemas. Structuration theory emphasizes how cultural schemas and their associated norms guide the allocation of resources. This article has shown that in the case of search engines, several schemas are at work simultaneously. The schemas clearly in the ascendant—the dominant market schema and the science-technology schema—provide little scope to raise issues of public welfare, fairness, or bias. Instead, they emphasize profit, in the case of the market schema, or progress and efficiency, in the case of the science-technology schema, or defense, in the case of the war schema.
Van Couvering conducted extensive interviews, providing a few excerpts in the article, I wish there were more because they are very interesting to read.