Comments on: Stop words and minimum term length http://fschiettecatte.wordpress.com/2008/02/09/stop-words-and-minimum-term-length/ Thoughts from the edge of the 'net Mon, 13 Oct 2008 11:56:50 +0000 http://wordpress.org/?v=MU hourly 1 By: noel http://fschiettecatte.wordpress.com/2008/02/09/stop-words-and-minimum-term-length/#comment-4107 noel Thu, 21 Feb 2008 19:03:33 +0000 http://fschiettecatte.wordpress.com/?p=404#comment-4107 I'm not experts like you guys, but I am a search engine user and here's my opinion on stop words. I think they should be indexed definitely but discarded during the search unless quoted. I remember typing dr who into feedster and being extremely annoyed that even "dr who" would not yield results I was after. Stop words aren't always stop words as you say. Context is crucial. Filter the query, not the index. I’m not experts like you guys, but I am a search engine user and here’s my opinion on stop words. I think they should be indexed definitely but discarded during the search unless quoted. I remember typing dr who into feedster and being extremely annoyed that even “dr who” would not yield results I was after. Stop words aren’t always stop words as you say. Context is crucial. Filter the query, not the index.

]]> By: François Schiettecatte http://fschiettecatte.wordpress.com/2008/02/09/stop-words-and-minimum-term-length/#comment-4102 François Schiettecatte Sat, 09 Feb 2008 18:06:14 +0000 http://fschiettecatte.wordpress.com/?p=404#comment-4102 I agree with you, intersecting 50 million document IDs would not be feasible, but there are a number of approaches you can take to deal with this issue. Architectural approaches would include sharding data across a number of machines, making sure that indices can be searched in memory without hitting storage. Within a machine you can also further shard data taking advantage of threading and multi-cpu machines. Algorithmic approaches would include document ID list truncation, for example a search for 'Vitamin A' would likely generate fewer documents for 'Vitamin" than for 'A' so you could limit the range of document IDs you check for the latter term based on the document IDs you get for the former term. You can could also apply some smarts whereby you could drop the search for 'A' if you searched for 'A Vitamin'. This is somewhat more complex to implement for obvious reasons, and I would steer clear of that one. In the past I have experimented with automatically turning a term into a stop term based on frequency and distribution and that worked well, so with a search like 'sitting in the laps of the gods', you could easily drop 'in', 'the' and 'of'. Of course you could not do this for phrases. I agree with you, intersecting 50 million document IDs would not be feasible, but there are a number of approaches you can take to deal with this issue.

Architectural approaches would include sharding data across a number of machines, making sure that indices can be searched in memory without hitting storage. Within a machine you can also further shard data taking advantage of threading and multi-cpu machines.

Algorithmic approaches would include document ID list truncation, for example a search for ‘Vitamin A’ would likely generate fewer documents for ‘Vitamin” than for ‘A’ so you could limit the range of document IDs you check for the latter term based on the document IDs you get for the former term.

You can could also apply some smarts whereby you could drop the search for ‘A’ if you searched for ‘A Vitamin’. This is somewhat more complex to implement for obvious reasons, and I would steer clear of that one.

In the past I have experimented with automatically turning a term into a stop term based on frequency and distribution and that worked well, so with a search like ’sitting in the laps of the gods’, you could easily drop ‘in’, ‘the’ and ‘of’. Of course you could not do this for phrases.

]]> By: Peter Zaitsev http://fschiettecatte.wordpress.com/2008/02/09/stop-words-and-minimum-term-length/#comment-4098 Peter Zaitsev Sat, 09 Feb 2008 15:51:41 +0000 http://fschiettecatte.wordpress.com/?p=404#comment-4098 You're right, stop words generally do not increase index size dramatically. The problem is they can slow down execution significantly. Think about intersecting 50mil document ID lists for "a" and "the" words. Unless search engine has special treatment for high frequency words it can slow things down really a lot. You’re right, stop words generally do not increase index size dramatically. The problem is they can slow down execution significantly.

Think about intersecting 50mil document ID lists for “a” and “the” words.

Unless search engine has special treatment for high frequency words it can slow things down really a lot.

]]>