| || |
Search Engine testing
I need some tips, approach for testing a search engine.
The search engine is solr (http://lucene.apache.org/solr/) which will be actually a document search engine where documents will be indexed in different Projects >> under different Categories.
There will be different search types implemented, such as
1. Concept Search,
2. Near Dupes,
3. Foreign language support,
4. Free text search etc...
What are the different FUNCTIONAL aspects we need to look??
1. For all the above mentioned test types, I would generate test data for each, where in I will be knowing what is the content of the data and for a given keyword and search type, what would be the expected output. By this way I will be testing the accuracy of the test result I get.
2. The application using this Search Engine is already in use and is run with other Search Engine. So here again I will be doing Parallel Testing for both the test engine and comparing with the same result. Here again my approach towards it would be the same as mentioned above, running the search from both the search engine on the same index.
I would like to know if there is any other aspect I need to look at, other than performance testing.
Couple of things I would like to know,
1. How can I test the score of the document/search result returned?
2. How can I test for different language support? (Search on CJK (Chinese, Japanese, Korean documents)
3. What are the general things that we need to look at while testing a search engine?
Any suggestion in response to this is appreciated.
Re: Search Engine testing
I've had quite a bit of web testing experience, but this will be a big project.
On this item, how do you know or how can you classify the document. I'm assuming that you mean this to mean "of a certain idea", so a search for documents in a category called widgets, since there is also free text search, will return me all documents related to Widgets.
How are you verifying that these documents are valid to the category or concept? Are you offering a grading scale for users to grade the results, in order to begin dropping lower-graded results?
This is a completely different type of search. What you should really do here is take a look at an application like WinDiff or a file comparison application. This type of technology is what your own technology SHOULD be based on. Reason being, if you shift a couple lines out of place, it still needs to realize that those are the only changes in the doc. I would take a look at the feature sets of some file comparison apps and then port that over to your own application.
Foreign Language Support
Ouch, that sucks for you. This is a difficult thing to test, especially at a search engine level. Ideally, if I was architecting this project, I would mandate that English should be the primary language, and that additional languages would become available as patches, shortly after the initial product release.
In your case, that doesn't sound like it will be possible, so you are faced with the daunting task of hitting on most of this stuff yourself. How can you really judge the results, though, if you are English and do not understand other languages?
I guess, number one, you could become really good friends with a free translator. Should terms be translated to english to also be relevent against English docs? So if someone in France did a search on "francais", would they get back English results that are related to "french"?
Of course, you will be dealing with Unicode characters here, also, which is cool because they are fun to play with sometimes. ASCII itself is a 7-bit character set. If this is in use then there could be some real problems as the most common Unicode standard is UTF-8 which is an 8-bit standard. There is an extension of the ASCII which is 8-bit, but there are also Unicode-16 and Unicode-32 standards.
In the end, differences in the way that encoding is handled for languages could present a problem.
Free Text Search
Free Text should be fun for you. This is where I would probably start, myself, because it will probably be the least restrictive environment to work in. Do huge search terms, try common mis-spellings, quoted text vs unquoted text, AND/OR searches. These are all criteria that people are used to.
Other than that, you can check for result relevance. So let's say I type in the word "Evolution". The the document titled, "The Evolution of the Automobile" get top ranking? Over, say, something titled "Human Evolution"? How do you judge these two to be different?
What I would do next is go around to some SEO (Search Engine Optimization) websites and see how people are "beating" the current search engines that are out there. That is, how can search engines be manipulated so that your document seems more relevant than the next? Even though your document has nothing to do with the topic at hand. You can use these same techniques in your own testing to try to expose problems in the search algorithm.
As for your additional questions:
1) I would ask the developers to publish a build for QA that will either, write out the "score" somewhere on the page, write the score to a database, or flat file for reading, so that you can review this as you go along. This would then need to be removed afterwards.
2) As I mention above, two words: Free Translator! Apart from that it would be outsourcing some of the testing to a contractor in that country, so you get the dialect correct.
3) Search engines are all about relevance. If the results aren't relevant, then why would anyone waste their time using it. This is what makes Google so successful for web searches. People know that, generally, they can find what they are actually looking for within the first couple pages of results, at most!
9 out of 10 people I prove wrong agree that I'm right. The other person is my wife.
Re: Search Engine testing
These are great suggestions by Brent, but there are some slight clarifications I would like to offer regarding international character support.
Although it only take 7 binary bits to encode the first 127 code points used to display characters (which is universally denoted as ASCII) the encoding of those characters is still 8 bits because the PC architecture parses data in 1 byte chunks.
UTF-8 is perhaps the most common transformation format of UCS-2; however, UTF-8 characters can be represented with 1, 2, 3 or 4 bytes (surrogate pairs).
UTF-16 is a true 16 bit encoding pattern; however, there are already more than the 65,535 characters that a 16-bit encoding format allows.
With regards to testing search algorithms with international characters another approach is to do a fuzzy match of the results. For example, if you enter a Japanese word (that you can copy from the Internet) do the results on the page 'mostly' match the submitted string and/or characters in the string.