This is an experimental addon (or patch) for the Zebra server to faciliate ranked searches with the Vector Space Model. To install the addon, copy the file 'rank1.c' in the index directory of the Zebra distribution to 'rank1.c.orig' and replace 'rank1.c' with 'zrank1.c'. Then recompile Zebra. (make sure you link with the math library, so add the linker option '-lm' if necessary). Introduction: ============= In the vector space model, queries and documents are represented as vectors of term weights. Positive weights characterise terms assigned to a document while a zero weight is used for terms that do not occur in the document. The most popular method to obtain term weights is the tf-idf weighting schema, where the weight of a term is calculated as the product of a term factor (tf) and an inverse document factor (idf). Combining the tf and idf factors (usually the product) and normalizing them yields a tf-idf vector (or term weight vector) modelling a document (or query). The similarity score between a query and a document (or the similarity between two documents) depends on the term weight vectors and is then computed by a similarity function combining these two weight vectors (most often the cosine between the vectors is computed). Weighting functions: ==================== 1) term frequency functions: locc denotes the in document frequency of a term (local frequency) none ('n'): tf = locc binary ('b'): tf = 1 max_norm ('m'): tf = locc/max_locc aug_norm ('a'): tf = K + (1 - K) * (locc/max_locc) ; K = 0.5 square ('s'): tf = locc * locc log ('l'): tf = log(locc) + 1 2) inverse document functions: gocc is the database frequency of a term (global frequncy) num_docs is the number of documents in the database none ('n'): idf = 1 tfidf ('t'): idf = log (num_docs / gocc) prob ('p'): idf = log ((num_docs - gocc) / gocc) freq ('f'): idf = 1 / n squared ('s'): idf = log (num_docs / gocc) ^ 2 3) weight normalisation functions: wt = tf * idf none ('n'): nwt = wt sum ('s'): nwt = wt / sum ( wt_i ) cosine ('c'): nwt = wt / sqrt ( sum ( wt_i ^2 )) fourth ('f'): nwt = wt / sum ( wt_i ^ 4 ) max ('m'): nwt = wt / max ( wt_i ) For example, the string 'atc' indicates that the augmented normalized term frequency ('a'), the usual tfidf weight ('t') and cosine normalisation ('c') is to be used for term weight caomputation. *) similarity function: - cosine ('c') others are (TODO, not yet implemented): - pivoted unqiue normalisation ('p') - jaccard ('j') - dice ('d') - minkwoski ('m') (Note: at the moment, there are 6 * 5 * 5 * 6 * 5 * 5 (* 1) ranking schemes selectable, but not all of them work). Example query (for yaz-client): =============================== find @attr 1=4 @attr 2=102 @attr 4=105 "where do i find literature on medical databases" Relation attribute 102 indicates that the results should be ranked by relevance and the query should be treated as free text (or wordlist). Pitfalls and problems: ====================== - The name of the ranking schema should be read from the Zebra configuration file. - Some weighting schemas require values to be calculated that are assigned constant values in this addon. For example, the db_f_max is the maximum frequency of a term in the database. - Ranking (or weight computation) is done online, e. g. immediately before records are retrieved. A faster way to get term weights would be to store them within inverted files. Adding this feature to Zebra would require major changes for the indexing process (as far as I can tell). - Stopwords are frequent words considered not useful for indexing documents (for example, "and", "the", "above" are common stopwords). Often these words do not carry semantic information. A stopword list might considerably reduce the size of the index and will probably enhance precision for natural language queries. In Zebra, stopword removal is possible with input filters. - Stemming is often used to map various morphologic forms of a concept to a single representation (for example, "countries" and "country" should both be stemmed to "country"). Using stemming for indexing is used to increase recall. In Zebra, stemming should be part of input filtering. Literature: =========== * Sebastian Hammer and Adam Dickmeiss and Heikki Levanto and Mike Taylor, Zebra - user's guide and reference, IndexData, 1995-2003. * Gerard Salton and Chris Buckley, "Term Weighting Approaches in Automatic Text Retrieval", Dept. of Computer Science, Cornell University, TR 87-881, November 1987. Also appeared in: Information Processing and Management, vol. 24, no. 5, pp. 513--523, 1988. * Justin Zobel and Alistair Moffat, Exploring the Similarity Space, SIGIR Forum 32(1), p. 18-34, 1998. http://citeseer.nj.nec.com/zobel98exploring.html * SMART related documents: http://www.dcs.gla.ac.uk/ftp/pub/Bruin/HTML/SMART.HTM Nomenclature / glossary: ======================== - Database and collection are used as synonyms - A Document is the indexed part of a record that is referred to in a query (for a title search, the title entry) * Ranking schema A ranking schema is identified by a seven character string (note: this may change in the future). The first three characters indicate the function to be used for the documents term frequency factor, the documents inverse document frequency factor and the function to combine these factors to assign a term weight. The middle character is the delimiter '-'. The last three characters indicate the functions for query term weighting. Note that different functions may be used for query and document vectors. The default similarity function calculates the cosine between a document term vector and the query term vector. * Term: an atomic concept used for indexing (a string), for example a word, a phrase or a number * Document: In Zebra, any part of a record that has index terms assigned to it. As data can (sometimes) be structured, document also refers to the smallest substructure with index terms (for example, a newspaper article may be structured into its title, abstract and its body of text components, which can be considered as different documents). * Term weighting function: the function used to combine and normalize tf and idf * Term frequency factor (tf) / Local weight: It indicates how important a term is to a particular document and depends on how many times a term occurs in a document. * Inverse document factor (idf) / Global weight: The global weight indicates how important a term is to the entire database based on the number of documents in which the term occurs. The inverse document frequency is based on the observation that a less frequently occurring term has better properties discriminating documents than a term that in more documents. * Normalisation: the normalisation function tries to compensate for ranking discrepancies (for example different document lengths). * Score: The score of a document indicates its similarity to the query (0 <= score <=1000) * Rank: The rank of a document is the position in the ranked result set. (first document: rank 1, etc.) TODO: ===== - replace 'fprintf' with 'yaz_log' - produce small test database and test cases - extend naming schema to eight chars (include similarity functions) - warn if schema is not fully available (e.g. if 'num_docs' or 'tf_max' are used) - optimize implementation (!) - Some structure elements are declared as integers ('int'). For larger databases, they might have to be declared as 'unsigned long int' - 'd_f_max_str' and 'f_max_str' are not really needed (in DS, RS) - 'db_num_docs' is the number of documents in the database. (probably the number of sysnos) This is computed for the DBs Explain record as 'recordCount' and should be avaialable as reg-> ... -> recordCount - 'db_terms' is the number of distinct terms used in the database. (probably the number of keys) - maybe maximum frequencies can be computed on-the-fly for result sets (in contrast to the whole set of records)