score fuzzy term by distance
When playing around with tantivy, i came up with a query including multiple fuzzy term queries. Unvortunately the results don't make much sense (e.g. imagine a text field "the quick brown fox" and some other completely different text fields and then do a boolean query with multiple fuzzy terms. The results are completely unusable, because even an exact search term results in some completely different term scored the same). I think it comes down to this: The single fuzzy term query doesn't make a difference between exact matches or some levenshtein distance. The test should find the second added document first (with a higher score), but doesn't.
#[cfg(test)]
mod test {
use tantivy::query::FuzzyTermQuery;
use tantivy::collector::TopDocs;
use tantivy::schema::Schema;
use tantivy::schema::{TEXT, STORED};
use tantivy::Index;
use tantivy::Term;
#[test]
pub fn test_fuzzy_term() {
let mut schema_builder = Schema::builder();
let country_field = schema_builder.add_text_field("country", TEXT | STORED);
let schema = schema_builder.build();
let index = Index::create_in_ram(schema);
{
let mut index_writer = index.writer_with_num_threads(1, 10_000_000).unwrap();
index_writer.add_document(doc!(
country_field => "WENN ROT WIE RUBIN",
));
index_writer.add_document(doc!(
country_field => "WENN ROT WIE ROBIN",
));
index_writer.commit().unwrap();
}
let reader = index.reader().unwrap();
let searcher = reader.searcher();
{
let term = Term::from_field_text(country_field, "robin");
let fuzzy_query = FuzzyTermQuery::new(term, 2, true);
let top_docs = searcher
.search(&fuzzy_query, &TopDocs::with_limit(100))
.unwrap();
assert_eq!(top_docs.len(), 2, "Expected 2 documents");
let (score, adr) = top_docs[0];
let document = index.schema().to_named_doc(&searcher.doc(adr).expect("document"));
let json = index.schema().to_json(&searcher.doc(adr).expect("document"));
println!("{}", json);
println!("{:?}", top_docs);
assert_eq!(document.0.get("country").expect("second document")[0].text().unwrap(), "WENN ROT WIE ROBIN");
}
}
}Somehow the distance needs to be considered in the scoring, else the fuzzy query doesn't make much sense.
Or is there some other way to achieve this, i didn't see?
Source: quickwit-oss/tantivy