146 lines
6.4 KiB
Markdown
146 lines
6.4 KiB
Markdown
# Sonic search results ranking
|
||
|
||
In search engines like Sonic, it’s common for results to be ranked using
|
||
[tf-idf] or its “evolution” [BM25]. However, to keep a very small index Sonic
|
||
doesn’t store some information that’s necessary to compute [term frequency], a
|
||
core element of both [tf-idf] and [BM25].
|
||
|
||
Out of simplicity, before `v1.7.0` Sonic used to return results that contained
|
||
all search terms (with some fuzzy matching), sorted by reverse ingestion order
|
||
(i.e. most recent documents first). While this was good enough for some use
|
||
cases, discrepancies in ingestion and query languages would silently affect
|
||
which words were considered stopwords and prevent some documents[^doc] from
|
||
showing up in results (because of the implicit `AND`)[^1].
|
||
|
||
[^doc]: Since this document is about information retrieval as a general concept, this document will use the term “document” to refer to Sonic objects as it’s the strandard term.
|
||
[^1]: For example, see [issue #262](https://github.com/valeriansaliou/sonic/issues/262).
|
||
|
||
In `v1.7.0`, we reworked Sonic’s ranking algorithm to yield better results.
|
||
|
||
## Ranking algorithm
|
||
|
||
For each search term, Sonic looks for documents containing exactly that search
|
||
term, then it performs fuzzy matching until enough results are found. For each
|
||
term, Sonic keeps track of the [Levenshtein distance] of the closest word found
|
||
in each document. Once that done, a documents’s score is calculated by summing
|
||
up its scores for each term, and documents are finally sorted by score. When
|
||
ties occur, results are sorted by reverse ingestion order[^rio].
|
||
|
||
[^rio]: This happens at no cost and is invisible in the code, but happens because it’s the order in which results are yielded by the index (on purpose).
|
||
|
||
Here is some more detailed and accurate pseudo-code:
|
||
|
||
```txt
|
||
MISSING_MATCH_SCORE = 100
|
||
|
||
fn query(query, app_config) -> [doc] {
|
||
limit = app_config.store.kv.retain_word_objects
|
||
mut alternates_try = app_config.search.query_alternates_try
|
||
|
||
// Updates the document’s score for a given term.
|
||
// Returns whether the document was inserted or not.
|
||
fn update_score(mut results_matrix, doc, score, term_idx, term_count) -> bool {
|
||
if results_matrix contains doc:
|
||
// Update score to keep minimum between existing and new score.
|
||
results_matrix[doc][term_idx] = min(results_matrix[doc][term_idx], score)
|
||
return false
|
||
else:
|
||
// By default, assign a high score to each term.
|
||
mut scores = [MISSING_MATCH_SCORE; term_count]
|
||
scores[term_idx] = score
|
||
results_matrix[doc] = scores
|
||
return true
|
||
}
|
||
|
||
mut results_matrix = []
|
||
|
||
// Look for exact matches.
|
||
for (index, term) in normalized(query):
|
||
for doc in find_exact_matches(term):
|
||
if len(results_matrix) < limit:
|
||
score = 0
|
||
update_score(results_matrix, doc, score, index, len(normalized(query)))
|
||
|
||
// Look for words containing `term` as prefix.
|
||
if len(results_matrix) < limit && alternates_try > 0:
|
||
for (index, term) in normalized(query):
|
||
for suggested_term in lookup_begins(term):
|
||
for doc in limit(find_exact_matches(term), alternates_try):
|
||
if len(results_matrix) < limit:
|
||
// Compute Levenshtein distance.
|
||
score = len(suggested_term) - len(term)
|
||
if update_score(results_matrix, doc, score, index, len(normalized(query))):
|
||
alternates_try -= 1
|
||
|
||
// Thresholds are configured by `search.typo_factor_word_lengths`
|
||
// (shown with the default value: `[4, 7, 10]`).
|
||
fn typo_factor(word_len) -> u32 {
|
||
if word_len <= 3: return 0
|
||
else if word_len <= 6: return 1
|
||
else if word_len <= 9: return 2
|
||
else : return 3
|
||
}
|
||
|
||
// Look for words like `term` (fuzzy matching).
|
||
if len(results_matrix) < limit && alternates_try > 0:
|
||
for (index, term) in normalized(query):
|
||
max_typo_factor = typo_factor(len(term))
|
||
mut typo_factor = 1
|
||
|
||
// Iterate on increasingly larger typo factors (Levenshtein distances).
|
||
while len(results_matrix) < limit && typo_factor <= max_typo_factor:
|
||
for (suggested_term, lev_distance) in lookup_typos(term, typo_factor):
|
||
for doc in limit(find_exact_matches(term), alternates_try):
|
||
if len(results_matrix) < limit:
|
||
// Save Levenshtein distance as score.
|
||
if update_score(results_matrix, doc, lev_distance, index, len(normalized(query))):
|
||
alternates_try -= 1
|
||
typo_factor += 1
|
||
|
||
// Computes final score for each document, summing individual term scores.
|
||
// NOTE: Takes missing matches into account thanks to `MISSING_MATCH_SCORE`.
|
||
fn flatten_matrix(results_matrix) -> [(doc, score)] {
|
||
mut list = []
|
||
|
||
for (doc, scores) in results_matrix:
|
||
list += (doc, sum(scores))
|
||
|
||
return list
|
||
}
|
||
|
||
// Creates a B-Tree of `[doc]` keyed by score, then flattens it.
|
||
fn sort_by_score(results_map) -> [doc] {
|
||
btree = btree_by_score(results_map)
|
||
return flatten(btree)
|
||
}
|
||
|
||
return sort_by_score(flatten_matrix(results_matrix))
|
||
}
|
||
```
|
||
|
||
### Algorithm properties
|
||
|
||
- At most `store.kv.retain_word_objects` are returned.
|
||
- Fuzzy matching yields at most `search.query_alternates_try` results.
|
||
- Memory usage is linear, proportional to the number of results.
|
||
- Although the pseudo-code doesn’t show it, most operations are lazy, meaning
|
||
only useful computations are executed.
|
||
- Scoring is calculated using integer values only, without a single floating
|
||
point computation, division, square root or any other expensive operation.
|
||
|
||
### Possible future improvements
|
||
|
||
- Assign a weight to each term, based on [inverse document frequency], to give
|
||
more precedence to rare terms.
|
||
- This would imply making some floating point operations, but they’d only be
|
||
`O(len(results))`, with idf computation being `O(1)`.
|
||
- This could be done in parallel from lookups, as it is independent.
|
||
- Parallelize some operations (e.g. each term scoring)?
|
||
- Overhead might be higher than gain, we’d have to benchmark.
|
||
|
||
[tf-idf]: https://en.wikipedia.org/wiki/Tf%E2%80%93idf
|
||
[BM25]: https://en.wikipedia.org/wiki/Okapi_BM25 "“Okapi BM25” on Wikipedia"
|
||
[term frequency]: https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Term_frequency "“Term frequency” in “tf-idf” on Wikipedia"
|
||
[inverse document frequency]: https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Inverse_document_frequency "“Inverse document frequency” in “tf-idf” on Wikipedia"
|
||
[Levenshtein distance]: https://en.wikipedia.org/wiki/Levenshtein_distance "“Levenshtein distance” on Wikipedia"
|