Add: Repeated Levenshtein dictionary search - #10
Draft
grouville wants to merge 13 commits into
Draft
Conversation
grouville
marked this pull request as draft
August 16, 2026 07:43
grouville
force-pushed
the
levenshtein-index-benchmarks
branch
from
August 16, 2026 08:48
47fc7f0 to
3597d94
Compare
Every implementation needs to receive the same queries. Add a small generator whose output is fixed by the dictionary, query count, mode, and seed. Mixed files evenly interleave exact words, one-edit changes, two-edit changes, and five-character extensions. Those labels only describe how a query was made. They are not treated as expected answers because another dictionary entry may still match. Byte and UTF-8 modes follow the same rules. UTF-8 edits operate on decoded characters, so one generated edit remains one edit. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
Add the StringZilla side of repeated fuzzy search over a dictionary that is built once and then left unchanged. The runner calls the public C batch API rather than the internal C++ index used while the algorithm was being developed. Dictionary construction and one output-sizing pass stay outside the query timer. The three sparse result arrays are then reused, while finding and writing every query and dictionary match remains inside the timer. Result sorting and file output stay outside it. The same runner covers byte and UTF-8 behavior, one or several CPU cores, cache eviction, repeated batches, and bounds one through four. It validates every returned query ID, dictionary ID, and distance before writing the shared result format. On the local Intel AVX2 run with 213,557 words and 10,000 queries, one core took 7.45 ms at k=1 and 66.60 ms at k=2. Four physical cores took 2.11 and 17.99 ms. The complete serial and parallel files matched RapidFuzz. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
Use RapidFuzz as the reference for the byte benchmark. It receives the same dictionary, queries, and inclusive distance bounds as StringZilla. It scans the complete dictionary for every query, so it checks correctness rather than serving as the closest indexed performance comparison. Both runners write the same file format. Comparing those files checks every dictionary ID and exact distance for every query, not only the final match count. The complete results matched at k=1 and k=2 on the recorded English run. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
Add the closest reusable index found for this workload, pinned to the official SymSpell-Rust repository. SymSpell normally lowercases words, counts an adjacent swap as one edit, and cannot preserve duplicate IDs. The runner therefore uses unique lowercase input and checks every suggestion again with plain Levenshtein distance. Sorting is only needed for the result file and stays outside the timer. On the recorded English run, StringZilla took 1.948 ms at k=1 and 41.555 ms at k=2. SymSpell took 20.931 and 455.857 ms. Both complete result files matched RapidFuzz. The comparison is part of the normal StringWars Cargo target and dependency lock instead of a separate Rust project. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
Add Rust fst 0.4.7 to the shared dictionary-search benchmark. fst returns dictionary IDs without exact distances and cannot keep duplicate keys. The comparable run therefore uses unique ASCII input, where its character behavior and StringZilla's byte behavior agree. On the English run, fst took 0.866, 5.469, 62.284, and 226.019 seconds at bounds one through four. StringZilla took 1.948 ms, 41.555 ms, 5.461 seconds, and 16.223 seconds. Match counts agreed, but fst does less output work, so that difference stays visible. The optional Unicode path first checks three one-character examples. The pinned fst version misses valid substitutions there, so the runner stops instead of timing answers that do not match. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
Add Tantivy 0.26.1 through its public fuzzy term query, with adjacent swaps disabled. The runner builds one in-memory index, waits for its indexing work to finish, then collects every matching document address. Build and query time are reported separately. Tantivy returns IDs without exact distances and supports bounds one and two, so the output states those differences. On the English run, Tantivy took 485.1 ms at k=1 and 4.183 seconds at k=2. StringZilla took 1.948 and 41.555 ms. Most of this commit is the generated Cargo lock update for Tantivy's search and indexing dependencies. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
Add Lucene 10.3.1 on OpenJDK 21 through its public fuzzy query and Levenshtein automaton. The normal fuzzy query applies a shorter-term rule that plain Levenshtein search does not have. Its different match count is reported but not used for a speed ratio. The automaton mode disables adjacent swaps and returns the expected totals, so that is the comparable mode. Lucene returns hit counts without dictionary IDs or exact distances. On the English run, its exact automaton took 1.676 seconds at k=1 and 14.965 seconds at k=2. StringZilla took 1.948 and 41.555 ms. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
Add the RapidFuzz reference for Unicode search. It validates and decodes UTF-8 before timing, then measures distance between Unicode characters. The helper extracts 348,980 unique lowercase terms from the Simplified Chinese dictionary without changing their text. StringZilla and RapidFuzz produced identical complete result files on both the small non-ASCII test and the natural Chinese data. The Chinese run returned 2,219,220 matches at k=1 and 343,237,926 at k=2. StringZilla took 16.017 ms and 5.149 seconds. The very large k=2 output is an important limit on broader speed claims. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
The first mixed workload changed sampled words mostly through substitutions. That was deterministic, but it did not exercise the insertion and deletion behavior that a Levenshtein index is designed to handle. Generate exact queries, each one-edit operation, several two-edit combinations, adjacent swaps, and a five-symbol extension in equal proportions. A swap still costs two plain Levenshtein edits. The mode name describes only how the query was made from its sampled source. It makes no claim about distance from other dictionary entries. The complete-result oracle remains responsible for the answer. Use the same selection and mutation rules for bytes and decoded UTF-8 codepoints. Skip source words that cannot support a requested operation instead of silently turning it into a different edit. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
The StringZilla runner used one untimed search to discover the exact result capacity, then timed the same batch with warm reader memory and perfect output sizing. That is a useful steady-state measurement, but it should not stand in for every query cost. Report four named modes instead: a fresh reader with sizing and retry, a warm pre-sized batch, a reusable growable service buffer, and single-query p50, p95, and p99 latency. Keep dictionary construction separate and print the work included by each line. A threshold-specialized run still builds the low-bound indexes independently. A shared run builds one index at the requested maximum and queries every smaller bound, which matches a deployed service that accepts different thresholds. Allow the exact StringZilla and RapidFuzz track to sweep through distance 254. The ecosystem adapters keep their smaller limits, so large-bound crossover results remain a separate complete-output comparison rather than being mixed into the low-bound table. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
Counting matches and returning every query ID, dictionary ID, and distance are different amounts of work. Report them as separate RapidFuzz modes so a count-only result is never compared directly with StringZilla full output. The materialized mode keeps one result vector across repeats, computes the same checksum fields as the StringZilla runner, and continues to write the common binary artifact used for exact result comparison. Allow the byte and UTF-8 correctness runners through distance 254. This supports the separate large-bound crossover sweep without changing the smaller limits of tools whose own APIs stop earlier. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
SymSpell does not return the same contract as StringZilla. Its native suggestions may use different distance behavior, and the adapter still needs to recover every exact plain-Levenshtein ID before a direct timing comparison is valid. The first compatibility adapter decoded strings and allocated dynamic-programming rows while processing every suggestion. That measured avoidable harness work rather than the cost of the lookup. Decode the dictionary and queries once, then reuse the verifier rows and match buffer. Report native SymSpell output and exact compatibility output as separate modes. The exact mode remains restricted to unique lowercase dictionaries because those are real limits of this adapter. The benchmark target compiles with the pinned SymSpell revision, and its tiny complete-output artifact matches the StringZilla runner. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
Repeated dictionary retrieval is not the same workload as the existing dense distance matrices, so give it one focused page and link that page from the root benchmark catalogue. Document the exact output contract before listing tools. StringZilla, RapidFuzz, and the exact SymSpell mode can be compared only when they return the same original IDs and distances. Native suggestions, unique terms, and counts remain useful context but do not support a direct speedup claim. Describe the generated edit shapes, cold and warm timing modes, threshold-specialized and shared indexes, required machine metadata, and the common binary result check. Keep the larger-bound crossover separate because several ecosystem tools stop at small distances. Results and raw machine logs are intentionally absent. They belong in the draft PR and archived run artifacts after the protocol passes, not as permanent numbers in the benchmark source tree. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
grouville
force-pushed
the
levenshtein-index-benchmarks
branch
from
August 17, 2026 21:07
76d0717 to
56e5489
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a benchmark for exact bounded Levenshtein search over a dictionary that is built once and searched many times.
It accompanies StringZilla #336. The existing
similarities/benchmark measures a full distance matrix. This benchmark covers a different use case, such as spell checking or autocomplete over a dictionary that stays unchanged between queries.What is compared
Every runner receives the same deterministic dictionary and query files. A complete result contains every query ID, original dictionary ID, and exact distance within an inclusive bound. Duplicate dictionary entries keep separate IDs. An adjacent swap counts as two plain Levenshtein edits.
StringZilla uses the public C batch API. RapidFuzz scans the full dictionary and writes the same result format, so it acts as the correctness reference. SymSpell also has an exact compatibility mode on the unique lowercase dictionaries it supports.
Those three modes can be compared directly only after their complete result files match. Native SymSpell, Rust
fst, Tantivy, and Lucene return less information or use different rules. They remain useful context, but the benchmark does not turn them into direct speedup claims.Method
The query generator gives equal weight to exact queries, substitutions, insertions, deletions, mixed two-edit cases, adjacent swaps, and longer extensions. The label records how a query was created from one source word. RapidFuzz still decides the complete answer against the whole dictionary.
StringZilla reports four timings separately:
cold_end_to_endincludes fresh reader memory, output sizing, allocation, retry, and result materialization.warm_presizedmeasures repeated batches after scratch memory and output capacity are available.steady_growablereuses a realistic grow-only output buffer and includes any required retry.single_query_latencyreports p50, p95, and p99 for individual queries.Dictionary construction is reported separately. Threshold-specific runs build one index per maximum bound. Shared-index runs build once and query the same index at several bounds.
Byte and UTF-8 modes have separate references. UTF-8 distance is measured between decoded Unicode codepoints. Every directly comparable runner writes the same binary result file, and timings are accepted only after those files match.
Current evidence
An earlier public API run used 213,557 English words, 10,000 mixed queries, and one Intel Core i5-9300H AVX2 core. Its StringZilla numbers represent the warm, pre-sized mode:
k=1k=2The complete files matched. This is promising, but it is not the final cross-tool result. The revised runners now build cleanly, and a fresh small workload matches RapidFuzz result for result at
k=1andk=2across all four StringZilla timing modes.The PR stays in draft until the revised protocol is rerun on the same machines with at least 20 measured repetitions, randomized runner order, pinned CPU placement, dictionary and query hashes, raw per-run output, construction time, retained index size, peak build memory, and reader scratch.
Scope
This benchmark covers bounded retrieval of every match for StringZilla #336. It does not cover weighted distance, normalized similarity, mutable dictionaries, or the separate nearest/top-N follow-up.
The main comparison covers the small bounds supported by the indexed tools. A separate StringZilla and RapidFuzz sweep checks larger bounds for correctness and performance cliffs without mixing those results into the low-bound ecosystem table.
Review order
fst, Tantivy, Lucene, and the UTF-8 reference separately.