Add sp_matmul_rs

CHANGELOG.md

@@ -5,6 +5,42 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.8.0] - Unreleased
+
+### Added
+
+* New [sp_matmul_rs](https://github.com/Bergvca/sp_matmul_rs) backend for the cosine-similarity calculation — a Rust
+  reimplementation of the sparse top-n matrix multiplication. It performs the block/chunk splitting internally and
+  adds further optimizations. This is now the **default** backend.
+* New `use_sp_matmul_rs` configuration keyword (default `True`). Set it to `False` to fall back to the original
+  `sparse_dot_topn` backend. This is slower but battle tested and thus more stable. 
+* New dependency on `sp_matmul_rs>=0.2.1`.
+* Added a test suite verifying that the `sp_matmul_rs` and `sparse_dot_topn` backends produce equivalent results.
+* New `chunk_cols` configuration keyword (default `None`) — the `sp_matmul_rs` counterpart to `n_blocks`. It sets the
+  column-chunk width of the backend's cache-blocked kernel. `chunk_cols` can be used to tune performance for matrices that
+are denser than the matrices normally expected in the string-matching use case. Only used when `use_sp_matmul_rs=True`
+(ignored, with a warning, by the `sparse_dot_topn` backend); `None` lets the backend derive the width from the detected
+L1d cache size.
+
+### Changed
+
+* Cosine similarities are now computed with `sp_matmul_rs` by default, yielding a large speed-up (e.g. fuzzy matching
+  of 663 000 names in under 18 seconds on a m5 pro using 15 cores).
+* When `use_sp_matmul_rs=True`, block splitting is handled internally by the backend; the automatic `n_blocks`
+  guesstimate and `OverflowError` fallback are only used with the `sparse_dot_topn` backend. Should the
+  `sp_matmul_rs` backend overflow its 32-bit result indices (`OverflowError`), `fit()` transparently retries it
+  with 64-bit indices (`idx_dtype=np.int64`), staying on the fast backend. Only if that retry still fails, or on
+  a `MemoryError`, does `fit()` fall back to the `sparse_dot_topn` backend with automatic block splitting.
+* Setting `n_blocks` explicitly while `use_sp_matmul_rs=True` logs a warning and ignores `n_blocks`, since blocking is
+  calculated automatically by `sp_matmul_rs`. Existing code that tunes `n_blocks` keeps working; set
+  `use_sp_matmul_rs=False` to make `n_blocks` effective again.
+
+### Fixed
+
+* `n_blocks` passed through the instance-method variants (e.g. `StringGrouper.match_strings`) is now honored; previously
+  a stale value captured at construction time (or a prior fit's automatic guess) was silently used instead.
+
+
 ## [0.7.2] - 2026-05-22
 
 ### Changed

README.md

@@ -36,24 +36,27 @@ within a single list or between two lists of strings. The full process is descri
 
 ## Speed
 
-**`string_grouper`** leverages the blazingly fast [sparse_dot_topn](https://github.com/ing-bank/sparse_dot_topn) libary
+**`string_grouper`** leverages the blazingly fast [sp_matmul_rs](https://github.com/Bergvca/sp_matmul_rs) (originally based on: [sparse_dot_topn](https://github.com/ing-bank/sparse_dot_topn))
 to calculate cosine similarities. 
 
 ```python
 s = datetime.datetime.now()
-matches = match_strings(names['Company Name'], number_of_processes = 4)
-
+matches = match_strings(names["name"], number_of_processes=15)
 e = datetime.datetime.now()
-diff = (e - s)
-str(diff)
+
+diff = e - s
+print(diff)
 ```
 Results in: 
 
-`00:05:34.65` On an Intel i7-6500U CPU @ 2.50GHz, where `len(names)` = 663 000
+`00:17.80` On an m5 pro, where `len(names)` = 663 000
 
 *in other words*,
-the library is able to perform fuzzy matching of 663 000 names in _five and a half minutes_
-on a 2015 consumer CPU using 4 cores. 
+the library is able to perform fuzzy matching of 663 000 names in _less than 18 seconds_
+on a 2026 consumer CPU using 15 cores. 
+
+**The latest version (0.8.0) with a significant speed up is not released on pypi yet.** Use this repository to install
+if you want to use the latest and greatest. 
 
 ## Simple Match
 
@@ -100,3 +103,10 @@ companies.groupby('name_deduped')['Line Number'].count().sort_values(ascending=F
 ## Documentation
 
 The documentation can be found [here](https://bergvca.github.io/string_grouper/)
+
+## Backends
+
+The library was originally developed using the [sparse_dot_topn](https://github.com/ing-bank/sparse_dot_topn) library,
+but has since been rewritten to use the [sp_matmul_rs](https://github.com/Bergvca/sp_matmul_rs) library, which is a Rust 
+implementation of the sparse matrix multiplication algorithm optimized with _Claude Fable_. To run the library with the
+original sparse_dot_topn backend, set `use_sp_matmul_rs` to `False`.

docs/performance.md

@@ -1,5 +1,10 @@
 ## Performance
 
+> **Note:** this page describes the `n_blocks` block-splitting mechanism of the legacy
+> `sparse_dot_topn` backend, which is used only when `use_sp_matmul_rs=False`.  The default
+> `sp_matmul_rs` backend (since version 0.8.0) performs block/chunk splitting internally and
+> ignores `n_blocks` (with a warning); its performance can instead be tuned with the
+> `chunk_cols` option.
 
 <b><a name="Semilogx"></a>Semilogx plots of run-times of `match_strings()` vs the number of blocks (`n_blocks[1]`) into which the right matrix-operand of the dataset (663 000 strings from sec__edgar_company_info.csv) was split before performing the string comparison.  As shown in the legend, each plot corresponds to the number `n_blocks[0]` of blocks into which the left matrix-operand was split.</b>
 ![Semilogx](https://raw.githubusercontent.com/Bergvca/string_grouper/master/images/BlockNumberSpaceExploration1.png)

docs/references/options_kwargs.md

@@ -24,7 +24,9 @@ All keyword arguments not mentioned in the function definitions above are used t
 
 * **`number_of_processes`**: The number of processes used by the cosine similarity calculation. Defaults to
  `number of cores on a machine - 1.`
-* **`n_blocks`**: This parameter is a tuple of two `int`s provided to help boost performance, if possible, of processing large DataFrames (see [Subsection Performance](#perf)), by splitting the DataFrames into `n_blocks[0]` blocks for the left operand (of the underlying matrix multiplication) and into `n_blocks[1]` blocks for the right operand before performing the string-comparisons block-wise.  Defaults to `None`, in which case automatic splitting occurs if an `OverflowError` would otherwise occur.
+* **`use_sp_matmul_rs`**: Selects the backend used for the cosine-similarity matrix multiplication.  If `True` (the default), the Rust [sp_matmul_rs](https://github.com/Bergvca/sp_matmul_rs) library is used, which performs block/chunk splitting internally.  Set it to `False` to fall back to the original [sparse_dot_topn](https://github.com/ing-bank/sparse_dot_topn) backend (slower, but battle-tested).
+* **`n_blocks`**: This parameter is a tuple of two `int`s provided to help boost performance, if possible, of processing large DataFrames (see [Subsection Performance](#perf)), by splitting the DataFrames into `n_blocks[0]` blocks for the left operand (of the underlying matrix multiplication) and into `n_blocks[1]` blocks for the right operand before performing the string-comparisons block-wise.  Defaults to `None`, in which case automatic splitting occurs if an `OverflowError` would otherwise occur.  Only used by the `sparse_dot_topn` backend (`use_sp_matmul_rs=False`); the default `sp_matmul_rs` backend splits the data internally and ignores `n_blocks` with a warning.
+* **`chunk_cols`**: The `sp_matmul_rs` counterpart to `n_blocks`: an `int` setting the column-chunk width of the backend's cache-blocked kernel.  This is a performance knob only — any value yields identical results.  Defaults to `None`, which lets `sp_matmul_rs` derive the width from the detected L1d cache size.  Only used by the `sp_matmul_rs` backend (`use_sp_matmul_rs=True`); the `sparse_dot_topn` backend ignores it with a warning.
 
 ## Other settings
 

pyproject.toml

@@ -31,6 +31,7 @@ dependencies = [
     "numpy>=2.0",
     "sparse-dot-topn>=1.1.0",
     "loguru>0.7.0",
+    "sp_matmul_rs>=0.2.1"
 ]
 
 [build-system]