Any plans to use Polars for string-grouper?

[audiomuze]

Hi, I've used string-grouper to great effect to make sense of data and find it an incredibly useful tool for handling data. I was wondering whether there are any plans to leverage Polars in string-grouper in future given the performance and memory handling advantages?

[GuillaumePressiat]

Hi, thanks for your feedback and question.

I recently participated in the update of string_grouper 0.7.1 to make it work again with sparse_dot_topn C++ extension last version (native from ING with block/chunk functionnality).

I already use pola.rs a lot and have thought a first step might be to make string_grouper fluent with polars in/out with typing.aliases (eg: if pl.Series is the input then pl.DataFrame as output, else pd.DataFrame). see here #93 (comment)

For a complete rewrite with pola.rs it's a broader step I've thought of but I won't take it these months.

• Pro : I think it's a good idea and could make the code leaner and easier to maintain in the long term
• Con : pandas is still there and a core part of many projects while the polars api is not 100% stable yet. It will asked maintenance in the short term.
• Finally, most of the computation time is done by sklearn ngrams tokenization first and sparse_dot_topn, the last one is greatly optimized thanks to C++.

To answer your question it's planned but not soon (at least for me).

Have you thought about rewriting sklearn ngrams tokens or sparse_dot_topn part in Rust or just replacing python pandas with python pola.rs?

[Bergvca]

Agreed that this would be very nice, however its unclear to me how much effort this would take. The main issue is the integration with other libraries like @GuillaumePressiat mentioned. I'm not super familiar with Polars (love the idea!), but did a quick check to run a TfidfVectorizer on a polars Series:

from sklearn.feature_extraction.text import TfidfVectorizer

company_names = df.select(pl.col('Company Name'))
vectorizer = TfidfVectorizer(min_df=1)
tf_idf_matrix = vectorizer.fit_transform(company_names)

This doesn't work, so Polars doesn't seem supported fully (yet?). Unless I'm doing something wrong. Implementing this in string_grouper would be a major undertaking and I feel out of scope for this library.

[Bergvca]

Apparently - i was doing something wrong. Simply running:

tf_idf_matrix = vectorizer.fit_transform(df['Company Name']) works, so it should be easy to add polars to string_grouper.

I've posted this on stack overflow, one of the reactions pointed to narwals:

Just for the overall goal, github.com/narwhals-dev/narwhals may be of interest. Packages in the ecosystem have started to use it to provide Polars support. scikit-learn itself appears to be in the process: github.com/scikit-learn/scikit-learn/issues/31049

It's a library to write dataframe-agnostic code, and seems very usefull. It's worth playing with at least.

[Bergvca]

I've played around a bit with Narwhals, it's very promising, as it would mean we could use many dataframe backends (not just Pandas and Polars) but migrating to it will not be a trivial task I'm afraid.

• Narwhals is based on the idea that you input a DataFrame, do some transformations on said DataFrame and output the updated DataFrame or an scalar/other python object, whereas StringGrouper has a bunch of functions were new dataframes or series are created within the function. This would mean we would have to specify (or guess, based on installed libaries) a back-end and create the DataFrame using that back-end
• We make a lot of use of the Series objects directly, Narwhals seems to push to use DataFrames (even with 1 column).
• We do some stuff on Panda's indexes where I don't see directly how it would be done in Narwhals.
• There will be a lot of lines to rewrite to get the same functionality

Having said that, the core functionality (getting the cosine similarity of tf-idf matrices) works fine in Narwhals/Polars. Migrating to it will just be a big overhaul of the code.

[MarcoGorelli]

Hey all, author of Narwhals, so I figured it might be helpful if I try answering some of the questions

• There are some functions available to create new dataframes / series: from_numpy, from_dict, from_arrow, new_series. Formulaic uses some of these
• Using Series is fine in Narwhals, just pass series_only=True or allow_series=True to from_native. All the expression methods should be supported on it, plus a few more (arg_true, arg_min, scatter, ...) https://narwhals-dev.github.io/narwhals/api-reference/series/. Plotly uses Series a fair bit
• In general, Narwhals preserves the pandas index. If you need to do anything pandas-specific, you always have a little block in which you do
  • to_native
  • do your pandas-specific logic
  • back to from_native. Bokeh does something like this
• It's true that a full rewrite can be a lot of work, but there's also value in doing in incrementally. For example, fairlearn has been incrementally narwhalifying their methods - so currently, some of the library is dataframe-agnostic and the rest is still pandas-specific, and the percentage of the code which is dataframe-agnostic is gradually increasing as a community effort

[GuillaumePressiat]

Hi @MarcoGorelli,

Many thanks for the detailed feedback and tracks.
For my part, I would find it almost easier to rewrite everything from scratch with polars. I took the project just to pass it to the last sparse_dot_topn version and I'm not too fine with highly pandas dependant part of the codebase (pandas indexes are not my thing).
I made it work again that's all (it was broken).
But your points on Narwhals are goods.
Doing it incrementally is surely a good way.

[GuillaumePressiat]

Hi,

There are actually some implementations of string matching directly in polars, see here:
https://github.com/ddotta/awesome-polars?tab=readme-ov-file#text-similarity--fuzzy-matching

One of them even appears to include a rewrite of sparse_dot_topn in Rust.
https://github.com/schemaitat/polars_sim?tab=readme-ov-file#references
To be tried