[SYSTEMDS-3645] FTBench extension: Polars

[LasseHenrich]

(For more details and the history of commits, ref. my working repo)

issue paper

For review, I recommend starting at T1_polars.py and/or transformUtils.py, working yourself up the transformations in numerical order. The T10_polars.py shown first in the preview is probably the most complex and not very instructive to start with.

Purpose

(ref. issue)
FTBench currently implements only a subset of the 15 transformations (T1 through T15) listed in the paper, on only a certain number of libraries. Our goal is to extend FTBench by completing transformations for the currently implemented libraries and/or adding transformations in new libraries.

E.g. Keras is missing T4-T9 and T13-T15, and other libraries like Pandas, polars, should be implemented.

Current implementations (before this PR with polars)

Transformation	Dask	Keras	NimbusML	SKLearn	SparkML	SystemDS
T1	✅	✅	-	✅	✅	✅
T2	✅	✅	-	✅	✅	✅
T3	✅	✅	✅	✅	✅	✅
T4	-	-	-	✅	-	✅
T5	-	-	-	✅	-	✅
T6	-	-	-	✅	-	✅
T7	-	-	-	✅	-	✅
T8	-	-	-	✅	-	✅
T9	-	-	-	✅	✅	✅
T10	-	✅	-	✅	-	✅
T11	-	✅	-	-	-	✅
T12	-	✅	-	✅	-	✅
T13	-	-	-	✅	-	✅
T14	-	-	-	✅	-	✅
T15	-	-	-	✅	✅	✅

Why polars

The benchmark should implement the current state-of-the-art for fast transformations to be of value. We therefore decided to add polars to the benchmark, which is written in Rust, heavily multi-threaded (in contrast to Pandas) and therefore very fast → It's an aggressive competitor for SystemDS, beating it will be difficult.

Implementation description

The polars implementation is heavily inspired by the sklearn implementation, since the latter covers by far the most transformations – aside from SystemDS, which works very differently. It's also the cleanest implementation, in the sense that it implements a transformUtils.py that (i) handles the most common transformation techniques centrally so that they don't have to be repeated, and (ii) directly reads from the spec files that SystemDS uses as well, making the code arguably the most maintainable implementation.

For maximizing comparability, I copied as much from the sklearn implementation as I could and stuck closesly to the overall structure. That e.g. means that (i) the polars implementation has some comments that the sklearn one has as well, (ii) there are a lot of "mirrors"-annotations referring to the sklearn code, and (iii) sometimes polars code is already refactorable (e.g. feature hashing is just implemented in T9_polars.py instead of transformUtils.py, mirroring sklearn). Much of sklearn's functionality (like CountVectorizer and FeatureHasher classes) is not supported in polars; when I had to implement own solutions using polars, I again focused on comparability.

There are only a few cases where I intentionally developed code that behaves slightly differently, mostly in cases where the sklearn implementation worked in a way I deemed incorrect or unnecessary – ref. problems.

Test results

Some test results comparing polars and sklearn:

Transformation	SKLearn	Polars	Notes
T1	[58.764 56.352 48.592]	[19.8 18.178 19.081]	sklearn implementation results in 5 more columns; ref. problems issue 1
T2	[6686.51 6621.844 5102.801]	[4656.006 2827.002 2398.719]	sklearn implementation results in 41 more columns; ref. problems issue 4
T3			couldn't test, criteo dataset unavailable
T4			couldn't test, criteo dataset unavailable
T5	[5197.465 5242.937 5326.531]	[783.457 752.535 709.507]
T6	[922.462 965.298 997.794]	[1841.982 1375.196 1701.007]	Tested with just 1M instead of the original >24M lines of crypto.csv
T7	[354174.542 362530.858 356551.95]	[2555.263 2918.198 2582.053]	Tested with just 1M instead of the original >24M lines of crypto.csv.
T8	[3033.269 2743.068 2784.111]	[363.089 255.003 270.836]
T9	[13971.371 12787.931 12301.793]	[5151.001 3943.462 3796.602]
T10	[2148.08 2082.393 2086.692]	[595.165 457.439 432.453]	Tested with a subset of AminerAbstract_small.txt with only 100k lines instead of the >280k in the original. Note that the polars implementation results in 4121 columns more, since its \w regex expression counts "¾" as a token boundary like punctuation.
T13	[7102.688 6312.152 6777.781]	[1097.658 1031.282 1062.567]
T14	[6059.047 6522.691 5878.785]	[774.276 709.495 754.022]	Tested with just 2M instead of 100M lines in data.csv

Missing

Still missing are the implementations for T11, T12, and T15.

1. T11 also doesn't exist in sklearn (just Dask and SystemDS), so it seems less important to me.
2. T12 is a bit more effort to implement since it doesn't use a spec, but it should still be manageable with a future PR.
3. T15 seems like the most effort, because of the Gaussian Naive Bayes that is used by the sklearn implementation.

Problems

1. KBinsDiscretizer is applied to all numeric columns in sklearn

Decision In polars, bin only encoders['bins'], leave pt columns as passthrough

sklearn's num_pipe

numeric = list(X.select_dtypes(include=np.float64).columns)  # ~> pt ∪ bins
num_pipe = Pipeline([('selector', ColumnSelector(numeric))])
if scale: num_pipe.steps.append(['normalize', StandardScaler()])
if isBin: num_pipe.steps.append(['biner', KBinsDiscretizer(...)])

Since numeric = pt ∪ bins and the pipeline is sequential, KBinsDiscretizer is applied to every numeric column that reaches it – including pt columns, not just the ones listed in encoders['bins'].

Example: adult_spec2.json has pt = [4], bins = [0, 2, 10, 11, 12]and isBinDC is True (all bin columns are also in dummycode), so postBin = 'onehot' → sklearn's num_pipe one-hot-encodes 6 columns into 5 bins each (30 output columns total), where 5 of those 30 columns come from binning+one-hot-encoding the passthrough column. My polars implementation treats them separately and therefore produces 5 fewer columns than sklearn, skipping the (in sklearn's case, ultimately discarded/overwritten) extra binning work.

This is likely a bug of the sklearn reference (ref. "TODO: support column specific methods and numbins" / "TODO: support mixed encoders" comments). Implications: (a) output shapes differ between the two implementations, and (b) sklearn performs more transformation work in this case.

2. scale=True scales categorical/binned output in sklearn

Implemented See phase 4 in polars' transformUtils.py

sklearn implementation when scale=True:

• num_pipe gets a StandardScaler() (mean+std) inserted before the binning step, applied to all of pt ∪ bins.
• cat_pipe gets StandardScaler(with_mean=False) appended after the one-hot/ordinal encoder, scaling the encoded categorical output.

Both specs that are worked on with scale=True (kdd_spec1.json, criteo_spec2.json) have isBin=True. As mentioned in the first issue, KBinsDiscretizer is applied to both pt and bins, overwriting StandardScaler with bin indices/one-hot → num_pipe's StandardScaler has no effect on sklearn's output for these two specs and is just dead computation.

So, the effect of scale=True comes just from cat_pipe's StandardScaler(with_mean=False) and standardizes the recode/dummy-coded categorical output → So "scale" in this benchmark suite means "standardize the encoded categorical/dummy output", which is exactly what cat_pipe does.

3. Mixed recode/dummycode handling in sklearn

Ignored, not important

sklearn's cat_pipe applies one encoder to all catCols (rc and dc) and is constructed like so:

if isDC:
    cat_pipe.steps.append(['onehot', OneHotEncoder()])
elif isRC:
    cat_pipe.steps.append(['recode', OrdinalEncoder()])

If any dc column exists, every categorical column (including pure-rc ones) gets one-hot encoded – as is also stated via "TODO: support mixed encoders". My Polars implementation instead (correctly) applies ordinal encoding to rc columns and to_dummies to dc columns independently.

However, I don't think this is a problem at the moment. I let AI check all specs in systemds/specs/: in every spec where recode and dummycode are non-empty, recode ⊆ dummycode → so there is currently no "rc-only" column that would be mis-encoded by sklearn's shortcut.

4. Empty equi-width bins drop to_dummies columns in polars

Ignored, not important

For a binned column that's also dummy-coded, a different number of one-hot columns is produced when some bins are empty:

• sklearn's KBinsDiscretizer reserves a column for every bin, including ones that contain zero rows
• In the polars implementation, to_dummies emits a column only for bins that actually occur in the data – empty bins therefore produce no column.

Measured on KDD98.csv for kdd_spec1: sklearn produces 1670 bin columns, polars 1629. 29 binned columns miss 1 bin, 6 miss two, so 41 fewer columns are produced.

Since empty bins carry no information, I don't care about trying to align the two implementations.