Customer Segmentation Prediction: Process and Insights

1. Project Overview

The goal of this project was to build a machine learning pipeline to predict customer segments into four distinct classes (A, B, C, D) based on demographic data. Rather than blindly optimizing for a high accuracy score, the project focused on diagnosing the limits of the data, applying strict regularization, and building custom tools to handle business constraints safely.

2. Data Preprocessing Pipeline

A robust ColumnTransformer pipeline was built to handle raw data safely without data leakage. The preprocessing steps included:

• Numeric Features (Age, Work_Experience, Family_Size): Missing values were filled with the median. A custom FunctionTransformer applied a log transformation (log1p) to handle skewness, followed by standard scaling.
• Binary Features (Gender, Ever_Married, Graduated): Missing values were filled with the most frequent value, followed by ordinal encoding.
• Ordinal Features (Spending_Score): Encoded as Low/Average/High, shifted by 1 using a custom ShiftByOne transformer, and scaled.
• Categorical Features (Profession, Var_1): Missing values were labeled as 'unknown' and then one-hot encoded.

3. Baseline Models and the "Data Ceiling" Insight

To understand the data, a baseline floor was established using a DummyClassifier (scoring ~25% accuracy). A Logistic Regression model was then trained, achieving ~51% cross-validated accuracy.

Key Insight: The Class B and C Overlap Instead of just looking at the final score, the logistic regression weight matrix was extracted and analyzed. The weights revealed that Segments B and C shared nearly identical demographic trajectories. This parallel feature importance proved that the data had an intrinsic "ceiling." The demographic features simply did not contain enough signal to perfectly separate Class B from Class C.

To prove this, a structural experiment was run: Classes B and C were temporarily combined into a single "BC" class. When predicting 3 classes instead of 4, the model's accuracy immediately jumped to ~63%. This confirmed the overlap. However, to meet real-world business constraints, the final model reverted to predicting all 4 original classes.

4. XGBoost and Heavy Regularization

Because the data ceiling was mathematically proven, running standard hyperparameter tuning to chase an 80%+ accuracy score would only lead to severe overfitting.

An XGBoost model was selected, but the tuning process (using RandomizedSearchCV and GridSearchCV) focused entirely on variance control and regularization. The final parameters included:

• max_depth=3: Shallow trees prevented the model from memorizing noisy data pockets.
• min_child_weight=9 to 16: Forced the model to be conservative and only make splits when backed by enough data.
• colsample_bytree=0.4: Forced trees to look at random subsets of features, breaking the reliance on overlapping features.
• learning_rate=0.05: Slowed down the learning process for stable convergence.

This heavy regularization successfully stopped the model from wildly overfitting the training data, anchoring its performance to a stable, realistic expectation for production.

5. Custom Engineering for Production

Standard scikit-learn tools were not enough for specific project needs, so two custom classes were engineered:

• average_precision_ovr: A zero-latency, multiclass scorer built to dynamically align targets and compute Average Precision seamlessly inside the cross-validation loops.
• ThresholdWrapper: A scikit-learn compatible meta-estimator (BaseEstimator, ClassifierMixin). By default, models pick the class with the highest probability. This wrapper allows a user to pass a dictionary (e.g., {'B': 0.35}) to manually lower the threshold for a specific segment. It uses fast NumPy vectorization to apply business rules safely without needing to retrain the model or cause data pipeline errors.

6. Evaluation Strategy: Handling Unlabeled Test Data

The provided test set for this project did not contain target labels, mimicking a blind deployment or competition setting.

Because the test data could not be scored directly, the model's generalization performance was rigorously evaluated using Out-of-Fold Stratified Cross-Validation (cv=3). Every row of the training data acted as an unseen test point at least once.

Correcting Optimization Bias: When testing the custom ThresholdWrapper (e.g., lowering Class B's threshold to capture more volume), evaluating the results on the standard training set caused inflated scores (optimization bias). To fix this and get honest metrics, the custom threshold logic was applied directly to the pristine, out-of-fold probability matrix generated by cross_val_predict. This ensured the final performance metrics reported were mathematically sound and reflected true real-world capability.