Porto Seguro's Safe Driver Prediction

Modular, production style project for the Porto Seguro's Safe Driver Prediction dataset. Contains EDA, notebooks, figures, training scripts and final artifacts.

Personal score: 0.28788
Highest score on Leaderboard: 0.29698

---

TLDR

Predictions on anonymised dataset of Porto Seguro's drivers. Spent a good amount of time looking at distributions of every feature. Gained preliminary understanding of feature importance by grouping column values by their target mean. Obtained confident feature importance after training model and checking it's feature importance. Settled to XGBoost after trying multiple models, which was more or less expected. After the creation of a few new features through engineering and several hours of tuning, was able to obtain the above results. A bunch of things that I tried did not help. These include ensembling, a few more columns that I engineered, and tuning a very large amount of parameters. Overall, a really fun project.

---

Objective

• Develop a robust probability-based model for binary classification on the selected dataset
• Learn handling and interpreting datasets with anonymized features, and understand their behaviour
• Apply effective data cleaning, feature engineering, preprocessing, and pipeline design for reliable model performance
• Try multiple models, hypothesize their performances and compare with true results
• Use visualizations to better understand the data and preserve key insights for future reference

---

Dataset

Porto Seguro's Safe Driver Prediction
Kaggle link: https://www.kaggle.com/competitions/porto-seguro-safe-driver-prediction/data
[I have not included the dataset in the repo. Kindly get it through Kaggle, or through various GitHub repos hosting the dataset]

---

Approach

General Approach taken:

1. Explore features and check basic performance on non imputed and imputed data
2. Train basic models to get a rough understanding of performances
3. Tune best performing model from step 2 to improve performance
4. Extract feature importance from tuned model, and try engineering new features from important features. Try various other preprocessing as well
5. Train on entire data once again to obtain best model

Cleaning

• Dropped all '_calc' columns.
• Dropped additional columns ['ps_ind_10_bin', 'ps_ind_13_bin', 'ps_ind_18_bin', 'ps_car_10_cat'] due to minimal meaningful contribution in prediction

Preprocessing

• Created new binary features using certain '_ind' features
• Created new categorical features using combinations of certain '_cat' and '_ind' features
• One Hot Encoding applied to categorical columns (helped slightly in improving performance)

Model

XGBoost trained using following parameters, obtained through extensive hyperparameter tuning using Optuna:
params = {
 "objective": "binary:logistic",
 'device':'cuda',
 'tree_method':'hist',
 'eval_metric':'auc',
 'max_depth': 3,
 'eta': 0.06443239542535328,
 'subsample': 0.7731240330866832,
 'colsample_bytree': 0.947824044178731,
 'colsample_bylevel': 0.9124205465696081,
 'colsample_bynode': 0.7234222779570076,
 'reg_alpha': 0.9699926009829554,
 'reg_lambda': 0.6172896412559273}

The model recieves input data that has been cleaned and processed by a preprocessing pipeline.

Models tried and evaluated with Optuna tuned hyperparameters

• Logistic Regression
• Random Forest
• AdaBoost
• Gradient Boosting (sklearn)
• XGBoost (final model)
• Model ensembling via bagging (specifically, averaging predictions from multiple XGBoost models).

Chose XGBoost to be the final model, since improvement in performance when using ensemble techniques was very small, and not worth the extra computation and resources.

Hyperparameter tuning

• Optuna for optimised and automated tuning
• Tuned multiple models and chose best performing model

Results

XGBoost had the highest performance

• Gini co-efficient on holdout set: 0.28632
• Gini co-efficient on Kaggle provided test data: 0.28788

Key Insights

• Engineering new features based on feature importance was very helpful beyond the 0.28 Gini range
• Leaving -1s as is performed much better than imputation of columns
• One Hot Encoding helped, but only slightly
• Ensembling multiple XGBoosters helped performance extremely slightly, hence was not used

Project Structure

src/       # Core pipeline and logic
datasets/      # train and test data
artifacts/      # saved model and preprocessor
notebooks/     # experimentation
reports/       # visualisations

How to Run

1. Get dataset from https://www.kaggle.com/competitions/porto-seguro-safe-driver-prediction/data or through any of the GitHub repos that host the dataset
2. Place train.csv and test.csv in datasets/ directory
   
3. Install dependencies:
   

pip install -r requirements.txt

4. Run program:
   

python main.py

5. Options:
   • Build and train model
   • Evaluate model on test data
   • Feed CSV files to obtain predictions

---

Reflections

Was pretty fun actually. Had a fun time tuning parameters for XGB, and the parallel coordinate plot is pretty satisfying to look at. Experimenting with different features to get new potential strong features was very fun and anticipatory too. Overall, enjoyed it. 10/10, would recommend! [ps: I've written this way too formally. I enjoy informal language with less jargon more actually. Whatever, maybe I'll use an informal tone for the next project]

---