Machine Learning Assignments

Consolidated coursework repository for the Machine Learning assignments. The repository collects the original notebooks, scripts, datasets, generated outputs, and previous per-assignment PDF reports in a clean structure that is easier to review, run, and maintain.

Main documentation: Machine_Learning_Assignments_Report.pdf

Author

• Mohammad Saleh Mahdinejad

Repository Overview

The final work is organized into twelve assignment folders:

Folder	Topic	Main artifacts
assignments/ml1-1-bias-variance-regularization	Bias, variance, overfitting, underfitting, and regularization	Original conceptual PDF report
assignments/ml1-2-real-estate-price-regression	Linear regression for house price prediction	Notebook, dataset, model outputs, figures, original PDF
assignments/ml1-3-mysterious-machine-polynomial-regression	Polynomial feature engineering and closed-form regression	Python solution, original PDF
assignments/ml1-4-gradient-descent-treasure-search	Batch GD, SGD, and Mini-Batch GD comparison	Notebook, generated figures/tables, original PDF
assignments/ml2-1-raisin-hinge-classifier	Binary linear classifier with hinge loss	Notebook, Raisin dataset, outputs, report builder, original PDF
assignments/ml2-2-penguins-logistic-regression	Binary logistic regression from scratch	Notebook, Penguins dataset, outputs, report builder, original PDF
assignments/ml2-3-balance-scale-ovr-logistic	Multiclass One-vs-Rest logistic regression	Notebook, Balance Scale dataset, outputs, report builder, original PDF
assignments/ml3-1-neural-network-concepts	Exercise 3 conceptual report	Original PDF report
assignments/ml3-2-neural-network-concepts	Exercise 3 conceptual report	Original PDF report
assignments/ml3-3-neural-network-concepts	Exercise 3 conceptual report	Original PDF report
assignments/ml3-4-iris-neural-network	Iris neural network from scratch	Python source, saved outputs, original PDF
assignments/ml-project-water-potability	Final water potability classification project	Notebook, dataset, generated outputs, final PDFs

Generated .zip archives and duplicate Word files were intentionally excluded from the repository. The source notebooks/scripts, datasets, final outputs, and PDF deliverables are preserved.

Setup

Create and activate a virtual environment, then install the required packages:

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

Launch Jupyter if you want to inspect or rerun the notebooks:

jupyter notebook

The notebooks were written around standard scientific Python tools: NumPy, pandas, matplotlib, seaborn, scikit-learn utilities for splitting data, and openpyxl for Excel input.

Assignment Details

ML1_1: Bias, Variance, and Regularization

This assignment is a conceptual analysis of model complexity and generalization. It explains why a model can achieve low training error while producing high test error, identifies that behavior as overfitting, and connects it to low bias and high variance.

The report also explains how regularization changes the objective function by adding a penalty term:

total loss = training loss + lambda * penalty

The main conclusion is that regularization normally reduces variance by limiting overly flexible weights, while potentially increasing bias if the penalty is too strong. The assignment also discusses the role of additional data: more data usually reduces variance, especially for flexible models, but it does not automatically fix high bias caused by an unsuitable model class.

ML1_2: Real Estate Price Regression

This project predicts house prices using a linear regression model implemented from scratch. The workflow covers:

• loading and profiling the house-prices dataset;
• identifying missing values in numerical columns;
• splitting the data before preprocessing to avoid information leakage;
• replacing missing values and outliers using training-set statistics;
• encoding categorical features such as brick construction and neighborhood;
• scaling numeric features;
• training linear regression with gradient descent;
• reporting RMSE, MAE, and R2 on train/test splits.

Key results:

Metric	Train	Test
RMSE	10,297.76	9,583.84
MAE	7,830.99	7,935.46
R2	0.8488	0.8883

The test R2 of 0.8883 indicates that the fitted model captures most of the target variance on the held-out test set. The output folder contains the correlation heatmap, categorical feature plots, regression plots, RMSE learning curve, preprocessing summaries, model coefficients, prediction samples, and train/test scaling outputs.

ML1_3: Mysterious Machine Polynomial Regression

This assignment reconstructs an unknown three-input function under the assumption that it is a third-degree polynomial. The code creates a 20-dimensional feature vector containing:

• a constant intercept term;
• first, second, and third powers of x, y, and z;
• pairwise interaction terms such as xy, xz, and yz;
• mixed terms such as x^2y, xy^2, x^2z, and yz^2;
• the three-way interaction xyz.

Training was performed with the closed-form normal equation:

W = (X^T X)^(-1) X^T y

The final submitted script keeps the learned weights embedded in the code and performs only inference. It reads x, y, and z, builds the same feature vector, computes the dot product with the learned weights, and applies the required ceiling-based two-decimal rounding rule.

Sample behavior from the original assignment:

Input (x, y, z)	Output
(5, 7, 8)	1313.3
(-10, -5, 2)	-503.32

ML1_4: Gradient Descent Treasure Search

This project compares three optimization strategies on a synthetic linear regression problem:

• Batch Gradient Descent;
• Stochastic Gradient Descent;
• Mini-Batch Gradient Descent.

The notebook generates 1,000 noisy samples from a linear relationship, defines the mean squared error objective, and records the parameter path, loss curve, number of updates, and execution time for each algorithm.

Final comparison:

Algorithm	w	b	Final MSE	Updates	Time (s)
Batch GD	2.0653	4.5868	0.2778	500	0.0207
SGD	2.0400	5.0138	0.2904	500,000	0.5864
Mini-Batch GD	2.0043	4.9797	0.2393	8,000	0.1782

Mini-Batch GD achieved the lowest final MSE in this run, while Batch GD required the fewest updates. SGD performed many more updates and showed noisier movement, which is expected because each update uses a single sample.

ML2_1: Raisin Binary Linear Classifier with Hinge Loss

This assignment implements a binary linear classifier for the Raisin dataset using hinge loss and mini-batch gradient descent. The workflow includes:

• loading 900 raisin samples from Excel;
• encoding the two classes, Kecimen and Besni;
• splitting data into training, validation, and test partitions;
• standardizing features using training statistics only;
• implementing hinge loss and subgradients from scratch;
• evaluating accuracy, precision, recall, F1-score, and confusion matrix;
• analyzing misclassified samples;
• training a two-feature visualization model using Area and Perimeter.

Test results:

Metric	Value
Accuracy	0.9037
Precision	0.9091
Recall	0.8955
F1-score	0.9023
Misclassified test samples	13
2D visualization model accuracy	0.8889

The final confusion matrix is:

[[62,  6],
 [ 7, 60]]

ML2_2: Penguins Logistic Regression

This project implements binary logistic regression from scratch for species classification on the Penguins dataset. The task filters the dataset to two species, drops rows with missing values, standardizes features, searches over learning rate and regularization strength, and evaluates the final model on a held-out test set.

Features used:

• bill_length_mm
• bill_depth_mm
• flipper_length_mm
• body_mass_g

Best validation settings:

Hyperparameter	Value
Learning rate	0.1
Lambda	0.0

Test results:

Metric	Value
Accuracy	1.0000
Precision	1.0000
Recall	1.0000
F1-score	1.0000

The confusion matrix shows no test errors:

[[31, 0],
 [ 0, 24]]

The project also studies the effect of regularization and hyperparameter selection through validation loss curves.

ML2_3: Balance Scale One-vs-Rest Logistic Regression

This assignment extends logistic regression to a three-class problem using the One-vs-Rest strategy. The model predicts whether a balance scale tips left, is balanced, or tips right from four integer-valued physical features:

• left weight;
• left distance;
• right weight;
• right distance.

The workflow includes label encoding, a torque-based auxiliary analysis, train/validation/test splitting, standardization using training statistics, hyperparameter search, multiclass evaluation, and error analysis.

Best settings and final results:

Metric	Value
Best learning rate	0.2
Best lambda	0.0
Best epochs	5,000
Test accuracy	0.9280
Test macro-F1	0.8620
Misclassified test samples	9

Confusion matrix:

[[53, 5, 0],
 [ 1, 9, 0],
 [ 0, 3, 54]]

The most difficult class is the balanced class because it occupies a narrow decision region between the left-heavy and right-heavy classes. This is reflected in the macro-F1 score and the confusion matrix.

ML3_1 to ML3_3: Neural-Network Theory Reports

These three folders preserve the conceptual PDF deliverables for Exercise 3. They cover the theoretical and analytical parts of the assignment series and are kept as standalone reports because the source material is document-based rather than notebook-based.

ML3_4: Iris Neural Network from Scratch

This project implements a small feed-forward neural network for Iris classification. The model uses four Iris features, one hidden layer, tanh hidden activation, softmax output, and cross-entropy loss.

Final setup:

Architecture: 4 -> 12 -> 3
Learning rate: 0.05
Epochs: 1500
Train/Test split: 120 / 30

Final results:

Metric	Value
Final train loss	0.0554
Final train accuracy	0.9750
Test loss	0.0735
Test accuracy	0.9667

The final confusion matrix contains one test error:

[[10, 0, 0],
 [ 0, 9, 1],
 [ 0, 0, 10]]

Final Project: Water Potability Classification

The final project evaluates water potability classification on a dataset with 3,276 samples, nine original input features, and 1,434 missing values. The workflow includes manual median imputation, manual standard scaling for scale-sensitive models, feature engineering, stratified train/test splitting, cross-validation, and final test-set evaluation.

All required models and two bonus models were implemented from scratch without importing ready-made model classes:

• Logistic Regression
• KNN
• SVM with RBF kernel
• Decision Tree
• Gradient Boosting
• AdaBoost
• Random Forest
• MLP Classifier
• Extra Trees
• Gaussian Naive Bayes

Key final results:

Selection criterion	Model	Value
Best F1	MLP Classifier	0.5516
Best accuracy	Decision Tree	0.6601
Best ROC-AUC	MLP Classifier	0.6575
Safety recommendation	Decision Tree	dangerous FP rate 0.1425

The report discusses why model selection differs by metric: MLP gives the best F1 and ROC-AUC, while Decision Tree is the recommended safety-oriented model because it has the lowest dangerous false-positive rate among competitive models.

Output and Documentation Policy

The repository tracks:

• source notebooks and scripts;
• original input datasets;
• generated output figures, tables, JSON summaries, and text summaries;
• per-assignment PDF reports;
• the consolidated root PDF report.

The LaTeX source for the consolidated report and the copied figure assets used only for building that report are kept locally under docs/ and ignored by Git, matching the requested repository cleanup policy.

Suggested Review Order

1. Open Machine_Learning_Assignments_Report.pdf for the complete written documentation.
2. Inspect each assignment notebook under assignments/*/notebooks.
3. Compare the generated metrics and figures under assignments/*/outputs.
4. Use the per-assignment PDFs under assignments/*/reports for the original submitted writeups.