A collection of hands-on data science projects covering Data Extraction, Cleaning, Manipulation, Exploratory Data Analysis, Dimensionality Reduction, Supervised & Unsupervised Machine Learning, and Time-Series Analysis.
- Overview
- Skills & Competencies
- Projects
- 1. Cafe Sales Data Cleaning & Time-Series Analysis
- 2. Multi-Source Data Merging & Inconsistency Rectification
- 3. Iris Dataset — Structured Q&A Exploratory Data Analysis
- 4. Multi-Dataset EDA — Iris, Wine & Breast Cancer
- 5. Fashion-MNIST — EDA & Dimensionality Reduction
- 6. Decision Tree Classification & Feature Selection
- 7. K-Means Clustering Analysis
- 8. Agglomerative (Hierarchical) Clustering
- 9. KNN Classification & Time-Series Analysis
- Tech Stack
- Repository Structure
- Getting Started
- Key Takeaways
This repository documents my data science learning journey — from raw data wrangling to building and evaluating machine learning models. Each project tackles real-world data challenges using Python's scientific computing ecosystem, demonstrating a progression from foundational data manipulation through advanced ML pipelines.
- Reading structured data from CSV and Excel (
.xlsx) files using Pandas - Fetching benchmark ML datasets via
sklearn.datasetsand OpenML - Image loading and vectorization using OpenCV (
cv2) - Multi-source data consolidation from separate files
- Missing value profiling and systematic imputation strategies
- Rule-based algebraic imputation (e.g.,
Total = Quantity × Price) - Dynamic lookup-table construction for categorical reconstruction
- Error marker normalization (
'UNKNOWN','ERROR'→NaN) - Multi-source deduplication with completeness-ranked backfilling
- String standardization across locations, payment methods, and card providers
- Geospatial validation (Location ↔ Pincode consistency)
- Forward filling for time-series date propagation
- Multi-level GroupBy aggregations and pivoting
- Dataset concatenation, merging, and deduplication
- Datetime parsing and feature extraction (
day,month,year) - Min-Max normalization and feature scaling
- Business rule enforcement (discount tiers, positive-spend validation)
- Univariate, bivariate, and multivariate distribution analysis
- Descriptive & inferential statistics (mean, median, IQR, skewness)
- Correlation & covariance matrix analysis
- Class-wise profiling and variance contribution analysis
- Multicollinearity detection among redundant features
- Outlier detection using IQR method
- Matplotlib: Line charts, bar plots, histograms, image grids
- Seaborn: Pair plots, KDE density curves, violin plots, boxplots, correlation heatmaps, annotated confusion matrices
- Dendrogram tree visualizations (SciPy)
- Cumulative explained variance curves
- 2D/3D latent space scatter plots (PCA/SVD projections)
- Principal Component Analysis (PCA) — variance-preserving compression
- Truncated SVD — linear latent projections
- Variance Thresholding — removing zero/low-variance features
- Cumulative variance analysis for optimal component selection
- High-dimensional image reduction (784 → 187 features retaining 95% variance)
- Decision Tree Classifier (Gini Impurity & Entropy criteria)
- K-Nearest Neighbors (KNN) with distance metric tuning
- Hyperparameter Tuning via
GridSearchCV - Feature Selection:
SelectKBest(ANOVA F-test), Recursive Feature Elimination (RFE),SelectFromModel - Tree pruning for overfitting mitigation
- Classification evaluation: Precision, Recall, F1-Score, Confusion Matrices
- K-Means Clustering with Elbow Method optimization
- Agglomerative Hierarchical Clustering (Ward, Complete, Average, Single linkage)
- Multiple distance metrics (Euclidean, Manhattan, Cosine)
- Clustering validation: Silhouette Score, Davies-Bouldin Index, NMI, Fowlkes-Mallows Index
- Cluster stability analysis across varying data splits
- Temporal trend visualization and daily sales aggregation
- Multi-sensor overlay plotting (Temperature, Light, CO₂ vs. Occupancy)
- Seasonal decomposition using
statsmodels - Forward-fill imputation for temporal sequences
📁 Path: Data Cleaning/DataCleaning.ipynb
| Attribute | Details |
|---|---|
| Dataset | dirty_cafe_sales.csv — ~10,000 café transactions with extensive data quality issues |
| Objective | Clean corrupted sales data, impute missing values using domain rules, and analyze daily sales trends |
| Techniques | Error marker normalization, algebraic imputation (S = Q × P), lookup-table mapping, forward-fill date propagation, GroupBy aggregation |
| Visualization | Matplotlib line chart — "Day Wise Spent" across all 365 days of 2023 |
| Key Finding | Daily sales ranged from ₹102.5 to ₹406.0 across 2023 after cleaning |
Skills Demonstrated: Data profiling • Rule-based imputation • Datetime feature engineering • Time-series aggregation & visualization
📁 Path: Data Extraction, Cleaning and Manipulation/DataCleaning_Manipulation and Extraction from 2 source files.ipynb
| Attribute | Details |
|---|---|
| Datasets | inconsistency_B1.xlsx (1,014 rows) + inconsistency_b2.xlsx (3,068 rows) |
| Objective | Merge two Excel sources, resolve duplicates via completeness ranking, fix pricing/location/payment inconsistencies |
| Techniques | Completeness-ranked backfilling, smart deduplication (4,082 → 3,068 rows), location-pincode validation, card provider & payment method standardization, discount tier enforcement |
| Key Finding | Intelligent deduplication preserved all valid information while eliminating 1,014 duplicate transaction records |
Skills Demonstrated: Multi-source ETL • Enterprise-grade string cleaning • Geospatial validation • Business rule enforcement
📁 Path: IRIS_EDA.ipynb
| Attribute | Details |
|---|---|
| Dataset | Iris — 150 samples, 4 features, 3 species |
| Objective | Comprehensive EDA through 18 structured analytical questions |
| Techniques | Descriptive statistics, GroupBy profiling, correlation analysis, distribution skewness evaluation |
| Visualizations | Histograms, KDE curves, boxplots, violin plots, pair plots, correlation heatmaps, scatter plots |
| Key Finding | Petal length and petal width are perfectly correlated (r > 0.96) and provide complete linear separability for Iris-setosa |
Skills Demonstrated: Structured EDA methodology • Statistical profiling • Feature separability analysis
📁 Path: IRIS_WINE_BreastCancer_EDA.ipynb
| Attribute | Details |
|---|---|
| Datasets | Iris (150 × 4), Wine (178 × 13), Breast Cancer Wisconsin (569 × 30) |
| Objective | Comparative variance profiling, scaling requirement justification, and multicollinearity diagnosis |
| Techniques | Variance ratio analysis, skewness evaluation, IQR outlier detection, multicollinearity assessment |
| Key Finding | In Wine dataset, proline alone accounts for 99.77% of raw variance — proving StandardScaler is mandatory before PCA/KNN |
Skills Demonstrated: Multi-dataset comparative analysis • Feature scaling justification • Multicollinearity detection
📁 Path: FashionMNIST_EDA_DimensionalityReduction.ipynb
| Attribute | Details |
|---|---|
| Dataset | Fashion-MNIST — 70,000 images (28×28), 10 clothing categories |
| Objective | High-dimensional image EDA and dimensionality reduction benchmarking |
| Techniques | Variance Thresholding, PCA, Truncated SVD, cumulative variance analysis |
| Visualizations | Image grids, class frequency charts, cumulative variance curves, 2D PCA/SVD scatter plots |
| Key Finding | 187 principal components retain 95% variance — a 76% dimensionality reduction with negligible information loss |
Skills Demonstrated: High-dimensional data processing • PCA/SVD analysis • Latent space visualization
📁 Path: DecesionTree and FeatureSelection (Wine, Iris, Digits).ipynb
| Attribute | Details |
|---|---|
| Datasets | Iris (150 × 4), Wine (178 × 13), Digits (1,797 × 64) |
| Objective | Build interpretable classifiers, compare feature selection methods, optimize via hyperparameter tuning |
| Techniques | Decision Trees (Gini & Entropy), GridSearchCV, SelectKBest (ANOVA), RFE, SelectFromModel |
| Visualizations | Tree structure diagrams, feature importance bar charts, confusion matrix heatmaps |
| Key Finding | flavanoids, proline, and color_intensity provide >70% Gini impurity reduction in Wine classification |
Skills Demonstrated: Interpretable ML • Feature selection (Filter/Wrapper/Embedded) • Hyperparameter optimization • Overfitting mitigation
📁 Path: KMeans_Clustering_Analysis.ipynb
| Attribute | Details |
|---|---|
| Datasets | Make Moons, Make Circles (synthetic), Trash Classification (Kaggle — 9 waste classes, 64×64 images) |
| Objective | Evaluate K-Means across synthetic and real-world image data with comprehensive validation |
| Techniques | K-Means, Elbow Method, PCA (95% variance), Silhouette Score, Davies-Bouldin Index, NMI, Fowlkes-Mallows Index |
| Key Finding | K-Means fails on non-convex shapes (moons/circles) due to linear centroid-based partitioning; PCA accelerates high-dimensional image clustering |
Skills Demonstrated: Unsupervised learning pipelines • Cluster optimization • Internal & external validation metrics • Image vectorization
📁 Path: Clustering(Agglomerative ~ Hierarchial).ipynb
| Attribute | Details |
|---|---|
| Datasets | Make Moons, Make Circles (synthetic), Fashion-MNIST (70,000 images) |
| Objective | Benchmark hierarchical clustering across linkage criteria and distance metrics |
| Techniques | Agglomerative Clustering (Ward/Complete/Average/Single), Dendrograms, PCA, Silhouette/DBI/NMI/FMI metrics, stability analysis |
| Visualizations | Dendrogram trees, 2D cluster scatter plots, metric comparison line charts |
| Key Finding | Single linkage accurately captures non-convex boundaries (moons/circles); Ward linkage excels on globular clusters |
Skills Demonstrated: Hierarchical clustering • Dendrogram interpretation • Linkage & distance metric comparison • Cluster stability assessment
📁 Path: KNN and TimeSeries Analysis.ipynb
| Attribute | Details |
|---|---|
| Datasets | Digits (1,797 × 64), Fashion-MNIST (70,000 × 784), Occupancy Detection (8,143 sensor records) |
| Objective | Optimize KNN for image classification and analyze environmental sensor time-series data |
| Techniques | KNN (K-tuning, Manhattan distance, distance-weighted voting), PCA, seasonal decomposition, multi-sensor overlay plotting |
| Key Findings | 98.9% accuracy on Digits (K=7); 87.2% accuracy on Fashion-MNIST with PCA + Manhattan KNN; Light & CO₂ spikes strongly indicate room occupancy |
Skills Demonstrated: Instance-based learning • Curse of dimensionality mitigation • Time-series decomposition • Sensor data analysis
| Category | Tools & Libraries |
|---|---|
| Language | Python 3.x |
| Data Manipulation | Pandas, NumPy |
| Machine Learning | scikit-learn (Classification, Clustering, Feature Selection, Decomposition, Metrics) |
| Visualization | Matplotlib, Seaborn |
| Image Processing | OpenCV (cv2) |
| Time-Series | statsmodels (seasonal decomposition) |
| Statistical Analysis | SciPy (hierarchical clustering, dendrograms) |
| Environment | Jupyter Notebook, Google Colab |
Data_Science/
├── 📁 Data Cleaning/
│ ├── DataCleaning.ipynb # Cafe sales data cleaning & time-series analysis
│ └── dirty_cafe_sales.csv # Raw cafe sales dataset
│
├── 📁 Data Extraction, Cleaning and Manipulation/
│ ├── DataCleaning_Manipulation and Extraction from 2 source files.ipynb
│ ├── inconsistency_B1.xlsx # Transaction source file 1
│ └── inconsistency_b2.xlsx # Transaction source file 2
│
├── IRIS_EDA.ipynb # Iris Q&A exploratory data analysis
├── IRIS_WINE_BreastCancer_EDA.ipynb # Multi-dataset comparative EDA
├── FashionMNIST_EDA_DimensionalityReduction.ipynb # Fashion-MNIST EDA & PCA/SVD
├── DecesionTree and FeatureSelection (Wine, Iris, Digits).ipynb
├── KMeans_Clustering_Analysis.ipynb # K-Means clustering evaluation
├── Clustering(Agglomerative ~ Hierarchial).ipynb # Hierarchical clustering benchmark
├── KNN and TimeSeries Analysis.ipynb # KNN classification & time-series
└── README.md
pip install pandas numpy scikit-learn matplotlib seaborn opencv-python statsmodels openpyxl- Clone this repository:
git clone https://github.com/<your-username>/Data_Science.git cd Data_Science
- Launch Jupyter Notebook:
jupyter notebook
- Open any
.ipynbfile and run all cells.
Note: Some notebooks were originally developed in Google Colab and may contain
google.colab.drivemount calls. These can be safely removed when running locally — just update the file paths to your local directory.
| # | Insight | Project |
|---|---|---|
| 1 | Rule-based algebraic imputation (S = Q × P) is more accurate than statistical imputation when domain equations exist |
Cafe Sales Cleaning |
| 2 | Completeness-ranked backfilling preserves maximum information during multi-source deduplication | Multi-Source Merging |
| 3 | Petal measurements provide perfect linear separability for Iris-setosa (r > 0.96) | Iris EDA |
| 4 | Feature scaling (StandardScaler) is mandatory before PCA/KNN — proline alone accounts for 99.77% raw Wine variance |
Multi-Dataset EDA |
| 5 | PCA achieves 76% dimensionality reduction on Fashion-MNIST (784 → 187 features) retaining 95% variance | Fashion-MNIST |
| 6 | Tree depth pruning (max_depth=3-4) effectively prevents overfitting without accuracy loss |
Decision Trees |
| 7 | K-Means fails on non-convex shapes; Single-linkage Agglomerative Clustering succeeds | Clustering Analysis |
| 8 | Distance-weighted Manhattan KNN + PCA achieves 87.2% on Fashion-MNIST | KNN Classification |
| 9 | Environmental sensors (Light, CO₂) serve as strong predictive signals for occupancy detection | Time-Series Analysis |
📬 Feel free to explore, fork, and connect!
If you found this helpful, consider giving a ⭐