Machine Learning Workflow: From Data to Predictions

Guide

Table of contents

• Introduction
• Understanding the Big Picture
• Part 1: Understanding ML Types and Tasks
  • A. ML Learning Paradigms (Definition 1)
  • B. ML Learning Paradigms (Definition 2)
  • C. Common ML Tasks
• Part 2: Core ML Components
  • A. Models (Algorithms)
  • B. Objective Functions (Loss Functions)
  • C. Learning Algorithms (Optimizers)
• Part 3: Model Development Workflow
• Part 4: ML Validation Strategies
• Part 5: Evaluation & Performance Metrics
• Part 6: Common ML Problems and Solutions
• Part 7: ML Libraries and Tools
• Part 8: Complete ML Workflow Example
• Summary Table: Quick Reference
• Resources

Introduction

Machine Learning is a systematic process that transforms raw data into predictive models and actionable insights. This comprehensive guide covers the entire ML workflow: from problem definition through data processing, model selection, training, evaluation, and deployment.

Understanding the Big Picture

The Complete ML Workflow

1. Problem Definition
   • Define objective
   • Identify ML type (supervised/unsupervised/etc.)
   • Choose task (classification/regression/etc.)
2. Data Collection & Exploration
   • Gather data from sources
   • Understand data distribution
   • Identify challenges
3. Data Processing
   • Cleaning & Preprocessing
   • Feature Engineering
   • [See Data Processing category for details]
4. Data Splitting
   • Training set (60-70%)
   • Validation set (10-15%)
   • Test set (15-20%)
5. Model Selection
   • Choose algorithm
   • Initialize hyperparameters
   • Set objective function
6. Model Training
   • Fit model to training data
   • Monitor validation performance
   • Tune hyperparameters
7. Evaluation & Validation
   • Test on held-out test set
   • Calculate performance metrics
   • Domain-specific metrics
8. Deploy & Monitor
   • Deploy to production
   • Monitor performance
   • Iterate if needed

---

Part 1: Understanding ML Types and Tasks

A. ML Learning Paradigms (Definition 1)

Supervised Learning

Learns from labeled data where each example has a target outcome.

When to use: When you have labeled data and want to predict specific outcomes

# Example: House price prediction
X_train = [
    [100, 2, 1],  # sq_ft, bedrooms, bathrooms
    [150, 3, 2],
    [200, 4, 2]
]
y_train = [200000, 350000, 500000]  # LABELS - house prices

model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict([[120, 2, 1]])  # Predict price for new house

Advantages:

• Direct feedback on correctness
• High accuracy with sufficient data
• Well-understood algorithms

Disadvantages:

• Requires labeled data (expensive)
• Doesn’t work for unlabeled data

Unsupervised Learning

Finds patterns in unlabeled data without predefined targets.

When to use: When you need to discover hidden patterns or group similar items

# Example: Customer segmentation
X = [
    [100, 25],  # spending, age (NO LABELS)
    [150, 30],
    [80, 22]
]

model = KMeans(n_clusters=3)
labels = model.fit_predict(X)  # Discover groups automatically

Advantages:

• Works with unlabeled data
• Discovers unexpected patterns
• Exploratory analysis

Disadvantages:

• No ground truth for evaluation
• Harder to validate quality
• Results may be meaningless

Semi-Supervised Learning

Uses small labeled dataset + large unlabeled dataset.

When to use: When labeling is expensive but you have some labels

# Example: Email classification with few labels
X_labeled = emails[:100]  # 100 labeled emails
y_labeled = labels[:100]

X_unlabeled = emails[100:]  # 10,000 unlabeled emails

model = LabelPropagation()
model.fit(X_labeled, y_labeled, X_unlabeled)

Use cases: Medical imaging, NLP tasks, fraud detection

Self-Supervised Learning

Uses data itself to generate labels through pretext tasks.

When to use: When you have massive unlabeled data

# Example: Image rotation prediction (pretext task)
# Original task: Classify if image is rotated 0°, 90°, 180°, 270°
# This trains encoder to understand image structure
# Then fine-tune on actual task with few labels

B. ML Learning Paradigms (Definition 2)

Reinforcement Learning

Learns through interactions with environment and rewards.

Agent takes Action → Environment responds with Reward
Agent learns to maximize cumulative rewards

Example: Game AI

# Pseudo-code
agent = DQNAgent()
for episode in range(1000):
    state = environment.reset()
    for step in range(100):
        action = agent.act(state)  # Choose action
        next_state, reward = environment.step(action)  # Get reward
        agent.remember(state, action, reward, next_state)  # Learn

Transfer Learning

Uses knowledge from one task to improve learning on another task.

# Example: Image classification using pre-trained ImageNet model
base_model = models.ResNet50(weights='imagenet')
base_model.trainable = False  # Freeze pre-trained weights

model = Sequential([
    base_model,
    Dense(256, activation='relu'),
    Dense(10, activation='softmax')  # New task: 10 classes
])

model.fit(X_train, y_train)  # Train only new layers

Active Learning

Intelligently selects which samples to label next.

Strategy: Label samples that model is most uncertain about

# Query the most uncertain predictions
uncertainty = model.predict_proba(X_unlabeled).max(axis=1)
most_uncertain_idx = np.argsort(uncertainty)[:10]  # 10 most uncertain

# Have human label these 10 samples
# Add to training set and retrain

Continual Learning

Learns from data streams without forgetting previous knowledge.

Challenge: Catastrophic forgetting when learning new task

# Learn task 1
model.fit(X_task1, y_task1)

# Learn task 2 (without forgetting task 1)
# Use replay buffer and regularization
model.fit(X_task2, y_task2, replay_buffer=buffer)

C. Common ML Tasks

Supervised Learning Tasks

Task	Input	Output	Example
Classification	Features	Class label	Email spam/not spam
Regression	Features	Continuous value	House price prediction
Recognition	Image	Object identity	Face recognition
Detection	Image	Objects + locations	Vehicle detection in video
Segmentation	Image	Pixel-wise labels	Medical image segmentation
Time Series	Sequences	Future values	Stock price prediction

Unsupervised Learning Tasks

Task	Input	Output	Example
Clustering	Features	Group assignments	Customer segmentation
Dimensionality Reduction	High-dim data	Low-dim representation	Visualization (PCA)
Outlier Detection	Features	Anomaly labels	Fraud detection
Association	Transactions	Rules	Market basket analysis

---

Part 2: Core ML Components

A. Models (Algorithms)

from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier, GradientBoostingRegressor
from sklearn.cluster import KMeans
import tensorflow as tf

Linear Regression

Predicts continuous values using linear relationship.

ŷ = β₀ + β₁x₁ + β₂x₂ + ... + βₙxₙ

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X_train, y_train)

# Coefficients
print(f"Intercept: {model.intercept_}")
print(f"Slopes: {model.coef_}")

# Predictions
predictions = model.predict(X_test)

Use when: Linear relationship, interpretability important

Logistic Regression

Classification using sigmoid function.

P(y=1|x) = 1 / (1 + e^(-z))  where z = β₀ + β₁x₁ + ...

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
model.fit(X_train, y_train)

# Probability predictions
proba = model.predict_proba(X_test)  # Returns [P(class=0), P(class=1)]

# Class predictions
predictions = model.predict(X_test)

Use when: Binary/multiclass classification, probability needed

Decision Tree

Hierarchical rules for classification/regression.

from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(max_depth=5)
model.fit(X_train, y_train)

# Visualize tree
from sklearn import tree
tree.plot_tree(model)

Use when: Non-linear relationships, interpretability needed

Limitations: Prone to overfitting, unstable

Support Vector Machine (SVM)

Finds optimal hyperplane to separate classes.

from sklearn.svm import SVC

model = SVC(kernel='rbf', C=1.0)
model.fit(X_train, y_train)

predictions = model.predict(X_test)

Use when: High-dimensional data, clear separation exists

Note: Slower on large datasets

Random Forest

Ensemble of decision trees voting for prediction.

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(
    n_estimators=100,     # Number of trees
    max_depth=10,
    min_samples_split=5
)
model.fit(X_train, y_train)

# Feature importance
importances = model.feature_importances_

Use when: Mixed feature types, feature importance needed

Advantages: Handles non-linearity, reduces overfitting, fast inference

B. Objective Functions (Loss Functions)

The objective function defines what the model tries to minimize during training.

For Regression

Mean Squared Error (MSE / L2):

MSE = (1/n) Σ(y_true - y_pred)²

from sklearn.metrics import mean_squared_error

mse = mean_squared_error(y_true, y_pred)

Use when: You want to penalize large errors more

Mean Absolute Error (MAE / L1):

MAE = (1/n) Σ|y_true - y_pred|

from sklearn.metrics import mean_absolute_error

mae = mean_absolute_error(y_true, y_pred)

Use when: You want to penalize all errors equally (robust to outliers)

Huber Loss:

Combines MSE and MAE - quadratic for small errors, linear for large

from sklearn.metrics import mean_squared_log_error

# Smooth transition between L1 and L2

For Classification

Cross-Entropy (Log Loss):

CE = -Σ y_true · log(y_pred)

from sklearn.metrics import log_loss

loss = log_loss(y_true, y_pred_proba)

Use when: Probability predictions matter, imbalanced classes

Hinge Loss:

Used in SVM: max(0, 1 - y_true · y_pred)

Kullback-Leibler Divergence (KL Divergence):

Measures difference between probability distributions

C. Learning Algorithms (Optimizers)

Algorithms that adjust model parameters to minimize loss.

import tensorflow as tf

# Common optimizers
sgd = tf.keras.optimizers.SGD(learning_rate=0.01)
adam = tf.keras.optimizers.Adam(learning_rate=0.001)
rmsprop = tf.keras.optimizers.RMSprop(learning_rate=0.001)

# Compile model with optimizer and loss
model.compile(optimizer='adam', loss='categorical_crossentropy')

Optimizer	Characteristics	Best For
SGD	Simple, stable	Small datasets
Momentum	Faster convergence	Faster training
Adam	Adaptive learning rates	Most tasks (default)
RMSprop	Handles sparse gradients	RNNs

---

Part 3: Model Development Workflow

Step 1: Data Preparation

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Load data
df = pd.read_csv('data.csv')

# For detailed preprocessing steps, refer to:
# Category: Data Engineering → Tabular Data Preprocessing and Post-Processing

# Split into features and target
X = df.drop('target', axis=1)
y = df['target']

# Split into train/val/test (60%, 20%, 20%)
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, random_state=42
)

# Normalize features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_val_scaled = scaler.transform(X_val)
X_test_scaled = scaler.transform(X_test)

print(f"Training: {len(X_train)}")
print(f"Validation: {len(X_val)}")
print(f"Test: {len(X_test)}")

Step 2: Model Selection

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC

# Start with simple model (baseline)
baseline_model = LogisticRegression()

# Try more complex models
model1 = RandomForestClassifier(n_estimators=100)
model2 = SVC(kernel='rbf')
model3 = GradientBoostingClassifier()

# Compare on validation set
models = [baseline_model, model1, model2, model3]
for model in models:
    model.fit(X_train_scaled, y_train)
    val_score = model.score(X_val_scaled, y_val)
    print(f"{model.__class__.__name__}: {val_score:.4f}")

Model Selection Strategy:

1. Start simple (linear model)
2. Increase complexity gradually
3. Monitor validation performance
4. Choose based on bias-variance tradeoff

Step 3: Hyperparameter Tuning

Hyperparameters control model behavior before training.

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

# Define parameter grid
param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [5, 10, 15, None],
    'min_samples_split': [2, 5, 10],
    'min_samples_leaf': [1, 2, 4]
}

# Grid Search: tries all combinations
grid_search = GridSearchCV(
    RandomForestClassifier(),
    param_grid,
    cv=5,  # 5-fold cross-validation
    scoring='accuracy',
    n_jobs=-1  # Use all CPUs
)

grid_search.fit(X_train_scaled, y_train)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV score: {grid_search.best_score_:.4f}")

# Use best model
best_model = grid_search.best_estimator_

Random Search (faster for large spaces):

random_search = RandomizedSearchCV(
    RandomForestClassifier(),
    param_grid,
    n_iter=20,  # Try 20 random combinations
    cv=5,
    random_state=42
)

Step 4: Training

# Train on full training set
best_model.fit(X_train_scaled, y_train)

# For neural networks - monitor training
from tensorflow import keras

history = model.fit(
    X_train_scaled, y_train,
    validation_data=(X_val_scaled, y_val),
    epochs=100,
    batch_size=32,
    callbacks=[
        keras.callbacks.EarlyStopping(
            monitor='val_loss',
            patience=10,  # Stop if no improvement for 10 epochs
            restore_best_weights=True
        )
    ]
)

# Plot training history
import matplotlib.pyplot as plt
plt.plot(history.history['loss'], label='Train Loss')
plt.plot(history.history['val_loss'], label='Val Loss')
plt.legend()
plt.show()

---

Part 4: ML Validation Strategies

A. Data Validation

Types of Validation

1. Data Engineering Validations

Checks data integrity and quality.

def validate_data_quality(df):
    """Check for data quality issues"""
    
    issues = []
    
    # Missing values
    if df.isnull().sum().sum() > 0:
        issues.append(f"Missing values: {df.isnull().sum()}")
    
    # Duplicates
    if df.duplicated().sum() > 0:
        issues.append(f"Duplicate rows: {df.duplicated().sum()}")
    
    # Data type issues
    for col in df.columns:
        if df[col].dtype == 'object':
            # Check for non-numeric in numeric column
            pass
    
    # Range checks
    for col in df.select_dtypes(include=[np.number]).columns:
        if (df[col] < 0).any():
            issues.append(f"{col} has negative values")
    
    return issues

issues = validate_data_quality(df)
for issue in issues:
    print(f" {issue}")

2. ML-based Data Validations

Uses ML to detect invalid data patterns.

# Anomaly detection on new data
from sklearn.ensemble import IsolationForest

anomaly_detector = IsolationForest(contamination=0.05)
anomalies = anomaly_detector.fit_predict(X_train) == -1

# Flag anomalies in incoming data
suspicious_samples = anomaly_detector.predict(X_new) == -1
print(f"Suspicious new samples: {suspicious_samples.sum()}")

B. ML Training Validation

1. Hold-out Validation

Simple train/test split (already covered above).

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

Limitation: High variance, only uses 80% for training

2. K-Fold Cross-Validation

Splits data into K folds and trains K times.

Fold 1: Train on [2,3,4,5] → Test on [1]
Fold 2: Train on [1,3,4,5] → Test on [2]
Fold 3: Train on [1,2,4,5] → Test on [3]
Fold 4: Train on [1,2,3,5] → Test on [4]
Fold 5: Train on [1,2,3,4] → Test on [5]

Average all 5 test scores

from sklearn.model_selection import cross_val_score

scores = cross_val_score(
    RandomForestClassifier(),
    X, y,
    cv=5,  # 5-fold cross-validation
    scoring='accuracy'
)

print(f"Fold scores: {scores}")
print(f"Mean: {scores.mean():.4f} (+/- {scores.std():.4f})")

Advantages: Uses all data for training, lower variance

Disadvantages: More expensive (K times slower)

3. Leave-One-Out Cross-Validation (LOOCV)

Special case: K = N (number of samples)

from sklearn.model_selection import LeaveOneOut

loo = LeaveOneOut()
scores = cross_val_score(model, X, y, cv=loo)

Advantage: Maximum utilization of data

Disadvantage: Very expensive (N models trained)

4. Stratified K-Fold (for imbalanced data)

Maintains class distribution in each fold.

from sklearn.model_selection import StratifiedKFold

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=skf)

Use when: Classes are imbalanced

5. Bootstrapping

Random sampling with replacement.

from sklearn.utils import resample

for i in range(100):
    # Sample with replacement
    X_boot, y_boot = resample(X, y)
    
    # Train model
    model.fit(X_boot, y_boot)
    
    # Evaluate on out-of-bag samples (not in this bootstrap)

C. Feature Validation

Feature Importance & Selection:

# Check which features matter
importances = model.feature_importances_

# Remove low-importance features
threshold = np.percentile(importances, 20)
selected_features = importances > threshold

X_selected = X[:, selected_features]

Feature Stability:

# Train with different random seeds
scores = []
for seed in range(5):
    model = RandomForestClassifier(random_state=seed)
    score = cross_val_score(model, X, y, cv=5).mean()
    scores.append(score)

print(f"Score stability: mean={np.mean(scores):.4f}, std={np.std(scores):.4f}")

---

Part 5: Evaluation & Performance Metrics

Classification Metrics

from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    confusion_matrix, roc_auc_score, roc_curve
)

# Predictions
y_pred = model.predict(X_test)
y_pred_proba = model.predict_proba(X_test)[:, 1]

# Basic metrics
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred)

print(f"Accuracy:  {accuracy:.4f}")   # (TP+TN)/(Total)
print(f"Precision: {precision:.4f}")  # TP/(TP+FP) - Of predicted positive, how many correct?
print(f"Recall:    {recall:.4f}")     # TP/(TP+FN) - Of actual positive, how many found?
print(f"F1-Score:  {f1:.4f}")         # Harmonic mean of precision & recall

# Confusion Matrix
cm = confusion_matrix(y_test, y_pred)
print(f"\nConfusion Matrix:\n{cm}")
#        Predicted
#        Neg  Pos
# Actual Neg [TN  FP]
#        Pos [FN  TP]

# ROC-AUC for probability threshold
auc = roc_auc_score(y_test, y_pred_proba)
print(f"ROC-AUC: {auc:.4f}")

# ROC Curve
fpr, tpr, thresholds = roc_curve(y_test, y_pred_proba)
plt.plot(fpr, tpr, label=f'ROC (AUC = {auc:.3f})')
plt.plot([0, 1], [0, 1], 'k--', label='Random')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend()
plt.show()

Metric Selection Guide:

Metric	Use Case	Formula
Accuracy	Balanced classes	(TP+TN)/Total
Precision	Minimize false positives	TP/(TP+FP)
Recall	Minimize false negatives	TP/(TP+FN)
F1-Score	Balance precision & recall	2·P·R/(P+R)
ROC-AUC	All thresholds	Area under ROC curve
PR-AUC	Imbalanced data	Area under P-R curve

Regression Metrics

from sklearn.metrics import (
    mean_squared_error, mean_absolute_error, r2_score,
    mean_absolute_percentage_error
)

y_pred = model.predict(X_test)

# Error metrics
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_test, y_pred)
mape = mean_absolute_percentage_error(y_test, y_pred)

print(f"MSE:  {mse:.4f}")
print(f"RMSE: {rmse:.4f}")
print(f"MAE:  {mae:.4f}")
print(f"MAPE: {mape:.4f}")

# Goodness of fit
r2 = r2_score(y_test, y_pred)
print(f"R²-Score: {r2:.4f}")  # 1.0 = perfect fit, 0 = baseline

Domain-Specific Metrics

# Medical imaging: Sensitivity/Specificity
sensitivity = recall_score(y_test, y_pred)  # True positive rate
specificity = cm[0, 0] / (cm[0, 0] + cm[0, 1])  # True negative rate

# Information Retrieval: Precision@K
from sklearn.metrics import ndcg_score
ndcg = ndcg_score(y_test, y_pred_proba)

# NLP: BLEU score
from nltk.translate.bleu_score import corpus_bleu
bleu = corpus_bleu([[ref]], [hyp])

# Time Series: MAPE
mape = np.mean(np.abs((y_test - y_pred) / y_test)) * 100

---

Part 6: Common ML Problems and Solutions

1. Overfitting

Problem: Model memorizes training data, poor generalization

# Symptom: High train accuracy, low test accuracy
train_score = model.score(X_train, y_train)  # 0.99
test_score = model.score(X_test, y_test)     # 0.70
print(f"Overfitting gap: {train_score - test_score:.2f}")

Visualizing overfitting:

# Plot learning curves
from sklearn.model_selection import learning_curve

sizes, train_scores, val_scores = learning_curve(
    model, X, y, cv=5, scoring='accuracy'
)

train_mean = np.mean(train_scores, axis=1)
val_mean = np.mean(val_scores, axis=1)

plt.plot(sizes, train_mean, label='Train')
plt.plot(sizes, val_mean, label='Validation')
plt.xlabel('Training Set Size')
plt.ylabel('Accuracy')
plt.legend()
plt.show()
# Overfitting: Train curve ↑ high, Val curve ↓ low, gap widens

Solutions:

# 1. More data
# 2. Regularization (L1/L2)
model = LogisticRegression(C=0.1, penalty='l2')  # Smaller C = more regularization

# 3. Early stopping (neural networks)
model.fit(X_train, y_train, early_stopping=True, validation_fraction=0.2)

# 4. Dropout (neural networks)
model = Sequential([
    Dense(128, activation='relu'),
    Dropout(0.5),  # Drop 50% of neurons
    Dense(64, activation='relu'),
    Dropout(0.5),
    Dense(10, activation='softmax')
])

# 5. Simplify model
model = RandomForestClassifier(
    max_depth=5,                # Limit depth
    min_samples_split=10,       # Require more samples to split
    min_samples_leaf=5          # Require more samples in leaf
)

# 6. Cross-validation to catch overfitting early
scores = cross_val_score(model, X, y, cv=5)
print(f"Consistent scores? {scores.std() < 0.05}")

2. Underfitting

Problem: Model too simple, doesn’t capture data patterns

# Symptom: Low train and test accuracy
train_score = 0.60
test_score = 0.58
print("Model too simple!")

Solutions:

# 1. More complex model
model = RandomForestClassifier(n_estimators=500, max_depth=None)

# 2. More features / Feature engineering
# See Data Processing category

# 3. Reduce regularization
model = LogisticRegression(C=10.0)  # Larger C = less regularization

# 4. Longer training
model.fit(X_train, y_train, epochs=1000)

# 5. Adjust hyperparameters
model = SVC(kernel='rbf', gamma='auto', C=100)

3. Imbalanced Classes

Problem: Classes have different frequencies

# Check class distribution
print(y.value_counts())
# Output: Class 0: 9500 samples, Class 1: 500 samples

# Problem: Model biased toward majority class

Solutions:

# 1. Resampling
from imblearn.over_sampling import SMOTE
from imblearn.under_sampling import RandomUnderSampler

smote = SMOTE(sampling_strategy=0.5)
X_resampled, y_resampled = smote.fit_resample(X, y)

# 2. Class weights
model = LogisticRegression(class_weight='balanced')

# 3. Different threshold
from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_test, y_pred_proba)
optimal_idx = np.argmax(tpr - fpr)
optimal_threshold = thresholds[optimal_idx]
y_pred_adjusted = (y_pred_proba >= optimal_threshold).astype(int)

# 4. Use PR-AUC instead of ROC-AUC
from sklearn.metrics import average_precision_score
ap = average_precision_score(y_test, y_pred_proba)

4. High Dimensionality (Curse of Dimensionality)

Problem: Too many features, few samples, difficult learning

# Symptom: n_features >> n_samples
print(f"Samples: {X.shape[0]}, Features: {X.shape[1]}")

Solutions:

# 1. Dimensionality reduction (PCA)
from sklearn.decomposition import PCA

pca = PCA(n_components=50)
X_reduced = pca.fit_transform(X)

# 2. Feature selection
from sklearn.feature_selection import SelectKBest, f_classif

selector = SelectKBest(f_classif, k=20)
X_selected = selector.fit_transform(X, y)

# 3. Regularization (L1 especially)
model = LogisticRegression(penalty='l1', solver='liblinear')

5. Hyperparameter Tuning Challenges

Problem: Many hyperparameters, huge search space

# Solutions already covered above:
# GridSearchCV, RandomizedSearchCV, Bayesian Optimization

# Bayesian Optimization (more efficient)
from skopt import gp_minimize

def objective(params):
    model = RandomForestClassifier(
        n_estimators=params[0],
        max_depth=params[1]
    )
    return -cross_val_score(model, X, y, cv=3).mean()

result = gp_minimize(
    objective,
    [(10, 500), (3, 20)],  # Search ranges
    n_calls=20
)

6. Data Quality Issues

Solutions covered in Data Processing category

7. Computational Resources

Problem: Model too large to fit in memory

# Solutions:
# 1. Mini-batch training
batch_size = 32
for i in range(0, len(X), batch_size):
    X_batch = X[i:i+batch_size]
    y_batch = y[i:i+batch_size]
    model.partial_fit(X_batch, y_batch)  # Incremental learning

# 2. Distributed training
import horovod.tensorflow as hvd
# Train across multiple GPUs/TPUs

# 3. Model compression
# Quantization, pruning, distillation

8. Model Interpretability

Problem: “Black box” models, hard to explain predictions

# 1. Feature Importance
import shap

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test)

# 2. LIME (Local Interpretable Model-agnostic Explanations)
import lime

explainer = lime.LimeTabularExplainer(X_train, mode='classification')
exp = explainer.explain_instance(X_test[0], model.predict_proba)

# 3. Partial Dependence Plots
from sklearn.inspection import PartialDependenceDisplay

PartialDependenceDisplay.from_estimator(model, X, [0, 1])
plt.show()

9. Bias and Fairness

Problem: Model discriminates against certain groups

# 1. Detect bias
# Check performance across demographic groups
for group in ['male', 'female']:
    mask = df['gender'] == group
    accuracy = model.score(X[mask], y[mask])
    print(f"Accuracy for {group}: {accuracy:.4f}")

# 2. Fairness metrics
from fairlearn.metrics import demographic_parity_difference

dpd = demographic_parity_difference(
    y_test, y_pred,
    sensitive_features=sensitive_attrs
)
print(f"Demographic Parity Difference: {dpd:.4f}")

# 3. Fairness-aware training
from fairlearn.postprocessing import ThresholdOptimizer

mitigator = ThresholdOptimizer(
    estimator=model,
    constraints='demographic_parity'
)
mitigator.fit(X_train, y_train, sensitive_features=sensitive_train)
y_pred_fair = mitigator.predict(X_test, sensitive_features=sensitive_test)

10. Model Deployment and Monitoring

Production considerations:

# 1. Model serialization
import joblib
joblib.dump(model, 'model.pkl')

# 2. Model versioning
import mlflow
mlflow.log_model(model, "model_v1")

# 3. Monitor performance drift
def check_data_drift(X_new, baseline_stats):
    """Check if new data distribution changed"""
    current_stats = X_new.describe()
    drift_detected = (current_stats - baseline_stats).abs() > threshold
    return drift_detected

# 4. Prediction logging
predictions_log = pd.DataFrame({
    'timestamp': time.time(),
    'input': X_new,
    'prediction': y_pred,
    'confidence': confidence
})

# 5. Retraining pipeline
if performance_degraded:
    # Retrain on recent data
    model = train_model(X_recent, y_recent)
    model.save('model_updated.pkl')

---

Part 7: ML Libraries and Tools

# Data Processing
import pandas as pd
import numpy as np

# Preprocessing
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from imblearn.over_sampling import SMOTE

# Model Selection & Training
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.ensemble import RandomForestClassifier, GradientBoostingRegressor

# Evaluation
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score
import matplotlib.pyplot as plt
import seaborn as sns

# Deep Learning
import tensorflow as tf
from tensorflow import keras
import torch

# Optimization
from sklearn.model_selection import GridSearchCV
from skopt import gp_minimize

# Interpretability
import shap
import lime

# Fairness
from fairlearn.metrics import demographic_parity_difference

# MLOps
import mlflow
import wandb

---

Part 8: Complete ML Workflow Example

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    confusion_matrix, roc_auc_score
)

# ============ 1. LOAD & EXPLORE ============
df = pd.read_csv('data.csv')
print(f"Shape: {df.shape}")
print(f"Missing values:\n{df.isnull().sum()}")

# ============ 2. PREPROCESSING ============
# [See Data Processing category for detailed steps]
df = df.dropna()
df = df.drop_duplicates()

X = df.drop('target', axis=1)
y = df['target']

# ============ 3. SPLIT DATA ============
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# ============ 4. NORMALIZE ============
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# ============ 5. MODEL SELECTION & TRAINING ============
model = RandomForestClassifier(
    n_estimators=100,
    max_depth=10,
    random_state=42
)

model.fit(X_train_scaled, y_train)

# ============ 6. VALIDATION ============
cv_scores = cross_val_score(model, X_train_scaled, y_train, cv=5)
print(f"CV Scores: {cv_scores}")
print(f"Mean: {cv_scores.mean():.4f}")

# ============ 7. EVALUATION ============
y_pred = model.predict(X_test_scaled)
y_pred_proba = model.predict_proba(X_test_scaled)[:, 1]

accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred)
roc_auc = roc_auc_score(y_test, y_pred_proba)

print(f"\n=== EVALUATION METRICS ===")
print(f"Accuracy:  {accuracy:.4f}")
print(f"Precision: {precision:.4f}")
print(f"Recall:    {recall:.4f}")
print(f"F1-Score:  {f1:.4f}")
print(f"ROC-AUC:   {roc_auc:.4f}")

# Confusion Matrix
cm = confusion_matrix(y_test, y_pred)
print(f"\nConfusion Matrix:\n{cm}")

# ============ 8. DEPLOY ============
import joblib
joblib.dump(model, 'final_model.pkl')
print("Model saved!")

---

Summary Table: Quick Reference

Stage	Key Activities	Tools
Problem Definition	Define task, choose ML type	Pen & paper
Data Collection	Gather raw data	APIs, databases
Preprocessing	Clean, normalize data	Pandas, NumPy
Feature Engineering	Create features	Domain knowledge
Data Splitting	Train/Val/Test splits	sklearn.model_selection
Model Selection	Choose algorithm	Compare candidates
Training	Fit to training data	sklearn, TensorFlow
Validation	K-fold, cross-validation	sklearn
Evaluation	Measure performance	Metrics, plots
Hyperparameter Tuning	Optimize parameters	GridSearchCV, Optuna
Deployment	Put in production	Docker, Flask, TensorFlow Serving
Monitoring	Track performance	MLflow, Prometheus

---

Resources

• Scikit-learn Documentation
• TensorFlow & Keras
• Fast.ai Practical Deep Learning
• Andrew Ng - Machine Learning Coursera
• Kaggle Competitions
• Papers with Code

---