MLP and CNN as classifier on MNIST Dataset

Project

Table of Contents

• Overview
• 1. Project Purpose
• 2. Data Loading and Preprocessing
  • 2.1. MLP Classifier (NumPy)
  • 2.2. CNN Classifier (PyTorch)
• 3. Model Architectures
  • 3.1. MLP Classifier (NumPy)
  • 3.2. CNN Classifier (PyTorch)
• 4. Training and Evaluation
• 5. Hyperparameter Search
• 6. Gradient Checking (MLP)
• 7. Visualization
• 8. Grid Search (CNN)
• Project repository

Overview

This post provides a comprehensive explanation of the project, combining the workflows and key concepts from the MLP-with-MNIST-hand-written, CNN_Classifier, and CNN.ipynb notebooks. The project demonstrates how to build, train, and optimize neural network classifiers for the MNIST dataset using both NumPy (MLP) and PyTorch (CNN).

1. Project Purpose

The project implements and compares two types of neural network classifiers:

• MLP Classifier (Multi-Layer Perceptron): Built from scratch using NumPy.
• CNN Classifier (Convolutional Neural Network): Built using PyTorch.

Both models are trained to recognize handwritten digits from the MNIST dataset, with experiments in data augmentation, hyperparameter search, and gradient checking.

---

2. Data Loading and Preprocessing

The MNIST dataset consists of 70,000 grayscale images, each 28×28 pixels in size, categorized into 10 classes representing the digits 0–9.

2.1. MLP Classifier (NumPy)

This code block demonstrates how to load and preprocess the MNIST dataset for use with a Multi-Layer Perceptron (MLP) classifier built in NumPy. The dataset is loaded from a compressed .npz file, separating training and test images and labels. Each image is reshaped from 28×28 pixels to a flat vector of 784 features and normalized to the range [0, 1] by dividing by 255, preparing the data for input into the neural network.

import numpy as np
data = np.load("data/mnist.npz")
x_train, y_train = data["x_train"], data["y_train"]
x_test, y_test = data["x_test"], data["y_test"]
x_train = x_train.reshape(x_train.shape[0], -1).astype(np.float32) / 255
x_test = x_test.reshape(x_test.shape[0], -1).astype(np.float32) / 255

2.2. CNN Classifier (PyTorch)

This code block shows how to load and preprocess the MNIST dataset for use with a Convolutional Neural Network (CNN) in PyTorch. It uses the torchvision library to download the dataset and apply transformations: converting images to tensors and normalizing pixel values to have a mean of 0.5 and a standard deviation of 0.5. The training and test datasets are then created with these transformations, preparing the data for input into the CNN model.

from torchvision import datasets, transforms
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])
train_dataset = datasets.MNIST('data', train=True, download=True, transform=transform)
test_dataset = datasets.MNIST('data', train=False, download=True, transform=transform)

3. Model Architectures

3.1. MLP Classifier (NumPy)

This step involves building a Multi-Layer Perceptron (MLP) classifier from scratch using NumPy. The model is implemented as a Python class with methods for initialization, forward propagation, backpropagation, weight updates, training, and evaluation.

3.1.1. Network Structure Example

• Input layer: 784 units (flattened 28x28 image)
• Hidden layer 1: 500 units (default)
• Hidden layer 2: 250 units (default)
• Output layer: 10 units (one per digit class)

3.1.2. Key Parameters

• hidden_dims: Tuple specifying the number of neurons in each hidden layer (e.g., (500, 250)).
• n_hidden: Number of hidden layers.
• method: Weight initialization method (‘glorot’, ‘normal’, ‘zero’).
• epochs: Number of training epochs.
• lr: Learning rate for parameter updates.

3.1.3. Steps to Build the Model

1. Define the Network Structure:
   • The network consists of an input layer, one or more hidden layers, and an output layer.
   • The number of hidden layers and their sizes are controlled by the hidden_dims and n_hidden parameters.
   • For MNIST, the input size is 784 (28x28 pixels), and the output size is 10 (digits 0-9).
2. Initialize Weights and Biases:
   • Weights and biases are stored in lists (self.weights, self.bias).
   • Initialization can be done using different methods: ‘glorot’ (Xavier), ‘normal’, or ‘zero’.
3. Implement Forward Pass:
   • Computes activations layer by layer using matrix multiplication and ReLU activation for hidden layers.
   • The output layer uses softmax to produce class probabilities.
4. Implement Backward Pass (Backpropagation):
   • Computes gradients of the loss with respect to weights and biases using the chain rule.
5. Update Parameters:
   • Updates weights and biases using the computed gradients and a learning rate.
6. Training Loop:
   • Repeats forward and backward passes for each training example, updating parameters each time.
7. Evaluation:
   • Computes loss and accuracy on validation or test data.

3.1.4. Main Code Block: Classifier Definition

import numpy as np
import math
import random

class Classifier(object):
    def __init__(self, hidden_dims=(500, 250), n_hidden=2):
        self.weights, self.bias = [], []
        # Input to first hidden layer
        self.weights.append(np.empty((hidden_dims[0], 784)))
        self.bias.append(np.zeros((hidden_dims[0])))
        # Hidden layers
        for i in range(n_hidden - 1):
            self.weights.append(np.empty((hidden_dims[i + 1], hidden_dims[i])))
            self.bias.append(np.zeros((hidden_dims[i + 1])))
        # Output layer
        self.weights.append(np.empty((10, hidden_dims[-1])))
        self.bias.append(np.zeros((10)))

    def initialize_weights(self, method="glorot"):
        for i, w in enumerate(self.weights):
            if method == "glorot":
                d = math.sqrt(6 / (w.shape[0] + w.shape[1]))
                self.weights[i] = np.random.uniform(low=-d, high=d, size=w.shape)
            if method == "normal":
                self.weights[i] = np.random.normal(loc=0, scale=1, size=w.shape)
            if method == "zero":
                self.weights[i] = np.zeros(shape=w.shape)

    def forward(self, input):
        self.cache = [input]
        for W, b in zip(self.weights, self.bias):
            self.cache.append(self.activation(W @ self.cache[-1] + b))
        return self.softmax(self.cache.pop())

    def activation(self, input):
        return np.maximum(0, input)  # ReLU

    def loss(self, prediction, label):
        return -math.log(self.softmax(prediction)[label])

    def softmax(self, input):
        m = np.max(input)
        return np.exp(input - m) / np.sum(np.exp(input - m))

    def backward(self, output, label):
        grad_pre_activation = np.asarray([
            out - 1 if i == label else out for i, out in enumerate(output)
        ]).reshape(-1, 1)
        self.grad_w, self.grad_b = [], []
        for i, (w, b) in enumerate(zip(reversed(self.weights), reversed(self.bias))):
            previous_hidden_layer = np.asarray(list(reversed(self.cache))[i])
            self.grad_w.insert(0, grad_pre_activation @ previous_hidden_layer.reshape(-1, 1).T)
            self.grad_b.insert(0, grad_pre_activation.reshape(-1))
            grad_previous_hidden_layer = w.T @ grad_pre_activation
            grad_pre_activation = grad_previous_hidden_layer * np.asarray([
                1 if x > 0 else 0 for x in list(reversed(self.cache))[i]
            ]).reshape(-1, 1)

    def update(self, lr):
        for i, (gw, gb) in enumerate(zip(self.grad_w, self.grad_b)):
            self.weights[i] = self.weights[i] - lr * gw
            self.bias[i] = self.bias[i] - lr * gb

    def train(self, inputs, labels, epochs=1, lr=0.001, verbose=True):
        total_loss = []
        for epoch in range(epochs):
            loss = []
            data = list(zip(inputs, labels))
            random.shuffle(data)
            inputs, labels = zip(*data)
            for i, (x, y) in enumerate(zip(inputs, labels), 1):
                pred = self.forward(x)
                loss.append(self.loss(pred, y))
                self.backward(pred, y)
                self.update(lr)
            total_loss.append(np.mean(loss))
        return total_loss

    def test(self, inputs, labels):
        loss, acc = zip(*[(self.loss(self.forward(x), y),
                           np.argmax(self.forward(x)) == y)
                          for x, y in zip(inputs, labels)])
        loss, acc = np.mean(loss), np.mean(acc)
        return np.mean(loss), np.mean(acc)

3.2. CNN Classifier (PyTorch)

3.2.1. Network Structure

The model is a Convolutional Neural Network (CNN) for binary classification, implemented as a PyTorch nn.Module. It consists of:

• Convolutional Layers: Three layers extract features from input images, each followed by max pooling and activation (ReLU).
• Fully Connected Layers (MLP): Two linear layers map features to a single output (for binary classification).

3.2.2. Key Parameters:

• activation: Activation function (default ReLU)
• n_filters: Number of filters in each conv layer (default (8, 16, 32))
• linear_size: Size of hidden layer in MLP (default 100)

3.2.2. Steps to Build the Model

1. Initialization (__init__)
   • Sets up convolutional and fully connected layers.
2. Forward Pass (forward)
   • Processes input through conv layers, flattens, passes through MLP, applies sigmoid for binary output.
3. Evaluation (evaluate)
   • Computes loss and accuracy on validation/test data.
4. Prediction (predict)
   • Outputs predicted labels for a dataloader.
5. Training (train)
   • Trains the model with early stopping, tracks loss and accuracy, saves best model.

3.2.3. Main Code Block: Classifier Definition

class Net(nn.Module):
    def __init__(self,
                 activation=nn.ReLU(),
                 n_filters=(8, 16, 32),
                 linear_size=100):
        super(Net, self).__init__()
        self.activation = activation
        self.n_filters = n_filters
        self.linear_size = linear_size
        self.conv = nn.Sequential(
            nn.Conv2d(3, self.n_filters[0], kernel_size=3, stride=1, padding=1),
            nn.MaxPool2d(2, 2), self.activation,
            nn.Conv2d(self.n_filters[0], self.n_filters[1], kernel_size=3, stride=1, padding=1),
            nn.MaxPool2d(2, 2), self.activation,
            nn.Conv2d(self.n_filters[1], self.n_filters[2], kernel_size=5, stride=1),
            nn.MaxPool2d(3, 3), self.activation)
        self.mlp = nn.Sequential(
            nn.Linear(self.n_filters[2] * 4 * 4, self.linear_size),
            self.activation, nn.Linear(self.linear_size, 1))
    def forward(self, x):
        x = self.conv(x)
        x = torch.sigmoid(self.mlp(x.view(-1, self.n_filters[2] * 4 * 4)))
        return x

4. Training and Evaluation

MLP Classifier:

• Trains for multiple epochs, updates weights using backpropagation.
• Evaluates accuracy and loss on validation/test sets.

clf = Classifier()
clf.initialize_weights("glorot")
g_loss = clf.train(x_train, y_train, epochs=10, lr=1e-3)

CNN Classifier:

• Trains using Adam optimizer and CrossEntropyLoss.
• Evaluates accuracy and loss on test set.

model = Net()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()
for epoch in range(10):
    model.train()
    for images, labels in train_loader:
        optimizer.zero_grad()
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

5. Hyperparameter Search

The goal of hyperparameter search is to find the best model configuration by systematically trying different values for key parameters and evaluating their performance. In this project, we perform a grid search over:

• Hidden layer sizes: Number of neurons in the first and second hidden layers (hidden_dims=(h1, h2)).
• Learning rate: The step size for gradient descent (lr).

Steps for Hyperparameter Search

1. Define the search space for each hyperparameter:
   • First hidden layer size: [400, 200, 100]
   • Second hidden layer size: [200, 100, 50]
   • Learning rate: [1e-1, 1e-2, 1e-3]
2. For each combination of these values:
   • Create a new MLP classifier with the given hidden layer sizes.
   • Initialize weights using Glorot (Xavier) initialization.
   • Train the model for 10 epochs with the selected learning rate.
   • Evaluate the model on the validation set.
   • Track and print the accuracy for each configuration.
   • Keep the best model found so far.

print("{:^10s} | {:^10s} | {:^10s} | {:^10s}".format("hidden 1", "hidden 2", "lr", "accuracy"))
print(49 * "-")
best_model = None
best_acc = -1
for h1 in [400, 200, 100]:
    for h2 in [200, 100, 50]:
        for l in [1e-1, 1e-2, 1e-3]:
            clf = Classifier(hidden_dims=(h1, h2))
            clf.initialize_weights("glorot")
            clf.train(x_train, y_train, epochs=10, lr=l, verbose=False)
            loss, acc = clf.test(x_valid, y_valid)
            if acc > best_acc:
                best_acc = acc
                best_model = clf
            print("{:^10d} | {:^10d} | {:^10g} | {:^10.1%}".format(h1, h2, l, acc))

6. Gradient Checking (MLP)

Gradient checking using the finite difference method is a crucial step to verify the correctness of your backpropagation implementation. Backpropagation is prone to subtle bugs, and this method provides a numerical way to estimate gradients and compare them to the analytical gradients computed by your code. the Finite Difference Method ensures your backpropagation code is correct by comparing it to a simple, reliable numerical estimate.

How Finite Difference Method Works For a given weight $w$, the gradient of the loss $L$ with respect to $w$ can be approximated as:

\[\frac{\partial L}{\partial w} \approx \frac{L(w + \epsilon) - L(w - \epsilon)}{2\epsilon}\]

where $\epsilon$ is a small value (here, $1/N$ for various $N$).

Steps

1. Select a single training example $(x, y)$.
2. Compute analytical gradients using backpropagation.
3. For a subset of weights in a chosen layer, perturb each weight by $+\epsilon$ and $-\epsilon$ and compute the loss.
4. Estimate the gradient using the finite difference formula.
5. Compare the estimated gradient to the analytical gradient.
6. Record and plot the mean and maximum absolute differences for various $\epsilon$ values.

# Select a single training example
x, y = x_train[0], y_train[0]
# Compute analytical gradients
clf.backward(clf.forward(x), y)
mean_abs_delta, max_abs_delta = [], []
N_values = [i * 10**exp for exp in range(0, 3) for i in range(1, 10)]
layer = 1  # Choose which layer to check
for N in N_values:
    delta = []
    for i, (W, gW) in enumerate(zip(clf.weights[layer], clf.grad_w[layer])):
        for j, (w, gw) in enumerate(zip(W, gW)):
            if len(clf.weights[layer]) * i + j > 10:
                break
            # Perturb weight by +epsilon
            clf.weights[layer][i][j] = w + (1.0 / N)
            loss1 = clf.loss(clf.forward(x), y)
            # Perturb weight by -epsilon
            clf.weights[layer][i][j] = w - (1.0 / N)
            loss2 = clf.loss(clf.forward(x), y)
            # Restore original weight
            clf.weights[layer][i][j] = w
            # Estimate gradient
            estimate_grad = (loss1 - loss2) / (2.0 / N)
            # Compare to analytical gradient
            delta.append(abs(estimate_grad - gw))
    max_abs_delta.append(np.max(delta))
    mean_abs_delta.append(np.mean(delta))

7. Visualization

• Plots training and validation loss curves.
• Visualizes sample predictions.

import matplotlib.pyplot as plt
plt.plot(train_losses, label='Training Loss')
plt.plot(valid_losses, label='Validation Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.show()

8. Grid Search (CNN)

Grid search systematically tries different hyperparameter combinations (e.g., learning rates, batch sizes, layer sizes) to find the best model configuration.

1. Define ranges for hyperparameters.
2. Train and evaluate models for each combination.
3. Select the best performing model.

for lr in [0.01, 0.001, 0.0001]:
    for batch_size in [32, 64, 128]:
        train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
        model = Net()
        optimizer = torch.optim.Adam(model.parameters(), lr=lr)
        # ...train and evaluate model...
        # ...track best accuracy...

Project repository

GitHub Code: MLP & CNN Classifier for MNIST