Neural Network from scratch

Access Repository 

Executive Summary

This project is a handwritten digit classifier built from scratch using only NumPy and Pandas, with no machine learning frameworks such as PyTorch or TensorFlow. It implements a full neural network training pipeline manually, including forward propagation, loss computation, backpropagation, and gradient-based optimization.

The model classifies 28×28 grayscale images of handwritten digits into one of ten classes (0–9) using a simple two-layer feedforward architecture. Every component of the network is implemented explicitly using matrix operations, giving full visibility into how neural networks learn from data at a low level.

Training is performed on 59,000 MNIST samples using full-batch gradient descent. The system uses ReLU activation in the hidden layer and softmax-based classification at the output layer, optimized using cross-entropy loss. The training process is fully vectorized for efficiency and stability.

Despite its minimal architecture and lack of external ML libraries, the model achieves strong learning behavior, reaching approximately 80–90% training accuracy. The project emphasizes correctness, interpretability, and mathematical transparency over abstraction, making it a clear demonstration of fundamental machine learning understanding.

Overall, this project showcases the ability to build and train a neural network from first principles, including data preprocessing, numerical linear algebra, and optimization, without relying on high-level frameworks.

Tech Stack & Project Structure

Language: Python3

Libraries:

1. Numpy

2. Pandas

3. Matplotlib

Structure:

NNProject/

├── NN.ipynb                      # Main notebook — all code lives here

└── digit_dataset/

    ├── mnist_train.csv           # Training images + labels

    └── mnist_test.csv            # Held-out test set (not used in notebook yet)

Dataset: MNIST

Network Architecture

784 inputs: One node per pixel after flattening the image.
10 hidden units: A small hidden layer keeps the network fast and easy to reason about while still learning non-linear patterns with ReLU activation function.
10 outputs: One score per digit class; Softmax activation converts scores to probabilities.

Forward Propagation

Forward propagation in this project is implemented as a fully vectorized computation over the entire batch of MNIST images. Each input batch of normalized 784-dimensional pixel vectors is first passed through a linear transformation using randomly initialized weights and biases, producing the hidden layer pre-activations. A ReLU activation function is then applied to introduce non-linearity by zeroing out negative values while preserving positive activations. The transformed hidden representations are passed into a second linear layer that maps them to 10 output logits, representing digit classes from 0 to 9. Finally, a softmax function converts these logits into a normalized probability distribution over classes. The implementation is fully batch-based, meaning all 59,000 training examples are processed simultaneously using matrix operations for efficiency and clarity.

Training and Optimization

The model is trained using full-batch gradient descent over the entire training dataset of 59,000 MNIST images. In each iteration, the network performs a complete forward pass followed by manual backpropagation to compute gradients for all weights and biases. These gradients are then used to update parameters using a fixed learning rate of 0.10. The training loop runs for 500 iterations, with periodic accuracy checks every 10 iterations to monitor convergence. The optimization objective is implicitly categorical cross-entropy loss, and training progresses by iteratively adjusting parameters in the direction that reduces classification error across the full dataset. This implementation prioritizes transparency of the optimization process over computational efficiency.

Backpropagation (Conceptual)

Backpropagation is implemented explicitly using matrix calculus and the chain rule to compute gradients for all trainable parameters. The process begins by converting integer labels into one-hot encoded vectors and computing the difference between predicted probabilities and true labels at the output layer. This error signal is propagated backward through the network using transposed weight matrices. The ReLU derivative is applied as a binary mask to block gradients flowing through inactive neurons in the hidden layer. The resulting gradients are then used to compute partial derivatives for both weight matrices and bias vectors. The implementation closely follows the mathematical structure of neural networks, translating each derivative directly into NumPy operations.

Results

The network demonstrates consistent learning behavior over the 500 training iterations. Starting from randomly initialized weights, the model begins with near-chance performance and steadily improves as gradient descent updates refine the decision boundaries. With only a single hidden layer of 10 neurons and approximately 7,950 trainable parameters, the model achieves roughly 85–90% training accuracy on the MNIST dataset. This result confirms that even a minimal neural architecture, when correctly implemented using backpropagation and vectorized computation, can learn meaningful representations of handwritten digits. The performance validates the correctness of the manual implementation rather than relying on architectural complexity.

How to Run

To run the project, install the required Python dependencies including NumPy, Pandas, and Matplotlib, and execute the Jupyter Notebook in sequence. Then visit github and clone the repo and run the program while having the dataset in the same folder.