Neural Network From Scratch 🧠

"Why use TensorFlow when you can spend 3x longer writing the same thing in NumPy and actually understand it?"

Built by following Neural Networks from Scratch in Python by Harrison Kinsley & Daniel Kukieła. Every layer, every gradient, every optimizer — hand-rolled. No model.fit() safety net here.

---

What's Inside

This repo is a full deep learning framework built on nothing but NumPy and existential dread.

neuralnet.py — The Engine Room

The entire framework lives here. If something breaks, it's definitely in here.

Component	What it does
Layer_Object	Dense layer. Weights init at 0.1 * randn so your gradients don't explode on day one
Layer_DropOut	Randomly murders neurons during training. Builds character
Activation_ReLU	max(0, x). Deceptively simple. Surprisingly powerful
Activateion_Softmax	Yes, the typo is load-bearing. Converts logits to probabilities via the Jacobian
Activate_Sigmoid	Squashes everything to (0, 1). The introvert of activation functions
Activation_linear	Does nothing. Somehow essential for regression
Loss_CatagoricalCrossEntropy	Punishes the model for being confidently wrong
Loss_BinaryCrossEntropy	Same punishment, but for binary decisions
Loss_MeanSquaredError	Squares your errors so big mistakes hurt more. Petty but effective
Loss_MeanAbsoluteError	Less petty. Uses sign() in the backward pass
Optimizer_SGD	The grandfather. Supports momentum so it doesn't get stuck in local minima
Optimizer_Adagrad	Accumulates squared gradients. Great until the cache grows to infinity
Optimizer_RMSProp	Adagrad but with a leaky memory (rho=0.9). Geoff Hinton's napkin math
Optimizer_Adam	Momentum + RMSProp + bias correction. The optimizer everyone just uses by default

Regularization: L1 and L2 on both weights and biases. Because overfitting is just memorization with a PhD.

Backprop: Fully manual. The chain rule, implemented by hand, one np.dot(dvalues, self.weights.T) at a time.

---

model.py — The Framework

Wraps everything into a clean Model class so you don't have to manually chain 47 .backward() calls.

model = Model()
model.add(Layer_Object(2, 512, weight_regularization_l2=5e-4))
model.add(Activation_ReLU())
model.add(Layer_DropOut(0.1))
model.add(Layer_Object(512, 3))
model.add(Activateion_Softmax())  # the typo stays

model.set(
    loss=Loss_CatagoricalCrossEntropy(),
    optimizer=Optimizer_Adam(learning_rate=0.5, decay=5e-5),
    accuracy=Accuracy_Catagorical()
)

model.finilize()  # also a typo. also stays.
model.train(X, y, validation_data=(X_test, y_test), epoches=10000, print_every=100)

Supports batched training, validation loops, model serialization via pickle, and the Softmax + CrossEntropy fused backward pass (because computing the full Jacobian every step is how you age prematurely).

---

The Scripts

classification.py — Spiral dataset, 3-class softmax classification. 512 neurons, Adam optimizer, 10% dropout. Achieves ~95% accuracy on data that would make logistic regression cry.

BinaryLogisticRegression.py — Same spiral data, 2 classes, sigmoid output + binary cross-entropy. The "is it a cat or not" of this repo.

regression.py — Fits a sine wave using a 3-layer network with linear output and MSE loss. Plots the result with matplotlib so you can feel good about yourself.

mnist_classification.py — The main event. Loads Fashion MNIST (60k training images), normalizes to [-1, 1], flattens 28×28 images to 784-dim vectors, trains a 3-layer network, saves it, then loads it back and classifies your own tshirt.png or pants.png. Uses ThreadPoolExecutor for parallel image loading because waiting is for people who don't understand concurrency.

---

Architecture: Fashion MNIST Model

Input (784)  →  Dense(128)  →  ReLU  →  Dense(128)  →  ReLU  →  Dense(10)  →  Softmax

Classes: T-shirt, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, Ankle boot

The trained model is saved as fashion_mnist.model (pickle). Load it with Model.load('fashion_mnist.model').

---

The Math, Briefly

Forward pass: output = activation(W·X + b)

Backward pass (chain rule, all the way down):

dL/dW = X^T · dL/dZ
dL/db = sum(dL/dZ)
dL/dX = dL/dZ · W^T

Adam update rule (the one everyone uses but few implement):

m = β₁·m + (1-β₁)·g          # momentum
v = β₂·v + (1-β₂)·g²         # RMS
m̂ = m / (1 - β₁ᵗ)            # bias correction
v̂ = v / (1 - β₂ᵗ)            # bias correction
W -= lr · m̂ / (√v̂ + ε)       # update

---

Setup

python -m venv env
env\Scripts\activate
pip install numpy nnfs opencv-python matplotlib

The fashion_mnist_images/ folder contains the dataset pre-organized into train/ and test/ subdirectories by label (0–9).

---

Files

neuralnet.py              # The whole framework
model.py                  # High-level Model class
classification.py         # Spiral data, 3-class
BinaryLogisticRegression.py  # Spiral data, 2-class
regression.py             # Sine wave fitting
mnist_classification.py   # Fashion MNIST + custom image prediction
fashion_mnist.model       # Saved trained model
tshirt.png / pants.png    # Test images for prediction

---

"A neural network is just a function composition with a really aggressive learning rate and a prayer."