Homework #1: Residual Networks and Model Compression

EE 641: Fall 2026

ImportantAssignment Details

Assigned: 26 August
Due: Tuesday, 01 September at 23:59

Gradescope: Homework 1 | How to Submit

WarningRequirements

Overview

This assignment reviews CNN construction and training, then applies two compression techniques — quantization and pruning — to pretrained networks.

Getting Started

Download the pretrained models: hw1-provided.zip

unzip hw1-provided.zip
  • models.py — CIFAR-100 model definitions (ResNet-56, MobileNetV2, VGG-16-BN)
  • cifar100_resnet56.pt — 0.86M parameters
  • cifar100_mobilenetv2.pt — 2.35M parameters
  • cifar100_vgg16_bn.pt — 15.30M parameters

Architectures and checkpoints are from chenyaofo/pytorch-cifar-models.

Build a model and load its checkpoint:

import torch
from models import resnet56

model = resnet56()
model.load_state_dict(torch.load('cifar100_resnet56.pt', map_location='cpu'))
model.eval()

The checkpoints were trained with this normalization:

import torchvision
import torchvision.transforms as T

MEAN = (0.5071, 0.4865, 0.4409)
STD  = (0.2673, 0.2564, 0.2762)

transform = T.Compose([T.ToTensor(), T.Normalize(MEAN, STD)])
testset = torchvision.datasets.CIFAR100(root='./data', train=False,
                                        download=True, transform=transform)

Evaluate top-1 accuracy on the CIFAR-100 test set. One evaluation is a single pass over the 10,000-image test set — about 15 seconds on a Colab GPU and a few minutes on CPU.

ImportantBaseline accuracies
Checkpoint Top-1 accuracy
cifar100_resnet56.pt 72.62%
cifar100_mobilenetv2.pt 74.35%
cifar100_vgg16_bn.pt 74.03%

Your evaluation harness must reproduce these values (within 0.05%). A larger deviation means the transform or checkpoint loading is wrong.

Problem 1: Residual Network for CIFAR-100

WarningRequirements

Implement the network from scratch. Do not use torchvision.models or any prebuilt architecture. You may use torchvision datasets and transforms, numpy, and matplotlib.

Construct a residual convolutional network for image classification on the CIFAR-100 dataset (https://www.cs.toronto.edu/~kriz/cifar.html) using PyTorch. Train your model to at least 65% accuracy.

Review the original ResNet paper:

K. He, X. Zhang, S. Ren and J. Sun, Deep Residual Learning for Image Recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778.

Implement the residual network described in the table below. Use a block-architecture to define the convX layers. Combine blocks to construct the end-to-end network. Experiment with different regularization, optimizers, and learning rate schedules and apply data pre-processing and augmentations to improve model performance and generalization.

Layer Name Architecture
conv1 3×3, 32, stride 1, padding 1
BatchNorm, ReLU, Dropout
conv2_x \(\begin{bmatrix} 3 \times 3, 32 \\ 3 \times 3, 32 \end{bmatrix} \times 2\), stride 1, padding 1
conv3_x \(\begin{bmatrix} 3 \times 3, 64 \\ 3 \times 3, 64 \end{bmatrix} \times 4\), stride 2, padding 1
conv4_x \(\begin{bmatrix} 3 \times 3, 128 \\ 3 \times 3, 128 \end{bmatrix} \times 4\), stride 2, padding 1
conv5_x \(\begin{bmatrix} 3 \times 3, 256 \\ 3 \times 3, 256 \end{bmatrix} \times 2\), stride 2, padding 1
output max-pool, 100-d fc, softmax

Table: Residual architecture. Building blocks are shown in brackets with the numbers of blocks stacked. Downsampling is performed by conv3_x, conv4_x, and conv5_x with a stride of 2.

Use a validation set held out from the training data for hyperparameter tuning (e.g., grid search, random search). Record final loss and accuracy values for each hyperparameter configuration that you evaluate. Plot the training and validation classification accuracy as a function of epoch on separate figures for at least three hyperparameter configurations. Apply your best trained model to the test set (used exactly once) and report the final loss and accuracy.

You may refer to the PyTorch ResNet reference implementation but you must implement the model from scratch: https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py

Deliverables

See Submission.

  1. Tested hyperparameter configurations with final loss and accuracy values.
  2. Accuracy and loss plots for at least three hyperparameter configurations.
  3. Determine and report the output size, parameter count, and MACs for each convX stage. Verify the total parameter count programmatically (sum(p.numel() for p in model.parameters())). Report the receptive field at the final stage and state which stage dominates parameters and which dominates MACs.
  4. The shortcut option you used (A or B, §3.3 of the paper) and why.

Problem 2: Post-Training Quantization

WarningRequirements

Implement quantization by hand with tensor operations — do not use torch.ao.quantization or any other quantization API. Build models with the provided models.py.

Quantization stores weights on a grid of \(2^b\) levels instead of in 32-bit floating point. In this problem, you will quantize pretrained CIFAR-100 classifiers and measure test accuracy as a function of bit width, scale granularity, and range calibration.

This problem uses cifar100_resnet56.pt and cifar100_mobilenetv2.pt.

Part A: Symmetric uniform quantization

For bit width \(b\), the integer grid is \(\{-n, \dots, n\}\) with \(n = 2^{b-1} - 1\). With range \(\alpha = \max |w|\) over the tensor, the scale is \(\Delta = \alpha / n\) and

\[ \hat{w} = \Delta \cdot \mathrm{clamp}\!\left(\mathrm{round}(w / \Delta),\; -n,\; n\right). \]

Implement quantize–dequantize: compute \(\hat{w}\) and store it back in FP32 (“fake quantization”). The accuracy effect is identical to integer storage. Only the memory savings are simulated.

Quantize the weight of every Conv2d and Linear module. Leave biases and BatchNorm parameters in FP32.

Sweep \(b \in \{8, 6, 4, 3, 2\}\) with one scale per tensor on both models. Plot top-1 accuracy vs bit width, both models on one figure, with the 1% chance level marked.

Part B: Per-channel scales

At 4 bits, repeat with one scale per output channel (dimension 0 of the weight tensor) instead of one per tensor. Report per-tensor vs per-channel accuracy for both models.

Per-channel scales differ from a single tensor scale only when the channel ranges differ. For one 3×3 convolution in ResNet-56 and one depthwise convolution in MobileNetV2, report the spread \(\max_c \alpha_c / \min_c \alpha_c\) across output channels. Use the spreads to explain the difference in recovery between the two models.

Part C: Range clipping

With min/max calibration a single outlier sets the range. Repeat the per-tensor measurements with \(\alpha\) set to the 99.9th percentile of \(|w|\), clamping weights above it. Evaluate at 8 and at 4 bits on both models.

Clipping trades error on the clamped outliers for resolution on the remaining weights. Explain why the trade improves accuracy at one bit width and degrades it at the other.

Deliverables

See Submission.

  1. Baseline accuracy table.
  2. Accuracy vs bit width figure. State the bit width at which each model first loses more than 1 point.
  3. 4-bit per-tensor vs per-channel table, the channel-range spreads, and your explanation of the difference between the two models.
  4. Clipping results at 8 and 4 bits and your explanation of when clipping helps and when it hurts.

Problem 3: Magnitude Pruning

WarningRequirements

Implement pruning by hand with binary masks — do not use torch.nn.utils.prune.

Pruning removes weights entirely. The magnitude criterion is the standard baseline: keep the largest weights, zero the rest. In this problem, you will prune two pretrained CIFAR-100 classifiers and measure test accuracy as a function of sparsity, with and without fine-tuning.

This problem uses cifar100_vgg16_bn.pt (15.30M parameters) and cifar100_resnet56.pt (0.86M parameters).

Part A: One-shot sparsity sweep

Implement global magnitude pruning: pool the weights of every Conv2d and Linear module, find the magnitude threshold below which a fraction \(s\) of all weights falls, and zero every weight below it. Keep a binary mask per layer. Leave biases and BatchNorm parameters untouched.

Sweep \(s \in \{0.3, 0.5, 0.7, 0.8, 0.9, 0.95\}\) on both models with no retraining. Plot top-1 accuracy vs sparsity, both models on one figure, chance level marked.

Explain the difference between the two curves using the parameter counts and where each architecture concentrates its weights.

WarningApple Silicon

torch.kthvalue is not implemented on MPS. Compute the global threshold on CPU. The masked forward passes run fine on MPS.

Part B: Fine-tune recovery

Prune VGG-16 at \(s = 0.7\), \(0.9\), and \(0.95\), then fine-tune each for one epoch on the CIFAR-100 training set: SGD, learning rate \(10^{-3}\), momentum 0.9, weight decay \(5 \times 10^{-4}\), batch size 128, standard augmentation (random crop with padding 4, horizontal flip). Re-apply the masks after every optimizer step so pruned weights stay zero.

One fine-tuning epoch takes about three minutes on a Colab GPU.

Report accuracy before and after fine-tuning at each sparsity.

Part C: Prune, then quantize

Quantize the surviving weights of the fine-tuned 70%-sparse VGG-16 to 8 bits on a symmetric per-tensor grid. Report the final accuracy and the weight-memory compression relative to dense FP32 (count \((1-s)\) of the weights at \(8/32\) the bits, ignoring index overhead). This composition is the first two stages of Deep Compression (Han et al., 2016).

Deliverables

See Submission.

  1. Accuracy vs sparsity figure for both models and your explanation of why they differ.
  2. Fine-tune recovery table: accuracy before and after one epoch at each sparsity.
  3. Combined prune-and-quantize accuracy and compression ratio.
TipSubmission 
README.md
q1/
└── q1.ipynb
q2/
└── q2.ipynb
q3/
└── q3.ipynb

Each problem directory contains a suitably annotated Jupyter notebook with inline figures and analysis. Python files with a separate summary of figures and answers/analysis are also acceptable.

Do NOT submit models or coefficient files.