Deep Architectures: Convolutions & ResNets

Explore spatial weight sharing in CNNs, residual skip connections, and the stabilization of deep networks.

1. Spatial Convolutions & Receptive Fields

Convolutional layers enforce translational equivariance and local feature spatial locality through parameter sharing of compact kernels \(K \in \mathbb{R}^{k \times k}\):

\[ S(i, j) = (I * K)(i, j) = \sum_{m} \sum_{n} I(i-m, j-n) K(m, n) \]

2. Residual Skip Connections (ResNets)

When networks exceed 20+ layers, training degradation occurs due to vanishing gradients during backpropagation. He et al. introduced residual identity shortcuts:

\[ \mathbf{y} = \mathcal{F}(\mathbf{x}, \{W_i\}) + \mathbf{x} \implies \frac{\partial L}{\partial \mathbf{x}} = \frac{\partial L}{\partial \mathbf{y}} \left( \frac{\partial \mathcal{F}}{\partial \mathbf{x}} + \mathbf{I} \right) \]

The identity term \(\mathbf{I}\) guarantees that gradients can flow unimpeded directly back to the earliest layers without exponential decay.

🎯 Module Mastery Certification Quiz

+100 XP

How do Residual Skip Connections solve the vanishing gradient problem?

By adding an identity term (+x) whose derivative adds an identity matrix I to the gradient, allowing signals to propagate directly.

By eliminating backpropagation entirely.

By converting all weights into 8-bit integers.

By sorting the activation vectors in ascending order.