DenseNet from scratch: ResNet adds a skip connection, DenseNet concatenates every one — top accuracy at a third of the params

ResNet gave every layer a shortcut to the loss by adding a residual: y = x + F(x) . DenseNet takes that idea to its logical extreme. Inside a dense block, every layer is wired to every layer before it — layer ℓ receives the concatenation of the feature-maps of all preceding layers, and passes its own maps to all that follow. I built an interactive breakdown of the channel math, and the surprising part is how thrifty this "connect everything to everything" turns out to be. Here's why.…