Convolutional Neural Networks: How AI Learned to See

Before CNNs, teaching a computer to recognize an image required telling it exactly what to look for — explicit feature extraction written by engineers. After CNNs, the computer learned what to look for from data. This shift — from engineered features to learned features — is arguably the most important transition in the history of computer vision.

Understanding CNNs is understanding how AI learned to see.

The problem with using plain neural networks for images

A fully connected neural network treats every input independently. Each input neuron connects to every hidden neuron, with no structural assumption about what the input represents.

For images, this is a disaster. Consider a modest 224×224 RGB image. It has 224 × 224 × 3 = 150,528 input values. A first hidden layer with 1,000 neurons would require 150,528 × 1,000 = 150 million parameters — for just the first layer.

Worse: fully connected networks have no built-in understanding that pixels near each other are related. Pixel (100, 100) and pixel (101, 100) are neighbors and should influence similar features, but a fully connected network treats them as completely independent inputs.

CNNs solve both problems with two ideas: local connectivity and weight sharing.

The convolution operation

The core operation in a CNN is a convolution: sliding a small filter (kernel) across the input and computing dot products at each position.

Concretely: a 3×3 filter contains 9 learned values. As it slides across a 224×224 image, at each position it:

1. Takes the 3×3 patch of pixels at that position
2. Multiplies each pixel value by the corresponding filter value (element-wise)
3. Sums all 9 products to produce a single output value

The result is a 2D map of how strongly the filter’s pattern was present at each location — called a feature map or activation map.

Why this is brilliant: The same 9 filter weights detect the same visual pattern everywhere in the image. A filter that detects horizontal edges detects them in the top-left corner and the bottom-right corner with the same weights. This is weight sharing — instead of 150 million parameters for a fully connected first layer, a 3×3 filter has just 9 parameters, shared across the entire image.

Multiple filters run in parallel, each looking for different patterns. A layer with 64 filters produces 64 feature maps — 64 different “views” of what patterns are present where.

Stride and padding

Stride: How many pixels the filter moves at each step. Stride 1 produces an output nearly the same size as the input. Stride 2 downsamples by 2× in each dimension. Larger strides produce smaller output maps and reduce computation.

Padding: Convolving without padding reduces the output size (a 3×3 filter on a 10×10 image produces an 8×8 output). Zero-padding adds a border of zeros around the input so the output can match the input size (“same” padding) or not (“valid” padding). Padding also lets the filter attend to pixels at the edges, which would otherwise be underrepresented.

Activation functions

After each convolution, element-wise non-linearity is applied — typically ReLU (Rectified Linear Unit):

ReLU(x) = max(0, x)

This introduces non-linearity (without which any sequence of linear transformations would collapse to a single linear transformation) and is computationally cheap. Variations like Leaky ReLU, PReLU, and GELU are used in more modern architectures.

Pooling: reducing spatial size while keeping what matters

After convolution and activation, pooling downsamples the feature maps — keeping the strongest signals while discarding spatial precision.

Max pooling: Take the maximum value in each region. A 2×2 max pool with stride 2 reduces each feature map to one-quarter its size. The intuition: “was this pattern present somewhere in this region?” — precise location doesn’t matter.

Average pooling: Take the mean instead of the maximum. Less commonly used in hidden layers; global average pooling (averaging the entire feature map to a single value per channel) is common at the network’s end.

Pooling provides:

• Spatial invariance — small shifts in the input don’t dramatically change the output
• Reduced computation in subsequent layers
• Some regularization (information compression)

Modern architectures (ResNet, EfficientNet) often use strided convolutions instead of max pooling, achieving similar downsampling with learned rather than fixed operations.

Building blocks to full networks

A typical CNN alternates:

1. Convolutional layer(s) + ReLU
2. Pooling (or strided conv)
3. Repeat, increasing depth and filter count
4. Flatten the feature maps into a vector
5. Fully connected layers for the final prediction

Early layers learn low-level features: edges, colors, simple textures. Middle layers combine those into mid-level features: textures, patterns, parts. Late layers encode high-level concepts: faces, objects, scenes.

This hierarchy of representation — simple features composing into complex concepts — is why CNNs work so well. It matches the structure of visual information.

Key architectural milestones

LeNet-5 (1998): Yann LeCun’s pioneering CNN for digit recognition. 7 layers, trained on MNIST. Proved the concept but couldn’t scale to complex natural images with the hardware of the era.

AlexNet (2012): The ImageNet moment. Krizhevsky, Sutskever, and Hinton trained an 8-layer CNN on GPUs and won the ImageNet competition by a 10-percentage-point margin over classical computer vision. This single result catalyzed the deep learning revolution. Key innovations: ReLU (instead of sigmoid/tanh), dropout regularization, GPU training, data augmentation.

VGGNet (2014): Showed that deep networks with simple 3×3 convolutions throughout outperformed more complex but shallower architectures. The “simplicity at depth” lesson: 16-19 layers of small filters works better than 5-8 layers of large filters.

GoogLeNet / Inception (2014): Introduced “inception modules” — parallel convolutions at different scales within a single layer. Allowed very deep networks (22 layers) without the parameter explosion, by using 1×1 convolutions to reduce dimensionality.

ResNet (2015): The skip connection breakthrough. Added residual connections — direct paths from earlier layers to later layers that bypass intermediate computations. This allowed training networks of 50, 101, even 152 layers without degradation. The insight: identity mappings are easier to learn when the network can pass information through unchanged. ResNet is still widely used in production systems.

EfficientNet (2019): Systematic scaling of width, depth, and resolution simultaneously using compound scaling coefficients. Achieved state-of-the-art ImageNet accuracy with significantly fewer parameters than contemporaries.

CNNs vs. Vision Transformers

Since 2020, Vision Transformers (ViT) have increasingly competed with and surpassed CNNs on large-scale image recognition benchmarks. ViTs apply the Transformer self-attention mechanism to image patches rather than tokens.

The tradeoffs:

• CNNs are more data-efficient: they bake in useful inductive biases (local connectivity, translation invariance) that ViTs must learn from data. CNNs work better with smaller datasets.
• ViTs scale better: with very large datasets and compute, ViTs outperform CNNs. They also generalize better to diverse tasks.
• Hybrid models (combining CNN feature extraction with Transformer attention) often beat both.

In 2026, production vision systems often use hybrid architectures or ViTs for tasks with sufficient data, while CNNs remain competitive for edge deployment, limited-data scenarios, and applications where inference efficiency is critical.

Practical implications

If you’re using a pre-trained CNN (ResNet, EfficientNet) via a library like PyTorch or TensorFlow:

Feature extraction: Use the convolutional backbone to produce rich feature representations from your images, then train a small classifier on top. Works well with limited data.

Fine-tuning: Start from pre-trained ImageNet weights, then continue training on your domain-specific data. Almost always beats training from scratch unless you have millions of domain images.

Transfer learning robustness: Features learned on ImageNet transfer surprisingly well to domains like medical imaging, satellite imagery, and manufacturing defect detection. The early layers (edge, texture detectors) are genuinely universal.

Choosing a backbone: For production inference with limited compute, use MobileNet or EfficientNet-B0/B1. For maximum accuracy when compute allows, EfficientNet-B7 or ConvNeXt. For research baselines, ResNet-50 remains the standard comparison point.

---

CNNs gave AI eyes. Understanding how they work — the local connectivity, weight sharing, hierarchical representation — gives you the conceptual vocabulary to evaluate claims about computer vision systems, debug model failures, and make informed architecture choices.

For the full picture of how CNNs are trained with backpropagation, see the 🟣 Technical article on backpropagation. For how CNNs compare to Vision Transformers in research, see the 🔴 Research series on deep learning.