Deep learning architectures

Why do networks need nonlinear activations?

Without them, any stack of linear layers collapses into a single linear map, so depth would add no expressive power.

What is the sigmoid function, and what is its main drawback in hidden layers?

It maps any number to between 0 and 1. It saturates for large inputs, where its gradient is near zero, which slows learning in deep stacks.

How does tanh differ from sigmoid?

It maps to between -1 and 1 and is centered at zero, which helps optimization, but it saturates in the same way.

What is ReLU?

max(0, x). Its gradient is 1 for positive inputs, so it does not saturate there, and it is cheap to compute.

What is a dead ReLU?

A unit whose input is negative for every example, so it always outputs zero and gets no gradient to recover.

What are Leaky ReLU and GELU?

Leaky ReLU gives negative inputs a small slope. GELU is a smooth curve close to ReLU that lets small negative values through; it is common in transformers.

What does softmax do?

Turns a vector of scores (logits) into probabilities that are positive and sum to 1, by exponentiating and normalizing.

What is backpropagation?

Computing the gradient of the loss with respect to every weight by applying the chain rule from the output back through each layer.

What is the vanishing gradient problem?

Gradients shrink as they pass back through many layers or time steps, so early layers learn very slowly.

What is the exploding gradient problem?

Gradients grow as they pass back through many layers, causing huge, unstable updates. Gradient clipping is a common fix.

Why do initialization schemes like Xavier and He exist?

To set the starting weight scale so activations and gradients keep a steady size through the layers. He initialization is designed for ReLU.

Why not initialize all weights to zero?

Every unit in a layer would compute the same thing and get the same update, so they would never become different.

What does batch normalization do?

Normalizes each feature using the batch's mean and variance, then applies a learned scale and shift. At inference it uses running averages from training.

What does layer normalization do?

Normalizes across the features of each single example or token, independent of the batch. It is the standard in transformers.

What is a residual (skip) connection?

Adding a block's input to its output, so the block learns a correction. It gives gradients a direct path and makes very deep networks trainable.

What is SGD with momentum?

Gradient descent on mini-batches that keeps a running average of past gradients, which smooths updates and speeds progress along consistent directions.

What is Adam?

An optimizer that keeps running averages of each parameter's gradient and squared gradient, and scales each parameter's step by them.

What is AdamW?

Adam with weight decay applied directly to the weights instead of through the loss, so decay works as intended alongside Adam's scaling.

What are warmup and cosine decay?

Warmup ramps the learning rate up over the first steps for stability. Cosine decay then lowers it smoothly toward a small value by the end.

What is gradient clipping?

If the gradient's overall size exceeds a limit, scaling it down to that limit while keeping its direction.

What is mixed-precision training?

Doing most math in 16-bit formats such as bfloat16 to save memory and time, while keeping sensitive parts in 32-bit.

What does a convolutional layer do?

Slides small learned filters across the input, computing a weighted sum at each position. The same filter is used everywhere, which saves parameters.

What is the formula for a convolution's output size?

(W - K + 2P) / S + 1, rounded down, where W is input size, K kernel size, P padding, and S stride.

How many parameters does a convolution have?

K × K × input channels × output channels, plus one bias per output channel. It does not depend on image size.

What are stride and padding?

Stride is how far the filter moves each step; stride 2 halves the size. Padding adds a border so edges are covered and size can be kept.

What is a receptive field?

The region of the input that affects one output value. Stacking layers widens it; three 3×3 layers see 7×7.

What does pooling do?

Summarizes small regions, for example by taking the max of each 2×2 block, which shrinks the feature map and adds some tolerance to small shifts.

What is a 1×1 convolution for?

Mixing and changing the number of channels at each position cheaply, without looking at neighbors.

What is a depthwise separable convolution?

A per-channel spatial filter followed by a 1×1 convolution that mixes channels. It uses far fewer parameters and operations than a standard convolution.

What is translation equivariance in CNNs?

Shifting the input shifts the feature map by the same amount, so a pattern is detected wherever it appears.

What is global average pooling?

Averaging each channel over all positions to get one number per channel, used before a CNN's final classifier.

What is transfer learning?

Starting from a model pretrained on a large dataset and adapting it to a new task, often by replacing the head and fine-tuning.

What is a recurrent neural network?

A network that processes a sequence one step at a time, combining each input with a hidden state carried from the previous step, using the same weights at every step.

What is backpropagation through time?

Unrolling an RNN over the sequence and backpropagating through all steps, summing gradients for the shared weights.

What is an LSTM?

A recurrent unit with a cell state and input, forget, and output gates. The mostly additive cell update lets information and gradients last over long spans.

What is a GRU?

A simpler gated recurrent unit with update and reset gates and no separate cell state. It often performs similarly to an LSTM with fewer parameters.

What is a bidirectional RNN?

Two RNNs, one reading left to right and one right to left, with their states combined. It needs the whole sequence up front.

What is a sequence-to-sequence model?

An encoder that reads an input sequence and a decoder that generates an output sequence, as in translation.

What is teacher forcing?

Feeding the true previous token to the decoder during training, rather than its own prediction.

What is attention?

A weighted mix of a set of value vectors, with weights from how well each key matches a query. It lets a model look directly at any part of the input.

How does self-attention's cost grow with sequence length?

Quadratically, because every token scores against every other token.

Why did transformers replace RNNs for most language tasks?

They train in parallel across positions, and any token can reach any other in one step, which makes long-range patterns easier to learn at scale.

What is a Vision Transformer?

A transformer applied to images by cutting them into fixed-size patches, embedding each patch as a token, and adding position information.

What is an embedding?

A learned dense vector representing a discrete item such as a word or user, where similar items end up close together.

What is an autoencoder?

A network that compresses its input into a small code and reconstructs the input from that code.

What are a VAE and a GAN?

A VAE is an autoencoder whose codes follow a simple distribution, so new samples can be drawn. A GAN trains a generator against a discriminator that tries to tell real from fake.

What is a diffusion model?

A generator trained to remove noise step by step. Sampling starts from pure noise and denoises over many steps.

What is knowledge distillation?

Training a smaller student model to match a larger teacher's output probabilities, so it keeps much of the quality at lower cost.

What is quantization?

Storing weights, and sometimes activations, in fewer bits, such as 8 or 4 instead of 16 or 32. It cuts memory and speeds inference with some accuracy cost.