Deep learning architectures

Question 1

Why does a network need a nonlinear activation between its layers?

Accepted Answer

Without one, a stack of linear layers collapses into a single linear map, no matter how deep.

Question 2

The largest value of the sigmoid's derivative is 0.25. Roughly how much can a gradient shrink after passing back through 10 sigmoid layers, ignoring the weights?

Accepted Answer

To about one millionth of its size, since 0.25 to the 10th is about 9.5 × 10⁻⁷.

Question 3

Why did ReLU largely replace sigmoid in hidden layers?

Accepted Answer

Its gradient is 1 for positive inputs, so it does not saturate there, and it is cheap to compute.

Question 4

After a large learning-rate spike, 40 percent of the ReLU units in a layer output zero for every input. What happened?

Accepted Answer

Those units died: their weights moved so their input is always negative, so they get no gradient and cannot recover.

Question 5

What does Leaky ReLU change compared with ReLU?

Accepted Answer

It gives negative inputs a small slope, such as 0.01, so units still get some gradient.

Question 6

Why is GELU common in transformer feed-forward layers?

Accepted Answer

It is a smooth version of ReLU that lets small negative values through, which tends to train a bit better in large models.

Question 7

What does backpropagation compute?

Accepted Answer

The gradient of the loss with respect to every weight, by applying the chain rule from the output back to the input.

Question 8

Why are weights not all initialized to zero?

Accepted Answer

Every unit in a layer would compute the same thing and get the same update, so they would never become different.

Question 9

He initialization sets weight variance to 2 divided by the number of inputs. Which activation is it designed for?

Accepted Answer

ReLU, which zeroes about half its inputs, so the variance is doubled to keep signal size steady.

Question 10

Gradient norms grow by about 10 times per layer going backward in a deep network. What is that called, and what is a common fix?

Accepted Answer

Exploding gradients, often handled with gradient clipping, better initialization, and normalization.

Question 11

Why is softmax followed by cross-entropy the usual output for multi-class classification?

Accepted Answer

Softmax turns logits into a probability distribution, and the gradient of cross-entropy with respect to the logits is simply the predicted minus the true distribution.

Question 12

Why do libraries offer a fused 'cross-entropy from logits' instead of computing softmax and then the log?

Accepted Answer

Computing them separately can overflow or take the log of zero; the fused version uses the log-sum-exp trick to stay stable.

Question 13

The universal approximation theorem says a single hidden layer can approximate any continuous function. Why do we still use deep networks?

Accepted Answer

The theorem says nothing about how many units you need or how to learn them; depth often reaches the same function with far fewer units.

Question 14

A new model will not train. What is a good first sanity check?

Accepted Answer

Try to overfit a single small batch. If loss will not go near zero, the bug is in the model, the loss, or the data pipeline.

Question 15

What does batch normalization do during training?

Accepted Answer

Normalizes each feature using the mean and variance of the current batch, then applies a learned scale and shift.

Question 16

What does batch normalization use at inference time?

Accepted Answer

Running averages of the mean and variance collected during training.

Question 17

Why does batch normalization behave badly with a batch size of 2?

Accepted Answer

The mean and variance of two examples are very noisy estimates, so training becomes unstable.

Question 18

Transformers use layer normalization rather than batch normalization. What does layer normalization normalize over?

Accepted Answer

The features of each single token, independently of the rest of the batch.

Question 19

What problem do residual connections, as in ResNet, solve?

Accepted Answer

Very deep plain networks trained worse than shallower ones. Adding the input back to each block gives gradients a direct path and makes the identity easy to learn.

Question 20

A residual block computes x + F(x). For the addition to work, what must be true?

Accepted Answer

F(x) and x must have the same shape, or the shortcut needs a projection such as a 1×1 convolution.

Question 21

What does momentum add to plain SGD?

Accepted Answer

A running average of past gradients, so steps keep moving in consistent directions and damp back-and-forth swings.

Question 22

What does Adam keep for each parameter?

Accepted Answer

Running averages of the gradient and of the squared gradient, used to scale each parameter's step.

Question 23

Why do transformer training runs usually start with a learning-rate warmup?

Accepted Answer

Early on, Adam's statistics are poor and the weights are random, so full-size steps can push training somewhere it never recovers from.

Question 24

What does a cosine learning-rate schedule do over a run?

Accepted Answer

Lowers the rate smoothly from its peak toward a small value by the end of training.

Question 25

What does gradient clipping by norm do?

Accepted Answer

If the full gradient's norm exceeds a limit, scales it down to that limit while keeping its direction.

Question 26

Why is bfloat16 popular for training large models?

Accepted Answer

It keeps the same exponent range as float32 with half the bits, so it rarely overflows and usually does not need loss scaling.

Question 27

With mixed-precision training in float16, why is loss scaling used?

Accepted Answer

Small gradients can round to zero in float16; multiplying the loss first keeps them in range, and the gradients are divided back before the update.

Question 28

You raise the batch size from 256 to 2,048 and keep the learning rate the same. What often happens?

Accepted Answer

Progress per epoch slows, because there are eight times fewer steps. A common starting point is to raise the learning rate and add warmup.

Question 29

A model does not fit in GPU memory at the batch size you want. Which technique keeps the same effective batch size?

Accepted Answer

Gradient accumulation: run several smaller batches, add up their gradients, then take one optimizer step.

Question 30

What does activation checkpointing trade?

Accepted Answer

It saves memory by not storing some activations in the forward pass, then recomputes them during the backward pass.

Question 31

What two ideas make a convolutional layer far smaller than a fully connected layer on an image?

Accepted Answer

Local connections, where each output looks at a small patch, and weight sharing, where the same filter slides across the whole image.

Question 32

What does it mean that convolution is translation equivariant?

Accepted Answer

Shifting the input shifts the feature map by the same amount.

Question 33

A 32×32 input passes through a 5×5 convolution with stride 1 and no padding. What is the output height and width?

Accepted Answer

28×28

Question 34

A 224×224 image goes through a 7×7 convolution with stride 2 and padding 3. What is the output size?

Accepted Answer

112×112

Question 35

How many parameters are in a 3×3 convolution from 3 input channels to 64 output channels, with a bias for each output channel?

Accepted Answer

1,792

Question 36

What does 'same' padding do for a 3×3 convolution with stride 1?

Accepted Answer

Adds a one-pixel border so the output keeps the input's height and width.

Question 37

Three 3×3 convolutions with stride 1 are stacked. What is the receptive field of one output pixel?

Accepted Answer

7×7

Question 38

Two stacked 3×3 convolutions cover the same 5×5 area as one 5×5 convolution. With C channels in and out, why are the two 3×3 layers often preferred?

Accepted Answer

They use 18C² weights instead of 25C², and add an extra nonlinearity in between.

Question 39

A 56×56 feature map goes through 2×2 max pooling with stride 2. What comes out?

Accepted Answer

28×28, keeping the largest value in each 2×2 block

Question 40

What is a 1×1 convolution used for?

Accepted Answer

Mixing or changing the number of channels at each pixel, cheaply, without looking at neighbors.

Question 41

A depthwise separable convolution splits a 3×3 convolution from 128 to 256 channels into a 3×3 depthwise step and a 1×1 pointwise step. Ignoring biases, how many weights does it use compared with the standard layer?

Accepted Answer

33,920 instead of 294,912, about 9 times fewer.

Question 42

Why does a classification CNN often end with global average pooling instead of flattening into a large dense layer?

Accepted Answer

It removes most of the parameters, reduces overfitting, and works for any input size.

Question 43

In a typical CNN, how do feature maps change as you go deeper?

Accepted Answer

Spatial size shrinks and the number of channels grows, as features become more abstract.

Question 44

You have 2,000 labeled X-ray images. What is the usual first approach with a CNN?

Accepted Answer

Start from a network pretrained on a large image set, replace the head, and fine-tune.

Question 45

When fine-tuning a pretrained CNN on a small dataset, why do people often freeze the early layers?

Accepted Answer

Early layers learn general features like edges that transfer well, and freezing them leaves fewer weights to overfit.

Question 46

When fine-tuning the whole pretrained network, why use a smaller learning rate than for training from scratch?

Accepted Answer

Large steps can quickly destroy useful pretrained features before the new head has learned anything.

Question 47

A model pretrained on daytime street photos is used on night-time infrared images. Why might transfer work poorly?

Accepted Answer

The input statistics and even the low-level features differ a lot, so more layers need retraining and more target data helps.

Question 48

What is the difference between image classification and semantic segmentation?

Accepted Answer

Classification gives one label per image; segmentation gives a label for every pixel.

Question 49

Why does U-Net connect encoder layers directly to matching decoder layers?

Accepted Answer

The decoder gets back fine spatial detail lost during downsampling, which sharpens boundaries.

Question 50

What does intersection over union (IoU) measure in object detection?

Accepted Answer

The overlap of a predicted box and a true box, divided by the area they cover together.

Question 51

A detector outputs five overlapping boxes around the same dog. What step keeps one?

Accepted Answer

Non-maximum suppression: keep the highest-scoring box and drop others that overlap it above an IoU threshold.

Question 52

Why do vision models often struggle when an object appears much smaller than in training images?

Accepted Answer

Convolution is not built to handle changes of scale, so features learned at one size may not fire at another without multi-scale design or augmentation.

Question 53

A Vision Transformer splits a 224×224 image into 16×16 patches. How many patch tokens does that give?

Accepted Answer

196

Question 54

Why did early Vision Transformers need much more training data than CNNs to match them?

Accepted Answer

They lack the built-in assumptions of locality and weight sharing, so they have to learn those patterns from data.

Question 55

When would you still pick a CNN over a Vision Transformer for a new image task?

Accepted Answer

When data is limited, latency or memory on a small device is tight, and the task is mostly local patterns.

Question 56

Can convolutions be used on sequences such as audio or text?

Accepted Answer

Yes. 1D convolutions slide along time, and dilated convolutions widen the receptive field without many layers.

Question 57

What does a recurrent neural network do at each time step?

Accepted Answer

Combines the current input with the previous hidden state to produce a new hidden state, using the same weights at every step.

Question 58

What is backpropagation through time?

Accepted Answer

Unrolling the RNN across the sequence and applying ordinary backpropagation through all the copies, adding up the gradients for the shared weights.

Question 59

Why does a plain RNN have trouble learning that a word at position 5 affects the prediction at position 200?

Accepted Answer

The gradient passes through about 195 multiplications by the recurrent weights and activation slopes, and usually vanishes on the way.

Question 60

What is truncated backpropagation through time?

Accepted Answer

Backpropagating only through a fixed window of recent steps, while still carrying the hidden state forward.

Question 61

What is the main idea that lets an LSTM keep information over long spans?

Accepted Answer

A cell state updated mostly by addition, controlled by gates, so gradients can flow back without being squashed at every step.

Question 62

In an LSTM, what does the forget gate control?

Accepted Answer

How much of the previous cell state is kept.

Question 63

Why is the LSTM forget-gate bias often initialized to a positive value such as 1?

Accepted Answer

So the gate starts near open, keeping memory by default until training learns what to drop.

Question 64

An LSTM layer has input size 100 and hidden size 256, with one bias vector per gate. How many parameters does it have?

Accepted Answer

365,568