3.4: Practice Problems

False. While the model is linear in the weights, we can capture nonlinear relationships through feature engineering. For example:

• Adding polynomial features: $x_1, x_1^2, x_1^3$
• Including interaction terms: $x_1 x_2$
• Creating transformed features: $\sin(x_1), \log(x_2)$

The key insight: the model $y = w_1 x_1 + w_2 x_1^2 + b$ is still linear in the weights $\mathbf{w}$, even though it represents a nonlinear (quadratic) relationship in the original feature $x_1$.

False. This cost function is problematic because:

1. Minimizing $\frac{1}{\text{residual}^2}$ actually maximizes the residual (the denominator)
2. As the error gets larger, the loss gets smaller, the opposite of what we want
3. Division by zero when prediction is perfect

A valid cost function should decrease as predictions improve, not increase.

True. This is a valid cost function because:

• It’s always positive: $e^{(\cdot)^2} > 0$
• It increases with larger errors
• It’s differentiable everywhere
• Minimum occurs at zero error

However, this loss is very aggressive, it exponentially penalizes outliers, making the model extremely sensitive to large residuals. In practice, squared error is preferred for its mathematical convenience and balanced behavior.

Advantages:

1. Interpretability, Weights directly show feature importance and direction of influence
2. Computational efficiency, Fast to train on moderate datasets
3. Closed-form solution, Direct solution via normal equations (no iteration needed)
4. Good baseline, Establishes performance floor for complex models
5. Well-understood theory, Statistical properties thoroughly studied

Disadvantages:

1. Assumes linearity, Poor fit for inherently nonlinear relationships (without feature engineering)
2. Sensitive to outliers, Squared error heavily penalizes large residuals
3. Multicollinearity, Performance degrades when features are highly correlated
4. Requires numerical features, Categorical variables need encoding
5. High bias, May underfit complex data (simple model hypothesis class)
6. Computational cost of direct solution, $O(D^3)$ for matrix inversion when $D$ is large

False. The number of parameters equals $D+1$ (or just $D$ if we count the bias separately):

• $D$ weights for the features: $w_1, w_2, \ldots, w_D$
• $1$ bias term: $b$ (or $w_0$ when absorbed)

The number of training examples $N$ determines how much data we have to estimate these parameters, but doesn’t affect how many parameters exist. In fact, we typically want $N \gg D$ to avoid overfitting.

Incorrect. Makayla’s result has the wrong shape:

Her computation:

• $(\mathbf{X}\mathbf{w} - \mathbf{t})^{\top}$ has shape $1 \times N$ (row vector)
• $\mathbf{X}$ has shape $N \times (D+1)$
• Product: $(1 \times N) \times (N \times (D+1)) = 1 \times (D+1)$, a row vector

Problem: The gradient must be a column vector with shape $(D+1) \times 1$ to match the shape of $\mathbf{w}$ for gradient descent updates: $\mathbf{w} \leftarrow \mathbf{w} - \alpha \nabla_{\mathbf{w}} \mathcal{E}(\mathbf{w})$.

Correct gradient:

This gives:

• $\mathbf{X}^{\top}$ has shape $(D+1) \times N$
• $(\mathbf{X}\mathbf{w} - \mathbf{t})$ has shape $N \times 1$ (column vector)
• Product: $((D+1) \times N) \times (N \times 1) = (D+1) \times 1$, correct shape

Original formulation:

This requires treating $b$ separately in all computations.

After absorbing bias:

Define augmented vectors with a dummy feature $x_0^{(i)} = 1$:

Now the model becomes simply:

Benefits:

1. Unified notation, No special case for bias in equations
2. Simpler code, Vectorized operations handle bias automatically
3. Cleaner gradient, Single formula for all parameters
4. Standard convention, Matches most ML libraries

The cost: each data point and weight vector increases in dimension by 1 (from $D$ to $D+1$).

Part (a): Deriving the Gradients

For each training example, the residual is:

Derivative with respect to $w$:

Using the chain rule on $(r^{(i)})^2$:

Summing over all examples and dividing by $N$:

Derivative with respect to $b$:

Similarly:

Summing over all examples:

Part (b): Solving for Optimal Parameters

Set both partial derivatives to zero.

From $\frac{\partial \mathcal{E}}{\partial b} = 0$:

Multiply both sides by $N$:

Factor out constants:

Note that $\sum_{i=1}^{N} x^{(i)} = N\bar{x}$ and $\sum_{i=1}^{N} t^{(i)} = N\bar{t}$ where $\bar{x}$ and $\bar{t}$ are the means. Dividing by $N$:

Solving for $b$:

Geometric interpretation: The regression line passes through the centroid $(\bar{x}, \bar{t})$ of the data.

From $\frac{\partial \mathcal{E}}{\partial w} = 0$:

Expand:

Substitute $b = \bar{t} - w\bar{x}$:

Group terms with $w$:

Therefore:

Alternative form using covariance and variance:

This is the classic formula for the slope of the best-fit line.

We’ll work through this systematically using intermediate variables and the chain rule.

(a) Define the prediction vector:

Taking the derivative:

This makes sense dimensionally: $\mathbf{y}$ is $N \times 1$, $\mathbf{w}$ is $(D+1) \times 1$, so the Jacobian must be $N \times (D+1)$, which is exactly $\mathbf{X}$.

(b) Define the residual vector:

Since $\mathbf{t}$ is constant with respect to $\mathbf{w}$:

(c) Define the squared norm:

This is a scalar. Taking the derivative with respect to the vector $\mathbf{r}$:

Explanation: For each component $r_j$, we have $\frac{\partial s}{\partial r_j} = 2r_j$ (from the chain rule on $(r_j)^2$). Stacking these gives $2\mathbf{r}^{\top}$ (a row vector).

(d) Define the cost function:

The derivative with respect to $s$:

(e) Apply the chain rule:

Now we chain everything together:

Substituting:

Simplifying:

Wait, wrong shape! This gives a row vector $(1 \times (D+1))$, but we need a column vector $((D+1) \times 1)$.

Correct form: Take the transpose:

Dimension check:

• $\mathbf{X}^{\top}$ is $(D+1) \times N$
• $\mathbf{r} = (\mathbf{X}\mathbf{w} - \mathbf{t})$ is $N \times 1$
• Product: $((D+1) \times N) \times (N \times 1) = (D+1) \times 1$ ✓

Key Insight: The gradient must be a column vector to match the shape of $\mathbf{w}$ for gradient descent updates. The chain rule naturally gives us $\mathbf{r}^{\top} \mathbf{X}$, but we transpose the entire expression to get the conventional form $\mathbf{X}^{\top} \mathbf{r}$.

The learning rate is too small. The loss has barely decreased (only 0.2 over 100 iterations), indicating the steps are too tiny.

Fixes:

1. Increase the learning rate, try 10x or 100x larger
2. Use adaptive learning rates, methods like Adam automatically adjust
3. Run for more iterations, though this is inefficient

How to diagnose: Plot the training curve. With a proper learning rate, you should see rapid initial decrease that levels off. A nearly flat line suggests the learning rate is too small.

We should increase $w$.

Reasoning:

• The gradient is negative (-3), meaning the cost decreases as $w$ increases
• Gradient descent update: $w \leftarrow w - \alpha \cdot (-3) = w + 3\alpha$
• The negative gradient causes $w$ to increase

Intuition: The gradient points toward steepest increase. A negative gradient means “increasing $w$ decreases cost.” We move opposite to the gradient, so we move in the direction that decreases cost.