1Classification Using Linear Regression

We will try to use linear regression to do classification. Although this is not a good idea in general (as discussed in the class), it will work for simple data. Concretely, we will use the height-weight data from the first exercise to predict the binary valued gender (sex). For better visualization, we will randomly sample 200 data points from the data.

Exercise 1:

Classification using linear regression.

•  Use least squares(y, tx) from the previous exercise to compute the weights w on the height-weight data. Please COPY your previous implementation to the template file of this exercise least squares.py.

•  Visualize the data points and the decision boundary with visualization() as in Figure 1.

Figure 1: classification by least square.

2 Logistic Regression

Exercise 2:

Implement logistic regression using gradient descent.

•  Fill in the notebook function sigmoid().

•  Fill in the two notebook functions calculate loss() and calculate gradient(). The first function should return the negative log-likelihood loss, while the second function should return the gradient of this loss w.r.t. the parameters w.

•  Implement gradient descent learning by gradient descent() for logistic regression. You should calculate the loss and its gradient w.r.t. w. Now, you can update the weights w, and the function should return the loss and these updated weights w.

•  Plot the predictions to get a visualization similar to the right part of Figure 1. Check if you get similar or different results.

Now that we have gradient descent, we can easily implement Newton’s method.

Exercise 3:

Newton’s method for logistic regression

wt+1 = wt − γ∇2L(wt)−1∇L(wt).

•  Fill in the notebook function calculate hessian(). Now, complete logistic regression() by using calculate loss(), calculate gradient() and calculate hessian(). This function logistic regression() should return the loss, gradient, and Hessian. Note that it should not update nor return the parameters w. • Your gradient descent code can now be turned into a Newton’s method algorithm to find the optimal parameters w. Please fill in the notebook function learning by newton method(), which does a single parameter update. The function should return the loss and the updated parameters. hint: Use np.linalg.solve instead of directly computing the inverse ∇2L(wt)−1.

2

Exercise 4:

Penalized logistic regression.

•  Fill in the notebook function penalized logistic regression(). This requires you to add the regularization term λ∥w∥2. As a sanity check, set λ to a very small value and see if it gives the same result as before. Once complete, please fill in the notebook function learning by penalized gradient(), and increase the value of λ and check whether (the norm of) w is shrinking or not. (Note: you should not include the penalty term in your loss value calculation).

•  Check if this gives the same answer as gradient descent. To debug, print the function value and the norm of the gradient in every iteration. All these values should decrease in every iteration.

3