A PDF copy of the syllabus can be found here: IMT_3801.pdf

The third homework assignment has been posted.

April 15: We showed that, under a strong convexity assumption, gradient descent requires $LaTeX: \mathcal{O}\bigl(\log(1/\varepsilon)\bigr)$ iterations to achieve the same level of accuracy, yielding a significantly faster convergence rate.

April 13: We showed that gradient descent requires $LaTeX: \mathcal{O}(1/\varepsilon)$ iterations to approximate the global minimum of a convex loss function to within accuracy $LaTeX: \varepsilon.$

Notes: GD_convergence.pdf

April 8: We reviewed selected problems from the second homework assignment and discussed the midterm projects.

April 6: We introduced stochastic gradient descent and used it to train a shallow neural network classifier.

March 25 & 30: We presented the backpropagation algorithm for feedforward neural architectures and briefly discussed the vanishing gradient problem, which arises with certain activation functions.

Notes: SGD.pdf

March 23: We proved a depth-efficiency result for ReLU neural networks, showing that functions in LaTeX: L^q(K) for a compact domain $LaTeX: K\subset\mathbb{R}^d$ can be approximated with arbitrary accuracy by networks with logarithmic depth in the dimension.

The second homework assignment has been posted.

March 18: We derived Cybenko’s theorem on universal approximation for shallow neural networks, emphasizing the role of discriminatory activation functions.

March 11: We discussed feedforward neural architectures and the role of activation functions in enabling universal approximation in suitable function spaces. In preparation for next week’s results, we reviewed a corollary of the Hahn–Banach theorem and extensions of Riesz’s representation theorem.

Notes: neural_networks.pdf

March 9: We covered the basics of linear and logistic regression and discussed how the cross-entropy loss emerges naturally from a maximum likelihood formulation under an independence assumption for the training labels.

The first homework assignment has been posted.

Notes: regression_models.pdf

March 4: We introduced the main themes of the course and covered practical matters, including the grading scheme and other key details.

My lecture notes on probability theory can be found here: probability.pdf