Introduction to Machine Learning (CSC 2515) - Fall 2024
CSC 2515 Fall 2024: Introduction to Machine Learning
Machine learning (ML) is a set of techniques that allow computers to learn from data and experience, rather than requiring humans to specify the desired behaviour manually. ML has become increasingly central both in AI as an academic field, and in industry. This course introduces the main concepts and ideas in machine learning, and provides an overview of many commonly used machine learning algorithms. It also serves as a foundation for more advanced ML courses.
By the end of this course, the students will learn about (roughly categorized)
- Machine Learning Problems: Supervised (regression and classification), Unsupervised (clustering, dimension reduction), Reinforcement Learning
- Models: Linear and Nonlinear (Basis Expansion and Neural Networks)
- Loss functions: Squared Loss, Cross Entropy, Hinge, Exponential, etc.
- Regularizers: l1 and l2
- Probabilistic viewpoint: Maximum Likelihood Estimation, Maximum A Posteriori, Bayesian inference
- Bias and Variance Tradeoff
- Ensemble methods: Bagging and Boosting
- Optimization technique in ML: Gradient Descent and Stochastic Gradient Descent
The students are expected to learn the intuition behind many machine learning algorithms and the mathematics behind them. Through homework assignments, they will also learn how to implement these methods and use them to solve simple machine learning problems. More details can be found in syllabus (to be posted) and piazza.
Announcements:
Teaching Staff:
- Instructor: Rahul G. Krishnan
- Office Hour: Wednesdays, 5:00PM - 6:00PM, Pratt 286
- TA: Michael Cooper
- Office hours: TBA
- TA: Brian Lester
- Office hours: TBA
Time & Location:
Tuesdays (Lectures):
- Location: BA 1220
- Time: 4:00PM - 6:00PM
Thursdays (Tutorials)
- Location: BA 1240
- Time: 10:00AM - 11:00AM
Suggested Reading
No required textbooks. Suggested reading will be posted after each lecture (See lectures below).
- (ESL) Trevor Hastie, Robert Tibshirani, and Jerome Friedman, The Elements of Statistical Learning, 2009.
- (PRML) Christopher M. Bishop, Pattern Recognition and Machine Learning, 2006.
- (RLIntro) Richard S. Sutton and Andrew G. Barto Reinforcement Learning: An Introduction, 2018.
- (LNRL) Amir-massoud Farahmand Lecture Notes on Reinforcement Learning, 2021. (draft)
- (DL) Ian Goodfellow, Yoshua Bengio and Aaron Courville (2016), Deep Learning
- (MLPP) Kevin P. Murphy, Machine Learning: A Probabilistic Perspective, 2013.
- (ISL) Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani, Introduction to Statistical Learning, 2017.
- (SSBN) Shai Shalev-Shwartz and Shai Ben-David Understanding Machine Learning: From Theory to Algorithms, 2014.
- (ITIL) David MacKay, Information Theory, Inference, and Learning Algorithms, 2003.
Lectures
This is a tentative schedule, especially the parts related to tutorials, and it will change.
Note on videos: The videos from Fall 2021 version of this course are available at YouTube. Use them if you have to miss a class or you need to review the material. The content of this year will mostly be the same (70-90%).
| Week (date) | Topics | Material | Week’s Schedule | Suggested Reading |
|---|---|---|---|---|
| 1 (Sept 2-6) |
Lecture: Introduction to ML & K-Nearest Neighbors | Lecture: KNN slides video - part 1 video - part 2 Tutorial: Probability review (slides) |
Lec: Tue Tut: Thu |
ESL 1. ESL 2.(1-3), 2.5, 13.3 |
| 2 (Sept 9-13) |
Lecture: Decision Trees | DT slides video - part 1 video - part 2 Tutorial: Linear Algebra Review (slides, preliminaries) |
Lec: Tue Tut: Thu |
ESL 9.2 |
| 3 (Sept 16-20) |
Lecture: Regression and Classification with Linear Models | Linear Regression slides video - part 1 video - part 2 video - part 3 video - part 4 Tutorials: Optimization and CV (slides, code) |
Lec: Tue Tut: Thu |
ESL 3.(1,2,4), 4.(1,2,4); PRML 4.3 |
| 4 (Sept 23-27) |
Lecture: Regression and Classification with Linear Models (Continued); Bias-Variance Decomposition | Tutorial: Multilabel Classification | Lec: Tue Tut: Thu |
|
| 5 (Sept 30-Oct 4) |
Lecture: Bias-Variance Decomposition (Continued); Ensembles (Bagging) | B&V slides video - part 1 video - part 2 Tutorial: Gaussian distribution (slides) |
Lec: Tue Tut: Thu |
ESL 7.3, 8.7, 15.1-2 |
| 6 (Oct 7-11) |
Lecture: Neural Networks | NN slides video - part 1 video - part 2 Tutorial: PyTorch Overview (colab) |
Lec: Tue Tut: Thu |
notes |
| 7 (Oct 14-18) |
Lecture: Probabilistic Models | Probabilistic Models Slides Tutorial: LLMs as Generative Models (slides) |
Lec: Tue Tut: Thu |
ESL 2.6.3, 6.6.3, 4.3.0 |
| 8 (Oct 21-25) |
Lecture: Project Proposal Walkthrough | Lec: Tue Tut: Thu |
||
(Oct 28-Nov 1) |
Reading Week: No Class | |||
| 9 (Nov 4-8) |
Lecture: Principal Component Analysis | PCA Slides | Lec: Tue Tut: Thu |
ESL 14.5.1 |
| 10 (Nov 11-15) |
Lecture: K-Means Clustering | K-Means and EM Tutorial: EM (slides) |
Lec: Tue Tut: Thu |
PRML 9.1-2 |
| 11 (Nov 18-22) |
Lecture: Research Presentations | Lec: Tue Tut: Thu |
||
| 12 (Nov 25-29) |
Lecture: Self-Supervised Learning | Lec: Tue Tut: Thu |
Note on slides: Many have contributed to the design of these slides. Credit goes to several members of the ML Group at the U of T, and beyond, including (recent past, as far as I know): Amir-Massoud Farahmand, Roger Grosse, Murat Erdogdu, Richard Zemel, Juan Felipe Carrasquilla, Emad Andrews, and myself.
Assignments and Coursework
These are the main components of the course. The details are described below. You need to use MarkUs to submit your solutions (You need UTorID).
A schedule of the deliverables for this course, and their due dates, is provided below.
| Assignment | Weight | Duration | Release Date | Release Date (Day) | Due Date | Due Date (Day) |
|---|---|---|---|---|---|---|
| Assignment 1 (data) | 0.133 | 2 weeks | September 18, 2024 | Wednesday | October 3, 2024 | Thursday |
| Assignment 2 | 0.133 | 2 weeks | October 4, 2024 | Friday | October 18, 2024 | Friday |
| Assignment 3 (data + starter code) | 0.133 | 2.5 weeks | October 22, 2024 | Friday | November 8, 2024 | Friday |
| Project (Proposal) | 0 | 2 weeks | October 11, 2024 | Friday | October 25, 2024 | Friday |
| Project | 0.3 | 5 weeks | October 25, 2024 | Friday | November 29, 2024 | Friday |
| Take-Home Test | 0.1 | 1 week | November 29, 2024 | Friday | December 5, 2024 | Thursday |
| Reading Assignments | 0.1 | Ongoing | N/A | N/A | N/A | N/A |
| - Reading Assignment 1 | October 17, 2024 | Thursday | November 7, 2024 | Thursday | ||
| - Reading Assignment 2 | November 7, 2024 | Thursday | November 28, 2024 | Thursday | ||
| Q&A | 0.1 | Ongoing | N/A | N/A | N/A | N/A |
| - Q&A (Lectures 1-2) | Start of Semester | N/A | September 16, 2024 | Monday | ||
| - Q&A (Lectures 3-4) | Start of Semester | N/A | September 30, 2024 | Monday | ||
| - Q&A (Lectures 5-6) | Start of Semester | N/A | October 14, 2024 | Monday | ||
| - Q&A (Lectures 7-8) | Start of Semester | N/A | November 4, 2024 | Monday | ||
| - Q&A (Lectures 9-10) | Start of Semester | N/A | November 18, 2024 | Monday | ||
| - Q&A (Lectures 11-12) | Start of Semester | N/A | December 2, 2024 | Monday | ||
| Bonus (Finding typos in the slides, active class participation, evaluating the class, etc.) |
0.05 | Ongoing | Start of Semester | N/A | End of Semester | N/A |
| Total: | 1.05 |
Questions & Answers (Q&A) Assignments
The goal is to encourage you to reflect on the content of lectures. You do this by writing down two questions based on the content of the assigned lecture(s). You also need to try to answer those questions and write your answers down. The questions should be beyond what is covered in the lecture or discussed in the class. The goal is to make you think deeper about the content. You can search on the Internet to try to find answer to your questions. You do not need to answer the questions successfully or completely. It is enough to show that you have seriously thought about them. There is no strict rule about how long your submission should be, but an equivalent of 0.5-1 page should be enough.
We will be grouping the Q&A assignments in groups of two lectures. For each group of two lectures, you need to submit one Q&A assignment consisting of four questions.
Reading Assignments
The following papers are a combination of seminal papers in ML, topics that we didn’t cover in lectures, or active research areas. You need to choose five (5) papers out of them, depending on your interest. We will post the papers as the course progresses (so check here often). Please read them and try to understand them as much as possible. It is not important that you completely understand a paper or go into detail of the proofs (if there is any), but you should put some effort into it.
After reading each paper:
- You should summarize it in a short paragraph (100-200 words). Highlight the main points of the paper. Ignore the less interesting aspects.
- Try to come up with one or two suggestions on how the method/idea described in the paper can be used or extended.
Note: This list only covers a tiny fraction of all great papers published in various ML venues. Many of my favourites are not included.
Papers
- Ming Yuan and Yi Lin, “Model selection and estimation in regression with grouped variables,” Journal of Royal Statistical Society (B), 2006. PDF
- Ali Rahimi and Recht, “Random Features for Large-Scale Kernel Machines,” NIPS, 2007. PDF
- Leon Bottou and Olivier Bousquet, “The Tradeoffs of Large Scale Learning,” Advances in Neural Information Processing Systems (NeurIPS), 2007. PDF
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in Neural Information Processing Systems (NeurIPS), 2012. PDF
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, “Deep Residual Learning for Image Recognition,” CVPR, 2016. PDF
- Ian Goodfellow, Jonathan Shlens, and Christian Szegedy, “Explaining and Harnessing Adversarial Examples,” ICLR, 2015. PDF
- Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, Micah Goldblum, “A cookbook for self supervised learning,” PDF
- Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, “Scaling Laws for Neural Language Models,” PDF
- Jean Kaddour, Aengus Lynch, Qi Liu, Matt J. Kusner, Ricardo Silva, “Causal Machine Learning: A Survey and Open Problems,” PDF
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, “Attention is all you need,” PDF
Computing Resources
For the homework assignments, we will use Python, and libraries such as NumPy, SciPy, and scikit-learn. You have two options:
-
The easiest option is probably to install everything yourself on your own machine.
-
If you don’t already have python, install it. We recommend using Anaconda. You can also install python directly if you know how.
- Optionally, create a virtual environment for this class and step into it. If you have a conda distribution run the following commands:
conda create --name csc2515 source activate csc2515 - Use pip to install the required packages
pip install scipy numpy autograd matplotlib jupyter sklearn
-
-
All the required packages are already installed on the Teaching Labs machines.