Introduction to Machine Learning

203.4770 (3770)

Semester A

General Course Information

Meeting Times: Monday 9-12, Room

Instruction Hour: Wednesday 11:00-12:00, Room 410 (Jacobs)

Instructor: Dr. Rita Osadchy

e-mail: rita [at]cs [dot]haifa.ac.il
Office: Jacobs 410
__________________________________________________________________

Course Description

Machine learning is concerned with the development of computer algorithms that are able to learn solving tasks given a set of examples of those tasks and some prior knowledge about them. Machine learning has a wide spectrum of applications including handwritten or speech recognition, image classification, medical diagnosis, stock market analysis, bioinformatics etc. The goal of this course is to present the main concepts of modern machine learning methods including some theoretical background.

Recommended Prerequisites

The course assumes some basic knowledge of probability theory and linear algebra,
for example, you should be familiar with

• Joint and marginal probability distributions
• Normal (Gaussian) distribution
• Expectation and variance
• Statistical correlation and statistical independence

Tutorials of the above topics.

Problems, Concepts, Methods, and Tools within in the course

The list is partial and be can changed.

Problems

• Regression and classification
• Density estimation
• Clustering
• Model selection
• Inference

Concepts

• Estimation, bias, variance, loss, Empirical risk, maximum likelihood
• Generalization, overfitting
• Regularization
• Capacity, VC-dimension
• Generative/discriminative models

Models and Methods

• Linear regression
• Classification Linear Models
• Non-parametric Models
• Support Vector Machine (SVM)
• Decision Trees
• Boosting

Tools

• Cross-validation
• Gradient descent

The course will furthermore use several real-life applications to illustrate the interest of statistical machine learning.

Requirements

1) Home assignments 0-20% of the final grade (could be done in pairs but the pairs should be the same for all assignments).

2) Final exam 80-100%

Announcements

§ NEW: You are allowed to bring your notes limited to a double-sided A4 page in handwriting or in font size 10 or larger.

§ NEW: Assinment 2 was distributed via email. Due 19/02/2017. If you have not received it send me an email.

§ Home Assinment 1 was distributed via email. Due 05/01/2017. If you have not received it send me an email.

• Moed B was moved to 3/03/2017 due to collisions with other courses. Let me know via email (until Nov. 23) if you have a problem with the new date.
• All announcement, home assignments, and guidelines will be distributed by email.
• Those who do not send their contact address on time will not be added to the contact list!!!
• You must send me an email to (rita[at]cs [dot]haifa.ac.il) by November 7 from your active address with the subject “ML course contact”.

Lecture Notes

Date	Topic	Lecture notes	Reading material
31.10	Overview Introduction to Classification Started probability tutorial (see below)	PDF PDF
7.11	Introduction to Probability Bayesian Decision Theory, ML, MAP classifiers	PDF PDF	D.H.S. “Pattern Classification” Sections: 2.1-2.3
14.11	Bayesian Decision Theory, ML, MAP classifiers Bayesian Decision Tutorial	PDF
21.11	Normal Variables and their discriminant functions Parametric Density Estimation: Maximum Likelihood Estimation	PDF PDF	D.H.S. “Pattern Classification” Sections: 2.4-2.6 D.H.S. “Pattern Classification” Sections: 3.2-3.4, 3.5(3.5.1 only).
28.11	MLE Tutorial Parametric Density Estimation: Bayesian Estimation. Naïve Bayes	PDF example PDF	D.H.S. “Pattern Classification” Sections: 3.2-3.4, 3.5(3.5.1 only).
5.12	No class
12.12	Non-parametric density estimation, Histogram, Parzen Window, KNN	PDF PDF	D.H.S. “Pattern Classification” 4.1-4.3.4, 4.4-4.5,4.5.4,4.5.5,4.6
19.12	PCA, FDA,MDA	PDF PDF	D.H.S. “Pattern Classification” 3.7-3.8
26.12	LDF MSE	PDF PDF	D.H.S. “Pattern Classification” 5.2-5.4, 5.5.1 5.7,5.8,5.8.1,5.8.4 D.T.S. 5.11
2.01	SVM Multiclass classification	PDF PDF	D.H.S. “Pattern Classification” 5.11
9.01	Decision Trees Linear Regression	PDF PDF PDF (additional)	DT_extra_reading Hastie, Tibshirani Friedman “ The Elements of Statistical Learning” 3.1,3.2, 3.3, 3.4.3, 3.4.5.
16.01	Linear Regression Boosting	Cont. PDF	http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.56.9855
23.01	Computational Learning Theory Complexity, VC dimension	PDF PDF
25.01	clustering	PDF

Home Assignments:

General Instructions

· We will have 2-3 assignments this semester.

· You should submit a pdf file of the report and your implementation (running code) in a digital form. Zip it together and send me by email.

· Identical (or very similar solutions) are not allowed!

Textbooks:

• R. O. Duda, P. E. D. Hart, and G.Stork, Pattern Classification New York, NY: Wiley, 2000.
• T. Hastie, R. Tibshirani, and J. Friedman: Elements of Statistical Learning, Springer-Verlag, 2001.
• Christopher Bishop, Pattern Recognition and Machine Learning. Springer, August 2006.

Probability tutorials:

http://www-stat.stanford.edu/~susan/courses/s116/

Linear Algebra tutorial:

Eigen value decomposition

MATLAB resources:

Matlab is installed in the computer labs in Jacobs building.
For a student license see:
http://www.haifa.ac.il/index.php/he/2015-11-19-07-16-50

Introductory Tutorial

MATLAB tutorial from Carnegie Mellon University

  Slightly more advanced Tutorial

• MATLAB tutorial from Michigan Technological University

  More complete references/tutorials/FAQs

• The MathWorks home page for MATLAB