362.2.5322  Machine Learning and Neural Network Design (Three Credit Hours)
Professor: Simon D. Levy Schedule: Monday / Wednesday 19:00 – 22:00, via Zoom Email: simon.d.levy@gmail.com 
Textbooks (optional): Although they are not required for the course, I have drawn on these books in creating the lecture slides. I suggest them to anyone interested in delving further into the material. Fundamentals of Deep Learning: Designing NextGeneration Machine Intelligence Algorithms (2nd Edition). N. Buduma, N. Buduma, and J. Papa. This was one of the first textbooks to cover the field of Deep Learning from first principles. Although other books may have surpassed it in popularity, it remains (in my view) the most comprehensive and easiest to understand. Deep Reinforcement Learning HandsOn (2nd Edition). M. Lapan, Packt Publishing 2020. A practical introduction focusing on programming exercises, with minimal mathematical theory. Foundations of Deep Reinforcement Learning: Theory and Practice in Python. L. Graesser & W.L. Keng, AddisonWesley Professional 2019. Consistent with its name, this book provides a nice balance between mathematical theory (which can get heavy at times!) and programming. 
Objectives
By the end of this course you will be able to

 Code up fundamental machinelearning algorithms like QLearning and Backpropagation
 Use the Gymnasium framework for testing Reinforcement Learning models.
 Use the PyTorch machinelearning framework to implement Deep Learning (DL) and Deep Reinforcement Learning (DRL) models.
 Use the new Isaac Gym physics simulation and learning environment to build and train physically realistic robots.
 Think critically about the current limitations of DRL for robotics, and the prospects for the future.
Approach
Starting from first principles, we will review the problem of learning and popular solutions, from classic linear regression through today’s deep policygradient algorithms. As you will see, some of these algorithms are simple enough to be coded up “by hand” in your favorite programming language; for others, we will rely on powerful software packages like PyTorch. Each of the algorithms we study will be motivated by the problems left unsolved by the previous algorithm:

 Supervised learning
 Linear regression
 Singlelayer perceptron
 Multilayer perceptron / backpropagation
 Reinforcement learning
 Discrete observations, discrete control (QLearning)
 Continuous observations, discrete control (Deep QLearning)
 Continuous observations, continuous control (Actor/Critic)
 Supervised learning
Prerequisites
To do well in this class, you should have some programming experience (Python, Matlab, Java, C/C++). Although the software libraries we will use are built around Python, I am familiar with those other languages and can help you make the transition to Python at a reasonable pace.
Grading

 Weekly problem sets: 100%, due 23:59 each Wednesday on Github
 Optional Final project: This should be a small project applying some of the ideas and tools we have learned in class to your own research, which you can substitute for two problem sets. If you choose this option, you will need to notify me as soon as possible.
Because of the fast pace of the course, no late work will be accepted. The only exceptions will be for reserveduty commitments and genuine family/medical emergencies.
Please note that you will get zero credit for a program that crashes when I try to run it. This will typically happen if you do not download and retest your code after submitting it to your repository. It is common for me to hear but the program ran on my computer; however, this is typically the result of submitting the wrong copy of your program, and is not a legitimate excuse for submitting something that doesn’t run – just as it is not an excuse for delivering a defective product to your customer. It is better to submit a partial solution that runs, than a complete solution that crashes.
Tentative Schedule
Monday 
Wednesday 

7 August Week 1:

Course Outline What is (A)I? Reading: Deep Learning Intro Article (here’s an update!) 
Backpropagation with hidden units Reading: Buduma Chapter 1 
14 August Week 2:

Logistic Regression & SoftMax Intro to PyTorch Code: pytorch.py Reading: Buduma Chapters 3,4 Reading: Buduma Chapter 2 
Reading: Buduma Chapter 5 Due: Assignment #1 
21 August Week 3:

Reading: Buduma Chapter 7 
Due: Assignment #2 
28 August Week 4: 
Reading; Attention is All You Need **************************** Due: Assignment #3 

4 September Week 5 
Due: Assignment #4 

11 September Week 6 
Policy Gradient, Part I  
18 September Week 7 
Project presentations 

25 September 
Due: Assignment #6 or final project 