**362.2.5322 | Machine Learning and Neural Network Design (Three Credit Hours)**

Professor: Simon D. Levy
Schedule: Monday / Wednesday 19:00 – 22:00, via Zoom
E-mail: simon.d.levy@gmail.com

Textbooks (optional): Although they are not required for the course, I have drawn on these books in creating the lecture slides. I suggest them to anyone interested in delving further into the material.

Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms (2nd Edition). N. Buduma, N. Buduma, and J. Papa. This was one of the first textbooks to cover the field of Deep Learning from first principles. Although other books may have surpassed it in popularity, it remains (in my view) the most comprehensive and easiest to understand.

Deep Reinforcement Learning Hands-On (2nd Edition). M. Lapan, Packt Publishing 2020. A practical introduction focusing on programming exercises, with minimal mathematical theory.

Foundations of Deep Reinforcement Learning: Theory and Practice in Python. L. Graesser & W.L. Keng, Addison-Wesley Professional 2019. Consistent with its name, this book provides a nice balance between mathematical theory (which can get heavy at times!) and programming.

Objectives

By the end of this course you will be able to

- Code up fundamental machine-learning algorithms like Q-Learning and Back-propagation
- Use the Gymnasium framework for testing Reinforcement Learning models.
- Use the PyTorch machine-learning framework to implement Deep Learning (DL) and Deep Reinforcement Learning (DRL) models.
- Use the new Isaac Gym physics simulation and learning environment to build and train physically realistic robots.
- Think critically about the current limitations of DRL for robotics, and the prospects for the future.

Approach

Starting from first principles, we will review the problem of learning and popular solutions, from classic linear regression through today’s deep policy-gradient algorithms. As you will see, some of these algorithms are simple enough to be coded up “by hand” in your favorite programming language; for others, we will rely on powerful software packages like PyTorch. Each of the algorithms we study will be motivated by the problems left unsolved by the previous algorithm:

1. Supervised learning
  1. Linear regression
  2. Single-layer perceptron
  3. Multi-layer perceptron / back-propagation
2. Reinforcement learning
  1. Discrete observations, discrete control (Q-Learning)
  2. Continuous observations, discrete control (Deep Q-Learning)
  3. Continuous observations, continuous control (Actor/Critic)

Prerequisites

To do well in this class, you should have some programming experience (Python, Matlab, Java, C/C++). Although the software libraries we will use are built around Python, I am familiar with those other languages and can help you make the transition to Python at a reasonable pace.

Grading

- Weekly problem sets: 100%, due 23:59 each Wednesday on Github
- Optional Final project: This should be a small project applying some of the ideas and tools we have learned in class to your own research, which you can substitute for two problem sets. If you choose this option, you will need to notify me as soon as possible.

Because of the fast pace of the course, no late work will be accepted. The only exceptions will be for reserve-duty commitments and genuine family/medical emergencies.

Please note that you will get zero credit for a program that crashes when I try to run it. This will typically happen if you do not download and re-test your code after submitting it to your repository. It is common for me to hear but the program ran on my computer; however, this is typically the result of submitting the wrong copy of your program, and is not a legitimate excuse for submitting something that doesn’t run – just as it is not an excuse for delivering a defective product to your customer. It is better to submit a partial solution that runs, than a complete solution that crashes.

Tentative Schedule

Monday

Wednesday

7 August

Week 1:

Course Outline What is (A)I?

Linear Least Squares

Perceptron Learning

Reading: Deep Learning Intro Article (here’s an update!)

Limits of Perceptrons

Back-propagation with hidden units

Back-prop II: Improvements