Date Lecture Topics
8/30 Lecture 1.1: Course introduction
[ slides | video ]

Multimodal core challenges
Course syllabus
9/1 Lecture 1.2: Multimodal applications
[ slides | video ]

Research tasks and datasets
Team projects
9/6 Lecture 2.1: Background concepts
[ slides | video ]

Gradient and optimization
Loss functions and neural networks
9/8 Lecture 2.2: Unimodal representations
[ slides | video ]

Dimensions of heterogeneity
Visual representations
9/13 Lecture 3.1: Unimodal representations
[ slides | video ]

Language representations
Signals, graphs and other modalities
9/15 Lecture 3.2: Multimodal representations
[ slides | video ]

Cross-modal interactions
Multimodal fusion
9/20 Lecture 4.1: Multimodal representations
[ slides | video ]

Coordinated representations
Multimodal fission
9/22 Lecture 4.2: Alignment and grounding
[ slides | video ]

Explicit alignment
Multimodal grounding
9/27 Lecture 5.1: Project Hours (live working sessions instead of lectures)
9/29 Lecture 5.2: Aligned representations
[ slides | video ]

Self-attention transformer models
Masking and self-supervised learning
10/4 Lecture 6.1: Aligned representations
[ slides | video ]

Multimodal transformers
Video and graph representations
10/6 Lecture 6.2: Multimodal Reasoning
[ slides | video ]

Structured and hierarchical models
Memory models
10/11 Lecture 7.1: Multimodal Reasoning
[ slides | video ]

Reinforcement learning
Discrete structure learning
10/13 Lecture 7.2: Multimodal Reasoning
[ slides | video ]

Logical and causal inference
External knowledge
10/18 Lecture 8.1: Fall Break – No lectures
10/20 Lecture 8.2: Fall Break – No lectures
10/25 Lecture 9.1: Generation
[ slides | video ]

Translation, summarization, creation
Generative models, VAEs, flows
10/27 Lecture 9.2: Generation
[ slides | video ]

GANs and diffusion models
Model evaluation and ethics
11/1 Lecture 10.1: Midterm presentations – No lectures
11/3 Lecture 10.2: Midterm presentations – No lectures
11/8 Lecture 11.1: Transference
[ slides | video ]

Modality transfer
Transfer and multitask learning
11/10 Lecture 11.2: Transference
[ slides | video ]

Multimodal co-learning
Co-training and self-training
11/15 Lecture 12.1: Quantification
[ slides | video ]

Heterogeneity and interactions
Biases and fairness
11/17 Lecture 12.2: New research directions
[ slides | video ]

Recent approaches in multimodal ML
11/22 Lecture 13.1: Thanksgiving Week – No lectures
11/24 Lecture 13.2: Thanksgiving Week – No lectures
11/30 Lecture 14.1: Language, Vision, Actions
[ slides | video ]

Motion and navigation
Robots and embodied AI
12/2 Lecture 14.2: Multimodal Applications
[ slides | video ]

Healthcare and affective computing
Artificial social intelligence
12/6 Lecture 15.1: Final project presentations – No lectures
12/8 Lecture 15.2: Final project presentations – No lectures