11-777 MMML | Schedule

Date	Lecture	Topics
8/30	Lecture 1.1: Course introduction [ slides \| video ]	Multimodal core challenges Course syllabus
9/1	Lecture 1.2: Multimodal applications [ slides \| video ]	Research tasks and datasets Team projects
9/6	Lecture 2.1: Background concepts [ slides \| video ]	Gradient and optimization Loss functions and neural networks
9/8	Lecture 2.2: Unimodal representations [ slides \| video ]	Dimensions of heterogeneity Visual representations
9/13	Lecture 3.1: Unimodal representations [ slides \| video ]	Language representations Signals, graphs and other modalities
9/15	Lecture 3.2: Multimodal representations [ slides \| video ]	Cross-modal interactions Multimodal fusion
9/20	Lecture 4.1: Multimodal representations [ slides \| video ]	Coordinated representations Multimodal fission
9/22	Lecture 4.2: Alignment and grounding [ slides \| video ]	Explicit alignment Multimodal grounding
9/27	Lecture 5.1: Project Hours (live working sessions instead of lectures)
9/29	Lecture 5.2: Aligned representations [ slides \| video ]	Self-attention transformer models Masking and self-supervised learning
10/4	Lecture 6.1: Aligned representations [ slides \| video ]	Multimodal transformers Video and graph representations
10/6	Lecture 6.2: Multimodal Reasoning [ slides \| video ]	Structured and hierarchical models Memory models
10/11	Lecture 7.1: Multimodal Reasoning [ slides \| video ]	Reinforcement learning Discrete structure learning
10/13	Lecture 7.2: Multimodal Reasoning [ slides \| video ]	Logical and causal inference External knowledge
10/18	Lecture 8.1: Fall Break – No lectures
10/20	Lecture 8.2: Fall Break – No lectures
10/25	Lecture 9.1: Generation [ slides \| video ]	Translation, summarization, creation Generative models, VAEs, flows
10/27	Lecture 9.2: Generation [ slides \| video ]	GANs and diffusion models Model evaluation and ethics
11/1	Lecture 10.1: Midterm presentations – No lectures
11/3	Lecture 10.2: Midterm presentations – No lectures
11/8	Lecture 11.1: Transference [ slides \| video ]	Modality transfer Transfer and multitask learning
11/10	Lecture 11.2: Transference [ slides \| video ]	Multimodal co-learning Co-training and self-training
11/15	Lecture 12.1: Quantification [ slides \| video ]	Heterogeneity and interactions Biases and fairness
11/17	Lecture 12.2: New research directions [ slides \| video ]	Recent approaches in multimodal ML
11/22	Lecture 13.1: Thanksgiving Week – No lectures
11/24	Lecture 13.2: Thanksgiving Week – No lectures
11/30	Lecture 14.1: Language, Vision, Actions [ slides \| video ]	Motion and navigation Robots and embodied AI
12/2	Lecture 14.2: Multimodal Applications [ slides \| video ]	Healthcare and affective computing Artificial social intelligence
12/6	Lecture 15.1: Final project presentations – No lectures
12/8	Lecture 15.2: Final project presentations – No lectures