The course will present the fundamental mathematical concepts in machine learning and deep learning relevant to the six main challenges in multimodal machine learning: (1) representation, (2) alignment, (3) reasoning, (4) generation, (5) transference and (5) quantification. These include, but not limited to, multimodal transformers, neuro-symbolic models, multimodal tensor fusion, mutual information and multimodal graph networks. The course will also discuss many of the recent applications of MMML including multimodal affect recognition, multimodal language grounding and language-vision navigation

  • Time: Tuesday and Thursday 10:10-11:30 AM
  • Content: CMU Canvas
  • Location: Remote teaching – Zoom (see links in CMU Canvas)
  • Discussion and Q&A: Piazza
  • Assignment submissions: Gradescope (for registered students only)
  • Online lectures: The lectures will be recorded and made available on CMU Canvas for registered students. External link to the lectures on our Youtube channel!
  • Contact: Students should ask all course-related questions on Piazza, where you will also find announcements.