In this lecture, we will cover the most recent approaches and principles at the frontier of learning and applying of foundation models in language, vision (image generation, image segmentation, image understanding) as well as multimodal (=multimedia) data processing, including
learning vision foundation models from natural language supervision, with applications to open-vocabulary image classification and retrieval, object detection, segmentation, and multimodal understanding;
learning vision foundation models via masked image modeling, with its extensions to multimodal pre-training
vision foundation model architecture design with transformer and beyond.
The lecture will be regularly updated to cover the latest developments in the field.
?
Exercises
There will be exercises each week. The solutions are discussed in the exercise lecture in the following week. The exercises are not mandatory (but essential for passing the exam).
