Multimodal Deep Learning 1 Introduction 1.1 Introduction to Multimodal Deep Learning . . . . . . . . . . 1.2 Outline of the Booklet . . . . . . . . . . . . . . . . . . . . . . 2 Introducing the modalities 2.1 State-of-the-art in NLP . . . . . . . . . . . . . . . . . . . . . 2.2 State-of-the-art in Computer Vision . . . . . . . . . . . . . . 2.3 Resources and Benchmarks for NLP, CV and multimodal tasks 3 Multimodal architectures 3.1 Image2Text . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Text2Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Images supporting Language Models . . . . . . . . . . . . . . 3.4 Text supporting Vision Models . . . . . . . . . . . . . . . . . 3.5 Models for both modalities . . . . . . . . . . . . . . . . . . . 4 Further Topics 4.1 Including Further Modalities . . . . . . . . . . . . . . . . . . . 181 4.2 Structured + Unstructured Data . . . . . . . . . . . . . . . . . 197 4.3 Multipurpose Models . . . . . . . . . . . . . . . . . . . . . . 209 4.4 Generative Art . . . . . . . . . . . . . . . . . . . . . . . . . . 226 5 Conclusion 6 Epilogue 6.1 New influential architectures . . . . . . . . . . . . . . . . . . . 237 6.2 Creating videos . . . . . . . . . . . . . . . . . . . . . . . . . 238 7 Acknowledgements Preface Author: Matthias Aßenmacher FIGURE 1: LMU seal (left) style-transferred to Van Gogh’s Sunflower painting (center) and blended with the prompt - Van Gogh, sunflowers via CLIP+VGAN (right). In the last few years, there have been several breakthroughs in the methodologies used in Natural Language Processing (NLP) as well as Computer Vision (CV). Beyond these improvements on single-modality models, large-scale multimodal approaches have become a very active area of research. In this seminar, we reviewed these approaches and attempted to create a solid overv