mobile icon

Cognitive Basics of the Modality Effect

Multimodal Interaction


November 2006 - October 2008


German Research Foundation (Deutsche Forschungsgemeinschaft)


screenshotAccording to the so-called modality principle, text, which accompanies pictures, should be presented auditorily and not visually. A great number of empirical studies have shown the advantage of presenting auditory text with picture in contrast to presenting visual text with picture. The theoretical explanation of the modality effect assumes a higher capacity of working memory when using auditory text. In the present project, we wanted to find out, to what extent other cognitive processes could explain the advantage of auditory text and pictures. Thereby our main focus was on the aspect of simultaneous processing of verbal material and pictorial information: While with picture and auditory text simultaneous perception of the material is possible, this is not the case when presenting visual text and picture. Here the learner has to chance his attention between picture and visual text, to accomplish searching processes and to undertake mental reconstructions of the material. Therefore, the modality effect could be the result of absent temporal and spatial contiguity with visual text and picture presentation. We explored these alternative explanations in cooperation with the University of Tuebingen (Katharina Scheiter) and the University of Saarland (Ralf Rummer).


  • University of Tuebingen
  • Saarland University


  • Schüler, A., Scheiter, K., Rummer, R., & Gerjets, P. (in press). Explaining the modality effect in multimedia learning: Is it due to a lack of temporal contiguity with written text and pictures? Learning and Instruction.
  • Schüler, A., Scheiter, K., Gerjets, P., & Rummer, R. (2008). Does a lack of contiguity with visual text cause the modality effect in multimedia learning? In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 2353-2358). Austin, TX: Cognitive Science Society.
  • Schüler, A., Scheiter, K., Gerjets, P., & Rummer, R. (accepted). The Modality Effect in Multimedia Learning: Theoretical and Empirical Limitations. Proceedings of the 8th International Conference of the Learning Sciences.