November 2006 - October 2008
German Research Foundation (Deutsche Forschungsgemeinschaft)
According to the so-called modality principle, text, which accompanies pictures,
should be presented auditorily and not visually. A great number of empirical
studies have shown the advantage of presenting auditory text with picture in
contrast to presenting visual text with picture. The theoretical explanation
of the modality effect assumes a higher capacity of working memory when using
auditory text. In the present project, we wanted to find out, to what extent other
cognitive processes could explain the advantage of auditory text and pictures.
Thereby our main focus was on the aspect of simultaneous processing of verbal
material and pictorial information: While with picture and auditory text simultaneous
perception of the material is possible, this is not the case when presenting
visual text and picture. Here the learner has to chance his attention between
picture and visual text, to accomplish searching processes and to undertake
mental reconstructions of the material. Therefore, the modality effect could
be the result of absent temporal and spatial contiguity with visual text and
picture presentation. We explored these alternative explanations in cooperation
with the University of Tuebingen (Katharina Scheiter) and the University of
Saarland (Ralf Rummer).
