mobile icon

Influence of spoken text on cognitive processing of complex pictorial presentations

WorkgroupRealistic depictions Lab
FundingDeutsche Forschungsgemeinschaft
Project description

The DFG-project "Influence of spoken text on cognitive processing of complex pictorial presentations" examines the influence of audio information texts on cognitive processing of art works as can be found in television documentaries or museums and exhibitions.

Based on models of multimedia learning and psychological findings on picture and scene perception, we examine in this DFG-project the influence of accompanying audio texts on the perception and processing of complex pictorial contents and related knowledge acquisition processes. In four laboratory experiments we will analyze to which extent audio texts (1) systematically reduce the interpretational openness of pictures, (2) foster the perception and processing of neglected pictorial elements, (3) perceptually and cognitively relate pictorial elements, and (4) contribute to the dual coding of the pictorial contents. Research objects are complex and detailed history paintings about historical events, which are typically used to communicate historical knowledge. In the four studies, we will examine the influence of auditive intro texts, named salient and non-salient pictorial elements, named relations between pictorial elements as well as the influence of text structure on the allocation of attention (gaze coherence, fixation times, amount of transitions, measured by eye tracking) and knowledge acquisition (free visual recall, text-picture-relation, transfer, memory related interindividual consistency). 

In addition to this project, several other studies are conducted which examine not only the influence of audio text design, but also the influence of visual design elements on the processing of paintings. Thereby, cinematographic techniques such as camera zooms and pans as well as cueing techniques from multimedia research play an important role.