• Press Information
  • Contact
  • deutsch | english
Multimodal interaction

Multimodal Interaction Lab

Hypermedia-based learning environments are characterized by a non-linear connection of information nodes containing multimedia materials. These hyperlinked environments can be explored by users in an interactive and self-directed way.

The lab investigates three types of hypermedia environments: First, the World Wide Web (WWW) as a global network of information nodes of diverse origin and quality that can be explored by means of web browsers and search engines. Here, our research mainly addresses processes of evaluating and integrating different sources of information during web search. Second, instructional hypermedia environments that are designed to support learners of different needs. In particular, we are interested in how innovative technologies can be used to adapt these environments to learners’ prerequisites and to their learning progress. Third, we analyze the potentials of novel hypermedia environments that are based on interactive displays (e.g., multitouch tables, smartphones, or tablets). Here, we focus on the direct and intuitive manipulation of information by means of multitouch interaction and on the use of research paradigms from the field of embodied cognition for studying processes of knowledge acquisition.

With regard to all abovementioned types of hypermedia environments, we focus on two pivotal issues: How to support processes of comprehension by means of hypermedia and how to avoid potential cognitive overload due to the large amounts of information, presentation modes, and interaction requirements provided to users. We assume that a deeper understanding of hyperlinked information can be best supported when hypermedia design features are matched to users' particular prerequisites. Accordingly, the lab comprises two lines of research, addressing on the one hand the design and affordances of hyperlinked information environments, and on the other hand user prerequisites and cognitive resources.

Design and affordances of hyperlinked information environments

This line of research investigates how design features of hypermedia environments influence the resulting affordances with regard to user behavior and cognitive processing. For instance, we study which features of web resources and search interfaces stimulate users to elaborate on the origin and quality of web information. Moreover, we analyze which affordances for information processing are provided when embedding static and dynamic visualizations within hypermedia environments. A third core area is related to informal learning contexts like museums and addresses the design of intuitive hypermedia environments based on interactive displays. Here, we focus on the issue of how a direct haptic interaction with representations by means of gestures (“embodied interaction”) can not only improve an intuitive usability but also facilitate the acquisition of conceptual structures.

User prerequisites and cognitive resources

This line of research investigates important cognitive prerequisites for successful hypermedia exploration. One focus of the lab is on optimizing the management of limited cognitive resources like working memory capacity. For instance, we study how neural signatures of working-memory processes can be used to measure different aspects of working-memory load during learning in real time. The long-term goal of this work is to develop innovative and adaptive hypermedia environments, which allow reducing the amount of cognitive overload, particularly for users with unfavorable learning prerequisites. A second – and complementary – focus is on developing hypermedia environments for learners with exceptionally favorable learning prerequisites. For instance, we develop and optimize hypermedia-based iPad environments that address the specific learner prerequisites of gifted primary school children to stimulate discovery learning, e.g., in domains such as understanding biodiversity or musical structures.

Team Multimodal interaction


Drawing digital self-portraits in Art museum: Which receptive and productive processes mediate social transfer effects on empathy and self-perception of adolscents?

What could self-portraits in a museum teach us? Do self-portraits mediate more than knowledge about drawing style and epochal features? In the project DigiSelbst, we will focus on visual arts (especially self-portraits) in Art museum and the use of digital media to design cultural learning opportunities.
In an interdisciplinary cooperation of IWM and the Herzog Anton Ulrich-Museum in Braunschweig, three courses for upper secondary schools will be developed. In an experimental setting, we will evaluate the effect of different course contents on social skills like empathy and self-perception. Thereby, we will design (1) an emotion course, which focuses on perception and interpretation of own and foreign emotions based on mimical expressions, (2) a self-concept course, which focuses on differential access to personal characteristics in context of social roles, (3) an epochal course, which focuses on the conveying of epochal knowledge, e.g. hair accessories or clothing style in the Baroque period. The aim of the study is to evaluate, if there is a supporting effect on social skills especially in the emotion and the self-concept course.
The core element of this course program is the graphical interaction with tablets. Digital drawing offer new ways of dealing with the course contents (e.g. zooming, editing) and increases the motivation of the participants. Furthermore, tablets offer a high self-reference through the creation of selfies and, additionally, enable the recording and detailed scientific examination of the drawing process.

EEG-based neural signatures of different types of working-memory load

The project is part of Cluster01 in the ScienceCampus Tuebingen "Informational Environments". The cluster aims at bridging the gap between neuroscience and instructional psychology via informatics by developing an online adaptive learning environment based on physiological load-measures. Specifically, the online adaption grounds on specific types of working memory load via neural signatures in the electroencephalogram (EEG) of learners by means of advanced braincomputer interface (passive BCI) methodologies.

Intuitive and collaborative interaction with digital media on tabletop-displays

This interdisciplinary project aims at the conception, implementation and optimization of an innovative multimedia guide for museum and exhibitions (EyeVisit). The main concept is to present digital information by utilizing interactive displays. According to interaction with real-world objects, EyeVisit allows the manipulation of digital visual information with intuitive gestures for moving, rotating and scaling.

Processing dynamic visualizations with mirror neurons

This project investigates the idea that the activation of specific areas in the brain is beneficial for learning about continuous processes with dynamic visualizations. These areas in the brain (the so-called human mirror-neuron-system) are used to understand and imitate actions of other persons. We addressed, whether gestures that correspond or do not correspond to the to-be-learned processes, activate the mirror-neuron-system and enhance learning outcomes.

Reception of the scientific and public discourse on nanosafety on the Internet

Nanotechnology is considered one of the key technologies of the 21st century. A general statement on the effect of nanomaterials on people and the environment however is not yet possible. As part of the Leibniz Research Alliance „Nanosafety“, this project focuses on how laypersons and experts deal with conflicting scientific information on this topic and how they make decisions based on this information. We look at this research question with special regard to the internet as a source of knowledge.

Supporting evaluation processes during web search

The WWW offers easy and fast access to a vast amount of information, which, however, can vary greatly in its quality. Previous empirical studies have shown that both school students and adults don’t often spontaneously pay attention to and critically evaluate where the information stems from. Therefore, this project investigates different approaches how Web users can be supported in their evaluation and selection of information during Web search.

The influence of hand proximity on cognitive and emotional information processing with multi-touch interfaces

The use and application of interactive multi-touch displays increase continuously. Thus, touch-based user interfaces like multi-touch tables, tablets or smartphones are used in many public sector institutions and private households. These interfaces enable users to directly manipulate external representations using their hands without the requirement of an indirect interaction device (e.g., mouse). The question arises how touch-based interactions should be implemented to facilitate the processing of information.

The influence of user- and text characteristics on source evaluation during Web search

The WWW is characterized by an unprecedented amount of information and a high heterogeneity of information quality. For a successful information search, for example, about a conflicting science-related or medical issue, thus, it is often important to evaluate the information sources regarding their trustworthiness and to compare and weigh (potentially conflicting) information from multiple sources. In this project we investigate through eyetracking analyses and log file data, verbal protocols, as well as argumentative summaries how Web users evaluate and process information during Web search about conflicting topics. Specifically, the project aims at examining how certain user characteristics and certain text characteristics influence source evaluations during Web search.

The Interactive Ward Round Table – A Cognitive Interface for the Integration of Multiple Documents in Medical Team Meetings

In the daily ward round physicians use several documents for the optimal treatment of a patient. Medical information systems support this process. However, they are typically not optimized to integrate different information sources into one coherent overall picture. The aim of this project is to develop and implement a new cognitive interface to support physicians in their cognitive processes when working with multiple documents in the daily ward round.

Former Projects

graduation papers

Der Einfluss von Hyperlinks auf das Textverständnis beim Lesen von Wikipedia-Artikeln

Online-Artikel (z.B. in Wikipedia) enthalten oftmals eine beträchtliche Anzahl an Hyperlinks zu weiterführenden Informationen. Wie beeinflussen das Vorhandensein solcher Links und deren Selektion das Textverständnis beim Lesen von Wikipedia(-artigen) Artikeln? Sind die Links ablenkend? Oder führt die Auswahl relevanter Links zu einem tieferen Textverständnis?

Methode: Experimentalstudien mit Blickbewegungsmessung

Ihre Ansprechpartnerin ist: Frau Dr. Yvonne Kammerer

Wie kann das Spiegelneuronensystem aktiviert werden und ist diese Aktivierung hilfreich für das Lernen über nicht-menschliche Bewegungen?

Dynamische Visualisierungen können besonders hilfreich sein, wenn Lernende sich die dargestellten Bewegungen mit Hilfe des eigenen Körpers vorstellen. Dies kann erreicht werden indem sogenannte Spiegelneurone durch menschliche Gesten aktiviert werden. Dabei stellt sich die Frage, ob es reicht, wenn Lernende die menschlichen Gesten gezeigt bekommen oder ob sie diese selbst ausführen müssen. Zusätzlich können diese Befunde in verschiedenen Lerndomänen und –inhalten überprüft werden.

Methode: Experimentalstudie mit Messung der Gehirnaktivierung (fNIRS)

Ihre Ansprechpartnerin ist: Frau Dr. Birgit Brucker

Interaktive Displays und embodied cognition

Interaktive Displays verwenden eine intuitive und niederschwellige Gestensteuerung zur Interaktion mit Informationseinheiten sowie zur Aktivierung von Informationsverknüpfungen (z.B. iPhones oder Multi-Touch Tabletop-Systemen). Wie wird die kognitive Repräsentation einer konzeptuellen oder räumlichen Struktur durch die Darbietung auf einem interaktiven Display beeinflusst, mit dem der Nutzer direkt physikalisch durch Gesten oder Tangible Objects interagieren kann?

Methode: Experimentalstudie, die auf grundlagen- oder anwendungsbezogenem Material beruhen kann

Ihr Ansprechpartner ist: Herr Prof. Dr. Peter Gerjets

Können dynamische Visualisierungen, die gleichzeitig dargeboten werden, das Lernen unterstützen?

Auf größeren interaktiven Oberflächen (wie z.B. Multitouch-Tischen) können leicht mehrere dynamische Visualisierungen gleichzeitig abgespielt werden. Können Lernende mit diesem Format (multiple Animationen) Zusammenhänge leichter erkennen oder lenken multiple Animationen die Lernenden eher ab, da die Aufmerksamkeit auf unterschiedliche Visualisierungen verteilt werden müssen?

Methode: Experimentalstudie am Multitouch-Tisch

Ihre Ansprechpartnerin ist: Frau Dr. Birgit Brucker

Entdeckung von Inkonsistenzen und Vertrauenswürdigkeitsbewertungen bei der Rezeption von Informationen im Internet

Informationen im Internet können widersprüchliche oder fehlerhafte Informationen enthalten. Bemerken Internetnutzer Inkonsistenzen innerhalb einer Webseite? Und welche Rolle spielt dabei die Art der Webseite (z.B. Forum oder Uni-Seite)? Bemerken Internetnutzer Inkonsistenzen zwischen mehreren Webseiten? Und welchen Einfluss hat dies auf die Bewertung der Vertrauenswürdigkeit der sich widersprechenden Webseiten?

Methode: Experimentalstudien mit Blickbewegungsmessung

Ihre Ansprechpartnerin ist: Frau Dr. Yvonne Kammerer

Hilft direktes Interagieren beim Lernen?

Interaktive Oberflächen (wie z.B. Smartphones, Tablets, Multitouch-Tische) erlauben dem Lerner eine direkte intuitive Interaktion mit den dargestellten Objekten und Inhalten. Ist diese direkte haptische Interaktion (embodied interaction) hilfreich für das Lernen? Mögliche Themen sind Vergleiche mit Standard-GUIs; Variationen der räumlichen und zeitlichen Aspekte der Interaktion oder der Ausrichtung der Präsentationsfläche (Tisch vs. Wand).

Methode: Experimentalstudie mit interaktiven Geräten; Usabilitystudien

Ihre Ansprechpartnerin ist: Frau Dr. Birgit Brucker