mobile icon
Multimodal interaction

Multimodal Interaction Lab

Hypermedia-based learning environments are characterized by a non-linear connection of information nodes containing multimedia materials. These hyperlinked environments can be explored by users in an interactive and self-directed way.

The lab investigates three types of hypermedia environments: First, the World Wide Web (WWW) as a global network of information nodes of diverse origin and quality that can be explored by means of web browsers and search engines. Here, our research mainly addresses processes of evaluating and integrating different sources of information during web search. Second, instructional hypermedia environments that are designed to support learners of different needs. In particular, we are interested in how innovative technologies can be used to adapt these environments to learners’ prerequisites and to their learning progress. Third, we analyze the potentials of novel hypermedia environments that are based on interactive displays (e.g., multitouch tables, smartphones, or tablets). Here, we focus on the direct and intuitive manipulation of information by means of multitouch interaction and on the use of research paradigms from the field of embodied cognition for studying processes of knowledge acquisition.

With regard to all abovementioned types of hypermedia environments, we focus on two pivotal issues: How to support processes of comprehension by means of hypermedia and how to avoid potential cognitive overload due to the large amounts of information, presentation modes, and interaction requirements provided to users. We assume that a deeper understanding of hyperlinked information can be best supported when hypermedia design features are matched to users' particular prerequisites. Accordingly, the lab comprises two lines of research, addressing on the one hand the design and affordances of hyperlinked information environments, and on the other hand user prerequisites and cognitive resources.

Design and affordances of hyperlinked information environments

This line of research investigates how design features of hypermedia environments influence the resulting affordances with regard to user behavior and cognitive processing. For instance, we study which features of web resources and search interfaces stimulate users to elaborate on the origin and quality of web information. Moreover, we analyze which affordances for information processing are provided when embedding static and dynamic visualizations within hypermedia environments. A third core area is related to informal learning contexts like museums and addresses the design of intuitive hypermedia environments based on interactive displays. Here, we focus on the issue of how a direct haptic interaction with representations by means of gestures (“embodied interaction”) can not only improve an intuitive usability but also facilitate the acquisition of conceptual structures.

User prerequisites and cognitive resources

This line of research investigates important cognitive prerequisites for successful hypermedia exploration. One focus of the lab is on optimizing the management of limited cognitive resources like working memory capacity. For instance, we study how neural signatures of working-memory processes can be used to measure different aspects of working-memory load during learning in real time. The long-term goal of this work is to develop innovative and adaptive hypermedia environments, which allow reducing the amount of cognitive overload, particularly for users with unfavorable learning prerequisites. A second – and complementary – focus is on developing hypermedia environments for learners with exceptionally favorable learning prerequisites. For instance, we develop and optimize hypermedia-based iPad environments that address the specific learner prerequisites of gifted primary school children to stimulate discovery learning, e.g., in domains such as understanding biodiversity or musical structures.

Team Multimodal interaction


EEG-based neural signatures of different types of working-memory load

The project aims at bridging the gap between neuroscience and instructional psychology via informatics by studying the use of brain-computer interface (BCI) methodologies for research in instructional psychology (e.g., the individual classification of working memory load in real-time). As long-term goal stands the development of an online adaptive learning environment based on physiological measures of basic cognitive processes. Specifically, the online adaption grounds on the detection and classification of specific neural signatures in the electroencephalogram (EEG) of learners by means of advanced BCI methodologies.

Intuitive and collaborative interaction with digital media on tabletop-displays

This interdisciplinary project aims at the conception, implementation and optimization of an innovative multimedia guide for museum and exhibitions (EyeVisit). The main concept is to present digital information by utilizing interactive displays. According to interaction with real-world objects, EyeVisit allows the manipulation of digital visual information with intuitive gestures for moving, rotating and scaling.

Potentials of neurophysiological measures for research on instructional design: The case of decorative pictures

The project examines by means of (neuro-)physiological measures (electroencephalography, [EEG], eye tracking, pupillometry) the effects of decorative pictures on cognitive (e.g., working memory, attention) and emotional-motivational processes during learning with textual materials. Decorative pictures are commonly used in the design of instructional materials as a kind of seductive details to arouse learners' interest in the topic. However, decorative pictures have resulted in mixed effects on learning outcomes with potential reasons still matter of debate. Increased cognitive load and attentional distraction have been hypothesized as reasons for detrimental effects of decorative pictures on learning.

Processing dynamic visualizations with mirror neurons

This project investigates the idea that the activation of specific areas in the brain is beneficial for learning about continuous processes with dynamic visualizations. These areas in the brain (the so-called human mirror-neuron-system) are used to understand and imitate actions of other persons. We addressed, whether gestures that correspond or do not correspond to the to-be-learned processes, activate the mirror-neuron-system and enhance learning outcomes.

Reception of the scientific and public discourse on nanosafety on the Internet

Nanotechnology is considered one of the key technologies of the 21st century. A general statement on the effect of nanomaterials on people and the environment however is not yet possible. As part of the Leibniz Research Alliance „Nanosafety“, this project focuses on how laypersons and experts deal with conflicting scientific information on this topic and how they make decisions based on this information. We look at this research question with special regard to the internet as a source of knowledge.

SALIENT: Search as Learning – Investigating, Enhancing and Predicting Learning during Multimodal Web Search

The Internet has become indispensable when it comes to searching for information. Such an information search can be understood as a self-regulated learning process: Constructing knowledge while surfing the seemingly endless data stream is a challenging task. The SALIENT project contributes to a better understanding of search as learning and develops methods to support the acquisition of knowledge through the Internet with the help of ranking and retrieval algorithms.

Supporting evaluation processes during web search

The WWW offers easy and fast access to a vast amount of information, which, however, can vary greatly in its quality. Previous empirical studies have shown that both school students and adults don’t often spontaneously pay attention to and critically evaluate where the information stems from. Therefore, this project investigates different approaches how Web users can be supported in their evaluation and selection of information during Web search.

The influence of hand proximity on cognitive and emotional information processing with multi-touch interfaces

The use and application of interactive multi-touch displays increase continuously. Thus, touch-based user interfaces like multi-touch tables, tablets or smartphones are used in many public sector institutions and private households. These interfaces enable users to directly manipulate external representations using their hands without the requirement of an indirect interaction device (e.g., mouse). The question arises how touch-based interactions should be implemented to facilitate the processing of information.

The influence of user- and text characteristics on source evaluation during Web search

The WWW is characterized by an unprecedented amount of information and a high heterogeneity of information quality. For a successful information search, for example, about a conflicting science-related or medical issue, thus, it is often important to evaluate the information sources regarding their trustworthiness and to compare and weigh (potentially conflicting) information from multiple sources. In this project we investigate through eyetracking analyses and log file data, verbal protocols, as well as argumentative summaries how Web users evaluate and process information during Web search about conflicting topics. Specifically, the project aims at examining how certain user characteristics and certain text characteristics influence source evaluations during Web search.

The Interactive Ward Round Table – A Cognitive Interface for the Integration of Multiple Documents in Medical Team Meetings

In the daily ward round physicians use several documents for the optimal treatment of a patient. Medical information systems support this process. However, they are typically not optimized to integrate different information sources into one coherent overall picture. The aim of this project is to develop and implement a new cognitive interface to support physicians in their cognitive processes when working with multiple documents in the daily ward round.

Former Projects

graduation papers