• Press Information
  • Contact
  • deutsch | english
Multimodal interaction

Multimodal Interaction Lab

Hypermedia-based learning environments are characterized by a non-linear connection of information nodes containing multimedia materials. These hyperlinked environments can be explored by users in an interactive and self-directed way.

The lab investigates three types of hypermedia environments: First, the World Wide Web (WWW) as a global network of information nodes of diverse origin and quality that can be explored by means of web browsers and search engines. Here, our research mainly addresses processes of evaluating and integrating different sources of information during web search. Second, instructional hypermedia environments that are designed to support learners of different needs. In particular, we are interested in how innovative technologies can be used to adapt these environments to learners’ prerequisites and to their learning progress. Third, we analyze the potentials of novel hypermedia environments that are based on interactive displays (e.g., multitouch tables, smartphones, or tablets). Here, we focus on the direct and intuitive manipulation of information by means of multitouch interaction and on the use of research paradigms from the field of embodied cognition for studying processes of knowledge acquisition.

With regard to all abovementioned types of hypermedia environments, we focus on two pivotal issues: How to support processes of comprehension by means of hypermedia and how to avoid potential cognitive overload due to the large amounts of information, presentation modes, and interaction requirements provided to users. We assume that a deeper understanding of hyperlinked information can be best supported when hypermedia design features are matched to users' particular prerequisites. Accordingly, the lab comprises two lines of research, addressing on the one hand the design and affordances of hyperlinked information environments, and on the other hand user prerequisites and cognitive resources.

Design and affordances of hyperlinked information environments

This line of research investigates how design features of hypermedia environments influence the resulting affordances with regard to user behavior and cognitive processing. For instance, we study which features of web resources and search interfaces stimulate users to elaborate on the origin and quality of web information. Moreover, we analyze which affordances for information processing are provided when embedding static and dynamic visualizations within hypermedia environments. A third core area is related to informal learning contexts like museums and addresses the design of intuitive hypermedia environments based on interactive displays. Here, we focus on the issue of how a direct haptic interaction with representations by means of gestures (“embodied interaction”) can not only improve an intuitive usability but also facilitate the acquisition of conceptual structures.

User prerequisites and cognitive resources

This line of research investigates important cognitive prerequisites for successful hypermedia exploration. One focus of the lab is on optimizing the management of limited cognitive resources like working memory capacity. For instance, we study how neural signatures of working-memory processes can be used to measure different aspects of working-memory load during learning in real time. The long-term goal of this work is to develop innovative and adaptive hypermedia environments, which allow reducing the amount of cognitive overload, particularly for users with unfavorable learning prerequisites. A second – and complementary – focus is on developing hypermedia environments for learners with exceptionally favorable learning prerequisites. For instance, we develop and optimize hypermedia-based iPad environments that address the specific learner prerequisites of gifted primary school children to stimulate discovery learning, e.g., in domains such as understanding biodiversity or musical structures.

Team Multimodal interaction


Development of hypermedia discovery learning environments for children with exceptional skills

Some types of instructions are more or less effective in enhancing learning outcomes for different groups of individuals. This depends, for example, on learners’ specific cognitive abilities. Optimal learning occurs when the type of instruction is exactly matched to a particular learner’s aptitude. This project is concerned with the development of adaptive forms of instructions for learners with high cognitive prerequisites. In particular, it focuses on gifted and high-performing children and investigates whether their learning success can be enhanced by specific types of instructions. It is assumed that gifted and high-performing children differ from averagely gifted children in the way they can take advantage of complex and open learning environments. This assumption is based on their higher working memory capacity which should enable them to cope better with complex instructional designs. On the one hand, such instructions can lead to a deeper elaboration of the learning material, while on the other hand they impose a high cognitive load on the learner. The project is conducted in cooperation with the Center for Educational Science and Psychology (Prof. Trautwein) of the University of Tuebingen.

EEG-based neural signatures of different types of working-memory load

The project is part of Cluster01 in the ScienceCampus Tuebingen "Informational Environments". The cluster aims at bridging the gap between neuroscience and instructional psychology via informatics by developing an online adaptive learning environment based on physiological load-measures. Specifically, the online adaption grounds on specific types of working memory load via neural signatures in the electroencephalogram (EEG) of learners by means of advanced braincomputer interface (passive BCI) methodologies.

Intuitive und collaborative interaction with digital media on tabletop-displays

Multi-Touch Tabletop System This interdisciplinary project aims at the conception, implementation and optimization of an innovative multimedia guide for museum and exhibitions (EyeVisit). The main concept is to present digital information by utilizing interactive displays. According to interaction with real-world objects, EyeVisit allows the manipulation of digital visual information with intuitive gestures for moving, rotating and scaling. Large optical multi-touch tabletop systems with a display diagonal of approximately 180 cm can be used to explore additional information of an exhibit. Mobile handhelds which are tracked via bluetooth are utilized to provide personalized multimedia material during the visit of an exhibition and can be connected to the tabletop system.

Processing dynamic visualizations with mirror neurons

This project investigates the idea that the activation of specific areas in the brain is beneficial for learning about continuous processes with dynamic visualizations. These areas in the brain (i.e., the so-called human mirror-neuron-system [hMNS]) are used to understand and imitate the actions of other persons. They are typically activated by human movements, but may be more generally used to also represent other biological or even non-biological movements (if these movements can be anthropomorphized by the observer). In the domain of learning about continuous processes, one effective instructional strategy to activate the hMNS might be to show learners not only the to-be-learned processes, but also human gestures displaying the to-be-learned dynamics in order to trigger an anthropomorphized encoding. It has already been shown that the production of such gestures during learning is beneficial for acquiring knowledge. Whether showing gesture-based interventions are really beneficial to acquire knowledge about continuous processes is investigated in the example domain of biological movements in a series of studies using neurophysiological methods (i.e., functional near-infrared-spectroscopy [fNIRS]).

Supporting evaluation processes during Web search

The World Wide Web (WWW) today is a major information resource – not only to answer simple facts, but also to search for more complex and conflicting science-related topics. The Web offers easy and fast access to a vast amount of information, which, however, can vary greatly in its quality. A successful information research – especially to answer complex and conflicting science-related ore medical issues – thus requires adequate evaluation strategies such as evaluating the information sources regarding their trustworthiness and comparing information from multiple sources. Previous empirical studies have shown that both school students and adults rather seldom engage in trustworthiness evaluations spontaneously. Therefore, this project investigates different approaches, how Web users can be supported in their evaluation and selection of information during Web search.

The influence of hand proximity on cognitive and emotional information processing with multi-touch interfaces

The use and application of interactive multi-touch displays increase continuously. Thus, touch-based user interfaces like multi-touch tables, tablets or smartphones are used in many public sector institutions and private households. These interfaces enable users to directly manipulate external representations using their hands without the requirement of an indirect interaction device (e.g., mouse). The question arises how touch-based interactions should be implemented to facilitate the processing of information.

The influence of user- and text characteristics on source evaluation during Web search

The WWW is characterized by an unprecedented amount of information and a high heterogeneity of information quality. For a successful information search, for example, about a conflicting science-related or medical issue, thus, it is often important to evaluate the information sources regarding their trustworthiness and to compare and weigh (potentially conflicting) information from multiple sources. In this project we investigate through eyetracking analyses and log file data, verbal protocols, as well as argumentative summaries how Web users evaluate and process information during Web search about conflicting topics. Specifically, the project aims at examining how certain user characteristics and certain text characteristics influence source evaluations during Web search.

Former Projects

graduation papers

Der Einfluss von Hyperlinks auf das Textverständnis beim Lesen von Wikipedia-Artikeln

Online-Artikel (z.B. in Wikipedia) enthalten oftmals eine beträchtliche Anzahl an Hyperlinks zu weiterführenden Informationen. Wie beeinflussen das Vorhandensein solcher Links und deren Selektion das Textverständnis beim Lesen von Wikipedia(-artigen) Artikeln? Sind die Links ablenkend? Oder führt die Auswahl relevanter Links zu einem tieferen Textverständnis?

Methode: Experimentalstudien mit Blickbewegungsmessung

Ihre Ansprechpartnerin ist: Frau Dr. Yvonne Kammerer

Wie kann das Spiegelneuronensystem aktiviert werden und ist diese Aktivierung hilfreich für das Lernen über nicht-menschliche Bewegungen?

Dynamische Visualisierungen können besonders hilfreich sein, wenn Lernende sich die dargestellten Bewegungen mit Hilfe des eigenen Körpers vorstellen. Dies kann erreicht werden indem sogenannte Spiegelneurone durch menschliche Gesten aktiviert werden. Dabei stellt sich die Frage, ob es reicht, wenn Lernende die menschlichen Gesten gezeigt bekommen oder ob sie diese selbst ausführen müssen. Zusätzlich können diese Befunde in verschiedenen Lerndomänen und –inhalten überprüft werden.

Methode: Experimentalstudie mit Messung der Gehirnaktivierung (fNIRS)

Ihre Ansprechpartnerin ist: Frau Dr. Birgit Brucker

Hypermedia-Lernen von Grundschülern auf dem iPad − Unterstützungsmaßnahmen und kognitive Fähigkeiten

Digitale Lerntechnologien wie z.B. iPads werden im schulischen Kontext immer häufiger eingesetzt. Hierbei stellt sich jedoch die Frage, inwieweit besonders jüngere Schulkinder von solchen Lerntechnologien und den damit verbundenen hypermedialen Lernumgebungen profitieren können. Welche Unterstützungsmaßnahmen sind für Kinder beim Umgang mit einer hypermedialen iPad-Lernumgebung hilfreich? Welchen Einfluss haben kognitive Fähigkeiten auf den Umgang mit einer iPad-Lernumgebung?

Methode: Feldstudie in der Schule

Ihr Ansprechpartner ist: Frau Jessica Kornmann

Interaktive Displays und embodied cognition

Interaktive Displays verwenden eine intuitive und niederschwellige Gestensteuerung zur Interaktion mit Informationseinheiten sowie zur Aktivierung von Informationsverknüpfungen (z.B. iPhones oder Multi-Touch Tabletop-Systemen). Wie wird die kognitive Repr채sentation einer konzeptuellen oder räumlichen Struktur durch die Darbietung auf einem interaktiven Display beeinflusst, mit dem der Nutzer direkt physikalisch durch Gesten oder Tangible Objects interagieren kann?

Methode: Experimentalstudie, die auf grundlagen- oder anwendungsbezogenem Material beruhen kann

Ihr Ansprechpartner ist: Herr Prof. Dr. Peter Gerjets

Interaktive Displays zur Informationserschließung

Wie können die bei interaktiven Displays gegebenen Affordanzen genutzt werden, um die Erschließung umfangreicher Informationsbestände und ihrer Verknüpfungen mit externen Objekten und externen Informationen zu optimieren. Exemplarisch soll dies im Rahmen der Entwicklung eines personalisierten Besucherinformationssystems in Kooperation mit dem Anton Ulrich-Museum in Braunschweig untersucht werden.

Methode: Befragung und Beobachtung von Museumsbesuchern, Feldstudie

Ihr Ansprechpartner ist: Herr Prof. Dr. Peter Gerjets

Können dynamische Visualisierungen, die gleichzeitig dargeboten werden, das Lernen unterstützen?

Auf größeren interaktiven Oberflächen (wie z.B. Multitouch-Tischen) können leicht mehrere dynamische Visualisierungen gleichzeitig abgespielt werden. Können Lernende mit diesem Format (multiple Animationen) Zusammenhänge leichter erkennen oder lenken multiple Animationen die Lernenden eher ab, da die Aufmerksamkeit auf unterschiedliche Visualisierungen verteilt werden müssen?

Methode: Experimentalstudie am Multitouch-Tisch

Ihre Ansprechpartnerin ist: Frau Dr. Birgit Brucker

Entdeckung von Inkonsistenzen und Vertrauenswürdigkeitsbewertungen bei der Rezeption von Informationen im Internet

Informationen im Internet können widersprüchliche oder fehlerhafte Informationen enthalten. Bemerken Internetnutzer Inkonsistenzen innerhalb einer Webseite? Und welche Rolle spielt dabei die Art der Webseite (z.B. Forum oder Uni-Seite)? Bemerken Internetnutzer Inkonsistenzen zwischen mehreren Webseiten? Und welchen Einfluss hat dies auf die Bewertung der Vertrauenswürdigkeit der sich widersprechenden Webseiten?

Methode: Experimentalstudien mit Blickbewegungsmessung

Ihre Ansprechpartnerin ist: Frau Dr. Yvonne Kammerer

Hilft direktes Interagieren beim Lernen?

Interaktive Oberflächen (wie z.B. Smartphones, Tablets, Multitouch-Tische) erlauben dem Lerner eine direkte intuitive Interaktion mit den dargestellten Objekten und Inhalten. Ist diese direkte haptische Interaktion (embodied interaction) hilfreich für das Lernen? Mögliche Themen sind Vergleiche mit Standard-GUIs; Variationen der räumlichen und zeitlichen Aspekte der Interaktion oder der Ausrichtung der Präsentationsfläche (Tisch vs. Wand).

Methode: Experimentalstudie mit interaktiven Geräten; Usabilitystudien

Ihre Ansprechpartnerin ist: Frau Dr. Birgit Brucker