The Multimodal Interaction lab conducts research into the interaction of digital information on the basis of different sensory and motoric modalities. The focus is on the multimodal handling of multiple information resources and on the use of sensor-based interaction modalities, such as multi touch or brain-computer interfaces.
Digital media expand the range of presentation formats for information offerings, and they often also enable users to interact with these presentation formats in multimodal environments in a self-regulated manner. On the one hand, the term multimodal refers to the contribution of different (1) sensory modalities (e.g., visual, auditory, haptic), (2) presentation formats (e.g., text, film, simulation), and (3) output interfaces (e.g., notebooks with keyboard and mouse, multi-touch tables, digital glasses for virtual and augmented reality, VR/AR) in the presentation of information.
On the other hand, it also refers to the diverse interaction devices for acting in multimodal environments that are enabled by different digital sensors (e.g., touch, motion, or physiological sensors).
The role of different modes of interaction for knowledge-related usage and processing processes is the focus of interest of the Multimodal Interaction lab. Forms of interaction investigated in the research group range from mouse, keyboard, and touch interactions to the use of control gestures and physical objects when dealing with virtual realities (VR) and interactive displays (embodied or tangible interaction) to physiology-based interaction possibilities (e.g., gaze control or brain-computer interfaces).
The relations between multimodal interaction opportunities and the cognitive or emotional processing of information are theoretically investigated mainly from the point of view of self-regulation (e.g., attentional control and working memory load), information retrieval (e.g., information search and comprehension of multiple documents), and the mental significance of physical experiences (e.g., space-valence associations and hand-proximity during information processing).
Deputy Head of Lab
+49 7071 979-222b.brucker@iwm-tuebingen.deTeam assistance
+49 7071 979-338p.ebert@iwm-tuebingen.deNicolas Bauer
Associated scientist
+49 7071 979-0n.bauer@iwm-tuebingen.deAssociated scientist
m.beukman@iwm-tuebingen.deAssociated scientist
y.kammerer@iwm-tuebingen.deJessica Kienzle
Associated scientist
+49 7071 979-0j.kienzle@iwm-tuebingen.deAssociated scientist
m.lachmair@iwm-tuebingen.deAssociated scientist
+49 7071 979-360c.scharinger@iwm-tuebingen.deMultimodal Interaction
Duration 01/2018 - open
Digitalization is permeating numerous areas of life and constantly offering new opportunities - in private life, at school, and in organizations. Apps are being used to combat the pandemic, for example; robots, artificial intelligence, and other digital assistants provide support in the classroom and at work. However, users are not always enthusiastic about new technologies and the changes these technologies bring with them from the very first minute, even if users can objectively benefit from these changes.
Go to projectMultimodal Interaction
Duration 12/2016 - 11/2019
What could self-portraits in a museum teach us? Do self-portraits mediate more than knowledge about drawing style and epochal features? In the project DigiSelbst, we will focus on visual arts (especially self-portraits) in Art museum and the use of digital media to design cultural learning opportunities.
Go to projectMultimodal Interaction
Duration 01/2017 - open
The project aims at bridging the gap between neuroscience and instructional psychology via informatics by studying the use of brain-computer interface (BCI) methodologies for research in instructional psychology (e.g., the individual classification of working memory load in real-time). As long-term goal stands the development of an online adaptive learning environment based on physiological measures of basic cognitive processes. Specifically, the online adaption grounds on the detection and classification of specific neural signatures in the electroencephalogram (EEG) of learners by means of advanced BCI methodologies.
Go to projectMultimodal Interaction
Duration 01/2008 - 12/2023
The WWW is characterized by an unprecedented amount of information and a high heterogeneity of information quality. For a successful information search, for example, about a conflicting science-related or medical issue, thus, it is often important to evaluate the information sources regarding their trustworthiness and to compare and weigh (potentially conflicting) information from multiple sources. In this project we investigate through eyetracking analyses and log file data, verbal protocols, as well as argumentative summaries how Web users evaluate and process information during Web search about conflicting topics. Specifically, the project aims at examining how certain user characteristics and certain text characteristics influence source evaluations during Web search.
Go to projectMultimodal Interaction
Duration 01/2008 - 12/2023
The WWW offers easy and fast access to a vast amount of information, which, however, can vary greatly in its quality. Previous empirical studies have shown that both school students and adults don’t often spontaneously pay attention to and critically evaluate where the information stems from. Therefore, this project investigates different approaches how Web users can be supported in their evaluation and selection of information during Web search.
Go to projectMultimodal Interaction
Duration 04/2010 - open
This interdisciplinary project aims at the conception, implementation and optimization of an innovative multimedia guide for museum and exhibitions (EyeVisit). The main concept is to present digital information by utilizing interactive displays. According to interaction with real-world objects, EyeVisit allows the manipulation of digital visual information with intuitive gestures for moving, rotating and scaling.
Go to projectMultimodal Interaction
Duration 07/2017 - 12/2020
In the daily ward round physicians use several documents for the optimal treatment of a patient. Medical information systems support this process. However, they are typically not optimized to integrate different information sources into one coherent overall picture. The aim of this project is to develop and implement a new cognitive interface to support physicians in their cognitive processes when working with multiple documents in the daily ward round.
Go to projectMultimodal Interaction
Duration 04/2023 - 09/2025
Students with very different learning abilities take part in STEM lessons. Adaptive teaching, which considers individual prerequisites and offers personalised learning opportunities, is a promising didactic concept for dealing with heterogeneity. In the MINT-ProNeD project, research-based training and counselling concepts are being developed to support STEM teachers in using digital technologies for adaptive teaching and in preparing for future technologies.
Go to projectMultimodal Interaction
Duration 01/2013 - open
The use and application of interactive multi-touch displays increase continuously. Therefore, touch-based user interfaces like multi-touch tables, tablets or smartphones can be found in many public facilities and private households today. These user interfaces allow direct manipulation of external representations with the hands without the need for additional and indirect interaction devices (e.g., mouse). Thus, the question arises as to how manual interactions should be designed to support information processing.
Go to projectMultimodal Interaction
Duration 11/2016 - 12/2020
Nanotechnology is considered one of the key technologies of the 21st century. A general statement on the effect of nanomaterials on people and the environment however is not yet possible. As part of the Leibniz Research Alliance „Nanosafety“, this project focuses on how laypersons and experts deal with conflicting scientific information on this topic and how they make decisions based on this information. We look at this research question with special regard to the internet as a source of knowledge.
Go to projectMultimodal Interaction
Duration 01/2018 - 12/2023
The technical development in the field of Virtual Reality currently shows a strong dynamic development. The high degree of immersion of this technology makes it possible to create for the user an intense feeling of presence in the virtual world. It is assumed that cognitive processes create a mental model of the self in the virtual world, which generates a feeling of "being in the virtual world". However, it is unclear from a cognitive science and learning psychology perspective in what ways this form of immersion can benefit the user in knowledge contexts.
Go to projectMultimodal Interaction
Duration 08/2008 - open
The communication and exchange of information often takes place across certain hierarchies - be it in organizations between managers and employees, or in schools between teachers and school management or students. Power structures such as these bring with them freedom, but also a great deal of responsibility for the more powerful people. It can be particularly difficult to live up to this responsibility when working together over long distances (e.g. in virtual contexts).
Go to projectMultimodal Interaction
Duration 09/2018 - 08/2021
The project examines by means of (neuro-)physiological measures (electroencephalography, [EEG], eye tracking, pupillometry) the effects of decorative pictures on cognitive (e.g., working memory, attention) and emotional-motivational processes during learning with textual materials. Decorative pictures are commonly used in the design of instructional materials as a kind of seductive details to arouse learners' interest in the topic. However, decorative pictures have resulted in mixed effects on learning outcomes with potential reasons still matter of debate. Increased cognitive load and attentional distraction have been hypothesized as reasons for detrimental effects of decorative pictures on learning.
Go to projectMultimodal Interaction
Duration 09/2021 - 09/2024
Under National Socialism, discrimination and murder were implemented with frightening efficiency with the help of bureaucratic routines. The crimes of the Nazi era could be presented as necessary and harmless processes with the use of administrative language and euphemistic terms in forms and correspondences. But how can this phenomenon be presented to visitors in an interesting, intuitive and interactive way using exemplary Nazi documents? Can visitors learn to read and assess Nazi documents with the right medium?
Go to projectMultimodal Interaction
Duration 10/2020 - 09/2024
This research project, which is part of the research network "Human-Agent-Interaction", investigates how laypersons (and literary experts) perceive and evaluate an artificial intelligence (AI) based writing tool that produces creative output such as short narrative texts and poems. How credible, creative, and easy to read are the outputs of such a GPT-3-based writing tool?
Go to projectMultimodal Interaction
Duration 04/2012 - open
This project investigates the idea that the activation of specific areas in the brain is beneficial for learning about continuous processes with dynamic visualizations. These areas in the brain (the so-called human mirror-neuron-system) are used to understand and imitate actions of other persons. We addressed, whether gestures that correspond or do not correspond to the to-be-learned processes, activate the mirror-neuron-system and enhance learning outcomes.
Go to projectMultimodal Interaction
Duration 03/2022 - 02/2025
Interacting with technologies and particularly, multitouch devices such as smartphones, tablets, or large-scale monitors (e.g., in museums) has become standard in daily life. To date, however, it is very unclear how manipulating digital objects physically through these devices, influence affective processing (e.g., when encountering potentially pleasant or unpleasant stimuli). • Against this backdrop, the project aims at advancing the understanding of affective processing through the manual interaction with digital objects. Theoretically, it seeks the integration of three pivotal topics discussed in cognition and emotion fields: (a) spatial affective biases linked to space-valence associations (up/right - positive; down/left - negative), (b) situated action-goal representations linked to approach-avoidance mechanisms, and (c) visuo-spatial effects linked to hand-proximity to visual stimuli. Methodologically, the project builds upon own prior exploratory research on manual interactions with digital affective pictures to develop novel experimental paradigms, which will be used to deepen three main research questions: (i) May interaction gestures with affective digital pictures unravel space-valence effects observed in standard experimental research; (ii) Which contextual aspects shape affective action-goal representations during manual interactions? (iii) What is the specific affective contribution of interacting with digital objects directly by hand? The findings will contribute to the field of human-computer interaction and particularly, how multimodal aspects shape together emotional processing during interactions with digital objects.
Go to projectMultimodal Interaction
Duration 07/2023 - 02/2026
Digital technologies offer opportunities to rethink and redesign art, music, and sports lessons (KuMuS). Digital technologies are a promising medium especially for the targeted promotion of individual learning requirements of pupils. To integrate these technologies innovatively and successfully into everyday life in schools, the "KuMuS-ProNeD" project aims to develop an integrative overall concept for teacher education in art, music and sport and thus complements the "MINT-ProNeD".
Go to projectMultimodal Interaction
Duration 01/2024 - 12/2026
Warning: project with empty content field
Go to projectMultimodal Interaction
Duration 01/2023 - 12/2025
Video-SRS is an interdisciplinary project that focuses on supporting video learning in mathematics by exploring and improving self-regulation. It combines insights from cognitive and educational psychology, mathematics education, and computer science to develop innovative approaches to this type of learning. The use of responsible machine learning algorithms plays a significant role in this process. Specifically, the collaborative project aims to identify and address self-regulation problems in video learning by automatically recognizing when such problems occur and providing appropriate and automated assistance. Similarly, the project aims to recognize suboptimal characteristics of instructional videos that could assist learners in selection and creators in the production of better videos. On the educational side, the project focuses on the study of learning for derivation, as there is a great need for learning in this area among German students. Methodologically, the project not only utilizes machine learning techniques, but also analyzes video materials, log file data from video platforms, and multimodal sensor data from individual video learners, such as eye movement data. Where possible, these analyses are triangulated to make the best possible statements about self-regulation problems and their solutions. The goal of the project is both to deepen theoretical insights into the role of self-regulation in video learning and to obtain practical approaches for optimizing learning.
Go to projectMultimodal Interaction
Duration 01/2022 - 12/2025
Warning: project with empty content field
Go to projectMultimodal Interaction
Duration 02/2023 - 02/2026
This dissertation project investigates the use of immersive virtual reality (IVR) as a medium of experience-based learning in museums. Two IVR-applications will be developed and integrated into several exhibitions, in cooperation with the Natural History Museum Stuttgart. The goal is to give visitors an understanding of the importance of biodiversity and its protection. Furthermore, the project will provide new insights into the use of IVR as tools of science communication in museums. <p><br></p> Although biodiversity conservation is already established as a socially relevant topic, there are still many species that are overlooked or unappreciated. A recent temporary exhibition of the Natural History Museum introduced visitors to the biodiversity of wasps in Germany and highlighted these inconspicuous insects with the help of the two IVR applications of the IWM.<p><br></p> In the “virtual wasp”, visitors can meet two newly discovered wasp species eye-to-eye, even though they only measure a few millimeters in real life, while the “virtual collection” showcases the size of the museum’s scientific collection. Both applications mainly aim to provide visitors with unique experiences above teaching hard facts and are meant to facilitate a new perspective on insects to motivate visitors to engage in biodiversity protection. <p><br></p>Museums use different ways to communicate their contents. Besides more established types of media (e.g., audio, video), in recent years, virtual reality has found its way into more and more exhibitions. Therefore, this project also aims to investigate what role VR applications can play in museums and how they are best applied to have a lasting effect on visitors.
Go to projectMultimodal Interaction
Duration 01/2024 - open
In the context of this project, a collaborative multi-touch table game is being developed for schools and museum education. Learners navigate through virtual spaces by using GPT-generated sentences or original sentences by an author, e.g. Kafka, as a "literary fingerprint" to progress and thereby learn something about an author's writing style. A dissertation is investigating the means by which such a game can generate curiosity and whether this has a beneficial effect on learning. The first step is to investigate the extent to which players acquire both explicit (declarative) and implicit knowledge about an author's literary style through the game. Furthermore, it will be analyzed whether and to what extent the game arouses curiosity – specifically, whether an authentic interest in the learning content is aroused or whether it merely serves as a means to an end – winning the game. Subsequent studies will examine which specific features or game elements – for example, a narrative or the option of group play as opposed to solo play – generate or intensify curiosity and what effects this has on the learning process. The educational content of the game will be developed in cooperation with the Center for Didactic Computer Game Research at the PH Freiburg and the German Literature Archive Marbach.
Go to projectMultimodal Interaction
Duration 06/2016 - open
The project „Visitors’ view on Obersalzberg“ focuses on two research questions: How can propaganda pictures get deconstructed? And how does the awareness of being in a historic place related to Nazi-history influence the perception, the processing and the judgements about associated pictures? Cooperation partner is the Institute for Contemporary History (Dokumentation Obersalzberg). Empiric findings are meant to flow into the realm of practice, the museum. Places of remembrance linked to the history of National Socialism (NS, Nazi) are gaining importance during the last two decades. Obersalzberg is a place with references to the Nazi-regime and was used as a setting for Nazi-propaganda. Today an educational center about history and politics is located there. The project is divided into two thematic blocks: on the one hand the deconstruction of propaganda, on the other the impact of historical places. A first experimental study already showed that prior information about the crimes that happened during the time of NS as well as the awareness of being at a place linked to that history had a negative influence on the personal mood. The negative personal mood in a next step influences the valence ratings of associated pictures. Within the project, both field studies and lab studies are conducted. Methodologically, a broad spectrum of objective (computer-based speech analyses, eye-tracking) and subjective measurements (questionnaires, rating scales) is used. Methods for the deconstruction of propaganda images are also investigated in virtual realities and with multi-touch surfaces. The empiric findings are meant to build a basement for further psychological research as well as to get transferred to the conception of the new permanent exhibition of the Dokumentation Obersalzberg.
Go to projectMultimodal Interaction
Duration 07/2017 - open
Students’ focused attention is a central prerequisite for classroom learning. Research shows that teachers differ substantially in how successful they are in eliciting and guiding their students’ attention, which is often referred to as “classroom management” and “cognitive activation”. Furthermore, they may also not always be aware of their students’ attentional focus, and this may be particularly true for beginning teachers (teachers who are in the induction phase or shortly after this stage).
Go to projectMultimodal Interaction
Duration 08/2024 - 04/2026
Our interdisciplinary longitudinal study investigates the evolving dynamics of human-AI interaction over six waves spanning one year. By examining individual, behavioral, and task-related variables, the project aims to uncover how users' trust in, perceptions of, self-efficacy, and willingness to engage with AI systems develop and interrelate over time. The insights gained from this research are essential for better understanding human-machine interaction, a critical foundation for fostering effective collaboration between users and AI systems. This knowledge will inform user-centered AI design and guide the ethical integration of these technologies into various aspects of everyday life.
Go to projectMultimodal Interaction
Duration 01/2011 - open
The online journal wissensdialoge.de aims at transferring the results of psychological research to the field. Its goal is to maintain a continuous dialogue with partners from relevant domains of application. The online journal addresses people who are interested in the topic, experts as well as practitioners from the fields of learning / leading / cooperating / changing / living / working.
Go to projectArticles (peer-reviewed) | Books and book chapters | Proceedings and special issues | Position papers and statements | Research data | Software | Other publications
Open AccessPreregistration 1 | 2 | 3 | 4 DataStudy materialCode
Open AccessPreregistrationDataCode
Open AccessPreregistration 1 | 2 | 3 | 4 | 5 DataCode