Multimodal Journaling
for Parents of Premature Babies

Motivation

Based on the findings of the large scale user study with parents of premature babies, two aspects of the implementation of lumila, the needs for seamless interaction with the tool and presentation of babies' developmental data, were revisited.

One of the feedbacks from the study was that parents do not want to spend their precious time bedside by staring at and typing into mobile devices. Instead, they want to focus on their baby's care. Interaction with any digital support tool must be seamless, with only the necessary amount of hints of affordance that support this, and does not distract the user, i.e. the parents, from their primary goal of being in an intensive care unit.

Next

The work for lumila continues, the goal is to bring multimodal interface capabilities to the parental engagement platform.
The work is going to be published: Multimodal journaling for parents of premature babies in Neonatal Intensive Care Units (journal submission, April 2018).

Implementation

A gesture & speech modality based multimodal user interface solution was developed and tested with potential users. The collected data was used to train a hybrid (stochastic - heuristic) multimodal integrator that is capable of fusing incoming modality input channels into a single semantic interpretation with some 96% accuracy.

Above: the dashboard of parents with the babies' developmental data, as used in the prototyping and user testing phase, is presented.

Below: the recorded multimodal interactions are visualized, using the same visualization technology as in an earlier project. The users executed a predefined scenario (steps from left to right), either using speech-only modality (top path), gesture-only (bottom path), or multimodal gesture+speech interaction patterns (middle path).