Month: December 2023

The goal of Project Butterfly (PBF) is to create a controlled immersive media environment for the adaptable and translatable therapeutic movement, which incorporates runtime data feedback on player movement performance and behavioral analysis. Its aims are to bridge the gap between therapists and at home users undergoing repetitive exercise and physical therapy through mapping movement by “on-the-fly” motion capture for gamified scenarios such as protecting a virtual butterfly and catching crystals. Pilot work was conducted to explore how Mirror Visual Feedback Therapy could be translated into an immersive virtual reality environment through exploring how users protect a virtual butterfly with head-mounted display systems and wearable soft robotic exosuits.

Here is a video of the first iteration prototype for PBF:

Researchers: Aviv Elor, Michael Powell (DANSER Labs), Evanjelin Mahmoodi, Mircea Teodorescu (DANSER Labs), Sri Kurniawan

Immersive virtual reality gaming has the potential to motivate individuals to perform intensive repetitive task-based therapy, and it can be combined with motion capture as a way to track therapy compliance and progress. This project explores the design and evaluation of an immersive virtual reality experience, titled “Project Star Catcher,” for those with weakness on one side of their upper bodies. Our game mechanics were adapted from constraint-induced movement therapy, an established therapy method where users are asked to use the weaker arm by physically binding the stronger arm. This adaptation innovates from physical to psychological binding by providing a dynamic reward system that promotes the use of the weaker arm. Players are rewarded by scoring points when performing a rehabilitative motion to catch falling stars in an immersive, cosmic virtual reality. Initial results indicate that immersive games like PSC provide a powerful medium for physical exercise with an increase of over 40% exercise compliance for adults of mixed ability. We are performing futher studies for a systematic comparision of VR devices with PSC, as well as applying affective computing techniques to understand user emotional response. This modular system enacts a behavioral playground that is flexible from studying VR for Therapy, physical task-based analysis, and runtime adaptive stimuli.

Here is a video overview of the pilot PSC goals:

Researchers: Aviv Elor, Evanjelin Mahmoodi, Nico Hawthorne, Michael Powell (DANSER Labs), Mircea Teodorescu (DANSER Labs), Sri Kurniawan

Traditional cognitive testing for detecting cognitive impairment can be inaccessible, expensive, and time consuming. This project aims to develop an automated online computerized neuropsychological testing (CNT) system for rapidly tracking an individual’s cognitive performance throughout the user’s daily or weekly schedule in an unobtrusive way. By utilizing embedded microsensors within tablet devices, the proposed context-aware system will capture ambient and behavioral data pertinent to the real-world contexts and times of testing to compliment psychometric results, by providing insight into the contextual factors relevant to the user’s testing efficacy and performance.

Our primary objectives for the project will be to:

• Develop an accessible, dynamic, online CNT system capable of capturing contextual data during testing and tracking temporal variations in the user’s test performance.
• Develop an information fusion system capable of capturing and analyzing ambient data from different sources during testing.

Researchers: Sean Smith, Breanna Baltaxe, Alex Cabral, Brookelyn Mcjunkin, Bronte Brillantes, Joshua Lopez, Trevor Parker

This project, funded through an NSF CAREER award and the NSF GRFP, aims to facilitate independence for blind authors in producing documents that meet the presentation ‘standards’ expected by sighted readers. This includes gathering design guidelines for tools that aim to help blind people format their docuemnts independetly by: 1) developmenting an impact-weighted taxonomy of common document presentation errors produced by blind authors, 2) exploring blind persons’ mental models and strategies for learning and coping with docuemnt formatting, and how these models and strategies contribute to the success of independent document formatting and layout activities; and 3) by iterating over the previous the development and evaluation of prototype tools.

Researchers: Lourdes Morales-Villaverde, Sri Kurniawan, Sonia Arteaga
Collaborators: Priya Bhattacharjee, Tiffany Thang, Peter Cottrell, Dustin Adams

We are developing a video game to aid young children undergoing speech therapy after cleft palate surgery. Children undergoing speech therapy must first unlearn their compensations and ommissions that they use to get around certain syllables they cannot do. Unlearning involves significant amounts of practice at home, and parents have extreme difficulty in motivating children to practice. Working directly with children who are undergoing speech therapy, we are designing a simple game using a novel speech recognition engine to help motivate them to perform their therapy exercises. We aim to accelerate the rate of recovery and give therapists more tools to use with children. We also intend to forward research in educational games.

Here is a video overview of our prototype game “Speech Adventure”:

Researchers: Zachary Rubin, Sri Kurniawan. In collaboration with UC Davis Medical Center.

Photography is a visual way to capture a moment in time. Photos can be used for artistic expression, and to remember significant events. Because photo taking, organizing and sharing traditionally requires visual information, those with no or limited sight often have problems with these activities. Previous work has made photo capturing without sight easier, however, there is little work that makes photo browsing and sharing blind-accessible. This project aims at facilitating independence for blind persons to take, organize and share photos through user-centered development of a smartphone application that can be used without sight. The work starts with an investigation of current practices of blind persons in these activities, continued with a review of existing applications, and finally the design and long-term evaluation of the application.

The overarching needs that this smartphone application aims to meet are the following:

• Photo Taking: Aiming, focusing, positioning, and framing; Easy way to get sighted help; Accessible device; Improving photo quality.
• Photo Organizing and Editing: Identifying what’s in the picture; Labeling the pictures; Manipulating pictures
• Photo Sharing: Easy way to get sighted help; Accessible photo sharing method.

Here is a video of the proposed system, Phodio, a blind accessible iPhone app designed to help blind people take and organize their photos for easier retrieval and sharing:

Researchers: DustinAdams, Sri Kurniawan. In collaboration with Harada, Sato, Asakawa, Takagi, IBM Research – Tokyo.

We developed a new method of providing activities that help individuals with developmental disabilities (DD) of all ages learn or reinforce basic skills such as adding money, identifying US currency, numbers, or letters. The goal is to make the process of learning and reviewing those skills more enjoyable and manageable for people with DD and their caregivers or guardians.

Our methods include working in collaboration with Imagine! and Hope Services, two not-for-profit organizations that provides support services to people with DD, to gather system requirements and developed and evaluate a prototype in the form of an online application which works primarily as a web-app on the iPad and includes activities to teach individuals with DD of all ages about numbers, letters, colors, and currency. To check out the prototype please visit: eLearning Basic Skills.

Here is a video of a user testing the activity on recognizing money in our system:

Video produced by Imagine!

Researchers: Lourdes Morales-Villaverde, Taylor Gotfrid, Kariina Caro, Luke Buschmann, Sri Kurniawan. In collaboration with Imagine! and Hope Services.