Effective Gesture-Based User Interfaces on Mobile Mixed Reality

To help people who are blind or have low vision (BLV) travel independently, virtual reality (VR) and mixed reality (MR) are used to train the BLV users to acquire spatial knowledge for trip planning. The VR apps utilizes auditory and haptic feedback to help BLV people interact with virtual environments to build mental maps. In this project, we improve a MR cane iPhone app by designing and integrating a laser pointer tool and a gesture-based menu to assist BLV users in learning spatial layout of a virtual environment more efficiently. A laser pointer allows BLV users to quickly explore a virtual environment. A gesture-based menu helps BLV users easily manipulate the user interface of the MR cane app.