A Computational Framework for Bionic Eyes

The Project

Retinal prostheses provide a promising solution to restore vision to the 200 million people who suffer from retinal degenerative diseases worldwide. However, current FDA-approved implants are limited to low-resolution grayscale images, making object identification and localization difficult for patients. Furthermore, patients will face a steep learning curve to adapt to the prostheses. To address these challenges, this study develops a novel methodology framework that consists of three integral components: 1) an optimal transportation theory (OT)-based virtual magnifier to localize and enlarge regions of interest (ROIs) while preserving important features and curvatures; 2) a real-time image optimization framework to encode the maximum amount of spatial and color information to patients through attention mechanisms as well as color scheme comparisons; and 3) an autoencoder-OT model to augment the optimized images. Computational experiments through distortion maps showed that the magnifier enlarged the ROIs with minimal area and angle distortion. Further, users were able to select important features and optimize ROI densities through a “digital knob” user interface. In contrast to current schemes, the image optimization framework demonstrated better visual quality, was computationally efficient (<380 ms on tested cases), and allowed for optimal color mapping through comparison studies. A prototype processing system confirmed the effectiveness of the proposed optimization framework over current prostheses. Finally, the AE-OT model augmented images from 6 datasets to generate an image library for patient training. This research offers an accurate, scalable, and optimized architecture that will enable the next generation epiretinal prosthesis.

View this project on YouTube

Team Comments

As a piano player, I’ve learned to appreciate the five senses, specifically hearing and touch. I have always wondered how devastating it could be if a human loses even one of these, and therefore restoring vision was a natural path that I wanted to pursue.

When I first began this project, I ran into a roadblock with my saliency maps: failure in identifying main objects when multiple objects were present. To avoid this issue, the idea of a magnification tool to reduce background information came to mind, hence the creation of the virtual magnifier.

Current image processing algorithms suffer from one critical issue: varying environmental conditions. My goal is to add thermal processing by implementing a 3D sensor to my prosthesis frame, therefore aiding patients in identifying heat-emitting objects in the dark.