Results of FTS project

Results of FTS project

HANDS-FREE CROWD-ASSISTED INDOOR NAVIGATION SYSTEM AND METHOD FOR GUIDING BLIND AND VISUALLY IMPAIRED PERSONS

BACKGROUND OF THE INVENTION 

TECHNICAL FIELD

This invention relates to a hands-free wearable electronic traveling aid (ETA) system for blind and visually impaired (BVI) people's real-time indoor, guided navigation. Specifically, the invention discloses a system comprising multi-sensory inputs and machine learning processes together with crowd-assisted interfaces for navigation routing and for solving problematic situations during navigation of a BVI user.

DESCRIPTION OF RELATED ART 

A survey of blind experts has shown that after outdoor navigation, the second most important ETA feature for BVI persons is indoor navigation and orientation, for example, in public institutions, supermarkets, office buildings, homes, etc. BVI persons need ETA for orientation and navigation in unfamiliar indoor environments with embedded features for the detection and recognition of obstacles (not only on the ground but also at eye-level) and desired destinations such as rooms, staircases, elevators, doors, and exits. To-date, BVI indoor navigation in unknown environments is still the most critical task for developers in this area due to a weak Global Positioning System (GPS) signal indoors and costly pre-arranged indoor infrastructural installations (such as WI-FI routers, beamers, RFID tags, 5G signals, etc.). Thus, some other special techniques or technologies are needed. Solutions tailored to function in very restrictive settings, tests lacking robustness, and the limited involvement of end-users were emphasized as major limitations of the existing ETA research initiatives. A tradeoff between the accuracy and costs of developing and deploying an indoor navigation solution was highlighted as a limiting factor after a thorough review of various technologies. Wi-Fi was pointed out as the most economically feasible alternative as long as such infrastructure is properly installed and the users can tolerate lower accuracy. In a semi-structured survey of BVI experts, a basic understanding of users' expectations and requirements for indoor ETA solutions was presented and enabled the identification of some new developments in the field.

A wide range of general-purpose social networks, web 2.0 media apps, and other smart ICT (information and communication technology) tools have been developed to improve people’s daily tasks, including navigation and orientation. Although they are not designed to meet the specialized requirements of BVI people, some features make them useful. For instance, text (and image) to voice, tactile feedback, and other additional enabling software and hardware solutions are helpful for this matter. However, the complexity and abundance of features pose a significant challenge for BVI persons. The volumes of information together with data from social networks confuse BVI users. In this way, Web 2.0 social networks do not guarantee specialized digital content accessibility for BVI users. Some more focused approaches are in demand. BVI people frequently use apps specifically designed for them to accomplish daily activities. However, N. Griffin-Shirley et al. emphasizes that persons with visual impairments would like to see both improvements in existing apps and new apps. Several, currently available navigation apps are primarily based on pre-developed navigational information but do not provide real-life support, experience-centric user approaches, and participatory Web 2.0 social networking. Other real-life social apps enable access to a network of sighted users and company representatives who are ready to provide real-time visual assistance for the BVI tasks at hand. However, these apps are not adapted to the specific BVI needs in the indoor routes while navigating, orientating, getting lost, etc. To our knowledge, there are no publications or patents, which describe a handsfree BVI indoor navigation approach with crowd-sourced navigational routes, which provides tactile and audio information to the BVI user and which uses facial EMG signals as a source for a user-controlled instructive interface. In the present invention, these issues are addressed.

SUMMARY OF INVENTION 

This invention integrally and innovatively deals with the following main technical problems that are known in the field of BVI ETA navigation and orientation applications indoors: 

  1. Real-time step-by-step ETA navigation approach, using an intelligent odometrybased guiding map that is supported on visual, sensory, and semantic information. 
  2. Employment of wearable, inconspicuous, and user-friendly headband with integrated indiscernible sensors and other devices. 
  3. Hands-free ETA system control interface. 
  4. A stand-alone computing device or smartphone can work as a means for the ETA system’s configuration setup, intermediate data processing, and GSM communication. 
  5. Real-time intuitive tactile display interface and myographic (EMG) user feedback.  
  6. Employment of machine learning (deep neural networks or other artificial intelligence-based methods) and robot navigation approaches for object recognition and estimation of the best navigational routes, custom routing, semantic analysis, location-aware search, and discovery. 
  7. BVI real-time access to a Web cloud server and continuously updated indoor navigational routes’ database. 
  8. Web crowd assisted continuous mapping of guiding navigational routes, using sighted users' help. 
  9. Real-time coordinated audio and tactile BVI guidance in complex, out of ordinary indoor situations, using sighted users' online help.

Above listed technical problems are addressed using an integrated ETA system approach, where proposed innovative hardware components and software applications work in a coordinated manner. 

Fig. 1. Functional prototype of wearable indoor navigation system

Fig. 2. Guided indoor navigation in action

Fig. 3. Prototype testing: round#1

Fig. 4. Prototype testing: round#2

Fig. 5. Prototype testing: round#3

Fig. 6. Smartphone app: screen views