Interfaz humano-computador para personas con limitación motriz de miembros superiores basada en gestos faciales

Abstract

Body language is important to communicate fluently with people. In the area of machine interaction, there are systems to automatically recognize facial gestures. In the case of people with motor limitation of upper limbs, facial gestures are the only way to communicate with the world, nonetheless current interfaces do not take into account the mobility reductions that most people with motor limitations experience during their periods of recovery. In order to contribute to the solution of this problem, a human machine interface is here described. Our solution uses computer vision techniques to detect, track and estimate head pose from images acquired by a webcam to generate commands into WhatsApp application. For avoiding user fatigue, the system automatically detects key interaction regions within WhatsApp such as the conversation selection area and the chat selection area by using desktop screenshots and pattern recognition techniques. The interface is complemented with modules to configure its operation and commanding computer cursor. Programming is realized through Python 3.6.8, OpenCV Version 4.2.0 and DLIB Version 19.8.1 libraries to process and analyze webcam images, PyAutogui Version 0.9.42 to acquire computer screen, emulate keyboard and cursor events, and Qt library Version 4.8 to expose a simple user interface. Performance is assessed by videos of people using four interaction commands with WhatsApp. Various types of lighting, backgrounds, camera distances, postures, and movement speeds are used during testing. The results show that the algorithm detects, tracks and estimates head pose in 85% of cases. Failures occur when there are light sources in front of the camera, darkness, or movement behind the face. Our application executes at 10 FPS of speed and uses 16% of an Intel Core i5 processor and 2 GB out of 8GB RAM. The platform is able to distinguish faces of similar skin tone, however is not very tolerant to background movements. The ease of interaction and the fast learning curve allow to conceive the interface developed for applications with upper-limb disabled people, such as tetraplegics and people with stroke.