Humans and Machines Work in Sync in a Cockpit Assistance System

Within the framework of the European Peggasus project, CSEM and its partners have developed a vision system that allows pilots to focus on important tasks in the cockpit, improving crew efficiency and maximizing flight safety. This new human-machine interface enables remote eye tracking and hand gesture recognition and draws on the latest developments in artificial intelligence and computer vision. The system has been tested successfully by ten professional pilots in a flight simulator.

Most commercial aircraft built since 1914 are equipped with an autopilot system. Despite the clear benefits these systems provide (such as enhanced flight safety and operations), new challenges have emerged owing to the increased complexity of modern cockpits. Today these cabins contain an array of large computer screens, many secondary dials, and hundreds of switches. That means pilots have to process vast amounts of information in high-stress, high-workload situations.

Video Credit: CSEM

A New Human-machine Interface

This is the problem that PEGGASUS – short for Pilot Eye Gaze and Gesture tracking for Avionics Systems using Unobtrusive Solutions – has set out to tackle. The research was coordinated by CSEM and financed by the EU, and the other consortium members include ETH Zurich, SERMA Ingénierie and Swiss International Air Lines. CSEM worked with these project partners to develop a novel human-machine interface in the form of a dashboard-mounted, non-intrusive camera system. The system can track a pilot’s gaze and recognize their hand gestures in real time, inside a cockpit. It gives pilots seamless and accurate feedback, thereby lightening their workload and improving their situational awareness when executing a multitude of complicated tasks.

“The PEGGASUS system addresses the unique challenges encountered in aeronautical context situations, such as extreme variations in light intensity,” says Andrea Dunbar, the consortium coordinator at CSEM. “It improves pilots’ situational awareness, and consequently decision-making, and makes it easier for the flight crew to work with aircraft controls.

Eye Gaze Tracking and Hand Gesture Recognition

PEGGASUS employs state-of-the-art computer-vision and machine-learning algorithms. Its algorithmic pipeline for eye tracking combines analytical and data-driven approaches, operating in real time at 60 frames per second with minimum latency while achieving eye-tracking accuracy of better than 1°.

To enable hand-gesture recognition, a deep neural network model was developed and incorporated into the final PEGGASUS system, providing a new form of interaction directly in the cockpit.

This one-of-a-kind system is the result of a collaborative effort among all consortium members, with each one drawing on its own expertise. CSEM developed the vision system and machine-learning algorithms for eye tracking and hand gesture recognition. SERMA Ingénierie integrated the PEGGASUS hardware system into a cockpit demonstrator for testing. ETH Zurich, with its know-how in human-computer interaction and aviation safety, designed and carried out the testing in a simulator study. Swiss International Air Lines supported the consortium by providing insight on use cases and ideas for future training applications.

Tested and Approved by Professionals

The PEGGASUS system was tested by ten professional pilots in a cockpit simulator provided by Lufthansa Aviation Training, Switzerland. They gave very positive feedback and rated the PEGGASUS vision system better than head-mounted eye-tracking systems in terms of comfort and low distraction.

“Based on many incident and accident analyses, the highest degree of flight safety can be obtained from an optimized human-machine interface,” says Gilad Scherpf, the head of Aviation Training Development at Lufthansa Group. “Any steps towards improving such an interface will be of great value to Europe’s avionics industry. What’s more, these interfaces can improve both flight operations and pilot training. Computer-assisted technology like the one used in PEGGASUS can be very effective in modern airliners, especially since air-safety levels will need to be maintained or increased as the degree of automation increases.”

Articles

• Cockpit camera system monitors pilots’ eye-gaze and tracks hand-gestures
  Source: Article published in CORDIS | European Commission (2022).
• "AI-based Hand Gesture Recognition System for Improved Human Machine Interaction in Cockpits" 
  Source: CSEM Scientific & Technical Report 2020, page 105.