Abstract

Hand Gesture Recognition (HGR) is a form of perceptual computing that allows artificial systems to capture and interpret human gestures. HGR has applications in human-machine interaction, virtual reality, augmented reality, and human behavior analysis. The human hand can assume a near-infinite number of poses and orientations to form myriad gestures, thus increasing the difficulty of the HGR task.

The hand skeleton of connected joints effectively describes the hand’s geometric shape and thus contains richer semantic gesture information while eliminating noise from individual differences in physical hand characteristics. The efficacy and computational efficiency of skeleton-based HGR frameworks can be significantly enhanced by transforming the hand gesture recognition task into an image classification task, taking advantage of the success of ML algorithms/architectures in the image classification domain.

This research thesis document presents a skeleton-based HGR framework that implements temporal information condensation via data-level fusion to encode spatiotemporal gesture information into RGB images. The spatiotemporal images are processed through an end-to-end ensemble multi-stream CNN architecture that generates the class probabilities.

The framework was extensively evaluated on the CNR, FPHA, LMDHG, SHREC2017, and DHG1428 HGR benchmark datasets. The framework’s performance was competitive with the SOTA for the evaluation datasets, within -4.10% and +6.86% of reported classification accuracies. The experimental results confidently demonstrated the viability of the temporal information condensation and data-level fusion techniques within the HGR domain.

School

School of Sciences and Engineering

Department

Robotics, Control & Smart Systems Program

Degree Name

MS in Robotics, Control and Smart Systems

Graduation Date

Winter 1-31-2023

Submission Date

12-20-2022

First Advisor

Dr. Maki Habib

Second Advisor

Dr. Mohamed Moustafa

Committee Member 1

Dr. Amr Goneid

Committee Member 2

Dr. Hazem Abbas

Committee Member 3

Dr. Mohamed El-Morsi

Extent

139 p.

Document Type

Master's Thesis

Institutional Review Board (IRB) Approval

Not necessary for this item

Download

Share

COinS