Hand Gesture Recognition CNN

A gesture recognition system built on a convolutional neural network trained on the Leap Gesture Recognition dataset, classifying hand gestures in real time from a camera feed.

Model

The network is a multi-layer CNN using batch normalisation, dropout and max pooling for generalisation, trained with categorical cross-entropy and the Adam optimiser. Training uses early stopping, model checkpointing and learning-rate reduction on plateau, with stratified train/test splitting and one-hot encoded labels.

Preprocessing

MediaPipe Hands detects and isolates the hand region before classification. Images are converted to grayscale, resized to 64x64 and normalised. Doing detection first is what makes the classifier robust: the CNN only ever sees a tightly cropped hand, so background and lighting variation matter far less.

Real-time inference and interface

OpenCV handles the live video feed, with bounding boxes extracted per frame and hand landmarks processed dynamically. A PyQt interface lets you test gestures and see predictions as they happen, which makes the failure modes obvious in a way that a confusion matrix does not.