Computer vision researchers have introduced an efficient deep learning architecture called STANet that recognizes isolated Indian Sign Language gestures with high precision while achieving an average inference latency of less than four milliseconds on a desktop graphics processing unit. Developed by investigators Surbhi Maheshwari, Amit Kumar Sinhal, and Arpana Sinhal, the hybrid model couples a time-distributed two-dimensional convolutional frame encoder with causal temporal convolutions and multi-head self-attention. By eliminating recurrent mechanisms like long short-term memory networks, the system captures complex temporal motion patterns across video clips while substantially minimizing computational overhead. Evaluated on stratified clip-level benchmarks, STANet achieved a classification accuracy of 88.36 percent on the public INCLUDE-50 dataset and 94.38 percent on a newly introduced conversational dataset named ISL-50. The research outlines a structured pipeline engineered to balance handshape representation against execution speed, delivering thorough benchmarks that define both the real-time processing capabilities and the generalization boundaries of vision-based sign language recognition.

The societal motivation for this development stems from an acute accessibility deficit across India, where approximately 63 million people live with significant hearing impairment among at least 430 million worldwide requiring disabling hearing loss care. Within India’s Deaf and Hard-of-Hearing community, Indian Sign Language serves as a primary communication medium, yet assistive digital recognition tools remain scarce. Prior engineering approaches have struggled to balance accuracy with computational efficiency. Frame-level convolutional networks effectively extract static hand shapes and finger placements, but fail on gestures whose meaning depends on movement trajectories. Three-dimensional convolutional models and recurrent architectures like bidirectional long short-term memory networks capture temporal dynamics, yet high parameter counts and sequential dependencies cause substantial inference delays that impede interactive use. Furthermore, Transformer models require massive training datasets and pre-training pipelines that are rarely accessible for regional sign vocabularies.

To address this technical gap, the investigators configured STANet to process video sequences through a streamlined modular design without recurrent layers. The spatial encoder processes input video clips formatted as tensors of twenty uniformly sampled frames resized to 64 by 64 pixels. Operating across each frame independently, a time-distributed convolutional neural network extracts spatial cues including hand shape, palm orientation, and bodily placement. This encoder consists of three successive stages: a 32-filter convolutional layer, a 64-filter convolutional layer, and a 128-filter convolutional layer. Each stage incorporates two-dimensional convolution, batch normalization, max pooling, and a dropout regularization rate of 0.2 to mitigate overfitting. Spatial dimensions are subsequently flattened for each frame, transforming the video into a compact feature vector sequence of length twenty. By utilizing time-distributed two-dimensional convolutions rather than computationally intensive three-dimensional convolutions, the model maintains a compact footprint while preserving the fine visual details necessary for gesture classification.

The resulting sequence of spatial representations enters a temporal processing pipeline featuring two causal one-dimensional convolutional blocks followed by multi-head self-attention. Each temporal convolutional block uses 128 filters with a kernel size of three, accompanied by batch normalization and a dropout probability of 0.3. The causal property ensures that temporal filtering references only past and current frames, maintaining compatibility with streaming applications. This temporal convolution stage captures movement trajectories and sequence-level transitions that differentiate gestures possessing identical static appearances. The output then enters a multi-head self-attention module configured with four heads and a key dimension of 64, matching the 128-dimensional temporal representations. Enhanced by residual connections and layer normalization with an epsilon of 1e-6, the attention mechanism dynamically focuses on the most informative frames within a gesture. Finally, a global average pooling layer aggregates the sequence into a unified representation, feeding a 256-unit dense layer with rectified linear activation and a fifty-class SoftMax classifier.

Experimental validation was conducted on two distinct benchmarks: the established INCLUDE-50 dataset and a custom conversational dataset named ISL-50. INCLUDE-50 contains 943 video clips representing fifty diverse vocabulary classes, including practical objects, temporal markers, and abstract signs such as court and priest. The clips average 2.53 seconds in duration and exhibit minor class imbalances. To address practical communicative needs, the researchers developed ISL-50, which comprises 800 balanced video clips across fifty culturally relevant categories including greetings, familial relations, daily activities, and practical objects like table, bag, and paper. Four native signers, comprising two men and two women aged 18 to 25, performed each sign sixteen times in a controlled indoor environment with uniform lighting and minimal background clutter. High-resolution recordings were captured using an Apple iPhone 13 Pro placed 1.5 to 2.0 meters away, with average clip lengths of 2.88 seconds. Stratified clip-level splits allocated eighty percent of data to training and validation and twenty percent to testing.

Under this stratified signer-dependent protocol, STANet attained notable classification scores on both evaluation benchmarks. On the 189 held-out test clips of INCLUDE-50, the architecture reached an 88.36 percent top-1 accuracy and a macro F1-score of 0.8841. On the 160-clip test set of ISL-50, STANet achieved a 94.38 percent top-1 accuracy, a 96.88 percent top-3 accuracy, and a macro F1-score of 0.9399. An analysis of the resulting confusion matrices indicated that classification errors were highly localized rather than dispersed broadly across classes. On INCLUDE-50, residual misclassifications concentrated in a compact group of visually ambiguous categories including T-shirt, girl, shoes, boy, hot, new, and small or little. On ISL-50, residual confusion centered almost exclusively on semantically or visually related pairs. Most notably, namaste was occasionally confused with thank you, and wife was confused with husband, because both pairs share identical handshapes and chest-level physical positioning, differing primarily in subtle directional movements that challenge 64 by 64 pixel resolutions.

When compared against earlier baseline models implemented under the same evaluation pipeline, STANet demonstrated clear computational and predictive advantages. A baseline three-dimensional convolutional model achieved top-1 accuracies of only 63.49 percent on INCLUDE-50 and 64.38 percent on ISL-50, while requiring 9.83 million parameters and 4.33 billion floating-point operations per clip. Adding causal temporal convolution to that baseline raised accuracy to 73.54 percent on INCLUDE-50 and 83.13 percent on ISL-50 while operating at 1.79 billion operations. The strongest recurrent baseline, combining three-dimensional convolutions with bidirectional recurrence and attention, reached 82.01 percent on INCLUDE-50 and 91.25 percent on ISL-50, but required 6.86 million parameters and 4.41 billion operations. In contrast, STANet delivered higher accuracy using just 3.47 million parameters and 1.79 billion operations per clip. On a desktop NVIDIA GeForce GTX 1650 graphics card, STANet required 3.488 milliseconds per clip on INCLUDE-50 and 3.481 milliseconds on ISL-50, cutting inference latency by more than half compared to the recurrent baseline.

To isolate the contribution of temporal modeling independent of encoder variations, the team conducted an encoder-controlled ablation holding the time-distributed convolutional backbone fixed. On ISL-50, the convolutional backbone alone reached 89.33 percent accuracy, while adding temporal convolution or self-attention individually produced marginal gains to 90.00 percent and 90.63 percent. On INCLUDE-50, the backbone achieved 82.25 percent, while individual additions yielded 83.00 percent and 83.69 percent. In contrast, combining both modules in STANet produced substantial improvements, reaching 94.38 percent on ISL-50 and 88.36 percent on INCLUDE-50. This demonstrates clear architectural synergy: temporal convolution compresses feature dimensions from wide spatial tensors to 128 channels, allowing self-attention to operate efficiently without expanding model size. Direct attention on spatial features required 10.65 million parameters, whereas STANet required only 3.47 million. However, paired McNemar significance tests on retained per-clip predictions yielded raw p-values between 0.05 and 0.21, none of which remained statistically significant after Holm correction due to small test-set sample sizes.

Sensitivity analyses confirmed the empirical validity of core architectural parameters governing temporal sampling and attention allocation. Evaluating sequence lengths across 10, 20, 30, and 40 frames on ISL-50 demonstrated an optimal peak at twenty frames, reaching 94.38 percent accuracy. Reducing sampling to ten frames degraded accuracy to 87.50 percent by omitting critical motion phases, whereas expanding sampling to thirty and forty frames caused minor declines to 93.13 percent and 92.50 percent due to sequence redundancy. Similarly, testing attention heads at a fixed key dimension of 64 showed that four heads achieved the highest accuracy, outperforming two heads at 91.25 percent and eight heads at 92.50 percent. Crucially, when evaluated under a leave-one-signer-out protocol where one signer was completely excluded across four folds, RGB accuracy collapsed from 94.38 percent to roughly 19 percent. This drastic drop confirmed that raw RGB representations overfit to signer-specific visual appearance, such as skin pigmentation and clothing, that fails to generalize across individuals.

To address this generalization limit, the researchers extracted MediaPipe skeletal landmarks and fed coordinates into STANet’s temporal processing core. This landmark variant recovered mean leave-one-signer-out accuracy to 58.4 percent, with a bootstrap 95 percent confidence interval between 56.7 and 60.1 percent, compared to just 26 percent achieved by feeding identical landmarks into a recurrent network. A two-stream architecture fusing RGB frames with landmarks reached 57.3 percent accuracy, showing no statistically significant advantage over landmarks alone according to paired McNemar testing. This confirms that structural pose geometry, rather than raw appearance, enables cross-signer generalization. The study emphasizes that while STANet achieves desktop processing times of roughly 3.48 milliseconds, practical mobile deployment requires further profiling. An experimental browser demonstration revealed that unoptimized video decoding and framing overhead dominate real-time latency. The authors conclude that extending isolated recognition to continuous sign translation, expanding signer cohorts, and deploying landmark-driven models to embedded hardware represent critical next steps for assistive sign interfaces.

Subject of Research: Isolated Indian Sign Language recognition using a compact spatial-temporal attention neural network

Article Title: An efficient spatial-temporal attention network for low-latency isolated Indian Sign Language Recognition

Article References: Maheshwari, S., Sinhal, A. K., & Sinhal, A. (2026). An efficient spatial-temporal attention network for low-latency isolated Indian Sign Language Recognition. Discover Computing, 29(1), Article 870. https://doi.org/10.1007/s10791-026-10623-x

Image Credits: AI Generated

DOI: 10.1007/s10791-026-10623-x

Keywords: Indian Sign Language, Sign Language Recognition, STANet, Spatial-Temporal Attention, Causal Temporal Convolution, Computer Vision, Deep Learning, Gesture Recognition, INCLUDE-50, ISL-50 Dataset, MediaPipe Landmarks, Inference Latency, Model Efficiency, Ablation Study, Assistive Technology, Leave-One-Signer-Out