From Pose to Pitch: Classifying Baseball Pitch Types with Projection-Gated ST-GCN ^†

Abstract

We present a skeleton-based approach to baseball pitch type classification from broadcast video. We leverage Human Pose Estimation and an ST-GCN architecture, improved with a projection-gated temporal downsampler, to learn kinematic signatures of the pitcher’s body, adaptively selecting the most informative frames, enabling pitch type classification without the need for ball tracking. On the MLB-YouTube dataset, our proposed method reaches ~62% six-class accuracy, highlighting body mechanics as a practical biometric cue.

1. Introduction

Biometrics in sport increasingly leverages kinematic signatures for recognition and assessment. In baseball, a pitcher’s coordinated joint timings, segment orientations, and inter-limb couplings form a movement biometric indicative of pitch intent and deception. While systems such as Statcast provide precise ball and arm trajectories, they require costly radar/multi-camera setups limited to elite venues [1]. A body-centric alternative is to infer a skeletal pose from broadcast video and classify pitch type directly from pitcher dynamics, avoiding explicit ball tracking.

Prior work follows two directions. Trajectory-centric methods analyze ball motion or pre-pitch context for discrimination and outcome prediction [2,3]. Movement-centric methods read the pitcher’s mechanics via motion capture, wearables, or vision: wearables show proximal kinematics (pelvis/trunk) are predictive of pitch categories and outcomes [4,5]; vision pipelines range from handcrafted features and 3D CNNs on full frames [6,7,8,9] to pose-guided models emphasizing the athlete [10]. Skeleton-based action recognition, modeling the body as a spatiotemporal graph, learns inter-joint relations directly [11,12].

2. Materials & Methods

Motivated by this biometrics perspective, we isolate the pitcher using OpenPose [13] and model body motion with an ST-GCN backbone [11]. To improve temporal selectivity, we replace uniform stride-based downsampling with a Projection-Gated Temporal Downsampler (PGTD) that halves the temporal rate by softly selecting and fusing non-overlapping frame pairs, sharpening informative kinematic events while suppressing noise. This body-centric, graph-based design aligns with evidence linking kinematic measurements to pitch categorization [4], while remaining practical in broadcast settings.

We insert PGTD between the spatial graph convolution (GCN) and the temporal convolution (TCN) Figure 1. From broadcast video, we estimate and isolate pitcher poses exactly as in [12]; the pose stream feeds our Projection-Gated ST-GCN. In the baseline, an input $X\in {\mathbb{R}}^{N\times C\times T\times V}$ (batch, channels, frames, joints) is processed by a spatial aggregation and then a $(k_t,1)$ temporal convolution with BN-ReLU-Dropout. Using normalized adjacency partitions ${\left\{{\widehat{A}}_k\right\}}_{k=1}^K$ and weights $W_k\in {\mathbb{R}}^{C\times C^{\prime }}$, the spatial operator appears inline as $\mathrm{GCN}\left(X\right)={\sum }_{k=1}^K{\widehat{A}}_{k}X{W}_k$. Temporal downsampling in original ST-GCN uses stride 2 in selected blocks, blindly discarding every other frame.

PGTD activates only in blocks with projection residuals (channel changes). When active, the TCN keeps stride 1 and the gate performs the halving. Given GCN output $X^{\prime }\in {\mathbb{R}}^{N\times C^{\prime }\times T\times V}$, PGTD yields $Y\in {\mathbb{R}}^{N\times C^{\prime }\times \lfloor T/2\rfloor \times V}$ by fusing non-overlapping pairs $(2u,2u+1)$. The value signal uses a pointwise tanh plus per-time “cv” normalization (layer-normalization across $[C^{\prime },V]$):

$$V=\mathcal{N}\left(tanh\left(X^{\prime }\right)\right).$$

(1)

Gating logits come from a depthwise temporal convolution (groups $=C^{\prime }$), kernel $k_t=3$:

$$G={\mathrm{DWConv}}_t\left(X^{\prime }\right).$$

(2)

For each window $(2u,2u+1)$ we compute a two-way softmax with temperature $\tau =1.0$,

$$W_u=\mathrm{softmax}\left({\displaystyle \frac{\left[G[:,:,2u,:],G[:,:,2u+1,:]\right]}{\tau }}\right),$$

(3)

and mix the pair at half rate:

$$Y[:,:,u,:]=W_{u,0}\odot V[:,:,2u,:]+W_{u,1}\odot V[:,:,2u+1,:].$$

(4)

Finally, the block sums a stride-1 TCN on Y with a temporally aligned residual via average pooling (kernel/stride 2):

$$Z={\mathrm{TCN}}_{\mathrm{stride}=1}\left(Y\right)+{\mathrm{AvgPool}}_t^{(k=2,s=2)}\left(R\right).$$

(5)

For odd T, the last frame of $X^{\prime }$ is dropped; residual pooling also yields $\lfloor T/2\rfloor$, avoiding padding. This placement (GCN → PGTD → TCN) replaces blind decimation with data-driven selection that emphasizes motion-salient frames before temporal filtering. Because the gate is depthwise and pointwise-free, its cost is modest, $O\left(C^{\prime }{k}_{t}TV\right)$, and it is applied only when channels change; identity blocks match the original ST-GCN. In practice, non-overlapping windows of 2, tanh values, cv normalization, $k_t=3$, $\tau =1.0$, and TCN stride 1 provide adaptive halving with minimal disruption and stable training.

We evaluate on MLB-YouTube [10], a fine-grained benchmark with 5759 segmented clips from 20 2017 MLB postseason broadcasts. Of these, 5244 are pitches labeled as changeup, curveball, fastball, knuckle-curve, sinker, slider with class counts 331, 338, 2974, 395, 213, 993; the remainder are other events. Clips are ∼6 s at 60 fps and 1280 × 720. The split provides 4177 training and 1067 test clips (80/20). Challenges include subtle inter-class kinematic differences, severe class imbalance, and complex visuals. Experiments ran on an Intel Core i7-12700 CPU and a single NVIDIA RTX 4060 Ti (16 GB).

3. Results

Trained with Adam ($lr = {10}^{-3}$) and Cross-Entropy loss on MLB-YouTube [10], the Projection-Gated ST-GCN attains 61.8% six-class accuracy at epoch 216. Class-wise accuracies show strong discrimination for fastball (78.41%) and knucklecurve (71.26%), solid changeup (65.62%) and curveball (62.32%), and remaining difficulty on sinker (50.00%) and slider (43.22%). The gap among precision (0.58), recall (0.62), macro-F1 (0.59), and weighted-F1 (0.69) reflects class imbalance, with abundant fastballs lifting the weighted score. Higher recall but lower precision for curveball suggests confusion with related breakers; slider remains most confounded, consistent with kinematic overlap. These findings corroborate

Table 1. Comparison with state-of-the-art methods on MLB-YouTube for six-class pitch classification.

Method	Parameters	FLOPs	Inference Time	Accuracy (%)
Random	—	—	—	17.0
I3D [10]	12.73 M	528.9 G	80.73 ms	34.5
InceptionV3 [10]	27.19 M	5.73 G	25.42 ms	36.4
ST-GCN [12]	3.07 M	2.51 G	2.06 ms	58.5
Projection-Gated ST-GCN [Ours]	3.09 M	3.32 G	2.53 ms	61.8

: projection-gated processing improves recognition from body motion alone, with adaptive selection particularly aiding classes with distinctive kinematic signatures. We also report parameter count, FLOPs, and inference time, indicating a practical accuracy/efficiency balance: from 3.07 M, 2.51 G, 2.06 ms for ST-GCN to 3.09 M, 3.32 G, 2.53 ms for our model.

4. Conclusions

Treating a pitcher’s movement as a biometric signal, we show that skeleton-based modeling can classify pitch types from broadcast video without ball tracking. Our Projection-Gated variant of ST-GCN replaces stride-2 decimation with content-adaptive selection between GCN and TCN, yielding 61.8% accuracy on MLB-YouTube with a modest trade-off, adding ∼0.47 ms of average inference time per clip for a 3.3% accuracy gain. Per-class trends indicate benefits for pitches with distinctive kinematic cues. Future work aims to mitigate class imbalance with different augmentation techniques, explore overlapping or learned windows to capture the different phases of the pitching movement, and using 3D poses when available, targeting further gains without sacrificing deployability.