Block-Wise State Encoding for Action-Masked Reinforcement Learning in Flexible Job-Shop Scheduling

This paper addresses the flexible job-shop scheduling problem (FJSP) as a constrained combinatorial optimization task with a large discrete action space. Although action-masked reinforcement learning has shown promise for such problems, the effect of structured vector-state encoding in scheduling has received less attention. The main contribution of this work is a structured block-wise state representation and a multi-branch feature extraction module for action-masked Proximal Policy Optimization (PPO). The proposed representation decomposes the scheduling state into three heterogeneous components capturing resource availability, operation readiness, and temporal attributes of operation–machine alternatives. Instead of flattening these signals into a single vector, the proposed encoder processes each block separately before aggregation, with the aim of preserving semantic structure during policy learning. To isolate the effect of representation design, we compare the proposed multi-branch encoder with a baseline single-branch multilayer perceptron under identical PPO hyperparameters and training conditions. Experiments on the Brandimarte MK benchmark suite show that the proposed architecture yields a lower best-achieved makespan on nine of ten instances and improves the best baseline result by up to 27.84%. Additional validation on selected Behnke and Geiger instances indicates that the BR encoder’s advantage extends to larger FJSP cases while preserving sub-second inference.