scRL: Utilizing Reinforcement Learning to Evaluate Fate Decisions in Single-Cell Data

Simple Summary

Understanding how cells develop into different types during growth and disease is crucial for advancing medicine, but current methods cannot pinpoint exactly when and where cells make these critical decisions. We developed a new artificial intelligence tool called single-cell reinforcement learning that treats cell development like a strategic decision-making game. Just as a chess player learns to make optimal moves, our system learns to identify the precise moments when cells decide their future fate—whether to become blood cells, immune cells or other specialized types. We tested this approach on various biological systems, including normal human blood cell development, cancer cells, mouse organ development and cells responding to radiation damage. Our method consistently outperformed fifteen existing state-of-the-art tools and successfully identified early decision points that occur before cells show obvious signs of commitment to specific lineages. Additionally, we discovered previously unknown regulatory factors that control these decisions. This breakthrough provides scientists with a powerful new way to understand how cells make developmental choices, which could lead to better treatments for diseases like cancer and improved strategies for regenerative medicine. By revealing the hidden decision-making logic of cellular development, this work opens new possibilities for controlling and directing cell fate in therapeutic applications.

Abstract

Single-cell RNA sequencing now profiles whole transcriptomes for hundreds of thousands of cells, yet existing trajectory-inference tools rarely pinpoint where and when fate decisions are made. We present single-cell reinforcement learning (scRL), an actor–critic framework that recasts differentiation as a sequential decision process on an interpretable latent manifold derived with Latent Dirichlet Allocation. The critic learns state-value functions that quantify fate intensity for each cell, while the actor traces optimal developmental routes across the manifold. Benchmarks on hematopoiesis, mouse endocrinogenesis, acute myeloid leukemia, and gene-knockout and irradiation datasets show that scRL surpasses fifteen state-of-the-art methods in five independent evaluation dimensions, recovering early decision states that precede overt lineage commitment and revealing regulators such as Dapp1. Beyond fate decisions, the same framework produces competitive measures of lineage-contribution intensity without requiring ground-truth probabilities, providing a unified and extensible approach for decoding developmental logic from single-cell data.

1. Introduction

1.1. Single-Cell Sequencing and Dimensionality Reduction

Single-cell sequencing now enables genome-wide measurements at the resolution of individual cells, exposing previously hidden heterogeneity across tissues and disease states [1,2,3]. Each experiment routinely profiles tens of thousands of genes in thousands to millions of cells, yielding data that are not only high-dimensional but also sparse, noisy and burdened by technical artefacts [4]. Consequently, DR (dimensionality-reduction) methods have become indispensable for extracting biological structure from such data [5].

A major downstream application of DR in the single-cell field is pseudotime analysis, which seeks to order cells along putative developmental trajectories [6,7]. Visualization techniques—including PCA (principal component analysis), t-SNE (t-distributed Stochastic Neighbour Embedding) and UMAP (Uniform Manifold Approximation and Projection)—facilitate exploratory analysis and often serve as the basis for trajectory inference [8,9,10]. Indeed, tools such as PAGA, Monocle 3, Slingshot, and CAPITAL exploit UMAP embeddings to reconstruct branching lineages [11,12,13,14]. Yet, despite their success in ordering cells, existing approaches seldom address where and when lineage commitment occurs, nor do they provide a quantitative framework to evaluate these fate decisions. In reality, differentiation is not strictly hierarchical but resembles a rugged landscape in which progenitors become lineage-restricted at context-dependent times [15]. Detecting the continuous states of fate decisions that guide this process therefore remains an open problem.

1.2. Manifold Learning for Dimensionality Reduction

Linear DR techniques, exemplified by PCA, are computationally efficient but fail to capture the nonlinear geometry typical of biological systems [16]. Manifold-learning algorithms address this limitation by assuming that high-dimensional observations reside on, or near, a low-dimensional manifold embedded in ambient space [10,17]. Isomap [18], LLE (Locally Linear Embedding) [19], t-SNE [20] and UMAP [10] each preserve different aspects of the manifold—geodesic distances, local linear reconstructions or neighbourhood probabilities—thereby recovering structure inaccessible to linear projections. Theoretical foundations draw on differential geometry, topology and graph theories, providing a flexible toolkit for biological data characterized by nonlinear relationships and multiple developmental branches [16].

1.3. Applications in Single-Cell Data Analysis

Manifold learning has become integral to single-cell workflows because it reveals continuous developmental paths, rare subpopulations and functional gradients that remain hidden in the original space [21,22]. UMAP visualizations, for example, readily separate major immune lineages and their subtypes from ${\mathrm{CD45}}^{+}$ scRNA-seq profiles [23]. Specialized adaptations align transcriptomic with epigenomic or electrophysiological modalities [24,25], and Gaussian-process latent variable models capture noisy gene expression as smooth functions of latent states [26]. Nevertheless, even state-of-the-art manifold approaches primarily describe trajectories; they do not explicitly evaluate the decisions that drive differentiation.

1.4. Reinforcement Learning for Cellular Differentiation Trajectories

RL (reinforcement learning) offers a principled framework for sequential decision-making in complex, dynamic environments [27]. An RL agent iteratively selects actions, observes transitions and updates its policy to maximize the cumulative reward—attributes that have delivered breakthroughs in robotics, game playing and navigation [28,29,30]. Conceptually, cellular differentiation mirrors this paradigm: (i) a transcriptomic profile represents the state, (ii) regulatory events such as gene activation or repression correspond to actions, (iii) differentiation steps define state transitions, and (iv) developmental constraints or lineage markers provide rewards [31].

RL is therefore well-suited to explore high-dimensional single-cell spaces, balance the discovery of novel routes with optimization of known trajectories and pinpoint branch points where fate decisions occur [32,33]. Deep RL extends these capabilities to raw, high-dimensional inputs by coupling neural networks with value- or policy-based optimization [34]. For scRNA-seq data—often exceeding 10,000 genes—such expressivity is essential [35]. Framing differentiation as an RL task promises to reveal regulatory logic that conventional pseudotime tools overlook, particularly in multi-lineage systems with complex branching topologies [36].

1.5. Our Contribution

Here we introduce scRL, a reinforcement-learning framework that integrates manifold learning, an actor–critic architecture and biologically informed reward functions to decode fate decisions from single-cell transcriptomes. Briefly, we (i) construct an interpretable latent space via LDA, (ii) embed this space onto a two-dimensional grid, preserving UMAP topology, and (iii) train an RL agent whose critic learns state values reflecting lineage potential. scRL thereby identifies pre-expression decision states, quantifies lineage and gene decision intensities, and uncovers regulatory factors. Across diverse datasets—including human hematopoiesis, acute myeloid leukemia, mouse endocrinogenesis, Dapp1 knockout and irradiation injury—scRL outperformed benchmark fate-inference and pseudotime methods, revealed novel regulators and mapped dynamic fate biases (see Results, Figure 1, Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10). By viewing differentiation through the lens of sequential decision optimization, scRL provides a rigorous, extensible framework for studying cell-fate decisions, with potential applications in regenerative medicine, oncology and developmental biology.

2. Materials and Methods

2.1. Architecture and Workflow of scRL

scRL couples single-cell manifold learning with an actor–critic algorithm in three sequential stages (Figure 1).

• Stage 1: Preprocessing and grid embedding (Figure 1A)

  (i)

  Dimension reduction. The gene count matrix is first projected into a latent space.

  (ii)

  Manifold construction and clustering. A UMAP embedding is built from the latent space matrix and Leiden clustering is applied to annotate coarse cell populations.

  (iii)

  Grid embedding. To obtain a lattice that preserves local topology yet yields fully connected paths, edges in the UMAP k-NN graph are filtered by a Canny-style detector; surviving edges are rasterized into an $m\times n$ grid. This transforms scattered points into a continuum that spans primitive to terminal states.

  (iv)

  Pseudotime assignment. Dijkstra’s shortest-path algorithm is run on the grid graph from a user-specified root cluster to derive a monotonic pseudotime score for every grid node.

  (v)

  Bidirectional projection. Each cell inherits the coordinates, pseudotime and neighbourhood of its nearest grid node; conversely, grid nodes maintain links back to the original transcriptomic space.

Figure 1. The comprehensive architecture of the scRL framework for cellular fate decision analysis. (A) The preprocessing pipeline, which integrates dimensionality reduction, clustering, an edge detection-based grid embedding algorithm, subpopulation mapping, pseudotime computation via Dijkstra’s algorithm and bidirectional projection between embedding spaces. Gene expression data and clustering labels form the basis for constructing reward environments, which operate in two modes: a decision mode (reward diminishing along pseudotime) and a contribution mode (reward intensifying along pseudotime). (B) Core functional modules comprising a gene module (gene potential and decision values), a lineage module (lineage potential and decision values) and a trajectory module (differentiation trajectory analysis). (C) The actor–critic reinforcement learning architecture, implemented as distinct neural network architectures within scRL. The critic (value network) processes the latent space—which we validated to be optimally represented by LDA for robust feature extraction—to output cell-state value, specifically the decision or contribution intensity. Concurrently, the actor (policy network) learns and outputs the optimal differentiation trajectories within the grid environment.

Figure 1. The comprehensive architecture of the scRL framework for cellular fate decision analysis. (A) The preprocessing pipeline, which integrates dimensionality reduction, clustering, an edge detection-based grid embedding algorithm, subpopulation mapping, pseudotime computation via Dijkstra’s algorithm and bidirectional projection between embedding spaces. Gene expression data and clustering labels form the basis for constructing reward environments, which operate in two modes: a decision mode (reward diminishing along pseudotime) and a contribution mode (reward intensifying along pseudotime). (B) Core functional modules comprising a gene module (gene potential and decision values), a lineage module (lineage potential and decision values) and a trajectory module (differentiation trajectory analysis). (C) The actor–critic reinforcement learning architecture, implemented as distinct neural network architectures within scRL. The critic (value network) processes the latent space—which we validated to be optimally represented by LDA for robust feature extraction—to output cell-state value, specifically the decision or contribution intensity. Concurrently, the actor (policy network) learns and outputs the optimal differentiation trajectories within the grid environment.

• Stage 2: Reward design and module definition (Figure 1B)

Gene expression levels create two complementary reward landscapes:

• Decision mode—the reward decays exponentially with pseudotime, emphasizing early contribution signals.
• Contribution mode—the reward increases with pseudotime, capturing cumulative lineage output.

These landscapes feed the following three functional modules:

• Gene module calculates gene-level potential and decision intensity.
• Lineage module outputs lineage potential and decision intensity.
• Trajectory module simulates differentiation by rolling out the learned policy on the grid.

• Definition of Decision and Contribution Intensity

• Decision Intensity quantifies the “fate intensity” or “decision value” of a cell state, particularly focusing on its significance in shaping future fate decisions before overt lineage commitment. It is derived from the state value learned by the critic when operating in the “decision mode,” where rewards are structured to emphasize early signals by decaying exponentially along pseudotime. Higher decision intensity therefore indicates a cell state with greater potential to influence downstream lineage choices, capturing the “pre-expression decision states” where crucial developmental commitments are made.
• Contribution Intensity quantifies the cumulative “lineage output” or “lineage contribution” from a given cell state over time. It is derived from the state value learned by the critic when operating in the “contribution mode,” where rewards are structured to emphasize cumulative output by increasing along pseudotime. Higher contribution intensity reflects a cell’s accumulated propensity or success in differentiating towards and populating a specific lineage over the course of its developmental trajectory.

• Stage 3: Actor–critic learning (Figure 1C)

• Critic (value network). A multilayer perceptron ingests latent coordinates and predicts the state value, $V\left(s\right)$, for each grid node.
• Actor (policy network). A parallel network outputs a stochastic policy, $\pi \left(a\right|s)$, that favors moves increasing expected cumulative reward.
• Optimization. Networks are updated with an advantage formulation and an adaptive learning rate until both cumulative reward and maximal $V\left(s\right)$ converge.

• Validation workflow (Figure A1)

We benchmarked scRL on human ${\mathrm{CD34}}^{+}$ haematopoietic cells (S-SUBS8) and mouse endocrinogenesis (GSE132188). Lineage-specific markers—GATA1, IRF8, EBF1 for hematopoiesis; Ngn3, Fev for endocrinogenesis—served as external validators of pre-expression decision states. Cluster-specific rewards were assigned to erythroid, myeloid and lymphoid (hematopoiesis) or early/late endocrine trajectories. After training

(a)

State values were back-projected onto the UMAP to visualize decision hotspots;

(b)

Training dynamics showed smooth convergence of the cumulative reward and stabilization of the maximum state value, confirming learning efficacy;

(c)

Tabular Q-learning on discretized state spaces reproduced the neural results, validating the actor–critic implementation.

2.2. Grid-Based Embedding Representation

To transform two-dimensional cell embeddings into a structured grid representation that preserves topological relationships while enabling systematic trajectory analysis, we developed a grid mapping algorithm that discretizes the continuous embedding space into a regular lattice structure.

Grid Generation: Given a two-dimensional embedding matrix, $\mathbf{X}\in {\mathbb{R}}^{m\times 2}$, representing m cells, we first identify the spatial boundaries:

$$\begin{array}{cc}\hfill {\mathbf{x}}_{\mathrm{right}}& =\mathbf{X}[arg\underset{i}{max}{X}_{i,1},:],{\mathbf{x}}_{\mathrm{left}}=\mathbf{X}[arg\underset{i}{min}{X}_{i,1},:]\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathbf{x}}_{\mathrm{top}}& =\mathbf{X}[arg\underset{i}{max}{X}_{i,2},:],{\mathbf{x}}_{\mathrm{bottom}}=\mathbf{X}[arg\underset{i}{min}{X}_{i,2},:]\hfill \end{array}$$

(2)

We then generate an $n\times n$ regular grid spanning the embedding space:

$$\begin{array}{cc}\hfill x_k& =x_{\mathrm{left}}+{\displaystyle \frac{k-1}{n-1}}(x_{\mathrm{right}}-x_{\mathrm{left}}),k=1,2,\dots ,n\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill y_l& =y_{\mathrm{bottom}}+{\displaystyle \frac{l-1}{n-1}}(y_{\mathrm{top}}-y_{\mathrm{bottom}}),l=1,2,\dots ,n\hfill \end{array}$$

(4)

The grid points are defined as $\mathbf{G}=\{(x_k,y_l):k,l=1,2,\dots ,n\}$, yielding $n^2$ total grid positions.

Boundary Detection: To identify the data boundary, we compute angular distances from spine points to all cells. For each spine point ${\mathbf{s}}_i$, we calculate:

$$\begin{array}{cc}\hfill {\theta }_{i,j}& =arctan\left({\displaystyle \frac{s_{i,x}-X_{j,x}+ϵ}{s_{i,y}-X_{j,y}+ϵ}}\right)\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill d_{i,j}& =\left|\right|{\mathbf{s}}_i-{\mathbf{X}}_j{\left|\right|}_2\hfill \end{array}$$

(6)

where $ϵ={10}^{-6}$ prevents division by zero. Boundary cells are identified as:

$${\mathcal{B}}_i=\{arg\underset{j:{\theta }_{i,j}=\theta }{max}{d}_{i,j}:\theta \in {\Theta }_i\}$$

(7)

where ${\Theta }_i$ represents the set of unique angular values from spine point ${\mathbf{s}}_i$.

Grid Masking: We apply a masking procedure to remove grid points in regions with insufficient cell density. Using j observer points, we compute angular distances and identify valid grid regions:

$${\mathcal{M}}_i=\{g\in \mathbf{G}:\mathrm{rank}\left(d_{i,g}\right)>max\left(\mathrm{rank}\left(d_{i,b}\right)\right)+j,\forall b\in \mathcal{B}\}$$

(8)

where $\mathrm{rank}\left(d_{i,g}\right)$ denotes the distance rank of grid point g from observer i.

Adjacency Construction: For the remaining mapped grid points, ${\mathcal{G}}_{\mathrm{mapped}}$, we construct an 8-connectivity adjacency matrix, $\mathbf{A}$:

$$A_{g_1,g_2}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\left|\right|pos\left(g_1\right)-pos\left(g_2\right){\left|\right|}_{\infty }=1\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(9)

where $pos\left(g\right)$ returns the $(i,j)$ grid coordinates of grid point g.

Boundary Refinement: Grid boundary points are identified as those with fewer than 8 neighbors:

$${\mathcal{G}}_{\mathrm{boundary}}=\{g\in {\mathcal{G}}_{\mathrm{mapped}}:\sum _{g^{\prime }}{A}_{g,g^{\prime }}<8\}$$

(10)

A pruning step ensures graph connectivity by iteratively adding adjacent points to isolated boundary regions, maintaining a single connected component in the final grid graph $\mathcal{G}=({\mathcal{G}}_{\mathrm{mapped}},\mathbf{A})$.

2.3. Pseudotime Alignment via Graph-Based Distance Computation

To establish a coherent temporal ordering across the grid representation, we developed a pseudotime alignment algorithm that leverages graph-based shortest path computation to propagate temporal information from user-defined starting points throughout the grid network.

Starting Point Selection: The algorithm supports two initialization modes. For single-cell initialization with a specified early cell, $c_0$, we identify the corresponding grid point:  

$$g_{\mathrm{start}}=arg\underset{g\in {\mathcal{G}}_{\mathrm{mapped}}}{min}\left|\right|{\mathbf{X}}_{c_0}-{\mathbf{G}}_g{\left|\right|}_2$$

(11)

For cluster-based initialization with an early cluster, ${\mathcal{C}}_{\mathrm{early}}$, we sample $n_s$ starting points from the intersection of cluster and boundary grids:

$$\mathcal{S}={\mathrm{sample}}_{n_s}(\{g\in {\mathcal{G}}_{\mathrm{mapped}}:\mathrm{cluster}\left(g\right)={\mathcal{C}}_{\mathrm{early}}\}\cap {\mathcal{G}}_{\mathrm{boundary}})$$

(12)

where ${\mathcal{G}}_{\mathrm{boundary}}$ represents boundary grid points if the boundary constraint is enabled.

Graph Construction: We construct an undirected graph, $\mathcal{H}=(\mathcal{V},\mathcal{E})$, from the grid adjacency matrix:

$$\mathcal{V}={\mathcal{G}}_{\mathrm{mapped}},\mathcal{E}=\{(g_i,g_j):A_{g_i,g_j}=1\}$$

(13)

where $\mathbf{A}$ is the 8-connectivity adjacency matrix from the grid construction phase.

Single-Component Pseudotime Calculation: For a connected graph, we apply Dijkstra’s algorithm from each starting point, $s\in \mathcal{S}$:

$$d_s\left(g\right)=\mathrm{Dijkstra}(\mathcal{H},s,g)\forall g\in \mathcal{V}$$

(14)

The mean pseudotime across all starting points is computed as:

$$\overline{d}\left(g\right)={\displaystyle \frac{1}{\left|\mathcal{S}\right|}}\sum _{s\in \mathcal{S}}{d}_s\left(g\right)$$

(15)

Multi-Component Handling: When the graph contains multiple connected components $\{{\mathcal{C}}_1,{\mathcal{C}}_2,\dots ,{\mathcal{C}}_k\}$, we identify the main component containing the majority of starting points:

$${\mathcal{C}}_{\mathrm{main}}=arg\underset{i}{max}|\mathcal{S}\cap {\mathcal{C}}_i|$$

(16)

For each secondary component, ${\mathcal{C}}_j$, where $j\ne \mathrm{main}$, we establish connection through the closest grid pair:

$$(g_{\mathrm{main}},g_j)=arg\underset{g_m\in {\mathcal{C}}_{\mathrm{main}},g_s\in {\mathcal{C}}_j}{min}\left|\right|{\mathbf{G}}_{g_m}-{\mathbf{G}}_{g_s}{\left|\right|}_2$$

(17)

The pseudotime offset for component ${\mathcal{C}}_j$ is determined by:

$${\tau }_j=\overline{d}\left(g_{\mathrm{main}}\right)$$

(18)

Pseudotime within secondary component ${\mathcal{C}}_j$ is calculated as:

$$d_j\left(g\right)=\mathrm{Dijkstra}(\mathcal{H}{|}_{{\mathcal{C}}_j},g_j,g)+{\tau }_j\forall g\in {\mathcal{C}}_j$$

(19)

Normalization: The final pseudotime values are normalized to the unit interval:

$$\psi \left(g\right)={\displaystyle \frac{d\left(g\right)-{min}_{g^{\prime }\in \mathcal{V}}d\left(g^{\prime }\right)}{{max}_{g^{\prime }\in \mathcal{V}}d\left(g^{\prime }\right)-{min}_{g^{\prime }\in \mathcal{V}}d\left(g^{\prime }\right)}}$$

(20)

where $d\left(g\right)$ represents the unified distance function across all components:

$$d\left(g\right)=\left\{\begin{array}{cc}\overline{d}\left(g\right)\hfill & \mathrm{if}g\in {\mathcal{C}}_{\mathrm{main}}\hfill \\ d_j\left(g\right)\hfill & \mathrm{if}g\in {\mathcal{C}}_j,j\ne \mathrm{main}\hfill \end{array}\right.$$

(21)

2.4. Bidirectional Projection Between Cells and Grid

To establish correspondence between the original cell embedding and the grid representation, we developed bidirectional projection methods that enable transfer of annotations and continuous variables between these two spaces using distance-based Gaussian kernel weighting.

Cluster Annotation Projection: Given cell cluster annotations $\mathbf{c}=\{c_1,c_2,\dots ,c_m\}$, where $c_i$ represents the cluster label for cell i, we project these discrete labels onto grid points using nearest neighbor assignment:

$$\mathrm{cluster}\left(g\right)=c_{j^{*}}\mathrm{where}{j}^{*}=arg\underset{j\in \{1,\dots ,m\}}{min}\left|\right|{\mathbf{G}}_g-{\mathbf{X}}_j{\left|\right|}_2$$

(22)

The cluster assignment for each mapped grid point, $g\in {\mathcal{G}}_{\mathrm{mapped}}$, is determined by the cluster label of its closest cell in the embedding space.

Grid-to-Cell Projection: For projecting continuous values from grid space back to cells, we employ a Gaussian kernel-weighted interpolation scheme. For each cell i, we identify the k nearest grid points:

$${\mathcal{N}}_k\left(i\right)=\{g_1,g_2,\dots ,g_k\}\mathrm{where}\left|\right|{\mathbf{X}}_i-{\mathbf{G}}_{g_j}{\left|\right|}_2\le \left|\right|{\mathbf{X}}_i-{\mathbf{G}}_{g_{j+1}}{\left|\right|}_2$$

(23)

The local distance variance is computed as:

$${\sigma }_i^2=\mathrm{Var}\left(\right\{\left|\right|{\mathbf{X}}_i-{\mathbf{G}}_g{\left|\right|}_2:g\in {\mathcal{N}}_k\left(i\right)\left\}\right)$$

(24)

Gaussian weights are calculated for each neighboring grid point:

$$w_{i,g}={\displaystyle \frac{exp\left(-\frac{\left|\right|{\mathbf{X}}_i-{\mathbf{G}}_g{\left|\right|}_2^2}{2{\sigma }_i^2}\right)}{{\sum }_{g^{\prime }\in {\mathcal{N}}_k\left(i\right)}exp\left(-\frac{\left|\right|{\mathbf{X}}_i-{\mathbf{G}}_{g^{\prime }}{\left|\right|}_2^2}{2{\sigma }_i^2}\right)}}$$

(25)

The projected value at cell i is computed as:

$$v_i=\sum _{g\in {\mathcal{N}}_k\left(i\right)}{w}_{i,g}·V\left(g\right)$$

(26)

where $V\left(g\right)$ represents the grid value at position g. For optional annotation weighting with factor $w_{\mathrm{ann}}$:

$$v_i^{\mathrm{weighted}}=v_i·log\left(w_{\mathrm{ann},i}\right)$$

(27)

Cell-to-Grid Projection: For projecting cellular data onto grid points, we reverse the projection direction. For each grid point, g, we identify the k nearest cells:

$${\mathcal{M}}_k\left(g\right)=\{i_1,i_2,\dots ,i_k\}\mathrm{where}\left|\right|{\mathbf{G}}_g-{\mathbf{X}}_{i_j}{\left|\right|}_2\le \left|\right|{\mathbf{G}}_g-{\mathbf{X}}_{i_{j+1}}{\left|\right|}_2$$

(28)

The local variance for grid point g is:

$${\sigma }_g^2=\mathrm{Var}\left(\right\{\left|\right|{\mathbf{G}}_g-{\mathbf{X}}_i{\left|\right|}_2:i\in {\mathcal{M}}_k\left(g\right)\left\}\right)$$

(29)

Gaussian weights for neighboring cells are:

$$w_{g,i}={\displaystyle \frac{exp\left(-\frac{\left|\right|{\mathbf{G}}_g-{\mathbf{X}}_i{\left|\right|}_2^2}{2{\sigma }_g^2}\right)}{{\sum }_{i^{\prime }\in {\mathcal{M}}_k\left(g\right)}exp\left(-\frac{\left|\right|{\mathbf{G}}_g-{\mathbf{X}}_{i^{\prime }}{\left|\right|}_2^2}{2{\sigma }_g^2}\right)}}$$

(30)

The projected data value at grid point g for feature f is:

$$D_{g,f}=\sum _{i\in {\mathcal{M}}_k\left(g\right)}{w}_{g,i}·D_{i,f}$$

(31)

where $D_{i,f}$ represents the data value for feature f in cell i.

Normalization: For non-negative projections, final values are normalized to the unit interval:

$${\tilde{v}}_i={\displaystyle \frac{v_i-{min}_j{v}_j}{{max}_j{v}_j-{min}_j{v}_j}}$$

(32)

2.5. Reinforcement Learning Environment Configuration

To model cellular fate decisions as a sequential decision-making process, we designed a reinforcement learning environment that transforms the grid representation into a MDP (Markov Decision Process), where agents learn optimal trajectories through reward signals derived from biological annotations.

State Space Construction: For each grid point $g\in {\mathcal{G}}_{\mathrm{mapped}}$, we construct a state representation using the k-nearest neighbor cells in the embedding space. Given latent features $\mathbf{Z}\in {\mathbb{R}}^{m\times d}$ from principal component analysis:

$${\mathcal{N}}_k\left(g\right)=\{i_1,i_2,\dots ,i_k\}\mathrm{where}\left|\right|{\mathbf{G}}_g-{\mathbf{X}}_{i_j}{\left|\right|}_2\le \left|\right|{\mathbf{G}}_g-{\mathbf{X}}_{i_{j+1}}{\left|\right|}_2$$

(33)

The state vector for grid point g is computed as:

$${\mathbf{s}}_g={\displaystyle \frac{1}{k}}\sum _{i\in {\mathcal{N}}_k\left(g\right)}{\mathbf{Z}}_i$$

(34)

Action Space Definition: The action space consists of 8 discrete directional movements corresponding to the 8-connectivity neighborhood:

$$\mathcal{A}=\{\mathrm{R},\mathrm{RT},\mathrm{T},\mathrm{LT},\mathrm{L},\mathrm{LB},\mathrm{B},\mathrm{RB}\}$$

(35)

For grid position $(i,j)$, the action mapping is defined as:

$$\varphi \left(a\right)=\left\{\begin{array}{cc}(i,j+1)\hfill & \mathrm{if}a=\mathrm{R}\hfill \\ (i+1,j+1)\hfill & \mathrm{if}a=\mathrm{RT}\hfill \\ (i+1,j)\hfill & \mathrm{if}a=\mathrm{T}\hfill \\ ⋮\hfill & \\ (i-1,j+1)\hfill & \mathrm{if}a=\mathrm{RB}\hfill \end{array}\right.$$

(36)

Discrete Reward Function: For lineage-specific trajectory learning, we define rewards based on cluster annotations and pseudotime progression. Given starting clusters ${\mathcal{C}}_{\mathrm{start}}$ and target clusters ${\mathcal{C}}_{\mathrm{end}}$:

$$R_d(s,a)=\left\{\begin{array}{cc}-1\hfill & \mathrm{if}\mathrm{transition}\mathrm{leads}\mathrm{to}\mathrm{masked}\mathrm{grid}\hfill \\ exp(-\beta ·{\widehat{t}}_{s^{\prime }})+\Delta t\hfill & \mathrm{if}{s}^{\prime }\in {\mathcal{C}}_{\mathrm{end}}(\mathrm{Decision}\mathrm{mode})\hfill \\ 1-exp(-\beta ·{\widehat{t}}_{s^{\prime }})+\Delta t\hfill & \mathrm{if}{s}^{\prime }\in {\mathcal{C}}_{\mathrm{end}}(\mathrm{Contribution}\mathrm{mode})\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(37)

where ${\widehat{t}}_{s^{\prime }}$ is the normalized pseudotime at next state $s^{\prime }$, $\beta$ is the decay coefficient and $\Delta t=\psi \left(s^{\prime }\right)-\psi \left(s\right)$ represents pseudotime progression.

Continuous Reward Function: For gene expression-guided learning, rewards are computed using projected continuous values:

$$\overline{r}\left(g\right)={\displaystyle \frac{1}{|{\mathcal{K}}_{\mathrm{reward}}|}}\sum _{k\in {\mathcal{K}}_{\mathrm{reward}}}{D}_{g,k}$$

(38)

where ${\mathcal{K}}_{\mathrm{reward}}$ represents reward gene sets and $D_{g,k}$ is the projected expression of gene k at grid g. The continuous reward function is:

$$R_c(s,a)=\left\{\begin{array}{cc}-1\hfill & \mathrm{if}\mathrm{transition}\mathrm{leads}\mathrm{to}\mathrm{masked}\mathrm{grid}\hfill \\ w_{\mathrm{reward}}·\overline{r}\left(s^{\prime }\right)-w_{\mathrm{punish}}·\overline{p}\left(s^{\prime }\right)\hfill & \mathrm{if}{s}^{\prime }\mathrm{is}\mathrm{valid}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(39)

where the weighting factors are defined as:

$$w_{\mathrm{reward}}=\left\{\begin{array}{cc}exp(-\beta ·{\widehat{t}}_{s^{\prime }})\hfill & \mathrm{Decision}\mathrm{mode}\hfill \\ 1-exp(-\beta ·{\widehat{t}}_{s^{\prime }})\hfill & \mathrm{Contribution}\mathrm{mode}\hfill \end{array}\right.$$

(40)

and $\overline{p}\left(s^{\prime }\right)$ represents the mean expression of punishment genes.

Episode Termination: Episodes terminate under the following conditions:

$$\mathrm{terminate}=\left\{\begin{array}{cc}\mathrm{True}\hfill & \mathrm{if}{s}^{\prime }\in {\mathcal{G}}_{\mathrm{boundary}}\setminus {\mathcal{G}}_{\mathrm{start}}\hfill \\ \mathrm{True}\hfill & \mathrm{if}{s}^{\prime }\in \mathrm{trajectory}\mathrm{history}\hfill \\ \mathrm{True}\hfill & \mathrm{if}R(s,a)=-1\hfill \\ \mathrm{False}\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(41)

with trajectory truncation after $T_{max}$ steps.

Experience Replay: Training experiences are stored in a circular buffer, $\mathcal{B}$, with capacity $N_{\mathrm{buffer}}$:

$$\mathcal{B}={\left\{({\mathbf{s}}_t,a_t,r_t,{\mathbf{s}}_{t+1},d_t)\right\}}_{t=1}^{\left|\mathcal{B}\right|}$$

(42)

where $d_t\in \{0,1\}$ indicates episode termination. During training, mini-batches of size B are uniformly sampled from $\mathcal{B}$ for gradient updates.

2.6. Reinforcement Learning Model Architecture and Training

To learn optimal cellular fate decision policies, we implemented three distinct reinforcement learning algorithms that leverage different approximation strategies and learning paradigms for trajectory optimization in the grid-based cellular environment.

Tabular Q-Learning: For discrete state spaces, we employ tabular Q-learning with $ϵ$-greedy exploration. The Q-table is initialized as:

$$\mathbf{Q}\in {\mathbb{R}}^{|{\mathcal{G}}_{\mathrm{mapped}}|\times 8}$$

(43)

The exploration probability decays exponentially with training steps:

$$ϵ\left(t\right)={ϵ}_{min}+({ϵ}_{max}-{ϵ}_{min})exp\left(-{\displaystyle \frac{0.01·t}{1000}}\right)$$

(44)

where ${ϵ}_{max}=0.9$, ${ϵ}_{min}=0.01$ and t represents the current step. Action selection follows:

$$a_t=\left\{\begin{array}{cc}arg\underset{a}{max}Q(s_t,a)\hfill & \mathrm{with}\mathrm{probability}1-ϵ\left(t\right)\hfill \\ \mathrm{uniform}\mathrm{random}\hfill & \mathrm{with}\mathrm{probability}ϵ\left(t\right)\hfill \end{array}\right.$$

(45)

Q-value updates use temporal difference learning:

$$Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha \left[r_t+\gamma \underset{a^{\prime }}{max}Q(s_{t+1},a^{\prime })-Q(s_t,a_t)\right]$$

(46)

where $\alpha$ is the learning rate and $\gamma$ is the discount factor.

Actor–Critic Architecture: For continuous state representations, we implement an actor–critic framework with separate policy and value networks. The policy network outputs action probabilities:

$${\pi }_{\theta }\left(a\right|s)=\mathrm{Softmax}\left(1+5·\sigma \left({\mathbf{W}}_{\pi }^{\left(2\right)}\mathrm{ReLU}\left({\mathbf{W}}_{\pi }^{\left(1\right)}\mathbf{s}+{\mathbf{b}}_{\pi }^{\left(1\right)}\right)+{\mathbf{b}}_{\pi }^{\left(2\right)}\right)\right)$$

(47)

where $\sigma$ denotes the sigmoid activation and the scaling factor ensures exploration. The value network approximates state values:

$$V_{\varphi }\left(s\right)={\mathbf{W}}_v^{\left(2\right)}\mathrm{ReLU}\left({\mathbf{W}}_v^{\left(1\right)}\mathbf{s}+{\mathbf{b}}_v^{\left(1\right)}\right)+{\mathbf{b}}_v^{\left(2\right)}$$

(48)

The temporal difference error is computed as:

$${\delta }_t=r_t+\gamma V_{\varphi }\left(s_{t+1}\right)(1-d_t)-V_{\varphi }\left(s_t\right)$$

(49)

where $d_t\in \{0,1\}$ indicates episode termination. Policy updates incorporate entropy regularization:

$${\mathcal{L}}_{\pi }=-\mathbb{E}\left[log{\pi }_{\theta }\left(a_t\right|s_t)·{\delta }_t\right]+{\lambda }_{\mathrm{ent}}\mathbb{E}\left[\sum _a{\pi }_{\theta }\left(a\right|s_t)log{\pi }_{\theta }\left(a\right|s_t)\right]$$

(50)

where ${\lambda }_{\mathrm{ent}}=0.01$ promotes exploration. The value network loss is:

$${\mathcal{L}}_v=\mathbb{E}\left[{\left(V_{\varphi }\left(s_t\right)-\left(r_t+\gamma V_{\varphi }\left(s_{t+1}\right)(1-d_t)\right)\right)}^2\right]$$

(51)

DDQN (Double Deep Q-Network): To address overestimation bias in deep Q-learning, we employ a double DQN architecture with target network stabilization. The policy network estimates action values:

$$Q_{\theta }(s,a)={\mathbf{W}}_q^{\left(2\right)}\mathrm{ReLU}\left({\mathbf{W}}_q^{\left(1\right)}\mathbf{s}+{\mathbf{b}}_q^{\left(1\right)}\right)+{\mathbf{b}}_q^{\left(2\right)}$$

(52)

Action selection uses decaying $ϵ$-greedy with:

$$ϵ\left(t\right)={ϵ}_{\mathrm{end}}+({ϵ}_{\mathrm{start}}-{ϵ}_{\mathrm{end}})exp\left(-{\displaystyle \frac{0.1·t}{{\tau }_{\mathrm{decay}}}}\right)$$

(53)

The target value computation follows the double Q-learning update:

$$\begin{array}{cc}\hfill a^{*}& =arg\underset{a}{max}{Q}_{\theta }(s_{t+1},a)\hfill \end{array}$$

(54)

$$\begin{array}{cc}\hfill y_t& =r_t+\gamma Q_{{\theta }^{-}}(s_{t+1},a^{*})(1-d_t)\hfill \end{array}$$

(55)

where $Q_{{\theta }^{-}}$ denotes the target network. The loss function employs smooth L1 loss:

$${\mathcal{L}}_{\mathrm{DQN}}=\mathbb{E}\left[\mathrm{SmoothL}1\left(Q_{\theta }(s_t,a_t)-y_t\right)\right]$$

(56)

Target network updates use either soft updates:  

$${\theta }^{-}\leftarrow (1-\tau ){\theta }^{-}+\tau \theta$$

(57)

or hard updates every C steps: ${\theta }^{-}\leftarrow \theta$.

Training Dynamics: All algorithms track performance through exponential moving averages. For value-based methods, we monitor the maximum Q-value:

$${\overline{Q}}_{max}\left(t\right)=0.005·\underset{a}{max}Q(s_t,a)+0.995·{\overline{Q}}_{max}(t-1)$$

(58)

For actor–critic, we track the state value:

$$\overline{V}\left(t\right)=0.005·V_{\varphi }\left(s_t\right)+0.995·\overline{V}(t-1)$$

(59)

Episode returns are accumulated and averaged over sliding windows of 100 episodes to assess learning progress and convergence.

Experience Replay: For off-policy DDQN training, experiences are stored in a circular buffer with capacity $N_{\mathrm{buffer}}$ and sampled uniformly for mini-batch updates once the buffer contains at least $0.1·N_{\mathrm{buffer}}$ transitions.

2.7. scRL Parameter Configuration and Data Structure Organization

The scRL framework employs a hierarchical parameter configuration system and structured data organization to ensure computational efficiency and biological interpretability across diverse single-cell analysis scenarios.

Core Data Container Architecture: scRL organizes computational results through a structured container class with five primary data categories:

$${\mathcal{D}}_{\mathrm{scRL}}=\{\mathcal{E},\mathcal{G},\mathcal{Q},\mathcal{S},\mathcal{T}\}$$

(60)

where $\mathcal{E}$ represents embedding data, $\mathcal{G}$ contains grid information, $\mathcal{Q}$ stores reinforcement learning components, $\mathcal{S}$ holds simulation data and $\mathcal{T}$ maintains trajectory records.

Grid Generation Parameters: The fundamental grid configuration employs two primary parameters. Grid resolution parameter n determines spatial discretization:

$$|{\mathcal{G}}_{\mathrm{total}}|=n^2$$

(61)

Observer parameter j controls boundary detection sensitivity:

$${\mathcal{S}}_{\mathrm{observer}}=j\times \left|{\mathcal{B}}_{\mathrm{spine}}\right|$$

(62)

where ${\mathcal{B}}_{\mathrm{spine}}$ represents the boundary spine points. Default values are configured as $n_{\mathrm{default}}=50$ and $j_{\mathrm{default}}=3$.

Parallel Processing Configuration: Computational parallelization is controlled by the parameter:

$$n_{\mathrm{jobs}}\in \{1,2,\dots ,N_{\mathrm{CPU}}\}$$

(63)

where $N_{\mathrm{CPU}}$ represents available processor cores. The default setting is $n_{\mathrm{jobs}}=8$.

Cluster Projection Parameters: Cell type annotation projection employs cluster assignment vectors:

$$\mathcal{C}=\{c_1,c_2,\dots ,c_m\}\mathrm{where}{c}_i\in \mathbb{N}$$

(64)

Color mapping utilizes predefined palettes or automatic generation:

$${\mathcal{P}}_{\mathrm{color}}=\{p_1,p_2,\dots ,p_{\left|\mathcal{C}\right|}\}\mathrm{where}{p}_i\in {[0,1]}^3$$

(65)

Pseudotime Alignment Parameters: Temporal ordering configuration includes sampling parameters:

$$n_{\mathrm{sample}}\in \{1,2,\dots ,|{\mathcal{C}}_{\mathrm{early}}\left|\right\}$$

(66)

Boundary restriction flag:

$$b_{\mathrm{boundary}}\in \{0,1\}$$

(67)

Key identifier for storage:

$$k_{\mathrm{add}}\in \left\{\mathrm{strings}\right\}$$

(68)

where the default configuration sets $n_{\mathrm{sample}}=10$, $b_{\mathrm{boundary}}=1$ and $k_{\mathrm{add}}=\text{‘}\mathrm{pseudotime}\text{’}$.

Projection Parameters: Bidirectional projection between cells and grids employs neighborhood parameters:

$$k_{\mathrm{neighbors}}\in \{1,2,\dots ,|{\mathcal{G}}_{\mathrm{mapped}}\left|\right\}$$

(69)

Weighting factors for optional annotation scaling:

$$w_{\mathrm{annotation}}\in {\mathbb{R}}^{+}$$

(70)

Normalization flags:

$$f_{\mathrm{negative}}\in \{0,1\}$$

(71)

Default settings utilize $k_{\mathrm{neighbors}}=15$.

Data Structure Hierarchy: The framework maintains structured data organization:

$$\begin{array}{cc}\hfill \mathcal{E}& =\{{\mathbf{X}}_{\mathrm{embedding}},{\mathcal{C}}_{\mathrm{clusters}},{\mathcal{P}}_{\mathrm{colors}},{\mathbf{D}}_{\mathrm{data}}\}\hfill \end{array}$$

(72)

$$\begin{array}{cc}\hfill \mathcal{G}& =\{n,{\mathbf{G}}_{\mathrm{grids}},{\mathcal{M}}_{\mathrm{masked}},{\mathcal{M}}_{\mathrm{mapped}},{\mathbf{A}}_{\mathrm{adjacency}}\}\hfill \end{array}$$

(73)

$$\begin{array}{cc}\hfill \mathcal{Q}& =\{{\mathbf{R}}_{\mathrm{rewards}},{\mathbf{M}}_{\mathrm{matrix}},k_{\mathrm{reward}}\}\hfill \end{array}$$

(74)

2.8. Embedding Techniques and Evaluation Framework

To transform high-dimensional single-cell data into interpretable representations and assess the quality of our grid-based trajectory inference, we employ multiple dimensionality reduction techniques and comprehensive evaluation metrics that capture both biological relevance and computational performance.

Principal Component Analysis: Given an $n\times m$ gene expression matrix, $\mathbf{X}$, with n cells and m genes, PCA identifies orthogonal axes that maximize variance:

$$\mathbf{Z}=\mathbf{X}\mathbf{W}$$

(75)

where $\mathbf{W}\in {\mathbb{R}}^{m\times k}$ contains the principal component weights and $\mathbf{Z}\in {\mathbb{R}}^{n\times k}$ represents cell embeddings in the reduced space. The columns of $\mathbf{W}$ are eigenvectors of the covariance matrix ${\mathbf{X}}^T\mathbf{X}$ corresponding to the k largest eigenvalues ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_k$. The proportion of variance explained by the i-th component is:

$${\mathrm{var}}_i={\displaystyle \frac{{\lambda }_i}{{\sum }_{j=1}^m{\lambda }_j}}$$

(76)

Nonlinear Embedding Methods: t-SNE constructs probability distributions over cell pairs in high-dimensional and low-dimensional spaces:  

$$\begin{array}{cc}\hfill p_{ij}& ={\displaystyle \frac{p_{j|i}+p_{i|j}}{2n}},p_{j|i}={\displaystyle \frac{exp(-||{\mathbf{z}}_i-{\mathbf{z}}_j|{|}^2/2{\sigma }_i^2)}{{\sum }_{k\ne i}exp(-||{\mathbf{z}}_i-{\mathbf{z}}_k{\left|\right|}^2/2{\sigma }_i^2)}}\hfill \end{array}$$

(77)

$$\begin{array}{cc}\hfill q_{ij}& ={\displaystyle \frac{(1+||{\mathbf{y}}_i-{\mathbf{y}}_j{\left|\right|}^2{)}^{-1}}{{\sum }_{k\ne l}(1+||{\mathbf{y}}_k-{\mathbf{y}}_l{\left|\right|}^2{)}^{-1}}}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hfill \end{array}$$

(78)

The embedding minimizes the Kullback–Leibler divergence: $C\left(\mathbf{Y}\right)={\sum }_{i\ne j}{p}_{ij}log{\displaystyle \frac{p_{ij}}{q_{ij}}}$.

UMAP constructs a fuzzy topological representation through weighted graphs with connection weights:

$$\begin{array}{cc}\hfill w_{ij}& =exp\left(-{\displaystyle \frac{d({\mathbf{x}}_i,{\mathbf{x}}_j)-{\rho }_i}{{\sigma }_i}}\right)\hfill \end{array}$$

(79)

$$\begin{array}{cc}\hfill v_{ij}& =(1+a·||{\mathbf{y}}_i-{\mathbf{y}}_j{\left|\right|}^{2b}{)}^{-1}\hfill \end{array}$$

(80)

The optimization objective combines attraction and repulsion terms:

$$\sum _{i,j}{w}_{ij}log{\displaystyle \frac{w_{ij}}{v_{ij}}}+(1-w_{ij})log{\displaystyle \frac{1-w_{ij}}{1-v_{ij}}}$$

(81)

Diffusion Maps: Starting with similarity matrix $\mathbf{W}$, where $W_{ij}=exp(-||{\mathbf{x}}_i-{\mathbf{x}}_j{\left|\right|}^2/ϵ)$, we compute the normalized transition matrix:

$$\tilde{\mathbf{P}}={\mathbf{D}}^{-1/2}\mathbf{W}{\mathbf{D}}^{-1/2}$$

(82)

where $\mathbf{D}$ is diagonal with $D_{ii}={\sum }_j{W}_{ij}$. Diffusion coordinates are defined by eigenvectors ${\mathsf{\varphi }}_k$ of $\tilde{\mathbf{P}}$:

$${\mathbf{\Phi }}_t\left({\mathbf{x}}_i\right)=({\lambda }_1^t{\varphi }_1\left(i\right),{\lambda }_2^t{\varphi }_2\left(i\right),\dots ,{\lambda }_k^t{\varphi }_k\left(i\right))$$

(83)

scVI (single-cell Variational Inference): scVI employs variational autoencoders to model single-cell gene expression through a hierarchical generative process. The latent representation $z_n\sim \mathcal{N}(0,\mathbf{I})$ captures cellular states, while library size factors $l_n\sim \mathrm{LogNormal}({\mu }_l,{\sigma }_l^2)$ account for technical variation. Gene expression rates are modeled through neural networks:

$${\rho }_{ng}=f_h^{\left(1\right)}{(z_n,s_n)}_g\mathrm{and}{\alpha }_{ng}=f_h^{\left(2\right)}{(z_n,s_n)}_g$$

(84)

The observation model follows a zero-inflated negative binomial distribution:

$$x_{ng}\sim \mathrm{ZINB}({\mu }_{ng},{\theta }_g,{\pi }_{ng})$$

(85)

where ${\mu }_{ng}=l_n{\rho }_{ng}$ and zero-inflation probability ${\pi }_{ng}=f_h^{\left(3\right)}{(z_n,s_n)}_g$. The variational approximation $q_{\varphi }\left(z_n\right|x_n,s_n)=\mathcal{N}({\mu }_n^{\varphi },{\sigma }_n^{\varphi })$ optimizes the evidence lower bound:

$$\mathcal{L}(\varphi ,h)={\mathbb{E}}_{q_{\varphi }}[log{p}_h\left(x_n\right|z_n,s_n)]-\mathrm{KL}[q_{\varphi }\left(z_n\right|x_n,s_n)\parallel p\left(z_n\right)]$$

(86)

Linear-scVI Extensions: Linear-scVI enhances interpretability by imposing linear structure on the latent space:

$$z_n=\mathbf{A}{c}_n+{ϵ}_n$$

(87)

where $c_n\sim \mathcal{N}(0,{\mathbf{I}}_k)$ represents interpretable linear factors and $\mathbf{A}\in {\mathbb{R}}^{d\times k}$ provides the transformation matrix. Cellular trajectories are parameterized through time-dependent dynamics:

$$c_n\left(t\right)=\mathbf{B}{t}_n+{\nu }_n$$

(88)

where $\mathbf{B}$ captures trajectory directions. The decoder employs linear combinations:

$$log{\rho }_{ng}={\mathbf{W}}_g^T{z}_n+b_g+log{l}_n$$

(89)

The modified ELBO incorporates linear constraints:

$${\mathcal{L}}_{\mathrm{linear}}={\mathbb{E}}_{q_{\varphi }}[logp\left(x_n\right|z_n)]-\mathrm{KL}[q_{\varphi }\left(c_n\right|x_n)\parallel p\left(c_n\right)]-\mathrm{KL}[q_{\varphi }\left(z_n\right|c_n,x_n)\parallel p\left(z_n\right|c_n)]$$

(90)

Latent Dirichlet Allocation: LDA identifies latent gene expression programs through topic modeling. The generative process defines gene expression programs ${\mathsf{\beta }}_k\sim \mathrm{Dirichlet}\left(\eta \right)$ and cell-specific topic proportions ${\mathsf{\theta }}_n\sim \mathrm{Dirichlet}\left(\alpha \right)$. Gene assignments follow $z_{ng}\sim \mathrm{Categorical}\left({\mathsf{\theta }}_n\right)$ and expression levels $w_{ng}\sim \mathrm{Categorical}\left({\mathsf{\beta }}_{z_{ng}}\right)$. Variational inference employs coordinate ascent updates:

$$\begin{array}{cc}\hfill {\varphi }_{ngk}& \propto exp\left\{\Psi \left({\gamma }_{nk}\right)+\Psi \left({\lambda }_{k{w}_{ng}}\right)-\Psi \left(\sum _v{\lambda }_{kv}\right)\right\}\hfill \end{array}$$

(91)

$$\begin{array}{cc}\hfill {\gamma }_{nk}& ={\alpha }_k+\sum _{g=1}^{G_n}{\varphi }_{ngk}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hfill \end{array}$$

(92)

$$\begin{array}{cc}\hfill {\lambda }_{kv}& ={\eta }_v+\sum _{n=1}^N\sum _{g:w_{ng}=v}{\varphi }_{ngk}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hfill \end{array}$$

(93)

External Evaluation Metrics: ARI corrects for random clustering effects:

$$\mathrm{ARI}={\displaystyle \frac{{\sum }_{ij}\left(\genfrac{}{}{0pt}{}{n_{ij}}{2}\right)-\left[{\sum }_i\left(\genfrac{}{}{0pt}{}{a_i}{2}\right){\sum }_j\left(\genfrac{}{}{0pt}{}{b_j}{2}\right)\right]/\left(\genfrac{}{}{0pt}{}{n}{2}\right)}{\frac{1}{2}\left[{\sum }_i\left(\genfrac{}{}{0pt}{}{a_i}{2}\right)+{\sum }_j\left(\genfrac{}{}{0pt}{}{b_j}{2}\right)\right]-\left[{\sum }_i\left(\genfrac{}{}{0pt}{}{a_i}{2}\right){\sum }_j\left(\genfrac{}{}{0pt}{}{b_j}{2}\right)\right]/\left(\genfrac{}{}{0pt}{}{n}{2}\right)}}$$

(94)

where $n_{ij}$ represents cells belonging to both true class i and predicted class j. NMI quantifies information-theoretic consistency:

$$\mathrm{NMI}(U,V)={\displaystyle \frac{2\times I(U,V)}{H\left(U\right)+H\left(V\right)}}$$

(95)

Internal Evaluation Metrics: ASW measures within-cluster cohesion versus between-cluster separation:

$$s\left(i\right)={\displaystyle \frac{b\left(i\right)-a\left(i\right)}{max\left\{a\right(i),b(i\left)\right\}}},\mathrm{ASW}={\displaystyle \frac{1}{n}}\sum _{i=1}^{n}s\left(i\right)$$

(96)

CH evaluates the ratio of between-cluster to within-cluster dispersion:

$$\mathrm{CH}={\displaystyle \frac{B\left(k\right)/(k-1)}{W\left(k\right)/(n-k)}}$$

(97)

DB assesses intra-cluster scatter relative to inter-cluster distances:

$$\mathrm{DB}={\displaystyle \frac{1}{k}}\sum _{i=1}^k\underset{j\ne i}{max}\left({\displaystyle \frac{S_i+S_j}{d_{ij}}}\right)$$

(98)

Comprehensive Scoring: To enable fair comparison across metrics with different scales and optimization directions, we apply min–max normalization:

$$X_{\mathrm{scaled}}={\displaystyle \frac{X-X_{min}}{X_{max}-X_{min}}}$$

(99)

For metrics requiring minimization (DB), we use $X_{\mathrm{scaled}}={\displaystyle \frac{X_{max}-X}{X_{max}-X_{min}}}$.

We define two key metrics to quantitatively assess the alignment of cell lineage intensity with learned dimensionality reduction embeddings.

MAS (Manifold Alignment Score): Given a lineage intensity vector, L, and the two-dimensional UMAP manifold represented by its orthogonal dimensions $U_1$ and $U_2$, the MAS is defined as the average of the absolute Pearson correlation coefficients between the lineage intensity vector and each UMAP dimension:

$$\mathrm{MAS}={\displaystyle \frac{1}{2}}\left(|\rho (L,U_1)|+|\rho (L,U_2)|\right)$$

(100)

where $\rho (X,Y)$ denotes the Pearson correlation coefficient between variables X and Y.

PAAC (Principal Axis Alignment Coefficient): Given a lineage intensity vector, L, and the first two principal components obtained from PCA, denoted as $P_1$ and $P_2$, the PAAC is defined as the average of the absolute Pearson correlation coefficients between the lineage intensity vector and these two principal components:

$$\mathrm{PAAC}={\displaystyle \frac{1}{2}}\left(|\rho (L,P_1)|+|\rho (L,P_2)|\right)$$

(101)

Similarly, $\rho (X,Y)$ denotes the Pearson correlation coefficient.

2.9. Comprehensive Validation Framework for scRL

To establish scRL’s robustness across diverse biological contexts, we designed a systematic validation strategy encompassing five complementary analytical dimensions, each addressing critical aspects of cellular fate decision modeling through carefully selected single-cell datasets.

Benchmark Performance Validation: We assessed scRL’s fundamental capabilities using human bone marrow ${\mathrm{CD34}}^{+}$ hematopoietic progenitor cells (5780 cells, S-SUBS8) and mouse endocrinogenesis datasets (2531 cells, GSE132188). These canonical differentiation systems enabled direct comparison with 8 existing methodologies across 5 unsupervised metrics, focusing on pseudotime ordering accuracy, lineage contribution strength quantification and fate decision intensity detection. For the hematopoietic dataset, we configured scRL with grid parameters ($n=50$, $j=3$), defining erythroid (Ery_1 [${\mathrm{Erythrocyte}}_1$], Ery_2 [${\mathrm{Erythrocyte}}_2$], Mega [Megakaryocyte]), myeloid (Mono_1 [${\mathrm{Monocyte}}_1$], Mono_2 [${\mathrm{Monocyte}}_2$], DCs [dendritic cells]), and lymphoid (CLP [Common Lymphoid Progenitor]) lineages as reward targets from HSC_1 (Hematopoietic Stem ${\mathrm{Cell}}_1$) starting points. The endocrinogenesis analysis targeted alpha, beta, delta, and epsilon cell fates from Ngn3 low/high and Fev⁺ progenitors, enabling systematic performance benchmarking across distinct developmental contexts.

Temporal Precedence Analysis: Using human phenotypical Hematopoietic Stem Cell datasets (GSE117498) containing HSC (Hematopoietic Stem Cell), MPP (Multipotent Progenitor), MLP (Multipotent Lymphoid Progenitor), CMP (Common Myeloid Progenitor), MEP (Megakaryocyte–Erythroid Progenitor), GMP (Granulocyte–Monocyte Progenitor), and PreBNK (Pre-B/NK Progenitor) populations, we validated whether scRL-identified fate decision states represent early specification events preceding observable lineage contributions. This analysis specifically examined the temporal relationship between computational fate predictions and experimentally characterized progenitor hierarchies. By targeting MEP and GMP fates from HSC origins, we established that scRL captures pre-contribution cellular states that systematically anticipate downstream differentiation events, confirming the framework’s predictive temporal resolution.

Multi-scale Decision Integration: The AML (acute myeloid leukemia) dataset (94,311 cells, GSE185993) provided a pathological context for validating coherence between molecular-level and cellular-level decision processes. This large-scale analysis demonstrated that fate decision intensity and gene decision intensity characterize equivalent cellular states, confirming scRL’s capacity to integrate multi-scale biological information. The dataset’s cellular heterogeneity and malignant transformation context enabled assessment of scRL’s robustness under conditions where normal developmental hierarchies are disrupted.

Perturbation Response Validation: We employed conditional Dapp1 knockout Hematopoietic Stem Cell datasets (10,224 cells, GSE277292) to validate the functional significance of scRL-identified dynamical genes through genetic perturbation analysis. Quality control criteria included total counts (1000–70,000), mitochondrial genes (<5%), ribosomal genes (10–50%) and expressed genes (>1000), followed by Harmony integration and Leiden clustering (resolution 0.6). This perturbation study confirmed that genetic alterations systematically modify fate decision landscapes, establishing biological relevance of scRL-captured decision intensities and validating the framework’s sensitivity to functional genetic modifications.

Pathological Condition Analysis: Radiation-induced hematopoietic injury datasets (41,252 cells, GSE278673) enabled validation of scRL’s performance under stress conditions that alter normal differentiation dynamics. Stringent quality control (total counts 5000–70,000, mitochondrial genes $<5\%$, ribosomal genes 10–50%, expressed genes 1500–7000) preceded Harmony integration and UMAP analysis (min_dist = 1.5). This pathological validation demonstrated scRL’s capacity to capture altered fate decision dynamics, differentiation biases and recovery processes, establishing the framework’s utility for analyzing cellular responses to environmental perturbations and pathological states.

3. Results

3.1. LDA Provides an Interpretable Latent Space for Single-Cell Lineage Analysis

We conducted a comprehensive comparison of dimensionality reduction methods including deep learning-based approaches (scVI, Linear scVI, and LDA) and traditional methods (PCA, ICA, FA, NMF, and Diff [Diffusion Map]) to identify techniques that effectively capture lineage relationships for investigating cell fate decision mechanisms. Using hematopoietic and pancreatic single-cell datasets, we systematically evaluated these approaches across varying numbers of highly variable genes (1000–5000) and latent dimensions (5–25) using five discriminative metrics: ARI, NMI, ASW, CH, and DB. LDA achieved the highest overall performance scores across all evaluation metrics (Figure 2A,B), with substantial improvements over baseline methods including scVI (+0.785 across HVG [highly variable gene] sizes, +0.713 across latent dimensions) and ICA (+0.675 and +0.618, respectively) (

Table 1. LDA performance improvements over baseline methods across HVG sizes.

Method	NMI	ARI	ASW	DB	CH	Overall
LDA vs. scVI	+0.267	+0.199	+0.334	+1.504	+2151.767	+0.785
LDA vs. ICA	+0.220	+0.149	+0.331	+1.938	+1857.164	+0.675
LDA vs. PCA	+0.064	+0.058	+0.303	+1.316	+1745.558	+0.511
LDA vs. LscVI	+0.081	+0.032	+0.246	+1.213	+1786.359	+0.459
LDA vs. FA	+0.012	−0.021	+0.190	+0.557	+1577.247	+0.261
LDA vs. Diff	−0.016	−0.045	+0.057	+0.585	+1568.978	+0.204
LDA vs. NMF	+0.007	+0.010	+0.159	+0.179	+375.229	+0.123

and

Table 2. LDA performance improvements over baseline methods across latent space sizes.

Method	NMI	ARI	ASW	DB	CH	Overall
LDA vs. scVI	+0.213	+0.106	+0.338	+1.516	+2643.360	+0.713
LDA vs. ICA	+0.127	+0.029	+0.334	+2.147	+2337.782	+0.618
LDA vs. LscVI	+0.075	+0.028	+0.326	+1.451	+2354.817	+0.560
LDA vs. PCA	−0.003	−0.056	+0.293	+1.576	+2273.306	+0.409
LDA vs. FA	−0.005	−0.084	+0.260	+0.686	+2383.781	+0.278
LDA vs. Diff	−0.042	−0.099	+0.124	+0.796	+2157.947	+0.198
LDA vs. NMF	−0.066	−0.117	+0.153	+0.536	+886.494	+0.075

). Individual latent component analysis showed that LDA exhibited strong subgroup specificity in intensity distributions within original cell type annotations (Figure 2C). Parallel analyses on the endocrinogenesis dataset confirmed these results, with LDA showing enhanced performance gains over scVI (+0.806 across HVG sizes, +0.839 across latent dimensions) and ICA (+0.789 and +0.765, respectively) (Figure A2,

Table A1. LDA performance improvements over baseline methods on endocrinogenesis dataset across HVG sizes.

Method	NMI	ARI	ASW	DB	CH	Overall
LDA vs. scVI	+0.258	+0.187	+0.379	+1.613	+1416.596	+0.806
LDA vs. ICA	+0.251	+0.157	+0.349	+1.869	+1411.359	+0.789
LDA vs. LscVI	+0.061	−0.006	+0.306	+1.254	+1061.209	+0.455
LDA vs. Diff	+0.012	+0.022	+0.165	+0.832	+1166.559	+0.347
LDA vs. PCA	+0.012	−0.060	+0.254	+0.971	+1022.558	+0.341
LDA vs. FA	−0.023	−0.100	+0.159	+0.145	+1085.451	+0.173
LDA vs. NMF	−0.074	−0.084	+0.117	+0.147	+744.455	+0.087

and

Table A2. LDA performance improvements over baseline methods on endocrinogenesis dataset across latent space sizes.

Method	NMI	ARI	ASW	DB	CH	Overall
LDA vs. scVI	+0.278	+0.186	+0.370	+1.579	+1506.128	+0.839
LDA vs. ICA	+0.189	+0.124	+0.337	+1.799	+1495.191	+0.765
LDA vs. LscVI	+0.099	+0.057	+0.329	+1.514	+1109.556	+0.605
LDA vs. Diff	+0.034	+0.028	+0.206	+0.906	+1323.790	+0.420
LDA vs. PCA	+0.027	-0.082	+0.266	+1.486	+1218.136	+0.410
LDA vs. FA	+0.023	-0.060	+0.200	+0.503	+1270.592	+0.313
LDA vs. NMF	+0.004	-0.054	+0.185	+0.428	+719.630	+0.236

).

Figure 2. LDA exhibits superior interpretability in comparative analysis of dimensionality reduction methods. (S-SUBS8) (A,B) Comparison of interpretability metrics including ARI, NMI, ASW, CH and DB for dimensionality reduction methods (scVI, ICA, PCA, LscVI, FA, Diff, NMF and LDA) evaluated across varying numbers of highly variable genes (1000, 2000, 3000, 4000, 5000) and latent space components (5, 10, 15, 20, 25). (C) Intensity distributions of latent components obtained by LDA across different subgroups, demonstrating subgroup specificity.

Figure 2. LDA exhibits superior interpretability in comparative analysis of dimensionality reduction methods. (S-SUBS8) (A,B) Comparison of interpretability metrics including ARI, NMI, ASW, CH and DB for dimensionality reduction methods (scVI, ICA, PCA, LscVI, FA, Diff, NMF and LDA) evaluated across varying numbers of highly variable genes (1000, 2000, 3000, 4000, 5000) and latent space components (5, 10, 15, 20, 25). (C) Intensity distributions of latent components obtained by LDA across different subgroups, demonstrating subgroup specificity.

3.2. Reinforcement Learning Captures Early Fate-Decision Signals

scRL constructs a grid embedding framework through edge detection-based algorithms, employing reinforcement learning to address the sequential decision-making problem inherent in cellular differentiation and quantify fate decision intensity at any developmental stage (Figure 3A). We utilized intensities from each LDA latent space component as lineage markers and projected them onto two-dimensional UMAP embeddings, revealing clear correspondence between these lineage intensities and distinct differentiation branches across both hematopoietic and pancreatic endocrinogenesis datasets (Figure 3B and Figure A3A). Critical to scRL’s innovation is its ability to identify pre-commitment cellular states: when projecting scRL-derived fate decision intensity onto developmental trajectories, peak decision regions consistently preceded observable lineage specification events, indicating that scRL captures regulatory checkpoints before overt lineage commitment becomes apparent. To quantitatively validate scRL’s superior interpretability, we compared its performance against percentile-based measures of LDA lineage intensities (95%, 90%, 85%, and 80% percentiles) using unsupervised metrics including ASW, CH, and DB (Figure 3C and Figure A3B). Quantitative analysis demonstrated scRL’s consistent superiority across both datasets, with substantial improvements of +0.986, +0.932, +0.735, and +0.375 over LDA at 80%, 85%, 90% and 95% percentiles, respectively, on the hematopoietic dataset (

Table 3. scRL performance improvements over LDA percentile variants on hematopoietic dataset.

Method	ARI	NMI	ASW	CH	DB	Overall
scRL vs. ${\mathrm{LDA}}_{80\%}$	+0.330	+0.209	+0.549	+1016.722	+0.495	+0.986
scRL vs. ${\mathrm{LDA}}_{85\%}$	+0.308	+0.192	+0.533	+981.563	+0.465	+0.932
scRL vs. ${\mathrm{LDA}}_{90\%}$	+0.253	+0.170	+0.434	+851.186	+0.350	+0.735
scRL vs. ${\mathrm{LDA}}_{95\%}$	+0.153	+0.091	+0.251	+482.741	+0.172	+0.375

), and comparable improvements of +0.848, +0.549, +0.371 and +0.287 over corresponding LDA percentile thresholds on the endocrinogenesis dataset (Figure A3,

Table A3. scRL performance improvements over LDA percentile variants on endocrinogenesis dataset.

Method	ARI	NMI	ASW	CH	DB	Overall
scRL vs. ${\mathrm{LDA}}_{80}$	+0.219	+0.122	+0.277	+310.840	+0.205	+0.848
scRL vs. ${\mathrm{LDA}}_{85}$	+0.176	+0.104	+0.196	+256.830	+0.095	+0.549
scRL vs. ${\mathrm{LDA}}_{90}$	+0.099	+0.055	+0.083	+194.023	+0.076	+0.371
scRL vs. ${\mathrm{LDA}}_{95}$	+0.025	−0.012	+0.049	+108.108	+0.112	+0.287

).

Figure 3. scRL demonstrates superior interpretability in fate decision intensity analysis. (S-SUBS8) (A) Workflow illustration of scRL’s grid embedding construction and reinforcement learning framework for fate decision intensity inference. (B) Projection of lineage intensities from LDA latent space components onto UMAP embeddings showing correspondence with differentiation branches in hematopoietic and pancreatic datasets. (C) Comparison of scRL fate decision intensity with LDA percentile variants (95%, 90%, 85%, 80%) using ASW, CH and DB interpretability metrics on hematopoietic dataset. The overall score includes data from ARI and NMI.

Figure 3. scRL demonstrates superior interpretability in fate decision intensity analysis. (S-SUBS8) (A) Workflow illustration of scRL’s grid embedding construction and reinforcement learning framework for fate decision intensity inference. (B) Projection of lineage intensities from LDA latent space components onto UMAP embeddings showing correspondence with differentiation branches in hematopoietic and pancreatic datasets. (C) Comparison of scRL fate decision intensity with LDA percentile variants (95%, 90%, 85%, 80%) using ASW, CH and DB interpretability metrics on hematopoietic dataset. The overall score includes data from ARI and NMI.

3.3. scRL Outperforms Alternative Approaches in Fate-Decision Inference

We evaluated scRL’s fate decision intensity performance against various machine learning and deep learning dimensionality reduction methods including Diff, FA, ICA, LscVI, NMF, PCA and scVI across hematopoietic and endocrinogenesis datasets (Figure 4 and Figure A4A). Using scRL-derived grid embeddings and pseudotime information as the foundation, we extracted 95%, 90%, 85% and 80% percentile intensities from each method as alternative fate decision measures, conducting experiments across HVG sequences from 1000 to 5000 and evaluating performance using unsupervised metrics ARI, NMI, ASW, DB and CH. scRL consistently achieved the highest overall scores across all experimental conditions (Figure 4 and Figure A4B). On the hematopoietic dataset, scRL showed substantial improvements over all baseline methods, with pronounced advantages against ICA-based approaches (overall improvements ranging from +0.614 to +0.811) and scVI-based methods (overall improvements from +0.609 to +0.733) (

Table 4. scRL fate decision performance improvements over baseline methods on hematopoietic dataset.

Method	NMI	ARI	ASW	DB	CH	Overall
scRL vs. ${\mathrm{Diff}}_{80\%}$	+0.121	+0.060	+0.224	+1.314	+966.124	+0.465
scRL vs. ${\mathrm{Diff}}_{85\%}$	+0.115	+0.046	+0.213	+1.265	+940.620	+0.441
scRL vs. ${\mathrm{Diff}}_{90\%}$	+0.100	+0.039	+0.256	+0.939	+927.518	+0.432
scRL vs. ${\mathrm{Diff}}_{95\%}$	+0.037	−0.007	+0.103	+0.638	+720.043	+0.228
scRL vs. ${\mathrm{FA}}_{80\%}$	+0.229	+0.142	+0.226	+2.317	+979.821	+0.647
scRL vs. ${\mathrm{FA}}_{85\%}$	+0.195	+0.114	+0.202	+1.892	+926.712	+0.563
scRL vs. ${\mathrm{FA}}_{90\%}$	+0.150	+0.076	+0.233	+1.090	+836.652	+0.466
scRL vs. ${\mathrm{FA}}_{95\%}$	+0.080	+0.009	+0.186	+0.578	+648.564	+0.296
scRL vs. ${\mathrm{ICA}}_{80\%}$	+0.346	+0.224	+0.243	+3.010	+995.896	+0.811
scRL vs. ${\mathrm{ICA}}_{85\%}$	+0.344	+0.224	+0.241	+2.829	+968.163	+0.785
scRL vs. ${\mathrm{ICA}}_{90\%}$	+0.325	+0.213	+0.231	+2.577	+926.928	+0.731
scRL vs. ${\mathrm{ICA}}_{95\%}$	+0.258	+0.162	+0.206	+2.548	+837.487	+0.614
scRL vs. ${\mathrm{LscVI}}_{80\%}$	+0.236	+0.170	+0.262	+1.989	+1014.079	+0.677
scRL vs. ${\mathrm{LscVI}}_{85\%}$	+0.213	+0.155	+0.254	+1.674	+991.179	+0.627
scRL vs. ${\mathrm{LscVI}}_{90\%}$	+0.169	+0.125	+0.239	+1.365	+952.584	+0.551
scRL vs. ${\mathrm{LscVI}}_{95\%}$	+0.119	+0.078	+0.208	+1.228	+883.328	+0.454
scRL vs. ${\mathrm{NMF}}_{80\%}$	+0.307	+0.262	+0.008	+0.095	+656.817	+0.455
scRL vs. ${\mathrm{NMF}}_{85\%}$	+0.311	+0.258	+0.093	+0.143	+817.645	+0.540
scRL vs. ${\mathrm{NMF}}_{90\%}$	+0.278	+0.237	+0.158	+0.169	+799.054	+0.544
scRL vs. ${\mathrm{NMF}}_{95\%}$	+0.160	+0.171	+0.096	+0.023	+349.374	+0.297
scRL vs. ${\mathrm{PCA}}_{80\%}$	+0.240	+0.166	+0.199	+2.434	+920.403	+0.651
scRL vs. ${\mathrm{PCA}}_{85\%}$	+0.206	+0.150	+0.179	+2.095	+861.993	+0.579
scRL vs. ${\mathrm{PCA}}_{90\%}$	+0.152	+0.110	+0.156	+1.697	+792.187	+0.474
scRL vs. ${\mathrm{PCA}}_{95\%}$	+0.089	+0.059	+0.159	+1.205	+702.267	+0.362
scRL vs. ${\mathrm{scVI}}_{80\%}$	+0.344	+0.226	+0.200	+1.731	+1061.220	+0.733
scRL vs. ${\mathrm{scVI}}_{85\%}$	+0.325	+0.216	+0.195	+1.542	+1040.949	+0.698
scRL vs. ${\mathrm{scVI}}_{90\%}$	+0.299	+0.202	+0.190	+1.390	+1020.921	+0.659
scRL vs. ${\mathrm{scVI}}_{95\%}$	+0.259	+0.181	+0.185	+1.272	+998.612	+0.609

). Validation on the endocrinogenesis dataset confirmed these findings, where scRL maintained superior performance with the most significant improvements observed against Diffusion Map approaches (overall improvements from +0.461 to +0.811) and ICA-based methods (overall improvements from +0.612 to +0.812) (

Table A4. scRL fate decision performance improvements over baseline methods on endocrinogenesis dataset.

Method	NMI	ARI	ASW	DB	CH	Overall
scRL vs. ${\mathrm{Diff}}_{80}$	+0.337	+0.236	+0.364	+1.436	+615.913	+0.811
scRL vs. ${\mathrm{Diff}}_{85}$	+0.310	+0.234	+0.389	+1.218	+610.215	+0.793
scRL vs. ${\mathrm{Diff}}_{90}$	+0.240	+0.213	+0.359	+1.222	+596.458	+0.722
scRL vs. ${\mathrm{Diff}}_{95}$	+0.102	+0.120	+0.224	+0.831	+534.217	+0.461
scRL vs. ${\mathrm{FA}}_{80}$	+0.187	+0.115	+0.213	+2.754	+503.551	+0.605
scRL vs. ${\mathrm{FA}}_{85}$	+0.152	+0.092	+0.190	+2.089	+461.722	+0.502
scRL vs. ${\mathrm{FA}}_{90}$	+0.100	+0.033	+0.177	+2.471	+440.212	+0.439
scRL vs. ${\mathrm{FA}}_{95}$	+0.037	−0.041	+0.134	+2.100	+418.401	+0.300
scRL vs. ${\mathrm{ICA}}_{80}$	+0.345	+0.201	+0.255	+2.819	+599.145	+0.812
scRL vs. ${\mathrm{ICA}}_{85}$	+0.331	+0.186	+0.244	+2.557	+592.465	+0.770
scRL vs. ${\mathrm{ICA}}_{90}$	+0.303	+0.165	+0.229	+2.227	+584.174	+0.709
scRL vs. ${\mathrm{ICA}}_{95}$	+0.248	+0.129	+0.210	+1.804	+563.078	+0.612
scRL vs. ${\mathrm{LscVI}}_{80}$	+0.171	+0.090	+0.219	+1.772	+465.352	+0.506
scRL vs. ${\mathrm{LscVI}}_{85}$	+0.134	+0.070	+0.210	+1.450	+435.556	+0.443
scRL vs. ${\mathrm{LscVI}}_{90}$	+0.085	+0.032	+0.192	+1.251	+393.053	+0.351
scRL vs. ${\mathrm{LscVI}}_{95}$	+0.016	−0.037	+0.164	+1.037	+297.412	+0.207
scRL vs. ${\mathrm{NMF}}_{80}$	+0.335	+0.202	+0.178	+0.066	+325.897	+0.499
scRL vs. ${\mathrm{NMF}}_{85}$	+0.287	+0.171	+0.167	+0.134	+332.902	+0.452
scRL vs. ${\mathrm{NMF}}_{90}$	+0.193	+0.120	+0.151	+0.123	+353.477	+0.362
scRL vs. ${\mathrm{NMF}}_{95}$	+0.113	+0.049	+0.134	+0.116	+291.898	+0.238
scRL vs. ${\mathrm{PCA}}_{80}$	+0.200	+0.111	+0.164	+2.066	+376.044	+0.495
scRL vs. ${\mathrm{PCA}}_{85}$	+0.160	+0.107	+0.142	+1.450	+304.960	+0.395
scRL vs. ${\mathrm{PCA}}_{90}$	+0.117	+0.074	+0.110	+1.214	+253.100	+0.298
scRL vs. ${\mathrm{PCA}}_{95}$	+0.032	−0.007	+0.095	+0.961	+252.420	+0.173
scRL vs. ${\mathrm{scVI}}_{80}$	+0.330	+0.203	+0.221	+1.741	+577.491	+0.713
scRL vs. ${\mathrm{scVI}}_{85}$	+0.312	+0.191	+0.217	+1.546	+568.437	+0.678
scRL vs. ${\mathrm{scVI}}_{90}$	+0.283	+0.173	+0.213	+1.392	+559.935	+0.634
scRL vs. ${\mathrm{scVI}}_{95}$	+0.237	+0.140	+0.206	+1.267	+548.500	+0.571

). scRL’s performance advantages were consistently maintained across different percentile thresholds, with higher improvements generally observed at lower percentile values (80–85% percentiles) compared with higher percentiles (95% percentile), indicating that scRL’s reinforcement learning framework effectively captures early fate decision signals that precede observable lineage contribution events.

Figure 4. scRL exhibits superior interpretability in fate decision intensity compared with various dimensionality reduction methods. (S-SUBS8) Comprehensive comparison of interpretability metrics for scRL fate decision intensity against Diff, FA, ICA, LscVI, NMF, PCA and scVI at 95%, 90%, 85% and 80% percentile intensities on hematopoietic dataset.

Figure 4. scRL exhibits superior interpretability in fate decision intensity compared with various dimensionality reduction methods. (S-SUBS8) Comprehensive comparison of interpretability metrics for scRL fate decision intensity against Diff, FA, ICA, LscVI, NMF, PCA and scVI at 95%, 90%, 85% and 80% percentile intensities on hematopoietic dataset.

3.4. Label-Free scRL Accurately Quantifies Lineage Contribution

When accurate lineage probabilities are unavailable or cannot be mapped to subgroup trajectory branches, scRL offers an alternative solution leveraging reinforcement learning principles with clustering labels as reward signals to evaluate lineage contribution intensity. We applied this approach to hematopoietic and endocrinogenesis datasets using KMeans clustering to obtain labels for distinct subgroups, with scRL computing lineage contribution intensity that exhibited clear subgroup specificity (Figure 5A). We evaluated scRL’s interpretability against established dimensionality reduction techniques (PCA, ICA, FA, NMF, Diff, scVI, LscVI) across varying KMeans subgroup numbers (4, 6, 8, 10, 12) and highly variable gene sequences (1000–5000) using ASW, DB and CH metrics. scRL achieved the best overall assessment scores across all experimental conditions (Figure 5B), with substantial improvements over baseline methods including pronounced advantages against scVI-based approaches (overall improvements from +0.856 to +0.924), ICA methods (+0.409 to +0.694) and PCA approaches (+0.329 to +0.515) across different cluster configurations (

Table 5. scRL lineage contribution performance improvements over baseline methods across different cluster numbers on hematopoietic dataset.

Method	ASW	DB	CH	Overall
scRL vs. ${\mathrm{PCA}}_4$	+0.247	+0.359	+5121.830	+0.353
scRL vs. ${\mathrm{ICA}}_4$	+0.256	+0.447	+6269.940	+0.409
scRL vs. ${\mathrm{FA}}_4$	+0.119	+0.100	+5613.054	+0.242
scRL vs. ${\mathrm{NMF}}_4$	+0.114	+0.105	+816.813	+0.105
scRL vs. ${\mathrm{Diff}}_4$	+0.041	+0.012	−2261.901	−0.048
scRL vs. ${\mathrm{scVI}}_4$	+0.551	+1.761	+8639.039	+0.899
scRL vs. ${\mathrm{LscVI}}_4$	+0.241	+0.350	+4669.743	+0.336
scRL vs. ${\mathrm{PCA}}_6$	+0.124	+0.235	+5058.890	+0.329
scRL vs. ${\mathrm{ICA}}_6$	+0.150	+0.312	+6535.205	+0.425
scRL vs. ${\mathrm{FA}}_6$	+0.058	+0.023	+5792.529	+0.254
scRL vs. ${\mathrm{NMF}}_6$	+0.042	+0.077	+2848.851	+0.131
scRL vs. ${\mathrm{Diff}}_6$	−0.086	+0.212	+2454.264	−0.024
scRL vs. ${\mathrm{scVI}}_6$	+0.368	+1.301	+7855.647	+0.859
scRL vs. ${\mathrm{LscVI}}_6$	+0.119	+0.255	+5366.970	+0.337
scRL vs. ${\mathrm{PCA}}_8$	+0.157	+0.310	+3963.771	+0.426
scRL vs. ${\mathrm{ICA}}_8$	+0.189	+0.353	+5198.874	+0.535
scRL vs. ${\mathrm{FA}}_8$	+0.120	+0.080	+4634.559	+0.375
scRL vs. ${\mathrm{NMF}}_8$	+0.103	+0.196	+2651.048	+0.265
scRL vs. ${\mathrm{Diff}}_8$	−0.047	+0.133	+3041.397	+0.091
scRL vs. ${\mathrm{scVI}}_8$	+0.360	+1.106	+5999.717	+0.924
scRL vs. LscVI8	+0.177	+0.320	+4190.457	+0.456
scRL vs. PCA10	+0.137	+0.237	+3330.795	+0.379
scRL vs. ICA10	+0.169	+0.280	+3284.407	+0.430
scRL vs. FA10	+0.069	+0.112	+3547.165	+0.236
scRL vs. NMF10	+0.061	+0.103	+2414.816	+0.210
scRL vs. Diff10	−0.069	+0.161	+2894.393	+0.059
scRL vs. scVI10	+0.308	+1.024	+4944.921	+0.856
scRL vs. ${\mathrm{LscVI}}_{10}$	+0.161	+0.321	+3663.236	+0.447
scRL vs. PCA12	+0.194	+0.465	+2821.714	+0.515
scRL vs. ICA12	+0.235	+0.578	+3874.281	+0.694
scRL vs. FA12	+0.096	+0.106	+2951.559	+0.284
scRL vs. NMF12	+0.114	+0.240	+2419.564	+0.334
scRL vs. Diff12	−0.064	+0.043	+2708.683	+0.120
scRL vs. scVI12	+0.300	+1.019	+4055.950	+0.899
scRL vs. LscVI12	+0.164	+0.311	+2998.675	+0.461

). Validation on the endocrinogenesis dataset confirmed these findings, where scRL maintained superior performance with clear subgroup specificity in both KMeans labels and contribution intensities, showing distinct intensity distribution patterns across lineage branches and consistently strong performance advantages over scVI (overall improvements from +0.856 to +0.924) and ICA (+0.409 to +0.694) across all cluster configurations (Figure A5A,B,

Table A5. scRL lineage contribution performance improvements over baseline methods across different cluster numbers on endocrinogenesis dataset.

Method	ASW	DB	CH	Overall
scRL vs. ${\mathrm{PCA}}_4$	+0.247	+0.359	+5121.830	+0.353
scRL vs. ${\mathrm{PCA}}_6$	+0.124	+0.235	+5058.890	+0.329
scRL vs. ${\mathrm{PCA}}_8$	+0.157	+0.310	+3963.771	+0.426
scRL vs. ${\mathrm{PCA}}_{10}$	+0.137	+0.237	+3330.795	+0.379
scRL vs. ${\mathrm{PCA}}_{12}$	+0.194	+0.465	+2821.714	+0.515
scRL vs. ${\mathrm{ICA}}_4$	+0.256	+0.447	+6269.940	+0.409
scRL vs. ${\mathrm{ICA}}_6$	+0.150	+0.312	+6535.205	+0.425
scRL vs. ${\mathrm{ICA}}_8$	+0.189	+0.353	+5198.874	+0.535
scRL vs. ${\mathrm{ICA}}_{10}$	+0.169	+0.280	+3284.407	+0.430
scRL vs. ${\mathrm{ICA}}_{12}$	+0.235	+0.578	+3874.281	+0.694
scRL vs. ${\mathrm{FA}}_4$	+0.119	+0.100	+5613.054	+0.242
scRL vs. ${\mathrm{FA}}_6$	+0.058	+0.023	+5792.529	+0.254
scRL vs. ${\mathrm{FA}}_8$	+0.120	+0.080	+4634.559	+0.375
scRL vs. ${\mathrm{FA}}_{10}$	+0.069	+0.112	+3547.165	+0.236
scRL vs. ${\mathrm{FA}}_{12}$	+0.096	+0.106	+2951.559	+0.284
scRL vs. ${\mathrm{NMF}}_4$	+0.114	+0.105	+816.813	+0.105
scRL vs. ${\mathrm{NMF}}_6$	+0.042	+0.077	+2848.851	+0.131
scRL vs. ${\mathrm{NMF}}_8$	+0.103	+0.196	+2651.048	+0.265
scRL vs. ${\mathrm{NMF}}_{10}$	+0.061	+0.103	+2414.816	+0.210
scRL vs. ${\mathrm{NMF}}_{12}$	+0.114	+0.240	+2419.564	+0.334
scRL vs. ${\mathrm{Diff}}_4$	+0.041	+0.012	−2261.901	−0.048
scRL vs. ${\mathrm{Diff}}_6$	−0.086	+0.212	+2454.264	−0.024
scRL vs. ${\mathrm{Diff}}_8$	−0.047	+0.133	+3041.397	+0.091
scRL vs. ${\mathrm{Diff}}_{10}$	−0.069	+0.161	+2894.393	+0.059
scRL vs. ${\mathrm{Diff}}_{12}$	−0.064	+0.043	+2708.683	+0.120
scRL vs. ${\mathrm{scVI}}_4$	+0.551	+1.761	+8639.039	+0.899
scRL vs. ${\mathrm{scVI}}_6$	+0.368	+1.301	+7855.647	+0.859
scRL vs. ${\mathrm{scVI}}_8$	+0.360	+1.106	+5999.717	+0.924
scRL vs. ${\mathrm{scVI}}_{10}$	+0.308	+1.024	+4944.921	+0.856
scRL vs. ${\mathrm{scVI}}_{12}$	+0.300	+1.019	+4055.950	+0.899
scRL vs. ${\mathrm{LscVI}}_4$	+0.241	+0.350	+4669.743	+0.336
scRL vs. ${\mathrm{LscVI}}_6$	+0.119	+0.255	+5366.970	+0.337
scRL vs. ${\mathrm{LscVI}}_8$	+0.177	+0.320	+4190.457	+0.456
scRL vs. ${\mathrm{LscVI}}_{10}$	+0.161	+0.321	+3663.236	+0.447
scRL vs. ${\mathrm{LscVI}}_{12}$	+0.164	+0.311	+2998.675	+0.461

).

Figure 5. scRL constructs superior lineage intensity using subgroup information compared with various methods. (S-SUBS8) (A) Distribution of 5 subgroups under KMeans clustering and corresponding subgroup lineage intensities obtained by scRL projected onto UMAP embedding, demonstrating subgroup specificity. (B) Comprehensive comparison of scRL lineage contribution intensity against PCA, ICA, FA, NMF, Diff, scVI and LscVI using ASW, DB and CH metrics. Evaluations conducted across KMeans subgroup numbers (4, 6, 8, 10, 12) and highly variable gene sequences (1000–5000) as experimental replicates.

Figure 5. scRL constructs superior lineage intensity using subgroup information compared with various methods. (S-SUBS8) (A) Distribution of 5 subgroups under KMeans clustering and corresponding subgroup lineage intensities obtained by scRL projected onto UMAP embedding, demonstrating subgroup specificity. (B) Comprehensive comparison of scRL lineage contribution intensity against PCA, ICA, FA, NMF, Diff, scVI and LscVI using ASW, DB and CH metrics. Evaluations conducted across KMeans subgroup numbers (4, 6, 8, 10, 12) and highly variable gene sequences (1000–5000) as experimental replicates.

3.5. scRL Rivals State-of-the-Art Pseudotime and Fate Inference Tools

We conducted comprehensive performance evaluations comparing scRL with established methods for pseudotime reconstruction (Wishbone [37], Palantir [38], DPT (Diffusion Pseudotime) [39], Monocle2 [40], Monocle3 [12]) and lineage probability inference (Palantir [38], FateID [41], CellRank [42]) using Manifold Alignment Score and Principal Axis Alignment Coefficient metrics. On the hematopoietic dataset, scRL demonstrated superior pseudotime reconstruction performance, achieving a Manifold Alignment Score of 0.773 (+0.091 improvement over the previous best method) and Principal Axis Alignment Coefficient of 0.848 (+0.127 improvement) (Figure 6A,B). For lineage probability assessment, scRL consistently outperformed all baseline methods with a Manifold Alignment Score of 0.323 (+0.085 improvement) and Principal Axis Alignment Coefficient of 0.373 (+0.048 improvement) (Figure 6C,D). Validation on the endocrinogenesis dataset confirmed these findings, where scRL maintained superior performance with pseudotime reconstruction achieving a Manifold Alignment Score of 0.865 (+0.061 improvement over the previous best method) and fate probability analysis demonstrating robust performance with a score of 0.674 (+0.312 improvement over the previous best method) (Figure A6A,B).

3.6. scRL Decision Intensities Reveal Pre-Contribution States in Hematopoiesis

We applied scRL to a human hematopoietic progenitor cell dataset containing diverse immune phenotypes [43], conducting pseudotime reconstruction and comprehensive evaluation of lineage contribution strength and fate decision intensity for MEP (Megakaryocyte–Erythroid Progenitor) and GMP (Granulocyte-Monocyte Progenitor) lineages (Figure 7A). scRL revealed distinct temporal dynamics where fate decision intensity and lineage contribution strength exhibited complementary temporal profiles along pseudotime trajectories, with pseudotime-weighted analysis demonstrating that decision-weighted values consistently preceded contribution-weighted values, indicating scRL captures earlier fate specification events before observable lineage contribution (Figure 7B). To validate scRL’s predictive capacity for identifying pre-expression cellular states, we analyzed key transcription factors GATA1 and CEBPA, which serve as master regulators of MEP and GMP differentiation, respectively, with GATA1 essential for erythroid and megakaryocytic development and CEBPA functioning as a master regulator of myeloid lineage contribution. Both transcription factors exhibited peak expression at terminal differentiation stages in their respective lineages, while scRL fate decision analysis demonstrated that both displayed high decision intensity values in more primitive regions compared with their contribution values, suggesting early fate specification events preceding actual gene expression (Figure 7C). Detailed temporal analysis revealed that, for both GATA1 in MEP lineage and CEBPA in GMP lineage, decision state values peaked at intermediate differentiation stages systematically preceding expression peaks, while contribution state values reached maximum intensity at terminal stages and exhibited stronger correlation with pseudotime compared with original expression patterns, with pseudotime-weighted analysis consistently showing decision-weighted values occurred earlier than contribution-weighted values (Figure 7D).

3.7. Integrated Gene- and Lineage-Level Decisions Resolve AML Branch Points

We applied scRL’s pseudotime analysis framework to an AML dataset to elucidate the relationship between gene-level and lineage-level decision-making processes during cellular differentiation [44]. Following cellular clustering and pseudotime reconstruction revealing distinct cellular populations and developmental trajectories (Figure 8A,B), we identified four distinct branches representing different lineage decision trajectories (Figure 8C). For each branch, we selected 10 marker genes based on differential expression analysis, requiring expression in at least 25% of cells within the target branch while being expressed in less than 25% of reference cells (Figure 8D), then conducted parallel analyses of gene decision values and lineage decision values across all four branches (Figure 8E,F). Correlation analysis revealed strong relationships between lineage and gene decision values within each branch, demonstrating coordinated decision-making processes at both molecular and cellular levels (Figure 8G), while temporal analysis showed that weighted average pseudotime of lineage decision values consistently preceded actual gene expression across all four branches (Figure 8H). Further characterization through binned pseudotime analysis revealed that lineage decision values across the four branches showed more distinct trajectory dynamics compared with mean Pearson correlation values with branch-specific cells (Figure 8I,J), while gene decision values along binned pseudotime provided clearer temporal dynamics compared with Pearson correlation values with branch-specific genes (Figure 8K,L).

Figure 6. Comparative performance analysis of scRL with pseudotime and fate inference methods. (S-SUBS8) (A) Pseudotime reconstruction performance comparison using Manifold Alignment Score across methods including Wishbone, Palantir, DPT, Monocle2, Monocle3 and scRL on hematopoietic dataset. (B) Principal Axis Alignment Coefficient comparison for pseudotime reconstruction methods. (C) Fate probability inference performance using Manifold Alignment Score comparing FateID, Palantir, CellRank and scRL. (D) Principal Axis Alignment Coefficient comparison for fate probability inference methods. Green bars indicate performance improvements over previous best methods.

Figure 6. Comparative performance analysis of scRL with pseudotime and fate inference methods. (S-SUBS8) (A) Pseudotime reconstruction performance comparison using Manifold Alignment Score across methods including Wishbone, Palantir, DPT, Monocle2, Monocle3 and scRL on hematopoietic dataset. (B) Principal Axis Alignment Coefficient comparison for pseudotime reconstruction methods. (C) Fate probability inference performance using Manifold Alignment Score comparing FateID, Palantir, CellRank and scRL. (D) Principal Axis Alignment Coefficient comparison for fate probability inference methods. Green bars indicate performance improvements over previous best methods.

Figure 7. scRL application to hematopoietic dataset reveals temporal precedence of decision over contribution states. (GSE117498) (A) Standard analysis workflow including pseudotime reconstruction and scRL-derived lineage contribution strength and fate decision intensity results for the two major lineages, MEP and GMP. (B) Temporal dynamics of fate decision intensity and lineage contribution strength along pseudotime, with bar plots showing pseudotime-weighted values for both intensities (lower values correspond to more primitive developmental stages). (C) Original expression patterns of key transcription factors GATA1 and CEBPA corresponding to MEP and GMP lineages, respectively, alongside scRL-derived contribution strength and decision strength when these genes serve as reward signals. (D) Temporal profiles along pseudotime showing original expression intensity, decision strength, contribution strength, and pseudotime-weighted decision and contribution intensities for GATA1 and CEBPA (lower values correspond to more primitive stages), with correlation comparison between contribution strength and original expression relative to pseudotime (higher values indicate superior differentiation trajectory matching).

Figure 7. scRL application to hematopoietic dataset reveals temporal precedence of decision over contribution states. (GSE117498) (A) Standard analysis workflow including pseudotime reconstruction and scRL-derived lineage contribution strength and fate decision intensity results for the two major lineages, MEP and GMP. (B) Temporal dynamics of fate decision intensity and lineage contribution strength along pseudotime, with bar plots showing pseudotime-weighted values for both intensities (lower values correspond to more primitive developmental stages). (C) Original expression patterns of key transcription factors GATA1 and CEBPA corresponding to MEP and GMP lineages, respectively, alongside scRL-derived contribution strength and decision strength when these genes serve as reward signals. (D) Temporal profiles along pseudotime showing original expression intensity, decision strength, contribution strength, and pseudotime-weighted decision and contribution intensities for GATA1 and CEBPA (lower values correspond to more primitive stages), with correlation comparison between contribution strength and original expression relative to pseudotime (higher values indicate superior differentiation trajectory matching).

Figure 8. Integrative analysis of lineage and gene decision states in acute myeloid leukemia progression. (GSE185993) (A) UMAP visualization of AML dataset colored by cellular clusters. Different colors represent distinct cell subpopulations. (B) Pseudotime trajectory inferred by scRL across the developmental landscape. (C) Four distinct branches identified within the UMAP embedding space. (D) Top 10 marker genes for each branch, selected based on log-fold changes with expression criteria of at least 25% cells in target branch and less than 25% in reference cells. (E) Gene set decision values computed for each of the four branches. (F) Lineage decision values determined for each branch using scRL framework. (G) Pearson correlation heatmap between gene set decision values (G1-4) and lineage decision values (L1-4) for each branch. (H) Comparison of average pseudotime weighted by lineage decision values versus gene expression for each branch. (I) Mean Pearson correlation with branch-specific cells along uniformly 50-binned pseudotime for the complete dataset (top) and trunk region (bottom). (J) Lineage decision values for each branch along uniformly 50-binned pseudotime for the complete dataset (top) and trunk region (bottom). (K) Mean expression of branch-specific gene sets along uniformly 50-binned pseudotime for the complete dataset (top) and trunk region (bottom). (L) Gene decision values of branch-specific gene sets along uniformly 50-binned pseudotime for the complete dataset (top) and trunk region (bottom).

Figure 8. Integrative analysis of lineage and gene decision states in acute myeloid leukemia progression. (GSE185993) (A) UMAP visualization of AML dataset colored by cellular clusters. Different colors represent distinct cell subpopulations. (B) Pseudotime trajectory inferred by scRL across the developmental landscape. (C) Four distinct branches identified within the UMAP embedding space. (D) Top 10 marker genes for each branch, selected based on log-fold changes with expression criteria of at least 25% cells in target branch and less than 25% in reference cells. (E) Gene set decision values computed for each of the four branches. (F) Lineage decision values determined for each branch using scRL framework. (G) Pearson correlation heatmap between gene set decision values (G1-4) and lineage decision values (L1-4) for each branch. (H) Comparison of average pseudotime weighted by lineage decision values versus gene expression for each branch. (I) Mean Pearson correlation with branch-specific cells along uniformly 50-binned pseudotime for the complete dataset (top) and trunk region (bottom). (J) Lineage decision values for each branch along uniformly 50-binned pseudotime for the complete dataset (top) and trunk region (bottom). (K) Mean expression of branch-specific gene sets along uniformly 50-binned pseudotime for the complete dataset (top) and trunk region (bottom). (L) Gene decision values of branch-specific gene sets along uniformly 50-binned pseudotime for the complete dataset (top) and trunk region (bottom).

3.8. scRL Pinpoints Dapp1 as an Early Regulator of HSC Fate

We developed a comprehensive workflow for discovering critical dynamical genes in HSC differentiation, establishing a systematic approach beginning with LSK (${\mathrm{Lin}}^{-}{\mathrm{Sca1}}^{+}{\mathrm{ckit}}^{+}$) single-cell data embeddings to identify differentiation starting points, employing velocity analysis for top dynamical genes, utilizing pseudotime analysis for earliest expressing genes, and intersecting highest expressed and most dynamic genes followed by time correlation scoring to identify Dapp1 as a critical early differentiation regulator (Figure 9A). Comprehensive trajectory analysis using Monocle2 identified 280 genes through intersection of dynamical genes and BEAM (Branching Expression Analysis Modeling) test significant genes with enrichment in hematopoietic differentiation-related GOBP (Gene Ontology Biological Process) terms, while gene correlation analysis using LSK datasets revealed Dapp1 exhibited strong correlation with myeloid lineage marker Dach1 and minimal correlation with lymphoid marker Dntt (Figure A7A). To validate scRL’s capacity to capture perturbation effects, we constructed a single-cell atlas from bone marrow LSK cells with conditional Dapp1 knockout, annotating cellular populations based on key transcription factors including Hlf and Tcf15 for primitive HSCs, Gata1 and Klf1 for erythroid bias, Id2 for dendritic cells, Irf8 for monocytes, Cebpe for neutrophils, Ebf1 for B cells, Hoxa9 for multipotency, Satb1 for lymphoid bias and Gfi1 for myeloid bias (Figure A7B). The Dapp1 knockout atlas revealed clusters annotated as early clusters (1, 2), multipotent clusters (3, 5, 7), lymphoid clusters (4, 9), myeloid cluster (6) and erythroid clusters (8, 10), with comparative analysis demonstrating significant disruption in lineage contribution homeostasis through altered cluster distributions and pseudotime patterns (Figure 9B,C). Pseudotime analysis and expression patterns of maturation markers Cd34, Mpo and Ctsg indicated impediment to cellular maturation following Dapp1 deletion (Figure 9D). Analysis along uniformly 50-binned pseudotime revealed Dapp1 deficiency led to significant perturbations in lineage decision trajectories with decreased decision values for myeloid and erythroid lineages while lymphoid lineage decision values increased (Figure 9E,F). Gene decision values for Mpo (myeloid), Itga2b (erythroid) and Dntt (lymphoid) showed notable alterations along pseudotime, with Dapp1 deletion decreasing decision values for Mpo and Itga2b while increasing Dntt decision values (Figure 9G,H). Detailed cluster-specific analysis revealed that, within earliest cluster (cluster 1), both myeloid lineage and Mpo gene decision trajectories were significantly disrupted with decreased decision values after knockout (Figure 9I,J), while, within multipotent clusters, lymphoid lineage and Dntt gene decision values were enhanced following Dapp1 deletion (Figure 9K). Comprehensive trajectory analysis comparing wild-type and knockout conditions demonstrated distinct alterations in myeloid, erythroid and lymphoid differentiation trajectories with cluster-specific responses to Dapp1 perturbation, where lymphoid lineage decisions remained largely unchanged in early clusters but were enhanced in multipotent clusters, while erythroid and myeloid decision values showed varying degrees of disruption across different developmental stages (Figure A7C and Figure A8A–E).

Figure 9. Lineage contribution of HSCs at the earliest primitive stage perturbed by knockout of critical dynamical gene Dapp1. (GSE277292) (A) Systematic workflow for identifying critical dynamical gene Dapp1 in HSC differentiation, beginning with LSK single-cell data embeddings to identify differentiation starting points, employing velocity analysis for top dynamical genes, using pseudotime for earliest expressing genes, intersecting highest expressed and most dynamic genes, and ranking through time correlation scoring. (B) Dapp1 knockout LSK atlas with annotated clusters: 1, 2 as early clusters, 3, 5, 7 as multipotent clusters, 4, 9 as lymphoid clusters, 6 as myeloid cluster and 8,10 as erythroid clusters. (C) Cluster distributions and pseudotime comparison between control and knockout groups. (D) Box plots comparing pseudotime and expression of maturation markers Cd34, Mpo and Ctsg between conditions. (E) Lineage decision values for myeloid, erythroid and lymphoid lineages along uniformly 50-binned pseudotime. (F) Box plots comparing lineage decision values across myeloid, erythroid and lymphoid lineages between conditions. (G) Gene decision values for Mpo, Itga2b and Dntt along uniformly 50-binned pseudotime. (H) Box plots comparing gene decision values for Mpo, Itga2b and Dntt between conditions. (I) Myeloid lineage and Mpo decision values within early cluster 1 along binned pseudotime. (J) Box plots comparing myeloid lineage and Mpo decision values within early cluster 1. (K) Box plots comparing lymphoid lineage and Dntt decision values within multipotent cluster 5.

Figure 9. Lineage contribution of HSCs at the earliest primitive stage perturbed by knockout of critical dynamical gene Dapp1. (GSE277292) (A) Systematic workflow for identifying critical dynamical gene Dapp1 in HSC differentiation, beginning with LSK single-cell data embeddings to identify differentiation starting points, employing velocity analysis for top dynamical genes, using pseudotime for earliest expressing genes, intersecting highest expressed and most dynamic genes, and ranking through time correlation scoring. (B) Dapp1 knockout LSK atlas with annotated clusters: 1, 2 as early clusters, 3, 5, 7 as multipotent clusters, 4, 9 as lymphoid clusters, 6 as myeloid cluster and 8,10 as erythroid clusters. (C) Cluster distributions and pseudotime comparison between control and knockout groups. (D) Box plots comparing pseudotime and expression of maturation markers Cd34, Mpo and Ctsg between conditions. (E) Lineage decision values for myeloid, erythroid and lymphoid lineages along uniformly 50-binned pseudotime. (F) Box plots comparing lineage decision values across myeloid, erythroid and lymphoid lineages between conditions. (G) Gene decision values for Mpo, Itga2b and Dntt along uniformly 50-binned pseudotime. (H) Box plots comparing gene decision values for Mpo, Itga2b and Dntt between conditions. (I) Myeloid lineage and Mpo decision values within early cluster 1 along binned pseudotime. (J) Box plots comparing myeloid lineage and Mpo decision values within early cluster 1. (K) Box plots comparing lymphoid lineage and Dntt decision values within multipotent cluster 5.

3.9. scRL Charts Dynamic Fate Decisions During Post-Irradiation HSC Recovery

We applied scRL to analyze HSC fate decision dynamics under pathological conditions by collecting LSK cells from mice subjected to total body ionizing radiation at multiple time points (days 2, 5, 8, 11, 14, 21, 30) and performing single-cell RNA sequencing to construct a comprehensive temporal atlas. Cellular populations were annotated based on characteristic marker expression patterns including Kit and Ly6a for LSK identification, Cd34 and Flt3 for Multipotent Progenitor components, and lineage-specific markers such as Il7r for lymphoid bias, Cd19 for B cells, Cd3e for T cells, and Itgam and Fcer1g for myeloid bias (Figure A9A). Eleven clusters were identified and annotated as HSC, MPP2/3 (Multipotent Progenitors 2/3), MPP4 (Multipotent Progenitors 4), MK (Megakaryocyte), Ery (Erythrocyte), Ma (Mast cell), Neu (neutrophil), Mo (monocyte), DC (dendritic cell), B cell and T cell, revealing dynamic changes in cellular composition across the recovery timeline (Figure 10A,B). Population dynamics analysis revealed HSPC (Hematopoietic Stem and Progenitor Cell) components, particularly HSCs, decreased dramatically after radiation, reaching their nadir around day 5, followed by gradual recovery (Figure 10D), while an erythroid contribution bias emerged after radiation, peaking at day 8 and persisting until day 21 (Figure 10E), and myeloid cell populations showed characteristic temporal changes throughout recovery (Figure 10F). Using pseudotime as a cellular primitivity metric, the recovery process demonstrated HSCs gradually regaining their primitive state with coordinated expression of primitive factors (Hlf, Meis1, Satb1, Hoxa9) indicating primitivity recovery, while myeloid factors (Spi1, Gfi1, Cebpa, Cebpe) and erythroid factors (Klf1, Gata1) exhibited lineage-specific temporal dynamics (Figure A9B and Figure 10C). scRL computation of lineage decision values for erythroid and myeloid lineages across all time points revealed characteristic spatial and temporal patterns projected onto UMAP space (Figure 10G,H), with comprehensive analysis showing distinct temporal patterns in both contribution and decision values (Figure A9C–E). Detailed analysis within HSPC subpopulations across uniformly 50-binned pseudotime demonstrated that erythroid-biased stem cell fate decision potential decreased dramatically immediately after radiation but rapidly returned to near-normal levels by day 8, while myeloid-biased stem cell proportions increased following irradiation and subsequently decreased as erythroid-biased populations recovered (Figure 10I,J). Correlation analysis revealed erythroid potential of HSPCs showed strongest association with overall HSPC proportions, whereas myeloid potential was closely correlated with myeloid cell proportions during early recovery phases, with validation through Pearson correlation coefficients confirming that myeloid and erythroid decision intensities exhibited positive correlations with their corresponding lineage-specific transcription factors while pseudotime demonstrated negative correlation with primitive transcription factor expression (Figure A9F and Figure 10K).

Figure 10. Erythroid bias of primitive HSCs correlated with their proportion in time series of irradiation recovery. (GSE278673) (A) LSK phenotype bone marrow cells with clusters annotated as HSC (Hematopoietic Stem Cell), MPP2/3 (Multipotent Progenitors 2/3), MPP4 (Multipotent Progenitors 4), MK (Megakaryocyte), Ery (Erythrocyte), Ma (Mast cell), Neu (neutrophil), Mo (monocyte), DC (dendritic cell), B (B cell) and T (T cell). Different colors represent distinct cell subpopulations throughout the analysis. (B) Single-cell atlas of irradiation-injured LSK at time points 2, 5, 8, 11, 14, 21 and 30 days post-radiation, colored with annotated cell types. (C) Box plot of pseudotime distribution at each time point post-irradiation. (D) HSPC (Hematopoietic Stem and Progenitor Cell: HSC, MPP2/3, MPP4) proportion at each time point. (E) Erythroid cells (MK, Ery, Ma) proportion at each time point. (F) Myeloid cells (Neu, Mo, DC) proportion at each time point. (G) Erythroid lineage decision value projected on UMAP space at different time points. (H) Myeloid lineage decision value projected on UMAP space at different time points. (I) Lineage decision atlas for each time point within HSPC populations, showing erythroid decision values (red dots) and myeloid decision values (blue dots) along uniformly 50-binned pseudotime. (J) Horizontal bar plot of erythroid-biased proportion (red) and myeloid-biased proportion (blue) of HSPC at different time points. (K) Horizontal bar plot of Pearson correlations between erythroid-biased and myeloid-biased HSPC proportions at each time point with respect to average pseudotime, HSPC proportion, erythroid proportion and myeloid proportion.

Figure 10. Erythroid bias of primitive HSCs correlated with their proportion in time series of irradiation recovery. (GSE278673) (A) LSK phenotype bone marrow cells with clusters annotated as HSC (Hematopoietic Stem Cell), MPP2/3 (Multipotent Progenitors 2/3), MPP4 (Multipotent Progenitors 4), MK (Megakaryocyte), Ery (Erythrocyte), Ma (Mast cell), Neu (neutrophil), Mo (monocyte), DC (dendritic cell), B (B cell) and T (T cell). Different colors represent distinct cell subpopulations throughout the analysis. (B) Single-cell atlas of irradiation-injured LSK at time points 2, 5, 8, 11, 14, 21 and 30 days post-radiation, colored with annotated cell types. (C) Box plot of pseudotime distribution at each time point post-irradiation. (D) HSPC (Hematopoietic Stem and Progenitor Cell: HSC, MPP2/3, MPP4) proportion at each time point. (E) Erythroid cells (MK, Ery, Ma) proportion at each time point. (F) Myeloid cells (Neu, Mo, DC) proportion at each time point. (G) Erythroid lineage decision value projected on UMAP space at different time points. (H) Myeloid lineage decision value projected on UMAP space at different time points. (I) Lineage decision atlas for each time point within HSPC populations, showing erythroid decision values (red dots) and myeloid decision values (blue dots) along uniformly 50-binned pseudotime. (J) Horizontal bar plot of erythroid-biased proportion (red) and myeloid-biased proportion (blue) of HSPC at different time points. (K) Horizontal bar plot of Pearson correlations between erythroid-biased and myeloid-biased HSPC proportions at each time point with respect to average pseudotime, HSPC proportion, erythroid proportion and myeloid proportion.

4. Discussion

The adaptation of reinforcement learning methodologies to biological systems represents a significant advancement in computational biology, addressing a fundamental gap in single-cell trajectory analysis: the identification of where and when fate decisions occur during cellular differentiation [45,46]. While traditional approaches primarily focus on ordering cells along developmental trajectories, scRL’s actor–critic framework recasts differentiation as a sequential decision process, enabling quantitative assessment of fate intensity and regulatory checkpoints that precede overt lineage commitment [32,33]. This paradigm shift from descriptive to mechanistic analysis represents a crucial step toward understanding the decision-making logic that governs cellular differentiation in both healthy and diseased contexts.

scRL’s superior performance across diverse biological systems—including human hematopoiesis, acute myeloid leukemia, mouse endocrinogenesis and perturbation studies—demonstrates its versatility in capturing early decision states that conventional methods fail to detect. By integrating LDA-derived interpretable latent spaces with biologically informed reward functions, scRL’s critic network learns state-value functions that quantify lineage potential, while the actor component traces optimal developmental routes across the manifold. This dual approach enables the framework to balance exploration of novel differentiation pathways with exploitation of established routes, consistently outperforming fifteen state-of-the-art methods across five independent evaluation dimensions. The identification of previously uncharacterized regulators such as Dapp1 further validates scRL’s capacity to uncover biologically meaningful insights that extend beyond trajectory reconstruction.

A critical consideration in single-cell trajectory inference is the potential confounding effect of doublets—cellular aggregates that arise during droplet-based capture and may create spurious intermediate states with hybrid transcriptional profiles [47,48]. However, rigorous quality control measures implemented in preprocessing pipelines substantially mitigate doublet-induced artifacts, as these aggregates typically exhibit abnormally elevated transcript counts enabling their identification and removal through standard filtering criteria. Any residual doublets manifest as sporadic noise points rather than coherent cellular populations, appearing as isolated outliers in two-dimensional manifold representations that minimally impact the structural integrity of inferred developmental pathways.

The reliance on two-dimensional embeddings, while computationally tractable and visually interpretable, represents a pragmatic compromise between analytical precision and biological interpretability [49,50]. Although high-dimensional single-cell data exhibit complex nonlinear structure that may not be fully preserved in 2D projections, the interpretability advantages substantially outweigh potential inaccuracies introduced by dimensional compression. This strategic design choice enables meaningful biological discovery and experimental validation within a comprehensible framework that facilitates integration of computational predictions with experimental observations.

While scRL’s graph-based pseudotime alignment provides robust temporal ordering across grid representations, several methodological limitations merit consideration. The requirement for user-defined starting points introduces subjectivity that may bias trajectory inference, as suboptimal selection of early cells or clusters propagates errors throughout Dijkstra’s shortest-path calculations [51]. More fundamentally, the assumption that shortest-path distances correspond to developmental time may not accurately reflect cellular differentiation dynamics, particularly in systems with rapid state transitions or non-monotonic developmental patterns [52,53]. Grid discretization compounds these issues by introducing artifacts through rasterization of continuous UMAP embeddings, potentially creating artificial barriers or connections that misrepresent the underlying biological manifold [54]. The multi-component handling strategy, despite its mathematical elegance, introduces approximation errors when connecting disconnected graph components, potentially mischaracterizing relationships between separated cellular populations [55]. Finally, the static, undirected graph structure overlooks the inherently directional nature of developmental processes, while the absence of uncertainty quantification limits reliability assessment, particularly in sparsely sampled regions where multiple plausible trajectories may exist [56].

The selection of grid embedding hyperparameters n and j critically influences grid construction and computational efficiency. Based on systematic parameter evaluation across grid resolution n (25, 50, 75, 100, 125) and border observer parameter j (3, 5, 8, 11, 15), we recommend $n=50$ and $j=3$ as optimal choices that balance embedding accuracy with computational tractability (Figure A10).

Beyond trajectory inference, scRL’s unified framework addresses broader challenges in developmental biology by providing competitive measures of lineage-contribution intensity without requiring ground-truth probabilities. This capability, combined with its ability to reconcile gene- and lineage-level information across healthy, diseased and perturbed systems, positions scRL as a versatile tool for regenerative medicine, oncology and developmental biology. The framework’s extensibility toward multi-omic and spatial modalities, coupled with ongoing efforts to streamline unsupervised identification of key subpopulations, promises to advance our mechanistic understanding of cell fate decisions and their therapeutic manipulation.

5. Conclusions

scRL brings reinforcement learning into single-cell biology, offering a principled, data-driven approach for charting differentiation landscapes and pinpointing regulatory checkpoints. Its ability to capture early decision states, reconcile gene- and lineage-level information, and generalize across healthy, diseased and perturbed systems positions scRL as a versatile tool for developmental biology, oncology and regenerative medicine. Ongoing work aims to integrate multi-omic and spatial modalities and to streamline unsupervised identification of key subpopulations, extending scRL’s reach toward a comprehensive, mechanistic atlas of cell fate decisions.