Dual-Domain Joint Learning Reconstruction Method (JLRM) Combined with Physical Process for Spectral Computed Tomography (SCT)

Abstract

Spectral computed tomography (SCT) enables material decomposition, artifact reduction, and contrast enhancement, leveraging symmetry principles across its technical framework to enhance material differentiation and image quality. However, its nonlinear data acquisition process involving noise and scatter leads to a highly ill-posed inverse problem. To address this, we propose a dual-domain iterative reconstruction network that combines joint learning reconstruction with physical process modeling, which also uses the symmetric complementary properties of the two domains for optimization. A dedicated physical module models the SCT forward process to ensure stability and accuracy, while a residual-to-residual strategy reduces the computational burden of model-based iterative reconstruction (MBIR). Our method, which won the AAPM DL-Spectral CT Challenge, achieves high-accuracy material decomposition. Extensive evaluations also demonstrate its robustness under varying noise levels, confirming the method’s generalizability. This integrated approach effectively combines the strengths of physical modeling, MBIR, and deep learning.

1. Introduction

Spectral computed tomography (SCT) has emerged as a pivotal imaging modality with growing applications in clinical diagnostics and industrial nondestructive testing. Unlike conventional CT systems (illustrated in Figure 1), which focus on reconstructing precise anatomical images through multi-angle scanning, SCT fundamentally addresses the critical yet often overlooked polychromatic nature of X-ray beams. This inherent spectral broadening phenomenon introduces nonlinear effects and ill-posed challenges in quantitative imaging reconstruction. The paradigm shift in SCT implementation involves strategic spectral differentiation through two primary approaches: (1) multi-spectral scanning using varied X-ray spectra, or (2) spectral separation from broad-spectrum emissions. Current spectral data acquisition methodologies encompass (1) kVp-switching techniques [1], (2) dual-source scanning systems [2], (3) dual-layer detector configurations, (4) sequential two-pass scanning protocols, and (5) photon-counting detectors [3,4]. These technological advancements have enabled sophisticated processing of polychromatic projection data, facilitating energy-resolved material decomposition through established algorithms [5,6,7,8]. The clinical significance of SCT lies in its dual capability; enhanced material discriminability through spectral signatures and precise quantification of mass density ratios in multi-component systems are particularly beneficial for tissue analysis and composite material characterization [9].

Compared to conventional CT systems, the reconstruction challenges in spectral computed tomography (SCT) demonstrate inherent nonlinearity and ill-posedness stemming from the subtle spectral differences during spectral separation, further exacerbated by potential geometric inconsistencies in real projection data. Current algorithmic approaches to address these challenges can be systematically classified using two taxonomies: (1) in the processing domain, methods are divided into image domain approaches [8,9,10,11,12,13,14] and projection domain methods [15,16]; (2) among decomposition strategies, methods can be categorized as direct mapping methods [5,10,11,12,13,14,15,17,18,19], iterative algorithms [16,20,21,22,23,24,25,26,27,28,29,30,31,32,33], or deep learning methods [34,35,36,37,38,39,40,41,42].

While image domain methods utilize diverse basis functions [10,11,15,18] to approximate nonlinear effects, they are fundamentally constrained by first-order linear assumptions, leading to persistent beam-hardening artifacts. Projection domain approaches theoretically allow for superior artifact suppression through nonlinear material decomposition [43,44]; however, they demand stringent geometric consistency that is rarely achievable in practical SCT implementations. Direct mapping techniques prioritize computational efficiency while compromising spectral information and exhibiting phantom-dependent accuracy. Iterative algorithms, including one-step spectral CT reconstruction approaches [32,33], are effective for ill-posed inverse problems; however, they confront computational bottlenecks in handling multi-spectral geometric inconsistencies and hyperparameter tuning challenges [45]. Deep learning methods achieve remarkable reconstruction quality given sufficient training data, but struggle with clinical data acquisition limitations and generalizability issues. The AAPM DL-Spectral CT Challenge [46] provides a critical benchmarking platform comparing data-driven and iterative approaches in terms of their ability to address spectral CT’s incomplete measurement problem, thereby promoting methodological advancements through standardized assessment.

This study examines whether deep neural networks can achieve accurate material-specific image decomposition to establish the foundation for data-driven deep learning-based SCT reconstruction algorithms [45]. We propose a novel iterative network architecture that integrates SCT physical modeling with model-based iterative reconstruction (MBIR) algorithms. Although the AAPM Challenge provides detailed physical processes (complete spectral and attenuation coefficient information), our approach reformulates the physical process and its associated parameters as learnable network components for enhanced flexibility. Following the sparse-view CT methodology within DL [47], high-accuracy reconstruction requires explicit incorporation of the forward model within the reconstruction framework, as demonstrated by iterative data consistency optimization. The proposed framework embeds physical processes via physics-constrained modules to enable precise forward modeling during iterative steps, ensuring correct residual propagation. For clinical noisy data, we implement adaptive regularization strategies to improve practical applicability. Our code is publicly available at JLRM code: https://github.com/GM0322/JLRM (accessed on 10 November 2024).

2. Preliminary Knowledge

The mathematical formulation of the SCT problem can be expressed as solving for $\mu (\overrightarrow{x},E)$ from K sets of polychromatic projection data under distinct spectra:

$$q_{k,L}=-ln\int S_k\left(E\right)e^{-{\int }_{t\in L}\mu (\overrightarrow{x},E)dt}dE+{ϵ}_{k,L},k=1,2,\dots K$$

(1)

where L denotes an X-ray projection path, $\mu (\overrightarrow{x},E)$ represents the linear attenuation coefficient at spatial position $\overrightarrow{x}=(x_1,x_2)$ with energy E, and $S_k\left(E\right)$, $k=1,2,\dots ,K$ is the normalized effective spectrum satisfying $\int S_k\left(E\right)dE=1$. The perturbation term ${ϵ}_{k,L}$ accounts for scatter/noise effects. The linear attenuation coefficient in SCT is typically decomposed as a linear combination of some predefined basis functions

$$\mu (\overrightarrow{x},E)=\sum _{i=1}^M{\theta }_i\left(E\right)f_i\left(\overrightarrow{x}\right),$$

(2)

where ${\theta }_i\left(E\right)$ are energy-dependent coefficients and $f_i\left(\overrightarrow{x}\right)$ correspond to spatial distributions. Symmetry here ensures complete spectral coverage, enabling quantitative analysis. This formulation has dual physical interpretations. (1) Physical effect decomposition: for $M=2$ with basis functions derived from photoelectric and Compton scattering effects [5,6], we have ${\theta }_1\left(E\right)=E^3$ (photoelectric) and ${\theta }_2\left(E\right)=f_{KN}\left(E\right)$ (Klein-Nishina function); then, ${\theta }_1\left(E\right)f_1\left(x\right)$ and ${\theta }_2\left(E\right)f_2\left(x\right)$ respectively indicate the photoelectric part and Compton scattering part of the attenuation. The reconstructed $f_i\left(x\right),i=1,2$ respectively quantify the spatial distributions of these effects, enabling computation of electron density and effective atomic number. (2) Basis material decomposition: ${\theta }_i\left(E\right)$ are interpreted as mass attenuation coefficients of selected basis materials, while $f_i\left(\overrightarrow{x}\right)$ represent the corresponding material density distributions, with M specifying the number of basis materials.

Substituting (2) into (1) provides

$$q_{k,L}=-ln\int S_k\left(E\right)e^{-{\int }_{t\in L}\sum _{i=1}^M{\theta }_i\left(E\right)f_i\left(\overrightarrow{x}\right)dt}dE+{ϵ}_{k,L},k=1,2,\dots K.$$

(3)

The discrete formulation of SCT is then expressed as

$$q_{k,L}=-ln\sum _{E=1}^{N_k}{S}_k\left(E\right)e^{-{\displaystyle \sum _{i=1}^M}{\theta }_{i,k}\left(E\right)R_k{f}_i\left(\overrightarrow{x}\right)}+{ϵ}_{k,L},k=1,2\dots K,$$

(4)

where $R_k,k=1,2,\dots ,K$, represents the $k-$th Radon transform. In the simplest terms, we can simplify this to $q_{k,L}=q\left(f\left(\overrightarrow{x}\right)\right)$.

3. Method

3.1. JLRM Architecture

We formulate SCT reconstruction as a nonlinear optimization problem to solve basis material images $f{\overrightarrow{x}}^{★}$ from acquired polychromatic projections $q^{★}$:

$$f\left({\overrightarrow{x}}^{★}\right)=arg\underset{f\left(\overrightarrow{x}\right)}{min}{\parallel q\left(f\left(\overrightarrow{x}\right)\right)-q^{★}\parallel }_2^2+{\lambda }_1{\mathcal{R}}_1\left(q\left(f\left(\overrightarrow{x}\right)\right)\right)+{\lambda }_2{\mathcal{R}}_2\left(f\left(\overrightarrow{x}\right)\right)$$

(5)

where $q\left(\overrightarrow{x}\right)$ corresponds to the nonlinear SCT physical process and ${\mathcal{R}}_1$ and ${\mathcal{R}}_2$ are network regularization terms in the projection and image domains, respectively, used in case of noise or scattering. In this way, ${\mathcal{R}}_1$ can more accurately describe the physical model for cases with noise or scattering, as ${\mathcal{R}}_2$ can effectively suppress the noise or scattering to obtain better reconstructed images.

Here, we adopt a simple alternating minimization [48,49] and incremental learning technique [50] to solve (5). Let $f^i$ denote the solution after the $i-$th iteration; then, $f^{i+1}$ can be obtained by solving the following two subproblems sequentially:

Subproblem 1: Projection domain

$$q^{i+1}=arg\underset{q}{min}{\parallel q\left(f^i\right)-q^{★}\parallel }_2^2+{\lambda }_1{\mathcal{R}}_1\left(q\left(f^i\right)\right).$$

Subproblem 2: Image domain

$$f^{i+1}=arg\underset{f}{min}{\parallel q^{i+1}\left(f\right)-q^{★}|}_2^2+{\lambda }_2{\mathcal{R}}_2\left(f\left(\overrightarrow{x}\right)\right).$$

For Subproblem 1, under noise-free conditions, the minimization of $q^{i+1}=arg\underset{q}{min}{\parallel q\left(f^i\right)-q^{★}\parallel }_2^2$ is implemented through a network architecture that exclusively contains the physical process module (as detailed in Section 3.1.1). When both the spectral information and attenuation coefficients are unknown, the trainable parameters in this network correspond to these spectral and attenuation components. Notably, when the spectrum is known or fixed, this network module becomes non-trainable with no learnable parameters, and this part does not require training. In noisy scenarios, the optimization comprises two components: (1) the physics-based forward module, and (2) a neural regularization item that suppresses noise and scattering artifacts in the projections generated by the physical module. To achieve noise suppression, we design an iterative network architecture in which the input $f^i$ first generates $q^i$ through physical modeling, which is then processed by a neural module to produce refined projections. These neural-processed results undergo weighted summation with the original $q^i$ to yield $q^{i+1}$:

$$q^{i+1}={\lambda }_1·{\mathcal{R}}_1\left(q^i\right)+q^i.$$

(6)

The final loss function computes the mean squared error (MSE) between the enhanced projection and the clean reference:

$$\mathcal{L}=\parallel q^{i+1}-q^{\prime }{\parallel }_2^2$$

(7)

with $q^{\prime }$ representing the ground truth, where $q^{\prime }=q^{★}$ for the noise-free case. In summary, the projection domain optimization process combining physical modeling and neural regularization can be formally expressed as follows:

$$q^{(i+1)}=\underset{\mathrm{Physical}\mathrm{forward}\mathrm{projection}}{\underbrace{q^i\left(f^{\left(i\right)}\left(\overrightarrow{x}\right)\right)}}+{\lambda }_1\underset{\mathrm{Neural}\mathrm{regularization}\mathrm{operator}}{\underbrace{{\mathcal{R}}_1\left(q\left(f^{\left(i\right)}\left(\overrightarrow{x}\right)\right)\right)}}.$$

(8)

For Subproblem 2, the same optimization framework could theoretically be applied; however, empirical studies revealed that this sequential approach suffers from catastrophic forgetting in which previously corrected material decompositions tend to regress in subsequent iterations. To address this limitation, we implement a continual learning paradigm with incremental learning retention [51]:

$$f^{i+1}=f^i+\Delta f$$

(9)

where the incremental update $\Delta f$ is governed by the dual components shown below.

$$\Delta f=\underset{\mathrm{Data}\mathrm{fidelity}\mathrm{term}}{\underbrace{\nabla \parallel q^{i+1}\left(f\right)-q^{★}{|}_2^2}}+{\lambda }_2\underset{\mathrm{Regularization}\mathrm{item}}{\underbrace{\nabla {\mathcal{R}}_2\left(f\left(\overrightarrow{x}\right)\right)}}$$

(10)

For the data fidelity pathway, this dual-pathway architecture features a combination of model-based iterative reconstruction with decomposition networks. This is expressed as $CNN$($MBIR(q^{(i+1)}-q^{★}))$, where $CNN$ is used to realize the decomposition of increments in the image domain; this part corresponds to our residual-to-residual learning strategy. The second pathway consists of regularization to further adaptively optimize noisy data.

In summary, the optimization problem can be decomposed into the following four steps:

• Step 1. Obtain initial value: $f^{\left(0\right)}=(f_1^{\left(0\right)},f_2^{\left(0\right)},f_3^{\left(0\right)})$
• Step 2. Nonlinear physical process: $q^{(i+1)}=q^i\left(f^{\left(i\right)}\left(\overrightarrow{x}\right)\right)+{\lambda }_1{\mathcal{R}}_1\left(q\left(f^{\left(i\right)}\left(\overrightarrow{x}\right)\right)\right)$
• Step 3. Residual increment: $\Delta f^{\left(i\right)}\left(\overrightarrow{x}\right)=$CNN(MBIR$(q^{(i+1)}-q^{★}))+{\lambda }_2\nabla {\mathcal{R}}_2\left(f^{\left(i\right)}\left(\overrightarrow{x}\right)\right)$
• Step 4. Update: $f^{(i+1)}\left(\overrightarrow{x}\right)=f^{\left(i\right)}\left(\overrightarrow{x}\right)+\Delta f^{\left(i\right)}\left(\overrightarrow{x}\right)$.

In our experiments, the MBIR uses the SART algorithm during the iteration process, which the other algorithm also support. The third step is worth noting; fe first calculate the projection residuals for reconstruction, then the results are input to the CNN to obtain the residuals of the base material images. In other words, we use s residual-to-residual strategy for network training, which can more effectively improve the image quality. In addition, it is important to note that the regularization term needs to be designed to make the method more suitable for situations with noisy data or scattering. For noise-free data, it is not necessary to use the regularization term.

Generally, the initial value has a pronounced influence on the convergence speed of the iterative algorithm; therefore, we incorporate a pre-decomposition module prior to the iterative process.

Our architecture integrates a spectral pre-decomposition module and an iterative reconstruction workflow, as visualized in Figure 2 under noise-free conditions. The pre-decomposition stage employs a hybrid MBIR-CNN pipeline, where the E-ART algorithm—notable for its spectral insensitivity and superior reconstruction fidelity compared to conventional analytical methods—provides robust initialization. For the iterative process, the three steps of the incremental method are consistent. As mentioned above, we use a residual-to-residual strategy for effective network training. Below, we show the pseudocode for the training stage (Algorithm 1) and test stage (Algorithm 2).

Algorithm 1 Training stage

1: Input: projection data $q=(q_1,q_2)$, ground truth data $f^{★}=(f_1^{★}\left(x\right),f_2^{★}\left(x\right),f_3^{★}\left(x\right))$, iteration number n, relaxation factor ${\omega }_1$ of E-ART and ${\omega }_2$ of ART, initial value $f^{\left(0\right)}=0$. 2: Output: $f^{n+1}=(f_1^{n+1},f_2^{n+1},f_3^{n+1})$ 3: # step 1: Pre-decomposition (projection domain to image domain) 4: $f^{\left(0\right)}=(f_1^{\left(0\right)},f_2^{\left(0\right)},f_3^{\left(0\right)})$       $=CN{N}_1($E-ART$(q;{\omega }_1,f^{\left(0\right)});{\mathsf{\Theta }}_1^{\left(0\right)},{\mathsf{\Theta }}_2^{\left(0\right)},{\mathsf{\Theta }}_3^{\left(0\right)})$ 5: $loss\_function=MSELoss(f^{\left(0\right)},f^{★})$ 6: train the networks ${\mathsf{\Theta }}_1^{\left(0\right)},{\mathsf{\Theta }}_2^{\left(0\right)},{\mathsf{\Theta }}_3^{\left(0\right)}$ 7: # step 2: iteration for residual-to-residual assign 8: for $j=0,2.\dots n$ do 9:        # nonlinear physical module (image domain to projection domain) 10:      $q_{k.L}^{(j+1)}\left(\overrightarrow{x}\right)=-ln{\displaystyle \sum _{E=1}^{N_k}}{S}_k\left(E\right)e^{-{\displaystyle \sum _{i=1}^M}{\theta }_{i,k}\left(E\right)R_{i,k}{f}_i^{\left(j\right)}\left(\overrightarrow{x}\right)}+{ϵ}_k,k=1,2,\dots ,K$ 11:      # residual-to-residual assign (projection domain to image domain) 12:      $\Delta f^{\left(j\right)}=$$CN{N}_2$(ART$(q^{(j+1)}-q^{★};{\omega }_2,f^j),f_1^{\left(j\right)},f_2^{\left(j\right)},f_3^{\left(j\right)}$;              ${\mathsf{\Theta }}_1^{\left(j\right)},{\mathsf{\Theta }}_2^{\left(j\right)},{\mathsf{\Theta }}_3^{\left(j\right)})$ 13:      $loss\_function=MSELoss(\Delta f^{\left(j\right)},f^{★}-f^{\left(j\right)}$) 14:      train the networks ${\mathsf{\Theta }}_1^{\left(j\right)},{\mathsf{\Theta }}_2^{\left(j\right)},{\mathsf{\Theta }}_3^{\left(j\right)}$ 15:      # update basis material images 16:      $f_1^{(j+1)}=f_1^{\left(j\right)}+\Delta f_1^{\left(j\right)},f_2^{(j+1)}=f_2^{\left(j\right)}+\Delta f_2^{\left(j\right)},f_3^{(j+1)}=f_3^{\left(j\right)}+\Delta f_3^{\left(j\right)}$ 17: end for

Algorithm 2 Test stage

1: Input: projection data $q=(q_1,q_2)$ iteration number n, relaxation factor ${\omega }_1$ of E-ART and ${\omega }_2$ of ART. 2: Output: $f^{n+1}=(f_1^{n+1},f_2^{n+1},f_3^{n+1})$ 3: # step 1: Pre-decomposition 4: $f^{\left(0\right)}=(f_1^{\left(0\right)},f_2^{\left(0\right)},f_3^{\left(0\right)})$       $=CN{N}_1($E-ART$(q;{\omega }_1,f^{\left(0\right)});{\mathsf{\Theta }}_1^{\left(0\right)},{\mathsf{\Theta }}_2^{\left(0\right)},{\mathsf{\Theta }}_3^{\left(0\right)})$ 5: # step 2: Iteration for residual-to-residual assign 6: for$j=0,2.\dots n$  do 7:      # nonlinear physical module (image domain to projection domain) 8:      $q_{k.L}^{(j+1)}\left(\overrightarrow{x}\right)=-ln{\displaystyle \sum _{E=1}^{N_k}}{S}_k\left(E\right)e^{-{\displaystyle \sum _{i=1}^M}{\theta }_{i,k}\left(E\right)R_{i,k}{f}_i^{\left(j\right)}\left(\overrightarrow{x}\right)}+{ϵ}_k,k=1,2,\dots ,K$ 9:      # residual-to-residual assign (projection domain to image domain) 10:      $\Delta f^{\left(j\right)}=$$CN{N}_2$(ART$(q^{(j+1)}-q^{★};{\omega }_2,f^j),f_1^{\left(j\right)},f_2^{\left(j\right)},$              $f_3^{\left(j\right)};{\mathsf{\Theta }}_1^{\left(j\right)},{\mathsf{\Theta }}_2^{\left(j\right)},{\mathsf{\Theta }}_3^{\left(j\right)})$ 11:      # update basis material images 12:      $f_1^{(j+1)}=f_1^{\left(j\right)}+\Delta f_1^{\left(j\right)},f_2^{(j+1)}=f_2^{\left(j\right)}+\Delta f_2^{\left(j\right)},f_3^{(j+1)}=f_3^{\left(j\right)}+\Delta f_3^{\left(j\right)}$ 13: end for

The proposed method is based on the complementary advantages of physical processing, MBIR, and deep learning. Physical processing realizes the accurate expression of physical phenomena, MBIRs achieve domain transformation, and deep learning can accurately perform material decomposition. The three parts are interdependent and complementary.

3.1.1. Physical Module for Nonlinear Physical Process

Only when the physical process can be accurately described by the model can the iterative reconstruction process continuously improve reconstruction accuracy. Here, to accurately describe the nonlinear physical process of SCT, its forward process is equivalently mapped into a network module. For noise-free cases, the discrete formulation of SCT can be expressed as Equation (4). As shown in Figure 3, we first perform a Radon transform for three basis materials. This can be achieved by a projection layer [52,53]. Here, we have overridden the interface where projection and back-projection operators are provided by the official framework. Next, the module is constructed according to the discrete polychromatic projection formula, where the spectral information and attenuation coefficients are defined as learnable parameters. Both sets of parameters are implemented through convolution operators. For the attenuation coefficients, this layer uses K convolution kernels with dimensions $N_k\times M\times 1\times 1$ ($ou{t}_{channels}\times i{n}_{channels}\times height\times width$). For spectral information, the K convolution kernels have dimensions $1\times N_k\times 1$. In noisy scenarios, a regularization network is added to the physical module to better model the physical process. It should be noted that because the competition provides the energy spectrum information, we do not update it after initialization during the network process.

3.1.2. CNN Network Structure Design

The network structure is also critical to the reconstruction results, since our network is trained module-to-module. If overfitting occurs, the results of subsequent iterations are disastrous. The CNNs for the pre-decomposition and iterative processes are structurally similar, with architectures as displayed in Figure 4. The biggest difference between them lies in the input channel configuration: CNN₁ uses two input channels, whereas CNN₂ employs six channels. Specifically, the CNN can be regarded as two segments. The left branch of the CNN architecture includes four blocks forming a stacked encoder, which implements a sequence of convolution, group normalization (GN), and ReLU activation layers. Given the input x and output y of one block, this process can be expressed as

$$y=ReLU\left(GN\left(C_3\left(ReLU\left(GN\left(C_2\left(ReLU\left(GN\left(C_1\left(x\right)\right)\right)\right)\right)\right)\right)\right)\right),$$

where $C_1,C_2,C_3$ are convolutional layers with kernel sizes of $3\times 3,1\times 1$, and $3\times 3$, respectively. The right branch of the CNN architecture includes four blocks as a stacked decoder. For each block, we first concatenate the feature map of the corresponding block on the left to achieve information fusion. Defining the last output as x, the output of the left branch as $x_0$, and the output as y, then $y=concatenation(C\left(x\right),x_0)$. A convolutional block identical to the stacked encoder is then applied.

It is worth noting that network operations such as downsampling, upsampling, and transpose convolution—which enlarge the receptive field—cannot be used. Although the projections are inconsistent and the pixels of basis material images exhibit domain-specific correlations, the spatial coherence caused by projection inconsistency is inherently limited. Enlarging the receptive field would exacerbate overfitting, a critical flaw that severely undermines the ability of subsequent iterations to achieve accurate material decomposition.

3.1.3. Residual-to-Residual Strategy

Based on the input and output images of our network, we can define training strategies such as residual-to-image, image-to-image, and others. For example, when the input is an image reconstructed from projection data and the output is the residual of basis material images, we define this as an image-to-image strategy; similarly, when the input is the residual of projection data reconstruction and the output is the residual of basis material images, we define this as a residual-to-residual training strategy.

The residual-to-residual strategy proposed based on experimental observations can effectively improve the accuracy of the reconstruction. As shown in Figure 5, regions with large errors which are difficult to identify in reconstructed images can be directly located in the residual image. Hence, the image-to-residual strategy may introduce new errors, introducing uncertainties that hinder quality improvement of reconstructed images; in other words, image-to-residual is less reliable for improving the image quality.

The commonly used ReLU activation function makes the images obtained by the network non-negative if the last layer of network uses the ReLU function. However, the residual image does not possess this property. In order to use the nonlinear ReLU activation function, we decompose the residual image $dx$ into the following form:

$$dx=d{x}^{+}-d{x}^{-}$$

(11)

where $d{x}^{+}=max\{dx,0\}$ and $d{x}^{-}=max\{-dx,0\}$ are the positive and negative parts of the residual image, respectively. Thus, the network output becomes an image with two non-negative channels.

3.2. Network Training

The proposed method was implemented using Python (version 3.9) under the PyTorch (version 2.3) framework. Each module was trained module-by-module, and the network was optimized using the Adam optimization method with $batch\_size=9$ for each iteration. In order to make the network more generalizable, we used the method of rotation, translation, and mirroring to enhance the data. At the same time, we used a single learning rate (lr = 1 × 10⁻⁴) for learning. The objective function for each module used an MSE loss function:

$$\underset{\mathsf{\Theta }}{min}L\left(\mathsf{\Theta }\right)=\sum _{i=1}^N\parallel CNN(x,\mathsf{\Theta })-y^{*}\parallel$$

(12)

where $x,y^{*}$ are the input/label data, $\mathsf{\Theta }$ denotes the network learnable parameters, and i is the traversal of the image. The complete training samples were randomly shuffled and the number of training epochs was set to 1000 as an empirical stopping criterion of each module. Comparative experiments were performed on a PC (AMD EPYC-Rome processor, Taichung, Taiwan/CN 128 GB RAM). Training was executed on a single-GPU Nvidia RTX A6000 (Santa Clara, CA, USA) with 48 GB of video memory.

We specifically emphasize the module-by-module training strategy, in which each component is independently parameterized. Below, we list the inputs and output of each module.

Pre-decomposition phase:

• Input: EART-decomposed basis material images
• Processing: Three dedicated networks individually process the inputs
• Output: Refined images of three basis materials

Physical Module:

• Condition: Activated when spectral information is unknown
• Input: Polychromatic projection data paired with reference labels
• Function: Updates the spectral information through physics-constrained learning

Residual-to-Residual Module:

• Input: Reconstructed images from projection residual and current basis material images
• Processing: Three separate networks generate material-specific residuals
• Output: Corrected residual images for each basis material

It is worth noting that all modules undergo isolated parameter updates without cross-module gradient interference.

4. Results

Experiments were carried out to demonstrate the behavior of the proposed method, including architecture validation, residual analysis, the performance of JLRM in noise suppression, the reconstruction method, the CNN network structure design, and the residual-to-residual strategy.

4.1. DL-Spectral CT Setup of AAPM Challenge

This challenge focuses on spectral CT, in which objects are scanned using X-rays with two spectra (i.e., $K=2$) under noise-free conditions. This challenge models an idealized form of fast kVp-switching, where the X-ray tube voltage alternates between two settings for consecutive projections. For each kVp setting, the beam spectrum is broad, and the polychromatic nature of the X-ray beam makes quantitative CT imaging particularly challenging when only one kVp setting is used. Scanned objects are simulated based on a breast model [54], which contains three tissue types ($M=3$): adipose, fibroglandular, and calcification. Ground truth images and simulated data are provided for 1000 cases, enabling participants to choose between data-driven or optimization-based techniques. The SCT imaging process is illustrated in Figure 6, which shows the spectral information and material-specific linear attenuation coefficients. Notably, the linear attenuation coefficients of the two materials are very close, a key challenge discussed in the experimental results. To ensure fairness, the test dataset includes ten additional cases provided in a “starting kit” with ground-truth images. The fan-beam geometric parameters of the SCT system are listed in

Table 1. System and geometric scanning parameters of the AAPM DL-Spectral CT Challenge dataset.

X-ray source to rotation center (mm)	500
X-ray source to detector distance (mm)	1000
Scanning angle (deg)	[0°, 360°]
Scanning angular interval (deg)	$1.40625$
Number of detector units	1024
Detector unit size (mm)	0.3574
Reconstructed image size (pixel)	$512\times 512$

. The DL-Spectral-CT Challenge aims to identify the image reconstruction algorithm that achieves the most accurate spatial maps of adipose, fibroglandular, and calcification from SCT projection data. Further details about the challenge setup and results are available in the official report [46].

4.2. Performance and Analysis of JLRM

We demonstrate that JLRM can be trained to accurately reconstruct basis material images from collected projection data in the DL-Spectral CT Challenge dataset. As described in Section 3, the JLRM method includes a pre-decomposition module and two iterative refinement modules. Figure 7 shows the basis material images these three modules. It is difficult to visually distinguish the image quality produced by each module. The last row displays the tissue label map of the three materials, revealing the proportion of each material per pixel. Figure 8 shows the difference images. The accuracy of the proposed method can be further improved, as the iterative modules continuously enhance reconstruction quality through comparison with differential images. Even in a display window approaching machine precision, the difference images after two iterations show negligible errors, validating the near-exact reconstruction capability of our method.

Table 2. Average RMSE for quantitative evaluation.

		Pre-Decomposition	1 Iteration	2 Iterations
adipose	training set	$2.53\times {10}^{-3}$	$9.24\times {10}^{-5}$	$6.79\times {10}^{-6}$
	test set	$2.87\times {10}^{-3}$	$7.09\times {10}^{-5}$	$7.54\times {10}^{-7}$
fibroglandular	training set	$2.68\times {10}^{-3}$	$1.20\times {10}^{-4}$	$5.54\times {10}^{-6}$
	test set	$3.06\times {10}^{-3}$	$9.74\times {10}^{-5}$	$6.67\times {10}^{-7}$
calcification	training set	$3.30\times {10}^{-7}$	$6.60\times {10}^{-7}$	$3.00\times {10}^{-7}$
	test set	$6.40\times {10}^{-7}$	$6.30\times {10}^{-7}$	$6.60\times {10}^{-7}$

provides the average RMSE for quantitative evaluation, confirming the accuracy of our reconstruction. The final submitted result in the AAPM challenge achieved an accuracy of $6.8\times {10}^{-7}$—approaching the single-precision floating-point limit (∼${10}^{-7}$). Compared to the second-place accuracy ($6.21\times {10}^{-6}$), our method demonstrates an order-of-magnitude improvement.

To better understand the behavior of each module in the proposed method, Figure 9 shows the loss function update curves for each material across modules. In the pre-decomposition module, the adipose and fibroglandular materials fail to achieve machine precision, but a higher proportion of calcification materials achieve machine precision. In the iterative refinement process, the accuracy is significantly improved using the residual-to-residual strategy. It is noteworthy that our framework employs a module-to-module training strategy; the initial 1000 iterations constitute the pretraining phase with exclusive loss updates, followed by two successive optimization windows (iterations 1000–2000 and 2000–3000) corresponding to first and second iterative refinement stages, respectively. Crucially, each training phase recommences with randomly initialized parameters, inherently inducing expected discontinuities in basis material decomposition across different iteration counts due to this deliberate parameter reinitialization protocol.

4.3. Residual Analysis

Because the projection data in the AAPM challenge are noise-free, a crucial criterion for evaluating a proper solver for this DL spectral-CT problem is minimizing the difference images $q_k^n-q_k^{★}$, which should be as low as possible. We analyze this aspect in Figure 10. For enhanced analytical clarity, we reconstructed the residual images from the projection residuals. Figure 11 displays these reconstructed residual images. Notably, the residual images in the image domain exhibit a monotonically decreasing trend. The final iteration results demonstrate that all residual pixel values fall within machine precision.

Some pixels with small residuals (marked by red boxes in Figure 11) cannot be updated effectively in the first iteration, and are shown to be more obvious in the residual image after one iteration, which facilitates further performance improvement. Certain regions with large errors can continuously shrink during the iteration process, although the updates still have larger errors, as marked by the blue boxes.

4.4. Performance of JLRM in Noise Suppression

To ensure universality, we also studied the case of noisy data. The statistics of X-ray measurements are often described using a Poisson distribution to model the incident intensity:

$$I_{noisy,k}\sim Poisson\left(I_{0,k}\times e^{-q_k}\right),k=1,\dots ,K$$

(13)

where $I_{noisy,k}$ and $q_k$ denote the received intensity of detectors and noise-free projection data, respectively. The noisy projection data $q_{noisy,k}=-ln\left({\displaystyle \frac{I_{noisy,k}}{I_{0,k}}}\right)$.

Table 3. Average RMSE of dataset with different noise levels for quantitative evaluation.

	Noise Level	$1\times {10}^5$	$5\times {10}^5$	$1\times {10}^6$
adipose	training set	$7.84\times {10}^{-2}$	$2.87\times {10}^{-2}$	$6.58\times {10}^{-3}$
	test set	$9.97\times {10}^{-2}$	$4.45\times {10}^{-2}$	$7.54\times {10}^{-3}$
fibroglandular	training set	$7.23\times {10}^{-2}$	$3.24\times {10}^{-2}$	$6.69\times {10}^{-3}$
	test set	$9.17\times {10}^{-2}$	$4.68\times {10}^{-2}$	$8.20\times {10}^{-3}$
calcification	training set	$7.07\times {10}^{-5}$	$3.54\times {10}^{-6}$	$2.72\times {10}^{-6}$
	test set	$8.92\times {10}^{-5}$	$5.47\times {10}^{-6}$	$1.03\times {10}^{-6}$

lists the accuracy of the proposed method under different noise levels. As shown in previous experiments, the observations for adipose and fibroglandular materials are consistent, primarily because their attenuation coefficients are nearly identical. Hence, these two materials are prone to decomposition errors, and their performance metrics align closely. The calcification material, however, has a significantly higher attenuation coefficient and a much smaller spatial extent. Even under high noise levels ($I_0=1\times {10}^5$), calcification achieves machine precision, while under lower noise ($I_0=1\times {10}^6$) it converges to this precision rapidly. To better demonstrate the noise characteristics, Figure 12 shows the results of FBP reconstruction applied directly to 80 kV projection data.

To further understand the effect of noise on the decomposition results of adipose and fibroglandular materials, we analyzed a set of results obtained from noise-free pre-decomposition. As in the residual analysis of Section 4.3, the obtained projection domain residuals are reconstructed to obtain the results shown in Figure 13, where the projection residuals are obtained by performing the physical process with the pre-decomposition results minus the noisy projections, i.e., $q_k^n-q_{noisy,k}$. Figure 13 demonstrates that decomposition errors in high-noise regions (e.g., red box, zoomed view in the upper-right corner) are challenging to visualize, whereas in low-noise regions (blue box, zoomed view in the upper-left corner) errors become visually apparent only in noise-free cases. Additionally, the primary decomposition errors in noisy reconstructions are localized to edge regions.

4.5. Reconstruction Method: FBP vs. E-ART

Next, we verify and analyze each part of the module mentioned in Section 3. We analyze reconstruction algorithms, including traditional analytical reconstruction (FBP) and iterative reconstruction algorithms (E-ART). For fairer comparison, we obtain higher-sampled projection data, i.e., 512 views of projection data (original 256 views), according to the projection operator provided by the official. The left two columns of Figure 14 display the reconstruction results of the FBP analytic algorithm. The right column shows the results of original projection data (256 views) using the E-ART iteration algorithm. The last row displays the profile of the mask in the first two rows.

There are linear artifacts caused by angle sparsity in the image reconstructed from the original data, and the reconstructed image quality is not high. Regarding the images in the middle column of the image and the cross-sectional view, it can be seem that the reconstructed image under the higher sampling projection data has a certain cup-shaped artifact. For the E-ART algorithm, although the base material images cannot be decomposed quickly in one iteration, the cup artifact is suppressed well and the linear artifact caused by sparse sampling can be effectively alleviated by setting the relaxation factor appropriately, which benefits subsequent network decomposition. It should be noted here that FBP ignores the energy spectrum and directly reconstructs the grayscale value. The grayscale value of the base material decomposition using E-ART is different, but can be used to analyze the advantages of E-ART over FBP and to facilitate subsequent decomposition.

4.6. Receptive Field: Small or Large

As argued in the CNN network structure design of Section 3, increasing the receptive field will aggravate the overfitting phenomenon, which is fatal to the goal of obtaining more accurate decomposition results in later iterations. Figure 15 shows the loss function curve update of the pre-decomposition module. During the training process of the proposed CNN, the changes of the training and test sets are essentially the same with the iterative update. For UNet, using pooling and upsampling layers causes the network to have a large receptive field, where other network parameters (such as the number of layers, channels and kernel size of convolution) are same as the proposed CNN. However, the experimental results show that adipose and fibroglandular materials display an obvious overfitting phenomenon after a period of training.

Figure 16 shows the performance of overfitting in the reconstructed image of adipose material, where fibroglandular material has the same phenomenon. The training set has higher accuracy for UNet, and the error in the difference image is quite small, while the test set has a significantly larger error.

Table 4. Average RMSE of dataset with different network structure for quantitative evaluation.

		Proposed CNN	UNet
adipose	training set	$2.53\times {10}^{-3}$	$1.84\times {10}^{-3}$
	test set	$2.87\times {10}^{-3}$	$6.23\times {10}^{-2}$
fibroglandular	training set	$2.68\times {10}^{-3}$	$2.56\times {10}^{-3}$
	test set	$3.06\times {10}^{-3}$	$2.04\times {10}^{-2}$
calcification	training set	$3.30\times {10}^{-7}$	$3.84\times {10}^{-7}$
	test set	$6.40\times {10}^{-7}$	$2.34\times {10}^{-7}$

also displays the RMSE performance metrics of the two network designs. In comparing the training and test set results, severe overfitting can be observed with the UNet architecture, while the proposed CNN model effectively mitigates this issue. This contrast not only highlights the robustness of our CNN design but also validates the effectiveness of the iterative training strategy in maintaining generalization capability.

4.7. Residual-to-Residual vs. Image-to-Residual

Next, we investigate the mapping strategy. Figure 17 displays partial results of two iterations for adipose and fibroglandular materials using the image-to-residual and residual-to-residual mapping strategies. The image-to-residual strategy introduces new reconstruction errors, leading to limited accuracy improvement and potential instability, as regions with large errors are visually indistinguishable in reconstructed images. In contrast, the residual-to-residual strategy achieves significantly higher accuracy. This improvement stems from two factors: (1) integration with the physical module to calculate residuals, and (2) avoidance of floating-point precision loss thanks to the residual-to-residual training strategy.

5. Summary

To solve the problem of high-precision base material decomposition in spectral CT, this paper proposes an algorithm fusing a physical model, model-based iterative reconstruction, and deep learning. Each part has clear functions and is constructed based on given problems and actual situations. The physical module, which is embedded and expressed as a network structure, can accurately describe the physical process of SCT imaging, while the model-based iterative reconstruction algorithm realizes information transfer between domains. Together, the three modules complement each other. We introduce some network skills or strategies in the proposed modules, which are based on observations related to SCT problems and can improve the reconstruction results. The key methodological innovations include:

• Synergistic integration of complementary components: By integrating physical processing (for nonlinear SCT physics), MBIR (for stable domain transformations), and deep learning (for material decomposition), the proposed method combines the interpretability of physics-driven models with the adaptability of neural networks. The physical component ensures fidelity to fundamental X-ray interactions, MBIR provides robustness against noise and incomplete data, and the neural network resolves ambiguities in material-specific attenuation profiles.
• Network-embedded physical model: A network-integrated physical model governs SCT projection data generation, enabling more accurate representation of SCT physical processes. The iterative process progressively enhances reconstruction accuracy through comprehensive physical modeling, where the SCT physical framework is implemented as a network module with learnable parameters including X-ray spectra $S_k\left(E\right)$ and attenuation coefficients $\theta \left(E\right)$.
• Residual-to-residual training: The training paradigm employs residual-to-residual mapping rather than conventional image-to-image or image-to-residual approaches. This strategy focuses on error-sensitive regions that are visually identifiable in residual images but indistinguishable in original reconstructions. This is a novel training paradigm that prioritizes error-sensitive regions by mapping residual projections to residual images.
• Network structure design: The network architecture addresses practical constraints through domain-specific design by (1) accounting for limited spatial correlations between SCT projection data and reconstructed images under inconsistent projections and (2) optimizing CNN receptive field size to balance feature extraction capability with prevention of overfitting and robustness degradation.

6. Limitations

While the proposed method demonstrates significant advancements in spectral CT (SCT) reconstruction, two critical limitations hinder its broader applicability and performance optimization:

• Modular disconnection in network training: The current framework adopts a modular reassembly strategy in which the physical model, MBIR, and deep learning components are trained in a distributed manner. Although this design simplifies implementation (e.g., pretraining the physics module independently), it restricts synergistic interactions between modules. For example, updates to the physical parameters (e.g., X-ray spectra $S_k\left(E\right)$) are based on localized loss functions rather than global feedback from the downstream decomposition task. This fragmented optimization prevents end-to-end error propagation, potentially underutilizing the physical model’s capacity to guide holistic reconstruction. A fully-integrated organic training framework in which modules co-evolve through shared gradients could better harmonize physics constraints with data-driven priors, likely improving reconstruction accuracy.
• Narrow data scope and limited spectral diversity: Validation relied primarily on the AAPM Challenge dataset, which uses synthetic projections generated from a single known X-ray spectrum. While effective for proof-of-concept, this setup overlooks real-world complexities such as spectral drift, detector nonlinearity, and patient-specific anatomical variations. Material decomposition accuracy may degrade when applied to clinical data with unknown or mixed spectra (e.g., dual-energy CT with vendor-specific beam hardening corrections), highlighting the need for validation across diverse scanner configurations and patient populations.

To address these limitations, future work should prioritize two key directions while preserving the method’s core innovations, specifically, unified training paradigms and data-driven generalization. (1) End-to-End Physics–Network Co-Training: Replacing the current module-to-module training strategy with a unified framework where physical parameters ($S_k\left(E\right),\theta \left(E\right)$) and network weights are jointly optimized through a shared loss function would enable the physical model to dynamically adapt to material decomposition feedback, closing the loop between projection generation and image reconstruction; for instance, spectral corrections could directly respond to errors, fostering tighter integration between physics and learning. (2) Multi-Source Data Augmentation and Spectral Transfer Learning: Training should be expanded to diverse clinical datasets with heterogeneous spectra (e.g., photon-counting CT, dual-source CT) and real-world artifacts (e.g., scatter, motion). A spectral domain adaptation module could be introduced to align simulated and clinical data distributions, while few-shot learning techniques might mitigate data scarcity for rare materials or pathologies.

Despite its limitations, the proposed method effectively bridges physics-based modeling and deep learning for spectral CT reconstruction, achieving state-of-the-art material decomposition accuracy on standardized benchmarks. While suboptimal in training cohesion, the modular design successfully demonstrates the value of embedding interpretable physical parameters ($S_k\left(E\right),\theta \left(E\right)$) within a learning framework. However, the reliance on synthetic data and fragmented optimization underscores a gap between theoretical innovation and clinical readiness.

Future efforts to unify training workflows and expand data diversity will be pivotal in translating this method into a robust clinical tool. By addressing these challenges, the proposed approach holds promise for advancing spectral imaging toward personalized diagnostics in which physics-guided AI synergizes with real-world clinical complexity to deliver precise and actionable insights.