Skip to Content
SensorsSensors
  • Review
  • Open Access

20 November 2026

58 Pages

Semantic and Multimodal Approaches to Content Restoration: An Image-Centered Review

,
,
,
and
Department of Cyber-Physical Systems Information Security, Moscow Institute of Electronics and Mathematics, HSE University, 101000 Moscow, Russia
*
Author to whom correspondence should be addressed.
†
These authors contributed equally to this work.
Sensors2026, 26(22), 7344;https://doi.org/10.3390/s26227344 
(registering DOI)
This article belongs to the Section Sensing and Imaging

Abstract

Restoring degraded digital content requires signal reconstruction and preservation of interpretable meaning. Although recent generative models produce perceptually convincing outputs, their semantic fidelity to the original content remains difficult to establish. This image-centered review compares classical and learning-based approaches to content restoration, using selected audio, text, and cross-modal studies as comparative context. We link distortion manifestation, semantic effects, and reversibility to validation requirements, and examine frequency-domain, variational, sparse, generative, transformer-based, and self-supervised approaches. We consider four ways of incorporating semantic information: global conditioning, structural guidance, cross-modal guidance, and shared multimodal latent representations. Evidence maps relate restoration tasks to method families and document coverage of evaluation metrics. The analysis identifies increasing integration of classical signal models with learned and generative priors, alongside substantial heterogeneity in datasets, degradation settings, and evaluation protocols. Signal fidelity, perceptual quality, and semantic correctness capture distinct aspects of restoration; improvements in one do not establish improvements in the others. Unresolved challenges include unsupported generated content, limited robustness under severe degradation and domain shift, computational cost, and the lack of unified semantic evaluation. We outline priorities in multimodal conditioning, uncertainty estimation, realistic benchmarking, and restoration provenance to support reliable reconstruction of semantically meaningful content.

1. Introduction

As digital communication systems, cloud services, multimodal databases, and generative models evolve, digital content restoration is becoming increasingly important. Images, audio, video, and text are used in communication systems, medical diagnostics, remote sensing, digital forensics, archival storage, and human–computer interaction. In these applications, content degradation can compromise not only the technical quality of a signal but also its semantic interpretability.
Content restoration encompasses methods aimed at reconstructing damaged, missing, or distorted portions of digital data. Traditionally, it has been treated as an inverse problem in digital signal processing, with the objective of bringing a degraded object closer to its original state according to technical or structural criteria. This approach is effective for simple, readily modeled distortions; however, under severe degradation, incomplete information, or semantically significant changes, restoration cannot be reduced to signal error minimization alone [1].
This review examines the evolution of approaches to content restoration, with an emphasis on images, from classical signal-oriented methods to modern models that use semantic representations, generative priors, and multimodal relationships. Here, semantics-oriented restoration refers to approaches that account not only for the physical or structural form of the data but also for their interpretable semantic content. Unlike low-level restoration, these methods seek to preserve the communicative function of content: object recognizability, speech intelligibility, the semantic integrity of text, and consistency across modalities. Audio and text restoration provide selected comparisons of semantic representations and evaluation, rather than equally extensive survey branches.
Digital content can undergo various modifications during its life cycle. These arise during data acquisition and digitization, processing, storage, and transmission, as well as from intentional manipulation. In this review, the term “distortion” denotes such modifications regardless of their source. Their effects can manifest at different levels, ranging from local signal degradation to disruption of the data structure or substitution of semantic content.
In a number of applications, simply improving perceptual quality is insufficient. In medicine, remote sensing, forensic examination, and automated recognition systems, semantically incorrect restoration can be more dangerous than no restoration. For these tasks, it is therefore essential to consider not only the technical quality of the result but also its semantic correctness and correspondence to the context [2]. Sensor-generated images make this distinction concrete: MRI reconstruction, cloud removal in optical imagery, and infrared–visible fusion restore observations that may subsequently support medical interpretation, scene analysis, or recognition [3,4,5,6]. Improvement in image-quality scores alone does not establish improvement in these downstream tasks.
Numerous content restoration methods have been proposed in recent years. Classical approaches include frequency transforms, variational methods, dictionary learning, and sparse coding. Modern machine learning methods employ deep generative models, diffusion architectures, transformers, masked autoencoders, pretrained vision–language models (VLMs), and multimodal fusion mechanisms. The diversity of tasks, modalities, degradation models, architectures, and quality metrics makes this field difficult to navigate and direct comparisons between existing approaches challenging.
In addition to content restoration itself, some of the studies considered address related fields: generative modeling, self-supervised learning, multimodal representation learning, missing-modality learning, and semantic communication. These fields pursue different objectives but share related mechanisms: latent spaces, semantic priors, reconstruction losses, and cross-modal alignment. Accordingly, this review considers them insofar as their methods apply to the restoration of degraded content [7].
Despite the substantial body of research, existing reviews are generally limited to a particular modality, task, or model family. Some focus on image inpainting, super-resolution, or all-in-one image restoration; others examine diffusion models, transformers, or masked autoencoders as distinct architectural directions. The present review compares these approaches through the relation between degradation characteristics, semantic information, and validation requirements, using existing task- and architecture-specific surveys as reference points [8,9,10]. These criteria provide a common basis for selecting hybrid restoration methods and evaluating the reliability of their outputs.
This image-centered review examines content restoration through the relationship between degradation, available semantic information, and validation of the reconstructed content. It brings classical and learning-based methods into a common comparison while treating algorithm families, learning paradigms, and the role of semantic guidance as complementary dimensions. Selected studies on audio, text, neural decoding, and semantic communication provide context for transferable mechanisms; they do not constitute equally comprehensive surveys of those fields.
The main contributions of this review are as follows:
•
Distortion manifestation, semantic effects, and reversibility are linked to the information used by restoration methods and to the validation required for their outputs;
•
Algorithm families and learning paradigms are analyzed separately from four functional roles of semantic information, allowing hybrid methods to be compared without forcing them into mutually exclusive classes;
•
Reported results are interpreted in relation to datasets, degradations, metrics, and computational conditions, distinguishing perceptual plausibility from fidelity to the observed target;
•
The synthesis identifies research questions concerning unsupported reconstruction, unreliable auxiliary context, and resource constraints in sensing applications.
The remainder of this article is organized as follows. Section 2 compares this review with existing reviews. Section 3 formulates the content restoration problem and proposes a classification of distortions that accounts for their semantic consequences. Section 4 examines existing restoration approaches, from classical digital signal processing methods to generative, transformer-based, and multimodal models. Subsequent sections discuss the limitations of the approaches considered and outline promising directions for future research.

2. Review of Previous Surveys

The rapid expansion of research on digital content restoration has led to numerous reviews focused on particular tasks, modalities, or method families. Clearly positioning the present review requires a systematic comparison with the most closely related studies in terms of subject matter and methodology, identifying both areas of overlap and fundamental differences.
A substantial body of review literature concerns image restoration. Zhang et al. (2023) [11] review deep learning-based image inpainting, organizing methods primarily by convolutional neural network (CNN) and generative adversarial network (GAN) architectures and application scenarios, with particular attention to feature fusion. The review covers research from 2000–2021 and contains 175 references. Xiang et al. (2023) [8] analyze approximately 150 publications from 2000–2022 and organize inpainting methods along three dimensions: filling strategy, network architecture, and loss function. Their taxonomy is supported by quantitative comparisons on standard benchmarks (Places2, CelebA) using peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and Fréchet inception distance (FID). The value of this work lies in its methodological rigor and reproducible comparison of methods. Quan et al. (2024) [12] include a section on text-guided image inpainting, treating textual information as guidance for restoration. In parallel, Li et al. (2025) [9] provide a detailed taxonomy of diffusion methods by learning paradigm and conditional guidance strategy. Finally, Jiang et al. (2025) [10] move from specialized architectures to a unified model robust to several degradation types simultaneously, consolidating the All-in-One concept and introducing an open, reproducible benchmark. Image restoration thus has an established review tradition covering diverse architectural paradigms and semantic approaches.
For audio restoration, Moliner & Välimäki (2024) [13] fulfill a similar role for diffusion methods as Li et al. do for images: a detailed comparison of diffusion-based audio inpainting with classical methods, including subjective evaluation using multiple stimuli with hidden reference and anchor (MUSHRA). Asaad et al. (2026) [14] compare frozen and fine-tuned HuBERT encoders for speech inpainting, finding different advantages in single- and multi-speaker settings. For images, Quan et al. achieved this step through text-guided control. Thus, by 2026, these two audio studies cover complementary aspects of audio restoration: signal-based and diffusion methods, and self-supervised learning (SSL) semantics for speech. Once we turn to the text modality, however, the scope of the review literature changes radically.
Zhang et al. (2025) [15], publishing “A Systematic Survey of Text Summarization”, consolidate more than 250 publications spanning two decades and trace three paradigm shifts—from statistical methods to seq2seq and then to large language model (LLM) approaches. The problem is that summarization compresses meaning rather than reconstructing it from a degraded state. The survey also discusses multimodal summarization as a future direction, including images, audio, and video alongside text. This discussion concerns understanding and compression rather than the reconstruction of damaged or missing content.
Alongside the reviews already discussed, a growing body of surveys on multimodal large language models (MLLMs) has emerged since 2023. Wu et al. (2023) [16] and Yin et al. (2024) [17] jointly provide a broad architectural description of MLLMs, from early cross-attention approaches to GPT-4V, including multimodal in-context learning strategies and hallucinations. Zong et al. (2025) [7] move toward a methodological foundation by classifying self-supervised multimodal learning (SSML) around three challenges: learning without labels, modality fusion, and working with unaligned data. This provides the methodological basis for many contemporary restoration methods, from masked autoencoders (MAE) [18] for images to HuBERT for audio. He et al. (2024) [19] extend their scope to four modalities (images, video, 3D, and audio), providing a systematic account of LLM applications to generation and editing across these four modalities. Huang et al. (2025) introduce a detailed classification of diffusion-based image editing methods, including text-guided inpainting, and formalize evaluation through a new benchmark, EditEval [20]. These surveys address multimodal understanding, generation and editing, including image inpainting. The present review instead uses degradation characteristics, semantic-conditioning roles, and validation requirements as shared criteria for comparing image restoration approaches.
A closely related review across the combined criteria is Sapkota et al. (2025) [21], “Multimodal Large Language Models for Image, Text, and Speech Data Augmentation.” The authors systematically cover the application of multimodal LLMs to three modalities—images (~24 studies), text (~45 studies), and speech (~35 studies)—within a unified analytical structure. Nevertheless, this is a review of data augmentation, not restoration. Augmentation starts with a complete signal and creates diverse variations to expand the training dataset; restoration starts with a damaged or incomplete signal and reconstructs what has been lost. Success in augmentation is measured by the semantic plausibility and diversity of synthetic examples rather than correspondence to a specific lost original. Consequently, Sapkota et al. do not examine diffusion-based audio inpainting, semantically guided image restoration systems (SPIRE [22], XPSR [23], SSP-IR [24]), or MAE [18]/SSL as mechanisms for signal reconstruction.
The reviews considered follow two trajectories. The first concerns signal restoration: it traces the evolution from GANs to diffusion and semantics, but within a single modality. The second concerns semantics and multimodality: it encompasses several modalities and a developed semantic framework but focuses on understanding, generation, and augmentation rather than reconstruction of degraded content. These trajectories already intersect in reviews of semantically guided image restoration. Individual methods such as SPIRE [22], XPSR [23], MaskSR [25], and SenSE [26] illustrate how semantic information can support restoration. The studies in Table 1 address restoration tasks and model families [8,9,10,11,12,13,14], as well as semantic processing, generation, editing, and multimodal learning [7,15,16,17,19,20,21]. We compare these approaches using three common criteria: degradation characteristics, the source of information used for reconstruction, and validation requirements. This comparison links method selection to the available observations and identifies validation requirements for hybrid restoration systems.
Table 1. Comparison of previous reviews by scope, classification basis, semantic guidance, and limitations relevant to restoration. Source counts refer to bibliography entries, rounded to the nearest ten.

3. The Content Restoration Problem

3.1. Distortions in Different Types of Content

Content restoration encompasses both the structural level—removing low-level damage such as noise, blur, and compression artifacts—and the semantic level: reconstructing meaning under substantial degradation, when bitwise accuracy is unattainable but the context retains sufficient information for a plausible reconstruction of the content. The relationship between these two levels determines the specific nature of the task for each modality.
Classical digital signal processing theory describes content quality using the signal-to-noise ratio as a basic criterion, capturing only energy characteristics without reflecting structural or semantic aspects of interpretation [27].
Contemporary approaches to semantic restoration assume a three-level understanding of quality. The technical level reflects the accuracy of the physical representation. The perceptual level accounts for subjective criteria such as sharpness, intelligibility, and naturalness. Preservation of semantic content and the ability to interpret it correctly are, in turn, represented by the semantic level. For multimodal content, cross-modal consistency must also be considered: its disruption undermines interpretation even when each modality individually has high quality [27,28].
Depending on the extent to which a reference signal is used, metrics are classified as full-reference, reduced-reference, or no-reference [27]. Objective pixelwise metrics—mean squared error (MSE) and PSNR—are computationally simple but correlate weakly with subjective perception. Structural metrics, particularly SSIM [29], and perceptual metrics based on neural network features, such as learned perceptual image patch similarity (LPIPS) [30], show better agreement with human judgments. Perceptual evaluation of speech quality (PESQ) [31] and short-time objective intelligibility (STOI) [32] are used for audio, while text is evaluated using lexical overlap measures (bilingual evaluation understudy (BLEU) [33] and recall-oriented understudy for gisting evaluation (ROUGE) [34]) and language-model-based semantic metrics (BERTScore) [35]. Developing unified metrics that adequately assess the semantic correctness of restoration in multimodal scenarios remains a fundamental unresolved problem [28], itself indicating a substantial gap in this field.
Distortion sources fall into three groups according to their origin: acquisition and digitization; processing, storage, and transmission; and intentional distortions, which are particularly relevant to forensic tasks. Distortions are additionally distinguished by their dependence on content, since content-dependent distortions require explicit consideration of the semantic structure of the signal.
Each distortion category manifests differently across modalities according to the information medium. For example, noise distortions appear as Gaussian or impulse noise in images, background hum and nonstationary interference in audio, and optical character recognition (OCR) or automatic speech recognition (ASR) errors in text. Reconstruction artifacts in neural network systems form a separate category: they improve quality metrics while compromising semantic fidelity.
A multidimensional classification along three orthogonal axes is proposed. By level of manifestation: signal-level, structural, and semantic. By effect on semantics: degrading, distorting, and neutral with respect to semantics (affecting perceptual quality without disrupting interpretation). By degree of reversibility: reversible, conditionally reversible (recoverable using contextual or cross-modal data), and irreversible. This classification formalizes the requirements for restoration methods and directly identifies a research gap: existing approaches do not provide a systematic link between the type of degradation and the strategy for its semantics-oriented removal.
Figure 1 summarizes distortion sources, content dependence, modality-specific manifestations, and the proposed three-axis classification.
Figure 1. Classification of distortions: sources, content dependence, modality-specific manifestations, and three orthogonal axes.

3.2. Effects of Distortions on Content Perception and Semantics

A substantial proportion of changes are irreversible, and reconstruction yields only a probabilistic estimate of the original content. Thus, the central objective of semantic restoration is to reconstruct the capacity for interpretation rather than reproduce the original signal in full. In a multimodal context, semantic correspondence across modalities can compensate for degradation in one of them. Given the differences between perceptual and functional quality, restoration methods should be optimized according to criteria for preserving semantically meaningful features rather than surrogate signal metrics [2,28].
Together, these observations reveal a research gap: existing descriptions of distortions and evaluation tools, which primarily target the signal level, are insufficient for semantics-oriented restoration of multimodal content. There is no systematization linking the type and nature of degradation to its semantic consequences and the requirements for restoration methods.
This review systematically examines existing restoration approaches—from classical signal-oriented methods to contemporary generative architectures and multimodal strategies—within the unified analytical framework established in this section.

4. Existing Approaches to Content Restoration

The analysis uses two complementary views: methodological families and learning paradigms, and the role of semantic information in restoration. These views are not mutually exclusive: a diffusion model can use a wavelet representation, a structural prior, and a textual condition within the same pipeline. The classification in Figure 1 is applied to the task setting rather than to the architecture alone. For each setting, we identify the level of distortion, its possible semantic effect, the information available for reconstruction, and the evidence needed to validate the result.
Table 2 applies these questions to representative methods reviewed below. Recoverability is conditional on the observations and assumptions: a category label does not guarantee that the original content can be uniquely recovered. Semantic effects also depend on severity and application; a signal-level change can be neutral for recognition in one case and obscure relevant content in another.
Table 2. Distortion categories, available information, and validation requirements.
This is an analytical mapping of task settings, not a reclassification claimed by the cited studies. The final row specifies a validation scenario, not a reported failure frequency. Proposed checks do not imply that every cited work performed them.
Historically, content restoration has relied on methods based on various mathematical transforms and statistical and physical models of distortions. In this review, these methods are referred to as classical. The 1999 and 2000 studies by Efros and Leung and Bertalmio et al. on pixel interpolation [1] are often regarded as pioneering contributions to digital content restoration. However, the earliest content restoration methods originated in digital signal processing theory; this group includes the Fourier transform and the least-squares method. Subsequently, machine learning methods became more widely used for content restoration because they offer better performance and depend less on the damage model, although classical methods continue to develop.

4.1. Classical Methods

4.1.1. Frequency-Transform-Based Methods. Wavelet Transforms

Content restoration in the spatial domain faces the fundamental problem of high computational complexity caused by the large number of elements requiring processing. In some cases, this problem can be circumvented by using the frequency domain instead of, or alongside, the spatial domain. Moving to the frequency domain allows an incoming signal to be evaluated in terms of its frequency distribution, enabling the use of patterns and models that are difficult to analyze in the spatial domain.
The historical foundation of frequency-domain methods is the Fourier transform, its inverse, and their subsequent developments for digital signal processing, particularly fast discrete Fourier transform algorithms. The Fourier transform moves signal analysis into the frequency domain, while its inverse reconstructs the original spatial representation. Frequency-domain manipulation can restore content when degradation results from substantial but separable changes in that domain, such as the addition of high-frequency Gaussian noise. The corresponding limitation is that restoration quality will be severely constrained if degradation-related changes are difficult to separate from the useful signal in the frequency domain. Table 3 shows comparative results for the Fourmer method.
Table 3. Comparative results of Fourmer (Zhou et al., 2023 [42]) for dehazing, deraining, and low-light enhancement. Each row describes a distinct task and dataset; the within-row comparisons are those reported in [42]. Values across rows do not form a common ranking.
Key takeaways. Fourmer improves the reported PSNR over the listed comparator in each setting, but its SSIM on LOL is lower than that of URetinex. Thus, even these within-benchmark comparisons do not identify a method that dominates every quality criterion. The table contains no runtime or memory measurements from which deployment efficiency could be inferred [42].
The discrete wavelet transform (DWT) provides an alternative representation by applying a sequence of low- and high-pass filters along each spatial dimension, forming a multiscale hierarchical structure that retains both frequency and spatiotemporal information.
Contemporary methods use the wavelet representation as an intermediate transform rather than a standalone restoration tool, exploiting a key property of wavelets: natural images and audio signals have substantially sparser representations in the wavelet domain than in the spatial or time domain, making the restoration problem more structured.
The fastWDM3D method proposed by Durrer et al. (2026) [3] demonstrates the effectiveness of wavelet representations in accelerating diffusion models. Rather than applying the diffusion process directly to spatial data, the method operates on wavelet coefficients. Its key technical innovation is a variance-preserving noise schedule adapted to a small number of time steps (500 times fewer than with a classical schedule). Experimental validation on the BraTS dataset demonstrates an SSIM of 0.8571, an MSE of 0.0079, and a PSNR of 22.26, with a restoration time of 1.81 s per volume, corresponding to an approximately 800-fold speedup over classical denoising diffusion probabilistic models (DDPMs). However, the inverse wavelet transform adds a computational stage and may introduce artifacts if high-frequency components are reconstructed insufficiently accurately. Table 4 shows the fastWDM3D results.
Table 4. Results of fastWDM3D [3] compared with the baseline WDM3D model on the BraTS 2023 test set (healthy brain tissue inpainting in magnetic resonance imaging (MRI)). This is the within-study comparison reported in [3]; inference times apply to that study’s evaluation setup.
Key takeaways. The comparison in [3] supports reducing diffusion steps for the specific 3D MRI inpainting task while retaining the reported reconstruction quality. The 1.81 s per-volume inference time is an application-specific result; it does not establish real-time performance for other volume sizes, hardware, or acquisition settings.
For the audio modality, Chithra K. R. et al. (2025) [43] developed an approach combining the continuous wavelet transform (CWT) with autoregressive modeling. The authors use matched wavelets adapted to local signal characteristics. The autoregressive model then predicts coefficients from the boundaries toward the center of the gap. The method is effective for short gaps, where boundary information reliably determines local spectrotemporal characteristics, but quality deteriorates for wide gaps: accumulated prediction errors cause phase distortions and a loss of timbral consistency in the central portion of the restored segment.
WFIL-NET, proposed by Cao et al. (2025) [44], uses wavelets differently: DWT serves as a downsampling operation before the image enters the neural network. Unlike classical pooling or convolutional downsampling, wavelet downsampling preserves all frequency components. An attention mechanism suppresses high-frequency noise in smooth regions and enhances detail at boundaries. Halving the spatial resolution along each dimension reduces the number of spatial elements fourfold, proportionally lowering the computational complexity of subsequent convolutional layers. However, the loss of spatial resolution requires more powerful decoders with several upsampling levels to accurately reconstruct fine details, partially offsetting the computational gain.
Huang et al. applied wavelet diffusion to the specialized task of cloud removal from satellite images, proposing WaveDiffDecloud [4]. Clouds and the Earth’s surface have different frequency characteristics: clouds primarily produce low-frequency brightness changes with blurred boundaries, whereas surface details contain high-frequency components. Wavelet decomposition naturally separates these components, allowing the diffusion model to generate textures in different frequency bands independently. Explicit modeling of frequency structure provides more accurate reconstruction than spatial methods, but the narrow focus on cloud characteristics and optical imagery limits the architecture’s applicability to other content restoration tasks. Figure 2 illustrates the WaveDiffDecloud architecture.
Figure 2. Architecture of WaveDiffDecloud: a conditional diffusion model in wavelet space with a StruTex-HFR module for cloud removal from satellite images [4].
The methods considered demonstrate a consistent trend: wavelet transforms are effective as auxiliary components that provide compact multiscale representations for different data modalities. Their main roles are to reduce problem dimensionality by moving to a sparse representation and to explicitly separate frequency components, simplifying subsequent processing by diffusion models, autoregression, or deep neural networks. As standalone restoration tools, wavelets have been displaced by more powerful generative models. The methods are compared in Table 5.
Table 5. Comparison of Fourier transform- and wavelet-based methods: quality metrics and application mode. Values summarize different signals, tasks, datasets, and evaluation protocols; they are representative results, not a direct ranking. Inference times refer to different input units and workloads.
Key takeaways. Frequency and wavelet representations play different roles here: direct filtering, prediction of missing coefficients, or an intermediate representation for a learned generator. The practical choice depends on whether the degradation is separable in the transform domain and whether a learned prior is needed. MRI-volume and satellite-image inference times describe different workloads and do not rank the methods by efficiency [3,4,43].

4.1.2. Variational Methods

Variational methods are a class of image restoration approaches based on minimizing an energy function that includes data fidelity and regularization terms. These methods have a strong mathematical foundation and allow explicit control over the balance between restoration accuracy and solution smoothness. Figure 3 shows the general structure of a variational method [45].
Figure 3. General structure of a variational image restoration method. Abbreviations: total generalized variation (TGV); alternating direction method of multipliers (ADMM); iterative shrinkage-thresholding algorithm (ISTA).
Traditionally, variational methods, such as those based on Total Variation (TV), effectively restore piecewise-smooth image regions by minimizing the gradient norm. Recent developments include the use of spline spaces to model variational restoration methods, providing a finite-dimensional approximation and a more efficient numerical implementation [45].
In recent years, variational methods have been integrated with modern generative models. Agarwal et al. (2026) [38] proposed VIPaint, an image restoration method that uses pretrained diffusion models through variational inference. Instead of heuristic sampling procedures, VIPaint employs a hierarchical variational inference algorithm that optimizes a non-Gaussian Markov approximation to the true diffusion posterior. The method analytically marginalizes missing features and uses a rigorous variational bound for optimization. VIPaint substantially outperforms existing approaches in both the plausibility and diversity of restorations and readily generalizes to other inverse problems, such as blurring and super-resolution. Figure 4 illustrates inpainting results obtained with VIPaint.
Figure 4. Qualitative image inpainting results of VIPaint-4 and VIPaint-2 on ImageNet-64 with random and rotated-window masks, compared with baseline methods [38].
Another approach uses probabilistic frameworks for diverse images with large arbitrary masks. Sumathi and Uma Devi (2025) [46] developed a probabilistic model that uses image-prior data to learn prominent features within StyleGAN3. The prior data are constructed using cosine similarity, the mean, and intensity. The image is reconstructed using a probabilistic maximum a posteriori (MAP) estimate. Variational inference with a modified Bayes-by-Backprop approach is then applied to obtain an optimal solution. Figure 5 illustrates the results of the probabilistic MAP method.
Figure 5. Image inpainting results of the probabilistic maximum a posteriori (MAP) method with Bayesian variational inference (Bayes-by-Backprop) based on StyleGAN3 (Sumathi & Uma Devi, 2025 [46]). Brain MRI (PSNR = 29.97 dB, SSIM = 0.86, LPIPS = 0.07).
Variational methods are also successfully applied to specialized image processing tasks. Wang et al. (2024) [47] proposed an effective image restoration algorithm based on a modified Gray–Scott model with an added data fidelity term to control edge sharpness and the stability of changes in pixel values during restoration. The method outperformed alternative techniques in both visual appeal and computational metrics, such as mean squared error and peak signal-to-noise ratio. Variational methods are compared in Table 6.
Table 6. Reported quality and computational characteristics of variational image restoration methods. Quality values were obtained for different tasks, datasets, masks, and resolutions and do not constitute a direct ranking of the methods.
Key takeaways. VIPaint and PF require a trained generator and image-specific posterior fitting, but support multiple completions [38,46]. Modified Gray–Scott, improved BM3D + TV, and GSR + SVTV use no pretrained neural generator; their image-specific computations comprise PDE iteration, block matching and filtering, and patch grouping with alternating optimization, respectively [47,48,49]. These workflow differences describe deployment requirements but do not establish a common runtime ranking.

4.1.3. Dictionary Learning and Sparse Coding Methods

Dictionary learning methods represent local image patches as sparse linear combinations of elements from an overcomplete dictionary, providing efficient compression and robustness to noise: a small number of active dictionary atoms capture essential structural patterns while ignoring random variations. Group Sparse Representation (GSR) extends this idea by exploiting Non-local Self-Similarity (NSS). Joint coding exploits correlations between similar patches, yielding more stable coefficient estimates than independent sparse coding [50,51,52].
The problem with the classical GSR scheme emerges at high noise levels: the NSS prior is extracted from the noisy image, and patches are grouped using their noisy versions. Under substantial degradation, patch similarity is distorted, causing dissimilar patches to be included in groups and consequently producing incorrect sparse coefficient estimates.
Zhang et al. (2025) [36] proposed the Corrected Group Sparse Residual Constraint (CGSRC) model, introducing an additional NSS prior from a prefiltered image. The two NSS priors correct each other: grouping based on x ^ provides a more reliable group structure, whereas coding y preserves accurate information from the observations. Experimental validation demonstrates improvements in PSNR and SSIM at high noise levels compared with classical GSR methods. Computational complexity, however, increases. The quality of the final result depends critically on the quality of the prefilter.
Ning et al. (2023) [53] extended the group sparse residual concept to a multiscale representation in MS-GSRC. MS-GSRC generalizes to various inverse problems—denoising, inpainting, and compressed sensing—by modifying the data fidelity term while retaining the general multiscale group scheme. Its main drawback is high memory consumption: dictionaries and coefficients must be stored for several scales, and high-resolution images can require several gigabytes of memory. Figure 6 illustrates the results of MS-GSRC.
Figure 6. Denoising results of MS-GSRC (multiscale group sparse representation with low-rank constraints) compared with BM3D and GSRC-NLP on BSD68. PSNR gain: +0.24 dB vs. BM3D; SSIM gain: +0.0153 vs. BM3D (averaged across noise levels) [53].
A fundamental limitation of dictionary learning methods is their dependence on the NSS prior: they are effective for textured images with recurring structural patterns (fabric textures, natural landscapes, brickwork), where many similar patches are available for grouping. For images with unique, irregular patterns, group coding quality deteriorates. Moreover, the computational complexity of the method scales proportionally to the number of pixels and the group and dictionary sizes, requiring substantial processing time for large inputs even on modern GPUs. This limits applicability to real-time tasks and video processing. The methods are compared in Table 7.
Table 7. Comparison of dictionary learning and sparse coding methods for image restoration: quality metrics and gains over BM3D. Datasets, noise levels, and averaging schemes differ. Relative gains apply only to the baseline and setting identified in each row; they are not directly comparable with absolute PSNR values.
Key takeaways. The reported MS-GSRC gains support multiscale grouping under its tested conditions: denoising gains are measured against BM3D, whereas the inpainting gain uses GSR as the baseline [53]. Grouping and dictionary construction remain important cost drivers. Practical use therefore depends on image self-similarity, memory requirements, and acceptable latency, not only the reported quality gain [36,50,53].

4.1.4. Limitations and Directions for Development

The key limitation of image restoration methods that do not use machine learning lies in their dependencies: on the distortion distribution for frequency-transform-based methods, on the selected statistical model for variational methods, and on NSS priors and the sparse coding method for dictionary methods. For dictionary methods, computational complexity also depends on hyperparameters such as input, group, and dictionary sizes, limiting their applicability to real-time tasks and stream processing.
Current directions for development can be divided into two broad areas. The first concerns standalone solutions. Key directions in this area address the limitations of the methods: the previously mentioned dependencies on priors, coding methods, and statistical models, as well as the computational complexity limitations of dictionary methods.
The second concerns combined use. This area includes cases where different methods are used together within a single processing pipeline. For example, wavelet-based methods are combined with machine learning models. Such approaches combine the high effectiveness of classical methods for distortion matching a predefined model with the strong generalization capability and versatility of machine learning methods.
For the classical formulations, the taxonomy directs attention to what remains observable. Patch similarity can constrain signal-level denoising, but the grouping itself becomes unreliable under strong corruption, as illustrated by CGSRC [36]. Inpainting instead removes observations and requires a prior to select among possible completions; VIPaint [38] combines variational inference with a learned diffusion prior for this purpose. The change in prior expands the set of plausible solutions, but does not by itself establish which missing details belonged to the original. Thus, a restoration family and a reversibility category describe different aspects of the problem.

4.2. Learned Model Families and Training Paradigms

This subsection compares learned model families and training paradigms: generative models, transformers, masked pretraining, pretrained vision–language backbones, and multimodal fusion architectures. These categories describe how a method is built or trained. Section 4.3 examines a complementary question: what semantic information the method uses and how that information constrains reconstruction. Within each family, the taxonomy in Figure 1 distinguishes the observed corruption from uncertainty introduced by missing or unreliable content; the same model family may therefore address several distortion categories.

4.2.1. Deep Generative Models and Restoration of Semantic Representations

Deep generative models can restore semantically consistent content by learning distributions from high-dimensional data. Their main advantage is the ability to produce perceptually realistic results that remain consistent with the input data, even under severe degradation. Figure 7, Figure 8 and Figure 9 illustrate the architectures of the generative models considered.
Figure 7. Architecture of LaMa: inpainting based on fast Fourier convolutions [39].
Figure 8. Architecture of IR-SDE: image restoration through mean-reverting stochastic differential equations [54]. Green/blue: start/end states of each process; orange: score term.
Figure 9. Architecture of SPIRE: image restoration guided by semantic prompts [22].
Diffusion model-based methods are used for multimodal content restoration, integrating semantic priors from textual descriptions and visual information from reference images, as well as contextual prompts obtained from pretrained language models.
•
MRIR (Multimodal Reference Image Restoration) integrates textual and visual information at two levels, extracting semantically meaningful descriptions from degraded images and using Contrastive Language–Image Pretraining (CLIP) and ControlNet encoders to control spatial structures at the pixel level [24].
•
A similar approach is implemented in SSP-IR (Semantic and Structure Priors for Image Restoration), which uses explicit semantic priors and implicit visual representations to prevent the generation of artifacts [24].
•
SPIRE (Semantic and Restoration Prompt-driven Image Restoration) extends this concept with instructions that allow users to control the degree of restoration [22].
The main advantage of diffusion-based methods is their ability to generate perceptually realistic results by using powerful generative priors from pretrained models. However, they have several limitations: high computational complexity, lack of control over the generative process, and dependence on large amounts of aligned data.
The need to make restoration controllable and reduce computational complexity motivates the development of hybrid architectures and efficient conditioning strategies, discussed in the next section. Comparative results are presented in Table 8.
Table 8. Comparison of deep generative models for image restoration. Results were reported under different datasets, degradations, and protocols and do not constitute a direct ranking. FID values from different evaluation sets should not be compared as if obtained under one protocol.
Key takeaways. LaMa reports worse LPIPS and FID for wide masks than for narrow masks, illustrating sensitivity to missing-region size within one evaluation setting [39]. The diffusion-based rows cover different restoration tasks and conditioning mechanisms [22,24,54]. They therefore identify available design choices, while selection for deployment additionally requires measured sampling cost and validation of generated details against the observations.

4.2.2. Transformer-Based Architectures for Semantic Restoration

Transformer-based architectures effectively model long-range dependencies and semantic relationships between different regions of the input data through self-attention. For image restoration, transformer models capture global contextual relationships to enable semantically correct reconstruction.
•
Traditional Vision Transformer (ViT)-based image restoration methods compute attention over all tokens of the input image, resulting in quadratic computational complexity and inefficient processing of high-resolution images.
•
SemanIR introduces the concept of sharing key semantics by constructing a sparse semantic dictionary containing only the semantic relationships essential for restoration. The resulting dictionary is shared by all transformer blocks within a stage. This approach reduces computational complexity to linear while retaining the ability to model semantically meaningful relationships [55]. Figure 10 illustrates the SemanIR architecture.
Figure 10. Architecture of SemanIR: a shared semantic dictionary with linear attention complexity [55].
For multimodal restoration, transformer architectures enable efficient integration of information from different modalities through cross-modal attention mechanisms. CSF-GAN [56], which is based on a generative adversarial architecture with embedded transformer blocks, introduces a semantic fusion module that combines textual information at the sentence and individual word levels to guide the process of inserting pixels into images.
The main advantage of transformer architectures is their ability to model complex semantic dependencies. Their main drawbacks are high computational complexity at high input resolutions, the need for substantial amounts of training data, and the limited interpretability of attention mechanisms. Transformer architectures are compared in Table 9.
Table 9. Comparison of transformer architectures for image restoration. Dataset names alone do not establish matching training and evaluation conditions. The reported results are representative; complexity expressions describe the specified attention operations rather than complete implementation costs.
Figure 11. Architecture of Restormer: a multiscale transformer with transposed channel attention [37]. Notes: “MDTA uses attention across channels rather than pixels → applicable to high-resolution images.” Results on GoPro: PSNR = 32.92 dB, SSIM = 0.9611. Result on Rain100L: PSNR = 38.99 dB, SSIM = 0.978. (a) MDTA; (b) gated-Dconv feed-forward network (GDFN). Red symbols mark the channel dimension.
Figure 12. Architecture of SwinIR: image restoration based on the Swin Transformer with window-based attention [40].
Key takeaways. Windowed, channel-based, and sparse-neighborhood attention reduce the dependence of attention cost on spatial input size through different assumptions [37,40,55,57]. The complexity expressions describe selected operations, not end-to-end latency or memory use. Resolution, channel width, neighborhood size, and training protocol must therefore accompany quality measurements when choosing a model for deployment.
The move toward more efficient representation and learning strategies motivates the development of masked autoencoders and self-supervised learning methods.

4.2.3. Masked Autoencoders and Self-Supervised Learning of Multimodal Representations

Masked autoencoders (MAE [18]) learn semantically meaningful representations through self-supervised pretraining tasks in which the model restores masked portions of the input data. For multimodal data, MAE enables the learning of shared, modality-invariant representations by jointly reconstructing multiple modalities from partially observed inputs. Table 10 presents MAE results on downstream tasks.
Table 10. Masked autoencoder (MAE) [18] and self-supervised learning methods: metrics on downstream tasks related to content restoration. The rows report different downstream tasks, architectures, and evaluation protocols. They do not directly compare restoration accuracy or constitute a common model ranking.
Key takeaways. Table 10 demonstrates transfer of pretrained representations to classification, detection, and segmentation [18,41,58,59,60]. These results support their use as candidate representation backbones, but they do not measure reconstruction of a degraded target. Restoration quality, robustness to the intended degradation, and adaptation cost require a separate evaluation.
SemMIM [61] proposes semantically enhanced cross-modal masked image modeling (MIM) for pretraining vision–language models. The key idea is to inject high-level semantics into local image patch encodings through self-supervised consistency learning and a shared encoding space. Experimental results confirm that MIM becomes more effective at cross-modal semantic alignment and that the method outperforms alternatives on downstream vision–language understanding tasks.
The Multimodal Autoencoder (MMAE) [62] for automated media content processing is trained on aligned image–audio–text triplets from the LUMA dataset, minimizing joint reconstruction losses across all modalities. This joint reconstruction naturally aligns representations in a shared embedding space, achieving semantic consistency across modalities.
Hi-End-MAE introduces encoder-driven reconstruction: decoder tokens query visible encoded representations. This reduces the role of the decoder and directly links representation quality to reconstruction capability. This approach demonstrates improved adaptability of attention to different anatomical contexts in medical image segmentation tasks [41].
Multimodal Channel-Mixing (MCM) [63] implements early modality fusion with a channel-mixing module. This module reduces channel redundancy, facilitates multimodal learning, and enables robust feature extraction. Improved performance is reported for the evaluated facial action unit detection tasks.
The advantages of MAE-based approaches include the ability to learn in a self-supervised manner without extensive manual annotation, efficient extraction of semantically meaningful representations through reconstruction tasks, and natural cross-modal alignment through joint reconstruction. Their main limitations are the need to carefully design masking strategies for different modalities, the computational cost of reconstructing high-dimensional data, and the absence of explicit mechanisms for controlling the semantic correctness of restored representations. Figure 13 illustrates the operating principle of MAE [18], and Table 11 presents the masking parameters.
Figure 13. Operating principle of MAE (Masked Autoencoder): an asymmetric encoder–decoder architecture with 75% of patches masked [18]. Gray denotes masked patches and mask tokens, light blue denotes encoded visible patches, and salmon denotes the decoder’s patch reconstructions.
Table 11. Masking designs in MAE and VideoMAE, with a controlled comparison of video masking strategies. The upper-panel accuracy values refer to different datasets and pretraining settings and do not directly rank the methods. The lower panel compares random and tube masking within VideoMAE under the same conditions on each dataset [58].
Key takeaways. Under matched VideoMAE settings, tube masking improves top-1 action-recognition accuracy over random masking by 1.3 percentage points on Something-Something V2 and 0.5 points on Kinetics-400 [58]. This supports temporally consistent masking for video pretraining.
Integrating semantic priors from large-scale pretrained models is a promising direction for improving the semantic accuracy of restoration.

4.2.4. Using Pretrained Vision–Language Models as Semantic Priors

Large-scale pretrained vision–language models, such as CLIP (Contrastive Language–Image Pretraining) [64], are trained on millions of image–text pairs to acquire rich cross-modal knowledge of semantic relationships between visual and textual content. These models can serve as powerful semantic priors for restoration tasks, guiding generative processes toward semantically meaningful results.
DA-CLIP (Degradation-Aware Contrastive Language–Image Pretraining) adapts the frozen CLIP image encoder by training an additional controller to predict high-quality feature embeddings for multitask image restoration. Integrating the embeddings into the restoration network through cross-attention enables the model to be guided toward learning highly accurate image reconstruction. The controller naturally classifies different degradation types, producing a feature that corresponds to the actual distortions in the input. On a mixed-degradation dataset, DA-CLIP reports strong performance on both degradation-specific and unified restoration tasks [65]. Figure 14 illustrates the DA-CLIP architecture.
Figure 14. Architecture of DA-CLIP: adapting a pretrained Contrastive Language–Image Pretraining (CLIP) encoder for degradation-aware multitask image restoration [65]. Closed blue and open red padlocks indicate frozen and trainable parameters, respectively; solid arrows show information flow, and dashed purple arrows show gradient flow.
XPSR (Cross-modal Priors for Diffusion-based Super-Resolution) uses multimodal large language models (MLLMs) to obtain semantic conditions for super-resolution. An attention module for semantic feature fusion efficiently combines cross-modal priors. A regularizer that suppresses the influence of distortions isolates semantically preserved information rather than unwanted degradations. The method demonstrates the ability to generate highly accurate and realistic images on synthetic and real-world datasets [23]. RestoreAgent uses an MLLM to autonomously assess the type and severity of degradation, determine suitable tasks, optimize their sequence, and perform restoration using the most appropriate models. Experiments demonstrated that RestoreAgent modules outperformed humans in handling complex degradations [66]. Figure 15 illustrates the XPSR architecture.
Figure 15. Architecture of XPSR: a cross-modal text prior for diffusion-based super-resolution (×4) [23]. Panel (a) describes semantic prior guidance from an MLLM via the CLIP text encoder.
Methods based on CMA (Cross-Modal Alignment) use a pretrained vision–language model (VLPM) as an encoder, introducing cross-modal alignment distillation and “within-sample” distribution distillation for image inpainting. Adversarial training further improves the model’s ability to effectively fill missing regions in complex structures [67]. Methods using a VLM prior are compared in Table 12.
Table 12. Comparison of methods using pretrained VLMs as semantic priors for image restoration. Tasks, datasets, and evaluation protocols differ, and CLIP-related or alternative scores are not interchangeable. Values are representative results rather than a direct ranking.
Key takeaways. Pretrained vision–language models provide degradation-aware embeddings, textual conditions, or control of a restoration pipeline [22,23,65,66]. These roles entail different conditioning and adaptation costs. CLIP image–text similarity, image-quality scores, and degradation-classification accuracy measure different properties; none alone establishes that generated details match the lost image content.
SEER (SElf-supERvised) [68] implements knowledge-driven semantic image restoration based on alignment of vision–language diffusion models. To this end, it introduces a heuristic channel-aware prompt control strategy that can dynamically adapt prompt representations. The Dual-modal integrative semantic restoration mechanism (DISR) integrates visual features with language information, using cross-attention to align semantics and ensure complementary information in the semantic latent space.
The key advantage of using pretrained vision–language models is access to extensive semantic knowledge acquired during large-scale pretraining, enabling semantically accurate restoration without training from scratch. Cross-modal alignment emerges naturally through a shared representation space. The main limitations include dependence on the quality of pretrained models and the difficulty of ensuring precise semantic control under severe input distortions and specific degradation types.
Integrating multimodal priors from different sources and modalities requires the development of effective fusion mechanisms, discussed in the next subsection.

4.2.5. Multimodal Fusion Strategies and Cross-Modal Alignment

Effective restoration of semantic content across modalities requires fusion methods that ensure consistent representations and preserve semantic information when transitioning between modalities. Fusion strategies differ in their level of integration, alignment mechanisms, and approaches to ensuring semantic consistency. Figure 16 shows the SAM architecture for semantic alignment across multiple images [69].
Figure 16. Architecture of SAM for semantic alignment across multiple images: assisted visual token extraction (Part A) and contextual semantic generation (Part B) [69].
SAM introduces bidirectional semantic guidance during visual token extraction to preserve linking information between images for coherent analysis and to align semantics before input to an MLLM. In the reported comparisons, the method achieves gains of 37% for group captioning and 22% for storytelling on the CIDEr (Consensus-based Image Description Evaluation) metric [69].
MMGInpainting (Multi-Modality Guided Inpainting) implements multimodal guidance for diffusion-based image inpainting, taking an image and text as conditions for content generation. An adapted U-Net architecture with Anchored Stripe Attention uses anchor points to efficiently model global contextual dependencies. The Semantic Fusion Encoder guides the reverse process of the diffusion model, iteratively performing denoising. Examining the interaction of different modalities provides users with useful guidance for a more controllable inpainting procedure [70].
Diffusion Bridges for Cloud Removal (DB-CR) addresses cloud removal from optical satellite images by fusing them with synthetic aperture radar (SAR) images. A dual-branch architecture with an efficient feature extractor and dedicated cross-modal fusion blocks is used to extract and combine features effectively. The diffusion-bridge formulation directly connects the distributions of cloudy and cloud-free images. This avoids the complexity of sampling from pure Gaussian noise and achieves highly accurate results efficiently [5]. Figure 17 illustrates the DB-CR architecture.
Figure 17. A schematic diagram of a method proposed by Hu et al., illustrating the inference process. Each iteration alternates between a reverse diffusion-bridge step, which estimates the cloud-free state, and a forward diffusion-bridge step, which progresses the state to the next diffusion timestep. The process starts from the cloudy image, progressively refining it toward a cloud-free output. T is the number of diffusion timesteps used in training the diffusion bridge. During inference, N = T/s diffusion timesteps are used, where s is the step size. z refers to the synthetic aperture radar (SAR) image. Architecture of DB-CR: a multimodal diffusion bridge with SAR cross-modal attention for cloud removal from satellite images [5].
Unbiased Missing-modality Multimodal Learning [71] decomposes restoration into three stages: global structure generation, modality transfer, and refinement of local details. A duplex cross-diffusion framework simultaneously processes diffusion for available and missing modalities, facilitating the learning of cross-modal dependencies. This approach effectively restores missing modalities while maintaining semantic consistency with the available input data.
Semantic Alignment for MLLMs addresses the problem of independently extracting visual tokens for each image. Introducing bidirectional semantic guidance preserves linking information and aligns the semantics of different images before they are processed by an LLM. The advantages of multimodal fusion strategies include the ability to use complementary information to improve restoration quality, achieve semantic consistency through explicit alignment mechanisms, and adapt to different configurations of available data types. Their main drawbacks are increased computational complexity when processing multiple modalities, the need for careful data synchronization and alignment during training, and the difficulty of assessing the semantic correctness of cross-modal relationships in restored results [69]. Fusion strategies are compared in Table 13.
Table 13. Comparison of multimodal fusion methods for image and content restoration tasks. The table includes restoration and related tasks with different evaluation criteria. Results across these tasks and datasets do not constitute a direct ranking of restoration methods.
Key takeaways. Table 13 distinguishes restoration systems [5,67,70] from fusion, representation-learning, and captioning systems that supply related mechanisms [6,69,72,73]. Auxiliary information is useful only to the extent that it is relevant and correctly aligned with the target. Practical evaluation should therefore include unavailable, degraded, or conflicting auxiliary inputs, as well as the cost of processing each additional modality.
Open questions include developing more efficient cross-modal attention mechanisms that reduce computational costs while preserving alignment quality, creating unified architectures capable of handling arbitrary combinations of modalities without retraining, and formulating evaluation metrics that adequately reflect the semantic consistency and perceptual quality of multimodal restoration.

4.2.6. Limitations and Future Directions

Modern machine learning-based methods for semantic content restoration demonstrate considerable progress in the use of deep generative models, transformer architectures, self-supervised learning methods, pretrained vision–language models, and multimodal fusion strategies. Diffusion models enable perceptually realistic restoration through powerful generative priors but require substantial computational resources and face controllability challenges. Transformer architectures effectively model semantic dependencies and enable cross-modal attention, but their computational complexity remains high when processing high-dimensional data. Masked autoencoders efficiently learn multimodal representations through self-supervised reconstruction but require carefully designed masking strategies. The use of pretrained vision–language models provides access to extensive semantic knowledge, but adapting frozen encoders to specific degradations remains challenging.
Key limitations of current approaches include dependence on large-scale annotated or synchronized multimodal data, vulnerability to shifts in data distributions, the emergence of semantic inconsistencies under severe degradation, and the lack of universal metrics for assessing the semantic correctness of restoration. Directions for further research include developing more efficient multimodal diffusion and transformer architectures, using multimodal large language models as semantic priors for content restoration, creating new metrics and evaluation protocols for semantic and cross-modal restoration quality, and improving the interpretability, reliability, and robustness of restoration systems.
The taxonomy also separates restoration from learning through reconstruction. In MAE [18] and Hi-End-MAE [41], masking defines a pretraining task; the downstream results in Table 10 evaluate the learned representation rather than exact recovery of a damaged image. For generative inpainting and super-resolution, the output itself is the reconstruction, so consistency with observed data and the status of generated details must be assessed. Cross-modal restoration adds a further condition: complementary measurements can reduce uncertainty only to the extent that they are aligned and informative for the missing content, as illustrated by optical–SAR cloud removal [5].

4.3. Roles of Semantic Information in Restoration

4.3.1. Sources and Reliability of Semantic Representations

Extracting semantic representations from degraded content is a key step in semantically guided restoration. A semantic representation is a compact description of a signal’s content that is invariant to its specific form. The main difficulty in obtaining such a representation is that feature extraction methods trained on clean data lose accuracy when applied to degraded data in which both form and content are distorted. The task therefore requires a representation that is simultaneously robust to input signal degradation and sufficiently rich for subsequent reconstruction or generation.
Early methods for extracting semantics for subsequent restoration relied either on handcrafted features or on unimodal encoders as sources of representations. However, these representations were often either tied to a specific set of categories, without generalizing to arbitrary content, or insufficiently robust to noise. A common limitation of early approaches is that they are generally optimized independently for each modality (intra-modal) and lack mechanisms for adapting to the specific degradation.
SPIRE [22] and SSP-IR [24] illustrate different forms of semantic conditioning. SPIRE exposes content and restoration conditions through prompts; SSP-IR combines explicit textual, implicit visual, and structural priors. Their architectures are discussed in Section 4.2. Here, the relevant distinction is whether a condition specifies global content, constrains structure, or supplies complementary information, and how reliably it can be extracted from the degraded input.
XPSR [23] and LLMRA [74] use an MLLM-generated description as a restoration condition. When this description is inferred from the degraded observation, it is not an independent measurement. Agreement between the description and the generated output therefore does not, by itself, establish fidelity to the unknown original.
An approach based on discrete semantic tokens is developing in parallel. In speech restoration, MaskSR2 [75] (2025) uses semantic knowledge distilled from a pretrained HuBERT teacher to condition masked acoustic modeling. SenSE [26] (2025) proposes a two-stage architecture for semantic speech enhancement: in the first stage, a specialized language model with a tokenizer converts a degraded audio stream into clean semantic tokens; in the second, these tokens, together with the mel spectrogram of the degraded speech, condition a flow-matching model that generates clean speech. This decomposition eliminates semantic ambiguity by separating semantics (“what was said”) from acoustics (“how it sounded”).
In Token-based Audio Inpainting (2026) [76], discrete diffusion is applied to sequences of semantic tokens from a pretrained tokenizer rather than to a continuous feature space. This enables semantically coherent restoration of long gaps (up to 750 ms).
Using MLLMs as semantic encoders substantially improves robustness to degradation. Token-based audio approaches improve the interpretability and controllability of restoration. Hierarchical encoders demonstrate good scalability at very low bitrates.
Nevertheless, several fundamental limitations remain. Dependence on MLLMs and diffusion models entails high computational costs. The content of implicit semantic representations is difficult to verify. Discrete audio token representations depend on the training data distribution. Table 14 presents the characteristics of the methods, and Figure 18 shows the CoSeR architecture [77].
Table 14. Comparison of methods for semantic extraction and content reconstruction. Results use different datasets, corruption settings, and semantic-score definitions. They are representative evaluations and do not directly rank the semantic priors or restoration methods.
Figure 18. Architecture of CoSeR: a cognitive embedding as a semantic bridge between the low-resolution (LR) input and diffusion-based reconstruction of a high-resolution (HR) image [77].
Key takeaways. The methods differ in what they extract as a semantic prior and how that prior controls reconstruction. HINT’s results on CelebA-HQ show lower PSNR and higher FID for larger masks, illustrating the influence of missing information within one method [78]. Selecting a prior requires checking its robustness to the actual degradation and its computational cost, rather than ranking unrelated CLIP-based scores.

4.3.2. Ways of Using Extracted Semantics for Content Restoration

Four complementary roles of semantic information are compared in Table 15. They describe how a condition is used, rather than mutually exclusive model families; one method may combine several roles. An inferred description or embedding provides a prior, whereas an auxiliary sensor observation may supply additional measured information. This distinction matters when assessing whether a plausible reconstruction is supported by the available evidence.
Table 15. Four roles of semantic information in restoration.
For example, XPSR derives textual conditions from the degraded image using LLaVA [23], whereas DB-CR uses co-registered SAR observations for optical cloud removal [5]. Both use cross-modal guidance, but the information supporting reconstruction differs. A description inferred from the degraded image does not provide an additional measurement of the scene; SAR provides such a measurement with different sensing properties. Validation should therefore examine sensitivity to erroneous descriptions in XPSR and to spatial or temporal mismatch between observations in DB-CR. This comparison connects the source of auxiliary information to the checks needed to assess reconstruction fidelity.
Using a semantic representation as a global generation condition is a common strategy. In this scheme, a semantic description is supplied to the generative model to modulate its output. Models that implement this scheme include SPIRE [22], SSP-IR [24], and SAGA-SR [84].
Several approaches extract semantics as structural guides and constraints rather than as a global textual or vector condition. The central idea is that many degradations disrupt object structure without completely destroying semantic identity; extracting a degradation-invariant structure makes it possible to “fix” a framework within which the generative model fills in the details. Examples of this strategy include the SAGI framework [85] and the CSF-GAN [56], GenSE [86], and SemMIM [61] models.
Cross-modal approaches use the semantics of one modality to restore another, either through explicit conditions (text → image, audio → video) or through implicit alignment in a shared embedding space. Examples include XPSR (2024) [23], LLMGA (2024) [87], SenSE [26], and GVSC [88].
Another class is restoration through a common latent space in which the representations of all participating modalities are aligned. The rationale for this approach is that a “correct” reconstruction of any modality is obtained by projection from a shared semantic manifold. Learning in this space relies on contrastive losses (CLIP-like) or mutual information objectives that align representations of different modalities for the same semantic concepts. These principles are implemented in mm-GESCO [82] and MM-LDM [83]. Related latent-domain approaches include CASC [89] and FedDISC [90].
Figure 19 illustrates the MindEye2 pipeline [91], and Table 16 and Table 17 present cross-modal reconstruction results.
Figure 19. MindEye2 pipeline: three-stage decoding of functional magnetic resonance imaging (fMRI) brain signals into CLIP space, followed by image reconstruction through Stable Diffusion XL [91].
Table 16. Reported reconstruction quality and inference time of fMRI-to-image methods. MindEye1 [92] and MindEye2 [91] quality results are from [91]; Brain-Diffuser quality results are from [93] and use a different evaluation protocol. Inference times are from the single-GPU comparison of Zhang et al. [94] and are not paired measurements for all quality values above.
Table 17. Applications of extracted semantics to cross-modal reconstruction: brain decoding, visual retrieval, and semantic compression. Metrics describe different objectives and evaluation protocols and are not a common scale of restoration performance. Brain-decoding results are also subject to the protocol distinction stated in Table 16.
Key takeaways. In the comparison reported in [91], MindEye2 improves retrieval, pixelwise correlation, and SSIM over MindEye1, while its two-way identification accuracy is slightly lower. The measures therefore capture different aspects of reconstruction. Brain-Diffuser results use another evaluation protocol [93] and should not be interpreted as a matched comparison with the MindEye models. In the inference-time comparison reported by Zhang et al. [94], Brain-Diffuser requires 4.87 s per image, whereas MindEye1 and MindEye2 require approximately 12 s.
Key takeaways. Neural decoding, semantic compression, and image description use shared semantic representations for different objectives [69,80,91,93,95,96]. Identification or retrieval accuracy does not imply pixelwise recovery, and captioning gains do not measure restoration of a damaged image. Application-level validation must connect the reported metric to the intended reconstruction task and its acquisition and computational constraints.
Despite considerable progress, several key challenges remain unresolved. First, each of the four strategies faces an urgent need for a unified evaluation of the semantic correctness of the result: existing metrics, including CLIP Score, FID, PESQ, and word error rate (WER), cover different aspects of quality and do not provide a single interpretable scale. Second, cross-modal and multimodal frameworks are difficult to scale to an arbitrary number of modalities without explicit retraining: aligning three or more modalities simultaneously requires an exponentially increasing volume of paired multimodal data. Third, the use of MLLMs as semantic oracles when the input image itself is degraded is limited by their robustness to severe degradation.
Three conclusions follow from the combined comparison. First, an architectural family does not determine whether a task is recoverable: a denoising observation, a missing image region, and an auxiliary sensor measurement impose different constraints. Second, semantic conditions inferred from the target input and independent complementary observations carry different evidential weight. Third, validation must follow the restoration objective: a representation-learning result, a perceptually plausible completion, and a faithful reconstruction are different outcomes. These distinctions connect the method comparisons to the evaluation limitations and research directions discussed in Section 5.

5. Discussion

5.1. Scope of the Studies and Diversity of Tasks

The reviewed studies span domain-specific and general-purpose restoration. Specialized applications include 3D magnetic resonance imaging (MRI) inpainting (fastWDM3D), cloud removal (WaveDiffDecloud), and inpainting of flute audio with missing segments (Chithra K. R. et al. [43]). Domain specialization improves performance within that domain at the expense of generalizability [3,4,43]. General-purpose systems such as DA-CLIP (Luo et al., 2024 [65]), SPIRE (Qi et al., 2025 [22]), and RestoreAgent (Chen et al., 2024 [66]) broaden task coverage, raising the question of a trade-off with task-specific performance. For example, DA-CLIP reports an average PSNR of 27.01 dB across ten degradation datasets, whereas IR-SDE reports 31.65 dB on the Rain100H deraining task. These values describe different evaluation settings [22,54,65,66].
The bibliography of this review comprises primary empirical studies alongside reviews and surveys, dataset, benchmark and metric papers, theoretical background sources, and contextually related research on neural decoding, semantic communication and multimodal summarization. The number of cited references is therefore not the denominator of the primary-study corpus. Of 108 bibliographic records, 103 remained after deduplication on DOI or normalized title and consolidation of preprint and journal versions; of these, 47 satisfy the review’s operational definition of content restoration, namely the reconstruction of a degraded, incomplete or corrupted target signal with reference to that signal [1,7,10,16]. Figure 20 maps these 47 studies across eight restoration tasks and twelve method families. Methodological categories are treated as non-exclusive, since classical transform-domain, sparse and variational formulations are now commonly embedded within learned, generative or semantically conditioned pipelines [22,23,24,38,39,44,48,49,51,56,67,85,97].
Figure 20. Evidence map of restoration tasks and method families. Bubble size and labels show study counts; crosses mark combinations with no eligible study. The 47 studies contribute 103 task–method assignments because categories overlap. Source: authors’ evidence matrix [3,4,5,22,23,24,38,39,44,48,49,51,56,65,67,70,77,79,84,85,97].
Figure 20 maps the distribution of evidence within the reviewed corpus, not methodological effectiveness. Image inpainting spans the broadest range of method families, while sparse and dictionary-based methods are concentrated in image denoising [36,48,49,50,51,52,53,97]. Diffusion and score-based priors connect inpainting, super-resolution, multi-degradation restoration, and remote-sensing and medical reconstruction [3,4,5,22,23,24,38,54,65,68,70,77,79,84,85]. The audio and speech rows cover fewer families, with masked and token-based models prominent in the reviewed speech examples [26,75,76].
Sparse and empty cells identify task–family combinations that are weakly represented or absent in the displayed task–method assignments, including sparse and dictionary-based approaches in the audio rows and vision–language or agentic control in denoising and deblurring. They do not establish that these combinations are absent from the literature. Nor do cell counts establish superiority, maturity, or reliability: the studies use different data and protocols, and multiple assignments may describe components of the same hybrid pipeline [22,23,24,36,42,48,49,50,51,52,53,54,66,74].
Input-modality combinations require a separate qualitative reading because they are not an independent axis of Figure 20. The reviewed image methods include text-guided inpainting [67,70], text conditions derived from a degraded image [23,24], and complementary sensor observations such as SAR and optical images [5] or infrared and visible images [6]. The speech examples instead emphasize semantic and acoustic representations [26,75,76]. Restoration of damaged text and controlled comparisons under jointly degraded auxiliary inputs are not systematically covered by the present tables. These are limitations of this review’s evidence base, not demonstrated absences of research in those areas.

5.2. Metrics Used and Lack of Comparability Between Evaluations

For clarity, five image measures used in this review are defined as follows. Let x and y denote a reference and a reconstructed image, represented using the same range, color space and spatial alignment.
PSNR measures pixelwise fidelity. With N compared scalar pixel values and maximum possible value L,
M S E x , y = 1 N ∑ i = 1 N x i − y i 2
P S N R x , y = 10   l o g 10 L 2 M S E x , y
PSNR is expressed in dB and is higher for smaller squared error; exact reconstruction gives an infinite value. L is 255 for unnormalised 8-bit samples and 1 for samples scaled to [0, 1]. The evaluated channels and border cropping must match when comparing results [51].
SSIM compares local luminance, contrast and structure [29]:
S S I M x , y   =   2 μ x μ y   +   C 1 2 σ x y   +   C 2 μ x 2   +   μ y 2   +   C 1 σ x 2   +   σ y 2   +   C 2
Here μ denotes a local mean, σ2 a local variance, and σxy the local covariance; C1 = (K1L)2 and C2 = (K2L)2 stabilize the expression. The original implementation uses K1 = 0.01 and K2 = 0.03. Image-level SSIM averages local scores; higher is better, but the window and channel protocol must be specified.
LPIPS measures distance between normalized deep features [30]:
L P I P S x , y   =   ∑ l 1 H l W l ∑ h , w ‖ w l   ⊙   f ^ l x h , w   −   f ^ l y h , w ‖ 2 2
f ^ l is the layer-l feature vector normalized across channels, wl contains learned channel weights, ⊙ denotes elementwise multiplication, and Hl and Wl are spatial dimensions. Lower values indicate greater perceptual similarity under the chosen backbone and weights; they do not establish factual correctness.
FID compares image distributions rather than individual reference–output pairs [98]:
F I D   =   ‖ μ r   −   μ g ‖ 2 2   +   T r Σ r   +   Σ g   −   2 Σ r 1 2 Σ g Σ r 1 2 1 2
μr, Σr and μg, Σg are the sample means and covariance matrices of Inception features for reference and generated image sets; Tr denotes the matrix trace. Lower is better. The symmetric matrix-square-root form above is equivalent to the usual Gaussian Fréchet-distance expression. Feature extraction, preprocessing, and sample count affect the estimate. FID is a distributional quality measure, not a test of the semantic correctness of an individual reconstruction.
CLIPScore evaluates image–text alignment [99]:
C L I P S c o r e I , c   =   2.5   m a x c o s v I , t c ,   0
where v and t are CLIP image and text embeddings, and c is a caption. Higher values indicate stronger alignment under the encoder; the original definition uses ViT-B/32. The scale must be stated if scores are multiplied by 100. CLIPScore is distinct from raw CLIP image–image or image–text cosine similarity and from CLIP-based identification accuracy. These measures cannot be substituted for one another without specifying their definitions.
Evaluation heterogeneity is the principal obstacle to quantitative synthesis in this field. Classical methods are optimized primarily for pixel-level metrics such as PSNR and SSIM, which assess signal- and structure-level correspondence to a reference [28,29]. Generative models also use LPIPS and FID to characterize feature-level similarity and distributional plausibility, respectively [30,98]. These objectives can diverge under the perception–distortion trade-off [2]. Audio and speech studies employ a partly non-overlapping set of measures, including PESQ, STOI and mean opinion scores [31,32], as well as cross-correlation [43]. Studies reconstructing images from brain signals report CLIP-based two-way identification accuracy, pixelwise correlation (PixCorr) and image retrieval accuracy [91], while semantically conditioned systems may additionally report embedding-based alignment or task-specific semantic measures [33,34,35,99]. Table A1 summarizes the methods, datasets, and reported restoration metrics. Figure 21 therefore reports, for each restoration task, the proportion of verified evaluation records in which each measure was explicitly disclosed. It deliberately does not compare numerical values, because the underlying results were obtained under incommensurable tasks, datasets, degradation models, reference definitions and metric implementations.
Figure 21. Metric-reporting coverage by restoration task. Cells show counts and within-task percentages of study × dataset × task records reporting each metric (61 records from 33 studies). Zero indicates no verified report of the metric; n/a indicates no evaluation records. Source: [2,14,25,26,28,29,30,31,32,43,75,76,84,86,98]. Darker teal shading indicates a higher within-task reporting percentage; white cells indicate 0%, and gray cells indicate n/a.
The coverage map separates consistently reported fidelity measures from inconsistently reported perceptual and semantic ones. PSNR was disclosed in 43 of 61 evaluation records (70%) and SSIM in 32 (52%), whereas LPIPS and FID each appeared in 20 records (33%); coverage is strongly task-dependent, reaching complete PSNR reporting in super-resolution, deblurring and multi-degradation restoration but only 45% in denoising, where several sources report qualitative or relative results only. LPIPS and FID coverage also varies across tasks: image inpainting reports these metrics in 47% and 41% of records, respectively; super-resolution reports FID in 62%, and remote-sensing and medical restoration reports LPIPS in 75%, while denoising reports neither. Uncertainty calibration, computational cost, latency and downstream task utility were reported too rarely to support tabulation at all. This imbalance limits what apparently favorable benchmark results can establish: maximizing pixelwise fidelity may reduce perceptual realism and does not ensure semantic correctness, while generative restoration may introduce plausible content that is not supported by the observed measurement [2,30,85]. We therefore recommend that restoration studies report at least one fidelity measure, one perceptual measure, and one task-relevant or semantic measure, together with the degradation model, reference definition and metric implementation used.

5.3. Interpretation of the Combined Findings

5.3.1. Dominant Patterns and Trends

The literature reviewed reveals several patterns indicating a shift in restoration solutions toward semantic generation. A substantial proportion of machine learning-based studies invoke semantics through semantic priors (CLIP, DA-CLIP, XPSR, SPIRE) and dictionaries (VLM, SemanIR), semantic structural constraints (SSP-IR, SAGI), and multimodal semantic alignment (MMAE, SemMIM). This indicates a shift in how the task is addressed: from minimizing signal distance to maximizing semantic consistency [22,23,24,55,61,62,64,65,85].
Approaches are also increasingly combined to compensate for limitations and complement one another’s strengths: diffusion models are hybridized with wavelet transforms (fastWDM3D, WaveDiffDecloud), variational inference (VIPaint), CLIP conditions (DA-CLIP), and transformers (SPIRE [22] with ControlNet, MRIR/SSP-IR). Transformers integrate CNN components (SemanIR with a CNN extractor) and global attention mechanisms [3,4,24,38,55,65].
Whereas early studies in the field focused on well-defined, single-degradation settings, modern methods explicitly target mixed degradations involving multiple distortions of different kinds simultaneously, making method development and evaluation more difficult [10].
MindEye2 and Brain-Diffuser reconstruct visual images from fMRI observations, whereas MRIR/SSP-IR derives semantic conditions from the degraded image using pretrained multimodal models [24,91,93]. These related reconstruction settings distinguish information supplied by an auxiliary observation from a condition inferred from the target input.
Sensor applications also expose a gap between restoration quality and decision reliability. Table 18 separates the evidence summarized in this review from the validation needed for downstream use; the final column states evaluation requirements, not additional results.
Table 18. Sensor applications and the evidence needed for downstream use.

5.3.2. Relationship to Existing Theoretical Frameworks

The studies reviewed can be situated within two competing theoretical paradigms: Bayesian estimation and semantic generation. Classical methods formulate restoration as MAP estimation, seeking the most probable clean signal given a noisy observation and a prior model. This paradigm is mathematically rigorous and provides a number of theoretical guarantees, provided that the prior model is correct. Deep generative methods, by contrast, follow a different paradigm: rather than minimizing the expected restoration error, they maximize the perceptual plausibility of the result within a distribution of “realistic” outputs [2,38,46].
The Information Bottleneck framework (Tishby et al. [100]) is useful for interpreting semantic methods: DA-CLIP [65] and SemanIR [55] implicitly implement the principle of sufficient statistics by retaining only semantically relevant information and discarding noise. In content restoration, Occam’s razor is reflected in sparse coding methods (MS-GSRC, CGSRC): a parsimonious representation of a signal using a minimal number of dictionary atoms reflects a Bayesian prior of compressibility. In neuroscience, the MindEye2 and Brain-Diffuser studies connect with neural coding theory [36,53,91,93,100].

5.4. Limitations and Critical Assessment

Limitations of the Primary Studies

The studies reviewed have several limitations in terms of generalizability. The first is data leakage. Pretraining on datasets containing images included in benchmarks creates the potential for leakage. For example, VLM-based methods that use MLLMs such as LLaVA-7B are trained on corpora that overlap with benchmarks, leading to inflated performance metrics [23,24].
A further limitation is reliance on synthetic degradations, which may not reproduce the complexity of real acquisition processes. Evaluation on both synthetic and real-world data is therefore important [23,24]. MRIR/SSP-IR reports PSNR values of 19.18 dB on DIV2K-val and 23.76 dB on RealSR [24], but these different test settings do not isolate a generalization effect or demonstrate a drop in performance on real data.
Semantic priors are used in the image and speech methods discussed above [26,75,85], but perceptual plausibility alone does not establish fidelity [2]. A key risk is semantic hallucination: the generation of plausible but incorrect structures, words, or attributes during restoration. Illustrative failures include an invented or erased image structure, a substituted word or speaker identity in restored speech, an OCR completion containing a plausible but incorrect name or number, and an image reconstruction that follows a conflicting caption or auxiliary observation. Figure 22 and Figure 23 provide published examples of an invented structure during image inpainting [101] and an incorrect digit generated from a damaged receipt [102]. They motivate checks beyond aggregate image or audio quality: consistency with retained measurements, verification of object attributes or recognized text, and comparison against the reference transcript or speaker identity when available. For multimodal methods, removing, corrupting, or deliberately contradicting the auxiliary input would test whether the reconstruction follows the observation or an unreliable condition.
Figure 22. Semantic hallucination during image inpainting: (a) reference, (b) masked input, and (c) Stable Diffusion output containing a pavilion absent from the reference (red box). Adapted from Yang [101], Figure 13, © The Author(s) 2026, under CC BY 4.0; original headings and panel D omitted, panel labels added.
Figure 23. OCR-based key-information extraction from an original and damaged receipt. The model replaces the missing digit with an incorrect digit. Reproduced from Hwang et al. [102], Figure 8, © 2026 Association for Computational Linguistics, under CC BY 4.0.
Finally, the analysis of computational resources is limited: a substantial proportion of the studies reviewed either do not report inference time or report it only for optimized configurations on specialized hardware. The absence of a standardized protocol for evaluating computational efficiency prevents a uniform ranking of practical efficiency across tasks and implementations.
Table 19 separates training or task adaptation from inference. It includes published runtime measurements where their test conditions can be identified, together with the reported hardware and memory limitations. The rows describe representative implementations, not a common benchmark; CPU timings from 2007, modern GPU measurements, two-dimensional images and three-dimensional volumes cannot be used to rank the families by speed. The two Restormer rows are a controlled depth–width ablation, not different final benchmark checkpoints. For fastWDM3D, the published abstract confirms the two-step, 1.81 s result; the detailed training setup and memory figure are taken from the authors’ version of the same work.
Table 19. Computational requirements: representative measurements and their conditions.
The within-paper comparisons expose two practical trade-offs. BM3D’s Fast profile reduces processing time at a small PSNR cost under moderate noise, while the wider Restormer ablation trains and runs faster but gives slightly lower PSNR. Diffusion cost also depends on the sampling schedule: fastWDM3D uses two steps, whereas XPSR uses twenty and additionally obtains semantic conditions from an MLLM. Step counts alone do not measure latency because network size, input dimensionality, guidance and conditioning differ. These observations support testing single-pass or reduced-step implementations when latency matters; they do not establish real-time or edge-device suitability. That decision requires end-to-end latency and peak inference-memory measurements on the target hardware, including semantic-condition generation. Foundation-model pretraining must also be distinguished from the task-specific adaptation costs reported here.

5.5. Research Prospects and Directions for Future Studies

The evidence assembled in Figure 20 and Figure 21 supports a research agenda defined by validation requirements rather than by an expectation that any single method family will supersede the others. Three findings constrain that agenda. First, methodological development is integrative: the same study frequently combines a signal-domain prior with a learned or generative component, so the 47 eligible studies produced 103 task–method assignments. Second, evaluation reporting is incomplete and unevenly distributed across tasks, which prevents quantitative pooling and makes claims of state-of-the-art performance difficult to verify independently. Third, semantically conditioned and generative restoration introduces a class of risk that fidelity metrics do not measure, namely the insertion of plausible but unsupported content [2,3,4,5,85]. Figure 24 organizes the resulting priorities into methodological streams, near-term validation requirements and long-term outcomes.
Figure 24. Research roadmap for content restoration. Five methodological streams are linked to near-term validation requirements and long-term outcomes. Arrows indicate methodological dependencies; the lower lane shows adjacent research excluded from the core corpus. Source: authors’ synthesis [3,4,5,18,22,23,24,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,70,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,95,96,97].
The roadmap implies concrete reporting obligations rather than aspirational goals. Matched benchmarks with realistic rather than synthetic degradations, and external validation under domain shift, are prerequisites for attributing an improvement to a method rather than to a degradation model [28,29,30,98]. Separating fidelity from plausibility and reporting calibrated uncertainty would make the perception–distortion trade-off explicit instead of implicit [2,85]. Ablation of individual modalities and testing under conflicting or missing inputs are required before a multimodal or vision–language-conditioned system can be claimed to exploit semantic information rather than dataset priors [5,56,65,66,67,68,70,74,77,79,84,85]. Disclosure of compute, latency and training conditions is a precondition for reproducibility, and explicit provenance marking of generated regions is a precondition for use in evidential, medical and remote-sensing settings, where a plausible but fabricated reconstruction is a safety failure rather than a quality defect [3,4,5,85].

5.5.1. Unified Metrics for the Semantic Correctness of Restoration

The comparisons in Table 6, Table 8, Table 12 and Table 14 distinguish reference fidelity, perceptual quality, and image–text alignment, but do not provide a common test of whether a restored detail is supported by the observation. This motivates a specific question: can an evaluation protocol detect semantic errors while remaining sensitive to useful restoration? A practical test would pair ordinary degradation benchmarks with controlled changes to object attributes, text, or spatial relationships, and compare metric responses with reference-based task judgments. PSNR and SSIM, perceptual measures, and semantic or task measures should be reported separately rather than collapsed into an unvalidated score [28,29,30,99]. Multimodal language models may assist this assessment, but their agreement with reference judgments and sensitivity to misleading context require evaluation before they are used as judges.

5.5.2. Theoretical Guarantees for Generative Restoration Methods

The generative and variational methods compared in Table 6 and Table 8 can produce plausible completions even when the missing information is not uniquely determined by the input. The resulting research question concerns the conditions under which a reconstruction can be accompanied by a useful guarantee, rather than whether perceptual quality can be increased further. Under a declared observation model, uncertainty estimates should be tested for calibration on held-out data and under model mismatch; errors should also be localized to task-relevant structures. Comparing sampled reconstructions with retained measurements tests data consistency; variation across samples can help characterize uncertainty about unobserved details. Guarantees must state their assumptions about the degradation process and data distribution instead of treating plausibility as evidence of fidelity [2,38].

5.5.3. Restoration Under Extreme Degradation and Incomplete Information

The reviewed inpainting and super-resolution studies use different masks, degradation levels, and auxiliary information, so their reported scores do not identify a common boundary of recoverability. A testable question is when a particular attribute or structure ceases to be supported by the observation, and how independent context changes that boundary. Experiments should progressively remove information while keeping the scene, evaluation target, and protocol fixed, then compare reconstruction with and without auxiliary input. Alongside quality scores, they should record task-relevant errors, uncertainty, and cases where the method cannot support a reliable conclusion. Information-theoretic limits motivate this analysis, but a general law of semantic recoverability cannot be inferred from the heterogeneous benchmarks reviewed here [2,38,103].

5.5.4. Cross-Modal Restoration as a Promising Paradigm

The semantic-guidance strategies in Section 4.3 and SAR-guided cloud removal [5] show how auxiliary information can constrain image reconstruction. They also expose a specific uncertainty: whether improvement persists when that information is incomplete, misaligned, outdated, or contradictory. The resulting research question is how a restoration system should weight the target observation against an unreliable auxiliary modality. Evaluation should compare matched, missing, corrupted, and conflicting auxiliary inputs while holding the target degradation fixed. Performance on intact regions, semantic errors, and uncertainty should be measured separately. Neural decoding studies provide related mechanisms for alignment [91,93,95], but validation of these mechanisms for a particular restoration task requires the corresponding target observations and controls.

5.5.5. Training Efficiency and Low-Resource Scenarios

The computational limitations in Section Limitations of the Primary Studies and the dependence of semantic models on pretraining motivate separating data efficiency from deployment cost. MindEye2 reports CLIP-based two-way identification accuracy of 79.2% with one hour of subject-specific data, compared with 93.0% under full training, in its brain-decoding setting [91]. This adjacent example motivates controlled tests of adaptation with limited data in restoration. The research question is how much task-relevant fidelity can be retained when adaptation data, training time, memory, or inference latency is constrained. Comparisons should use the same degradation protocol and distinguish pretraining, adaptation, and inference budgets. Few-shot adaptation, augmentation, contrastive objectives, and active selection of training examples can then be assessed as concrete interventions rather than grouped under a general low-resource claim.
Deployment-oriented research should also distinguish restoration provenance from output quality. A perceptually convincing restoration may contain content inferred from a generative prior rather than recovered from the observation [2]. Systems intended for medicine, forensics, archives, journalism, or scientific imaging should therefore preserve the original input, record the model version and restoration parameters, identify generated regions where feasible, and communicate uncertainty to users. Provenance specifications such as those of the Coalition for Content Provenance and Authenticity (C2PA) provide mechanisms for recording source assets and editing actions [104]. Evaluation should also address traceability, the potential for misleading reconstructions, and the effect of restored content on downstream human or automated decisions.

6. Conclusions

This image-centered review examines content restoration through the relationship between degradation, semantic information, and validation, with selected audio, text, and cross-modal studies providing comparative context. The main methodological trend is integration: transform-domain, variational, and sparse models remain relevant within learned and generative pipelines. Semantic information can act as a global condition, a structural guide, a cross-modal constraint, or part of a shared latent representation. These roles complement algorithm families and learning paradigms; they do not define mutually exclusive classes. Their usefulness depends on the information retained in the observation and on the reliability of the condition supplied to the model.
The comparative evidence supports several practical conclusions. Results obtained with different datasets, degradations, or metric implementations cannot establish a universal ranking. Better perceptual quality or embedding alignment does not by itself demonstrate that a recovered detail is correct. Computational comparisons also require explicit training, adaptation, and inference conditions. For sensor applications, including medical imaging and remote sensing, restoration should therefore be evaluated both against the observed data and against the requirements of the subsequent task. Improvements in reconstruction scores must be distinguished from demonstrated improvements in diagnosis, recognition, or scene interpretation.
The resulting research priorities concern validation as much as model design: detecting unsupported semantic changes, quantifying uncertainty under severe information loss, testing unreliable or conflicting auxiliary inputs, and measuring performance under realistic resource constraints. Controlled experiments and transparent reporting are needed to determine when semantic guidance adds evidence and when it merely adds plausibility. Some lost information cannot be uniquely recovered, and a credible output may remain a conditional reconstruction. Preserving the original observation, documenting the restoration process, and communicating uncertainty are therefore central to reliable use of semantically guided methods.

Author Contributions

Conceptualization, O.E. and R.M.; methodology, O.E. and R.M.; formal analysis, V.B., B.N. and L.R.; investigation, V.B., B.N. and L.R.; writing—original draft preparation, V.B., B.N. and L.R.; writing—review and editing, V.B., B.N., L.R., O.E. and R.M.; supervision, O.E. and R.M. V.B., B.N. and L.R. contributed equally to this work. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Russian Science Foundation, grant number 24-11-00340.

Institutional Review Board Statement

Not applicable.

Data Availability Statement

The data discussed in this review are available in the cited publications. The extracted comparisons are presented in the tables and Appendix A.

Acknowledgments

The supercomputer of HSE University was used to conduct the computational research.

Conflicts of Interest

The authors declare no conflicts of interest. The funder had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Appendix A. Summary Table of the Methods Reviewed (Table A1)

Table A1. Summary comparison of the image restoration and multimodal processing methods reviewed in Section 4. Missing-data notation: —1, not reported in the cited evaluation; —2, not located during verification; —3, not applicable to the reported task or comparison; —4, unavailable with the reason not established in the current evidence record. ↓: lower values are preferable.

References

  1. Jam, J.; Kendrick, C.; Walker, K.; Drouard, V.; Hsu, J.G.-S.; Yap, M.H. A comprehensive review of past and present image inpainting methods. Comput. Vis. Image Underst. 2021, 203, 103147. [Google Scholar] [CrossRef] [Scilit]
  2. Blau, Y.; Michaeli, T. The Perception-Distortion Tradeoff. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 6228–6237. [Google Scholar] [CrossRef] [Scilit]
  3. Durrer, A.; Bieder, F.; Friedrich, P.; Menze, B.; Cattin, P.C.; Kofler, F. fastWDM3D: Fast and Accurate 3D Healthy Tissue Inpainting. In Deep Generative Models, Daejeon, South Korea, 23 September 2025; Springer: Cham, Switzerland, 2026; Volume 16128, pp. 171–181. [Google Scholar] [CrossRef] [Scilit]
  4. Huang, Y.; Wang, Z.; Luo, M.; Qiu, S. WaveDiffDecloud: Wavelet-domain conditional diffusion model for efficient cloud removal. Comput. Geosci. 2026, 209, 106121. [Google Scholar] [CrossRef] [Scilit]
  5. Hu, Y.; Lohit, S.; Kamilov, U.S.; Marks, T.K. Multimodal Diffusion Bridge with Attention-Based SAR Fusion for Satellite Image Cloud Removal. IEEE Trans. Geosci. Remote Sens. 2025, 63, 5639612. [Google Scholar] [CrossRef] [Scilit]
  6. Yi, X.; Xu, H.; Zhang, H.; Tang, L.; Ma, J. Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 17–21 June 2024; pp. 27026–27035. Available online: https://openaccess.thecvf.com/content/CVPR2024/html/Yi_Text-IF_Leveraging_Semantic_Text_Guidance_for_Degradation-Aware_and_Interactive_Image_CVPR_2024_paper.html (accessed on 6 September 2026).
  7. Zong, Y.; Mac Aodha, O.; Hospedales, T.M. Self-Supervised Multimodal Learning: A Survey. IEEE Trans. Pattern Anal. Mach. Intell. 2025, 47, 5299–5318. [Google Scholar] [CrossRef] [Scilit]
  8. Xiang, H.; Zou, Q.; Nawaz, M.A.; Huang, X.; Zhang, F.; Yu, H. Deep learning for image inpainting: A survey. Pattern Recognit. 2023, 134, 109046. [Google Scholar] [CrossRef] [Scilit]
  9. Li, X.; Ren, Y.; Jin, X.; Lan, C.; Wang, X.; Zeng, W.; Wang, X.; Chen, Z. Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey. Int. J. Comput. Vis. 2025, 133, 8078–8108. [Google Scholar] [CrossRef] [Scilit]
  10. Jiang, J.; Zuo, Z.; Wu, G.; Jiang, K.; Liu, X. A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trends. IEEE Trans. Pattern Anal. Mach. Intell. 2025, 47, 11892–11911. [Google Scholar] [CrossRef] [Scilit]
  11. Zhang, X.; Zhai, D.; Li, T.; Zhou, Y.; Lin, Y. Image inpainting based on deep learning: A review. Inf. Fusion 2023, 90, 74–94. [Google Scholar] [CrossRef] [Scilit]
  12. Quan, W.; Chen, J.; Liu, Y.; Yan, D.-M.; Wonka, P. Deep Learning-Based Image and Video Inpainting: A Survey. Int. J. Comput. Vis. 2024, 132, 2367–2400. [Google Scholar] [CrossRef] [Scilit]
  13. Moliner, E.; Välimäki, V. Diffusion-Based Audio Inpainting. J. Audio Eng. Soc. 2024, 72, 100–113. [Google Scholar] [CrossRef] [Scilit]
  14. Asaad, I.; Jacquelin, M.; Perrotin, O.; Girin, L.; Hueber, T. Is self-supervised learning enough to fill in the gap? A study on speech inpainting. Comput. Speech Lang. 2026, 99, 101922. [Google Scholar] [CrossRef] [Scilit]
  15. Zhang, H.; Yu, P.S.; Zhang, J. A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models. ACM Comput. Surv. 2025, 57, 277:1–277:41. [Google Scholar] [CrossRef] [Scilit]
  16. Wu, J.; Gan, W.; Chen, Z.; Wan, S.; Yu, P.S. Multimodal Large Language Models: A Survey. In Proceedings of the 2023 IEEE International Conference on Big Data (BigData), Sorrento, Italy, 15–18 December 2023; pp. 2247–2256. [Google Scholar] [CrossRef] [Scilit]
  17. Yin, S.; Fu, C.; Zhao, S.; Li, K.; Sun, X.; Xu, T.; Chen, E. A survey on multimodal large language models. Natl. Sci. Rev. 2024, 11, nwae403. [Google Scholar] [CrossRef] [Scilit]
  18. He, K.; Chen, X.; Xie, S.; Li, Y.; Dollár, P.; Girshick, R. Masked Autoencoders Are Scalable Vision Learners. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 19–24 June 2022; pp. 15979–15988. [Google Scholar] [CrossRef] [Scilit]
  19. He, Y.; Liu, Z.; Chen, J.; Tian, Z.; Liu, H.; Chi, X.; Liu, R.; Yuan, R.; Xing, Y.; Wang, W.; et al. LLMs Meet Multimodal Generation and Editing: A Survey. arXiv 2024, arXiv:2405.19334. Available online: https://arxiv.org/abs/2405.19334 (accessed on 6 September 2026).
  20. Huang, Y.; Huang, J.; Liu, Y.; Yan, M.; Lv, J.; Liu, J.; Xiong, W.; Zhang, H.; Cao, L.; Chen, S. Diffusion Model-Based Image Editing: A Survey. IEEE Trans. Pattern Anal. Mach. Intell. 2025, 47, 4409–4437. [Google Scholar] [CrossRef] [Scilit]
  21. Sapkota, R.; Raza, S.; Shoman, M.; Paudel, A.; Karkee, M. Multimodal Large Language Models for Image, Text, and Speech Data Augmentation: A Survey. arXiv 2025, arXiv:2501.18648. Available online: https://arxiv.org/abs/2501.18648 (accessed on 6 September 2026).
  22. Qi, C.; Tu, Z.; Ye, K.; Delbracio, M.; Milanfar, P.; Chen, Q.; Talebi, H. SPIRE: Semantic Prompt-Driven Image Restoration. In Computer Vision—ECCV 2024, Milan, Italy, 29 September–4 October 2024; Springer: Cham, Switzerland, 2025; pp. 446–464. [Google Scholar] [CrossRef] [Scilit]
  23. Qu, Y.; Yuan, K.; Zhao, K.; Xie, Q.; Hao, J.; Sun, M.; Zhou, C. XPSR: Cross-modal Priors for Diffusion-based Image Super-Resolution. In Proceedings of the Computer Vision—ECCV 2024, Milan, Italy, 29 September–4 October 2024; Springer: Cham, Switzerland, 2025; Volume 15069, pp. 285–303. [Google Scholar] [CrossRef] [Scilit]
  24. Zhang, Y.; Zhang, H.; Cheng, Z.; Xie, R.; Song, L.; Zhang, W. SSP-IR: Semantic and Structure Priors for Diffusion-Based Realistic Image Restoration. IEEE Trans. Circuits Syst. Video Technol. 2025, 35, 6259–6272. [Google Scholar] [CrossRef] [Scilit]
  25. Li, X.; Wang, Q.; Liu, X. MaskSR: Masked Language Model for Full-band Speech Restoration. In Proceedings of the Interspeech 2024, Kos, Greece, 1–5 September 2024; pp. 2275–2279. [Google Scholar] [CrossRef] [Scilit]
  26. Li, X.; Xie, H.; Wang, Z.; Zhang, Z.; Xiao, L.; Wang, S.; Xie, L. SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancement. arXiv 2025, arXiv:2509.24708. Available online: https://arxiv.org/abs/2509.24708 (accessed on 6 September 2026).
  27. Zhou, W.; Amirpour, H.; Timmerer, C.; Zhai, G.; Le Callet, P.; Bovik, A.C. Perceptual Visual Quality Assessment: Principles, Methods, and Future Directions. arXiv 2025, arXiv:2503.00625. Available online: https://arxiv.org/abs/2503.00625 (accessed on 6 September 2026).
  28. Ghazanfari, S.; Garg, S.; Flammarion, N.; Krishnamurthy, P.; Khorrami, F.; Croce, F. Towards Unified Benchmark and Models for Multi-Modal Perceptual Metrics. arXiv 2024, arXiv:2412.10594. Available online: https://arxiv.org/abs/2412.10594 (accessed on 6 September 2026).
  29. Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef] [Scilit]
  30. Zhang, R.; Isola, P.; Efros, A.A.; Shechtman, E.; Wang, O. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 586–595. [Google Scholar] [CrossRef] [Scilit]
  31. Rix, A.W.; Beerends, J.G.; Hollier, M.P.; Hekstra, A.P. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. In Proceedings of the 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221), Salt Lake City, UT, USA, 7–11 May 2001; Volume 2, pp. 749–752. [Google Scholar] [CrossRef] [Scilit]
  32. Taal, C.H.; Hendriks, R.C.; Heusdens, R.; Jensen, J. An Algorithm for Intelligibility Prediction of Time–Frequency Weighted Noisy Speech. IEEE Trans. Audio Speech Lang. Process. 2011, 19, 2125–2136. [Google Scholar] [CrossRef] [Scilit]
  33. Papineni, K.; Roukos, S.; Ward, T.; Zhu, W.-J. Bleu: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA, 7–12 July 2002; pp. 311–318. [Google Scholar] [CrossRef] [Scilit]
  34. Lin, C.-Y. ROUGE: A Package for Automatic Evaluation of Summaries. In Proceedings of the Text Summarization Branches Out, Barcelona, Spain, 25–26 July 2004; pp. 74–81. Available online: https://aclanthology.org/W04-1013/ (accessed on 6 September 2026).
  35. Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K.Q.; Artzi, Y. BERTScore: Evaluating Text Generation with BERT. In Proceedings of the 8th International Conference on Learning Representations (ICLR 2020), Virtual, 26–30 April 2020; Available online: https://iclr.cc/virtual_2020/poster_SkeHuCVFDr.html (accessed on 3 October 2026).
  36. Zhang, T.; Li, W.; Wu, D.; Gao, Q. Corrected Group Sparse Residual Constraint Model for Image Denoising and Inpainting. Circuits Syst. Signal Process. 2025, 44, 1184–1213. [Google Scholar] [CrossRef] [Scilit]
  37. Zamir, S.W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F.S.; Yang, M.-H. Restormer: Efficient Transformer for High-Resolution Image Restoration. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 19–24 June 2022; pp. 5718–5729. [Google Scholar] [CrossRef] [Scilit]
  38. Agarwal, S.; Hope, G.; Heo, J.; Sudderth, E.B. VIPaint: Image Inpainting with Pre-Trained Diffusion Models via Variational Inference. In Proceedings of the 29th International Conference on Artificial Intelligence and Statistics, Tangier, Morocco, 2–5 May 2026; Volume 300, pp. 4240–4248. Available online: https://proceedings.mlr.press/v300/agarwal26a.html (accessed on 3 October 2026).
  39. Suvorov, R.; Logacheva, E.; Mashikhin, A.; Remizova, A.; Ashukha, A.; Silvestrov, A.; Kong, N.; Goka, H.; Park, K.; Lempitsky, V. Resolution-robust Large Mask Inpainting with Fourier Convolutions. In Proceedings of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 4–8 January 2022; pp. 3172–3182. [Google Scholar] [CrossRef] [Scilit]
  40. Liang, J.; Cao, J.; Sun, G.; Zhang, K.; Van Gool, L.; Timofte, R. SwinIR: Image Restoration Using Swin Transformer. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Virtual, 11–17 October 2021; pp. 1833–1844. [Google Scholar] [CrossRef] [Scilit]
  41. Tang, F.; Yao, Q.; Ma, W.; Wu, C.; Jiang, Z.; Zhou, S.K. Hi-End-MAE: Hierarchical encoder-driven masked autoencoders are stronger vision learners for medical image segmentation. Med. Image Anal. 2026, 107, 103770. [Google Scholar] [CrossRef] [Scilit]
  42. Zhou, M.; Huang, J.; Guo, C.-L.; Li, C. Fourmer: An Efficient Global Modeling Paradigm for Image Restoration. In Proceedings of the 40th International Conference on Machine Learning, Honolulu, HI, USA, 23–29 July 2023; Volume 202, pp. 42589–42601. Available online: https://proceedings.mlr.press/v202/zhou23f.html (accessed on 6 September 2026).
  43. Chithra, K.R.; Remesh, A.; Sinith, M.S. Audio Inpainting Using Autoregressive Modeling of CWT Coefficients with Matched Wavelets. In Proceedings of the 2025 27th International Conference on Digital Signal Processing and Its Applications (DSPA), Moscow, Russia, 26–28 March 2025; pp. 1–6. [Google Scholar] [CrossRef] [Scilit]
  44. Cao, Y.; Ma, R.; Zhao, K.; An, P. WFIL-NET: Image inpainting based on wavelet downsampling and frequency integrated learning module. Multimed. Syst. 2025, 31, 21. [Google Scholar] [CrossRef] [Scilit]
  45. Rudin, L.I.; Osher, S.; Fatemi, E. Nonlinear total variation based noise removal algorithms. Phys. D Nonlinear Phenom. 1992, 60, 259–268. [Google Scholar] [CrossRef] [Scilit]
  46. Sumathi, G.; Uma Devi, M. High-resolution image inpainting using a probabilistic framework for diverse images with large arbitrary masks. Front. Artif. Intell. 2025, 8, 1614608. [Google Scholar] [CrossRef] [Scilit]
  47. Wang, J.; Wu, X.; Xu, H.; Kim, J. An efficient image inpainting algorithm based on a modified Gray–Scott model. Signal Process. 2024, 214, 109265. [Google Scholar] [CrossRef] [Scilit]
  48. Jia, H.; Yin, Q.; Lu, M. Blind-noise image denoising with block-matching domain transformation filtering and improved guided filtering. Sci. Rep. 2022, 12, 16195. [Google Scholar] [CrossRef] [Scilit]
  49. Jung, M. Group sparse representation and saturation-value total variation based color image denoising under multiplicative noise. AIMS Math. 2024, 9, 6013–6040. [Google Scholar] [CrossRef] [Scilit]
  50. Elad, M.; Aharon, M. Image Denoising via Sparse and Redundant Representations over Learned Dictionaries. IEEE Trans. Image Process. 2006, 15, 3736–3745. [Google Scholar] [CrossRef] [Scilit]
  51. Dabov, K.; Foi, A.; Katkovnik, V.; Egiazarian, K. Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering. IEEE Trans. Image Process. 2007, 16, 2080–2095. [Google Scholar] [CrossRef] [Scilit]
  52. Zhou, M.; Chen, H.; Paisley, J.; Ren, L.; Li, L.; Xing, Z.; Dunson, D.; Sapiro, G.; Carin, L. Nonparametric Bayesian Dictionary Learning for Analysis of Noisy and Incomplete Images. IEEE Trans. Image Process. 2012, 21, 130–144. [Google Scholar] [CrossRef] [Scilit]
  53. Ning, W.; Sun, D.; Gao, Q.; Lu, Y.; Zhu, D. Natural image restoration based on multi-scale group sparsity residual constraints. Front. Neurosci. 2023, 17, 1293161. [Google Scholar] [CrossRef] [Scilit]
  54. Luo, Z.; Gustafsson, F.K.; Zhao, Z.; Sjölund, J.; Schön, T.B. Image Restoration with Mean-Reverting Stochastic Differential Equations. In Proceedings of the 40th International Conference on Machine Learning, Honolulu, HI, USA, 23–29 July 2023; Volume 202, pp. 23045–23066. Available online: https://proceedings.mlr.press/v202/luo23b.html (accessed on 3 October 2026).
  55. Ren, B.; Li, Y.; Liang, J.; Ranjan, R.; Liu, M.; Cucchiara, R.; Van Gool, L.; Yang, M.-H.; Sebe, N. Sharing Key Semantics in Transformer Makes Efficient Image Restoration. In Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 10–15 December 2024; Volume 37, pp. 7427–7463. [Google Scholar] [CrossRef] [Scilit]
  56. Zhang, S.; Wang, S.; Zhang, Q.; Zhao, L.; Huo, W.; Hou, S.; Fu, C. CSF-GAN: Cross-modal Semantic Fusion-based Generative Adversarial Network for Text-guided Image Inpainting. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, Montreal, QC, Canada, 16–22 August 2025; pp. 2377–2385. [Google Scholar] [CrossRef] [Scilit]
  57. Chen, X.; Wang, X.; Zhou, J.; Qiao, Y.; Dong, C. Activating More Pixels in Image Super-Resolution Transformer. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 18–22 June 2023; pp. 22367–22377. [Google Scholar] [CrossRef] [Scilit]
  58. Tong, Z.; Song, Y.; Wang, J.; Wang, L. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. In Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA, 28 November–9 December 2022; Volume 35, pp. 10078–10093. [Google Scholar] [CrossRef] [Scilit]
  59. Bao, H.; Dong, L.; Piao, S.; Wei, F. BEiT: BERT Pre-Training of Image Transformers. In Proceedings of the 10th International Conference on Learning Representations (ICLR 2022), Virtual, 25–29 April 2022; Available online: https://iclr.cc/virtual/2022/oral/6324 (accessed on 3 October 2026).
  60. Chen, X.; Ding, M.; Wang, X.; Xin, Y.; Mo, S.; Wang, Y.; Han, S.; Luo, P.; Zeng, G.; Wang, J. Context Autoencoder for Self-Supervised Representation Learning. Int. J. Comput. Vis. 2024, 132, 208–223. [Google Scholar] [CrossRef] [Scilit]
  61. Liu, H.; Shi, Y.; Xu, H.; Yuan, C.; Ye, Q.; Li, C.; Yan, M.; Zhang, J.; Huang, F.; Li, B.; et al. Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italy, 20–25 May 2024; pp. 14664–14675. [Google Scholar] [CrossRef] [Scilit]
  62. Benhammou, Y.; Kalyan, S.; Kumar, S. Reconstruction-Driven Multimodal Representation Learning for Automated Media Understanding. arXiv 2025, arXiv:2511.17596. Available online: https://arxiv.org/abs/2511.17596 (accessed on 6 September 2026).
  63. Zhang, X.; Yang, H.; Wang, T.; Li, X.; Yin, L. Multimodal Channel-Mixing: Channel and Spatial Masked AutoEncoder on Facial Action Unit Detection. In Proceedings of the 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 4–8 January 2024; pp. 6065–6074. [Google Scholar] [CrossRef] [Scilit]
  64. Radford, A.; Kim, J.W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. Learning Transferable Visual Models from Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, Virtual, 18–24 July 2021; Volume 139, pp. 8748–8763. Available online: https://proceedings.mlr.press/v139/radford21a.html (accessed on 3 October 2026).
  65. Luo, Z.; Gustafsson, F.K.; Zhao, Z.; Sjölund, J.; Schön, T.B. Controlling Vision-Language Models for Multi-Task Image Restoration. In Proceedings of the 12th International Conference on Learning Representations (ICLR 2024), Vienna, Austria, 7–11 May 2024; Available online: https://openreview.net/pdf?id=t3vnnLeajU (accessed on 3 October 2026).
  66. Chen, H.; Li, W.; Gu, J.; Ren, J.; Chen, S.; Ye, T.; Pei, R.; Zhou, K.; Song, F.; Zhu, L. RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models. In Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 10–15 December 2024; Volume 37, pp. 110643–110666. [Google Scholar] [CrossRef] [Scilit]
  67. Zhou, Y.; Long, G. Improving Cross-modal Alignment for Text-Guided Image Inpainting. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia, 2–6 May 2023; pp. 3445–3456. [Google Scholar] [CrossRef] [Scilit]
  68. Wu, S.; Jiang, J.; He, X.; Xu, Y.; Zhu, Y.; Jiang, W.; Li, H. SEER: Knowledge-driven semantic image restoration with vision-language diffusion alignment. Knowl.-Based Syst. 2025, 329, 114464. [Google Scholar] [CrossRef] [Scilit]
  69. Wu, T.; Li, M.; Chen, J.; Ji, W.; Lin, W.; Gao, J.; Kuang, K.; Zhao, Z.; Wu, F. Semantic Alignment for Multimodal Large Language Models. In Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne, Australia, 28 October–1 November 2024; pp. 3489–3498. [Google Scholar] [CrossRef] [Scilit]
  70. Zhang, C.; Yang, W.; Li, X.; Han, H. MMGInpainting: Multi-Modality Guided Image Inpainting Based on Diffusion Models. IEEE Trans. Multimed. 2024, 26, 8811–8823. [Google Scholar] [CrossRef] [Scilit]
  71. Dai, R.; Li, C.; Yan, Y.; Mo, L.; Qin, K.; He, T. Unbiased Missing-modality Multimodal Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Honolulu, HI, USA, 19–23 October 2025; pp. 24507–24517. Available online: https://openaccess.thecvf.com/content/ICCV2025/html/Dai_Unbiased_Missing-modality_Multimodal_Learning_ICCV_2025_paper.html (accessed on 6 September 2026).
  72. Girdhar, R.; El-Nouby, A.; Liu, Z.; Singh, M.; Alwala, K.V.; Joulin, A.; Misra, I. ImageBind: One Embedding Space to Bind Them All. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 18–22 June 2023; pp. 15180–15190. Available online: https://openaccess.thecvf.com/content/CVPR2023/html/Girdhar_ImageBind_One_Embedding_Space_To_Bind_Them_All_CVPR_2023_paper.html (accessed on 6 September 2026).
  73. Singh, A.; Hu, R.; Goswami, V.; Couairon, G.; Galuba, W.; Rohrbach, M.; Kiela, D. FLAVA: A Foundational Language and Vision Alignment Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 19–24 June 2022; pp. 15638–15650. Available online: https://openaccess.thecvf.com/content/CVPR2022/html/Singh_FLAVA_A_Foundational_Language_and_Vision_Alignment_Model_CVPR_2022_paper.html (accessed on 6 September 2026).
  74. Jin, X.; Shi, Y.; Xia, B.; Yang, W. LLMRA: Multi-modal Large Language Model based Restoration Assistant. arXiv 2024, arXiv:2401.11401. Available online: https://arxiv.org/abs/2401.11401 (accessed on 6 September 2026).
  75. Liu, X.; Li, X.; Serrà, J.; Pascual, S. Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility. In Proceedings of the ICASSP 2025—2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Hyderabad, India, 6–11 April 2025; pp. 1–5. [Google Scholar] [CrossRef] [Scilit]
  76. Dror, T.; Shoham, I.; Buchris, M.; Gal, O.; Permuter, H.; Katz, G.; Nachmani, E. Token-Based Audio Inpainting via Discrete Diffusion. In Proceedings of the 14th International Conference on Learning Representations (ICLR 2026), Rio de Janeiro, Brazil, 23–27 April 2026; Available online: https://proceedings.iclr.cc/paper_files/paper/2026/hash/20888d00c5df685de2c09790040e0327-Abstract-Conference.html (accessed on 3 October 2026).
  77. Sun, H.; Li, W.; Liu, J.; Chen, H.; Pei, R.; Zou, X.; Yan, Y.; Yang, Y. CoSeR: Bridging Image and Language for Cognitive Super-Resolution. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 17–21 June 2024; pp. 25868–25878. [Google Scholar] [CrossRef] [Scilit]
  78. Chen, S.; Atapour-Abarghouei, A.; Shum, H.P.H. HINT: High-quality INPainting Transformer with Mask-Aware Encoding and Enhanced Attention. IEEE Trans. Multimed. 2024, 26, 7649–7660. [Google Scholar] [CrossRef] [Scilit]
  79. Wu, R.; Yang, T.; Sun, L.; Zhang, Z.; Li, S.; Zhang, L. SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 17–21 June 2024; pp. 25456–25467. [Google Scholar] [CrossRef] [Scilit]
  80. Song, J.; Yang, L.; Feng, M. Extremely low-bitrate Image Compression Semantically Disentangled by LMMs from a Human Perception Perspective. arXiv 2025, arXiv:2503.00399. Available online: https://arxiv.org/abs/2503.00399 (accessed on 6 September 2026).
  81. Li, X.; Guo, Q.; Lin, D.; Li, P.; Feng, W.; Wang, S. MISF: Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 19–24 June 2022; pp. 1859–1868. [Google Scholar] [CrossRef] [Scilit]
  82. Fu, W.; Xu, L.; Wu, X.; Wei, H.; Wang, L. Multimodal generative semantic communication based on latent diffusion model. In Proceedings of the 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP), London, UK, 22–25 September 2024; pp. 1–6. [Google Scholar] [CrossRef] [Scilit]
  83. Sun, M.; Wang, W.; Qiao, Y.; Sun, J.; Qin, Z.; Guo, L.; Zhu, X.; Liu, J. MM-LDM: Multi-Modal Latent Diffusion Model for Sounding Video Generation. In Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne, Australia, 28 October–1 November 2024; pp. 10853–10861. [Google Scholar] [CrossRef] [Scilit]
  84. Im, J.; Nam, J. SAGA-SR: Semantically and Acoustically Guided Audio Super-Resolution. In Proceedings of the ICASSP 2026—2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 4–8 May 2026; pp. 1706–1710. [Google Scholar] [CrossRef] [Scilit]
  85. Giakoumoglou, P.; Karageorgiou, D.; Papadopoulos, S.; Petrantonakis, P.C. SAGI: Semantically Aligned and Uncertainty Guided AI Image Inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Honolulu, HI, USA, 19–23 October 2025; pp. 16090–16101. Available online: https://openaccess.thecvf.com/content/ICCV2025/html/Giakoumoglou_SAGI_Semantically_Aligned_and_Uncertainty_Guided_AI_Image_Inpainting_ICCV_2025_paper.html (accessed on 6 September 2026).
  86. Yao, J.; Liu, H.; Chen, C.; Hu, Y.; Chng, E.; Xie, L. GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling. In Proceedings of the 13th International Conference on Learning Representations (ICLR 2025), Singapore, 24–28 April 2025; Available online: https://openreview.net/pdf?id=1p6xFLBU4J (accessed on 3 October 2026).
  87. Xia, B.; Wang, S.; Tao, Y.; Wang, Y.; Jia, J. LLMGA: Multimodal Large Language Model based Generation Assistant. In Proceedings of the Computer Vision—ECCV 2024, Milan, Italy, 29 September–4 October 2024; Springer: Cham, Switzerland, 2025; Volume 15096, pp. 389–406. [Google Scholar] [CrossRef] [Scilit]
  88. Yin, H.; Qiao, L.; Ma, Y.; Sun, S.; Li, K.; Gao, Z.; Niyato, D. Generative Video Semantic Communication via Multimodal Semantic Fusion with Large Model. IEEE Trans. Veh. Technol. 2026, 75, 1701–1706. [Google Scholar] [CrossRef] [Scilit]
  89. Chen, W.; Yang, Q. CASC: Condition-Aware Semantic Communication with Latent Diffusion Models. In Proceedings of the 2025 IEEE International Conference on Communications Workshops (ICC Workshops), Montreal, QC, Canada, 8–12 June 2025; pp. 1723–1728. [Google Scholar] [CrossRef] [Scilit]
  90. Qiu, X.; Cheng, J.; Fang, Y.; Zhang, W.; Lu, Y.; Zhang, Y.; Li, C. Federated Dialogue-Semantic Diffusion for Emotion Recognition under Incomplete Modalities. In Proceedings of the Advances in Neural Information Processing Systems, San Diego, CA, USA, 2–7 December 2025, and Mexico City, Mexico, 30 November–5 December 2025; Neural Information Processing Systems Foundation, Inc.: San Diego, CA, USA, 2025; Volume 38, pp. 95833–95862. [Google Scholar] [CrossRef] [Scilit]
  91. Scotti, P.S.; Tripathy, M.; Torrico, C.; Kneeland, R.; Chen, T.; Narang, A.; Santhirasegaran, C.; Xu, J.; Naselaris, T.; Norman, K.A.; et al. MindEye2: Shared-Subject Models Enable fMRI-To-Image with 1 Hour of Data. In Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria, 21–27 July 2024; Volume 235, pp. 44038–44059. Available online: https://proceedings.mlr.press/v235/scotti24a.html (accessed on 3 October 2026).
  92. Scotti, P.S.; Banerjee, A.; Goode, J.; Shabalin, S.; Nguyen, A.; Cohen, E.; Dempster, A.J.; Verlinde, N.; Yundler, E.; Weisberg, D.; et al. Reconstructing the Mind’s Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors. In Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA, 10–16 December 2023; Volume 36, pp. 24705–24728. [Google Scholar] [CrossRef] [Scilit]
  93. Ozcelik, F.; VanRullen, R. Natural scene reconstruction from fMRI signals using generative latent diffusion. Sci. Rep. 2023, 13, 15666. [Google Scholar] [CrossRef] [Scilit]
  94. Zhang, X.; Quan, R.; Wang, W.; Yang, Y. Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction. In Proceedings of the International Conference on Learning Representations (ICLR), Rio de Janeiro, Brazil, 23–27 April 2026; Available online: https://proceedings.iclr.cc/paper_files/paper/2026/hash/6fe6a2ba2594521d15af3b1f2162d79c-Abstract-Conference.html (accessed on 4 November 2026).
  95. Lu, Y.; Du, C.; Zhou, Q.; Wang, D.; He, H. MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion. In Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada, 29 October–3 November 2023; pp. 5899–5908. [Google Scholar] [CrossRef] [Scilit]
  96. Kalantari, F.; Faez, K.; Amindavar, H.; Nazari, S. Improved image reconstruction from brain activity through automatic image captioning. Sci. Rep. 2025, 15, 4907. [Google Scholar] [CrossRef] [Scilit]
  97. Scetbon, M.; Elad, M.; Milanfar, P. Deep K-SVD Denoising. IEEE Trans. Image Process. 2021, 30, 5944–5955. [Google Scholar] [CrossRef] [Scilit]
  98. Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; Hochreiter, S. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; Volume 30. Available online: https://proceedings.neurips.cc/paper/2017/hash/8a1d694707eb0fefe65871369074926d-Abstract.html (accessed on 3 October 2026).
  99. Hessel, J.; Holtzman, A.; Forbes, M.; Le Bras, R.; Choi, Y. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic, 7–11 November 2021; pp. 7514–7528. [Google Scholar] [CrossRef] [Scilit]
  100. Tishby, N.; Pereira, F.C.; Bialek, W. The information bottleneck method. In Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing, Monticello, IL, USA, 22–24 September 1999; pp. 368–377. Available online: https://www.princeton.edu/~wbialek/our_papers/publications_wbialek.html (accessed on 3 October 2026).
  101. Yang, C. A coarse-to-fine restoration framework integrating deep semantic inpainting and texture-aware refinement for Jinling school landscapes. Sci. Rep. 2026, 16, 19267. [Google Scholar] [CrossRef] [Scilit]
  102. Hwang, T.; Heo, I.; Lee, S.J.; Jung, S. Perceptual Hallucination in Vision–Language Models: Definition, Analysis and Verification. In Proceedings of the Findings of the Association for Computational Linguistics: ACL 2026, San Diego, CA, USA, 2–7 July 2026; pp. 24710–24725. [Google Scholar] [CrossRef] [Scilit]
  103. Shannon, C.E. A Mathematical Theory of Communication. Bell Syst. Tech. J. 1948, 27, 379–423. [Google Scholar] [CrossRef] [Scilit]
  104. Coalition for Content Provenance and Authenticity. Content Credentials: C2PA Technical Specification. Version 2.2. Available online: https://spec.c2pa.org/specifications/specifications/2.2/specs/C2PA_Specification (accessed on 13 September 2026).
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Article metric data becomes available approximately 24 hours after publication online.