QMRNet: Quality Metric Regression for EO Image Quality Assessment and Super-Resolution

Round 1

Reviewer 1 Report

This paper proposes to explore the EO image super-resolution and quality assessment. Experiments show the superiority of the proposed methods.

1. The introduction section is too long. It would be better to divide the introduction part into related work and the introduction itself for better organization.

2. Although the paper targets SR and its quality assessment, many SR specifically quality metrics are missing for references and better for analysis, including Quality assessment of image super-resolution: balancing deterministic and statistical fidelity (the extension of SFSN metric), Single image super-resolution quality assessment: a real-World dataset, subjective studies, and an objective metric, Blind quality assessment for image superresolution using deep two-stream convolutional networks, etc.

3. The ablation study can be tested in the experiments to further verify the effectiveness of each proposed component.

4. The reviewed IQA models are relatively old. To provide a better understanding for readers, more recently published DL-based IQA works are recommended to be added, such as Graph IQA, lifelong IQA, meta IQA, etc.

5. It is suggested to further improve the presentation of this paper. For example, the figures are not clear enough.

n/a

Author Response

First of all, we would like to thank the Editor and Reviewers for reviewing the submission in that fast pace while giving precise questions, issues and changes to the submission. (Please download the new attached manuscript to see the changes in the submission.)

Response 1:

We have unified the “1. INTRODUCTION” with its sub-sections, removed and re-wrote all pages. We have mainly erased comments and references with Algorithms and Metrics which are not posteriorly used nor mentioned in the rest of the article, a total of 12 references will be erased. 

In section “2. DATASETS AND RELATED WORK” we re-wrote and made new comments in relation to Responses 2, 3 and 4. 

Response 2:

SRIF, RealSRQ and DeepSRQ have been included and commented in “2. DATASETS AND RELATED WORK”. We want to pinpoint SRIF, RealSRQ and DeepSRQ have shown deep learning models as No Reference-Metrics for SR, but have been only tested in generic datasets CVIU, SRID, QADS and Waterloo. These models compare the metrics with Root-Mean-Square Error (RMSE), the Spearman’s rank order correlation coefficient (SRCC or SROCC), the Pearson’s linear correlation coefficient (PLCC) and the Kendall’s rank order correlation coefficient (KROCC). Although we did not initially include RMSE, KRCC/KROCC and PLCC, we considered medR and R@X metrics as better approximation for assessing N-rank order classification, as these are hard to assess, specially as each distortion parameter has distinct precision ranges. These have been mentioned in the “1. INTRODUCTION” and  “4. EXPERIMENTS-Evaluation metrics”. 

Response 3:

We have considered to design QMRNet architecture with multiparameter prediction (-MH for Multihead) and (-MB for Multibranch), see Figure 2 and Table 5. QMRNet-MH considers one encoder and #N classifiers per EO parameter while QMRNet-MB considers #N encoders + #N classifiers (a whole QMRNet per parameter). All comments have been done in “3. PROPOSED METHOD” and new results have been included in Tables 5, 11 and Section “4. EXPERIMENTS”. Also in qualitative evaluation of EO (Figures 5, 6, 7 and 10, 11, 12) we have included colormaps of the sum of differences (∆R+∆G+∆B) of each Algorithm with respect the Original HR.

Response 4:

We have considered RER, MTF y FWHM, SNR and GSR as specific EO evaluation metrics, including general blur and sharpness distortions. Traditional blind IQA methods (i.e. BLIINDS-II, BRISQUE, CORNIA, HOSA or RankIQA) as well as latest deep blind image quality assessment models such as WaDIQaM (deepIQA), NR-IQA, Meta-IQA, GraphIQA and LCSA propose to benchmark distortion-aware datasets (e.g. LIVE, LIVEC, CSIQ, KonIQ10k, TID2013 and KADID-10k) with already distorted images and MOS/CMOS. These train and assess upon annotated exemplars such as gaussian blur, lens blur, motion blur, color quantization, color saturation, etc.

(In relation to this and references mentioned in Response 2), 

Most of these models do not integrate their own modifiers that able customization of ranking metrics (i.e. are limited to the available synthetic annotations from the aforementioned datasets). Some of these could include geo-reference annotations from the actual EO missions, such as the GSD, Nadir angle, atmospheric data... The usage of customizable modifiers allows the finetuning on distortions on any existing domain, HR EO images for our case. It has also not been demonstrated for IQA methods to integrate with super-resolution model benchmarking and re-training. Understanding and building the mechanics of distortions (geo-metrics and modifiers) is thus key for the generation of the necessary samples to train a network with enough samples to represent the whole domain. These comments have been included in “2. DATASETS AND RELATED WORK”.

Metrics in IQUAFLOW currently include own implementations, as well as piq, Pillow, OpenCV and Pytorch-Torchvision transforms. We selected the mentioned in the submission to represent distinctly balanced metrics to observe effects and differences over datasets, resolutions and distortions. Any other FR- or NR- metric has not initially been included to keep the briefness of the article, focusing in EO-specific quality assessment metrics.

In “5. CONCLUSION” we included comments with regards to the use of QMRNet and IQUAFLOW in compression parameters and datasets included in “2. DATASETS AND RELATED WORK” in future.

Response 5: 

For this comment, we totally agree the acronyms as well as both Figures and Tables were not easing the understanding of the article. We have rewrote the whole article, including visuals and text formatting for better representing the “3. PROPOSED METHOD” (Figures 1-4) and “4. EXPERIMENTS”. Moreover, formatting of Tables 1-11 has been updated. Specifically in Tables 4-11 includes Bold text for Algorithms having lower distortion than HR, Underlined text represents top-1 best performance, and Italic represents same value for most cases. The effects present in these tables have been added in “4. EXPERIMENTS”. We have specified In qualitative evaluation (Figures 5, 6, 7 and 10, 11, 12) the colormaps of the sum of differences (∆R+∆G+∆B) of each Algorithm with respect the Original HR.
We have updated results in all tables, and the new results in Tables 5, 6 and 11. 

Author Response File: Author Response.pdf

Reviewer 2 Report

Most of quality assessment metrics are designed and developed for natural images instead of EO images. This research bridges the gap by building a network to predict EO image quanlity from different perspectives (e.g. gaussian blur,  sharpness, a rescaling to a distinct GSD,  noise) trained on the Inria-AILD data set. Here are some issues.

1. The figures in the manuscript are very blur. Please insert the high-quality figures. 

2. Besides the metrics listed in Fig.9, the authors must display the super-resolved images by optimizing QMRNet-loss. The readers will be interestd in whether the QMRNet-loss results in images with better quality.

3. This work is carried out on the Inria-AILD data set, where the reviewer found that the training samples are three-channel images, instead of multi-spectral or hyper-spectral images. Many deep learning based image quanlity assessment methods have been developed for natural images which are also three-channel images. Therefore, the authors should compare at least one existing methods to demonstrate the superiority of QMRNet for EO images. 

Author Response

First of all, we would like to thank the Editor and Reviewers for reviewing the submission in that fast pace while giving precise questions, issues and changes to the submission. (Please download the new attached manuscript to see the changes in the submission.)

Response 1:

We have replaced qualitative samples from Figures 5, 6, 7 with distinct crop sizes (96x96, 128x128 and 256x256) and datasets (including UCMerced-2100, UCMerced380, XView, Inria-AILD-val and Inria-AILD-test) for testing the adaptation strategies of each Algorithm. Distinct images of crops, forest, buildings, roads, intersections, cars, airplanes, boats, sea, beach and much other objects are included in Figures 5, 6, 7 and 10, 11, 12. Low resolution examples have been included in Figure 7.

Response 2:

We have run novel examples from datasets UCMerced-380, UCMerced-2100 and Inria-AILD-test in Figures 10, 11, 12. In all Tables, results and formatting has been updated for better visualization of metrics assessment. We have to pinpoint that in qualitative evaluation of EO (Figures 5, 6, 7 and 10, 11, 12) we have included image colormaps of the sum of differences (∆R+∆G+∆B) of each Algorithm with respect the Original HR. This can show where are differences localized in each Algorithm example with respect the HR, whether trying to improve original I_HR (with contour sharpening or deep-generated features) or trying to reach I_HR approximations given I_LR downsampled inputs. Distinct resolutions of 96x96 and 256x256 show changes in different resolutions given its GSD.

Response 3:

We have included ECODSE dataset, which has been considered for EO hyperspectral image classification, delineation (segmentation) and alignment of trees. ECODSE has available NEON Photographs, LiDAR data for assessing canopy height and the hyperspectral images with 426 bands ranging from 383 to 2.512 nm with a spectral resolution of five nm. The terrain is photographed with a mean altitude of 45 m.a.s.l. and the mean canopy height is approximately 23 m. All comments, results, links and references have been included in Table 1 and Sections “2. DATASETS AND RELATED WORK” and “4. EXPERIMENTS”.

In Table 6 we show validation results for QMRNet’s prediction of snr and rer in hyperspectral images. By changing the first convolutional layer of QMRNet's Encoder backbone for input channels we can classify the quality metric with multiple channels. Overall medR is around 5.0 and recall rates around 20 % for R@1 (exact match), 56 % for R@5 (5 closest categories) and 80 % for R@10 (10 closest categories), given the (fine-grained) N=40 distortion categories. We want to pinpoint the lower precision due to the hardness of the approximation task, given the hyperspectral resolution (i.e. 80x80 at ~60cm/px) and the very few examples (43 examples). Although the hardness of the dataset task, QMRNet with ResNet18 is able measure whether a parameter is in a specific range of quality metric (e.g. snr, rer) in hyperspectral images. We have re-adapted QMRNet network and IQUAFLOW framework readout for our own snr and rer metrics, as Pillow and OpenCV not being able to read multichannel images.  Not every metric can be integrated, as most PyTorch and OpenCV transforms (and thus the modifiers) are not implemented for channels beyond RGB, as precisely work with these channels.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The authors have addressed my comments.

n/a

Reviewer 2 Report

No more comments. This is a high-quanlity paper. The reviewer recommends strong accept.