Human Perceptuality-Aware Tone-Mapping-Based Dynamic Voltage Scaling for an AMOLED Display Smartphone

Abstract

In recent years, people have wanted to watch high dynamic range imagery which can give high human visual satisfaction on smartphones and demand longer smartphone battery time. However, compression of dynamic range using tone-mapping operators is required in smartphones because most smartphone displays currently have a low dynamic range, and this causes loss of local contrast and details to compress dynamic range. Thus, in this paper we propose a novel dynamic voltage scaling scheme tightly coupled with a modified tone-mapping operator to achieve high power saving as well as good human perceptuality on an AMOLED display smartphone. In order to perform a human perceptuality-aware voltage control, we control display panel voltage to save power consumption and use a well-adjusted global tone-mapping operator to convert image brightness and unsharp masking to enhance local contrast and details and control. We implement the proposed scheme on the AMOLED display Android smartphone and experiment with various high dynamic range image databases. Experimental results show that not only tone-mapped images but also general images are improved in terms of human visual satisfaction and power saving, compared to conventional techniques.

1. Introduction

Recently, active matrix organic light emitting diode (AMOLED) displays have evolved into ultrahigh definition (UHD) displays [1,2]. In addition, a market research by IHS Markit predicts that the market for high dynamic range (HDR) imagery, which is one of the most important elements in UHD (or 4K) technology, will expand sharply and have a market share of over 60% by 2020 [3]. In order to better support HDR, ARM, a global chip IP company, recently launched Assertive Display (AD)-5, the next generation display system chip, which is involved in the HDR display pipeline and improved color and gamut control [4].

The HDR image should be displayed on a display that satisfies the HDR standard (HDR display), which expresses the HDR well. However, since many displays now have a low dynamic range (LDR) display of eight bits per channel (also called standard dynamic range (SDR)), preprocessing is required to show the HDR image on the LDR display. This preprocessing is called tone mapping and it is possible to display various image qualities according to the design of the tone-mapping operator (TMO).

Meanwhile, in recent years, in order to run SNS, games, videos and so on, smartphones have been designed for a close and continuous interaction with users. AMOLED displays are one of the technologies with the largest power consumption among smartphone devices, so power saving in smartphones is important. As shown in [5], AMOLED display power consumption of an AMOLED display smartphone is over 20% of noncall usage. Therefore, optimizing the power consumption of AMOLED displays is crucial to extend the battery life of smartphones and to increase the convenience and satisfaction of users. In addition, a recent report from the National Resources Defense Council (NRDC) shows that using HDR on the same UHD images consumes more than 47% more power [6]. Therefore, research on evaluating and processing image quality in terms of low power is much required when displaying HDR images on AMOLED displays.

In this paper, we propose a technique to obtain optimal human visual satisfaction and power saving by performing TMO control considering HDR image and LDR AMOLED display of smartphone and performing dynamic voltage scaling (DVS) in conjunction with it. In addition to TMO control, unsharp masking is tuned to fit the smartphone to further improve local contrast and details. The proposed technique is implemented by controlling the framebuffer and backlight device by implementing it at hardware abstract layer (HAL) level in Android smartphones equipped with AMOLED display. At this time, human visual satisfaction is reflected by making a lookup table (LUT) by performing quality evaluation by actual human vision. The contributions of this paper are summarized as follows.

• Compared to prior work, a DVS technique which targets tone-mapped images on the smartphone is proposed for the first time.
• In order to consider power saving as well as image quality, Reinhard TMO is modified and applied to DVS together with unsharp masking.
• Runtime image quality control is performed at the HAL level of the Android platform based on the results from the user study.

2. Background and Related Work

2.1. HDR Images on LDR Displays

Among smartphones, Samsung Galaxy S3 is measured to have a maximum of 330 nits and Galaxy S7 has a maximum of 493 nits, which does not yet provide HDR [7,8]. Among smartphones, Sony Xperia XZ2 provides a HDR video mode for LCDs [9]. Since most of the AMOLED display smartphones, which are the target of this research, are equipped with LDR display devices, it is important to study HDR images on LDR displays.

TMO is an operation that performs range reduction when the range of the HDR image is larger than the dynamic range that the display can represent [10]. A lot of studies have been performed on them and they are divided into global and local TMOs [11]. Global TMO has the advantage of efficient computation time by applying the same operation to all pixels, and local TMO performs calculations based on neighboring pixel information and provides more contrast and high level of detail. Among them, Reinhard TMO [12] is basically a well-designed one [11]. Thus, in this paper, we focus on Reinhard TMO. It can be selected as global or local. Reinhard global TMO tends to saturate the bright areas too much and the local contrast of the local regions becomes larger due to local dodging and burning methods of Reinhard local one [13].

2.2. Related Work

Low-power techniques on OLED displays. The most effective low-power technique for OLED displays is DVS. DVS is a technique that performs pixel compensation to keep the luminance at the same level while varying the supply voltage applied to the AMOLED display panel. Shin et al. researched on a pioneering DVS work for OLED displays, but the target was a passive matrix OLED display [14]. Park et al. combined human visual perceptuality-aware color transformation and DVS [15]. Lee et al. proposed an entropy-based scene change detection method and combined it with DVS [16]. Hong et al. proposed a practical local gamma correction-based DVS scheme for Android smartphones [17]. These all target AMOLED displays, but only Hong et al. employed an Android smartphone. In addition, they all do not consider tone-mapped images which comes from HDR images through TMOs unlike our work (Among these works, we compared Hong’s results with ours in Section 5).

There are some remarkable low-power OLED techniques. He et al. adopted dynamic resolution scaling for power saving in AMOLED smartphones [18]. RAVEN [19] employed a frame skipping technique for both low power and human visual quality-awareness running mobile games on AMOLED smartphones. Kim et al. adjusted the refresh rate of a framebuffer according to the frame per second (FPS) for apps on AMOLED smartphones for power saving [20]. Recently, Yan et al. proposed a subpixel shutoff technique for power saving while maintaining the display quality on AMOLED smartphones [21]. These works did not fulfill a real human visual evaluation while our work did; our work focuses on combining DVS and TMOs.

Another important low-power OLED technique is color transformation. Jin et al. searched the optimal HVS-aware and low-power direction in the CIE Lab color space for general images and developed an assessment metric by combining hue, saturation, and power ratio for each direction on AMOLED display smartphones [22]. PARVAI is a grid-based low-power color transformation technique, which employs hue and saturation constraints and splits a screen into multiple grids [23]. It exploited Structural SIMilarity (SSIM) [24] and global contrast loss (GCL) to assess HVS-awareness. PARVAI is likely to cause color distortion and blocking artifacts because it minimizes only the blue channel and grid-based color transformation, respectively. Lin et al. divided a still image into several regions using a saliency model and applied different scaling to each region [25]. They employ SSIM as an image quality assessment metric. These works all did not treat the image quality control of tone-mapped images and TMOs on LDR displays. In contrast, we focus on DVS.

Tone-mapping techniques on OLED displays. Previously, Iranli et al. proposed a tone-mapping method to preserve the perceived brightness for backlight scaling in LCDs [26]. This work targeted LCDs. Chen et al. developed a low-power tone-mapping technique for OLED displays based on video classification using a power feature-based Hidden Markov Model (HMM) classifier in video stream [27]. They applied different dynamic tone-mapping schemes to each classified video category and utilized SSIM for determining the image quality. They focused on three tone-mapping techniques of color range mapping shown in [28], saturation tuning, and hue tuning. This work does not consider power-aware control of TMOs for DVS. In addition, we employed real human visual evaluations for image quality assessment through user study instead of SSIM. T. Shiga et al. [29] used the Helmholtz-Kohlraush effect, where humans perceive a high saturation image as a brighter image even if their luminance are same, to save the power consumption of OLED display while minimizing the deterioration of human visual satisfaction. They saved power by lowering the luminance of pixels and compensated for bright, which is decreased due to the luminance decrease, by increasing saturation. However, their work is not about DVS, so the display power saving ratio is only 8% when the color change is not significant. A. Bhojan [30] used Energy Efficient Color Mapping (EECM) to convert a color of each video frame to a color that consumes less power, and then used gamma correction to lower the luminance, which is closely related to power saving, of each pixel. However, their research also is not a DVS technique that controls the supply voltage of the display, so the power saving effect is not significant. As a result, there is no research combined with the DVS technique that controls the supply voltage of the display among the techniques that apply tone mapping to the OLED display. Therefore, there is a need to associate a DVS, which has more power saving effect, with a tone-mapping technique.

3. Motivational Studies

We find that an LDR image, which is obtained after a TMO is applied to a HDR image, has lowered dynamic ranges and degraded details together with human visual appearance loss (we call this image an HDR-like LDR image). Now we test HDR-like LDR images for the combinations of supply voltage control and TMOs on an AMOLED display smartphone to find the relation between image quality and power consumption. Since our work targets an LDR display and HDR-like LDR images still have much room for contrast and detail enhancement, we need to investigate how to improve their contrast and local details while reducing the power consumption of an LDR display. To this end, we utilized well-designed Reinhard TMO and controlled the brightness of the panel. We used a Galaxy S3 smartphone [31] with an AMOLED display panel of a 1280 × 720 resolution and its Android platform version is 6.0 Marshmallow. The used tone-mapped image is arbitrarily selected from among the best MOS images Yeganeh’s TMO database [32] and we treat the best MOS images as the HDR-like LDR image.

Figure 1a–d are taken for a Galaxy S3 screen using a SONY DSC-RX100 III camera with ISO200, WB5500K, shutter speed 1/30 and F-stop 3.5 under normal light. Figure 1e,i contain locally cropped enlargements of Figure 1a. Figure 1a is a HDR-like image resized by a resolution of 1280 × 720 with a brightness value of 200. In this image, the shadow is so dark that the tree and grass details in shadow are not easily discernible. In addition, the snow on the mountain is obscure. Figure 1b is the result obtained when modified Reinhard global TMO is applied to Figure 1a and the brightness is set to 100. In this case, 27.3% power saving is achieved, but when we compare Figure 1f,j with Figure 1e,i, respectively, we can see that image quality is degraded slightly.

To enhance the local contrast and details, we applied the Reinhard local TMO and Reinhard global TMO + unsharp masking [33,34]. As we can see in Figure 1g,k, which are locally cropped enlargements of Figure 1c, Reinhard local TMO improves the details of a bush, a tree, and snow to give better human visual satisfaction over Figure 1f,j as well as a big power saving of 25.9%. However, we observe that more local detail enhancement occurs in Figure 1h,l which corresponds to the case of Reinhard global TMO + unsharp masking. Figure 1h has much clearer view than Figure 1e–g. Figure 1l also has clearer than Figure 1i–k. The power saving in Figure 1d also reaches 25.1%, which is comparable to the power saving amount in Figure 1c. In addition, Reinhard local TMO requires high overhead operations such as Fast Fourier Transform (FFT) as described in [12]. This can result in high system power including high CPU power and large execution time which may be inappropriate to mobile devices like smartphones.

Based on these observations, we believe that it is necessary to find the optimum combination of TMO brightness control parameter, local detail enhancement parameter, and AMOLED display panel voltage, which results in the optimal balance of power saving and human visual satisfaction for the converted image. In addition, making the execution times of TMO and DVS appropriate for run-time applications in constrained smartphone environments needs to be considered.

4. Design of Tone-Mapping-Based DVS

4.1. Unsharp Masking for Smartphones

Unsharp masking is a technique to generate a detail-enhanced image with enhanced edges by adding the resulting image of a high pass filter to the original image [33,34]. The process of unsharp masking is shown as (1). In (1), $y\left(m, n\right)$, $x\left(m, n\right)$ and $z\left(m, n\right)$ respectively represent enhanced image, original image and high pass filtered image. Thus, we can create a detail-enhanced image using (1) and use the λ to control the detail enhancement.

$$y\left(m,n\right)=x\left(m,n\right)+\lambda z\left(m,n\right)$$

(1)

A commonly used high pass filter for high pass filtered image $z\left(m,n\right)$ in (1) is the Laplacian of Gaussian (LoG) [35,36,37]. When the LoG kernel size is M × M, (M = 2h + 1), it is common to set h ≥ 4σ for the edge-detection application. However, if the application is local contrast enhancement, a smaller h is also possible [38]. Ref. [38] introduces parameter k dependent on σ and kernel radius h. If the target application is a contrast enhancement, there is no problem to reduce h to satisfy k < 1.1, and it is advantageous in view of CPU time [38]. In our study, considering the computing power of the smartphone, it is difficult to set σ more than 2, and taking σ < 0.5 has little effect on smoothing [35], a combination of h = 2σ, kernel size = 5 × 5, σ = 1 has been adopted to balance the aforementioned considerations.

Figure 2a shows x (m, n) in (1) as the result image of the modified Reinhard global TMO, which will be described in Section 4.2 and Figure 2b shows y (m, n) in (1) as the result image of the unsharp masking. The image in Figure 2b has more detail than Figure 2a to compensate for the detail loss of the global TMO.

Table 1. Comparison of execution times of Reinhard TMOs without and with UM (unsharp masking) (unit: s).

Image Database	Reinhard Local TMO	Reinhard Global TMO	Reinhard Global TMO + UM
15 images [30]	14.36	0.83	1.40
10 images [35]	9.08	0.54	0.97
Avg.	0.94	0.05	0.09

compares the execution times of Reinhard local and global TMO and Reinhard global TMO + unsharp making using MATLAB 2014b in a desktop computer with Intel i7-3770 CPU, 16GB DRAM, and Windows 10. The image database uses HDR-like images representing the best MOS for the 15 tone-mapped images of [32] and 10 tone-mapped images of [39] as reference images. Since these images have different resolutions, we resize the images to a resolution of 1024 × 1024. Reinhard local TMO had a mean execution time of 0.94 s, whereas Reinhard global TMO had a time of 0.05 s, and local TMO required 18 times more time than global TMO. This shows that Reinhard local TMO requires much more execution time than Reinhard global TMO, although the ratio of execution time difference is less than that of [12] due to the difference of experimental environment.

Also, Reinhard local TMO enhances local detail and enhances image quality. However, its improving image quality is comparable to that from well-modified global TMO and it has no significant benefit in terms of power saving. Therefore, in this study, we aim to obtain image quality and power saving by using a combination of global TMO and unsharp masking to meet smartphone’s computing power and memory limit instead of detail and contrast enhancement by local TMO.

4.2. Modified Tone-Mapping Operator

Since we used the Reinhard global TMO as a pixel compensation method for DVS, we modified several parameters to control the brightness and contrast of a target image. In global Reinhard TMO, after a linear scaling in (2), by ${\mathit{L}}_{\mathit{w}\mathit{h}\mathit{i}\mathit{t}\mathit{e}}$ the luminance above a certain value is mapped to the output luminance by passing through the luminance mapping function which is clamped to the max luminance that the display can express as shown in (3).

$$\mathit{L}\left(\mathit{x},\mathit{y}\right) = \frac{\mathit{\alpha }}{{\overline{\mathit{L}}}_{\mathit{i}\mathit{n}\mathit{p}\mathit{u}\mathit{t}}}{\mathit{L}}_{\mathit{i}\mathit{n}\mathit{p}\mathit{u}\mathit{t}}\left(\mathit{x},\mathit{y}\right)$$

(2)

$${\mathit{L}}_{\mathit{d}} \left(\mathit{x},\mathit{y}\right) = \frac{\mathit{L}\left(\mathit{x},\mathit{y}\right)\left(\mathit{1}+\frac{\mathit{L}\left(\mathit{x},\mathit{y}\right)}{{\mathit{L}}_{\mathit{w}\mathit{h}\mathit{i}\mathit{t}\mathit{e}}^{\mathbf{2}}}\right)}{\mathbf{1}+\mathit{L}\left(\mathit{x},\mathit{y}\right)}$$

(3)

In (2), ${\mathit{L}}_{\mathit{i}\mathit{n}\mathit{p}\mathit{u}\mathit{t}}\left(\mathit{x},\mathit{y}\right)$ means an input luminance and ${\overline{\mathit{L}}}_{\mathit{i}\mathit{n}\mathit{p}\mathit{u}\mathit{t}}$ represents the log average of an input image. In (3), ${\mathit{L}}_{\mathit{d}}\left(\mathit{x},\mathit{y}\right)$ means an output luminance. In this study, the user-specified parameter α used in Reinhard TMO is set as (4).

$$\mathit{\alpha }={\overline{\mathit{L}}}_{\mathit{i}\mathit{n}\mathit{p}\mathit{u}\mathit{t}}\times {\mathit{L}}_{\mathit{w}\mathit{h}\mathit{i}\mathit{t}\mathit{e}}$$

(4)

By doing so, we can use the ${\mathit{L}}_{\mathit{w}\mathit{h}\mathit{i}\mathit{t}\mathit{e}}$ value as a parameter to control the brightness. A high ${\mathit{L}}_{\mathit{w}\mathit{h}\mathit{i}\mathit{t}\mathit{e}}$ value means that the image is much brighter after conversion, and a lower ${\mathit{L}}_{\mathit{w}\mathit{h}\mathit{i}\mathit{t}\mathit{e}}$ value means that the image is slightly brighter after conversion.

4.3. Measuring Human Perceptuality

We employed a user study instead of objective image assessment metrics to evaluate the image quality correctly, focusing on the enhancement of local contrast and details. The user study was conducted for 27 subjects consisting of 20 undergraduates and 7 graduates with normal or corrected visions in the university campus. The test methodology was adopted from that proposed in [17]. In a darkroom, a subject evaluated two images displayed on individual Galaxy S3 smartphones concurrently by scoring the image similarity against the reference image on a centered smartphone. Twenty images with different average brightness are selected from [32,39,40] and adjusted to have different TMO parameters and different brightness (i.e., supply voltage) values for evaluations.

We used Reinhard global TMO as our main pixel compensation method to improve the image quality degraded due to DVS. For the Reinhard global TMO, ${\mathit{L}}_{\mathit{w}\mathit{h}\mathit{i}\mathit{t}\mathit{e}}$ was adjusted to 2, 3, and 4. To enhance the local contrast and details, unsharp masking described in Section 4.1 was also used by changing λ to 4 and 7 in (1). The brightness was changed to 50, 100, 150, and 200. We found that the highest possible brightness on Galaxy S3 indoors was 198 and thus we limited the largest brightness value to 200.

Considering the battery capacity of the smartphone, we divided it into low-power mode and Optimal HVS mode. When the battery capacity is sufficient (≥20), it operates in Optimal HVS mode and choose one of the two highest MOS score with the highest power savings. When battery capacity is low (≤20), it operates in low-power mode and chooses one of the two highest power saving ratios with the highest MOS score.

4.4. Tone-Mapping-Based DVS Scheme

The overall flow of the proposed tone-mapping-aware DVS scheme is shown in Figure 3. First, when an input image comes, the view surface of the Android framework is executed for screen composition. Through the Surfaceflinger and HWComposer, the processed image is transferred to the Framebuffer HAL in the form of a buffer containing pixel values, and the R, G and B pixel values can be read by accessing this buffer. The proposed method computes the luma value using the read RGB values and reads the battery information from the battery driver to access the LUT of the power mode. Once the optimal parameters (${\mathit{L}}_{\mathit{w}\mathit{h}\mathit{i}\mathit{t}\mathit{e}}$, λ) and brightness values are determined in the LUT, pixel conversion and display backlight value scaling are performed. The proposed pixel compensation technique, combination of Reinhard TMO and unsharp masking is applied to the image and it is transferred to the framebuffer driver. Then, the resultant image is shown on the AMOLED display.

Display backlight scaling uses the background process of the Android operating system using a native app. The variable brightness value in the HAL is passed to the framebuffer driver and the background process reads the information of the framebuffer driver to recognize the variable brightness value to synchronize the brightness values. Then, backlight scaling is performed by controlling the supply voltage of the AMOLED display by transmitting variable brightness information to the light driver.

5. Experiments and Results

5.1. Experimental Setup

Our experimental setup is shown in Figure 4. HDR-like images used in the experiments were selected as images not used in the user study among images existing in [32,39] and also included the normal LDR images. The proposed method is compared with the normal mode of the smartphone and LGC-DVS [17]. The target device was a Galaxy S3 equipped with a 4.8-inch HD super AMOLED display panel. We implemented our proposed method with LGC-DVS at Android HAL level. We also use the Monsoon power monitor [41] as a tool for the power measurement. The experimental images were selected from 20 different luma values selected for the TMO databases [32,39,40] and LIVE database [42], and the techniques were applied to these images and power measurements were performed. We put the smartphone to airplane mode for accurate measurement. A camera was used to shoot the AMOLED display screen displaying the results of the proposed method and other methods.

The power consumed in the smartphone display for each image is calculated as shown in (5).

$${\mathit{P}}_{\mathit{d}\mathit{i}\mathit{s}\mathit{p}\mathit{l}\mathit{a}\mathit{y}}^{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}} \left(\mathbf{m}\mathbf{W}\right)={\mathit{P}}_{\mathit{s}\mathit{m}\mathit{a}\mathit{r}\mathit{t}\mathit{p}\mathit{h}\mathit{o}\mathit{n}\mathit{e}}^{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}}-{\mathit{P}}_{\mathit{s}\mathit{m}\mathit{a}\mathit{r}\mathit{t}\mathit{p}\mathit{h}\mathit{o}\mathit{n}\mathit{e}}^{\mathit{R},\mathit{G},\mathit{B}=\mathbf{0}}$$

(5)

In (5), ${\mathit{P}}_{\mathit{s}\mathit{m}\mathit{a}\mathit{r}\mathit{t}\mathit{p}\mathit{h}\mathit{o}\mathit{n}\mathit{e}}^{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}}$ represents the power consumed by all components of smartphone when an image is displayed, and ${\mathit{P}}_{\mathit{s}\mathit{m}\mathit{a}\mathit{r}\mathit{t}\mathit{p}\mathit{h}\mathit{o}\mathit{n}\mathit{e}}^{\mathit{R},\mathit{G},\mathit{B}=\mathbf{0}}$ represents the power consumed by all components of smartphone when an image with R, G, and B values of all pixels is displayed. Since AMOLED displays have independent light-emitting elements for each pixel, if true black image (R, G, B = 0) is displayed, the dynamic power consumed by the display is zero. Therefore, ${\mathit{P}}_{\mathit{s}\mathit{m}\mathit{a}\mathit{r}\mathit{t}\mathit{p}\mathit{h}\mathit{o}\mathit{n}\mathit{e}}^{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}}-{\mathit{P}}_{\mathit{s}\mathit{m}\mathit{a}\mathit{r}\mathit{t}\mathit{p}\mathit{h}\mathit{o}\mathit{n}\mathit{e}}^{\mathit{R},\mathit{G},\mathit{B}=\mathbf{0}}$ means the dynamic power consumed by the AMOLED display when the image is displayed.

$$\mathbf{D}\mathbf{i}\mathbf{s}\mathbf{p}\mathbf{l}\mathbf{a}\mathbf{y} \mathbf{p}\mathbf{o}\mathbf{w}\mathbf{e}\mathbf{r} \mathbf{s}\mathbf{a}\mathbf{v}\mathbf{i}\mathbf{n}\mathbf{g} \left(\%\right)=\frac{{\mathit{P}}_{\mathit{d}\mathit{i}\mathit{s}\mathit{p}\mathit{l}\mathit{a}\mathit{y}}^{\mathit{n}\mathit{o}\mathit{r}\mathit{m}\mathit{a}\mathit{l} \mathit{m}\mathit{o}\mathit{d}\mathit{e}}-{\mathit{P}}_{\mathit{d}\mathit{i}\mathit{s}\mathit{p}\mathit{l}\mathit{a}\mathit{y}}^{\mathit{m}\mathit{e}\mathit{t}\mathit{h}\mathit{o}\mathit{d}} }{{\mathit{P}}_{\mathit{d}\mathit{i}\mathit{s}\mathit{p}\mathit{l}\mathit{a}\mathit{y}}^{\mathit{n}\mathit{o}\mathit{m}\mathit{a}\mathit{l} \mathit{m}\mathit{o}\mathit{d}\mathit{e}}}\times \mathit{100}$$

(6)

The power saving ratio used in this paper is the ratio of saved power caused by DVS technique to normal mode power. The power-saving ratio is calculated as shown in (6). In (6), ${\mathit{P}}_{\mathit{d}\mathit{i}\mathit{s}\mathit{p}\mathit{l}\mathit{a}\mathit{y}}^{\mathit{m}\mathit{e}\mathit{t}\mathit{h}\mathit{o}\mathit{d}}$represents the dynamic power consumed by smartphone display when the technique is applied, and ${\mathit{P}}_{\mathit{d}\mathit{i}\mathit{s}\mathit{p}\mathit{l}\mathit{a}\mathit{y}}^{\mathit{n}\mathit{o}\mathit{r}\mathit{m}\mathit{a}\mathit{l} \mathit{m}\mathit{o}\mathit{d}\mathit{e}}$ represents the dynamic power consumed by the smartphone display when the technique is not applied.

5.2. Experimental Results

Figure 5a–l are taken for a Galaxy S3 screen using a SONY DSC-RX100 III camera with ISO 200, WB 5500K, shutter speed 1/30 and F-stop 3.5 under normal light environment. Figure 5a,d,g,j are reference images and the used smartphone was set with the default setting (display backlight value = 200). Figure 5b,e,h,k are the results of LGC-DVS and the display backlight was set by LGC-DVS. Figure 5c,f,i,l are the results of the proposed method and the display backlight was set by the proposed method. In Figure 5, the average luma means the average of the RGB values of the corresponding image, and the display power saving means the percentage of display power saving against the reference images when LGC-DVS and the proposed method are applied. Although the average luma determines the image brightness, the brightness we actually see is heavily influenced by the display backlight value. The backlight value achieves power saving, but the visual perceptuality may be harmed. Thus, image compensation is required for the decreased visual satisfaction. Figure 5c shows a 37.13% decrease in display power against Figure 5a by lowering the backlight value although its average luma is higher than that of Figure 5a. Looking at the sky between the central trees of the image, we can see that Figure 5a is clearly a brighter image. However, we can see that the visual satisfaction in Figure 5c is higher because the dark objects are well visible and the detail of the objects is alive. This phenomenon can also be observed when comparing Figure 5d and Figure 5f. We notice that Figure 5d is much brighter than Figure 5f (see sky brightness outside the window), but visual satisfaction is higher in Figure 5f. This is because the boundaries between objects are clearly distinguished and the details of objects are improved. Therefore, we can know that the parameters (${\mathit{L}}_{\mathit{w}\mathit{h}\mathit{i}\mathit{t}\mathit{e}}$, λ, display brightness value) found through the user study is both efficient in terms of human visual satisfaction and power saving.

Figure 5b,e shows that the image compensation is quite passive. Comparing them, the average luma value of the converted image is not higher than that of the original image, so pixel compensation rarely occurs, which may not compensate enough the visual loss due to the decreased backlight value. The same phenomenon occurs when comparing Figure 5e,d. Since the decreased backlight is not compensated, the bottom area where a person, chairs, and drawers exist in Figure 5e is not well distinguished, so human visual satisfaction is considerably reduced. However, as shown in Figure 5f, the proposed method appropriately increases the average luma using the parameters found through the user study. Thus, the objects are well distinguished in the dark part. At the same time, due to sharpening which makes details better, the human visual satisfaction is high. Though not as much as LGC-DVS, the display power reduction of the proposed method is fairly significant.

Table 2. Comparison of power consumption and power-saving ratio.

	Normal	LGC-DVS		Proposed Method
	Power (mW)	Power (mW)	Power Saving (%)	Power (mW)	Power Saving (%)
TMO images	295.89	151.84	45.51	174.29	37.66
LDR images	251.69	134.75	44.21	148.91	37.20
Avg.	269.38	141.59	44.73	159.07	37.38

shows the average power consumption values of LGC-DVS and the proposed method, and their power saving ratios compared to the normal mode for 20 images, respectively. Twenty images were selected from 12 images in the TMO image database [32,39,40], and 8 images from the LIVE database [42]. The brightness of the image was selected from various images ranging from 0 to 200. LGC-DVS [17] and the proposed method show that power is lower than the normal mode in almost all cases. Both of these techniques reduce power consumption compared to the normal mode when the backlight is 200 because the display backlight value is decreased enough.

In terms of the power consumption, LGC-DVS shows very good performance. However, LGC-DVS has a very low increase of pixel values for compensation and thus the total visual quality is degraded. In contrast, the proposed method has a large luma value (Avg. 139.01) compared to the normal mode. Such proper pixel compensation can improve visual satisfaction that has been reduced due to AMOLED display dimming. In addition, the proposed method shows a sufficient power saving rate of more than 35% against the normal mode in both TMO and LDR images.

In this technique, the color of image must be changed compared to color of the original image since the modified tone-mapping operator is used to change all pixel values of original image. In order to satisfy human visual satisfaction despite color change as much as possible, we selected a parameter combination of high MOS scores among all parameter combinations through a user study in Section 4. However, it is necessary to accurately analyze degree of color change caused by pixel conversion and display DVS techniques. To do this, we conducted an experiment on how much color error the proposed technique had compared to normal mode. The experiment of color error was conducted using the Macbeth chart [43], which consists of 24 colors which emulate common natural colors such as skin colors, sky and foliage. These standard colors can be used to measure color error. We displayed images with the standard colors on the smartphone display and measured the color using KONICA MINOLTA’s CA-P410 [44]. The color coordinate values in the CIE Lab color space with and without the proposed technique are shown in Figure 6. We calculated the Euclidian distance between these two coordinates because the distance between two different colors in the L * a * b * space is proportional to the difference in color perceived by humans. As a result, the average Euclidian distance between results with and without this technique was 2.34, which is close to just noticeable difference (JND).

6. Conclusions

In this paper, we have presented a novel DVS scheme based on TMO control for tone-mapped images on an AMOLED LDR display smartphone to achieve both power saving and human perceptuality optimally. We conducted power-aware image quality control using a modified well-designed TMO and also apply unsharp masking to enhance local contrast and details. In extensive experiments, the proposed method saved on average 37.38% of display power for tone-mapped images maintaining the human visual satisfaction by enhancing local contrast and details effectively. Compared to a prior well-designed technique, it showed better human perceptuality for tone-mapped images and general images. Consideration of system power remains future work, including performance acceleration in executing TMOs.