Color Harmonization, Deharmonization and Balancing in Augmented Reality

: Color schemes play a crucial role in blending virtual objects with the real environment. Good color schemes improve user’s perception, which is of crucial importance for augmented reality. In this paper


Introduction
Modern digital technologies such as Augmented Reality (AR) allow us to enjoy new experiences in exciting ways. AR superimposes digital information (3D models, text, images, videos) directly into the real environment by providing additional information to the user [1][2][3][4]. In most cases, AR content is generated by 3D artists and application developers who define its visual properties without any prior knowledge about the realworld environment in which virtual objects will be integrated. This makes it difficult to predict how digital content will appear and if it will fit into its real scenario, with adverse consequences on the AR visualization.
Perceptual issues in AR have been previously documented [5,6]. The color scheme used, as well as the variety of an environment, can hinder a correct perception. This could lead to depth problems [7]. Specific AR colors can hinder the outcomes of augmentation based on the similarity with the chosen color scheme. A typical example occurs with labels where text legibility is problematic due to uncontrollable environmental conditions such as colors and textures of the background on which the augmenting information is overlaid [8][9][10]. This can lead to perceptual issues for users who may not be able to appropriately distinguish virtual text from real objects.
Another common problem occurs where the color of digital content may lead to a poor perception due to a conflict with the colors of the real background for interior design [11,12], where virtual furniture is displayed in an AR environment so that virtual information and real-world scenes can co-exist in harmony. In this way, users can pick the perfect furniture to fit their homes or offices, taking into account their own creativity and imagination. Therefore, virtual objects blended with real ones contribute to an aesthetically pleasing environment with marginal mental efforts by users in choosing colors to be applied to virtual content. Another example of the pertinence of this problem is related to AR applications for gaming and entertainment [13,14], where colors associated with 3D models of characters are usually predefined by developers. In these circumstances, it is very difficult to recognize if colors are fit to ensure a good visibility of virtual information.
To address these issues, this paper proposes a set of recoloring methods to recolor AR content that is integrated into the real world. Given an arbitrary real background, the colors associated with virtual objects are automatically modified by taking into account real colors. In particular, the recoloring process can follow three different approaches, i.e., harmonic, disharmonic, and balance.
The first approach aims to harmonize the colors of virtual objects to make them consistent with the colors of the real background and reach a more pleasing effect to a human eye. The second approach is based on the generation of disharmonious colors with respect to real ones to be associated with virtual objects. The aim is to improve the visibility of the virtual content by obtaining a high contrast between real and virtual objects. The third approach combines both the aforementioned ones and aims to generate harmonious colors while ensuring a good contrast between virtual and real objects. To evaluate the proposed recoloring methods, a preliminary experimentation has been carried out with three different case studies in an indoor controlled environment.
The rest of the paper is structured as follows: Section 2 is a review of the related works, while Section 3 describes the proposed AR recoloring methodology in detail. Experiments and discussions are detailed in Section 4, followed by conclusions.

Related Work
A variety of recoloring techniques has been proposed in the literature for modifying colors of images or videos to match the desirable color style. A typical example is the adjustment of colors to improve the aesthetic appearance and provide a pleasant effect to a human eye. In other cases, the recoloring process is used to enhance the quality of composite images or make some regions more visible than others by preserving the original colors as much as possible. Recoloring methods have been also proposed for AR application fields to enhance the quality of the integration of the virtual information into the real world.

Recoloring Techniques for Images and Videos
In [15][16][17], images are corrected by means of statistical transformation from the original image to the recolored one to adjust its visual appearance. In [18], dominant colors are exploited to find local region correspondences between two images and perform a statistical transfer. A variety of tools to seamlessly edit the content of an image-including replacements or mixing of regions with those of a source image-was introduced in [19]. Specific modifications related to textures, illumination, and colors may be performed in manually selected regions. Similarly, Jia et al. [20] developed a user-friendly system for image compositing that allows for drawing a boundary on the target object, then dragging and dropping it on a different image.
A gradient-domain compositing technique is used to improve quality of boundary selection between foreground and background regions of an image, and therefore to minimize artifacts after the process [21]. Unfortunately, inadequate results occur when images are taken from difference sources or in different conditions. Sunkavalli et al. [22] presented a multi-scale technique that allows for transferring the appearance of one image to another by a two-steps algorithm that consists of a harmonization process, followed by a compositing process. Similar techniques based on the use of multi-scale methods are proposed in [23,24]. Different solutions based on the propagation of desired colors to similar pixels were proposed in [25][26][27][28][29][30][31][32][33][34]. Furthermore, a large set of composite pictures was analyzed to assess their naturalness [35]. At the same time, a harmonization process based on the estimation of color distribution and the recoloring of pixels was used in order to improve the realism of the composite images. In the same context, Zhu et al. [36] presented a learning approach to predict the realism score for composite images as perceived by a human observer, followed by an adjustment of foreground colors.
An end-to-end deep convolutional neural network for image harmonization was proposed in [37], where the context and semantic information would improve harmonization. By contrast, Huang et al. [38] engaged in video harmonization by designing a methodology based on the use of an end-to-end CNN that overcomes the flicker artifacts that occur when the image harmonization process is applied directly to videos. Moreover, Hou and Zhang proposed a method to recolor images starting from predefined color distributions. Eight different color concepts were generated starting from a database containing images grouped by semantic contents [39]. In [40,41], combinations of colors associated to semantic works were adopted to adjust colors on an image by taking into account emotions and the desired editing style.
Other methods were proposed to allow non-expert users to recolor an image by editing a color palette [42,43]. Dominant colors are automatically extracted from an image by means of a k-means algorithm to generate a palette to be used for adjusting the colors of pixels. Zhang et al. [44] proposed a variant of the k-means algorithm to create a palette of representative colors and recolor an image. A color decomposition optimization process is used to express colors of the image as a linear combination of colors of the palette. Tang et al. [45] presented a different approach based on the computing of the convex hull vertices of pixels in RGB color space. Generated palettes better represent the colors contained in an image. In [46,47], an extension of previous methods to 5D RGBXY space was proposed for extracting colors from an image and generating a palette for the recoloring process by means of the optimization process.
To generate aesthetically pleasing recoloring results, images harmonization techniques were also proposed. Cohen-Or et al. [48] enhanced the harmony of the colors contained in images or photographs, where the recolored image remains as much faithful as possible to the original one. The solution is based on the harmonic schemes proposed by [49,50] that define relationships among harmonic colors on the hue wheel in HSV color space. In this way, the best set of colors is calculated by solving an optimization problem through a cut-graph optimization while ensuring the spatial coherence of colors. The main limitation of this approach is that it cannot recognize automatically disconnected areas that belong to the same object, so that these could be shifted to a different sector of the template.
Sawant et al. [51] presented an approach to harmonize both images and videos taking inspiration from Cohen-Or's work [48]. In particular, in the harmonization of videos, they observed a flickering effect in the post-processed video due to the lack of continuity in changing colors between adjacent frames. To solve this issue, authors introduced two different improvements. Firstly, they concentrated on processing of the frames in groups rather than individually. Secondly, an overlapping of frames between two adjacent groups was utilized to avoid inter-group flickering effect.
Based on [48], Huo et al. [52] presented a technique to find the best sets of colors to harmonize images based on the use of the predominant hue in the image. In this way, the best angle and the best template to minimize the harmonization function are found by a continuous use of a graph-cut optimization to guarantee the color coherence. Tang et al. [53] proposed an improvement in colors harmonization process to preserve the coherence of pixels. A two-level graph cut algorithm was utilized which consists of two main parts: a region-level graph and a pixel-level graph. Firstly, the region-level graph allows for shifting regions with similar hue values to the same sector of the selected template, even if they are not close to each other. Subsequently, it generates seeds as pixels that can be interpreted as constraints under which the pixel-level graph harmonizes the colors of the image. The same approach was extended to videos in which foreground and background are treated separately by also employing optical flow estimation, region growing, and foreground extraction algorithm.
Wan et al. [54] developed an algorithm to harmonize composite images. Three different cases of harmonization were proposed: (1) harmonization of both foreground and background; (2) harmonization of the background without adjusting the foreground; (3) harmonization of the foreground maintaining the background unaltered. An assigning process was followed to change the hue value of pixels by employing a novel cost function for optimization. This was formulated by considering that pixels very close to each other, which have similar inharmonic hues, need to be harmonized in the same way. Baveye et al. [55] focused their efforts on improving works proposed by Cohen et al. [48] and Matsuda [49]. The authors have introduced a saliency map to accurately predict which areas of images are more visually attractive and fit for driving the estimation of the harmonic template. Additionally, the use of a new color mapping function and a dedicated color segmentation was adopted to obtain consistent results after the harmonization process.

Recoloring Techniques for Augmented Reality Content
Past approaches provide undeniably accurate results, though they are just focused on static images and videos. In this regard, other solutions were proposed to extend the recoloring process to the AR with aim to improve the quality of the integration of virtual objects in the real environment.
Gruber et al. [56] proposed a solution for harmonizing the combinations of colors of both real and virtual objects in an AR-based scenario. Authors introduced the concept of constrained color harmonization by identifying different types of color-to-object assignments. The downside is that this solution allows for harmonizing colors following only the approach proposed in previous works. In [57,58], researchers investigated what adaptations can be chosen to improve the legibility of graphical symbols superimposed on real-world images by proposing four different types of adaptation by varying several parameters (including thickness and color luminosity of the border, size of symbols, color of the letters or digits inside symbols). Results showed how to combine types of adaptation to improve symbol salience.
In spite of this, there are no experiments to prove the effectiveness of their solution for an augmented visualization. Thomas et al. [13] addressed the issue of selecting the appropriate colors to be assigned to monsters in the ARQuake game by proposing a mobile AR application. The experiments were conducted considering nine colors with four intensities in different scenarios. Although the results confirm the existence of an appropriate set of colors depending on the conditions, the choice is still limited to a few colors.
Ryffel et al. [59] proposed a mobile AR application for a museum context. The application allows visitors to change the colors of a reconstructed virtual painting on the basis of the real one in an interactive way. A user can therefore produce a personal and recolored version of the painting. The recoloring starts from the generation of layers by means of an automatic soft color segmentation algorithm that significantly reduces the manual labor required by the user. Various results are presented by the authors that confirm the usefulness of the proposed solution. Nevertheless, this work has a different target from ours.
Tong et al. [60] presented an AR application to view pieces of furniture in different colors and styles within the real scenario to enhance user experience. Their approach exploits functionalities of deep-learning based on semantic segmentation and fast-speed color transformation to recognize and segment furniture in the users' field of view, and to overlap a virtual color layer on top of them. Furthermore, the application runs on a headmounted AR headset device. Unfortunately, selected colors are not harmonized with the ones contained in the real background. In this context, a set of novel methods to automatically recolor augmented reality content according to colors of the background is proposed. Three different approaches are presented for satisfying different needs in different settings, aiming to improve the quality of the AR visualization.

Materials and Methods
This section introduces the developed methodology for recoloring virtual objects integrated into the real world according to three different approaches-harmonic, disharmonic, and balance. In particular, the virtual content is recolored through a set of colors that are generated starting from the real background. Figure 1 shows the steps of the algorithm adopted for the generation of colors. The algorithm consists of three main stages including color space conversion, color scheme optimization, and color generation. In the first step, a certain number of images, which are representative of the real background, are captured through the device camera. Images are then analyzed to compute the overall color histogram that is representative of real colors. Subsequently, these data are processed to identify a set of colors to be used during the colors' generation phase. To this end, the algorithm exploits concepts from color theory known as color schemes. In the last step, a palette of colors is extracted from the set of colors to perform the recoloring of virtual objects.

Color Space Conversion
The first step of the proposed algorithm is the color space conversion. First, a set of images is obtained by means of the device camera by taking pictures of the real scenario. Next, each image is analyzed to read colors from pixels of images in RGB color space that expresses a color by using three channels: red (R), green (G), and blue (B), defined in the range [0,255]. Afterwards, RGB values are converted into HSV ones that consist of hue, saturation, and value channels. In detail, the RGB values are first divided by 255 to change the range from [0,255] to [0,1], and then converted from RGB to HSV color space through equations reported in [61]. These new values are used to compute the color histogram of the original images using L bins (typically L = 360). Each color's frequency is normalized according to the highest one of the whole distribution.

Color Scheme Optimization
In the second step, a set of colors from which the specific colors to be assigned to virtual objects will be extracted is generated by applying one of the three strategies previously mentioned, i.e., harmonic, disharmonic, and balance approaches. These approaches are based on the notion of color harmony, which in turn is based on schemes developed by Matsuda [49,50] and widely accepted in applicable fields involving colors. Figure 2 shows the eight different harmonic types (or templates) defined over the hue channel of the HSV color wheel.
Each template consists of gray colored sectors that identify a distribution of harmonic hues with a specific relationship over the hue wheel in terms of radial position. In general, templates could consist of shades of the same colors (i,V,T), be composed by complementary colors (I,Y,X) or by a more complex combination (L). In this work, all types of template have been considered, except for the type N, as it corresponds to gray-scale images.
Let T m denote a specific template and α its orientation on the hue wheel by defining a harmonic scheme. The templates can be rotated in arbitrary angles in the range [0,2π], which have been discretized into 360 orientations. In addition, if T m is the chosen template with m ∈ (i, L, T, V, X, Y), then E T m (a) (p) represents the hue of edge boundary of template T m with orientation α, which is the closest to the hue of the pixel H(p). A collection of colors that fall into the gray sectors is considered to be harmonic. On the basis of these last considerations, it is possible to define a function F that represents the distance between the histogram of the image and a template T m . In particular, each hue h over the color wheel, which is representative of one pixel, can be related to border sectors in term of arc-length according to the Equation (1) as proposed in [48]. In addition, the following formula was extended by also considering the contribution of the light channel as reported in [55,62]. where: • || · || is the arc-length distance measured in radians on the hue wheel; • H, S, and V are hue, saturation, and light channels of pixel p, respectively.
Hues of those pixels of input images that are already inside gray sectors of templates in their initial configurations are not considered in the summation as the distance with the sector borders is equal to zero by convention. The F function can be optimized by adopting Brent's algorithm [48] and using one of the recoloring approaches proposed in this paper. In this way, augmented reality content is recolored to accommodate different needs that occur in different situations, as mentioned in the introduction.

Harmonic Approach
The first approach allows for determining a set of harmonic colors according to the background. Therefore, if these colors are assigned to virtual objects, they will appear more harmonic with the real ones and well-integrated into the real environment by ensuring a pleasant effect to a human eye. In order to do this, it is possible to follow the approach presented by Cohen-Or et al. [48], in which an optimization problem is defined and solved. In particular, the function F can be minimized under template T m to get the α angle as reported in the following Equation (2).
The template T m is therefore rotated counterclockwise of α angle. In this way, it allows us to identify a region of harmonic colors on the hue wheel which is useful for the recoloring process.

Disharmonic Approach
The second proposed approach is intended to provide a solution for the generation of inharmonic colors with respect to the real ones. Following this strategy, it is possible to improve the distinctiveness of virtual objects from the real background. In some cases, in fact, virtual objects appear to be too similar to the real ones and this affects their visibility. Therefore, it is essential to recolor virtual objects by using actually different colors from those in the background. Unlike the previous approach, a maximization problem is defined and solved. Considering the template T m , it is required to estimate the angle α that maximizes the cost function F as shown in the following equation:

Balance Approach
The third approach is related to those cases in which it is necessary not only to harmonize virtual objects with the real background, but also to ensure a good visibility. Therefore, it is possible to satisfy these conditions by combining the two first approaches. This one only works with templates that have two sectors, i.e., types L, I, Y, and X. Previously, the term E T m (a) (p) has been considered as the closest border of a specific harmonic template to the H(p). Let us now consider the two closest borders that belong to both sectors of the template. In particular, E 1 represents the first border and E 2 the second one. Thus, the Equation (1) can be written again by subtracting the term relative to the second border of the template to the first one as follows: and then, minimized the function to get al pha angle: By doing so, the optimization of the F function assumes a different meaning from the previous one. In fact, by minimizing both terms of the Equation (5), it is as if the first and the second part followed a harmonic and disharmonic approach, respectively. One sector will identify the hue distribution that fits well with the colors of the real background, while the other sector will be as far as possible from the real color distribution. In this way, the virtual content appears harmonized with the real scenario, but at the same time, high contrast and good visibility is ensured.

Color Generation
In the last step, colors are generated starting from the region identified by the template. The generation of colors is based on the following rules:

•
If harmonic or disharmonic approaches are selected, colors can be extracted from each sector; • if a balance approach is selected, colors are extracted only from the sector that maximizes its distance with real color distribution. Figure 3 depicts the pseudo-code of the algorithm dedicated to the generation of colors from the color schemes.
Different parameters are given as input to the algorithm, including the number of colors to be extracted N c , the size of sectors depending on the selected template S i , and the minimum distance between two nearby colors on the hue wheel in terms of hue L b . The latter allows us to avoid generating colors that are too similar to each other. When the number of generated colors is not sufficient for recoloring virtual objects, as it is not possible to satisfy the constraint about the minimum distance L b , then the remaining colors are generated by using the third part of the algorithm. In this case, the colors generated during the previous steps are considered as base colors from which other ones are generated. One base color is determined by three channels in the HSV color space including hue, saturation, and value. The hue channel is defined on the basis of the angular position on the hue wheel, while the last two channels are initially set to 100. Thus, other colors can be obtained starting from this by changing the saturation and value in HSV color space as shown in Figure 3 without modifying the hue of the color. Therefore, it is possible to generate various shades of the base color that are different to each other. Once the generation of palette is completed, the colors are assigned to virtual objects randomly. Therefore, the virtual content appears recolored according to one of the approaches previously described.

Case Studies
A preliminary experimentation has been carried out by considering three different case studies to assess the validity of the proposed methods. In particular, the first case study was consisted of recoloring of two different porcelain design objects. It has been considered a porcelain owl family to integrate in an office room, and a porcelain rabbit family to place on living-room furniture. These models have been downloaded from [63]. The second set of visual experiments was performed by using a human avatar in front of a bookcase. It has been recolored by changing the colors of the clothes (shirt, trousers, shoes) by leaving the skin color unaltered. This model has been downloaded from [64]. In the third case study, the famous painting The Last Supper by Leonardo Da Vinci has been taken under examination. Labels containing the names of the apostles on a colored background have been added to the painting and recolored during experiments. The three case studies just described are depicted in Figure 4. For each case study, the combination of all templates with the three proposed approaches has been considered for a total of 72 example conditions (4 scenarios × 18 strategies). Furthermore, some data was collected including the covered area, color distribution, and the palette of generated colors.
The recoloring approaches have been implemented into a demo AR application based on ARCore TM SDK [65], an open-source framework adopted for building AR applications for Android TM devices, such as smartphones and tablets. It provides different useful functionalities to create AR experiences, and in this case, the recognition of physical markers has been adopted for aligning virtual objects with real ones. In order to improve the quality of the AR visualization, information about the lighting of the real scenario has been collected and used by means of ARCore functionalities. In particular, they have been adopted for the generation of shadows of virtual objects and making these objects more realistic by enhancing the immersive experience for users.
As far as the hardware is concerned, the AR application has been programmed on a Samsung S9 [66] mobile device. This is an Android smartphone equipped with a M3 Mongoose (4 × 2.8 GHz + 4 × 1.7 GHz Cortex-A55) processor, a 12MP rear camera, 6.2-inch touch-screen with 2960 × 1440 resolution, and integrated inertial tracking sensors (accelerometer, compass, and gyro). The luminosity of the device has been set to 100%. Since all the computations are carried out on the device itself, there are no other external hardware components required for the data processing.

Results
In this section, results of the recoloring process for each case study are reported. For each visual example, the color scheme is presented by indicating the covered area as well. The latter is a percentage value that represents how much a template with the optimized angle fits well the color distribution of the input image. Additionally, the virtual objects are showed with different colors depending on the strategy applied to the recoloring process.

Porcelain Design Objects
The first case study deals with the recoloring of two different types of porcelain design objects. In particular, in the first case, a porcelain owl family placed on a table in an office room has been considered, while in the second case, a porcelain rabbit family was positioned on living-room furniture. In the recoloring of these objects has been considered the need to make them more visually pleasing when they blended with real objects. All templates have been combined with the three proposed approaches to obtain different results. Figure 5 depicts the color schemes and the generated color palettes at the end of experiments about the recoloring of the porcelain owl family. Considering the harmonic approach, it is possible to observe how the templates are oriented to fit the distribution of the colors of the real background. The real distribution is ranged by a region on the hue wheel close to red and oranges hues. The covered area percentage (CA) varies between a minimum value of 39% (template i) and a maximum value of 95.8% (template X). In the case of the disharmonic approach, the orientation of the templates is such as to maximize the distance among sectors of the specific template and the distribution of the real colors.
The minimum value of the CA is reached by applying the template i with a value of 2%, while the maximum value 14.2% is relative to the template X. By adopting the third approach, one sector of the harmonic scheme is brought closer to the real distribution, while the second sector is far from it, and it is used to generate the palette of colors. In this case, the maximum and minimum values are equal to 11.2% (template I) and 44.5% (template X), respectively. In Table 1, percentages of covered areas by the selected template are reported for each experiment.  Figure 6 shows the porcelain owl family placed on a desk close to two other real objects with different colors. Three colors were extracted from the palette and assigned to the virtual objects by which the porcelain owl family is composed. When the harmonic approach is used for recoloring objects, the extracted colors are close to the real ones. In particular, the application of i and I templates produces colors very similar to each other (i.e., shades of brown). On the contrary, V, L, Y, and X templates generate colors closer to brown and dark yellow hues. Template T generates shades of brown and green. Different from the harmonic approach, the application of the disharmonic strategy produces discordant colors with respect to the real background. Blue, violet, and green colors are generated and applied to the virtual objects. By using the last approach, instead, L and I templates generates shades of blue and violet. By applying the Y template, instead, shades of light blue are obtained. The adoption of the X template proposes blue color and shades of violet. As previously mentioned, it has been also considered the recoloring of a porcelain rabbit family placed on a living-room furniture. Figure 7 depicts an example of the recolored virtual objects obtained by applying the Y template and the three proposed strategies.

Human Avatar
In this case, we have focused on the recoloring of clothes of an avatar. The avatar was placed in an indoor environment in front of a bookcase. By applying the proposed method to the recoloring of clothes of the human avatar, a twofold goal should be achieved, i.e., a better harmonization with real objects by also maintaining a good visibility. Moreover, three different colors have been extracted from the generated palette to recolor the shirt, the trousers, and the shoes. Figure 8 depicts the colors schemes and the generated color palettes. The harmonic approach identifies a region of colors that are close to the distribution of real colors. Furthermore, in this case, the real distribution covers a large region on the color wheel. More specifically, as it is possible to observe from the schemes, the range of colors varies from orange to blue. The minimum value of the covered area is reached by applying the template i-type with a value of 27.4%, while the maximum value is relative to the adoption of the template T with a percentage value that is equal to 91.1%.
The application of the disharmonic strategy leads towards the identification of a region of colors that are far from the real distribution. In particular, the covered area reaches the maximum value (27.4%) and the minimum value (2%) in the cases of template T and template i, respectively. By applying the last approach, the covered area varies between a minimum value of 24.6% (template I-type) and a maximum value of 32.2% (template X). In Table 2, percentages of covered areas by the selected template are reported for each experiment.  Figure 9 shows different instances of the human avatar with the recolored clothes. As previously mentioned, only three colors have been used to change the colors of clothes. The application of the harmonic approach ensures similar colors to be assigned to virtual objects. In detail, templates i and I generate three similar colors, which are shades of orange. Furthermore, the adoption of V and Y templates produces two similar colors to the previous ones, but the third is a shade of yellow applied to the shirt and the trousers, respectively. The application of the X template generates a bit different recoloring than the previous ones. In particular, colors are shades of yellow and brown. By using T and L templates, a shade of brown is applied to the shirt, while shades of blue and violet are assigned to the trousers and shoes. The disharmonic approach, instead, produces colors that ensure a high contrast with the real one. When I and X templates are selected, the clothes of a human avatar are recolored with shades of violet. In the cases of V and T templates, shades of green are proposed. The application of L and Y templates generated colors represented by shades of blue and violet. Finally, the template i produces shades of blue. The balance approach generates shades of blue in the case of L, I, and X templates, while shades of light blue when Y template is selected. High-resolution images about these visual examples can be found in the Supplementary Materials.

Painting
In this case, we have considered the recoloring of the label background that contains the names of the characters in The Last Supper by Leonardo Da Vinci. In this last case, it is expected that the textual content is stand out from the background in order to improve its legibility. Figure 10 shows how the real distribution covers a region of colors that goes from red to light green counterclockwise. The three strategies have been adopted to generate and recolor the virtual content. When the harmonic approach is used, templates orient themselves to cover most of the region of real distribution. CA ranges between a minimum value of 28.6% (template i-type) and a maximum value of 99% (template T). On the contrary, the disharmonic approach generates synthetic colors with high contrast with the real ones. The CA reaches the maximum value (9.2%) and the minimum value (0.2%) in the cases of template X and i, respectively. By using the latter approach, the CA reaches the minimum value in the case of I template, while the highest one is attained with the adoption of the X template. In Table 3, percentages of covered areas by the selected template are reported for each experiment.  Figure 11 shows different ways to recolor the background of each label using a single color. The first approach generates very similar colors. In particular, shades of brown are proposed in the case of V, T, X templates. Instead, i and I templates produce shades of dark yellow. A light red color is obtained when L and Y templates are selected. The disharmonic approach returns colors that can be classified in three groups. The first one is related to the application of V and T templates that generate a color similar to green color. The second group, instead, is represented by shades of blue that are obtained when i and Y templates are used. The third group proposes shades of violet (templates L, I, and X). When the balance approach is adopted, shades of blue are generated and applied to the labels. High-resolution images about these visual examples can be found in the Supplementary Materials.

Discussion
Our AR-based method allows for recoloring virtual objects according to the colors of the real background. To this end, color schemes by [48,50] have been used to identify a set of possible colors in HSV color space. In addition, three different strategies for recoloring have been also implemented and, in particular, the harmonic, disharmonic, and balance approaches. Three different case studies have been considered in our analysis. More specifically, two different porcelain design objects, a human avatar positioned in front of a bookcase, and finally, the famous painting, The last supper by Leonardo Da Vinci in which labels containing the names of apostles have been added to the piece.
Considering the recoloring of the porcelain owl family, the most interesting results are obtained when the harmonic approach is adopted. Colors assigned to virtual objects are harmonic with the real background. Therefore, the porcelain owl family appears aesthetically pleasing to the user and well-integrated to the real environment. This is particularly true in the cases of i,V, I, Y, and X templates, where shades of brown are assigned to the virtual content. Differently, the application of T template produce a shade of green that is not so harmonic with the background. Since the goal of placing a piece of furniture in a room for interior design applications is to reach optimal integration into the real environment, the harmonic approach seems the best choice. Similar results have been obtained with the recoloring of the porcelain rabbit family placed on living-room furniture.
Regarding the recoloring of the clothes of the human avatar, interesting recoloring is obtained when both harmonic and balance approaches are adopted. In the first case, i, V, I, Y, and X templates produce similar results that make the avatar harmonic with the real background. The use of T and L templates, instead, allows for recoloring virtual objects with colors that are closer to blue and violet hues. More interesting results are obtained when the balance approach is applied. In fact, the clothes of the avatar are recolored by using shades of blue. In this way, clothes appear harmonic with the real background, and at the same time, a good level of visibility is ensured. The templates L and Y generate similar colors really close to blue and light blue. On the contrary, template I and X recolor virtual clothes with shades of dark blue and violet. In this case, the use of the balance approach seems to be the best as it allows for achieving a compromise between harmony and visibility of virtual objects blended with real ones. In the recoloring of the labels contained in the painting, the main goal is to ensure text legibility. Differently from the previous case studies, the application of the disharmonic approach seems to bring the best result. In particular, the recolored labels can be categorized into three different groups. The first group includes V and T templates according to which shades of green have been generated and applied to the virtual labels. The second one, instead, contains the results obtained when i and Y templates are used. In particular, shades of blues have been associated with virtual objects. The third group proposes shades of violet (templates L, I, and X). However, it can be observed that the balance approach leads towards similar results, though the too dark colors do not ensure optimal visibility in some parts of the painting.
Overall, our AR method proposes an interesting method to integrate virtual objects into the real scenario by automatically choosing colors to be associated to the augmented content. Thanks to shadows and environment lighting conditions estimated through the ARCore framework, it has been possible to achieve high quality in the integration of virtual information into the real world. The limitations are mostly related to the lack of usability and perception studies. No specific bench-marking tests have been performed to measure the time required by the recoloring process to recolor virtual objects. The optimization process could be performed on a server that returns results to the smartphone through the Web to reduce processing times.

Conclusions
This paper contributes a set of novel recoloring methods for AR scenarios to improve AR visualization by satisfying different needs in different settings. In particular, color schemes have been adopted in order to generate a set of colors with which to recolor the virtual content according to colors of the real background. Differently from works reported in Section 2, our approach has been proposed for satisfying different requirements and achieve different objectives depending on applications without restricting our proposal to specific needs. In this regard, three different strategies have been developed-harmonic, disharmonic, and balance-to improve AR visualization.
The harmonic approach has been proposed to harmonize the virtual content with the real background by assigning harmonic colors, i.e., colors that fit well with real ones. The disharmonic approach, instead, has been adopted to provide a solution for the generation of inharmonic colors avoiding too similar colors with those contained in the image background by improving the visibility of virtual information. The last approach has been used to achieve a compromise between the last two goals by providing a harmonic recoloring of the virtual content but, at the same time, ensuring its distinctiveness from real objects. In this context, three different case studies with their own requirements have been also proposed to assess the validity of the recoloring strategies.
In the first case study, virtual design objects have been augmented in an office and in a home environment. In the second one, instead, a human avatar has been positioned in front of a bookcase. In the last case study, the famous painting "The Last Supper" by Leonardo Da Vinci that contained virtual labels with the names of the apostles has been considered. In all these cases, the aim was the recoloring of virtual information by using the three strategies and all color schemes. What emerged from the outcomes is a great potential of the proposed approaches that are able to automatically improve the quality of the AR visualization by modifying original colors with new ones generated on the basis of the real background.
In the future, field experimentation will be carried out on industrial case studies with end-users to assess the proposed approaches. In this regard, we will implement several improvements. It will be of crucial importance to consider the quality of the rendering in order to provide a more realistic and pleasant AR visualization. It will be also developed a specific logic for choosing the saturation and the value of each new generated color according to the lightning conditions of the real background. Furthermore, it will be very interesting to investigate a local recoloring technique, which takes into account the area immediately adjacent to the virtual content.