Order Space-Based Morphology for Color Image Processing

Abstract

Mathematical morphology is a fundamental tool based on order statistics for image processing, such as noise reduction, image enhancement and feature extraction, and is well-established for binary and grayscale images, whose pixels can be sorted by their pixel values, i.e., each pixel has a single number. On the other hand, each pixel in a color image has three numbers corresponding to three color channels, e.g., red (R), green (G) and blue (B) channels in an RGB color image. Therefore, it is difficult to sort color pixels uniquely. In this paper, we propose a method for unifying the orders of pixels sorted in each color channel separately, where we consider that a pixel exists in a three-dimensional space called order space, and derive a single order by a monotonically nondecreasing function defined on the order space. We also fuzzify the proposed order space-based morphological operations, and demonstrate the effectiveness of the proposed method by comparing with a state-of-the-art method based on hypergraph theory. The proposed method treats three orders of pixels sorted in respective color channels equally. Therefore, the proposed method is consistent with the conventional morphological operations for binary and grayscale images.

1. Introduction

Mathematical morphology is a theory that originated in the 1960s for treating the shape of objects in images based on set theory and lattice theory. Mathematical morphology has a wide range of applications in digital image processing. At the present time, there are a large number of theoretical and practical studies on mathematical morphology, where the development of basic techniques for binary and grayscale images is almost complete. For a comprehensive introduction to morphological image analysis, one may refer to the book by Soille [1].

Unlike the case of binary or grayscale images, mathematical morphology for color images and higher-dimensional images is still under study because of the difficulty and uncertainty of ordering vector data, such as color vectors or pixels, in color images. That is, there is no canonical ordering of pixels in color images, and an arbitrary choice of order function that assigns an order to any pixel will provide good results for some problems or images and bad results for others.

Recently, a number of morphological techniques for color images have been presented to overcome the above difficulty. Sartor and Weeks  [2] proposed a method based on the $(C-Y)$ color space, which has the luminance coordinate (Y) and the chromatic coordinates $(R-Y,B-Y)$, where a generalized distance between a color and reference color is used for ordering color vectors. For computing the generalized distance, one requires a $3\times 3$ covariance matrix. They also provided a brief introduction to lattice theory. Angulo also proposed a method to use a reference color for computing color distance, where the distance-based and lexicographical approaches are generalized in RGB, LSH and Lab color spaces [3]. d’Ornellas and Costa proposed an ordering relation for colors that is based on the natural ordering of spectra [4]. As a result, they derived a color-model-independent approach for color mathematical morphology. However, in compensation for the independency, their method requires reconstructing the fundamental spectrum from a conventional 3D color vector. Benavent et al. proposed a method based on the color histogram of a quantized color image [5], where the pixels are sorted on the basis of the frequency recorded in the histogram. In their method, the results may be affected by the number of quantized colors. Comer and Delp defined erosion and dilation for color morphology using marginal ordering, which uses only one component of a vector for ordering pixels, and reduced ordering, which uses a dimensionality reduction function for ordering three-dimensional pixel data [6]. Zhao also proposed a method for color morphology based on marginal ordering, where no new color artifacts are produced [7]. Van de Gronde and Roerdink pointed out that the false color problem lies in violating certain invariances, and demonstrated the effectiveness of hue-invariant and rotation-invariant methods [8]. Gimenez and Evans [9] analyzed the convex color sieve (CCS) by Gibson et al. [10] and the vector area morphology sieve (VAMS) by Evans [11] algorithmically and experimentally. Evans also described a chapter devoted to color area morphology scale spaces [12]. Burgeth et al. extended morphological maximum and minimum operations to tensor-valued settings by employing Loewner ordering for symmetric matrices [13]. Naegel and Passat extended the gray-level definition of component trees to multi-value images, and compared some strategies for color image processing based on component trees in the color image filtering and color document binarization fields [14]. Tushabe and Wilkinson proposed a color filtering method using max-tree representation, where a color image is converted into a grayscale image that is then subjected to the conventional filtering techniques, and demonstrated three restitution mechanisms that reconstruct the filtered image back into color: nearest color, nearest neighbor and mean of parent approaches [15]. Lezoray et al. proposed a method for the graph-based ordering of color vectors, and demonstrated the proposed mathematical morphology operations in RGB, IHSL [16], $L^{*}{a}^{*}{b}^{*}$ and CIECAM02 [17] color models [18]. Wang et al. proposed color morphological operators on a hypergraph to avoid the loss of details caused by fixed structuring elements [19]. The hypergraph-based method has two parameters for controlling spatial and color similarities between pixels.

In this paper, we propose a method for morphologically operating color images on the basis of an order space, which is a three-dimensional space with the axes of orders of pixels in three color channels. Specifically, we take the RGB (red, green and blue) color channels of RGB color images as a typical example, and sort the pixels in each channel separately. As a result, we obtain a triplet of orders for each pixel, which is then represented as a point in the order space. After that, we compute a single order from the obtained triplet of orders by a function from 3D to 1D, and then apply morphological operations based on the single order of pixels. In our experiments, we compere the proposed method with Wang’s hypergraph-based method [19], and demonstrate the performance of the proposed method in a noise-removal task. Moreover, we present the fuzzy version of the proposed order space-based method, which improves the performance further.

Since a color image is a stack of three grayscale images corresponding to three color channels, it would be consistent with the conventional morphological operation for binary and grayscale images to take into account the triplet of orders of pixels in the color version of the morphological operations. Therefore, the proposed order space-based method can be a plausible candidate for the extension of a grayscale morphology to a color one.

The rest of this paper is organized as follows: Section 2 briefly summarizes binary and grayscale morphological operations. Section 3 proposes the order space-based morphological operations for color images. Section 4 proposes an exponentially weighted averaging method for fuzzifying the order space-based method. Section 5 shows the experimental results obtained from a standard image dataset. Section 6 discusses the results obtained in Section 5. Finally, Section 7 concludes this paper.

2. Morphology for Binary and Grayscale Image Processing

In this section, we briefly summarize the basics of mathematical morphology for binary and grayscale images in preparation for the subsequent sections. In the following definitions of morphological operations, since binary images can be regarded as a special case of grayscale images, we focus on grayscale morphology [20].

Let $G=\left[g_{ij}\right]$ be a grayscale image, where $g_{ij}\in \{0,1,\dots ,L-1\}$ for $i=1,2,\dots ,m$ and $j=1,2,\dots ,n$ denotes the pixel value at the ith row and jth column in G with m rows and n columns, where $L-1$ denotes the maximum pixel value, e.g., $L=256=2^8$ for an 8-bit grayscale image. The grayscale pixel values $g_{ij}=0$ and $g_{ij}=L-1$ denote black and white pixels, respectively. To perform a morphological operation on G, we need a structuring element S, which is a set of relative coordinates, e.g., a $3\times 3$ ‘Cross’ (or ‘Rhombus’) structuring element $S=\left\{\right(-1,0),(0,-1),(0,0),(0,1),(1,0\left)\right\}$ and a ‘Square’ one $S=\left\{\right(-1,-1),(-1,0),(-1,1),(0,-1),(0,0),(0,1),(1,-1),(1,0),(1,1\left)\right\}$, which are illustrated in Figure 1. We assume that all structuring elements used in this paper are point-symmetric, which is a reasonable assumption for isotropic image processing tasks and simplifies the definitions of morphological operations as follows. (In a formal description, dilation is defined as the Minkowski set addition by a given structuring element. On the other hand, erosion is defined as the Minkowski set subtraction by the reflection of the structuring element. With the symmetric structuring elements, we can skip the reflection step.)

The dilation of G by a symmetric structuring element S is defined as

$$D(G,S)=\left[d_{ij}\right],d_{ij}=\underset{(k,l)\in S}{max}\left\{g_{i+k,j+l}\right\},$$

(1)

and the erosion of G by the same S is defined as

$$E(G,S)=\left[e_{ij}\right],e_{ij}=\underset{(k,l)\in S}{min}\left\{g_{i+k,j+l}\right\},$$

(2)

where we note that if $i+k\notin \{1,2,\dots ,m\}$ or $j+l\notin \{1,2,\dots ,n\}$, then the corresponding value $g_{i+k,j+l}$ is excluded from the candidates for output pixel.

The above dilation and erosion operations change the mean brightness of a given image. To compensate for such bias caused by dilation and erosion operations, they are often combined with each other as follows: the opening of G by S is defined as

$$O(G,S)=D\left(E\right(G,S),S),$$

(3)

and the closing of G by S is defined as

$$C(G,S)=E\left(D\right(G,S),S),$$

(4)

i.e., the opening operation is composed of an erosion followed by a dilation, and the closing operation is composed of a dilation followed by an erosion. The opening and closing operations are also combined with each other as follows: the open–closing of G by S is defined as

$$OC(G,S)=C\left(O\right(G,S),S),$$

(5)

and the close–opening of G by S is defined as

$$CO(G,S)=O\left(C\right(G,S),S),$$

(6)

i.e., the open–closing operation is composed of an opening followed by a closing, and the close–opening operation is composed of a closing followed by an opening (the definitions of the open–closing and close–opening operations are convertible into each other as mentioned by Schevenels and Sigmund [21]).

Here, we note that if $L=2$, then these definitions are reduced to those for binary images.

3. Order Space-Based Morphology for Color Image Processing

The above morphological operations for binary and grayscale images include the maximum and minimum selection, which is based on the assumption that the pixels in an image can be sorted in ascending or descending order of their pixel values. For a binary or grayscale image, it is easy to sort the pixels by their values. However, for a color image, it is not easy because each color pixel has three values, e.g., red, green and blue components, the orders of which may conflict with each other. Therefore, sorting each color component separately frequently causes undesirable colors.

In this section, we propose a method for morphologically processing color images in an appropriate manner, which can alleviate the drawback of the naive separable method.

Let $F=\left[f_{ij}\right]$ be an RGB color image, where $f_{ij}=[f_{ij}^{\mathrm{R}},f_{ij}^{\mathrm{G}},f_{ij}^{\mathrm{B}}]$ for $i=1,2,\dots ,m$ and $j=1,2,\dots ,n$ denotes the RGB color vector at the position $(i,j)$ in F, where $f_{ij}^{\mathrm{R}},f_{ij}^{\mathrm{G}}$ and $f_{ij}^{\mathrm{B}}$ denote the red (R), green (G) and blue (B) components of the pixel $f_{ij}$, respectively. Then, we can sort the color components covered by a structuring element in ascending order of their values, e.g., $f_{i+k\left(a_1^{\mathrm{R}}\right),j+l\left(a_1^{\mathrm{R}}\right)}^{\mathrm{R}}\le f_{i+k\left(a_2^{\mathrm{R}}\right),j+l\left(a_2^{\mathrm{R}}\right)}^{\mathrm{R}}\le \cdots \le f_{i+k\left(a_5^{\mathrm{R}}\right),j+l\left(a_5^{\mathrm{R}}\right)}^{\mathrm{R}}$ for the R component with a structuring element S with five pixels, as shown in Figure 1a, where $a_1^{\mathrm{R}},a_2^{\mathrm{R}},\dots ,a_5^{\mathrm{R}}$ denote the sorted indices of pixels covered by S, and $k\left(a_1^{\mathrm{R}}\right)$ and $l\left(a_1^{\mathrm{R}}\right)$ denote the vertical and horizontal relative coordinates of the pixel corresponding to $a_1^{\mathrm{R}}$, respectively.

Let $\xi =a_{\eta }^{\mathrm{R}}$ (for $\eta =1,2,\dots ,5)$ be a function where a pixel index $\xi$ is related to an order $\eta$ given by the above sorting procedure. Then, we consider the inverse function of $\xi =a_{\eta }^{\mathrm{R}}$ as $\eta =o_{\xi }^{\mathrm{R}}={\left(a^{\mathrm{R}}\right)}_{\xi }^{-1}$, where an order $\eta$ is related to a pixel index $\xi$. That is, each pixel $\xi$ has the corresponding order $\eta$. According to the order, we can arrange the five pixels on an axis as shown in Figure 2a, where each black point denotes a pixel $\xi$, which is placed at the position designated by the order $\eta$.

Similarly, we can sort the set of pixels in the ascending order of the G components to have the sorted indices $a_1^{\mathrm{G}},a_2^{\mathrm{G}},\dots ,a_5^{\mathrm{G}}$ satisfying $f_{i+k\left(a_1^{\mathrm{G}}\right),j+l\left(a_1^{\mathrm{G}}\right)}^{\mathrm{G}}\le f_{i+k\left(a_2^{\mathrm{G}}\right),j+l\left(a_2^{\mathrm{G}}\right)}^{\mathrm{G}}\le \cdots \le f_{i+k\left(a_5^{\mathrm{G}}\right),j+l\left(a_5^{\mathrm{G}}\right)}^{\mathrm{G}}$. We also consider the inverse function of $\xi =a_{\eta }^{\mathrm{G}}$ as $\eta =o_{\xi }^{\mathrm{G}}={\left(a^{\mathrm{G}}\right)}_{\xi }^{-1}$. Then, we can plot the five pixels on a two-dimensional coordinate system with the axes $o^{\mathrm{R}}$ and $o^{\mathrm{G}}$ as shown in Figure 2b, where we note that no two pixels are aligned with one another vertically and horizontally.

By sorting the B components, we also have the sorted indices $a_1^{\mathrm{B}},a_2^{\mathrm{B}},\dots ,a_5^{\mathrm{B}}$ satisfying $f_{i+k\left(a_1^{\mathrm{B}}\right),j+l\left(a_1^{\mathrm{B}}\right)}^{\mathrm{B}}\le f_{i+k\left(a_2^{\mathrm{B}}\right),j+l\left(a_2^{\mathrm{B}}\right)}^{\mathrm{B}}\le \cdots \le f_{i+k\left(a_5^{\mathrm{B}}\right),j+l\left(a_5^{\mathrm{B}}\right)}^{\mathrm{B}}$, which are summarized as a function $\xi =a_{\eta }^{\mathrm{B}}$ for $\eta =1,2,\dots ,5$, the inverse function of which is given by $\eta =o_{\xi }^{\mathrm{B}}={\left(a^{\mathrm{B}}\right)}_{\xi }^{-1}$. Then, we can plot each pixel on a three-dimensional coordinate system with the axes $o^{\mathrm{R}},o^{\mathrm{G}}$ and $o^{\mathrm{B}}$ as shown in Figure 2c. We call the space determined by the above coordinate system order space. As a result, an RGB color pixel identified by an index $\xi$ is mapped to a point on the order space as $(o_{\xi }^{\mathrm{R}},o_{\xi }^{\mathrm{G}},o_{\xi }^{\mathrm{B}})$. This is a natural consequence of the generalization of the basics of morphological operations for binary and grayscale images, since there are no advantages and disadvantages among the three color channels. Therefore, it is reasonable to construct morphological operations for color images on the basis of the 3D order space.

Algorithm 1 summarizes the procedure for mapping from an RGB color space to the order space. In Algorithm 1, line 3, the function ‘argsort’ sorts the arguments $f_1^{\mathrm{X}},f_2^{\mathrm{X}},\dots ,f_{\left|S\right|}^{\mathrm{X}}$ in ascending order as $f_{a_1^{\mathrm{X}}}^{\mathrm{X}}\le f_{a_2^{\mathrm{X}}}^{\mathrm{X}}\le \cdots \le f_{a_{\left|S\right|}^{\mathrm{X}}}^{\mathrm{X}}$, and returns their indices $a_1^{\mathrm{X}},a_2^{\mathrm{X}},\dots ,a_{\left|S\right|}^{\mathrm{X}}$, where $\left|S\right|$ denotes the number of elements of S. Each value of the coordinates in the order space is given at line 5.

Algorithm 1 Mapping from RGB color space to order space

Require:  a set of pixels $\{f_{i+k,j+l}=[f_{i+k,j+l}^{\mathrm{R}},f_{i+k,j+l}^{\mathrm{G}},f_{i+k,j+l}^{\mathrm{B}}]|(k,l)\in S\}$ Ensure:  a set of coordinates in order space $\{(o_{\xi }^{\mathrm{R}},o_{\xi }^{\mathrm{G}},o_{\xi }^{\mathrm{B}})|\xi \in \{1,2,\dots ,\left|S\right|\left\}\right\}$  1: Assign serial numbers $\xi \in \{1,2,\dots ,|S\left|\right\}$ to all pixels in the required set to have a set of re-indexed pixels $\{f_{\xi }=[f_{\xi }^{\mathrm{R}},f_{\xi }^{\mathrm{G}},f_{\xi }^{\mathrm{B}}]|\xi \in \{1,2,\dots ,\left|S\right|\left\}\right\}$.  2: for $\mathrm{X}\in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$ do  3:     $[a_1^{\mathrm{X}},a_2^{\mathrm{X}},\dots ,a_{\left|S\right|}^{\mathrm{X}}]=\mathrm{argsort}(f_1^{\mathrm{X}},f_2^{\mathrm{X}},\dots ,f_{\left|S\right|}^{\mathrm{X}})$  4:     for $\xi \in \{1,2,\dots ,|S\left|\right\}$ do  5:         $o_{a_{\xi }^{\mathrm{X}}}^{\mathrm{X}}=\xi$  6:     end for  7: end for  8: Return $\{(o_{\xi }^{\mathrm{R}},o_{\xi }^{\mathrm{G}},o_{\xi }^{\mathrm{B}})|\xi \in \{1,2,\dots ,\left|S\right|\left\}\right\}$

In the 3D order space, an RGB color pixel with an index $\xi$ is represented by a triplet $(o_{\xi }^{\mathrm{R}},o_{\xi }^{\mathrm{G}},o_{\xi }^{\mathrm{B}})$. To define dilation and erosion for an RGB color image, we need to reduce the triplet $(o_{\xi }^{\mathrm{R}},o_{\xi }^{\mathrm{G}},o_{\xi }^{\mathrm{B}})$ to a singlet $o_{\xi }$. If we have $o_{\xi }^{\mathrm{R}}\le o_{{\xi }^{\prime }}^{\mathrm{R}},o_{\xi }^{\mathrm{G}}\le o_{{\xi }^{\prime }}^{\mathrm{G}}$ and $o_{\xi }^{\mathrm{B}}\le o_{{\xi }^{\prime }}^{\mathrm{B}}$ for two pixels $\xi$ and ${\xi }^{\prime }$, then we can assign an order satisfying $o_{\xi }\le o_{{\xi }^{\prime }}$. For example, the following equations for obtaining a singlet $o_{\xi }$ from the triplet $(o_{\xi }^{\mathrm{R}},o_{\xi }^{\mathrm{G}},o_{\xi }^{\mathrm{B}})$ satisfy this condition:

$$o_{\xi }^S=o_{\xi }^{\mathrm{R}}+o_{\xi }^{\mathrm{G}}+o_{\xi }^{\mathrm{B}},$$

(7)

$$o_{\xi }^P=o_{\xi }^{\mathrm{R}}{o}_{\xi }^{\mathrm{G}}{o}_{\xi }^{\mathrm{B}},$$

(8)

$$o_{\xi }^M=\mathrm{med}\left\{o_{\xi }^{\mathrm{R}},o_{\xi }^{\mathrm{G}},o_{\xi }^{\mathrm{B}}\right\},$$

(9)

where ‘med’ means to select the median value from a given set of elements. We chose these equations here for their simplicity and easy implementation. Other choices may include weighted average, mode and nonlinear transformations of the order triplet. For example, let us consider three pixels in 3D order space: $a=(1,2,3),b=(3,1,2)$ and $c=(2,3,1)$. Then, the above three Equations (7)–(9) give the same orders $o^S=1+2+3=6,o^P=1\times 2\times 3=6$ and $o^M=\mathrm{med}\{1,2,3\}=2$ for $a,b$ and c. To assign different orders to different points in 3D order space, we need more sophisticated equations than the above.

Figure 3 shows the contour lines of $o_{\xi }^S,o_{\xi }^P$ and $o_{\xi }^M$ in their order spaces, where the top, middle and bottom rows correspond to $o_{\xi }^S,o_{\xi }^P$ and $o_{\xi }^M$, respectively. In each row, the five figures from left to right show cross sections for $o^{\mathrm{B}}=1,2,\dots ,5$. In each figure, the horizontal and vertical axes denote $o^{\mathrm{R}}$ and $o^{\mathrm{G}}$. For example, in Figure 3a, feasible orders are denoted by color points, and contour lines for fixed values of $o^S=6,7,\dots ,10$ are denoted by color lines with the corresponding colors. For fixed $o^S$ and $o^{\mathrm{B}}$, we have a straight line $o^{\mathrm{G}}=(o^S-o^{\mathrm{B}})-o^{\mathrm{R}}$ with slope $-1$. On the other hand, in the middle row of Figure 3, for fixed $o^P$ and $o^{\mathrm{B}}$, we have a hyperbola $o^{\mathrm{G}}=(o^P/o^{\mathrm{B}})/o^{\mathrm{R}}$. In the bottom row of Figure 3, we have no analytical expressions for the contour lines. However, we can draw them as shown in this figure. It is interesting that there are contour ‘areas’ as shown in Figure 3l–n with light blue, light orange and light green colors.

Once a function from $(o_{\xi }^{\mathrm{R}},o_{\xi }^{\mathrm{G}},o_{\xi }^{\mathrm{B}})$ to $o_{\xi }$ is given like in (7)–(9), we can define the dilation and erosion of F by S as follows:

$$D(F,S)=\left[d_{ij}\right],d_{ij}=f_{{\xi }^{max}},{\xi }^{max}=arg\underset{\xi \in \{1,2,\dots ,|S\left|\right\}}{max}{o}_{\xi },$$

(10)

$$\begin{array}{c}\hfill E(F,S)=\left[e_{ij}\right],e_{ij}=f_{{\xi }^{min}},{\xi }^{min}=arg\underset{\xi \in \{1,2,\dots ,|S\left|\right\}}{min}{o}_{\xi }.\end{array}$$

(11)

Next, we combine the above dilation and erosion operations for defining opening and closing operations as follows:

$$O(F,S)=D\left(E\right(F,S),S),$$

(12)

$$C(F,S)=E\left(D\right(F,S),S).$$

(13)

Here, we define an image $Z=\left[z_{\xi }\right]$ whose pixel value $z_{\xi }$ denotes the order $o_{\xi }$ of the corresponding pixel of F. Then, the above morphological operations correspond to the grayscale morphological operations applied to Z. Therefore, idempotence, increasingness and (anti-)extensiveness hold for the application of morphological opening and closing operations to Z as well as the conventional opening and closing operations.

Combining the above opening and closing operations, we can define open–closing and close–opening operations as follows:

$$OC(F,S)=C\left(O\right(F,S),S),$$

(14)

$$CO(F,S)=O\left(C\right(F,S),S),$$

(15)

which are consistent with the definitions of the corresponding operations for binary and grayscale images summarized in the previous section.

We applied the proposed order space-based morphological operations to a color image in Figure 4, and obtained the images in Figure 5, where the top, middle and bottom rows were obtained with $o^S,o^P$ and $o^M$ in (7)–(9), and the six columns from left to right show the results of dilation, erosion, opening, closing, open–closing and close–opening operations, respectively. We observe that the three kinds of ordering, $o^S,o^P$ and $o^M$, give similar output images to each other.

4. Fuzzy Morphological Operations by Exponentially Weighted Averaging

In this section, we extend the above order space-based morphological operations, which are crisp operations, to fuzzy ones by introducing an exponentially weighted averaging method.

Assume that an order $o_{\xi }$ for $\xi \in \{1,2,\dots ,|S\left|\right\}$ is obtained from a set of triplets $\{(o_{\xi }^{\mathrm{R}},o_{\xi }^{\mathrm{G}},o_{\xi }^{\mathrm{B}})\}$. Then, we define a fuzzy dilation by introducing an exponentially weighted averaging to (10) as follows:

$$D^{\mathrm{FUZ}}(F,S,\alpha )=\left[d_{ij}^{\mathrm{FUZ}}\right],d_{ij}^{\mathrm{FUZ}}=\frac{{\displaystyle \sum _{\xi =1}^{\left|S\right|}exp\left(\alpha o_{\xi }\right)f_{\xi }}}{{\displaystyle \sum _{\xi =1}^{\left|S\right|}exp\left(\alpha o_{\xi }\right)}},$$

(16)

where exp denotes the exponential function defined by $exp\left(x\right)=e^x$, where e is a constant called Euler’s number and $\alpha$ is a positive constant for controlling the fuzziness. Here, we note that the second equation in (16) includes the softmax function [23], which is the gradient of the log-sum-exp function [24]. Therefore, the equation can be rewritten as

$$d_{ij}^{\mathrm{FUZ}}=arg\underset{d}{min}\sum _{\xi =1}^{\left|S\right|}\frac{\partial }{\partial o_{\xi }}\left(log\sum _{\nu =1}^{\left|S\right|}exp\left(\alpha o_{\nu }\right)\right){∥f_{\xi }-d∥}^2,$$

(17)

which explicitly describes the relationship between the exponentially weighted averaging and the log-sum-exp function, where $\partial /\partial o_{\xi }$ denotes a partial derivative with respect to $o_{\xi }$ and $\parallel ·\parallel$ denotes the Euclidean norm. This expression accepts vectors $f_{\xi }$ and d of any dimension. Therefore, we can fuzzify morphological operations for not only color images but also binary and grayscale images by the proposed method.

Similarly, we can define a fuzzy erosion as follows:

$$E^{\mathrm{FUZ}}(F,S,\alpha )=\left[e_{ij}^{\mathrm{FUZ}}\right],e_{ij}^{\mathrm{FUZ}}=\frac{{\displaystyle \sum _{\xi =1}^{\left|S\right|}exp\left(-\alpha o_{\xi }\right)f_{\xi }}}{{\displaystyle \sum _{\xi =1}^{\left|S\right|}exp\left(-\alpha o_{\xi }\right)}}.$$

(18)

Combining these operations, we have fuzzy opening and closing operations as follows:

$$O^{\mathrm{FUZ}}(F,S,\alpha )=D^{\mathrm{FUZ}}\left(E^{\mathrm{FUZ}}\left(F,S,\alpha \right),S,\alpha \right),$$

(19)

$$C^{\mathrm{FUZ}}(F,S,\alpha )=E^{\mathrm{FUZ}}\left(D^{\mathrm{FUZ}}\left(F,S,\alpha \right),S,\alpha \right).$$

(20)

From these, we further define fuzzy open–closing and close–opening operations as follows:

$$O{C}^{\mathrm{FUZ}}(F,S,\alpha )=C^{\mathrm{FUZ}}\left(O^{\mathrm{FUZ}}\left(F,S,\alpha \right),S,\alpha \right),$$

(21)

$$C{O}^{\mathrm{FUZ}}(F,S,\alpha )=O^{\mathrm{FUZ}}\left(C^{\mathrm{FUZ}}\left(F,S,\alpha \right),S,\alpha \right).$$

(22)

The value of $\alpha$ is empirically determined in the following section.

5. Experimental Results

In this section, we show the experimental results of noise removal from color images by the proposed order space-based morphological operations, which are compared with the state-of-the-art method of Wang et al. [19]. Figure 6 shows a noisy input image including 10% impulse noise.

Figure 7 shows the output images by Wang’s hypergraph-based method [19] and the proposed method with ‘Cross’ and ‘Square’ structuring elements in Figure 1a and Figure 1b, respectively, where the order $o^S$ in (7) is used. Wang’s method has two parameters, $\mu$ and $\beta$, which are set as $\mu =250$ and $\beta =1$ by adjusting for this noise removal task through preliminary experiments. The top row in this figure shows the results of Wang’s method with a structural hypergraph denoted by ‘HG’ as an alternative to structuring elements for the dilation, erosion, opening, closing, open–closing and close–opening operations, the output images of which are arranged from left to right in columns, where we observe that the dilation Figure 7a and erosion Figure 7b tend to enlarge impulse noise, but the following operations Figure 7c–f can suppress the noise effectively. The middle and bottom rows in Figure 7 show the results of the proposed method with ‘Cross’ and ‘Square’ structuring elements shown in Figure 1a and Figure 1b, respectively. The $3\times 3$ ‘Square’ structuring element corresponds to the structuring hypergraph with $\beta =1$ in Wang’s method in terms of size.

Table 1. PSNR for output images in Figure 7 with ‘HG’, ‘Cross’ and ‘Square’ structuring elements (SEs). In each row, the highest value is boldfaced.

SE	Dilation	Erosion	Opening	Closing	Open-Closing	Close-Opening
‘HG’ [19]	15.31	16.22	20.11	20.52	20.88	21.59
‘Cross’	18.82	18.80	22.96	23.11	25.14	25.13
‘Square’	17.79	17.73	23.67	23.97	25.74	25.83

shows the values of the peak signal-to-noise ratio (PSNR) [25] between the output images in Figure 7 and the original noise-free image in Figure 4, where Wang’s method with the ‘HG’ structuring element achieves the highest PSNR 21.59 dB by the close–opening operation. On the other hand, the proposed method with ‘Cross’ and ‘Square’ structuring elements achieves 25.14 and 25.83 dB by open–closing and close–opening, respectively.

Figure 8 shows the noise removal results on the SIDBA image dataset [22], where the top row lists all 12 original color images, which are corrupted by impulse noise as shown in the second row, where the noise density increases from the left to right columns. We adopted open–closing and close–opening operations for removing the noise on the basis of the observation in Table 1, where those operations achieved higher PSNR values than the other morphological operations. In this figure, we compared the proposed method based on the ordering $o^S,o^P$ and $o^M$ in (7)–(9), computed with the $3\times 3$ ‘Square’ structuring element in Figure 1b with Wang’s method using the parameters $\mu =250$ and $\beta =1$.

The performance of noise removal is evaluated by the mean PSNR among the 12 different images as summarized in Figure 9, where Figure 9a,b show the results of open–closing and close–opening, respectively. In each graph, the vertical and horizontal axes denote the mean PSNR among the 12 images and the density of impulse noise varying from 10% to 60%, respectively, and the line colors, blue, orange, green and red correspond to Wang’s method and the proposed methods with $o^S,o^P$ and $o^M$, respectively. In both graphs, Wang’s method (blue line) obtains higher values of mean PSNR than the proposed methods with $o^P$ and $o^M$ (green and red lines), and the proposed method with $o^S$ (orange line) outperforms Wang’s method.

We also compared the performance of grayscale and color morphological operations as illustrated in Figure 10, where a color original image (top left box) is corrupted by Gaussian or impulse noise to produce a noisy image (top right box). Both images are converted to grayscale images through the outside clockwise and counterclockwise routes. Then, the standard method for grayscale images is applied to those grayscale images (bottom leftmost and rightmost boxes). On the other hand, color original and noisy images pass through inner routes, where Wang’s and our methods for color images are applied to them. Then, the resultant images are converted to grayscale images (bottom middle two boxes). Finally, we compute PSNR or the structural similarity index measure (SSIM) [26] for the resultant grayscale images processed by the same method. Figure 11 shows the results of Gaussian noise removal with the SIDBA dataset [22], where the vertical axes denote Figure 11a PSNR and Figure 11b SSIM, and the horizontal axes denote the compared methods. In this figure, the grayscale operation achieved the highest PSNR and SSIM values, and the proposed method outperformed Wang’s method.

On the other hand, Figure 12 shows the results of impulse noise removal, where the settings of axes are the same as Figure 11. In Figure 12a, the proposed method achieved the highest PSNR value, while in Figure 12b, the grayscale operation achieved the highest SSIM value. The reason why the grayscale operation achieves higher values than color ones is speculated to be that the conversion from color to gray has a noise-reduction effect. This speculation is investigated in

Table 2. Comparison of the quality of noisy images used in Figure 10 between color and gray. In each row, the highest value is boldfaced.

Noise	Quality Measure	R	G	B	Gray
Gaussian	PSNR	18.90	19.06	19.03	22.71
Gaussian	SSIM	0.306	0.350	0.315	0.417
Impulse	PSNR	12.16	11.94	12.01	16.63
Impulse	SSIM	0.118	0.144	0.122	0.183

, where color noisy images are compared with grayscale noisy images by PSNR and SSIM for Gaussian and impulse noises. The values of the image quality measure for R, G and B components are improved by grayscale conversion for all combinations of noises and measures. This result shows that the grayscale conversion has a noise-reduction effect.

In terms of the color morphological operations in this experiment, the proposed method achieved better PSNR and SSIM values than Wang’s method.

For impulse noise removal, it is well known that simple marginal processing based on marginal ordering works well, although it can generate false colors. Thus, we compared the proposed method with the marginal open–closing and Max-tree-based area open–closing proposed by Tushabe and Wilkinson [15], where we adopted an area threshold of $T=150$ and a nearest color (NC) approach for image restoration as recommended in their paper. Figure 13 shows the results of noise removal, where 12 color images in the SIDBA dataset [22] were corrupted by 50% impulse noise, and we applied the three open–closing operations to those noisy images. The performance of noise removal was evaluated with PSNR and SSIM, which are shown in Figure 13a and Figure 13b, respectively. In Figure 13a, the proposed method achieved the highest PSNR value among the three methods. On the other hand, in Figure 13b, the simple marginal method achieved the highest SSIM value, which matches the above knowledge that the marginal method removes impulse noise well.

Figure 14 shows the output images computed by the three methods, where the image in Figure 14a given by the simple marginal open–closing operation appears to be the best among three output images. However, the image in Figure 14a includes new colors that do not exist in the input image. This violates a desirable property that color morphological operations should have, i.e., no new colors should be introduced [27].

Next, we show the experimental results for the fuzzy morphological operations described in Section 4. Here, we would like to determine the value of $\alpha$ in the fuzzy operations with the open–closing operation, which outperforms the close–opening operation as shown in Figure 9; that is, the orange line in Figure 9a exceeds that in Figure 9b. Figure 15 shows the PSNR values for different values of $\alpha$ varying from 0.1 to 1.0, where we applied the fuzzy open–closing in (21) to the noisy image in Figure 6. We observe that $\alpha =0.5$ achieves the highest PSNR value, and use $\alpha =0.5$ below.

Figure 16 shows the results of the proposed fuzzy morphological operations defined in (16)–(22). A visual comparison between Figure 16 and Figure 7 demonstrates the effectiveness of the proposed fuzzification, and Figure 17 shows the effectiveness quantitatively by comparing the mean PSNR values computed with the SIDBA dataset [22] between crisp and fuzzy operations, where the vertical and horizontal axes denote the mean PSNR and noise density as well as the settings from Figure 9, and Figure 17a,b which show the results of open–closing and close–opening operations, respectively. For both operations, the fuzzy ones denoted by purple lines outperform the crisp ones denoted by orange lines.

6. Discussion

In the above experimental results, we demonstrated the effectiveness of the proposed order space-based morphological operations, which are compared with Wang’s method based on hypergraph theory [19]. Generally, color images have three color channels, e.g., red, green and blue channels for RGB color images, and we can sort the pixel values in each channel separately. Therefore, it would be appropriate to consider three orders derived from the three channels in morphological operations for color images. The proposed method is a method by which the obtained three orders, which form a 3D order space, can be unified into a single order of pixels in a suitable manner for a given application. In this paper, we presented three ways of unifying three orders corresponding to three color channels into a single order, i.e., $o^S,o^P$ and $o^M$ defined in (7), (8) and (9), respectively. In the application of morphological operations to noise removal demonstrated in Section 5, Wang’s method [19] achieved better performance than the proposed method with $o^P$ and $o^M$, but the remaining option $o^S$ outperformed Wang’s method, which shows the potential of the proposed order space-based method. The ways of unifying three orders are not restricted to the three methods presented in this paper. Investigating better ways to unify plural orders into one will be a promising future research direction.

We also extended the proposed order space-based morphological operations to their fuzzy version, where we proposed a method for fuzzifying morphological operations by exponentially weighted averaging. Conventional morphological operations for binary and grayscale images include maximum and minimum selection. The max function for selecting the maximum value from a set of scalar values can be approximated by the log-sum-exp function [24], which has a relationship with fuzzy techniques [28]. However, it is difficult to use the log-sum-exp function directly for fuzzifying morphological operations for color images because of the dimensional discrepancy between scalar and vector data. On the other hand, the proposed exponentially weighted averaging method can be used for both scalar and vector data. Therefore, we can also fuzzify conventional morphological operations for binary and grayscale images by the proposed method. In the above experiments, we demonstrated the effectiveness of the fuzzification of the order space-based morphological operations in the noise-removal application. The applicability of the proposed fuzzification method to any dimensional data will expand its application areas into the area of multidimensional data processing.

7. Conclusions

In this paper, we proposed a method for applying morphological operations to color images. The proposed method is based on a three-dimensional (3D) order space, which has three coordinate axes corresponding to the orders of pixels in three color channels, e.g., R (red), G (green) and B (blue) color channels for RGB color images. That is, a color pixel is represented by a point on the 3D space specified by the set of orders of pixels sorted in R, G and B color channels, separately. Then, the triplet was shrunk to a single order by an appropriate function, e.g., sum, product and median functions were investigated in this paper. We conducted experiments on noise removal with a standard image dataset, and demonstrated that the sum function achieved the best performance among the three functions and a state-of-the-art method based on hypergraph theory by Wang et al. [19]. We can consider other options for the shrinkage functions and the structuring elements in morphological operations that may further improve the performance.

We also proposed an exponentially weighted averaging method for fuzzifying the order space-based morphological operations. As a result, the noise removal performance was further improved by the fuzzification for both open–closing and close–opening operations, both of which achieved better performance than dilation, erosion, opening and closing operations in a conventional crisp situation.

Our future work will include the extension of the proposed method for color image processing to multidimensional data processing, which will also be fuzzified by the proposed exponentially weighted averaging method to improve the performance.