# An Image Retrieval Framework Design Analysis Using Saliency Structure and Color Difference Histogram

^{1}

^{2}

^{3}

^{4}

^{5}

^{6}

^{7}

^{*}

## Abstract

**:**

## 1. Introduction

- An innovative model is proposed for the retrieval of plant images in which a hybrid combination of color difference histogram (CDH) and saliency structure histogram (SSH) descriptors are used to design a detailed feature map.
- The CDH descriptor uses the hue and saturation values of a leaf image to compute the color features.
- The SSH descriptors use first-order statistical features (FOSF) of a leaf image to compute the shape features.
- A new feature map is generated by concatenating features extracted from CDH and SSH descriptors to enhance the image retrieval accuracy.
- Euclidean distance approach is used as a distance metric for measuring the similarity index between the query image and the database images.
- The performance of the proposed model is evaluated using precision, recall, and F-measure.

## 2. Related Work

## 3. Proposed Model for Leaf Image Retrieval

#### 3.1. Loading Dataset Images and Query Image

#### 3.2. Image Resizing

#### 3.3. Features Extraction Using CDH Descriptor

#### 3.3.1. Image Conversion from RGB to L*a*b* Color Space

_{n}, y

_{n}and z

_{n}are the values of x, y and z for illuminant (reference white point) and $\left[\begin{array}{ccc}{\mathrm{x}}_{\mathrm{n}}& {\mathrm{y}}_{\mathrm{n}}& {\mathrm{z}}_{\mathrm{n}}\end{array}\right]$=$\left[\begin{array}{ccc}0.950450& 1.000000& 1.088754\end{array}\right]$ [1,25].

#### 3.3.2. Edge Detection Using Sobel Operator

_{xx}) and vertical (g

_{yy}) edges in every pixel, then summing the computed dot product of g

_{xx}and g

_{yy}with the color value, creating g

_{xy}. Sobel operator is preferred over other methods as it has a high-quality result on edge detection and also has the property to detect with very fast speed. Moreover, the gradient vector is composed of both magnitude and direction components. Sobel operator can be expressed using the convolution template using Equations (6) and (7).

_{xx}.* Z, Fx = g

_{yy}.* Z provides the size of the gradient as shown by Equation (9).

#### 3.3.3. Quantization of L*a*b* Color Space

#### 3.3.4. HSV Feature Map

#### 3.4. Features Extraction Using SSH Descriptor

#### 3.5. Hybrid Feature Map Using Concatenation of HSV Feature Map and FOSF Feature Map

#### 3.6. Computation of Euclidean Distance between Dataset Image and Query Image Features

_{1}, b

_{1}) and n = (a

_{2}, b

_{2}), then the Euclidean distance between these points is calculated as described by Equation (14).

#### 3.7. Sorting of Distances and Retrieving Images Using Thresholding

## 4. Model Evaluation

#### 4.1. Performance Parameters

- Precision (P
_{r}) is the ratio of the relevant images recovered to the total set of images extracted as stated by Equation (15).

- Recall (R
_{e}) is computed by dividing the number of relevant images extracted by the total number of images in that dataset. The equation used to compute this parameter is represented by Equation (16).

- F-measure (F
_{m}) is a combining measure of precision and recall stated by Equation (17).

#### 4.2. Image Retrieval Results Using Only CDH Descriptor

#### 4.3. Image Retrieval Results Using Only SSSH Descriptor

#### 4.4. Image Retrieval Results Using Combination of CDH and SSH Descriptor

#### 4.5. Comparative Analysis of the Proposed Model with State-of-the-Art Techniques

## 5. Conclusions and Future Scope

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Patil, J.K.; Kumar, R. Plant Leaf Disease Image Retrieval Using Color Moments. IAES Int. J. Artif. Intell. (IJ-AI)
**2013**, 2, 36–42. [Google Scholar] [CrossRef] - Patil, J.; Kumar, R. Analysis of content based image retrieval for plant leaf diseases using color, shape and texture features. Eng. Agric. Environ. Food
**2017**, 10, 69–78. [Google Scholar] [CrossRef] - Bama, B.S.; Valli, S.M.; Raju, S.; Kumar, V.A. Content based leaf image retrieval (CBLIR) using shape, color and texture features. Indian J. Comput. Sci. Eng.
**2011**, 2, 202–211. [Google Scholar] - Liu, G.-H.; Yang, J.-Y. Content-based image retrieval using color difference histogram. Pattern Recognit.
**2013**, 46, 188–198. [Google Scholar] [CrossRef] - Smeulders, A.W.; Worring, M.; Santini, S.; Gupta, A.; Jain, R. Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell.
**2000**, 22, 1349–1380. [Google Scholar] - Ahmad, J.; Sajjad, M.; Mehmood, I.; Baik, S.W. SSH: Salient structures histogram for content based image retrieval. In Proceedings of the 2015 18th International Conference on Network-Based Information Systems, Taipei, Taiwan, 2–4 September 2015; pp. 212–217. [Google Scholar]
- Pahalawatta, K.K. A Plant Identification System Using both Global and Local Features of Plant Leaves. Doctoral Dissertation, Department of Computer Science and Software Engineering, University of Canterbury, Christchurch, New Zealand, 2008. [Google Scholar]
- Nam, Y.; Hwang, E.; Kim, D. A similarity-based leaf image retrieval scheme: Joining shape and venation features. Comput. Vis. Image Underst.
**2008**, 110, 245–259. [Google Scholar] [CrossRef] - Chugh, H.; Gupta, S.; Garg, M. Image Retrieval System-An Integrated Approach. IOP Conf. Ser. Mater. Sci. Eng.
**2021**, 1022, 012040. [Google Scholar] - Wang, Z.; Chi, Z.; Feng, D.; Wang, Q. Leaf image retrieval with shape features. In International Conference on Advances in Visual Information Systems; Springer: Berlin/Heidelberg, Germany, 2000; pp. 477–487. [Google Scholar]
- Mokhtarian, F.; Abbasi, S. Matching Shapes With Self-Intersections: Application to Leaf Classification. IEEE Trans. Image Process.
**2004**, 13, 653–661. [Google Scholar] [CrossRef] - Soille, P. Morphological image analysis applied to crop field mapping. Image Vis. Comput.
**2000**, 18, 1025–1032. [Google Scholar] [CrossRef] - Fu, H.; Chi, Z. A two-stage approach for leaf vein extraction. In Proceedings of the International Conference on Neural Networks and Signal Processing, Istanbul, Turkey, 14–17 December 2003; Volume 14, pp. 208–211. [Google Scholar]
- Huang, J.; Kumar, S.R.; Mitra, M.; Zhu, W.J.; Zabih, R. Image indexing using color correlograms. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, Puerto Rico, 17–19 June 1997; pp. 762–768. [Google Scholar]
- Manjunath, B.S.; Ohm, J.R.; Vasudevan, V.V.; Yamada, A. Color and texture descriptors. IEEE Trans. Circuits Syst. Video Technol.
**2001**, 11, 703–715. [Google Scholar] - Manjunath, B.S.; Salembier, P.; Sikora, T. (Eds.) Introduction to MPEG-7: Multimedia Content Description Interface; John Wiley & Sons: Hoboken, NJ, USA, 2002. [Google Scholar]
- Ahmad, J.; Jan, Z.; Khan, S.M. A fusion of labeled-grid shape descriptors with weighted ranking algorithm for shapes recognition. arXiv
**2014**, arXiv:1406.3949. [Google Scholar] - Yang, M.; Kidiyo, K.; Joseph, R. A survey of shape feature extraction techniques. Pattern Recognit.
**2008**, 15, 43–90. [Google Scholar] - Achard, C.; Devars, J.; Lecassagne, L. Object image retrieval with image compactness vectors. In Proceedings of the 15th International Conference on Pattern Recognition, ICPR-2000, Barcelona, Spain, 3–7 September 2000; Volume 4, pp. 271–274. [Google Scholar]
- Narayanan, S.S.; Vinod, D.; Athisayamani, S.; Robert Singh, A. Combination of Local Feature Extraction for Image Retrieval. In Proceedings of Third International Conference on Sustainable Computing; Springer: Singapore, 2022; pp. 319–328. [Google Scholar]
- Chugh, H.; Gupta, S.; Garg, M.; Gupta, D.; Juneja, S.; Turabieh, H.; Na, Y.; Bitsue, Z.K. Image Retrieval Using Different Distance Methods and Color Difference Histogram Descriptor for Human Healthcare. J. Health Eng.
**2022**, 2022, 9523009. [Google Scholar] [CrossRef] - Ghrabat, M.J.J.; Ma, G.; Maolood, I.Y.; Alresheedi, S.S.; Abduljabbar, Z.A. An effective image retrieval based on optimized genetic algorithm utilized a novel SVM-based convolutional neural network classifier. Hum.-Cent. Comput. Inf. Sci.
**2019**, 9, 31. [Google Scholar] [CrossRef] - Li, W.; Duan, L.; Xu, D.; Tsang, I.W.-H. Text-based image retrieval using progressive multi-instance learning. In Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011; pp. 2049–2055. [Google Scholar] [CrossRef]
- Minarno, A.E.; Suciati, N. Batik image retrieval based on color difference histogram and gray level co-occurrence matrix. Telecommun. Comput. Electron. Control
**2014**, 12, 597–604. [Google Scholar] - Gonzales, R.C.; Woods, R.E. Digital Image Processing, 3rd ed.; Prentice Hall: London, UK, 2007; pp. 424–447. [Google Scholar]
- Burger, W.; Burge, M.J. Principles of Digital Image Processing: Core Algorithms; Springer: Berlin/Heidelberg, Germany, 2009. [Google Scholar]
- Mehta, K.; Kumar, Y. Implementation of efficient clock synchronization using elastic timer technique in IOT. Adv. Math. Sci. J.
**2020**, 9, 4025–4030. [Google Scholar] [CrossRef] - Caballero, C.; Aranda, M.C. Plant species identification using leaf image retrieval. In Proceedings of the ACM International Conference on Image and Video Retrieval, Xi’an, China, 5–7 July 2010; pp. 327–334. [Google Scholar]

**Figure 4.**(

**a**) L*a*b* Image Converted from RGB Image (

**b**) Edge Detected Image Converted from L*a*b* Image using Sobel Operator.

**Figure 8.**Retrieval Results for Top 25 Images Retrieved using Combination of CDH and SSH Descriptor.

**Figure 9.**Performance Chart for F-Measure at (

**a**) 40% Threshold (

**b**) 75% Threshold (

**c**) 80% Threshold (

**d**) 85% Threshold.

Features | Mathematical Expression | Values |
---|---|---|

Mean | $\mathrm{M}=\overline{\mathrm{x}}=\frac{\sum {\mathrm{i}\mathrm{x}}_{\mathrm{i}}}{\mathrm{n}}$ | 67.192 |

Variance | $\mathrm{V}=\frac{\sum {\left({\mathrm{x}}_{\mathrm{i}}-\overline{\mathrm{x}}\right)}^{2}}{\mathrm{n}-1}$ | 3.56 × 10^{3} |

Standard Deviation | $\mathrm{s}\mathrm{d}=\mathrm{s}=\sqrt{\frac{\sum {\left({\mathrm{x}}_{\mathrm{i}}-\overline{\mathrm{x}}\right)}^{2}}{\mathrm{n}-1}}$ | 59.696 |

Skewness | $\mathrm{S}\mathrm{K}=\frac{\sum \mathrm{i}{\left({\mathrm{x}}_{\mathrm{i}}-\overline{\mathrm{x}}\right)}^{3}}{\mathrm{n}\times {\tilde{\mathrm{s}}}^{3}}$ $\text{where}\tilde{\mathrm{s}}=\mathrm{s}\times \sqrt{\left(\mathrm{n}-1\right)/\mathrm{n}}$ | 1.107 |

Kurtosis | $\mathrm{K}=\frac{\sum \mathrm{i}{\left({\mathrm{x}}_{\mathrm{i}}-\overline{\mathrm{x}}\right)}^{4}}{\mathrm{n}\times {\tilde{\mathrm{s}}}^{4}}$ | 3.536 |

Smoothness | $\mathrm{S}=1-\left(\frac{1}{1+{\mathrm{s}}^{2}}\right)$ | 0.9997 |

Uniformity | $\mathrm{U}=\sum {\left(\frac{\text{histogram of image}}{\text{number of elements}}\right)}^{2}$ | 0.0087 |

Entropy | $\mathrm{E}=-{\displaystyle {\displaystyle \sum}_{\mathrm{x}=0}^{\mathrm{m}-1}}{\displaystyle {\displaystyle \sum}_{\mathrm{y}=0}^{\mathrm{n}-1}}\mathrm{f}\left(\mathrm{x},\mathrm{y}\right){\mathrm{log}}_{2}\mathrm{f}\left(\mathrm{x},\mathrm{y}\right)$ | 7.326 |

Image Index | 1 | 2 | 3 | 4 | 5 |
---|---|---|---|---|---|

Retrieved Image | |||||

Computed Distance | 0 | 2.973 | 4.06 | 4.1954 | 4.2329 |

Image Index | 6 | 7 | 8 | 9 | 10 |

Retrieved Image | |||||

Computed Distance | 4.3112 | 4.3843 | 4.4677 | 4.7242 | 4.9537 |

Threshold Value | Retrieved Images | True Positive | Actual Images | Precision | Recall | F-Measure |
---|---|---|---|---|---|---|

40% | 4 | 2 | 25 | 0.5 | 0.08 | 0.137931 |

75% | 39 | 20 | 25 | 0.444444 | 0.8 | 0.571429 |

80% | 45 | 20 | 25 | 0.512821 | 0.8 | 0.625 |

85% | 62 | 19 | 25 | 0.306452 | 0.76 | 0.43678 |

Type of Leaf Image | Retrieved Images | True Positive | Actual Images | Precision | Recall | F-Measure |
---|---|---|---|---|---|---|

Elm | 45 | 20 | 25 | 0.444444 | 0.8 | 0.571429 |

Maple | 38 | 20 | 25 | 0.526316 | 0.8 | 0.634921 |

Coleus | 38 | 22 | 25 | 0.578947 | 0.88 | 0.698413 |

Croton | 46 | 22 | 25 | 0.478261 | 0.88 | 0.619718 |

Type of Leaf Image | Retrieved Images | True Positive | Actual Images | Precision | Recall | F-Measure |
---|---|---|---|---|---|---|

Elm | 97 | 25 | 25 | 0.257732 | 1 | 0.409836 |

Maple | 97 | 25 | 25 | 0.257732 | 1 | 0.409836 |

Coleus | 81 | 25 | 25 | 0.308642 | 1 | 0.471698 |

Croton | 64 | 19 | 25 | 0.296875 | 0.76 | 0.426966 |

**Table 6.**Performance Parameters of Image Retrieval using Combination of CDH & SSH Descriptor at 80% Threshold.

Type of Leaf Image | Retrieved Images | True Positive | Actual Images | Precision | Recall | F-Measure |
---|---|---|---|---|---|---|

Elm | 24 | 24 | 25 | 1 | 0.96 | 0.979592 |

Maple | 15 | 12 | 25 | 0.8 | 0.48 | 0.6 |

Coleus | 26 | 22 | 25 | 0.846154 | 0.88 | 0.862745 |

Croton | 26 | 22 | 25 | 0.846154 | 0.88 | 0.862745 |

Sr. No. | Author(s) | Technique Used | Features Used | Recall |
---|---|---|---|---|

1 | Yunyoung Nam et al. [8] | An adaptive grid-based matching algorithm | Shape & Venation features | 0.90 |

2 | Carlos et al. [28] | Contour descriptor | Shape features | 0.81 |

3 | B.Sathya Bama et al. [3] | Log-Gabor wavelet | Texture features | 0.62 |

Scale Invariant Feature Transform (SIFT) | 0.60 | |||

4 | Proposed | CDH descriptor | Color features | 0.8 |

SSH descriptor | Shape Features | 1.00 | ||

Combination of CDH & SSH descriptor | Color & shape features | 0.96 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Chugh, H.; Gupta, S.; Garg, M.; Gupta, D.; Mohamed, H.G.; Noya, I.D.; Singh, A.; Goyal, N.
An Image Retrieval Framework Design Analysis Using Saliency Structure and Color Difference Histogram. *Sustainability* **2022**, *14*, 10357.
https://doi.org/10.3390/su141610357

**AMA Style**

Chugh H, Gupta S, Garg M, Gupta D, Mohamed HG, Noya ID, Singh A, Goyal N.
An Image Retrieval Framework Design Analysis Using Saliency Structure and Color Difference Histogram. *Sustainability*. 2022; 14(16):10357.
https://doi.org/10.3390/su141610357

**Chicago/Turabian Style**

Chugh, Himani, Sheifali Gupta, Meenu Garg, Deepali Gupta, Heba G. Mohamed, Irene Delgado Noya, Aman Singh, and Nitin Goyal.
2022. "An Image Retrieval Framework Design Analysis Using Saliency Structure and Color Difference Histogram" *Sustainability* 14, no. 16: 10357.
https://doi.org/10.3390/su141610357