# Harvest Stage Recognition and Potential Fruit Damage Indicator for Berries Based on Hidden Markov Models and the Viterbi Algorithm

^{1}

^{2}

^{3}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

## 2. Theoretical Background

#### 2.1. Markov Chains

- ${a}_{ij}\ge 0$
- ${\sum}_{j=1}^{N}{a}_{ij}=1$.

#### 2.2. Hidden Markov Models (HMMs)

- $\mathbf{N}$: Number of states. The set of possible states can be denoted by $S=\{{s}_{1},\cdots ,{S}_{N}\}$. The state of the system at time t is denoted ${q}_{t}$.
- $\mathbf{M}$: Number of measurements associated with each state. Each measurement corresponds to a physical outcome from the system that can be acquired using the appropriate sensors.
- The transition probability distribution between system states $A=\left\{{a}_{ij}\right\}$ where:$$P[{q}_{t}={S}_{j}|{q}_{t-1}={S}_{i}]={a}_{ij},\phantom{\rule{0.277778em}{0ex}}\phantom{\rule{1.em}{0ex}}1\le i,j\le N$$
- The measurement probability distribution conditional of the state j, $B=\left\{{b}_{j}\left(k\right)\right\}$:$$\begin{array}{cc}\hfill {b}_{j}\left(k\right)=P\left[{O}_{k}\right|{q}_{t}={S}_{j}],& 1\le j\le N\hfill \\ & 1\le k\le M\hfill \end{array}$$
- The initial probability distribution of system states $\pi $, where:$${\pi}_{i}=P[{q}_{1}={S}_{i}],\phantom{\rule{1.em}{0ex}}1\le i,j\le N$$

#### 2.3. Viterbi Algorithm

## 3. Materials and Methods

#### 3.1. The Blueberry Harvesting Process

#### 3.2. A Modular Distributed Monitoring System for the Harvesting Process: The “Smartbin”

#### 3.3. Data Acquisition Campaign

- $Temp\_1$: Temperature measurement acquired every 15 s using a sensor that is located near the bottom of the bin.
- $Temp\_2$: Temperature measurement acquired every 15 s using a sensor that is located near one of the four the external edges of the bin.
- $Ac{c}_{x}$: Acceleration measurement in x-axis acquired ten times per second with an IMU located inside the bin.
- $Ac{c}_{y}$: Acceleration measurement in y-axis acquired ten times per second with an IMU located inside the bin.
- $Ac{c}_{z}$: Acceleration measurement in z-axis acquired ten times per second with an IMU located inside the bin.
- $Weight$: Net weight of the “smartbin” acquired every 15 s with a sensor that is located at the bottom of the bin.

#### 3.4. Proposed Methodology for Online Harvesting Stage Detection

- (1)
- “Picking”($S1$): The pickers, provided with a 3.5 L plastic box hung around the neck by a harness, cover the orchard prepared in rows approximately 100 m long. Picking lasts 20 to 40 min per box, depending on the picker’s experience and the volume of fruit on the shrubs. During this stage it is possible to measure high energy vibration signals and high temperatures.
- (2)
- “Wait” ($S2$): When the box is full, the picker goes to the storage center (shaded area), where he/she delivers the box for counting. The full boxes remain at the warehouse waiting for the tractor–trailer to take them to the local packing area.
- (3)
- “Transport” (full bin) ($S3$): The tractor–trailer transports full boxes from the warehouse to the local packing area.
- (4)
- “Cooling” (freezer tunnel) ($S4$): The fruit is admitted to packing via a conveyor table, where a cooling system lowers its temperature using a freezing tunnel.

- (1)
- Inertial measurement unit (IMU): Data acquired by the IMU. A simple pre-processing algorithm is implemented to complement this information with an average of the total energy in the vibration signal every 15 s over the time window containing the last 15 s of measurements.$$IMU\_Energ{y}_{t}=\sum _{j=t-14}^{t}ac{c}_{x}{\left(j\right)}^{2}+ac{c}_{y}{\left(j\right)}^{2}+ac{c}_{z}{\left(j\right)}^{2}$$
- (2)
- Temperature measurements: Besides the information provided by sensors $Temp\_1$ and $Temp\_2$, a simple pre-processing algorithm is implemented to measure the difference in readings between both temperature sensors.$$Delta\_T\left(t\right)=Temp\_2\left(t\right)-Temp\_1\left(t\right)$$

#### 3.5. Proposed Methodology for Fruit Damage Indicator

## 4. Obtained Results in Experimental Campaign

## 5. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## Appendix A

Algorithm 1 Viterbi Algorithm ($\lambda =(A,B,\pi ),O$) | |

Inputs:$\lambda =(A,B,\pi ),O$ | |

Output:${Q}^{*}=\{{q}_{1}^{*},{q}_{2}^{*},\cdots ,{q}_{T}^{*}\}$ | |

1: for $i=1,\cdots ,N$do | ▹ Initialization |

2: ${\delta}_{1}\left(i\right)={\pi}_{i}{b}_{i}\left({O}_{1}\right)$ | |

3: ${\psi}_{1}\left(i\right)=0$ | |

4: for $j=1,\cdots ,N$, $t=2,\cdots ,T$ do | ▹ Recursion |

5: ${\delta}_{t}\left(j\right)=\underset{1\le i\le N}{max}\left[{\delta}_{t-1}\left(i\right){a}_{ij}\right]\xb7{B}_{j}\left({O}_{t}\right)$ | |

6: ${\psi}_{t}\left(i\right)=\underset{1\le i\le N}{argmax}\left[{\delta}_{t-1}\left(i\right){a}_{ij}\right]$ | |

7: ${P}^{*}=\underset{1\le i\le N}{max}\left[{\delta}_{T}\left(i\right)\right]$ | |

8: ${q}_{T}^{*}=\underset{1\le i\le N}{argmax}\left[{\delta}_{T}\left(i\right)\right]$ | |

9: for $t=T-1,T-2,\cdots ,1$ do | ▹ Reconstruction of state sequence |

10: ${q}_{t}^{*}={\psi}_{t+1}\left({q}_{t+1}^{*}\right)$ | |

11: return ${Q}^{*}$ |

## References

- Muñoz, M. Boletin de fruta fresca. Octubre de 2018. Available online: https://www.odepa.gob.cl/contenidos-rubro/boletines-del-rubro/boletin-de-fruta-fresca-octubre-de-2018 (accessed on 2 October 2019).
- Li, Z.; Thomas, C. Quantitative Evaluation of Mechanical Damage to Fresh Fruits. Trends Food Sci. Technol.
**1991**, 32, 138–150. [Google Scholar] [CrossRef] - Aliasgarian, S.; Ghassemzadeh, H.; Moghaddam, M.; Ghaffari, H. Mechanical Damage of Strawberry during Harvest and Postharvest Operations. World Appl. Sci. J.
**2013**, 22, 969–974. [Google Scholar] [CrossRef] - Vergano, P.; Testin, R.; Newall, C. Distinguishing among Bruises in Peaches Caused by Impact, Vibration, and Compression. J. Food Qual.
**1991**, 14, 285–298. [Google Scholar] [CrossRef] - Lin, X.; Brusewitz, G. Peach Bruise Thresholds Using the Instrumented Sphere. Appl. Eng. Agric.
**1994**, 10, 509–513. [Google Scholar] [CrossRef] - Sanford, K.; Lidster, P.; McRae, K.; Jackson, E.; Lawrence, R.; Stark, R.; Prange, R. Lowbush Blueberry Quality Changes in Response to Mechanical Damage and Storage Temperature. J. Am. Soc. Hortic. Sci.
**1991**, 116, 47–51. [Google Scholar] [CrossRef][Green Version] - Martínez-Romero, D.; Serrano, M.; Carbonell, A.; Castillo, S.; Riquelme, F.; Valero, D. Production Practices and Quality Assessment of Food Crops; Kluwer Academic Publishers: Dordrecht, The Netherlands, 2004; pp. 233–252. [Google Scholar]
- Gallardo, R.K.; Stafne, E.T.; DeVetter, L.W.; Zhang, Q.; Li, C.; Takeda, F.; Williamson, J.; Yang, W.Q.; Cline, W.O.; Beaudry, R.; et al. Blueberry Producers’ Attitudes toward Harvest Mechanization for Fresh Market. Horttechnology
**2018**, 28, 10–16. [Google Scholar] [CrossRef] - Moggia, C.; Graell, J.; Lara, I.; González, G.; Lobos, G.A. Firmness at Harvest Impacts Postharvest Fruit Softening and Internal Browning Development in Mechanically Damaged and Non-damaged Highbush Blueberries (Vaccinium corymbosum L.). Front. Plant Sci.
**2017**, 8, 1–11. [Google Scholar] [CrossRef] [PubMed] - Timm, J.; Bollen, A.; Rue, B.D.; Woodhead, I. Apple Damage and Compressive Forces in Bulk Bins During Orchard Transport. Appl. Eng. Agric.
**1998**, 14, 165–172. [Google Scholar] [CrossRef] - Swedberg, C. Driscoll’s Monitors Its Berry Shipments in Real Time. Available online: https://www.rfidjournal.com/articles/view?8011/3 (accessed on 2 October 2019).
- Ampatzidis, Y.G.; Vougioukas, S.G.; Whiting, M.D. A wearable module for recording worker position in orchards. Comput. Electron. Agric.
**2011**, 78, 222–230. [Google Scholar] [CrossRef] - Ampatzidis, Y.; Whiting, M.; Scharf, P.; Zhang, Q. Development and evaluation of a novel system for monitoring harvest labor efficiency. Comput. Electron. Agric.
**2012**, 88, 85–94. [Google Scholar] [CrossRef] - Galeas, P.; Muñoz, C.; Huircan, J.; Fernandez, M.; Segura-Ponce, L.A.; Duran-Faundez, C. Smartbins: Using Intelligent Harvest Baskets to Estimate the Stages of Berry Harvesting. Sensors
**2019**, 19, 1361. [Google Scholar] [CrossRef] [PubMed] - Rabiner, L.R. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE
**1989**, 77, 257–286. [Google Scholar] [CrossRef] - Viterbi, A. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Trans. Inf. Theory
**1967**, 13, 260–269. [Google Scholar] [CrossRef] - Forney, G.D. The viterbi algorithm. Proc. IEEE
**1973**, 61, 268–278. [Google Scholar] [CrossRef] - Ryan, M.; Nudd, G. The Viterbi Algorithm; Springer Science: New York, NY, USA, 2009. [Google Scholar] [CrossRef]
- Jackson, E.; Sanford, K.; Lawrence, R.; McRae, K.; Stark, R. Lowbush blueberry quality changes in response to prepacking delays and holding temperatures. Postharvest Biol. Technol.
**1999**, 15, 117–126. [Google Scholar] [CrossRef] - He, Y.; Wang, J.; Wang, X. The fruit quality change of less mature blueberry in different postharvest storing temperatures. IOP Conf. Ser. Earth Environ. Sci.
**2019**, 237, 8–12. [Google Scholar] [CrossRef] - Nunez-barrios, A.; Nesmith, D.S.; Chinnan, M.; Prussia, S.E. Dynamics of Rabbiteye Blueberry Fruit Quality in Response to Harvest Method Dynamics of Rabbiteye Blueberry Fruit Quality in Response to Harvest Method and Postharvest Handling Temperature. Small Fruits Rev.
**2005**, 4, 73–81. [Google Scholar] [CrossRef]

**Figure 2.**Blueberry orchard in Yungay, Chile. (

**a**) Top view showing storage centers and packing house. (

**b**) Land view from storage center 6.

**Figure 3.**Basket conditioned to on line measure of weight, two temperatures and accelerations; (

**a**) basket view, (

**b**) schematic front view and (

**c**) schematic lateral view.

Harvesting Cycle (${\mathit{N}}_{\mathit{i}}{\mathit{c}}_{\mathit{j}}$: ${\mathit{bin}}_{\mathit{i}}$, ${\mathit{cycle}}_{\mathit{j}}$) | Damage Index |
---|---|

${N}_{5}{c}_{1}$ | 1.4762 |

${N}_{2}{c}_{1}$ | 1.3710 |

${N}_{1}{c}_{1}$ | 1.3386 |

${N}_{4}{c}_{2}$ | 1.3017 |

${N}_{3}{c}_{2}$ | 1.2905 |

${N}_{2}{c}_{2}$ | 1.1239 |

${N}_{1}{c}_{2}$ | 1.1119 |

${N}_{5}{c}_{2}$ | 1.0218 |

${N}_{4}{c}_{1}$ | 0.8002 |

${N}_{3}{c}_{1}$ | 0.7484 |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Orchard, M.; Muñoz-Poblete, C.; Huircan, J.I.; Galeas, P.; Rozas, H. Harvest Stage Recognition and Potential Fruit Damage Indicator for Berries Based on Hidden Markov Models and the Viterbi Algorithm. *Sensors* **2019**, *19*, 4421.
https://doi.org/10.3390/s19204421

**AMA Style**

Orchard M, Muñoz-Poblete C, Huircan JI, Galeas P, Rozas H. Harvest Stage Recognition and Potential Fruit Damage Indicator for Berries Based on Hidden Markov Models and the Viterbi Algorithm. *Sensors*. 2019; 19(20):4421.
https://doi.org/10.3390/s19204421

**Chicago/Turabian Style**

Orchard, Marcos, Carlos Muñoz-Poblete, Juan Ignacio Huircan, Patricio Galeas, and Heraldo Rozas. 2019. "Harvest Stage Recognition and Potential Fruit Damage Indicator for Berries Based on Hidden Markov Models and the Viterbi Algorithm" *Sensors* 19, no. 20: 4421.
https://doi.org/10.3390/s19204421