Int. J. Environ. Res. Public Health 2007, 4(1), 34-38; doi:10.3390/ijerph2007010006

Articles
System of Indexes and Indicators for the Quality Evaluation of HACCP Plans based on the Results of the Official Controls conducted by the Servizio di Igiene degli Alimenti della Nutrizione (Food and Nutrition Health Service) of the Local Health Authority of Foggia, Italy
Michele F. Panunzio *, Antonietta Antoniciello and Alessandra Pisano
Food and Nutrition Health Services, Local Health Authority, Foggia, Italy
*
Correspondence to Dr. Michele F. Panunzio. E-mail: mpanunz@tiscali.it
Received: 31 January 2006 / Accepted: 23 January 2007 / Published: 31 March 2007

Abstract

: Within the realm of evaluating self-monitoring plans, developed based on the Hazard Analysis and Critical Control Points (HACCP) method and adopted by food companies, little research has been done concerning the quality of the plans. The Servizio di Igiene degli Alimenti e della Nutrizione (Food and Nutrition Health Service) of the Local Health Authority of Foggia, Italy, has conducted research with the aim to adopt a system of indexes and indicators for the qualitative evaluation of HACCP plans. The critical areas considered were the following: simplicity, specificity, feasibility and adherence. During the period from January 2004 to June 2005, the evaluation grid was used in examining 250 HACCP self-monitoring plans of food companies. For the analysis of the determining factor four groups were considered, with reference to HACCP self-monitoring plans designed: group 1 - with the aid of a qualified team; group 2 - with the aid of an unqualified team; group 3 - with the aid of an unqualified expert; group 4 – without the aid of an expert. The mean values of the measures elaborated decrease towards insufficiency moving from group 1 to group 4. In particular, collaboration by teams of unqualified experts brought about drafting unacceptable HACCP plans on the levels of specificity and adherence, with respect to the HACCP method. The method proposed of the analysis of the indexes and indicators beginning with an evaluation sheet can also help the individual company to better adjust contribution by internal or external professionals to the company.

Introduction

Hygienic-sanitary self-monitoring in the food industry was made mandatory in Italy by Italian Legislative Decree no. 155/97. It was modelled on the practicable method of the system called HACCP (Hazard Analysis and Critical Control Points) designed in the late 1950’s by the Pillsbury Company to provide safe food for NASA. This system is based on the use of hazard-analysis technology at critical control points (CCP) and moves control from the final production phase to the entire production chain through monitoring individual CCP. These points are identified with the aid of a decision tree for each phase of the production flow chart that represents a potential hazard [17].

The owner of the food industry has a series of obligations, decreed by and clarified in article no. 3 of Italian Leg. Decree no. 155/97, including that to adopt a self-monitoring plan that is specific, simple, feasible and economically compatible. The control bodies on the other hand, must not act as repressors but must collaborate with the manager of the food industry by using the tools of notice-prescription.

At the end of June 1999 and March 2000, the part of Italian Leg. Decree no. 155/97 and subsequent, which concerns sanctions, conceded companies with fewer than five people in charge a further period of time to conform to the new self-monitoring system.

The phase from when the decree was issued (end of May 1997) to the expiry of the deferments is an interlocutory period between food companies and control bodies. This is an important phase for the realization of self-monitoring systems that reconcile the correct balance between the needs of a company’s balance sheet and its hygienic-sanitary requirements.

The interlocutory relation that was established with the food companies is also manifested in the preliminary examination of the HACCP self-monitoring plans and in setting out the necessary corrections.

Subjectivity as an analysis element is implicit in the evaluation of HACCP plans as the system is modelled on a generic operational method that must be adapted to every individual company rather than on an ordered set of total rules that must be strictly applied. Nevertheless, this subjectivity could lead to absurd situations when the evaluations of an HACCP plan itself are distinctly conflicting. This is the reason for the need to arrange for qualitative analysis tools that express subjectivity using objective and controllable parameters [813].

The goal of this work was that to:

  • Build a system of indexes and indicators for the evaluation of the correctness of HACCP plans;

  • Validate this system and study its determining factors.

Material and Methods

The following four critical areas were considered for examining and evaluating HACCP plans:

  • Specificity: Intended as pertinence of an HACCP plan in an unequivocal, precise manner that is determining to the individual corporate reality.

  • Simplicity: The HACCP plan must not have superfluous elements.

  • Feasibility: The HACCP plan must be practicable and not taken from a book of dreams;

  • Adherence: Adoption of the HACCP method and principles for hazard analysis at critical control points.

An evaluation grid (Table 1) was built based on these areas, that be applied when examining HACCP plans. For each Question point, there is a score ranging from 1 (minimum) to 10 (maximum) for each critical area taken into consideration. Scores are elaborated and expressed in their relevant specificity (ISp), simplicity (ISe), feasibility (IFe) and adherence (IAd) indexes, as indicated below:

ISp = Σ Sp ( tot ) / number of people in charge ISe = Σ Se ( tot ) / number of people in charge IFe = Σ Fe ( tot ) / number of people in charge IAd = Σ Ad ( tot ) / number of people in charge
where the numerator is equal to the sum of the score obtained in the relevant critical area and the denominator is represented by the number of company people in charge taken into consideration.

These indexes were the basis on which the completeness indicator (ICo) was built:

ICo = ISp + ISe + IFe + IAd

The completeness indicator is obtained therefore, by the sum of the four indexes.

Intention of the Study

During the period from January 2004 to June 2005, the Servizio di Igiene degli Alimenti e della Nutrizione – SIAN – (Food and Nutrition Health Service) of the FG/3 Local Health Authority of Foggia applied the evaluation grid when examining 250 HACCP self-monitoring plans of food companies. These companies all fell within the territorial area of the FG/3 Local Health Authority office.

The evaluation method was standardized by using a common training period for those who evaluated the HACCP plans. As a first step, the data was subjected to a repeatability and validity test in order to evaluate the reliability of the data collected.

Statistics

All the statistical analyses were carried out using the Statistical Package for the Social Sciences (SPSS for Window 5.0.1). The descriptive statistics were carried out to characterize the distribution of the evaluation of the HACCP plans.

Use of comparison between groups within variable stratification was used for the analysis of the determining factor taken into consideration for the quality of the HACCP plans, as specified below:

  • Group 1 - HACCP self-monitoring plans designed with the aid of a qualified team

  • Group 2 - HACCP self-monitoring plans designed with the aid of an unqualified team

  • Group 3 - HACCP self-monitoring plans designed with the aid of an unqualified expert

  • Group 4 - HACCP self-monitoring plans designed without the aid of an expert

A qualified team is intended as a group of people including various professional figures, who have a degree in biology, veterinary science, food science and technology, agriculture and medicine and who have followed at least one HACCP qualification course organized by public or private bodies.

A study of the indexes and completeness indicator was made using a T test for independent data in order to compare groups within the stratification by using the median rather than the mean.

Values where p<0.001 were considered highly significant while values where p<0.05 were considered significant on the average.

An acceptable quality threshold for the indexes was established for median values ≥ 45, while those with a median ≥ 180 were used for the completeness indicator.

Results

Quality Levels of the HACCP Plans

During the period from January 2004 to June 2005, 250 HACCP plans of food companies located within the territorial area of the FG/3 LHA were randomly analysed and evaluated.

Quality Levels

The variable under examination resulted being significantly associated with the distribution of the groups.

Figure 1 indicates the specificity indexes for each group. They range from a median of 47.38 for group 1 to a median of 44.44 for group 2 (group 1 versus group 2 where p > 0.05); of 23.22 for group 3 (group 1 versus group 3 where p < 0.001; group 2 versus group 3 where p < 0.001); of 22.86 for group 4 (group 1 versus group 4 where p < 0.001; group 2 versus group 4 where p < 0.001; group 3 versus group 4 where p is not significant).

Figure 2 indicates the simplicity indexes for each group. The median decreases from 47.28 for group 1 to 43.14 for group 2 (group 1 versus group 2 where p<0.05); to 23.88 for group 3 (group 1 versus group 3 where P<0.001; group 2 versus group 3 where p<0.001); to 22.93 for group 4 (group 1 versus group 4 where p<0.001; group 2 versus group 4 where p<0.001; group 3 versus group 4 where p<0.05).

The feasibility indexes are represented by Figure 3. The median values range from 50.47 for group 1 to 44.56 for group 2 (group 1 versus group 2 where p<0.05); to 22.70 for group 3 (group 1 versus group 3 where p<0.001; group 2 versus group 3 where p<0.001); to 22.50 for group 4 (group 1 versus group 4 where p<0.001; group 2 versus group 4 where p<0.001; group 3 versus group 4 where P is not significant).

Figure 4 indicates the adherence indexes. The median ranges from 50.91 for group 1 to 44.46 for group 2 (group 1 versus group 2 where p<0.05); to 24.62 for group 3 (group 1 versus group 3 where p<0.001; group 2 versus group 3 where p<0.001); to 21.96 for group 4 (group 1 versus group 4 where p<0.001; group 2 versus group 4 where p<0.001; group 3 versus group 4 where p<0.001).

The completeness indicator per group is indicated by Figure 5. The median values range from 196.03 for group 1 to 176.59 for group 2 (group 1 versus group 2 where p<0.05); to 94.42 for group 3 (group 1 versus group 3 where p<0.001; group 2 versus group 3 where p< 0.001); to 90.25 for group 4 (group 1 versus group 4 where p<0.001; group 2 versus group 4 where p<0.001; group 3 versus group 4 where p<0.05).

Discussion

The aim of our research was to build a system of indexes and indicators for the quality evaluation of HACCP plans. Beginning with the evaluation sheet, four indexes for each critical area (specificity, simplicity, feasibility and adherence) as well as the completeness indicator were put together, the latter representing a global summary measure of the indexes themselves.

During the period from January 2004 to June 2005, 250 HACCP plans from just as many food companies were examined using SIAN’s evaluation grid of the FG/03 LHA. The values found show an unsatisfactory mean qualitative level in the four critical areas of specificity, simplicity, feasibility and adherence. A more in-depth analysis indicated that when drawing up the self-monitoring plans, the contribution of personnel with specific HACCP system training was associated with better qualitative levels. Particularly, companies that used a system of a qualified team of experts achieved satisfactory mean values in the four indexes and an excellent mean qualitative level in the completeness indicator, on the contrary to other companies that made use of collaboration with unqualified experts or didn’t make use of collaboration at all [1420].

The mean values of the measures elaborated decrease towards insufficiency moving from group 1 to group 4. In particular, collaboration by teams of unqualified experts brought about drafting unacceptable HACCP plans on the levels of specificity and adherence, with respect to the HACCP method. On the other hand, collaboration by a single unqualified expert and no aide from any collaboration brought about the drafting of poor HACCP plans with respect to the profile of specificity, simplicity, feasibility and adherence.

That is to say (if it were confirmed by further studies) that it highlights that the necessity to involve qualified personnel must be oriented towards using qualified expertise; should this not be the implemented, a control body will revise the HACCP plan, resulting in all the consequences related to ordinary sanctions that a revision brings about [2123].

The method proposed of the analysis of the indexes and indicators beginning with an evaluation sheet can also help the individual company to better adjust contribution by internal or external professionals to the company.

Ijerph 04 00034f1 200
Figure 1:. Comparison between specificity indexes (mean ± SD). **, p<0.001; NS = not significant.

Click here to enlarge figure

Figure 1:. Comparison between specificity indexes (mean ± SD). **, p<0.001; NS = not significant.
Ijerph 04 00034f1 1024
Ijerph 04 00034f2 200
Figure 2:. Comparison between simplicity indexes (mean ± SD).*, p<0.05;**, p<0.001.

Click here to enlarge figure

Figure 2:. Comparison between simplicity indexes (mean ± SD).*, p<0.05;**, p<0.001.
Ijerph 04 00034f2 1024
Ijerph 04 00034f3 200
Figure 3:. Comparison between feasibility indexes (mean ± SD). *, P<0.05; **; P<0.001; NS = not significant.

Click here to enlarge figure

Figure 3:. Comparison between feasibility indexes (mean ± SD). *, P<0.05; **; P<0.001; NS = not significant.
Ijerph 04 00034f3 1024
Ijerph 04 00034f4 200
Figure 4:. Comparison between adherence indexes (mean ± DS). *, P<0.05; **, P<0.001.

Click here to enlarge figure

Figure 4:. Comparison between adherence indexes (mean ± DS). *, P<0.05; **, P<0.001.
Ijerph 04 00034f4 1024
Ijerph 04 00034f5 200
Figure 5:. Comparison between completeness indicators (mean ± SD). *, P<0.05; **, P<0.001.

Click here to enlarge figure

Figure 5:. Comparison between completeness indicators (mean ± SD). *, P<0.05; **, P<0.001.
Ijerph 04 00034f5 1024
Table Table 1:. HACCP plan evaluation grid

Click here to display table

Table 1:. HACCP plan evaluation grid
HACCP Self-Monitoring Plan, company _ _ _ _ _ _ _ _ _
Question pointCritical areas
SpSeFeAd
1. Are the specific flow charts of the production activity indicated?
2. Were the CCP identified with the aid of a decision tree?
3. Is the hazard described in detail and not generically described as physical, bacteriological or chemical?
4. Have the specific preventive measures for the hazard under examination been identified?
5. Are the critical limits expressed as objective, quantitative and measurable parameters?
6. Are the corrective actions to overcome the critical limits described?
7. Is control specifically organized by procedures and frequencies?
8. Is responsibility specified for managing the phase in which the hazard is present?
9. Are the control sheets present for monitoring the self-monitoring plan CCP?
Total Score

Sp = specificity; Se = simplicity; Fe = feasibility; Ad = adherence. Scores from 1 (minimum) to 10 (maximum) are attributed to each critical area for the relevant question point.

References

  1. Sperber, WH. The Modern HACCP System. Food Technology 1991, 45, 116–118.
  2. Pierson, MD; Corlett, DA. HACCP: Principles And Applications; Van Nostrand Reinhold: New York, 1992; pp. 13–23.
  3. National Advisory Committee On Microbiological Criteria For Foods (NACMCF). Hazard Analysis and Critical Control Points System. Int J Food Micr. 1992, 16, 1–23.
  4. National Restaurant Assoc. The Education Foundation Publication. 1993. HACCP Reference Book. Applied Food Service Sanitation 1995, 35–47.
  5. FDA. Food Code; U. S. Public Health Service, U.S. Department of Health and Human Services: Washington, DC, 1993; p. 50.
  6. Stecchini, ML; Del Torre, M. The food safety management system. Vet Res Commun 2005, 29, 117–121.
  7. van der Spiegel, M; Luning, PA; Ziggers, GW; Jongen, WM. Evaluation of performance measurement instruments on their use for food quality systems. Crit. Rev. Food Sci. Nutr. 2004, 44, 501–512.
  8. Higgins, CL; Hartfield, BS. A systems-based food safety evaluation: an experimental approach. J Environ Health 2004, 67, 9–14.
  9. McCabe-Sellers, BJ; Beattie, SE. Food safety: emerging trends in foodborne illness surveillance and prevention. J. Am Diet Assoc. 2004, 104, 1708–1717.
  10. Stecchini, ML; Del Torre, M. The food safety management system. Vet Res Commun 2005, 29, 117–121.
  11. Abernathy, T; Hart, R. Evaluation of a HACCP pilot program for the food service industry. Can J. Public Health 2004, 95(6), 470–472.
  12. Binkley, M; Ghiselli, R. Food safety issues and training methods for ready-to-eat foods in the grocery industry. J. Environ Health 2005, 68, 27–31.
  13. Fortin, ND. The hang-up with HACCP: the resistance to translating science into food safety law. Food Drug Law J 2003, 58, 565–593.
  14. Herrera, AG. The hazard analysis and critical control point system in food safety. Methods Mol. Biol. 2004, 268, 235–280.
  15. Kassem, M; Salem, E; Ahwal, AM; Saddik, M; Gomaa, NF. Application of hazard analysis and critical control point system in the dairy industry. East Mediterr Health J 2002, 8, 114–128.
  16. Buntain, BJ. Emerging challenges in public health protection, food and safety security: veterinary needs in the USDA’s Food Safety and Inspection Service. J. Vet Med. Educ. 2004, 31, 334–340.
  17. Elmi, M. Food safety: current situation, unaddressed issues and the emerging priorities. East Mediterr Health J 2004, 10, 794–800.
  18. Lan, KS. To ensure high standards of food safety through an integrated haccp system. Asia Pac J Clin Nutr. 2004, 13, 164–168.
  19. Smith, DeWaal C; Guerrero Brito, GR. Safe Food International: a blueprint for better global food safety. Food Drug Law J 2005, 60, 393–405.
  20. Sneed, J; Strohbehn, C; Gilmore, SA. Food safety practices and readiness to implement HACCP programs in assisted-living facilities in Iowa. J. Am Diet Assoc 2004, 104, 1678–1683.
  21. van der Spiegel, M; Luning, PA; Ziggers, GW; Jongen, WM. Evaluation of performance measurement instruments on their use for food quality systems. Crit. Rev Food Sci. Nutr. 2004, 44, 501–512.
  22. Strohbehn, CH; Gilmore, SA; Sneed, J. Food safety practices and HACCP implementation: perceptions of registered dieticians and dietary managers. J. Am Diet Assoc. 2004, 104, 1692–1669.
  23. Taylor, E; Taylor, JZ. Using qualitative psychology to investigate HACCP implementation barriers. Int. J. Environ Health Res. 2004, 14, 53–63.
Int. J. Environ. Res. Public Health EISSN 1660-4601 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert