Development and Reliability Review of an Assessment Tool to Measure Competency in the Seven Elements of the Risk Management Process: Part One—The RISKometric
Abstract
:1. Introduction
- Establishing the context;
- Risk identification;
- Risk analysis;
- Risk evaluation;
- Risk treatment;
- Communication and consultation, and
- Monitoring and reviewing.
1.1. 360° Performance Review Definition
1.2. 360° Performance Review—Multiple Sources of Feedback
1.3. The Collection and Delivery of Feedback
1.4. Benefits and Limitations of 360° Performance Review Assessment
- a limited or non-existent frame of reference;
- the rater using generalities rather than specifics when rating;
- the rater using an ancient history of the ratee based on memory, or
- the rater not being specifically equipped to make the rating, e.g., new to the job.
1.5. Rating Scales and Reliability and Validity Concerns (Biases)
1.6. Summary of Literature
2. Aim
- firstly, test further the RISKometric tool, whereby individuals’ RISKometric results were compared with their performance in a risk scenario exercise; so providing validity for the tool, and
- secondly, use the individual performance results to assemble collectively-, marginally- or sub-optimised teams, who undertook the risk scenario exercise to examine any team-effect on performance.
3. Method
3.1. Participants
3.2. Procedure and Material
- raters would give an honest rather than a ‘rosy’ critique, and
- raters had observed them in a risk management process.
- an introduction page that explained the purpose of the study, instructions, ethics/informed consent, an option to terminate their participation at this stage should they want to;
- a question asking whether the respondent was the participant or a rater;
- each participant’s unique six-digit identification code;
- a question asking participants their level in the organisation (for participants only);
- a question asking raters what their relationship was to the participant (i.e., upline, peer, downline colleague) (for raters);
- the survey proper that for each element included: the PEI, a question asking respondents to give a competence rating, and an open question that allowed respondents to provide extra information to support rating, and
- a ‘thank you’ for completing the survey.
3.3. Analysis Strategy
4. Results
5. Discussion
6. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Marling, G.J. Optimising Risk Management Team Processes. Ph.D. Thesis, University of Queensland, Brisbane, Australia, 2015. [Google Scholar]
- Marling, G.J.; Horberry, T.; Harris, J. Development and validation of plain English interpretations of the seven elements of the risk management process. Safety 2019, 5, 75. [Google Scholar] [CrossRef] [Green Version]
- Tornow, W. Perceptions or reality: Is multi-perspective measurement a means to an end? Hum. Resour. Manag. 1993, 32, 221–229. [Google Scholar] [CrossRef]
- Hoffman, R. Ten reasons why you should be using 360-degree feedback. HR Mag. 1995, 40, 82–86. [Google Scholar]
- Lepsinger, R.; Lucia, A. The Art and Science of 360˚ Feedback; Jossey-Bass-Pfeiffer: San Francisco, CA, USA, 2009. [Google Scholar]
- Peters, H. Peer coaching for executives. Train. Dev. 1996, 50, 39–42. [Google Scholar]
- Handy, L.; Devine, M.; Heath, L. 360˚ Feedback: Unguided Missile or Powerful Weapon? Ashridge Management Research Group: Berkhamsted, UK, 1996.
- Coates, D. Don’t tie 360 feedback to pay. Training 1998, 35, 68–75. [Google Scholar]
- Espinilla, M.; De Andrés, R.; Martínez, F.J.; Martínez, L. A 360-degree performance appraisal model dealing with heterogeneous information and dependent criteria. Inf. Sci. 2003, 222, 459–471. [Google Scholar] [CrossRef]
- Yukl, G.; Lepsinger, R. How to get the most out of 360˚ feedback. Training 1995, 32, 45–50. [Google Scholar]
- Jones, J.E.; Bearley, W.L. 360˚ Feedback: Strategies, Tactics and Techniques for Developing Leaders; HRD Press: Armherst, MA, USA, 1996. [Google Scholar]
- Mabey, C. Closing the circle: Participant views of a 360 degree feedback programme. Hum. Resour. Manag. J. 2001, 11, 41–53. [Google Scholar] [CrossRef]
- McCarthy, A.; Garavan, T.N. 360° feedback process: Performance, improvement and employee career development. J. Eur. Ind. Train. 2001, 25, 5–32. [Google Scholar] [CrossRef]
- Hannum, K.M. Measurement Equivalence of 360°-Assessment Data: Are different raters rating the same constructs? Int. J. Sel. Assess. 2007, 15, 293–301. [Google Scholar] [CrossRef]
- Nowack, K. 360˚ feedback: The whole story. Train. Dev. 1993, 47, 69–73. [Google Scholar]
- Hogg, B. The AMA Competency Programme. Development Centers: Realizing the Potential of Your Employees through Assessment and Development; The Tata-McGraw-Hill Training Series: London, UK, 1989. [Google Scholar]
- Craig, S.B.; Hannum, K. Research update: 360-degree performance assessment. Consult. Psychol. J. Pr. Res. 2006, 58, 117–124. [Google Scholar] [CrossRef]
- Doherty, E.G.; Brodsky, D. Educational perspectives the 360-degree assessment: A new paradigm in trainee evaluation. NeoReviews 2011, 12, 191–197. [Google Scholar] [CrossRef] [Green Version]
- Carson, M. Saying it like it isn’t: The pros and cons of 360-degree feedback. Bus. Horiz. 2006, 49, 395–402. [Google Scholar] [CrossRef]
- Penny, J.A. Exploring differential item functioning in a 360-degree assessment: Rater source and method of delivery. Organ. Res. Methods 2003, 6, 61–79. [Google Scholar] [CrossRef]
- Smither, J.W.; Walker, A.G.; Yap, M.K.T. An examination of the equivalence of web-based versus paper-and-pencil up- ward feedback ratings: Rater- and ratee-level analyses. Educ. Psychol. Meas. 2004, 64, 40–61. [Google Scholar] [CrossRef]
- Denisi, A.S.; Kluger, A.N. Feedback effectiveness: Can 360-degree appraisals be improved? Acad. Manag. Perspect. 2000, 14, 129–139. [Google Scholar] [CrossRef]
- Hackman, J.R.; Oldham, G.R. Work Redesign; Addison-Wesley: Reading, MA, USA, 1980. [Google Scholar]
- Becker, T.E.; Klimoski, R.J. A field study of the relationship between the organizational feedback environment and performance. Pers. Psychol. 1989, 42, 343–358. [Google Scholar] [CrossRef]
- Church, A.H.; Bracken, D.W. Advancing the state of the art of 360-degree feedback guest editors’ comments on the research and practice of multirater assessment methods. Group Organ. Manag. 1997, 22, 149–161. [Google Scholar] [CrossRef]
- London, M.; Beatty, R. 360° feedback as a competitive advantage. Hum. Resour. Manag. 1993, 32, 353–372. [Google Scholar] [CrossRef]
- Hazucha, J.F.; Hezlett, S.A.; Schneider, R.J. The impact of 360-degree feedback on management skills development. Hum. Resour. Manag. 1993, 32, 325–351. [Google Scholar] [CrossRef]
- Folger, R.; Cropanzano, R. Organizational Justice and Human Resource Management; SAGE Publications: Yhousand Oaks, CA, USA, 1998. [Google Scholar]
- Hegarty, W.H. Using subordinate ratings to elicit behavioral changes in supervisors. J. Appl. Psychol. 1974, 59, 764–766. [Google Scholar] [CrossRef]
- Atwater, L.; Roush, P.; Fischthal, A. The influence of upward feedback on self- and follower ratings of leadership. Pers. Psychol. 1995, 48, 35–59. [Google Scholar] [CrossRef]
- Reilly, R.; Smither, J.; Vasilopoulos, N. A longitudinal study of upward feedback. Pers. Psychol. 1996, 49, 599–612. [Google Scholar] [CrossRef]
- Walker, A.; Smither, J. A five-year study of upward appraisal feedback: What managers do with their results matters. Pers. Psychol. 1999, 52, 393–423. [Google Scholar] [CrossRef]
- Smither, J.; London, M.; Vasilopoulos, N.; Reilly, R.; Millsap, R.; Salvemini, N. An examination of the effects of an upward feedback program over time. Pers. Psychol. 1995, 48, 1–34. [Google Scholar] [CrossRef]
- Atwater, L.; Waldman, D.; Atwater, D.; Cartier, P. An upward feedback field experiment: Supervisors’ cynicism, reactions, and commitment to subordinates. Pers. Psychol. 2000, 53, 275–297. [Google Scholar] [CrossRef]
- Morgan, A.; Cannan, K.; Cullinane, J. 360° feedback: A critical enquiry. Pers. Rev. 2005, 34, 663–680. [Google Scholar] [CrossRef]
- Wexley, K.; Klimoski, R. Performance appraisal: An update. In Research in Personnel and Human Resources Management, 2nd ed.; Rowland, K., Ferris, G., Eds.; JAI Press: Greenwich, CT, USA, 1984. [Google Scholar]
- Kane, J.; Lawler, E. Methods of peer assessment. Psychol. Bull. 1978, 8, 555–586. [Google Scholar] [CrossRef]
- Cardy, R.; Dobbins, G. Performance Appraisal: Alternative Perspectives; Southwestern Publishing: Cincinnati, OH, USA, 1994. [Google Scholar]
- Bettenhausen, K.; Fedor, D. Peer and upward appraisals: A comparison of their benefits and problems. Group Organ. Manag. 1997, 22, 236–263. [Google Scholar] [CrossRef]
- Lawler, E.; Mohrman, A.; Resnick, S. Performance appraisal revisited. Organ. Dyn. 1984, 13, 20–35. [Google Scholar] [CrossRef]
- Meyer, H. A solution to the performance appraisal feedback enigma. Acad. Manag. Perspect. 1991, 5, 68–76. [Google Scholar] [CrossRef]
- Napier, N.; Latham, G. Outcome expectancies of people who conduct performance appraisals. Pers. Psychol. 1986, 39, 827–837. [Google Scholar] [CrossRef]
- Bernardin, H.J.; Beatty, R. Can subordinate appraisals enhance managerial productivity? Sloan Manag. Rev. 1987, 28, 63–73. [Google Scholar]
- London, M.; Wohlers, A.; Gallagher, P. 360˚ feedback surveys: A source of feedback to guide management development. J. Manag. Dev. 1990, 9, 17–31. [Google Scholar] [CrossRef]
- Kaplan, R. 360-degree feedback PLUS: Boosting the power of co-worker ratings for executives. Hum. Resour. Manag. 1993, 32, 299–314. [Google Scholar] [CrossRef]
- Bracken, D. Multisource (360˚) feedback: Surveys for individual and organizational development. In Organizational Surveys: Tools for Assessment and Change; Kraut, A., Ed.; Jossey-Bass: San Francisco, CA, USA, 1996; pp. 117–143. [Google Scholar]
- Moses, J.; Hollenbeck, G.; Sorcher, M. Other people’s expectations. Hum. Resour. Manag. 1993, 32, 283–297. [Google Scholar] [CrossRef]
- Kanouse, D. Why multi-rater feedback systems fail. HR Focus 1998, 75, 3–4. [Google Scholar]
- Rou, T.V.; Rou, R. The Power of 360 Degree Feedback; Sage Publications: New Delhi, India, 2014. [Google Scholar]
- McEvoy, G.; Buller, P. user acceptance of peer appraisals in an industrial setting. Pers. Psychol. 1987, 40, 785–797. [Google Scholar] [CrossRef]
- Fedor, D.; Bettenhausen, K.; Davis, W. Peer reviews: Employees’ dual roles as raters and recipients. Group Organ. Manag. 1999, 24, 92–120. [Google Scholar] [CrossRef]
- Murphy, K.; Cleveland, J. Performance Appraisal: An Organizational Perspective; Allyn & Bacon: Boston, MA, USA, 1991. [Google Scholar]
- McEvoy, G.; Beatty, R. Assessment centres and subordinate appraisal of managers: A seven-year longitudinal examination of predictive validity. Pers. Psychol. 1989, 42, 37–52. [Google Scholar] [CrossRef]
- Atkins, P.W.B.; Wood, R.E. Self- versus others’ ratings as predictors of assessment center ratings: Validation evidence for 360-degree feedback programs. Pers. Psychol. 2002, 55, 871–904. [Google Scholar] [CrossRef]
- Fox, S.; Caspy, T.; Reisler, A. Variables affecting leniency, halo and validity of self-appraisal. J. Occup. Organ. Psychol. 1994, 67, 45–56. [Google Scholar] [CrossRef]
- Zou, P.X.W.; Sunindijo, R.Y. Skills for managing safety risk, implementing safety task, and developing positive safety climate in construction project. Autom. Constr. 2013, 34, 92–100. [Google Scholar] [CrossRef]
- Burnham, M. Targeting zero. Professional safety. J. Am. Soc. Saf. Eng. 2015, 60, 40–45. [Google Scholar]
Element | Correlation Coefficient (r) | Descriptives | ||||||
---|---|---|---|---|---|---|---|---|
Self | Upline | Peers | Downline | Median | Quartile Array 1 (Q1) | Quartile Array 3 (Q3) | Interquartile Range (IQR = Q3 − Q1) | |
Context | ||||||||
Self | 1 | 3.00 | 2.25 | 4.00 | 1.75 | |||
Upline | −0.258 | 1 | 3.00 | 3.00 | 4.00 | 1.00 | ||
Peers | 0.053 | 0.444 * | 1 | 2.00 | 1.00 | 3.00 | 2.00 | |
Downline | −0.034 | 0.661 ** | 0.849 ** | 1 | 2.00 | 2.00 | 3.00 | 1.00 |
Risk identification | ||||||||
Self | 1 | 3.00 | 3.00 | 4.00 | 1.00 | |||
Upline | −0.012 | 1 | 4.00 | 3.00 | 4.00 | 1.00 | ||
Peers | 0.153 | 0.397 | 1 | 2.00 | 2.00 | 3.00 | 1.00 | |
Downline | 0.306 | 0.383 | 0.875 ** | 1 | 2.00 | 2.00 | 4.00 | 2.00 |
Risk analysis | ||||||||
Self | 1 | 3.00 | 2.25 | 4.00 | 1.75 | |||
Upline | 0.022 | 1 | 4.00 | 2.00 | 4.00 | 2.00 | ||
Peers | −0.077 | 0.601 ** | 1 | 1.00 | 1.00 | 2.00 | 1.00 | |
Downline | 0.025 | 0.606 ** | 0.895 ** | 1 | 1.50 | 1.00 | 2.75 | 1.75 |
Risk evaluation | ||||||||
Self | 1 | 3.00 | 2.00 | 4.00 | 2.00 | |||
Upline | 0.087 | 1 | 3.00 | 2.50 | 4.00 | 1.50 | ||
Peers | 0.174 | 0.111 | 1 | 1.00 | 1.00 | 2.00 | 1.00 | |
Downline | 0.262 | 0.311 | 0.763 ** | 1 | 1.50 | 1.00 | 2.00 | 1.00 |
Risk treatment | ||||||||
Self | 1 | 3.00 | 3.00 | 4.00 | 1.00 | |||
Upline | −0.187 | 1 | 3.00 | 2.00 | 4.00 | 2.00 | ||
Peers | −0.116 | 0.091 | 1 | 2.00 | 1.00 | 2.00 | 1.00 | |
Downline | 0.129 | 0.152 | 0.765 ** | 1 | 2.00 | 1.00 | 3.00 | 2.00 |
Communication and consultation | ||||||||
Self | 1 | 3.00 | 2.00 | 4.00 | 2.00 | |||
Upline | 0 | 1 | 4.00 | 2.50 | 4.00 | 1.50 | ||
Peers | −0.195 | 0.091 | 1 | 1.00 | 1.00 | 2.00 | 1.00 | |
Downline | 0.020 | 0.113 | 0.797 ** | 1 | 1.00 | 1.00 | 2.00 | 1.00 |
Monitoring and reviewing | ||||||||
Self | 1 | 3.00 | 2.00 | 4.00 | 2.00 | |||
Upline | −0.149 | 1 | 3.00 | 2.00 | 4.00 | 2.00 | ||
Peers | 0.276 | 0.182 | 1 | 1.00 | 1.00 | 2.00 | 1.00 | |
Downline | −0.351 | 0.249 | 0.429 * | 1 | 1.00 | 1.00 | 1.75 | 1.75 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Marling, G.; Horberry, T.; Harris, J. Development and Reliability Review of an Assessment Tool to Measure Competency in the Seven Elements of the Risk Management Process: Part One—The RISKometric. Safety 2021, 7, 1. https://doi.org/10.3390/safety7010001
Marling G, Horberry T, Harris J. Development and Reliability Review of an Assessment Tool to Measure Competency in the Seven Elements of the Risk Management Process: Part One—The RISKometric. Safety. 2021; 7(1):1. https://doi.org/10.3390/safety7010001
Chicago/Turabian StyleMarling, Garry, Tim Horberry, and Jill Harris. 2021. "Development and Reliability Review of an Assessment Tool to Measure Competency in the Seven Elements of the Risk Management Process: Part One—The RISKometric" Safety 7, no. 1: 1. https://doi.org/10.3390/safety7010001