Safety Occurrence Reporting amongst New Zealand Uncrewed Aircraft Users

Abstract

Safety reporting has long been recognised as critical to reducing safety occurrences by identifying issues early enough that they can be remedied before an adverse outcome. This study examines safety occurrence reporting amongst a sample of 92 New Zealand civilian uncrewed aircraft users. An online survey was created to obtain the types of occurrences that these users have had, how (if at all) these are reported, and why participants did or did not report using particular systems. This study focussed on seven types of occurrences that have been highlighted by the Civil Aviation Authority of New Zealand as being reportable using a CA005RPAS form, the template for reporting to the authority for uncrewed aircraft occurrences. The number of each type of occurrence was recorded, as well as what percentage of occurrences were reported using a CA005RPAS form, an internal reporting system, or were non-reported. Qualitative questions were used to understand why participants did or did not report using particular systems. Categorical and numerical data were analysed using Chi-Squared Tests of Independence, Kruskal–Wallis H Tests, and Mann–Whitney U Tests. Qualitative data were analysed using thematic analysis. The findings reveal that 85.72% of reportable safety occurrences went unreported by pilots, with only 2.74% of occurrences being self-reported by pilots using a CA005RPAS form. The biggest reason for non-reporting was that the user did not perceive the occurrence as serious enough, with not being aware of reporting systems and not being legally required to report also being major themes. Significant differences were observed between user groups, providing policy implications to improve safety occurrence reporting, such as making reporting compulsory, setting minimum training standards, having an anonymous and non-punitive reporting system, and through working with member-based organisations.

1. Introduction

While some types of uncrewed aircraft (UA) have been used within the civilian space for a long time (such as aeromodellers flying model aircraft), the proliferation of remotely-piloted aircraft and autonomous uncrewed aircraft (often called “drones”) has accelerated in recent years [1], with the International Civil Aviation Organisation beginning to look at this issue as early as 2006 [2]. With applications ranging from aerial survey, mapping, aerial photography and video, along with inspection in the agricultural, security, energy, and construction industries, the benefits and range of uses continues to grow [3,4]. Worldwide, UA have become “more accessible, affordable, adaptable and more capable of anonymity” [5] p. 1. As UA utilisation expands, the potential for occurrences also increases. This highlights the importance of the communication of hazard information to mitigate risks and provide safety solutions. As technology progresses, UA are now performing Beyond Visual Line of Sight (BVLOS) operations and the amount of autonomy is also increasing, highlighting the need for safety systems that are fit for purpose [1]. While there is always a degree of inherent risk within aviation, the benefits of achieving a task must be perceived to outweigh any associated risks, particularly those toward crewed aircraft [6]. UA are inexpensive to purchase and because no one is aboard the aircraft, the user may take greater risks. Considering this, the importance of being proactive and identifying potential risks and vulnerabilities in advance ensures UA remain beneficial and do not become a danger to safety as the industry expands [7]. Occurrence reporting and monitoring of UA activities may be one method to reduce the risk of further occurrences [4].

With the development of UA technology and the likelihood of the UA industry expanding into shared airspace, along with increased numbers of UA operating BVLOS [1], the objective of this research is to examine the types of UA safety occurrences that users are having, how (if at all) these UA safety occurrences are being reported, and why they are being reported (or non-reported) using particular systems. An online survey was created to obtain this information from users. Our study was restricted to UA users in New Zealand, and only measured UA safety occurrences for each user between 2015 and 2022. This was because the current regulatory framework came into effect in 2015. With the data obtained, similarities and differences between users on their reporting (or non-reporting) of safety occurrences can be examined to ensure that risks are managed collectively, and safety improvements are made for the benefit of the industry.

The next section will present a literature review, which examines the applicable regulations in New Zealand and compares these with other jurisdictions and discusses relevant past literature in relation to safety occurrence reporting. Next, the methods are presented, including how the survey was created, how participants were recruited, our sample, and the how the data were analysed. The results are then presented, both quantitative (i.e., number of types of occurrences, number reported or non-reported using particular systems, and so on) and qualitative (i.e., the reasons for reporting or non-reporting using particular systems, as well as use of alternative safety performance measures). The results are then discussed, highlighting a number of potential strategies for improving safety reporting amongst UA users in New Zealand. Finally, the study is concluded and limitations and opportunities for future research are provided.

2. Literature Review

2.1. Terminology

Worldwide, several terms are used to describe UA. Colloquially, they are often referred to as drones, with more formal terms including Unmanned/Uncrewed Aerial Vehicles (UAVs), Unmanned/Uncrewed Aircraft Systems (UASs), Remotely Piloted Aircraft Systems (RPAS), and model aircraft [4,8]. Nuances exist between these terms; for example, UAVs refer to all uncrewed aircraft, while RPAS excludes autonomous aircraft, and model aircraft tends to refer to scale versions of crewed aircraft [4,8,9]. This study is interested in all types of UA, and henceforth we have consistently used the UA abbreviation throughout the paper.

2.2. Applicable Regulations

UA operations are primarily governed by two Civil Aviation Rule (CAR) Parts, CAR Parts 101 and 102. CAR Part 101 outlines a series of general operating rules, while CAR Part 102 outlines a process for certificating organisations that want to execute operations outside of those general operating rules [10,11]. Crewed aircraft incident and accident notification is regulated under CAR Part 12 but gives dispensation for UA to file occurrence reports [12]. According to CAR102.11, UA operators are required to produce an exposition to apply for Part 102 certification. The exposition must outline processes for a hazard register. The hazard register identifies risks and known hazards involved with the operation, the mitigation measures taken to avoid them along with details of the operation and equipment to be used [10]. While not being an explicit legal requirement to submit CA005RPAS forms to report occurrences, an advisory circular (outlining acceptable means of compliance) recommends that UA operators conducting operations under Part 102 should report occurrences in this manner [13]. This advisory circular lists seven events where reporting using a CA005RPAS form is recommended. These are “ injury to persons; and loss of control; and fly-away; and motor or structural failure; and incidents involving manned aircraft; and incursion into airspace where not authorised; and damage to third party property” [13] p. 16. Nonetheless, the advisory circular does encourage UA operators to report any occurrences that they deem necessary, stating that regular recording of statistical data may help establish the reliability of UA and used when updating regulations [13]. However, with UA listed as a low-consequence and low-regulatory priority in the Civil Aviation Authority of New Zealand’s (CAANZ) current safety strategy, these regulations may not be revised for some time [14].

New Zealand does not significantly differ in most respects with other jurisdictions with regard to regulation of UA [8]. However, internationally, there are differences with regard to the requirement for UA occurrence reporting. For example, Australia, the United States, the United Kingdom, and member states of the European Union Aviation Safety Agency (including the 27 European Union member countries, Switzerland, Iceland, Norway, and Liechtenstein) all provide very explicit thresholds for when UA occurrence reporting is required [15,16,17,18,19]. The specific thresholds nonetheless vary. In this respect, New Zealand diverges from these major jurisdictions by only providing an advisory circular, which, because it relates to CAR Part 102, implies it is not relevant to CAR Part 101 where more operations occur. Australia, the United States, and the European Union also provide confidential reporting systems, which are for all incidents, including those that do not reach the threshold for formal reporting [15,20,21]. These systems are aimed at collecting, analysing, and sharing safety information with users, industry, and regulators. They also emphasise that they are there to help learn from occurrences and will not be used to assign blame. There is no confidential reporting system available in New Zealand, and there is also no explicit statement on the CA005RPAS form to say that reports of UA occurrences will not be used to assign blame, though it does highlight the purpose of reporting is to help monitor risk, learn from incidents and accidents, and help to reduce the chances of accidents occurring [22].

2.3. Safety Occurrence Reporting

Reporting occurrences is an essential element of safety, providing information for the improvement and development of technology in addition to aiding UA pilots to develop their skills and knowledge to avoid further occurrences. Occurrences can be a forewarning system of a gap or weakness in the safety hazard management system. Incidents enable industry to develop learning to avoid more serious occurrences [23]. For an organisation to avoid occurrences leading to major accidents, it is crucial for occurrence information to be recorded and the information filtered to those affected, ensuring risk management strategies are adapted to avoid occurrences turning somewhat more serious [23]. For the sake of brevity, key findings and approaches from past studies related to safety occurrence reporting are summarised in

Table 1. Abbreviated summary of past research relevant to UA safety occurrence reporting.

Study	Industry	Approach	Relevant Findings 1
Cooke and Rohleder [23]	Non-specific	Simulating a model of a safety and incident learning system	Incident learning systems help to reinforce safety culture Organisations should implement rewards systems to encourage incident reporting When occurrence information is shared with industry, other organisations may learn and avoid the same occurrence
Cermelli et al. [24]	Petrochemical	Analysis of 5800 reported occurrences from the construction phase of a large petrochemical complex in the Middle East	Only 10–20% of near misses are self-reported
Nazeri et al. [25]	Crewed Aviation	Data mining and analysis of accidents and incidents pertaining to large aircraft commercial flights in the United States from 1995 to 2004	Incident reporting data were subject to bias and may be underreported A mixture of influences, when combined, create the situations that are more likely to result in an accident
Kyriakidis et al. [26]	Rail Transport	Precursor analysis from 18 major metros between 2002 and 2009 and a questionnaire to representatives in 11 metro organisations to assess safety maturity	More safety-conscious metros tend to record more accident precursors and top event and do so more rigorously Even metros that collect data likely do so only for more severe incidents Safety culture is appropriate to reduce accident precursors caused by human performance failures
Wild et al. [27]	Crewed and Uncrewed Aviation	Analysis of 152 UA accidents and incidents between 2006 and 2015 and comparison with crewed aircraft occurrences	UA occurrences were focussed on technology, whereas crewed aircraft occurrences were focussed on human factors
Ghasri and Maghrebi [4]	Uncrewed Aviation	Analysis of 138 UA accidents and incidents reported to the Australian Transport Safety Board between 2000 and 2018	Technology and equipment play a prevailing role in UA occurrences Data patterns from reported occurrences can be biased because of different attitudes among operators towards reporting occurrences
Kasprzyk and Konert [28]	Uncrewed Aviation	Content analysis of current European and US legislation covering UA accidents and serious incidents	The shortage of occurrence data does not provide sufficient information for improving safety and avoiding serious occurrences in the future
Konert and Kasprzyk [17]	Uncrewed Aviation	Content analysis of legislation covering incidents	There is a lack of consistent standards for reporting serious incidents and accidents involving UA UA incident reporting helps to improve safety policies and facilitate airspace integration
CAAUK [29]	Uncrewed Aviation	Survey of 32,933 UA users in the open and special categories	Only 25% of reports filed each month come from the pilots themselves UA occurrences are underreported by approximately ten times when compared with crewed aircraft occurrences Knowledge of the legal system, accessibility, the difficulty of filing reports, and the fear of repercussions were the key reasons why UA operators did not report occurrences
Henderson [30]	Uncrewed Aviation	Survey of 812 UA users to examine means of mitigating operational risks	UA users need to view risk more holistically as there is a tendency to focus only on the airworthiness of the UA and not other sources of risk

¹ We have been selective about which findings are immediately relevant to the study at hand—only a portion of the findings from any of these studies are reported here; for full results, readers will need to consult the sources themselves.

.

Past research on occurrence reporting highlights some consistent themes. Firstly, occurrence reporting allows for learning and improvement of safety over time, not just for the affected organisation, but also for the industry as a whole when information is shared. Secondly, reporting has an attitudinal element, and organisational safety culture is often cited as having an influence on reporting, as well as the vice versa, where acting upon safety reports helps to reinforce a safety culture. Thirdly, underreporting is a common issue, with UA occurrence reporting estimated to be much lower than for crewed aircraft, and where only a minority of reports are self-reported by UA pilots. Lack of seriousness of an occurrence, knowledge of the legal system, usability of reporting systems, and fear of repercussions are cited as the key barriers to reporting. Fourthly, most past research has used reported occurrences as data, which is acknowledged by those studies as subject to bias because they are only the reported occurrences. This highlights the novel approach of this study by capturing participant occurrences, whether reported or unreported. Finally, no study appears to have asked UA users qualitative questions to understand why they are reporting or not, and if so, why they use particular systems.

3. Methods

3.1. Materials

An online survey was created to examine what types of safety occurrences UA users are having, how (if at all) these are being reported, and why they are being reported (or non-reported) using particular reporting systems. To provide some clear guidelines about what should be reported and what systems are available, this survey was designed for New Zealand UA users. Gender and age were collected for descriptive purposes. Users were categorised by user type (recreational, semi-professional, or professional), recency of flying, hours flown in the last 12 months, whether they had passed a course of UA operations, whether they had passed an Operational Competency Assessment (OCA) (the local term for a UA flight examination), whether they were a member of Model Flying New Zealand (MFNZ) (this is a nation-wide member organisation for aeromodellers), whether they were a member of UAVNZ and/or Aviation New Zealand (UAVNZ is an industry and professional body for UA operators, and Aviation New Zealand is its parent organisation representing the wider commercial aviation sector), which Rule Part they operated under (Part 101, Part 102, both, or unsure), and whether they pilot aircraft of 15 kg or more (as this is where some qualification or certification becomes required). These categorisations were used for later statistical analyses. Advisory Circular AC102-1 outlines seven types of occurrences that CAANZ would like reported using a CA005RPAS form, which are [13] p. 16:

• Injury to persons (which includes the operator)
• Loss-of-control incidents
• Fly-aways
• Motor and structural failures
• Incidents involving manned aircraft
• Incursion into airspace where not authorised
• Damage to third party property

Because this advisory circular was published alongside the current regulatory framework in 2015, this study limited its scope to occurrences between 2015 and 2022. The survey asks participants to tick a box as to whether they have had any of each occurrence within this time period, and if they had, they were asked a follow-up question to obtain the number of each type of occurrence that they had within this timeframe. Participants who had no reportable occurrences during this time period were asked follow-up questions to see whether they had reported anything else, while participants who had a reportable occurrence were asked to provide percentages for how many of their occurrences had been reported using a CA005RPAS form, how many had been reported using an internal reporting system, how many had been reported using both the CA005RPAS form and an internal reporting system, and how many were not reported using either system. Qualitative questions were used to obtain the reasons why they did or did not report using particular systems and whether they had any alternative ways of measuring safety performance for their operations. There were two reasons why qualitative questions were used for this purpose. The first was to avoid the issue of self-generated validity where answers may be influenced by asking questions about measures that may not exist in long-term memory [31,32,33,34]. The second was to be consistent with a heterophenomenological epistemology, whereby it is important to recognise that each person lives in their own subjective reality, influenced by their life experiences and what they believe about those experiences [35,36]. By allowing participants to describe their reasoning in their own words, we build a better understanding of why they think they may behave in a particular manner [37], which is of interest when considering their behaviour in relation to safety occurrence reporting. The full list of questions in the survey, including display logic, is provided in Appendix A.

3.2. Procedure

An online survey hosted via Qualtrics was used to collect data. This was available from the 12 September 2022 until the 8 October 2022. Participants were recruited through posts on social media, through a link in an Aviation New Zealand weekly newsletter, and through encouraging participants to refer others onto the survey. Posts were made on the following Facebook forums to recruit participants: (1) Kiwi Pilots, (2) DJI Drones New Zealand Operators’ Group, (3) Drone Fishing New Zealand, (4) Multirotors New Zealand, (5) New Zealand Drone Photography, (6) NZ Drone Photography, and (7) Drones on Farm NZ. A post was also made on LinkedIn using the personal account of the second author. Participants were asked how they found out about the survey before completing the survey. When clicking on the link, participants were presented with an information sheet about the study. Three recruitment criteria were applied:

• Participants had to reside in New Zealand
• Participants had to have flown an unmanned aircraft at least once since 2015
• Participants had to be 16 years or older (age to give consent to participate in New Zealand)

The use of convenience sampling was pragmatic—there are no reliable data about how many UA users there are in New Zealand, nor the split of different user types, though some estimations have been made [8]. Thus, this study simply aimed to make sure that there was a reasonable chance that different user types would be exposed to the recruitment materials posted across multiple forums. The “push-out” approach of social media recruitment (recruiting while they are engaged in other unrelated online activities) has been shown to provide demographically diverse samples [38]. Recruitment via Facebook has been shown to gather samples that are similarly representative to more traditional methods [39], with differences between Facebook data sets and comparison data sets being practically insignificant in their magnitude [40]. However, some researchers have found that it results in over-representation of young white women [41].

This project was peer-reviewed and deemed to be low risk. Consequently, it was not reviewed by one of the University’s Human Ethics Committees but was registered as a low-risk study on the Massey University Human Ethics Database.

3.3. Sample

The survey obtained 110 responses during the study period. However, only 92 responses were complete enough to be useful (determined by completing at least 69% of the questions). Out of this sample of 92 participants, 83 (90.22%) were male, 6 (6.52%) were female, 1 (1.09%) was nonbinary, and 2 (2.17%) preferred not to say. The mean age was 42.78 (SD = 16.25), with one participant who did not provide age. All participants were current users of UA. Of these, 49 (53.26%) classified themselves as recreational users (primarily for enjoyment), 21 (22.83%) classified themselves as semi-professional (where less than 50% of the participants work time is spent on activities related to unmanned aircraft operations), and 22 (23.91%) classified themselves as professional users (where more than 50% of the participants work time is spent on activities related to unmanned aircraft operations).

To ensure that UA users from various groups were not over-represented, we asked participants how they found out about the survey. The majority of participants, 75 (81.52%), found out via social media, 14 (15.22%) found out from the Aviation New Zealand email, and 3 (3.26%) were referred by a friend.

Regarding flight recency, 62 (67.39%) had flown within the last month, 18 (19.57%) had flown within the last 6 months, 6 (6.52%) had flown within the last year, 6 (6.52%) had flown more than a year ago. With regards to flight currency, 18 (19.57%) had flown less than 5 h within the last 12 months, 19 (20.65%) had flown 5–10 h within the last 12 months, 16 (17.39%) had flown 10–25 h within the last 12 months, 13 (14.13%) had flown 25–50 h within the last 12 months, and 26 (28.26%) had flown more than 50 h within the last 12 months. There were 34 (36.96%) participants who had completed a course on UA operations, and 40 (43.48%) who had passed an operational competency assessment. In terms of member-based organisations, 21 (22.83%) were members of MFNZ, and 20 (21.74%) were members of UAVNZ and/or Aviation New Zealand.

Most of the participants (55, 59.78%) operated only under Part 101 of the CARs, while three (3.26%) operated only under a Part 102 Operator’s Certificate, and 17 (18.48%) operated under both Part 101 and Part 102. Concerningly, 17 (18.48%) participants were unsure which set of CARs they were operating under. Only 6 (6.52%) participants operated UA with a mass of more than 15 kg.

3.4. Analysis

Participants were coded according to their user type, recency of flying, hours flown in last 12 months, whether they had completed a course or OCA, whether they were members of MFNZ or UAVNZ, and which Rule Part they used for operations. Chi-Squared Tests of Independence [42] were run to see whether occurrence reporting (both to CAANZ and internally) as well as non-reporting were associated with particular participant groups. Effect size is reported with Cramer’s V [43]. The percentage of occurrences that were reported (using any means) was calculated for each participant as a numerical value. To see whether differences exist between the percentage of reported occurrences based upon user type, recency of flying, and hours flown, Kruskal–Wallis H tests [44] were run. Distributions were checked for similarity by visual inspection of a boxplot. Pairwise comparisons using Dunn’s [45] procedure were performed and a Bonferroni [46] correction was applied. For other participant groupings (which only involve two groups), Mann–Whitney U tests [47] were used to see whether differences existed in the percentage of occurrences that were reported. Distributions were assessed to be similar based upon visual inspection, and directionality was assessed according to mean ranks and distributions using an exact sampling distribution for U [48].

For qualitative responses, a process consistent with Braun and Clarke’s [49] fifteen-point checklist for a good thematic analysis was used. First, all responses to each qualitative question were collated. Next, they were divided into single units of thought, which this study has called “statements”. This was important as sometimes a participant may outline multiple ideas within the same response, so they may have multiple statements for a particular question. Definitions for themes were created so that statements could be thematically classified. Definitions for themes did not overlap, so participants would only be grouped into multiple themes if they made statements that expressed ideas consistent with multiple themes.

4. Results

4.1. Descriptive Results

Out of the participants, 50 (54.35%) participants had a reportable occurrence in the period of 2015 to 2022. A summary of the types of occurrences is presented in

Table 2. Types of occurrences observed among users.

Type of Occurrence	No. Participants (%)	Observed Number	Mean (Sample)	Mean (Sub-Sample)
Injury to person	8 (8.70%)	9	M = 0.10, SD = 0.33	M = 1.13, SD = 0.33
Loss of control	34 (36.00%)	271	M = 2.97, SD = 11.62	M = 7.97, SD = 18.05
Fly-away	12 (13.04%)	34	M = 0.36, SD = 1.65	M = 2.83, SD = 3.72
Motor or structural failure	22 (23.91%)	122	M = 1.33, SD = 5.54	M = 5.55, SD = 10.24
Loss of separation with manned aircraft	3 (3.26%)	7	M = 0.08, SD = 0.54	M = 2.33, SD = 1.89
Airspace incursion	3 (3.26%)	7	M = 0.08, SD = 0.54	M = 2.33, SD = 1.89
Damage to third-party property	0	0	N/A	N/A
No reportable occurrences	42 (45.65%)	N/A	N/A	N/A
Total	50 1 (54.35%)	450	M = 4.89, SD = 18.79	M = 9.00, SD = 24.76

¹ This number cannot be computed from the earlier numbers of participants for each occurrence type as a single participant may have had multiple types of occurrences.

, showing the number of participants having each occurrence, the total number of each occurrence observed, the mean number for each occurrence across the sample (n = 92), and the mean number for each occurrence across the sub-sample of only those users who had that occurrence type.

Out of the 50 participants that indicated that they had reportable occurrences, 45 completed the follow-up questions about the percentage of occurrences that were reported using a CA005RPAS form (only), reported internally (only), reported both using a CA005RPAS form and internally, and the percentage that were not reported.

Table 3. Prevalence of reporting systems to report occurrences.

Reporting System	Mean Percentage	No. Participants with >0%
CA005RPAS Form (only)	2.67% (SD = 14.97%)	2
Internal Reporting (only)	21.24% (SD = 38.63%)	12
Both CA005RPAS Form and Internal Reporting	4.96% (SD = 19.14%)	4
Not reported	71.13% (SD = 43.66%)	33
Reported to CAANZ 1	7.62% (SD = 23.93%)	5
Reported Internally 2	26.20% (SD = 41.84%)	12
Reported (any means)	28.87% (SD = 43.66%)	12

¹ Computed by adding percentages for CA005RPAS form (only) and those who used that form as well as internal reporting. ² Computed by adding percentages for internal reporting (only) and those who reported internally and submitted a CA005RPAS form.

provides a summary of the prevalence of reporting systems to report occurrences.

Using the total number of reportable occurrences for each of the 45 participants who answered the follow-up questions, we can now multiply the percentage reported to CAANZ, internally, and the percentage non-reported by the number of reportable occurrences. This is important as different users had different numbers of reportable occurrences; it is not just the average percentages that are important, but the percentage of actual reportable occurrences. The sample of 45 participants who answered the follow-up questions had a total of 427 reportable occurrences,

Table 4. Number and percentage of occurrences that were reported using different systems.

Reporting System	No. Occurrences	% Occurrences (n = 427)
Reported to CAANZ 1	11.69	2.74%
Reported Internally 2	56.17	13.15%
Reported (any means)	60.97	14.28%
Non-Reported (any means)	366.03	85.72%

¹ Computed by adding percentages for CA005RPAS form (only) and those who used that form as well as internal reporting, and then multiplying by the user’s observed number of occurrences. ² Computed by adding percentages for internal reporting (only) and those who reported internally and submitted a CA005RPAS form, and then multiplying by the user’s observed number of occurrences.

shows how those occurrences are divided amongst reporting systems.

For this particular sample, despite the mean percentage for non-reporting being 71.13% across users, the percentage that were actually not reported was 85.72%. This was because some of the users with the highest number of occurrences had 0% reporting rates, skewing the percentage of occurrences upwards.

4.2. Quantitative Results

For the sake of brevity, statistically significant results from the Chi-Squared Tests of Independence, Kruskal–Wallis H Tests, and Mann–Whitney U Tests are reported in abbreviated form in Figure 1. In this figure, user groups are highlighted as being associated with either reporting to CAANZ via submitting a CA005RPAS Form, reporting using an internal process, or non-reporting of occurrences. These associations are based upon the results of the Chi-Squared Tests of Independence. Differences between groups in the overall percentage of occurrences that were reported (using either CA005RPAS Forms or an internal process) are shown in the bottom right corner. These are comparisons between groups determined using Kruskal–Wallis H Tests (for when there are more than two groups), or Mann–Whitney U Tests (for comparison between two groups). Full statistical reporting is provided in Appendix B. Effect sizes for statistically significant results are also reported in Appendix B.

4.3. Qualitative Results

To help explain the quantitative findings, participants were asked to explain why they reported using particular systems, and if they indicated that they had alternative ways of measuring safety performance, they were also probed on what alternative ways they were using. Thematic analyses were performed to make sense of the qualitative data and are reported in this section. Sub-sections divide the different thematic analyses, and an explanation is provided before each one about the number of participants who were asked the relevant question.

4.3.1. Non-Use of CA005RPAS Forms

Participants who did not report any occurrences using a CA005RPAS form were asked why they did not report any occurrences using this system. There were 40 participants who were presented with this question, however, only 31 participants provided an answer. These 31 participants made a total of 49 statements regarding why they did not use the CA005RPAS form to report any occurrences.

Table 5. Reasons for not using CA005RPAS forms to report any occurrences.

Theme	Explanation	Statements		Participants		Example Quotes
		No.	%	No.	% 1
Awareness	Statements that reflect not being aware about the form	8	16.33%	8	25.81%	“I haven’t heard of it”, “I don’t know what the form is”
Own Property	Operations over own property	4	8.16%	4	12.90%	“…whilst flying on my own property”, “A test flight of a new cinewhoop on my front lawn…”
Rules-Based	Not required to do so under the CARs	9	18.37%	9	29.03%	“I didn’t realise that I had violated any rule at the time”, “Don’t need to under Part 101 as far as I know”
Seriousness	Occurrence was perceived as not being serious enough to report	25	51.02%	14	45.16%	“Very minor incidents while learning to fly my first drone”, “Nothing was damaged except my drone”
No Comment	Did not want to make a comment	1	2.04%	1	3.23%	“No comment”
Uncategorised	Statements that do not fit in any other theme	2	4.08%	2	6.45%	“I didn’t want to waste the time of officials”

¹ Percentage calculated as of the number of participants who responded to the question, which was 31 (rather than the full sample of 92 participants).

presents the themes from these qualitative answers.

4.3.2. Use of Both CA005RPAS Forms and Internal Processes

Participants who indicated that they reported some occurrences using both a CA005RPAS form and through an internal process were asked why they used both systems. Only four participants were presented with this question due to the lack of users who used both reporting systems for the same occurrence. All four of these participants provided answers, making a total of five statements.

Table 6. Reasons for using both a CA005RPAS form and an internal process.

Theme	Explanation	Statements		Participants		Example Quotes
		No.	%	No.	% 1
Organisational Policy	A procedure that the participant’s organisation requires	2	40.00%	2	50%	“Internal reporting as per company SMS policies”, “Organisational requirement”
Uncategorised	Statements that do not fit into any other theme	3	60.00%	3	75%	“To make it useful”, “Reporting was required due to the failure that occurred”

¹ Percentage calculated as of the number of participants who responded to the question, which was 4 (rather than the full sample of 92 participants).

presents the thematic analysis from their answers.

4.3.3. Use of Internal Reporting Instead of a CA005RPAS Form

Participants who indicated that they reported some occurrences internally, but not using a CA005RPAS form were asked why they reported only internally. Ten participants were presented with this question, with all ten participants providing answers, making a total of 13 statements.

Table 7. Reasons for reporting only internally and not using a CA005RPAS form.

Theme	Explanation	Statements		Participants		Example Quotes
		No.	%	No.	% 1
Organisational Policy	A procedure that the participant’s organisation requires	2	15.38%	2	20.00%	“We use a standard health and safety model… So that the club level patterns can be seen”
Response to Occurrence	The organisation’s response to the occurrence	3	23.08%	3	30.00%	“…rules and procedures refined to ensure minor incidents don’t escalate”, “…Drone was not flown again until after software reinstall”
Rules-Based	Not required to do so under the CARs	3	23.08%	3	30.00%	“Reporting via 005 was not required in the regulation”, “As both incidents were under 101 conditions there was no requirement to report them”
Seriousness	Occurrence was perceived as not being serious	3	23.08%	3	30.00%	“It had only gone less than 1 metre. I regained control and landed it”
Uncategorised	Statements that do not fit in any other theme	2	15.38%	2	20.00%	“Form didn’t exist”

¹ Percentage calculated as of the number of participants who responded to the question, which was 10 (rather than the full sample of 92 participants).

presents the thematic analysis from their answers.

4.3.4. Non-Reporting Using Either CA005RPAS Form or Internal Systems

Participants who indicated that they did not report some occurrences using either a CA005RPAS form or an internal system were asked why they did not report those occurrences using either system. There were 33 participants who were asked this question, with 27 providing answers, making a total of 42 statements.

Table 8. Reasons for not reporting occurrences using CA005RPAS forms or internal systems.

Theme	Explanation	Statements		Participants		Example Quotes
		No.	%	No.	% 1
Activity Type	Reporting was dependent upon the purpose of the flight	9	21.43%	8	29.63%	“…training flights to improve my flying skill…”, “…whilst engaged in recreational flying”
Awareness	Statements indicating lack of knowledge about occurrence reporting processes	6	14.29%	6	22.22%	“…didn’t even know reporting incidents was a requirement in UAV flight or possible”, “I have not even heard of the form”
No Internal process	Lack of organisational requirements for reporting	2	4.76%	2	7.41%	“…our internal process at work hadn’t been set up”, “Not a whole lot of internal processes in a sole trader company…”
Rules-Based	Not required to do so under the CARs	5	11.90%	5	18.52%	“Don’t need to under part 101 as far as I know”, “I am completely unaware of having to report to anyone”
Seriousness	Occurrence was perceived as not being serious	16	38.10%	11	40.74%	“Engine failure on model aircraft is a daily occurrence and does not require 005 reporting”, “…no-one has ever been killed by the recreational use of multi-rotor drones”
Uncategorised	Statements that do not fit into any other theme	4	9.52%	3	11.11%	“Not worth the hassle”, “…all safety precautions taken were always successful in mitigating any risk of injury or third party damage”

¹ Percentage calculated as of the number of participants who responded to the question, which was 27 (rather than the full sample of 92 participants).

presents the thematic analysis from their answers.

4.3.5. Alternative Ways of Monitoring Safety Performance

All participants were asked whether they used alternative ways of monitoring safety performance outside of occurrence reporting using CA005RPAS forms and internal reporting systems. Participants received different wording for this question based upon whether they were reporting occurrences or not. Eight participants out of the 13 that were reporting occurrences (including one who did not have a reportable occurrence but reported anyway) indicated that they also used alternative systems for monitoring safety performance. For those who were not reporting occurrences using CA005RPAS forms or using internal systems (whether those occurrences were reportable or not), 6 participants indicated that they used alternative systems for monitoring safety performance, while 61 participants indicated that they used no alternative systems. There were 12 participants that did not answer either version of the question. Of the 14 participants who indicated that they used alternative ways of monitoring safety performance, only 12 provided information about those alternative systems.

Table 9. Alternative systems used by participants to monitor safety performance.

Theme	Explanation	Statements		Participants		Example Quotes
		No.	%	No.	% 1
Hazard Mitigation	Actions taken to eliminate or decrease the likelihood of an occurrence	5	33.33%	3	25.00%	“Test, test, test and test every aircraft and system prior to deploying…”, “…physical inspection of models, transmitter range checks before we fly as recommended by individual clubs, and model flying rules and requirements…”
Reporting systems	Other methods of incident reporting	5	33.33%	5	41.67%	“…MFNZ rules/procedures”, “The main company (manned aviation) used Air Maestro for SMS, it made sense to also roll the the [sic] UAV division into this also”
Uncategorised	Statements that do not fit in any other theme	5	33.33%	5	41.67%	“DroneLogBook”, “…we also have a safety officer…”

¹ Percentage calculated as of the number of participants who responded to the question, which was 12 (rather than the full sample of 92 participants).

presents the thematic analysis based upon the responses of the 12 participants who were using alternative systems (whether in combination with CA005RPAS and internal reporting systems, or in lieu of those systems). These participants made a total of 15 statements.

5. Discussion

The results have indicated that a very large portion of reportable occurrences in New Zealand are going unreported, with each user on average reporting only 28.87% of their occurrences. Because of differences in the numbers of occurrences between users, the actual percentage of occurrences reported was only 14.28%. Reporting to the Civil Aviation Authority using a CA005RPAS form was particularly low, with an average of 7.62% of occurrences per user, and an observed rate of 2.74% of occurrences across this study’s sample. Given the importance of safety occurrence reporting to improving organisational systems and for identifying common hazards across the sector and how these should be regulated, it is important to increase these percentages. The focus of this discussion section is on identifying ways that safety occurrence reporting might be improved, based upon the results of this study and also by examining the academic literature. It has been divided into five core areas of discussion: (1) the role of training and assessment, (2) working with member-based organisations, (3) seriousness of occurrences, (4) regulatory considerations, and (5) exploring confidential reporting systems.

5.1. The Role of Training and Assessment

One of the most useful findings from the statistical analyses was that having completed a course on UA operations and having passed an OCA both acted to improve reporting rates and decrease the likelihood of non-reporting occurrences. For the issuance of pilot licenses in crewed aviation, there are requirements in terms of passing both theory examinations and flight examinations [50]. Pilots operating under a Part 102 Operator’s Certificate will need to have completed a theory course covering general aviation knowledge and UA-specific knowledge, as well as have passed an OCA [13]. However, no such requirement exists for operators under Part 101, whether flying commercially or not. As part of their Enabling Drone Integration discussion document of 2021, the Ministry of Transport in New Zealand are proposing to introduce a basic pilot qualification for all UA users to complete [51]. If this proposal is implemented, then there may be the opportunity to ensure that occurrence reporting is covered as part of this basic pilot qualification. Regardless of what approach is taken exactly, the results of this study suggest that the lack of any educational requirements to enter the aviation system as a UA user may be one of the drivers behind low reporting rates among users. A less formalised approach may also be to provide an online resource on CAANZ’s website about occurrence reporting for UA users, which could provide clear guidelines on what should be reported and how this information will be used by the authority.

5.2. Working with Member-Based Organisations

This study found that members of UAVNZ/Aviation New Zealand were more likely to report an occurrence using the CA005RPAS form and were less likely to non-report occurrences. There were no statistically significant differences for MFNZ members. Both organisations have codes of conduct and can self-regulate their members. The difference in the observed results is likely because UAVNZ is a professional and industry body, meaning its members are commercial operators. MFNZ is an organisation for recreational operators who want to partake in aeromodelling. Unlike UAVNZ, MFNZ does have an accident reporting form and requires members to fill it out whenever someone is injured, when the model pilot files for insurance, or when the model has deviated into controlled airspace without permission [52]. However, this is also consistent with only reporting more serious occurrences (see later discussion).

One recent occurrence analysed on the UK’s Confidential Human Factors Incident Reporting Programme (CHIRP) shows the significant amount of knowledge to be gained from occurrence reporting, analysing and discussion. In this report, a UA pilot and spotter had carried out a sizeable amount of hazard mitigation and planning prior to a training flight in a park. Regardless of this preparedness, plans were overthrown by an uninvolved person who walked across the UA’s landing area and after a delay the UA ended up landing with minimal battery remaining. Lessons learnt and shared from this could help other new drone pilots in their hazard mitigations and preparations for training flights [53].

There is the opportunity for CAANZ to work more proactively alongside UAVNZ and MFNZ to ensure that their members report occurrences, even those that may not be strictly required by either organisation’s internal policies. Working with organisations that have the ability to self-regulate may be an effective way of improving occurrence reporting, and member-based organisations have shown ability to improve performance in risk mitigation for UA operations [30]. Membership to MFNZ currently costs NZD 90 (USD 56.55) for an adult, while membership to UAVNZ currently costs NZD 253 (USD 158.98) for a company, with both organisations have other forms of membership available.

5.3. Seriousness of Occurrences

One reoccurring theme throughout this study was the perceived seriousness of the occurrence by participants. Although there is an argument for UA aircraft to have a safe place for training and currency practice where mistakes can be made, and lessons learnt, there is also an argument to be made for all occurrences being reported for statistical purposes. While these data could be recorded through a system similar to the Aviation Safety Reporting System run by NASA for the US aviation industry, CAANZ are currently the only organisation in New Zealand that collects occurrence information on the aviation industry. The importance of statistical data being available to CAANZ is highlighted in the organisation’s Regulatory Safety and Security Strategy for the 2022 to the 2027 period, where the organisation’s regulatory direction is towards an “intelligence-led” and “risk-based” assessment process, where CAANZ state “We rely in large part upon high-quality reporting by participants of occurrences” [14] p. 19. “In short, we rely on data and information to provide intelligence that informs the formation of our strategic and operational policies and plans….” [14] p. 19. Without the data or with only limited data, the decisions made by the industry regulator may not be accurate to what the industry needs for safety and growth. The shortage of occurrence data may also limit the authority’s ability to improve safety and avoid serious occurrences in the future [28].

Additionally, it is worth highlighting the importance of occurrence data being collected so that it can be shared within the industry so that other UA pilots and operators can avoid similar occurrences. The importance of understanding the UA modes, what could go wrong and the manual flight currency to recover prior to an occurrence was illustrated by the M600 Pro serious incident in 2019. In this occurrence the UA experienced a GPS-compass error where the UA reverted to a mode requiring manual control. However, both the pilot and observer did not recognise the error message or the loss of initial control as reason to take manual control and the UA continued drifting with the wind until it collided with a building [54]. The AAIB reported the operator had 10 s between the GPS-compass error and losing VLOS, and although CAAUK required the UA pilot to prove competence prior to carrying out commercial operations, there was no requirement at the time to maintain currency in emergency procedures. As a result of this serious incident the AAIB recommended UA pilots regularly practice manual emergency actions should automation ability be lost and raised the importance of recording minor occurrences where prior to this the UA had a minor similar occurrence in the weeks prior [54]. UA data and research are essential because UA are still in their infancy and the industry is growing [4].

5.4. Regulatory Considerations

A number of participants stated they do not submit occurrence reports because the current civil aviation regulations do not require an occurrence report to be submitted. This study also found those who operated under CAR Part 101, which does not require a hazard register, were associated with the non-reporting of occurrences. Those that operated under CAR Part 102, which includes a hazard register had reporting percentages higher than those who operated under CAR Part 101. Although the current New Zealand regulations may not specifically require occurrences to be reported, occurrence data are vital for regulations to evolve. A collision between a Robinson R44 helicopter and a Phantom 4 UA in Israel underlines the importance of reporting even when an occurrence happens within the law. In this accident, both operators were operating within the bounds of current regulations and were both approved to be in the areas of operation [55].

In 2019, an Alauda Airspeeder MK II entered controlled airspace affecting traffic in the Gatwick Airport area until its battery depleted resulting in an uncontrolled landing in a field. This shows the importance of report data for manufacturers to improve. During the resultant investigation, the AAIB found poor quality design and build contributed to the accident which saw parts within the kill-switch system detach from the UA circuit board [56]. The Australian-based organisation was compliant with Australian UA regulations and held a licence in accordance with CASA regulations. The UA was flying in the UK under an exemption approved by CAAUK for which no inspection by CAAUK was required [56]. On the day the exemption was issued, a test flight was carried out by the operator, without CAAUK present, where the UA experienced a heavy landing and damage to its landing gear. This was attributed to a power loss due to a battery fault. Although this previous day’s occurrence was required under the exemption to be reported to CAAUK, no occurrence report was received to either the organisations base State authority CASA, the ATSB, or to CAAUK [56]. With UA’s rapid advancement at the time, CAAUK team responsible for signing off on the exemption had little experience in the area of UA and insufficient resources [56]. This occurrence highlights how critical it is for information and in-depth knowledge of new technology for both operators and regulators, prior to approvals being granted and flights carried out. This occurrence also highlights the importance of the reporting and investigation of occurrences to develop knowledge and learning.

Like other jurisdictions have done, it would be beneficial for CAANZ to provide an explicit list of thresholds for when occurrence reporting is required, and for this to be on an accessible part of their website. Relying on users to come across AC102-1 means that those who do not have formal training are unlikely to know how to report using this system. Some of the thresholds that have been used internationally centre around the weight of the craft, airspace incursions, damage to property exceeding a certain amount, injury to uninvolved persons, loss of separation with crewed aircraft, and if the operation was commercial [15,16,18,19,57]. This study argues that any thresholds should be entirely risk-based and assessed using robust means such as the Joint Authorities for Rulemaking on Unmanned Systems’ Specific Operations Risk Assessment (commonly called JARUS SORA) [58]. The current thresholds applied in AC102-1 could be used but need to be more accessible for users, such as through CAANZ’s website, and clearly explained. A statement indicating that submitted CA005RPAS forms will not be used to assign blame or liability may also be appropriate if the purpose of data collection is to investigate and learn from occurrences, and would be more in line with the International Civil Aviation Organisation’s Annex 13 requirements for air accident investigation [59].

5.5. Exploring Confidential Reporting Systems

This study discovered that CAANZ only received CA005RPAS forms from UA users for 2.74% of the occurrences within the sample over the study time period. This was similar to the CAAUK report which found only 25 percent of the 44 reports filed each month were by the UA pilots or operators themselves [60]. However, internal reporting was higher, with 13.15% of occurrences reported. The US confidential reporting system run by NASA, the ASRS, is aimed at collecting data and identifying deficiencies within the industry with the aim of improving safety. The ASRS system treats those that voluntarily report occurrences with immunity from prosecution so long as the act was not deliberate [20]. The EU and Australia have similar systems with ECCAIRS and REPCON, which are also aimed at improving safety.

6. Conclusions

This study presents the results obtained from a sample of 92 UA users in New Zealand regarding what sorts of occurrences they have had, how these have been reported (if at all), and why they chose to report using such systems (or non-report). There were 450 reportable safety occurrences from within the sample between 2015 and 2022, with the most common occurrences being loss of control and fly-aways. Concerningly, the average reporting rate (combining both internal reporting and the use of CA005RPAS forms) per user was only 28.87%, with the actual percentage of occurrences reported being even lower at 14.28%. This suggests that individual organisations and the industry as a whole are missing out on important safety information that may help to prevent occurrences escalating into accidents that result in injury, damage to property, or collisions with other aircraft. More so, reporting rates to CAANZ via CA005RPAS forms were particularly low, with the average reporting rate to CAANZ per user at only 7.62%, and only 2.74% of all occurrences being reported to CAANZ. This suggests that the regulator may be ill-informed about how to improve safety outcomes for the UA sector as it is not receiving sufficient data to lead evidence-based decisions. The statistical analyses and qualitative questions provide some potentially fruitful avenues for improving reporting rates amongst UA users in New Zealand. Namely, standards for training and assessment, working more effectively with member-based organisations, encouraging even non-serious occurrences to be reported, reconsideration of the current CARs, and the introduction of a confidential reporting system similar to what is used in the US, EU, and Australia. While these results are only directly applicable to New Zealand, the approach taken and the themes elucidated should also help guide other jurisdictions in the pursuit of improving safety occurrence reporting amongst UA users.

7. Limitations and Future Research

The key limitation of this research was the sample size, with only 92 valid responses. While understandable due to the topic area of the research and the number of qualitative questions that were presented to users, this does mean that the study only had the statistical power to find medium effect sizes. Small effect sizes may have been missed simply because of lack of statistical power. Nonetheless, the contribution of the qualitative themes may allow for a more structured questionnaire to be developed in the future and administered to a larger sample to check that the findings are indeed generalisable, and to provide enough statistical power to find small effect sizes. While we believe that this study has captured a useful and pragmatic cross-section of UA users in New Zealand, the convenience sampling method combined with anonymous responses means that we cannot be certain that the sample is representative. Nonetheless, we have described the recruitment methods in such a way that they could be replicated, and a similar result obtained.

Aside from generalisability, it would be valuable to have more in-depth discussions with UA users to understand why non-reporting of occurrences is so high. While some important conclusions can be made based upon this research, future research may benefit from focus groups or similar approaches being employed to properly understand why these conclusions hold true and to see how behaviours might be able to be changed in the future. That can then again be developed into a more structured approach, but with a solid foundation for making assumptions.

The final limitation is to highlight that we only observed self-reporting rates from the UA users themselves. In particular for CA005RPAS forms, it is possible for someone other than the UA user to file a report to CAANZ. In the United Kingdom, only a quarter of occurrence reports came from the UA pilots themselves [60]. Thus, it is possible that CAANZ is aware of some of the occurrences identified in this study from others reporting those occurrences. Reports from UA pilots are arguably more useful as these will contain operational and technical details that will not be able to be deduced from simply observing an occurrence. Regardless, it is important to highlight that our study is focussed only on UA users self-reporting the occurrences, rather than total occurrence reporting, including from third parties. Thus, reporting numbers inclusive of other parties may be higher than those reported in this study.