Establishing Training and Certiﬁcation Criteria for Visual Observers of Unmanned Aircraft Systems

: Safe integration of Unmanned Aircraft Systems (UAS) into airspace generally occupied by manned aircraft and other aviation stakeholders is a pressing global challenge. In the United States, efforts are being made to integrate small and large UAS into the National Airspace System (NAS). Whereas regulations for the civil operation of small UAS (25 kg and lighter) have already been adopted, those for larger unmanned systems are still being crafted. Thus, a two-part mixed methods study was conducted to examine three pivotal issues in the safe operation of large UAS: (1) What kind of visual observer skills are needed to execute safe UAS operations; (2) Should visual observers involved in UAS operations receive formal training; and (3) Should visual observers be required to pass a certiﬁcation exam? In the ﬁrst phase, subject matter experts identiﬁed various vigilance, trajectory estimation and communication skills that were vital to performing visual observer duties successfully and elaborated on their training regimens. In the second phase, survey participants were approximately evenly split on the need for formal classroom/online and hands-on visual observer training. Furthermore, participants generally favored visual observers having to pass a classroom/online certiﬁcation exam, whereas they were against a practical (hands-on) exam.


Introduction
The establishment of visual observer training and certification requirements for unmanned aircraft system (UAS) operations is a component in ensuring the safe and efficient integration of UAS into the U.S. National Airspace System (NAS). As stated in the Federal Aviation Administration's (FAA) Unmanned Aircraft Systems (UAS) Operational Approval policy notice [1], visual observers are expected to be responsible for: (1) Helping UAS pilots keep the aircraft within visual line of sight and (2) Exercising see-and-avoid responsibilities by preventing the unmanned aircraft from creating a collision hazard and maintaining compliance with federal regulations, including Operating Near Other Aircraft [2], Right-of-Way Rules: Except Water Operations [3] and Right-of-Way Rules: Water Operations [4]. So, to ensure the safety of UAS operations, visual observers must be able to visually scan the airspace effectively, track aircraft and make accurate and reliable estimates of (relative) aircraft position, assess the need for a potential avoidance maneuver and communicate that need to the UAS pilot in a timely manner [5].
In 2015, the FAA released a Notice of Proposed Rulemaking for civil UAS operations [6]. The proposed language stated that flights were to be limited to small UAS (sUAS) operated within visual line of sight for the sUAS pilot during daylight hours in visual meteorological conditions. Some simple operational tasks, like flying a predefined route in dedicated airspace, allow sUAS pilots to maintain the aircraft in their line of sight throughout flight. However, more complex operational tasks, like flying an ad-hoc route in shared airspace, typically require the pilot to actively maneuver the aircraft. This can result in fewer opportunities to maintain visual contact with the sUAS and perform see-and-avoid tasks, and, consequently, reduced situation awareness and increased cognitive load. So, in the following scenarios the Notice of Proposed Rulemaking suggested that sUAS operations would necessitate a visual observer [6,7]:

•
Operations conducted above 500 feet above ground level or beyond 1500 feet laterally from the pilot-in-command must have at least one dedicated visual observer, even if visual line of sight can be maintained by the pilot-in-command.

•
When the pilot-in-command determines that one or more visual observers are necessary for the flight crewmember to maintain the safety of the operation.

•
When the pilot-in-command is (expected to be) in a "heads-down" or any situation that precludes the ability to perform see-and-avoid duties.

•
When the pilot-in-command is within an enclosure, at least two visual observers are required.
The sUAS Notice of Proposed Rulemaking also specified the following guidelines for visual observers [6,7]:

•
In order to be certified, applicants for a visual observer certificate would be required to pass a practical test with either a certified sUAS pilot or instructor.

•
Medical standards and operational limitations in this proposed rule ensure that the pilot and visual observer are capable of scanning the airspace of intended operations. Aids to vision, such as binoculars, must be used with care to ensure that the total overall viewing of the airspace is not inadvertently limited. An FAA second-class medical certificate is required for commercial operations.

•
The pilot-in-command must be able to see or ensure that a visual observer is able to see the aircraft throughout the entire flight well enough to: Know its location, determine its attitude and direction to exercise effective control, observe the airspace for other air traffic or hazards and determine its altitude.

•
The visual observer will be required to always know where the sUAS is and to discern the attitude and trajectory in relation to conflicting traffic, obstacles, or inclement weather. Because of the level of vigilance that would be required in scanning the surrounding airspace, a visual observer would be prohibited from supporting more than one aircraft operation at a time.

•
The visual observer will need to be in close proximity (within 10 feet) to the pilot-in-command and should be able to communicate directly, exchange non-verbal signals and share the same relative visual references. A backup communications system is required for operations where the pilot-in-command is in an enclosure and cannot directly see at least one visual observer.

•
Operations that involve the hand-off of UAS control and/or see-and-avoid duties among multiple pilots and/or visual observers, sometimes referred to as 'daisy-chain,' 'relay,' or 'leap-frogging,' would not be authorized under this proposed rule.
In 2016, many of these guidelines were reiterated in the FAA's Small Unmanned Aircraft Regulations [8] for operations in the NAS. Similar to the Notice of Proposed Rulemaking, these regulations state that flights are limited to sUAS (25 kg or less) operated within visual line-of-sight in visual meteorological conditions. However, the proposed operational ceiling was lowered by 30 m and, notably, all certification requirements for visual observers were dropped. This has resulted in a potential regulatory gap.
Moreover, while systems larger that 25 kg are allowed to operate in the NAS with special certification from the FAA, general guidelines for such systems are still under development. Much care must be taken with this process because operating larger unmanned platforms inherently carries more risk due to the increased momentum of the aircraft and sturdier construction compared to their smaller counterparts. In addition, larger UAS platforms are flown with the pilot-in-command inside an enclosure much more frequently. Thus, it follows that flying such aircraft in the NAS will require visual observers to maintain visual line of sight in a variety of operational scenarios and settings. Furthermore, the added risk in such operations provides a clear impetus to standardize visual observer training and establish performance benchmarks such that visual observers can become certified to serve in the operation of large UAS (heavier than 25 kg). Although prior experimental work has examined visual observers' performance in field settings (e.g., [5]) and unmanned crewmember aviation training studies exist (e.g., [9]), no prior research has investigated training and certification of visual observers.
Thus, a two-phase mixed methods study was conducted to address this timely and complex issue. Phase 1 was a phenomenological study that consisted of guided subject matter expert interviews. Phase 2 was a descriptive study which consisted of a broad survey of aviation stakeholders that focused on two pivotal issues: Should visual observers receive formal training and should visual observers be required to pass an exam?

Materials and Methods
The initial phase of this research (New Mexico State University Institutional Review Board #12314) was a phenomenological inquiry and consisted of interviewing three subject matter experts who were licensed manned aircraft pilots and also performed the roles of UAS pilot, visual observer and mission commander. These individuals were selected due to their deep understanding of all parts of manned and unmanned aircraft operations in the NAS. The interviews lasted approximately 60 min and were guided by the research questions noted in Appendix A. These interviews were transcribed and coded to examine subject matter experts' background, training, assessment of vital skills and technologies needed to perform visual observer duties, assessment of UAS operation risks in various conditions and assessment of various current and potential UAS regulations (see Tables A1-A6).
The second phase (New Mexico State University Institutional Review Board #15418) of this research consisted of a brief descriptive survey of aviation stakeholders (see Appendix B) that was distributed in aviation forums, social networking site groups of NAS stakeholders and via email (snowball sampling).

Phase 1
All subject matter experts reported that they underwent a combination of computer-based and practical hands-on training before engaging in visual observer duties. The lengths and particulars of the training programs varied but all were completed over a few days. All subject matter experts were licensed manned and unmanned aircraft pilots prior to visual observer training, thus the majority of the training was redundant with their prior knowledge.
Open-coding was performed on the interview transcripts (available in the Supplementary Materials); tables of the generated research questions and codes are noted in Appendix A. The subject matter experts unanimously stated that visual observers should be well-versed in FAA rules and regulations regarding manned and unmanned aircraft operations in the NAS. As noted in Table A5, subject matter experts stated that visual observers must establish proficiency in the following skills that are needed to successfully accomplish see-and-avoid duties:

•
Tracking unmanned and manned aircraft in various lighting and meteorological conditions: 1.
Must be able to maintain visual-line-of-sight to the aircraft 2.
Must be able to re-engage visual contact after loss and/or distraction • Scanning airspace for approaching air traffic:

3.
Must be able to shift visual depth of field • Informing pilot of impending near midair collision or some other danger with enough time for the pilot to take appropriate action:
Must use appropriate verbiage when communicating with the pilot 6.
Must be able to use global bearings and local landmarks to identify positions of UAS and other air traffic 7.
Must be able to estimate aircraft flight paths, altitudes and closure rates in order to determine the likelihood of a near midair collision 8.
Must be able to determine and communicate correct course of action and a safe deviation from the flight path to avoid a potential near midair collision

Phase 2
Phase 2 consisted of a broad survey of aviation stakeholders (see Figure 1 for participants' roles in aviation). As shown in Table 1, participants were approximately evenly split when answering the questions of whether visual observers involved in operations of UAS 25 kg and heavier should receive formal classroom/online and hands-on training (survey data available in the Supplementary Materials). While participants favored having to pass a formal classroom/online exam, the difference in Yes/No response counts was not significant. Furthermore, participants were generally against a formal practical exam; this was the only significant difference identified by χ 2 (chi-squared) tests.  Multinomial logistic regression and maximum likelihood ratio tests were then conducted to determine whether participants' roles in aviation and age predicted their responses to the questions in Table 1. The models could not significantly predict Yes/No responses to the first three questions. However, participants' roles did significantly predict responses to the final question of whether visual observers should take a formal practical exam (χ 2 (12) = 25.4, p = 0.013) and accounted for approximately 28% of the variance (Nagelkerke pseudo R 2 = 0.28). More specifically, commercial UAS pilots (χ 2 (1) = 3.6, p = 0.058) were marginally more likely to respond 'No,' whereas air traffic controllers (χ 2 (1) = 3.2, p = 0.073) and aircraft technicians (χ 2 (1) = 8.1, p = 0.004) were significantly more likely to say 'Yes.' Furthermore, older participants were marginally more likely to respond 'Yes' (χ 2 (1) = 3.55, p = 0.06).
Fewer than half of the participants provided reasons for their Yes/No answers. When examining the reasons for why participants felt that training and examination were necessary, participants noted the usefulness of classroom training in understanding NAS regulations and manned/unmanned aircraft operations, as well as increased risk due to platform size. For example, one participant stated, "Visual observers must be familiar with aeronautical information and performance standards in order to judge performance and assess hazards properly." Another stated, "Knowledge of airspace, meteorology and human factors and crew resource management will help make for a more competent and safe crew." Those participants which felt that practical training was needed stated that hands-on knowledge is irreplaceable. For example, one participant noted that, "(Visual observers need to) practice acquiring and re-acquiring UAS at different distances and altitude, (and to) understand how to determine the attitude and distance from the (ground control station) of the UAS," and another said, "(Hands-on training is needed) because practical lessons provide a more realistic perspective to real world situations." When recommending training regimens, participants mainly suggested online/print materials, such as the sUAS airmen certification standards manual [10], the ALC-451: small Unmanned Aircraft Systems (sUAS) online course [11], the Aeronautical Information Manual [12] and commercially-available visual observer self-study guides. One participant that thought practical training was necessary mentioned simulator training as an option. Furthermore, another participant stated, "(Hand-on testing should be a) continuation of the formal practical training, another day would be the test day, where a spotter would be paired up with an operator who will then ensure the spotter goes through the list of operations correctly and may even test them by trying something not allowed."

Discussion
To ensure safe UAS operations, visual observer duties primarily require them to assist UAS pilots in effective see-and-avoid. To date, visual observer training regimens have yet to be standardized and feature both practical and classroom/computer-based components. As Williams and Gildea [7] note, there are no adequate human perceptual models of aircraft detection, primarily due to the large number of variables involved. Thus, given some acceptable level of visual competency (such as medical clearance), it is likely that a visual observer training regimen should focus on non-perceptual aspects of operations, such as rules, regulations, procedures and related matters. Although 14 CFR § 107 [8], the Small UAS Rule, did not specify a requirement for visual observer certification, visual observer certification may be appropriate for personnel engaged in operations involving larger UAS, which carry added risk. While aviation stakeholders acknowledge the additional risk of operating a UAS larger than 25 kg, there is currently a lack of consensus regarding whether training and/or certification needs to be mandated. Although participants in Phase 2 somewhat favored classroom/online certification, the difference in Yes/No counts was not statistically significant. However, this trend is indicative of numerous factors that make operations of such aircraft more dangerous than their lighter counterparts. Notably, significantly more participants thought that practical (hands-on) testing was not necessary than those that did. Moreover, while UAS and manned aircraft pilots were well represented in the sample, other NAS stakeholders were not. This limitation underlines the need for additional research in this domain that carefully considers the needs of all those that occupy the sky.
Furthermore, due to the human factors of UAS operations [5,7,13,14], one training and certification solution may not be optimal. Licensed pilots have already mastered many, if not all, of the essential skills needed to carry out visual observer duties. In addition, licensed pilots have already been trained and certified in their knowledge of rules and regulations pertaining to operations in the NAS. Likewise, manned aircraft pilots have a multitude of hours performing seek-and-avoid duties, are well-versed in aviation verbiage and are trained in maintaining proper cockpit discipline. Thus, it appears that pilots and non-pilots should require different training regimens and certification criteria, akin to the distinction between licensed pilots and non-licensed individuals made by the FAA when persons apply for sUAS certification.  Table A2. Research Question: What types of training and certification did participants receive prior to engaging in aviation duties?

Manned Pilot Training Program
Training program involves both classroom and field training "I mean the pilot training was excellent and I had previously taken ground school before I did that. . . . So, the pilot training and just actually working with aircraft, the flight training, the hands-on stuff."

Open Code Properties Example
Non-practical Testing Classroom or computer-based testing "Paper tests. Performance tests are useful vs. paper tests. Knowledge tests; I understand they're important but sometimes there is too much weight put on them especially for people that don't take test well. And there are a lot of people in high stress situations in regards to test taking that just don't do well." " . . . they required us to take weekly quizzes that was fine. An open book quiz annually that was harder than it sounds because they ask very obscure questions you had to go through a bunch of books to find it, that was fine. But then they gave us a really really difficult instrument exam every year and I'm an instrument instructor and they gave us an instrument exam that had nothing to do with reality or flying UAV's or anything and it was a thing everybody had trouble with and never needed. So, it was very frustrating and a waste of time." "I thought that the 55-pound weight was too big for the speed that they're allowed. That's a lot of mass and can cause a lot of damage really fast and danger . . . . There's no doubt that the training required to fly the Predator is a lot more extensive than what is required to fly a multi-copter. It was, it is and has to be."

Components/Configuration
Fixed wing aircraft vs multi-rotor aircraft; Rotor guards "I have a multi-copter and I've crashed it twice and in both cases I threw a propeller blade. I didn't do anything wrong; I didn't hit anything . . . . you're just flying along and one blade came off. It didn't hit anybody it just turned upside down . . . I've never had a wing come off an airplane, if it did that'd be catastrophic so you got to weigh that." Distance/Line of Sight Ability of UAS pilots and/or visual observers to see the UAS and scan surrounding airspace; Weather; Time of Day; Proximity to Hazards "We're flying at low altitude and for our flights we generally have three observers. We have one on each end of the flight area and one would be external pilot. So, we have people around that are an extra set of eyes. And the external pilots found that even with the larger aircrafts they didn't want it as far away as they could possibly see it. And then the radio controls too, the larger one is probably large enough that it could get out of range of the radio controls." "So the size [matters], an Aerostar or Tigershark [has] got all the light systems on it with a strobe then the risk I don't really foresee that risk being any different than during the day. But if you're looking at something like the Phantom, it's got lights on it, you can see it from the ground but is a pilot in the air going to see it? Nope. So large UAS, sure I don't see any reason why a large system can't fly beyond line of sight with light systems on it at night, as long as they're equipped with transponders, ASB, something of that sort and well-lit but small systems to be honest there is no reason to fly them at night."