Assessing and Advancing Safety Management in Aviation

A safety management system (SMS) is the overall set of procedures, documentation, and knowledge systems as well as the processes using them, which are employed within an organisation to control and improve its safety performance. Safety management systems are often observed as being bureaucratic, distinct from actual operations, and being too much focused on the prevention of deviations from procedures rather than on the effective support of safety in the real operational context. The soft parts of advancing safety in organisations, such as the multitude of interrelations and the informal aspects in an organisation that influence safety, are often only considered to a limited extent. As a way forward, this paper presents two coupled approaches. Firstly, a generic tool for assessing the maturity of safety management of aviation organisations is presented, which accounts for recent insights in effectively incorporating human factors. This assessment tool provides insight into the strong and weak topics of an organisation’s SMS. Secondly, an overview is given of a range of approaches that aim to improve the safety of aviation organisations by strengthening relevant organisational processes and structures, with a focus on human factors. The relations of these approaches with SMS are discussed, and the links with topics of the SMS maturity assessment tool are highlighted.


Introduction
A safety management system (SMS) is the overall set of procedures, documentation, and knowledge systems as well as the processes using them, which are employed within an organisation to control and improve its safety performance. The aviation industry has an extensive tradition on processes and procedures towards assuring the safety of air transport operations. The International Civil Aviation Organization (ICAO) is the agency of the United Nations that develops Standards and Recommended Practices (SARPs) for a safe, efficient, secure, economically sustainable, and environmentally responsible civil aviation sector. The ICAO SARPs for SMS [1,2] discern four main parts (Table 1): P1 safety policy and objectives; P2 safety risk management; P3 safety assurance; and P4 safety promotion.
SMS part P1 describes the safety objectives of an organisation and the principles, processes, and methods of the organisation's SMS to achieve them, including the accountabilities and responsibilities of the organisation's personnel. Part P2 describes the safety risk management processes to assure that the safety risks encountered in aviation activities are controlled to achieve an organisation's safety performance targets. Key processes are hazard identification and safety risk assessment, which lead to requirements for system design and operations that need to be assured by the organisation. Such safety assurance processes and activities to determine whether the SMS is operating according to expectations and requirements are addressed in part P3. It includes the continuous monitoring of its internal processes and operating environment to detect changes or deviations that may introduce new safety risks or the degradation of existing risk controls. Such changes or deviations then need to be addressed by the safety risk management process (P2). Part P4 describes the training, education, and communication in the SMS to achieve a positive safety culture with proper knowledge and awareness of all people in the organisation. There exist interrelations between each of the SMS components. Table 1. SMS parts in ICAO SARPs [1,2].

P1. Safety Policy and Objectives
Management commitment Definition of a safety policy and safety objectives. The safety policy describes the organisational commitment regarding safety, the provision of resources for implementation of the safety policy, the safety reporting procedures, and the delineation between acceptable and unacceptable behaviour. The safety objectives form the basis for safety performance monitoring in reflection of the organisation's commitment.

Safety accountabilities and responsibilities
Designation of an accountable executive for the implementation and maintenance of the SMS, the definition of lines of safety accountability in the organisation, the definition of management levels with the authority to decide about safety risk tolerability, and the documentation and communication of the safety responsibilities, accountabilities, and authorities in the organisation.

Appointment of key safety personnel
Appointment of a safety manager who is responsible for the implementation and maintenance of the SMS.
Coordination of emergency response planning Establishment and maintenance of an emergency response plan for accidents and incidents in aircraft operations and other aviation emergencies, which is well coordinated with the emergency response plans of related organisations.

SMS documentation
Development and maintenance of an SMS manual that describes the safety policies and objectives, SMS requirements, SMS processes and procedures, and the related accountabilities, responsibilities, and authorities. This documentation also includes SMS operational records.

P2. Safety Risk Management
Hazard identification Developing and maintaining a process for the identification of hazards associated with an organisation's aviation products or services, including reactive and proactive methods.

Safety risk assessment and mitigation
Developing and maintaining a process that ensures analysis, assessment, and control of the safety risks associated with identified hazards.

P3. Safety Assurance
Safety performance monitoring and measurement Developing and maintaining means to verify the organisation's safety performance by relating safety performance indicators with safety performance targets and to validate the effectiveness of safety risk control.

The management of change
Developing and maintaining a process to identify changes that may affect the level of safety risk associated with an organisation's aviation products or services and to identify and manage the safety risks that may arise from those changes.
Regulators, such as the Federal Aviation Administration (FAA) and the European Union Aviation Safety Agency (EASA), provide standards and guidance material for safety management to aviation service providers that build upon the ICAO SARPs [3][4][5][6][7].
The interacting components of an SMS relate with the people and processes at the various levels in an organisation (Figure 1), including staff, middle management, and executives. In such organisational contexts, safety intelligence aims to ensure that managers have the right information when making decisions that can affect safety. Safety mindfulness is about creating flows of information that support the staff's capability to remain mindful of safety when carrying out their activities and that support managers in maintaining safety oversight. Safety culture refers to the norms, values, and practices shared by all people in the organisation in relation to safety and risk. Typically, a level of safety in an organisation is materialized by the way that operations are conducted ("work as done" rather than "work as imagined") at the sharp end, which is influenced by decisions and operating conditions created by middle and top management at the blunt end. As such, the level of safety is influenced by the safety culture, by the safety mindfulness and safety intelligence processes, and by the safety management processes. have the right information when making decisions that can affect safety. Safety mindfulness is about creating flows of information that support the staff's capability to remain mindful of safety when carrying out their activities and that support managers in maintaining safety oversight. Safety culture refers to the norms, values, and practices shared by all people in the organisation in relation to safety and risk. Typically, a level of safety in an organisation is materialized by the way that operations are conducted ("work as done" rather than "work as imagined") at the sharp end, which is influenced by decisions and operating conditions created by middle and top management at the blunt end. As such, the level of safety is influenced by the safety culture, by the safety mindfulness and safety intelligence processes, and by the safety management processes. The effort in maintaining and documenting an SMS can be considerable, as it includes a multitude of components and relates in various ways to the design and conduct of operations in an organisation. Notwithstanding the variety of components considered in SMS standards, including human factors and safety culture, safety management systems are often observed as being bureaucratic, distinct from actual operations, and being too much focused on the prevention of deviations from procedures rather than on the effective support of safety in the real operational context. The soft parts of advancing safety in organisations, such as the multitude of interrelations and the informal aspects in an organisation that influence safety, are often only considered to a limited extent. A resulting reductionist view of safety management may separate human and technical factors and thus undermine holistic views on safety in organisations [8].
A holistic view on human contributions to safety is advocated by safety management from a Safety-II perspective [9]. A key foundation of Safety-II is the recognition that human performance, individually or collectively, is variable, and that such performance variability is indispensable to effectively get the work done in the actual working context of organisations. Resilience engineering [10] supports the Safety-II perspective, indicating the ability of a sociotechnical system to adjust its functioning to sustain required operations notwithstanding changes and disturbances and the "engineering" of the sociotechnical system to achieve such ability. Resilience engineering stresses the key role of perfor-  The effort in maintaining and documenting an SMS can be considerable, as it includes a multitude of components and relates in various ways to the design and conduct of operations in an organisation. Notwithstanding the variety of components considered in SMS standards, including human factors and safety culture, safety management systems are often observed as being bureaucratic, distinct from actual operations, and being too much focused on the prevention of deviations from procedures rather than on the effective support of safety in the real operational context. The soft parts of advancing safety in organisations, such as the multitude of interrelations and the informal aspects in an organisation that influence safety, are often only considered to a limited extent. A resulting reductionist view of safety management may separate human and technical factors and thus undermine holistic views on safety in organisations [8].
A holistic view on human contributions to safety is advocated by safety management from a Safety-II perspective [9]. A key foundation of Safety-II is the recognition that human performance, individually or collectively, is variable, and that such performance variability is indispensable to effectively get the work done in the actual working context of organisations. Resilience engineering [10] supports the Safety-II perspective, indicating the ability of a sociotechnical system to adjust its functioning to sustain required operations notwithstanding changes and disturbances and the "engineering" of the sociotechnical system to achieve such ability. Resilience engineering stresses the key role of performance variability by human operators to adjust for changing demands and conditions in the working context. Based on an understanding of performance variability and disturbances in an operational context and their safety implications, the "engineering" of resilience typically encompasses activities such as training, creating flexibility, providing sufficient resources, establishing a learning culture, etc. Main challenges for organisations towards improving their safety performance often lie in keeping the organisation resilient for changing demands and conditions and advancing the informal, human-related, soft elements. However, Safety-II has not provided a ready-made, pragmatic solution that meshes well with SMS and operational processes.
In order to persuade organisations to change the way they do business safely, two things are needed. The first is a compelling reason to change, and the second is a straightforward way to change that is not too disruptive to their existing business model. These two aspects are, broadly speaking, the "why", followed by the "what and the how". The reason to change is often not so easy to see from inside an organisation. However, organisations and, in particular, their leaders always keep an eye on the competition and wish to ensure they are keeping up with their peers and competitors. A safety maturity scheme is therefore one way in which organisations can perceive that they can do better, and if such schemes are backed up by regulators, then there can be an industry-wide drive to "raise the bar" in areas where there are common weak spots. This has to an extent already happened in air traffic organisations (particularly in Europe) in the field of safety culture, which was brought into the existing safety maturity schemes, and now almost all ANSPs in Europe carry out periodic safety culture surveys [11].
The "what and the how" is the more methodological part of this paper, focusing on key human factors approaches relating to safety culture, mindfulness, and safety intelligence, which can be brought inside an organisation's existing SMS without too much disruption. The following sections "unpack" these two approaches.
As a way forward, therefore, this paper presents two coupled approaches. Firstly, a generic tool for assessing the maturity of safety management of aviation organisations is presented, which accounts for recent insights in effectively incorporating human factors. This assessment tool provides insight into the strong and weak topics of an organisation's SMS. Secondly, an overview is given of a range of approaches that aim to improve the safety of aviation organisations by strengthening relevant organisational processes and structures, with a focus on human factors. The relationships between these approaches and SMS are discussed, and the links with topics of the SMS maturity assessment tool are highlighted.

Development of an SMS Maturity Assessment & Refinement Tool (SMART)
This section deals with the objective of assessing the level of maturity of a safety management system within a variety of aviation organisations, including recent insights for effectively incorporating human factors in safety management. There are several existing methods to assess the maturity of a safety management system, which include the following: • EASA questionnaire. As part of the acceptable means of compliance and guidance material for the implementation and measurement of safety key performance indicators [12], EASA has published a questionnaire for measurement of the effectiveness of safety management. The questionnaire is based on a maturity survey in the ATM Safety Framework [13], which was developed by EUROCONTROL to support ANSPs in assessing the maturity of their SMS. This maturity survey comprises eleven study areas. The study areas are specified in more detail by one to four topics per study area and 26 topics in total. For each of these topics maturity levels are defined on a five-point scale.
• CANSO maturity scheme. The Civil Air Navigation Services Organisation (CANSO) has published a Standard of Excellence in SMS [14]. It includes a definition of SMS maturity along five levels for its thirty-six SMS objectives. The development of the CANSO scheme used the above-mentioned publications of EASA [12] and EUROCON-TROL [13], but it has also added some items, and it provides some useful formulations. • Shell SMS assessment. The SMS HSE MS self-assessment questionnaire of Shell [15] lists safety management topics and related current aviation practices, typical supporting evidence, and interpretation/guidance for aircraft operators. It consists of thirty-two topics distributed over eight groups, which are scored on a four-point scale.
These methods for SMS maturity assessment are focused on particular air transport organisations, namely air navigation service providers (EASA questionnaire, CANSO maturity scheme) or aircraft operators (Shell SMS assessment). All approaches for SMS maturity assessment are mostly based on traditional perspectives on safety and safety management, and they lack insights from recent research in advancing safety management.
To overcome these limitations, a generic air transport SMS Maturity Assessment and Refinement Tool (SMART) was developed. For this development, the EASA questionnaire [12] and the CANSO SMS maturity scheme [14] were used as a basis. Their topics were generalized, reformulated, and combined where appropriate. For instance, specific Air Traffic Management (ATM) systems, procedures, or relationships were removed to allow more general use of the questions. Next, we analysed what questions from the Shell questionnaire [15] could add new aspects, leading to addition of some questions. Furthermore, insights from developments in research in human factors, Safety-II, and resilience engineering were used as a basis for some topics. Finally, the topics in SMART were structured following the SMS components of ICAO Annex 19. Details on the development and coupling with the earlier questionnaires of [12,14,15] can be found in Appendix A of [16].
An overview of the topics for each of the SMS components in SMART is shown in Figure 2, including nine topics for Safety Policy and Objectives, six topics for Safety Risk Management, nine topics for Safety Assurance, and eight topics for Safety Promotion. For each of these 34 topics, five levels of maturity from A to E are defined, ranging from none or a bare minimum approach for addressing the topic (level A) to a most advanced way for addressing the topic, well beyond what is required (level E). A listing of all topics and descriptions of the levels A to E is provided in full detail in Appendix A, and some examples are provided next.
For example, the levels for senior management visibility and involvement (topic 1.8) range from senior management not communicating at all about safety performance to senior managers driving the process for safety excellence, being role models for safety, ensuring all staff have safety results and activity targets in their appraisals, and being personally involved in safety improvement efforts. For safety risk control (topic 2.3), the highest level includes a Safety-II perspective, wherein performance variability is considered as normal and used as a basis to define a range of performance indicators that reflect the work-asdone in the organisation. Levels of safety assurance regarding variations with respect to procedures and standards (topic 3.7) range from complete denial of such variations to systematic processes for organisational learning from variations. Levels for safety lessons (topic 4.5) range from safety lessons being learned only by those who experience them to systematic processes for gathering and effective dissemination of safety lessons. For example, the levels for senior management visibility and involvement (top range from senior management not communicating at all about safety performance ior managers driving the process for safety excellence, being role models for safe suring all staff have safety results and activity targets in their appraisals, and bein sonally involved in safety improvement efforts. For safety risk control (topic 2.3), the est level includes a Safety-II perspective, wherein performance variability is cons as normal and used as a basis to define a range of performance indicators that refl work-as-done in the organisation. Levels of safety assurance regarding variations w spect to procedures and standards (topic 3.7) range from complete denial of such tions to systematic processes for organisational learning from variations. Levels for lessons (topic 4.5) range from safety lessons being learned only by those who expe them to systematic processes for gathering and effective dissemination of safety les

SMART Application Schemes
SMART aims to support continuous improvement of SMS and safety in org tions. The basis of the tool use is the multi-level perspective on SMS topics (Appen incorporated in a questionnaire. In the questionnaire, a user is asked for each topic dicate a level A to E and to provide an explanation that justifies the selection. T portance of providing an explanation needs to be stressed to a user since this is trigger thinking and discussion regarding the way that the SMS topics are handled

SMART Application Schemes
SMART aims to support continuous improvement of SMS and safety in organisations. The basis of the tool use is the multi-level perspective on SMS topics (Appendix A) incorporated in a questionnaire. In the questionnaire, a user is asked for each topic to indicate a level A to E and to provide an explanation that justifies the selection. The importance of providing an explanation needs to be stressed to a user since this is key to trigger thinking and discussion regarding the way that the SMS topics are handled in an organisation and may be improved. In particular, the evaluation of levels of the topics in SMART is a means towards continuous improvement, and achieving maximum scores for the topics is not an end in itself.
The tool has been designed for use in several application schemes or configurations of user groups:
These application schemes are presented next. This paper does not provide results for their use in organisations.

Single User-Single Organisation
The single user-single organisation case is the most straightforward application scheme. The single user can, for instance, be a safety manager or a senior member of staff from a safety department. The single user scores the topics and provides a justification. The results and the justification process help him/her to understand the strong and weak points of the organisation's SMS. This can support the identification of ways to improve weak points of the SMS.

Multiple Users-Single Organisation
The multiple users-single organisation case uses the tool in a more advanced way, as illustrated in Figure 3. It consists of the following three elements: • SMART survey. The questionnaire is completed by personnel from whom it is expected to have a reasonable overview over SMS topics, such as safety managers, staff of a safety department, and other managers. The objective is to obtain a multitude of opinions about the SMS topics from different perspectives in the organisation.
People are asked to provide their opinions for the scores of the SMS topics as well as explanations of their findings. • SMART analysis. The results of the survey are collected and analysed by independent analysts with expertise in SMS. This analysis provides statistics of the scores on the various SMS topics, pointing to views on strong and weak points and to differences in opinions on the SMS topics. The analysis of the explanations provided by the participants leads to initial insights in reasons for the scores. • SMART workshop. The results of the analysis serve as the main input for one or several workshops with participants of the survey, depending on the size and distribution of the survey group. Each workshop is facilitated by the researchers who performed the SMART analysis. The objectives of the workshop are to achieve an improved understanding between the participants of the way that the SMS works in practice in the organisation and to arrive at ways to improve the organisation's SMS and the ways that it can be effectively applied. Discussion of the differences in the views of the participants is important to arrive at these ends.

Multiple Users-Multiple Organisations
The multiple users-multiple organisations case extends the application of SMART from a single to multiple organisations. The objective is to improve the safety management systems of different organisations and to improve the interactions between the SMSs of interacting organisations. It is expected that a workshop format involving participants from different organisations is a suitable way to achieve these objectives. A basis of such workshop can be results from single-or multiple-users cases per organisation. The workshop is an opportunity for organisations to learn from the strong points of other organisations and to improve the inter-organisational safety management relations.

Advancing Safety Management
This section addresses the objective to provide an overview of a range of methods that aim to improve the safety of aviation organisations by strengthening relevant organisational processes and structures and especially by supporting the ways that people at all levels in the organisation understand, behave, and interact in safety-relevant situations. These methods cover the following research areas: top management, middle management, safety mindfulness, safety dashboards, safety culture, and agile response capability. For each of these areas, the links with safety management systems are discussed. Finally, the connections of the methods with the topics of the SMS Maturity Assessment Research Tool are explained.

Top Management-Safety Wisdom
Interviews were held with sixteen senior executives from organisations across the European aviation industry (airlines, airports, air traffic management, regulators, research) about their perspectives on safety performance in their organisations [17]. Each executive was interviewed by two interviewers with backgrounds in aviation and organisational safety. A topic guide was used to maintain standardised questions, such as "What kind of safety information do you consult?", "How does safety information feed into the decision-making process?", and "How do you monitor the impact of business decisions on safety?". These interviews have provided a range of views by top managers of air transport organisations about their roles in safety management, such as maintaining safety under pressure, searching for evidence, seeing around the corner, and accountability and responsibility at the top [17]. These insights and practices, known as safety wisdom (to distinguish them from safety intelligence), can inspire and support top managers in improving their contributions and leadership in the safety performance and safety management of their organisation.

Advancing Safety Management
This section addresses the objective to provide an overview of a range of methods that aim to improve the safety of aviation organisations by strengthening relevant organisational processes and structures and especially by supporting the ways that people at all levels in the organisation understand, behave, and interact in safety-relevant situations. These methods cover the following research areas: top management, middle management, safety mindfulness, safety dashboards, safety culture, and agile response capability. For each of these areas, the links with safety management systems are discussed. Finally, the connections of the methods with the topics of the SMS Maturity Assessment Research Tool are explained.

Top Management-Safety Wisdom
Interviews were held with sixteen senior executives from organisations across the European aviation industry (airlines, airports, air traffic management, regulators, research) about their perspectives on safety performance in their organisations [17]. Each executive was interviewed by two interviewers with backgrounds in aviation and organisational safety. A topic guide was used to maintain standardised questions, such as "What kind of safety information do you consult?", "How does safety information feed into the decision-making process?", and "How do you monitor the impact of business decisions on safety?". These interviews have provided a range of views by top managers of air transport organisations about their roles in safety management, such as maintaining safety under pressure, searching for evidence, seeing around the corner, and accountability and responsibility at the top [17]. These insights and practices, known as safety wisdom (to distinguish them from safety intelligence), can inspire and support top managers in improving their contributions and leadership in the safety performance and safety management of their organisation.
The strongest links of top management safety wisdom are with SMS part P1 (safety policy and objectives). This concerns the top management role in defining and keeping up-to-date the authorities, responsibilities, and accountabilities for the management of safety in the organisation. The highest organisational level must recognise its role in the SMS and actively support the development, implementation, maintenance, and promotion of the SMS throughout the organisation (including support departments). There should be continuous monitoring of the effectiveness and efficiency of the various SMS processes, including the search for "weak signals" about safety, and senior management should take effective measures to control the performance of the SMS. Senior management visibility and involvement are important for effective safety management. A strong safety commitment is characterized by senior managers actively participating in and/or supporting safetyrelated activities, such as training, reward and recognition schemes, safety workshops, safety conferences, and audits. They jointly develop and discuss both safety results and activity-improvement targets with staff and company contractors. They should be fully aware of the high priority areas for improvement identified in the SMS and the status of the follow-up remedial programme. Senior managers drive the process for safety excellence, and they are role models for safety. In relation to SMS part P3, concerning safety assurance, senior management support an integrated risk management system for all relevant key performance areas, such as finance, quality, security, and environment. This systematically addresses all types of risks and their relations, including assessment of costs associated with accidents and incidents and of costs and benefits of risk mitigating measures.

Middle Management
Interviews were held with 48 middle managers (who are not safety managers) at a range of European ANSPs, aircraft manufacturers, airlines, and airports about their views on safety and their roles and ways of working in the organisation [18,19]. Semistructured interviews were conducted, and a qualitative content analysis method was used through a data-driven coding frame. It was analysed how middle managers take safety into consideration in their work, including their information management, decision making, and influencing of others. Middle managers have to balance multiple goals in getting the work done, and their leadership role is crucial for the level of safety performance that is achieved in operations. This led to guidance on how best to harness the middle management role in organisational safety.
There are various SMS topics that are relevant to the work by middle managers. It follows from the interviews that for many of these topics, middle managers do not relate to the formalities of a SMS but rather feel responsible for getting the work done while dealing with uncertainty and disturbances that may affect the operations. As such, they serve multiple goals (including productivity, finance, quality, security, environment), of which safety is only one. Doing so, they perform a kind of integrated risk management although without necessarily explicitly assessing all risks. The management of information and knowledge transfer is focused on the overall practices for getting the work done appropriately without special focus on safety. They recognized that it is important to have experts that have a good knowledge of the overall operations and the interactions between its human and technical elements in order to get the work done effectively, efficiently, and safely.
The interviewed middle managers indicated that for understanding the level of safety that they manage, they typically rely more on their own judgements and those of key staff than on using formal safety indicators. They considered safety management practices to be mostly relevant for safety management personnel although some middle managers took safety management training to obtain a broader and/or deeper safety picture. An openreporting culture was considered important by the interviewed middle managers as part of a management style where one listens to the field. The topic where the strongest connection with the formal SMS seems to exist is for management of change. The interviewed middle managers considered it important to involve all actors impacted by a particular change and to recognize when a change would require a safety risk assessment.

Safety Mindfulness
The safety mindfulness concept describes flows of information that support mindfulness about safety in an organisation and how such mindfulness impacts decisions and actions to mitigate risks both directly within operations as well as in the management of system improvement [20][21][22][23][24]. The flow of information and the sharing and transformation of knowledge that is grounded in operational constraints represent the core activity. This dynamic information sharing requires nurturing by supportive social relations: both good coordination and leadership across relevant operational units as well as amongst management groups and teams dedicated to improvement. Horizontal escalation supports sharing of information across interdependent operations (e.g., between different teams) such that risk-inducing interdependencies may be recognized and avoided. Vertical escalation supports sharing insights and observances at the sharp end, including "weak signals", up the hierarchy. This leads to better and more coherent safety intelligence throughout the organisation and "safety as experienced" rather than "safety as imagined". Metrics were developed in [21] to measure safety mindfulness, ranging from metrics for safety mindfulness principles to metrics that focus on the risk perception and its adaptation of people in the organisation. Questionnaires and workshops are the means to acquire such metrics. Two software applications were designed [22,24]: (1) a social media app that allows operators to share and learn safety-related experiences in their work and (2) an improvement manager app that supports information transfer for change management. The safety mindfulness metrics and apps have not yet been applied in operations.
Given the broad scope of the safety mindfulness concept describing the flows of safety-related information as well as the governance structure for accountability, decision making, and acting in support of safety, the safety mindfulness concept relates with all SMS parts. The safety mindfulness metrics may be used as safety indicators in support of safety assurance (SMS part P3). The safety mindfulness IT applications, such as the social media app that would allow operators to share experiences in their work, may support various SMS topics. They may support the identification of hazards and disturbances for safety risk management (SMS part P2), as the apps can be used to report hazards, disturbances, and changing circumstances by staff. In support of safety assurance (SMS part P3), such apps may be used to report safety occurrences and to suggest corrective actions by staff. Furthermore, they may be used by staff to report variations with respect to procedures and standards in operational conduct and to suggest improvements. In support of management of change, the improvement manager app was designed to provide a structured way to control information flows for change processes. Recording and dissemination of safety lessons learned (SMS part P4) may be supported by the safety mindfulness app.

Safety Dashboards
The core purpose of safety dashboards is to provide a structured overview of safetyrelevant information that effectively supports decision-making processes that may affect safety in an organisation. Best practices for the design and use of safety dashboards at Executive Board level were identified, leading to several prototypes for air navigation service providers [25,26]. Key areas on safety dashboards address (1) operational safety, e.g., graphs of monthly incidents, top contributory factors, and hotspot information; (2) people and culture, e.g., trends of reporting rate for safety occurrences, participation in safety training, and progress on safety culture initiatives; (3) technical systems, e.g., number and duration of technical issues and ratio between planned/unplanned maintenance; and (4) change management, e.g., trend of corrective actions, status of change projects, and top change recommendations coming from external bodies (e.g., regulator). These results support organisations to develop and fine-tune safety dashboards to provide the information that is suitable for their needs. As such, they can lay an effective foundation for the information transfer in the organisation's safety management system and ensure the key safety intelligence rises to the top of the organisation.
There exist various ways by which safety dashboards can support safety management. In support of safety risk management (SMS-P2), they provide data that support the identification of threats to operations. The research has highlighted a need for a quantitative risk model capable of connecting multiple indicators that provides an overall risk picture, which may be integrated in future safety dashboards [25]. For safety assurance (SMS-P3), dashboards provide high-level data of operational safety, human factors, technical systems, and change management. For integrated risk management, the research has highlighted a need for integrating indicators for different KPAs, thus visualizing capacity, economic, and safety indicators in a single dashboard. In support of safety culture measurement and an improvement (SMS-P4), safety dashboards include indicators for monitoring the progress of safety culture actions and the participation to safety initiatives.

Safety Culture
The entirety of norms, values, and practices with respect to safety and risk by people in an organisation forms an important foundation for the effectiveness of its safety management and the level of safety that can be achieved in its operational conduct. As such, safety culture is at the core of the human contributions to safety in any air transport organisation. Understanding the weaknesses and strengths in an organisation's safety culture and effecting means to support weaker aspects are key contributions towards avoiding safety occurrences. An ATM safety culture survey approach was extended to other areas of the air transport system, including airlines, airport organisations, and airframe manufacturers. A questionnaire-based analysis was done in a pan-European safety culture study of pilots [27]. This involved more than 7000 airline pilots who indicated their level of agreement on a 5-point Likert scale with about 55 statements concerning safety culture topics, such as just culture and reporting, communication and learning, risk handling, and commitment to safety. A "safety culture stack" approach was developed for measuring, evaluating and improving safety culture of various organisations in the aviation system [28,29]. First, questionnaire-based analyses were performed for each of the organisations. Next, these measurements set a basis for interaction and collaboration in workshops between these organisations, with the objective of making the total stack system safer. This was applied for a case at Luton Airport, involving key organisations, such as airport, airline, ground handlers, air traffic services, fire services, and de-icing services.
There are two types of relations of safety culture with safety management systems. The first type considers the intrinsic relation of safety culture with safety management. It stipulates that the effectiveness of safety management depends on the safety culture in the organisation. If elements of safety culture, such as safety commitment or willingness to report incidents, are at low levels, it may for instance be harder to achieve effective safety assurance (P3) and safety promotion (P4). Additionally, the safety management system can influence the level of safety culture; in particular, the actions and results of safety management may influence the beliefs of the personnel and the way that the work is done in the organisations. As an example, if there is inadequate just culture in an organisation, staff may be reluctant to report incidents or near misses.
The second type considers the relation with SMS parts. SMS part P4 of ICAO Annex 19 [1] explicitly describes training, education, and communication as part of safety management to achieve a positive safety culture with proper knowledge and awareness of all people in the organisation. A key aspect is a safety culture measurement and improvement programme, where safety culture is assessed regularly, weaknesses are identified, and there is a continuing safety culture improvement. This considers all safety culture dimensions, including the promotion of a just and open culture for reporting and investigation of occurrences, which as noted above, is an important basis for effective safety management. The safety culture stack approach adds relations regarding the interactions with external partners. These concern the improvement of safety-related interfaces with external partners (e.g., identification of newly developing risks), the harmonisation of procedures (e.g., for ground handling services at an airport) to reduce complexity whilst allowing for flexibility, active sharing of safety data and information, and sharing and learning best practices on operational safety and SMS practices with industry stakeholders.

Agile Response Capability
Development of approaches for an agile response capability (ARC) addresses events focusing on sudden crises. The ARC of actors in the air traffic system refers to their ability to anticipate/detect events, control them, and "bounce back" after they have happened. Doing so involves adapting their organisation and resource use, learning, and self-monitoring as well as the ability to coordinate activities with other actors. New approaches were developed for the development of exercises and preparedness plans for air transport crisis situations [30][31][32]. These structured approaches support single organisations and multiple interacting organisations to augment their intra-and inter-organisational capability of detecting and flexibly responding to dynamic crisis scenarios. They support air transport organisations to be better prepared for crisis situations and to be more resilient and resourceful when crises occur.
There are several relations of the ARC methods with SMS components. The strongest relation exists with the development of emergency/contingency response planning and exercises in SMS part P1. The ARC methodology provides approaches for development of guidance for analysing and aiding the intra-and inter-organisational capability of detecting and flexibly responding to dynamic crisis scenarios. As a core method, ARC supports the development of scenarios for exercises and preparedness planning, including combinations of hazards and disturbances in crisis situations (SMS-P2). It is a key aspect of the ARC approaches to analyse and support the interactions between different organisations for their agile response capability. As such, they improve safety-related interfaces and sharing of safety information with external parties as part of SMS parts P3 and P4. Training in safety and safety management (SMS-P4) is also supported by the ARC approach.

SMS Impact of the Organisational Safety Approaches
The previous sections provided an overview of approaches to advance safety in organisations. A qualitative evaluation of these approaches was made for the topics of the SMS Maturity Assessment Research Tool as presented earlier in Section 2. This qualitative evaluation considers three levels for the extent to which the approaches addressed an SMS maturity topic: (1) a topic was not addressed; (2) some aspects of a topic were addressed; and (3) the core of a topic was addressed. A graphical description of the results of this qualitative evaluation is shown in Figure 4, where darker colours depict a larger impact.
An overview of the SMS topics that are addressed most effectively by the research streams for advancing safety in organisations is provided in Table 2. For each of these topics, the research streams and related sections of this paper as well as references to papers and reports are provided. It can be observed that the research streams especially support the SMS parts P1 safety policy and objectives, P3 safety assurance, and P4 safety promotion. The considered research streams have not focussed in detail on topics of P2 safety risk management, such as hazard identification and risk assessment. While this is a key part of safety management, it is already covered by a considerable volume of literature, and the considered research streams emphasized less-researched organisational processes that impact safety.

Discussion
Safety management systems are often observed as being bureaucratic, distinct from actual operations, and being too much focused on the prevention of deviations from procedures rather than on the effective support of safety in the real operational context. The soft parts of advancing safety in organisations, such as the multitude of interrelations and the informal aspects in an organisation that influence safety, are often only considered to a limited extent. As a way forward, this paper has presented a generic tool for assessing the maturity of safety management of aviation organisations and coupled this with recent research approaches that aim to improve the safety of aviation organisations by strengthening relevant organisational processes and structures, with focus on human factors.
The development of the SMS Maturity Assessment Research Tool (SMART) was based upon the SMS structure posed by existing standards [1,2], and it built upon existing maturity surveys either focussed on air navigation service providers [12,14] or aircraft operators [15]. The survey questions were generalized for use by general service providers in the aviation domain (airlines, airports, air traffic control centres, and related organisations) and reformulated where appropriate using recent insights in human factors and safety management. The use of existing standards and surveys as the basis of SMART and the integration of recent insights herein implies a gradual change in assessment of the effectiveness of approaches for advancing safety in organisations. Such evolution in thinking about safety management is needed to be able to achieve effective improvements. Given the large safety interests in the aviation domain, revolutionary changes in safety management are very unlikely to attain common ground.
The main objective of the use of SMART is to trigger thinking and discussion about the way that safety is being addressed in an organisation and thereby to support improvement. The evaluation of the levels of the topics in SMART is a means towards this end, and achieving maximum scores is not an end in itself. By performing surveys and explaining their views and in particular by discussing survey results in workshops, people in an organisation can be stimulated to think about the way that safety is handled and to debate how this works out for different people with various roles and responsibilities. It is expected that this will support insight into the effectiveness of an organisation's existing SMS and supports its advancement. For this, the presented research streams provide effective links that organisations may apply to improve their safety performance.
At present, each aviation organisation must have its own SMS. However, in operations where one organisation must rely heavily on others, there is an argument for at least some kind of alignment of safety, making safety "interoperable". Furthermore, in such areas of interdependency for safety-such as at an airport-the argument is strong because if one organisation suffers a safety-related event, it can impact others. The application of SMART from a single to multiple organisations enables those different organisations to compare and discuss safety issues as well as to share safety information and improving the coherence of working methods and training by working together to develop common best practices. This may help organisations learn from each other about practical ways for effective safety management. This approach has been embodied in the Safety Stack approach [28,29], developed and implemented by a number of organisations at London Luton Airport who decided to "partner for safety". The Safety (Culture) Stack approach uses results of safety culture surveys as a basis for workshop interactions. An additional perspective may be achieved by a multiple organisations application of SMART. These two multi-organisational approaches provide different though partly overlapping perspectives on the way that safety is handled by organisations. A safety culture survey provides a broad overview about the safety attitudes of a large part of the personnel and about their perceptions on the way safety is handled in operations. An SMS maturity survey provides views on the effectiveness of approaches in the organisation's SMS according to a group of experts on SMS topics (such as safety managers, staff of a safety department, other managers). Attitudes and perceptions throughout the organisations can best be understood in combination with knowledge on safety performance and SMS details to arrive at organisational measures that support advancing the level of safety.
The design of the SMART questionnaire generalized previous questionnaires with extensions based on recent safety management literature, and the types of user groups are based on general insights for the use of questionnaires and workshops. SMART is an agile system, where both the questionnaire and the implementation of use cases can be adapted following feedback from their application and from developing views on effectively achieving safe performance in an organisation. While SMART was developed in the context of the aviation domain, its questions are all formulated in a general, domain agnostic way. As such, it can be a suitable basis to evaluate the maturity of SMSs in other domains (e.g., maritime and rail transport, process industry).
In support of improving human-related, soft aspects impacting safety in organisations, an overview of a range of methods were presented in this paper. The results of the research on top management can support executives in improving their contributions and leadership in the safety performance and safety management of their organisation. The middle management research has provided new insights and guidance that organisations can use to harness the role of middle management in organisational safety. The safety dashboard research supports organisations to develop and fine-tune safety dashboards that lay an effective foundation for the information transfer in the organisation's safety management system. The safety mindfulness research has provided new methods and techniques that support the flows of information in an organisation to keep everyone mindful of their role in assuring safety and to provide them the knowledge to do so. The safety culture research has provided a broadened safety culture assessment and enhancement approach and a safety culture stack approach, which supports improving the safety culture and safety interfaces of a stack of interrelated air transport organisations. The agile response capability research has provided new approaches for the development of exercises and preparedness plans for air transport crisis situations, which support organisations to advance their intra-and inter-organisational capability of detecting and flexibly responding to dynamic crisis scenarios.

Conclusions
Every organisation is unique in the operations it conducts, in the education of its personnel, in the relations it has with other organisations, in its history, in its organisational culture, in its relations with regulators, etc. As a result, every organisation has its own strong and weak elements in assuring the safety of its operations. As such, there cannot be a one-size-fits-all approach for advancing safety in organisations, but this needs to be based on a careful analysis of the organisation at hand, leading to tailored solutions. The SMS maturity assessment approach in combination with the overview of methods for improving human-related, soft aspects in safety management were developed to support such analysis and tailoring.
The SMS Maturity Assessment Research Tool (SMART) provides 34 questions with five maturity levels each along the four main parts of an SMS. Although stemming from the aviation domain, the questions are formulated in a general way, thus supporting potential application in various domains. SMART can be used in various ways by single or multiple users and intra-or inter-organisations. Its main objective is to trigger thinking and discussion to support improvement rather than achieving maximum scores per se. Improvement of safety-related, soft aspects in organisations is supported by a number of discussed research streams that support top management in safety leadership, harness middle managers in their role in organisational safety, allow fine-tuning of safety dashboards, support safety mindfulness in organisations, broaden safety culture assessment, and support improvement of an organisation's agile response capability.

Acknowledgments:
We thank all participants of Future Sky Safety P5 for the fruitful discussions of topics contributing to this study. We thank Lennaert Speijker for his managerial support. We thanks the anonymous reviewers for their comments helping us to improve the paper.

Conflicts of Interest:
The authors declare no conflict of interest.

Appendix A. SMS Maturity Assessment & Refinement Tool (SMART)
This appendix provides all topics of the SMS Maturity Assessment and Refinement Tool (SMART). The topics are grouped by each of the four SMS parts according to ICAO Annex 19 [1]: (1) safety policy and objectives; (2) safety risk management; (3) safety assurance; and (4) safety promotion. Each topic is scored on a level ranging from A to E. Safety authorities, responsibilities, and accountabilities have been identified but not yet formalised. Line managers assume responsibility for safety. C Authorities, responsibilities, and accountabilities for the management of safety have been defined and documented. This includes an accountable executive who, irrespective of other functions, has ultimate responsibility and accountability on behalf of the organisation for the implementation and maintenance of the SMS. Delineation of responsibility for the development, oversight, and implementation of the SMS is clearly understood. D All of Level C plus: Procedures are in place to address the need to review safety authorities, responsibilities, and accountabilities after any significant organisational change. E All of Level D plus: Safety authorities, responsibilities, and accountabilities are periodically reviewed to determine whether they are suitable and effective (i.e., continuous improvement of safety management).

Safety management function
A A safety management function has not yet been appointed to develop the SMS. B A safety management function has been appointed to develop and maintain the SMS. C The safety management function is independent of line management and develops and maintains an effective SMS.
The safety manager has access to the resources required for the proper development and maintenance of the SMS.

D
All of Level C plus: The highest organisational level recognises its role in the SMS and actively supports the development, implementation, maintenance, and promotion of the SMS throughout the organisation (including support departments). E All of Level D plus: There is clear evidence that the highest organisational level plays a proactive role in the continuous improvement of the SMS.

Implementation and management of the SMS
A There is no SMS in place. The need for an SMS implementation plan may have been recognised. B An SMS is partially implemented, but it does not yet meet standards established through safety regulatory requirements. A compliance gap analysis has been performed, and an SMS implementation plan has been developed towards improvement. C The essential parts of the SMS are implemented, and the organisation meets the standards established through safety regulatory requirements. The requirements expressed in the SMS implementation plan have been completed. D All of Level C plus: All parts of the SMS are implemented, and the coupling between the SMS processes have been shown to be functional. E All of Level D plus: There is continuous monitoring of the effectiveness and efficiency of the various SMS processes, and management takes effective measures to control the performance of the SMS. Latest insights on effective safety governance are used for this purpose.

Consistency with regional/international safety standards
A There is little awareness of the regional or international safety standards. B There is an awareness of the regional and international safety standards. Work has started in some areas. C Regional and international safety standards are known and met as required. D All of Level C plus: There is a process in place to address the need for timely and consistent compliance with regional or international safety standards. E All of Level D plus: The organisation has a structured mechanism to address the need for ongoing and consistent compliance with regional or international safety standards. It contributes to a regional or international dialogue to improve these standards.

SMS documentation
A Operations manuals do not contain any specific safety management procedures. B The documentation of SMS processes and procedures has started and is progressing according to the SMS implementation plan containing as a minimum: (a) safety policy and objectives; (b) SMS requirements; (c) SMS processes and procedures; (d) accountabilities, responsibilities, and authorities for SMS processes and procedures; and € SMS outputs. C The documentation of the essential parts of the SMS processes and procedures is complete. The processes and procedures ensure that the organisation is compliant with all applicable safety and regulatory requirements. D All of Level C plus: There is clear evidence that the safety and safety management documentation is readily available to all personnel in the organisation. This documentation details safety and safety management processes and procedures that meet or exceed the applicable safety and regulatory requirements. E All of Level D plus: Processes are in place and are being applied to continuously improve the SMS documentation.

Emergency/Contingency response procedures and plan
A The organisation does not have redundant capabilities or back-up systems. Relevant external emergency organisations are unfamiliar with the operational hazards in the company, and the organisation has not defined an incident command structure in relationship with these external agencies. B There are procedures and some redundant capabilities and resources to cope with abnormal and unexpected situations. An incident command structure is identified. Regulatory emergency response requirements are met, while a comprehensive emergency response plan is under development. External emergency agencies are familiar with operational hazards in the company. C All primary systems have redundant capabilities, and emergency/contingency response procedures have been developed, documented, and distributed to appropriate staff. The emergency/contingency response plan is properly coordinated with the emergency/contingency response plans of those organisations it must interface with during the provision of its services. D All of Level C plus: The emergency/contingency response plan and procedures are defined in a flexible, adaptive way, properly allowing ranges of variations in the crises situation. They have been rehearsed through desktop or operational exercises. E All of Level D plus: The emergency/contingency response plans, procedures, and processes are regularly exercised and revised to keep them up-to-date. This includes exercises and coordination with all relevant external agencies, thus creating an agile response capability for the entire air transport system.

Safety policy
A The need for a safety policy may have been recognised, but one does not exist. B The organisation recognises that the implemented policy needs to be signed by an accountable executive and communicated to all employees and stakeholders. A draft safety policy is available, which reflects the organisation's commitment to safety and its priority. The policy is communicated to staff throughout the organisation and visibly endorsed by an accountable executive. C The safety policy has been finalised and signed by an accountable executive. It presents the organisation's commitment to both safety and its adequate resourcing. There is a periodic review of the policy to assure that it continues to be relevant and appropriate. D All of Level C plus: Updates to the policy are undertaken when the accountable executive changes or if the organisation believes that the policy does not adequately address the organisation's commitment to safety. E All of Level D plus: The organisation benchmarks its safety policy against other organisations and high reliability industries. Gaps and deficiencies are addressed in the policy and actioned through the SMS.

Senior management visibility and involvement
A Senior managers do not communicate explicitly about their expectations for safety performance, and they are not involved in safety management processes. B Senior managers communicate their flight safety, occupational safety, and, where appropriate, environmental protection expectations to staff reporting to them, but they do not refer to related SMS processes. They review reactive safety indicators, such as incidents and accidents, but they are unconvinced about the value of proactive safety indicators as part of safety management. C Senior managers discuss and review with staff and subcontractors progress against meeting specific safety result targets and needed activities, usually during appraisals. They participate in the development of objectives and target setting for safety indicators. They review the progress both in the development and the content of the SMS and safety cases. They make available the resources and expertise needed for SMS tasks, evaluation, and development. D All of Level C plus: Senior managers actively participate in safety-related activities, such as training, reward and recognition schemes, safety workshops, safety conferences, and audits. They jointly develop and discuss both safety results and activity improvement targets with staff and company contractors. They are fully aware of the high priority areas for improvement identified in the SMS and the status of the follow-up remedial programme. E All of Level D plus: Senior managers drive the process for safety excellence, and they are role models for safety. They ensure that all staff have safety results and activity targets in their appraisals. They are personally involved in safety improvement efforts.

Sub-contractors
A The safety competence of sub-contractors is not considered. B Sub-contractor safety competence is assessed in the light of the risks to be managed during the contract prior to the invitation to tender and award of contract. C Sub-contractor acceptance is conditional upon receiving a description of how safety risks will be systematically managed and interfaces managed on that particular activity. D All of Level C plus: Compliance with the sub-contractors own SMS is audited within an audit programme defined in the contract. Actions to be taken in the event of different levels of non-compliance are defined in the contract. E All of Level D plus: The SMS of sub-contractors are subject to continuous improvement during the course of projects and contracts in consultation with the company. A single approach is used for the identifications of hazards, which is used for designs or changes to sociotechnical systems supporting operations. A limited number of hazards is thus identified. C A number of approaches are used for the identification of hazards as part of assessment of new designs or changes to sociotechnical systems. A broad set of hazards is thus identified. D All of Level C plus: Hazards are systematically identified on the basis of feedback from operations (coupling with safety assurance), including changes in operational circumstances. E All of Level D plus: Disturbances and variations in operations are systematically identified irrespective of the potential effect on safety. This is done for new designs, changes to sociotechnical systems, changing circumstances, and on the basis of feedback from operations.

Risk assessment for design and change
A The level of risk is assessed for each identified hazard separately on a scale from low to high safety risk. This assessment is based on the judgement of a single or few people in the organisation. B The level of risk is assessed for each identified hazard separately by judgement of the likelihood of the hazard and the severity of its consequences. The assessment is based on the consultation of several people in the organisation, including operators. C The level of risk is assessed for scenarios, which represent combinations of hazards in a specific operational context, by judgement of the severity levels of the potential consequences of the scenario and the likelihood of these severity levels. The assessment is based on the consultation of several people in the organisation and on quantitative data of the operations. D All of Level C plus: For complex scenarios, models are used which represent in detail the complexity of the dynamics and interactions in the sociotechnical system. The assessment addresses a broad range of disturbances and operational variations to determine the likelihood of safety occurrences. Computer simulation of agent-based models, quantified risk models (fault and event trees), and Functional Resonance Analysis Method (FRAM) models may all be used to support such assessment. The level of uncertainty in the risk assessment results is indicated. E All of Level D plus: The models of scenarios also represent the effects on key performance areas other than safety of the identified disturbances and operational variations. As such, an overall view is attained of the implications of a scenario and of the trade-offs that an operator may need to make in balancing safety with other performance areas.

Safety risk control
A There is little understanding of the need to control risk even when risks are recognised. The basic strategy is that the personnel is warned for particular risks. B Safety risk control is implemented by posing detailed requirements on human error and system failures such that the safety risk is considered acceptable. C To mitigate safety risks that are considered unacceptable, there is development by an interdisciplinary design team of new processes, equipment, training, or staffing arrangements. Residual risk levels are assessed by the design team. Managers can sign off residual risk levels over certain thresholds. D All of Level C plus: New designs for mitigation of unacceptable risks are assessed in a complete cycle of the safety risk management process to assess that the achieved risk is acceptable, and the proposed design does not introduce new hazards. This is done by an assessment team that is independent from the design team. The level of uncertainty in assessed risk levels is included in the risk tolerability decision making. E All of Level D plus: Performance variability that has been considered as normal in the safety risk assessment is used as a basis to define a range of performance indicators that reflect the work-as-done in the organisation. These performance indicators form a basis for measurement in safety assurance processes. Fatigue-related risk is considered as an operational hazard, but there is no formal risk-based system by which to manage it. Policy has been developed that recognises the need for a formal risk-based approach to fatigue-related risk. C A formal risk-based system that focuses on fatigue-related risk is being implemented, which addresses responsibilities of both management and operational personnel, and methods for assessing and managing fatigue risk. D All of Level C plus: Compliance with fatigue-related risk procedures is continually assessed. Processes are in place to assess and continually improve approaches for fatigue-risk management. E All of Level D plus: The organisation uses the data and information from internal and external sources to continually improve its approach to managing fatigue-related safety risk.

Sufficiency of resources
A Risks inherent in operations and emergency procedures are not considered in determining the resource levels. B Risks inherent in operations and the emergency procedures are taken into account in determining the resource levels. C The level of resources required to assure safety in terms of numbers and function of personnel are fully described in a safety case (i.e., to ensure "adequate" personnel and resources). D All of Level C plus: The actual resourcing meets the requirements described in the safety case in number and competency. E All of Level D plus: Changes to resourcing levels and competencies and associated risks are assessed as part of the change control procedure within the company. Symptoms of under-resourcing are recognised, acknowledged, and addressed.

Maintenance
A The maintenance program meets the regulatory requirements. B Activities within maintenance program are commensurate to the risks they impose. Quality and integrity of the systems are proportional to the risk. C There is a data-driven assurance of quality and integrity of systems (e.g., aircraft), facilities, and equipment; effective operation and maintenance of critical equipment; and thorough records of inspection, maintenance, repair, and alteration. D All of Level C plus: There are mature reliability programs in place and a well-developed, customized maintenance program. Systems, equipment, and facilities are managed in accordance with industry best practices. E All of Level D plus: There is continuous improvement of the maintenance management based on latest insights in safety management systems.
Appendix A.3. SMS Part P3: Safety Assurance 3.1 Integrated risk management and safety-related internal interfaces for key performance areas (such as finance, quality, security, and environment) A The various management systems of key performance areas operate in isolation, and safety-related interfaces are not considered. B Safety-related relations between management systems of key performance areas are managed on an informal or ad hoc basis with a basic understanding of their boundaries and relationships. C Safety-related relations between management systems of key performance areas are managed with a solid understanding of their boundaries and relationships. D All of Level C plus: There is an integrated risk management system for all relevant key performance areas, which systematically addresses all types of risks and their relations. This includes assessment of costs associated with accidents and incidents and of costs and benefits of risk-mitigating measures. E All of Level D plus: A learning process is in place for continuous improvement of the integrated risk management system.

Safety-related interfaces with external parties
A Safety-related interfaces with external parties are only considered to a limited extent. B Safety-related interfaces with external parties are managed on an ad hoc basis, and contractual arrangements are negotiated and implemented. C Formal risk management processes are used for all relations with external parties. Safety requirements are specified and documented in appropriate agreements. D All of Level C plus: External services and suppliers are surveyed/audited and systematically monitored to assure consistency with the agreements and to identify the development of new risks. Agreements and levels of coordination with external parties are revised as necessary. E All of Level D plus: A learning process is in place for continuous improvement of the safety management processes for external parties.

Reporting and investigation of safety occurrences
A There is an informal system in place for reporting safety occurrences, but reports are not reviewed systematically. The reporting system is not organization wide. Investigation is done on an ad hoc basis and with little or no feedback. B There is a plan to formalise the existing reporting and investigation system. There is commitment from management to allocate resources to implement this system. The reporting system is widespread but does not yet cover the whole organisation. Feedback is given on an ad hoc basis. C The system in place is commensurate with the size of the organisation. The organisation has a complete and formal system that records all reported information relevant to the SMS, including incidents and accidents. Corrective and preventive actions are taken in response to event analysis. D All of Level C plus: Identified safety-related risks and deficiencies are actively and continuously monitored and reviewed for improvement. E All of Level D plus: Personnel who report safety occurrences and problems are empowered to suggest corrective actions, and there is a feedback process in place.

Monitoring of safety indicators
A Ad hoc safety performance data related to individual incidents are available, but there is no systematic approach for measuring safety performance There are no indicators, thresholds, or formal monitoring system in place to measure safety achievements and trends. B There is a plan to implement a monitoring system. The implementation of some qualitative and quantitative techniques and indicators in certain parts of the organisation has started. C The safety monitoring system has been implemented and documented. Indicators and targets have been set, which are limited to meeting the safety regulatory requirements to verify the safety performance of the organisation. D All of Level C plus: A broader set of indicators is used, and safety performance is measured using statistical and other quantitative techniques. All indicators are tracked against thresholds/targets on a regular basis, including trend analysis. Internal comparative analysis is done, and external comparative analysis has begun. Results are used to drive further safety improvements across the organisation. E All of Level D plus: Safety indicators cover all aspects of the system/operations, and they include indicators for performance variability of work-as-done in the organisation. There are comprehensive metrics in place to measure and monitor indicators and thresholds throughout the system. Internal and external comparative analysis is well established.

Operational safety surveys and audits
A There is no plan to conduct systematic operational safety surveys and audits. Operational safety surveys, audits, and gap assessments are conducted on an ad hoc basis (e.g., when deficiencies in the system or in working arrangements are found). B There is a plan in place to formalise the conduct of systematic operational safety surveys and audits. A limited number of operational safety surveys and SMS audits have been carried out. C Internal operational safety surveys and audits are conducted on a periodic basis. Based on the output of operational safety surveys and audits, a process is in place that requires the development and implementation of appropriate improvement plans.

D
All of Level C plus: Internal or external operational safety surveys and audits are carried out in a systematic way. There is a process in place to monitor and analyse trends and identify areas that require follow-up operational safety surveys or audits. Follow-up operational safety surveys, audits, and gap assessments are conducted in all areas affecting operational safety. Operational safety surveys and audits are actively reviewed to assess opportunities for system improvement. E All of Level D plus: Independent (external) operational safety surveys and audits are periodically conducted. The outputs from operational safety surveys and audits are incorporated as appropriate into operations. There is a process in place that requires external data to be considered when selecting areas to be subject to operational safety surveys and audits.

3.6
Auditing and improvement of SMS methods A There is no formal process that maintains the SMS, nor is there an identified authority (or authorities) responsible for the updates. SMS audits are conducted on an ad hoc basis. B A process to maintain safety management procedures exists. The authority (or authorities) responsible for the updates are partially identified. The procedures are kept up-to-date on an ad hoc basis. C SMS audits are conducted on a periodic basis. The process to maintain SMS documentation is defined and practised. D All of Level C plus: Internal or external SMS audits are carried out systematically. There is a process in place to monitor and analyse trends and identify areas that require follow-up SMS audits. SMS audits are actively reviewed to assess opportunities for system improvement. There is a formal process in place to periodically review safety and safety management procedures and ensure that they remain relevant, consistent with industry practice, and effective. The authority (or authorities) responsible for the updates are clearly identified. E All of Level D plus: Independent (external) SMS audits are periodically conducted. Changes within the organisation that could affect the safety management framework are subjected to formal review. New insights about improving SMS in the scientific literature are actively followed and the organisation participates in studies to evaluate the effectiveness of such innovations for its organisation.

Variations with respect to procedures and standards (work-as-done)
A It is considered that there are no variances in the work-as-done with respect to procedures and standards. Non-compliance with procedures and standards is denied and is not recorded. B Procedures for variances with respect to procedures and standards exist, but they are impractical, and few variances are reported. C There is a system for reporting variances in work-as-done with respect to procedures and standards, which is well documented and communicated to the employees. There are records for variances for many types of work all over the organisation. D All of Level C plus: Reasons of reported variances are analysed on an ad-hoc basis. Lessons learned range from better training and education to changes in company procedures. They are systematically communicated to people who reported the variances and to others who are involved. E All of Level D plus: Safety assurance includes processes that systematically use the feedback from reported variances for organisational learning. Performance variability is explicitly considered (assumed) in safety risk management and reported variances are compared with the assumptions made. Company procedures are updated if needed, and active collaboration with industry stakeholders is sought to change standards.

Auditor competency
A Company uses mainly unqualified and/or inexperienced resources for SMS audits. B Personnel involved in audits first undergo formal SMS audit training. There is a process describing the required competency for auditors. C Safety and audit personnel as well as personnel in other parts of the organisation periodically undergo audit training. D All of Level C plus: Relevant personnel undergo an audit training and competency development program. The company has been subject to external audits by peers. E All of Level D plus: Company works with individually tailored development programs aligned with best practices, and it frequently uses external audits by peers.

Management of change
A No change management processes are in place although the organisation recognises that impacts of change need to be managed. B Some change management procedures exist, and they are applied on an ad hoc basis.
C A systematic set of change management processes are used to address how the impact of change can be assessed from a risk perspective; how to involve stakeholders; how to document and evaluate the impacts; and who will determine whether a change is authorised or not. D All of Level C plus: Quantitative approaches for risk assessment are used. Risk control functions are being monitored following the change. E All of Level D plus: The organisation continually looks to refine its approach to change management on the basis of experience within the organisation and using knowledge of state-of-the-art in management of change.
Appendix A.4. SMS Part P4: Safety Promotion 4.1 Safety culture measurement and an improvement programme A The organisation does not see the need to have a safety culture measuring mechanism in place. B The organisation is aware of the need to have periodic measurements of safety culture in place as well as an improvement plan. However, what will be measured and when is still being defined. C Safety culture is measured, and results are available. An improvement plan addresses the need for individuals to be aware of and support the organisation's shared beliefs, assumptions, and values regarding safety. D All of Level C plus: The organisation assesses its safety culture on a regular basis and implements improvements to any identified weaknesses. Safety culture enablers and barriers are identified, and solutions to reduce barriers are being implemented. E All of Level D plus: The organisation is gathering data on safety culture on a continuous basis, and it is constantly reflecting on the effects of all decision-making and changes on safety culture.

Promotion of a just and open culture for reporting and investigation of occurrences
A Management believes there are no issues regarding the existing reporting and investigation culture and therefore does not see the need for any activity or dialogue with the staff in this area. B Discussions between staff and management to improve reporting and investigation policies and culture are underway. C Safety data-sharing and publication policies are well known and supported by the staff. Safety data are sufficiently protected from external interference within legal limits. D All of Level C plus: Within the organisation, the line between acceptable and unacceptable mistakes is clearly established and known by the staff. Just reporting and investigation culture principles are in place and systematically applied within the organisation. E All of Level D plus: There is a clear and published policy on how dialogue with judicial authorities and media is established and followed.

4.3
Knowledge transfer of safety management standards and practices A Staff have limited knowledge of the safety policy, SMS processes, and procedures. B Limited communication is presented as to why particular safety actions have been taken and/or safety management procedures introduced. Internal communications within the organisation does not focus on safety and its management. C Communication strategies are being developed to ensure that staff are aware of the safety management practices which are relevant to their position. Specific communication strategies are being implemented to address situations where procedures have changed or when critical safety action has been taken. The safety policy is prominently displayed in a language understood by all staff and contractors. All staff have a personal copy of the safety policy. D All of Level C plus: Communication mediums are regularly assessed for effectiveness. Gaps and deficiencies are acknowledged and addressed. The personal relevance of the safety policy and changes therein is communicated to all staff by their immediate supervisors or as appropriate. E All of Level D plus: Safety is a key focus of internal communication. The organisation is looking to increase the number of mediums through which safety messages are sent within the organisation. All staff are able to explain what responsibilities they have to and what they have to do in their work to fulfil the requirements of the safety policy.

4.4
Training and competency in safety and safety management A Staff and contractors are provided sparsely with training for safety and safety management activities. B Staff and contractors are provided with training and education, but spaces are limited, and planning is ad hoc. C An annual planning process for training is in place. The plan considers all staff and contractors, and the training addresses all safety management practices that they may be called upon to apply and contribute to. D All of Level C plus: There is a process for the training provider(s) to receive feedback on the effectiveness of the training programmes. Based on this feedback, the training programmes are revised to improve effectiveness. E All of Level D plus: There is regular measurement of the level of competency of staff and contractors in safety management practices, and this is used in planning and improvement of training. There is a minimum number of SMS personal that has a suitable academic background. Latest scientific insights on effective safety management and training are used for the development of the training programs.

4.5
Recording and dissemination of safety lessons learned A Safety lessons learned are known only to those who experience them. B Safety lessons are recorded and shared on an ad hoc basis rather than systematically. C The process for sharing safety lessons learned is systematic and operational, and the majority of data is shared with appropriate personnel. The rationale for taking action and making changes to procedures is explained to staff. Safety-critical information is disseminated to all appropriate staff. D All of Level C plus: All safety lessons learned are systematically shared across the organisation at all appropriate levels. Corrective actions are taken to address lessons learned. E All of Level D plus: There is clear evidence that the dissemination process of the internal lessons learned is embedded across the organisation at all levels and is periodically reviewed.

Sharing of safety information and knowledge with industry stakeholders
A Safety data and information are treated as confidential and internal in the organisation as well as for industry stakeholders. B Safety data and information are shared internally, but the organisation is reluctant or unwilling to share data with industry stakeholders. C Safety data and information is shared internally, nationally, and with international bodies when it is required by regulation. D All of Level C plus: There is a clear and published policy that encourages the proactive sharing of safety-related information with other parties. E All of Level D plus: Safety data and information are actively shared internally, nationally, with recognised international bodies, and with other industry stakeholders. The organisation has a process in place to receive and act on safety data and information from external stakeholders.

4.7
Publication of safety performance information to the general public A Safety-related performance information is not made available to the public under any circumstances. B A limited amount of safety-related performance information is made available but only to selected authorities. C High-level safety-related performance information is made available to the general public according to applicable requirements. D All of Level C plus: Safety performance information not governed by applicable requirements is also made available to the public. E All of Level D plus: The organisation voluntarily makes available appropriate safety-related performance information to the general public. The achieved safety levels and trends are transparent to the general public.

4.8
Sharing and learning best practices on operational safety and SMS practices A There is no structured approach to learn and share best practices with the industry. The organisation has the capability to identify and adopt industry best practices on an ad hoc basis. There are no plans to release and share best practices with industry stakeholders. B There is an ad hoc structure in place to gather information on operational safety and SMS best practices. Some initial implementation has begun. Some internal best practices are spread across units within the organisation, but there is no systematic structure for the adoption of best practices. Sharing of best practices takes place in response to requests for assistance from industry stakeholders. C A structure has been established to identify applicable operational safety and SMS best practices from the industry to enable improvements to the SMS. Best practices are shared with industry stakeholders as required by regulation. D All of Level C plus: Industry best practices are periodically reviewed to provide the most current information, which is then assessed for applicability and adopted as appropriate. Safety-related best practices are shared to a wide extent with industry stakeholders. E All of Level D plus: All relevant best practices are readily accessible to appropriate personnel. The organisation actively cooperates with industry and academic partners in developing best practices.