Incident reporting schemes have been proposed as primary means of ensuring that failures in existing systems are not propagated into future applications. The success of the FAA/NASA’s Aviation Safety Reporting System (ASRS) and of the independent Confidential Human Factors Incident Reporting Programme (CHIRP) has led to the development of similar schemes throughout the world. While these initiatives have made significant contributions to aviation safety, this paper looks for further ways in which we might improve the reporting of human error.

Incident reporting; human "error"; safety.

Incident reports are primary mechanisms for improving safety in many different industries. For instance, the Chemical Safety and Hazards Investigation Board operates a scheme that covers all users and producers of chemicals within the United States. A national reporting system for the UK railway network has been proposed in the aftermath of the Paddington/Ladbroke Grove crash. Similar schemes also operate at local and regional hospitals throughout Europe. Many of these initiatives stem from the perceived success of incident reporting in aviation [1]. In particular, the FAA/NASA’s Aviation Safety Reporting System (ASRS) and the UK’s Confidential Human Factors Incident Reporting Programme (CHIRP) are often cited as prototypes for the wider application of confidential and anonymous reporting systems [2].

There are many different factors that contribute to the perceived success of incident reporting systems within aviation. For instance, they can be used to pick up failures that might eventually lead to more serious consequences had they not been detected in time. The findings of national and regional schemes can be exchanged so that trends can be identified and lessons can be shared across national borders. They also help to keep staff "in the loop". Most reporting systems include mechanisms by which the findings of previous submissions can be fed back to other workers. This helps to raise awareness about the potential for failure. Such feedback also reminds operators about possible detection factors and about the safeguards that can be deployed to mitigate any future risks [2].

A number of well-known problems affect the utility of incident reporting schemes as means of addressing the causes of human error. For instance, there is a tendency for some schemes to rely on reminder statements that encourage individuals and teams to "be careful" or to "work harder" on aspects of their performance [3]. Incident reports result in reminder statements to staff rather than requests for manufacturers to improve the design of their devices. Safety concerns end up as appeals to improve human performance rather than as management initiatives to avoid the precursors of operator "error". This paper shows how some of these problems can affect aviation reporting schemes. Later sections go on to identify a range of techniques that have been used to address these concerns within the ASRS and CHIRP systems.

Both the ASRS and CHIRP are well established. The ASRS was set up in 1976 and now receives an average of more than 2,600 reports per month. It encourages submissions about incidents with a wide range of technical, organizational and meteorological causes. In contrast, the much smaller scale CHIRP is specifically focussed on human factors issues. It was established in 1982 and in 1998-99 received a total of 228 reports. These two schemes, therefore, differ both in their scale and in their intent. There are also some notable similarities. Both the ASRS and CHIRP rely upon individuals returning paper-based forms. Figure 1 illustrates an Air Traffic Control report from the CHIRP systems. There are further similarities. Both schemes are confidential rather than anonymous. Any necessary information, which cannot be obtained directly from the form, can be elicited during follow-up interviews. Any identifying features in the report are then removed prior to further analysis.

Figure 1 CHIRP Air Traffic Control Forms (Feb. 2000).

Figure 2 illustrates an excerpt from an ASRS air traffic management incident. An increase in traffic led a controller to request an increase in separation for aircraft entering his sector. This was not achieved and there was a loss of separation. This is typical of reports that describe how environmental factors, such as increased traffic, create the preconditions for aviation incidents.

ACCESSION NUMBER: 426859

DATE OF OCCURRENCE: 9901

NARRATIVE: I WAS WORKING THE LOW ALT (0-FL230) WINDSOR SECTOR AT ZOB. THIS SECTOR IS AN OUTBOUND SECTOR FOR DETROIT APCH CTL. THERE WAS EXCESSIVE TURB IN THE HIGH ALT SECTOR OVER MY AIRSPACE SO THERE WERE SEVERAL OVERFLT ACFT IN MY SECTOR THAT NORMALLY WOULD NOT BE THERE. ALSO, ADDITIONAL ACFT NOT ON PREFERRED ROUTING IMPACTED COMPLEXITY. TFC WAS BUILDING AND SECTOR BECAME EXTREMELY BUSY. DETROIT WAS SUPPOSED TO PROVIDE 15 MI IN TRAIL OVER 2 FIXES INTO MY SECTOR. THE AIRPLANES BEGAN TO ENTER AT 10 MI IN TRAIL, THEN QUICKLY BEGAN TO COMPRESS. I CALLED DTW TO ASK FOR MORE SPACING. AT THAT POINT MY SUPVR TRIED TO PUT A TRACKER IN TO HELP. DUE TO CONFUSION OVER VSCS PROC, I LOST COM WITH ACFT 3 OR 4 TIMES. MY D-SIDE WAS STRUGGLING TO PUT IN REVISED ROUTING FROM TMU AND COULD NOT PROVIDE ASSISTANCE. DETROIT HANDED ME A DC9 (ACFT Y) FOLLOWED BY THE A320 (ACFT X) WITH A 70-90 KT OVERTAKE WITH ONLY 5 MI SEPARATION. NEEDLESS TO SAY, 5 MI SEPARATION WAS LOST QUICKLY AT 13000 FT. LOA STATES THAT DTW SHALL PROVIDE 5 MI INCREASING BUT TODAY THEY WERE SUPPOSED TO PROVIDE 15 MI INCREASING. THE TRACKER NEVER MADE IT IN. I WAS DISTR BY ALL OF THE CONFUSION. I'M NOT SURE WHO ACCEPTED THE HDOFS. I ISSUED NO CTL INSTRUCTION TO EITHER ACFT. SEPARATION WAS LOST IMMEDIATELY UNTIL I CLBED THE LEAD ACFT.

Figure 2: ASRS Air Traffic Control Incident (Aug., 1999)

Both the ASRS and CHIRP publish newsletters that help to keep crews, air traffic controllers and engineers informed about incidents such as that illustrated in Figure 2. The CHIRP Feedback newsletter has a circulation of approximately 30,000. The ASRS’s Callback is distributed to 85,000 aviation professionals. Both are written in an informal and accessible style. Both provide an overview of the incidents that have been reported in previous months. They also focus on specific "critical" topics, such as ramp safety. The ASRS also publishes the DirectLine journal, which is intended for operators and flight crews of commercial carriers and corporate fleets.

The ASRS and CHIRP are amongst the most successful incident reporting schemes currently being operated in the aviation industry. Callback received Flight Safety Foundation publication awards in 1981 and 1987. The Aviation/Space Writers Association's (AWA) Award of Excellence was awarded to its editors in 1982 and 1992. In 1999, 99% of the 28,384 people that responded to a recent Feedback survey stated that "CHIRP makes a useful contribution to the promotion and improvement of flight safety". Only 0.6 % disagreed with this statement. It is, however, important to look beyond these figures and awards to ask see what lessons can be learnt from the success of the ASRS and CHIRP. It is also important to identify any ways in which these existing schemes might be improved. For example, the following excerpt describes an incident and proposes a solution:

(ASRS Callback, Issue 247, January 2000).

The previous citation illustrates the strengths and weaknesses of many incident report schemes. They provide first-hand insights into usability problems that affect pilots, air traffic controllers and engineers. They also provide common sense solutions that help users to combat the challenges that poorly designed equipment can create. However, there is also a danger that immediate remedies to individual incidents will fail to address the root cause of a problem [4]. The noise-correcting headphones were clearly not fit for purpose. The recommended "solution" of removing one headphone whilst providing a short-term fix for individual pilots does little to increase confidence that such problems may not recur in future products. The previous excerpt, therefore, illustrates the first potential problem for incident reporting schemes. By focussing on the immediate symptoms of a problem, they may obscure deeper usability issues. Many reporting systems address this weakness by publishing more sustained investigations into several incidents. For instance, the ASRS does this through their DirectLine publication. There are, however, a number of further concerns about the treatment of human error in different reporting systems.

An important benefit of incident reporting systems is that they enable lessons that have been learnt in one organization to be transferred to other companies in the same industry [5]. Similarly, lessons that have been learnt in commercial operations can be propagated to general aviation. In practice, however, it can be difficult to ensure that lessons are transferred in this manner. A particular problem is that many publications tend to respond to incidents in a piecemeal way. Similar incidents can provoke different responses in different schemes. This is particularly significant because many pilots read and contribute to incident reporting schemes in more than one country. For instance, the following Feedback excerpt offers a slightly different perspective on the problems that are posed by ambient noise:

This illustrates how the individual incident reports that are received by different national schemes offer different perspectives on a common problem. The ASRS Callback article suggests that only one headset should be used during landing. In contrast, the CHIRP reporter argues that this practice poses a significant risk to flight safety. The feedback associated with the second report resolves this contradiction by arguing that the partial/full use of headsets should be dependent on ambient noise. However, this insight would not be readily apparent to many of the national crews who only have access to the first publication. This, therefore, illustrates a second concern about aviation incident reporting systems. Different national systems have offered different forms of advice to crews and engineers in response to very similar incidents [6]. These differences do not reflect underlying differences in SOP’s or in the regulatory environments. Instead, they are symptomatic of the inconsistencies that can arise when immediate or "stop-gap" advice is provided in response to individual human factors reports.

Previous paragraphs have argued that in a justifiable desire to provide rapid feedback, many incident-reporting systems focus on short-term solutions rather than the underlying causes of usability problems. It has also been argued that this can lead to inconsistencies between major incident reporting systems. This inconsistency prevents individuals from receiving a coherent message about common safety problems in an international industry. The following excerpt from the CHIRP Feedback publication illustrates a further concern.

This documents "yet another" cockpit data entry problem [7]. The corrective advice has the same common sense motivation as the ASRS report’s advice about noise reducing headsets. It is well founded and sensible. However, it reiterates ideas that have formed a standard part of CRM for almost twenty years. It is depressing that such data entry problems continue to be reported so long after regulatory authorities have acted to implement CRM training. UK Aeronautical Information Circular (AIC) 143/1993 (Pink) states that all crew must have completed an approved CRM course before January 1995. JAR OPS sub-part N, 1.945(a)(10) and 1.955(b)(6) and 1.965(e) extended similar requirements to all signatory states during 1998. The previous excerpt, therefore, illustrates a third problem for incident reporting schemes. Without further support and analysis, there is a danger that the pro-active role of incident reporting systems will be lost and that these schemes will continue to recommend well-known solutions to well-known problems. An alternative interpretation of the previous CHIRP report is that existing CRM training has not been as successful as many people would claim [8].

The previous section argued that many incident reporting systems reiterate well-known solutions to well known problems. It is important not to dismiss the value of this for many safety-critical environments. Incident reporting systems play an important role in raising awareness about safety issues. By documenting incidents that have occurred to others, the intention is to alert colleagues to the potential for future failures. Publications such as Feedback and Callback, as well as DirectLine and the ASRS Operational Bulletins, play a critical role in increasing the visibility of these common occurrences:

(CHIRP Feedback, Issue No. 53, January 2000)

The reiteration of well-known safety recommendations also raises a number of more fundamental concerns about the utility of incident reporting systems for the long-term safety of many applications. There is a considerable body of human factors research that points to the dangers of any reliance on reminders. Unless people are continually reminded then they are likely to forget the importance of safety precautions over time [9]. The previous excerpt was chosen carefully because ramp safety is an area where there has been, and continues to be, a high level of injuries and accidents. The incidents recur frequently within both the ASRS and the CHIRP reports and the same advice continues to be offered to broadly similar incidents. For example, this topic was addressed in DirectLine no. 8 and in Feedback no. 53. It might, therefore, be argued that sustained action to remove the fundamental causes of these injuries and accidents would be more effective than the repeated reminders offered by incident reporting systems.

The previous paragraphs have argued that incident reporting systems help to raise awareness about common safety concerns. They have also argued that by emphasizing short-term, well-known remedies, there is also a danger that they may take attention away from the underlying usability problems that affect many aspects of the aviation industry. Of course, it can be argued that the individuals who operate the systems, fly the aircraft and submit the reports can do little to address these deeper concerns. An individual pilot or air traffic control officer can do little to change the design of the complex systems that they operate. Callback and Feedback might not, therefore, provide an appropriate forum for discussing the root causes of aviation incidents.

• Standardize PDC formats, so that pilots will know where to look for routing information and revisions.

• Show only one clearance line in a PDC, and insert any revisions into the clearance line. For example, instead of showing a route revision this way:

• Make the revision section more visible by tagging it ("REVISION") or highlighting with asterisks or other eye -catching notation (*****).

• Provide flight crews with training in how to recognize PDC revisions."

ASRS Operational Issues Bulletin 96-01, December 1996.

The previous citation from the ASRS’ Operational Issues Bulletin provides a more detailed causal analysis of incident reports. The annual DirectLine reports publish the results of similar studies. These two publications address the concerns that were raised in previous sections. They "facilitate a better understanding of aviation safety issues". However, they have a more limited audience than Feedback or Callback. Research into the effectiveness of incident reporting systems in other domains has stressed the importance of providing the individuals who contribute to an incident reporting scheme with direct feedback about the effect of their contributions on fundamental and long running concerns. For example, incident reporting systems have had little impact on workload issues in the aviation industry [7]. Neither does there seem to be any decline in the usability problems that recur with each new generation of computer systems [9].

There is no counterpart to the ASRS Bulletins within the CHIRP scheme. However, there are good examples of more sustained causal analyses of usability issues within Feedback. For example, the following except draws attention to both the usability and the safety issues that can arise from poorly designed computer systems.

The comment that aviation is a "closed community" and that "troublemakers and stirrers do not keep jobs" captures a feeling that is often repeated in confidential interviews and in published incident reports. Some authors have claimed that incident reporting schemes help to encourage a more open culture [1]. Recent moves to develop incident reporting schemes within the UK railways and within many hospitals have been justified by claims that they will protect individuals who point out potential threats to safety. Incident reports, such as the one cited above, illustrate that these schemes can do very little to effect the underlying safety culture of the companies and organizations within particular industries.

Incident reporting schemes help to ensure that failures in existing systems are not propagated into future applications. The success of the FAA/NASA’s Aviation Safety Reporting System (ASRS) and of the independent Confidential Human Factors Incident Reporting Programme (CHIRP) has led to the development of similar schemes throughout the world. While these initiatives have made significant contributions to aviation safety, this paper looks for further ways in which we might improve the reporting of human error:

1. Many of the publications that are provided to the individuals who contribute incident reports tend to focus on direct solutions to their everyday problems. There is a danger, however, that these short-term panaceas will obscure deeper recurring safety issues, such as the continuing usability concerns with many computer applications in the aerospace industry.
2. Many reporting systems address the concerns described above by publishing overviews that address the common causal features in several incidents. CHIRP has introduced editorial overviews. The ASRS has sister publications such as DirectLine. Research into incident reporting systems in other domains stresses the importance of providing the individuals who contribute to an incident reporting system with direct feedback about the effect of their contributions on long running concerns as well as short-term fixes [1, 3].
3. Different national systems have offered different forms of advice to crews and engineers in response to very similar incidents. There are few examples of direct contradictions, however, there are many cases of inconsistency in the advice that is offered to pilots, air traffic controllers and engineers.
4. Many reporting systems re-iterate the importance of solutions that are recommended as part of standard training activities. The need for such reminders, and the incidents that generate them, raises wider concerns. For instance, some training practices, such as CRM [8], have failed to address many of the problems that continue to be reported with cockpit communication and coordination.
5. There is a considerable body of human factors research that points to the dangers of any reliance on the reminders that are mentioned in the previous paragraph. Unless people are continually reminded then they are likely to forget the importance of safety precautions.
6. Incident reporting schemes have not had a radical effect on the underlying safety culture of some/many companies and organizations within the aviation industries.

Thanks are due to the members of the Glasgow Accident Analysis Group and to the Glasgow Interactive Systems Group.