Improving Accident Reports

Chris Johnson

Glasgow Accident Analysis Group,
Department of Computing Science,
University of Glasgow,

Email: johnson@dcs.gla.ac.uk

Abstract

Accident reports are intended to explain the causes of human error and system failure. They are based upon the evidence of many different teams of experts and are, typically, the result of a lengthy investigation process. They are important documents because they ultimately help to shape legislation. They also guide the intervention of regulatory authorities who must reduce the impact and frequency of human 'error' in the workplace. There are, however, a number of problems with current practice. In particular, accident reports often contain fallacious arguments. Lines of analysis may ignore contradictory evidence and alternative hypotheses. This paper, therefore, presents seven guidelines or heurisitics that are intended to improve the quality of argument in accident reports. Such principles are of little benefit unless analysts have tools that help them to meet these requirements. This paper, therefore, goes on to show how graphical extensions to Knuth's 'literate programming' can be used to avoid the weaknesses of existing accident reports.

Keywords: accident analysis; argument; logic; reasoning; human error; system failure.

1. INTRODUCTION

Accident reports are published by a range of national and international bodies in response to major failures. They contain the findings of multi-disciplinary teams including systems engineers, human factors specialists, software engineers, meteorologists etc. They are critical to the subsequent planning, development and maintenance of complex systems. Accident reports are frequently cited during the public enquiries that precede the development of major production processes, such as Sizewell B, or transport infrastructure, such as the new runway at Manchester airport (Storey, 1996). Accident reports are also cited in the safety cases that companies present to industry regulators. For example, the Cullen (1990) report into the Piper Alpha fire made a series of detailed recommendations about the preparation of these cases for UK offshore oil installations. At a government level, accident reports influence the detailed committee work that shapes all major legislation. For instance, the Taylor (1989) report into the Hillsborough stadium disaster has had a major impact on all subsequent legislation governing large-scale public meetings in the UK.

Accident reports play a critical role in determining the ways in which information systems are introduced into particular domains. They often act as a forcing function to speed up the deployment of existing interactive systems. For example, the Hiden (1989) enquiry into the Clapham rail crash led to a 'priority programme' for the installation of crew communications systems on Britain's railways (recommendations 61-68). Further recommendations in the same report led to changes in the systems between signal boxes and the emergency services (recommendations 81-85). In addition to these high level, infrastructure requirements, accident reports may also contain more detailed recommendations that constrain the development of interactive systems. The Hidden enquiry recommended that all signal box control systems should enable their operators to switch all automatic signals to red in an emergency (recommendation 88).

Given the importance of accident reports for the development of interactive systems, it is surprising that there has been relatively little research into the usability and utility of these documents (Love and Johnson, 1997). The mass of relevant literature about safety-critical interface design (Norman, 1990, Reason, 1990) and even the usability of design documents in general (Moran and Carrol, 1995) is not matched in the field of accident reporting. This omission is all the more surprising because a number of errors and inconsistencies have weakened recent accident reports (Johnson, 1997). This paper, therefore, identifies a range of techniques that can be used to improve the quality of documents that are intended to explain the causes of human 'error' and system 'failure'.

1.1 The Embley Case Study

A collision between the bulk ship River Embley and the Royal Australian Naval patrol boat HMAS Fremantle will be used to illustrate the remainder of this paper (Marine Incident Investigation Unit, 1997). This accident has been chosen because it was the result of complex interactions between several different operators and several different systems. For instance, the crew of the River Embley were equipped with a GPS display, two radars - one of which provided an Automatic Radar Plotting Aid, a gyro compass and bearing repeaters, automatic steering systems and a course recorder plotter. There were also two VHF radio sets and a watch alarm on the bridge. This collision was also the result of complex interactions between the various members of both crews. These interactions were affected not only by the information provided by their on-board systems but also by individual levels of experience and training within the crews. Finally, this accident has been chosen because it typifies the many 'near-misses' that contribute most to our understanding of human 'error' and system 'failure'. Although the patrol boat collided with the River Embley, only slight injuries were sustained. Nobody was seriously hurt and no pollution resulted from the collision.

At 21:00hrs on 13th March 1997, three patrol boats were approaching the Heath reef, part of the Great Barrier Reef, from the South. The River Embley was a deep draught vessel and so was obliged to keep to the Eastern side of a two-way route off the reef. VHF contact was established between the bridge of the HMAS Fremantle and the River Embley. A few minutes after 21:00, the lead patrol vessel Fremantle crossed ahead of the Embley followed by the second patrol boat, in line. The third vessel altered course to pass between the Embley and Heath reef. HMAS Fremantle made a number of small alterations to her course and at about 21:08 the rudder was put 20 degrees to starboard. The patrol boat collided with the River Embley.

1.2 Outline of the Paper

This section has introduced the argument that is presented in this paper and a case study has been introduced. Section 2 goes on to identify four principles that are intended to improve the presentation and handling of evidence in accident reports. These guidelines are heuristic in the sense that they have emerged through discussions with commercial and regulatory bodies as well as through the more formal analysis that is presented in this paper. Section 3 builds on this and argues that as much care must be paid to the structure of argument in an accident report as is paid to the reliability and coverage of the available evidence. Three further guidelines are identified. Principles are of little benefit unless analysts have tools that help them to meet such requirements. Section 4, therefore, presents concrete means of improving the evidence and analysis in accident reports. Literate investigation techniques extend ideas from literate programming (Knuth, 1984) and contextual task analysis (Cockton, Clarke, Gray and Johnson, 1996) to explicitly capture the dependencies that exist between evidence and argumentation in accident reports. Section 5 presents the conclusions that can be drawn from this paper and suggests directions for future work.

2. THE EVIDENCE

This section argues that valuable contextual information may be lost when accident investigators filter and interpret the mass of evidence that is obtained during an enquiry. In consequence, it can be difficult for readers to reconstruct the ways in which human 'error' and system 'failure' contribute to major accidents.

2.1 Eye Witness Accounts

First-hand accounts provide a valuable source of evidence for accident investigators. There are, however, a number of well understood problems (Leveson, 1995). Eye witnesses are often the victims of major accidents. Others are so traumatised that they cannot remember the events leading to an accident. Strong feelings of blame and the fear of judicial sanction can also bias the accounts provided by individuals. As a result, first-hand testimonies are often contradictory (Reason, 1990). The Australian Maritime Incident Investigation Unit (MIIU) report identifies such a discrepancy:

"There is an apparent ambiguity between the angle on the bow at which the patrol boats navigation lights were first sighted, as recalled by the Pilot, Third Mate and Lookout on board River Embley. At interview on River Embley's bridge, the Lookout recalled the lights were to starboard of the foremast light. His position was some 14m from the centre line of the ship. Allowing for parallax, although the patrol boats appeared to the starboard side of the mast, viewed from the centre line they would have been fine on the port bow, as described by the Pilot and Third Mate" (page 27).

The difficulties that people face in recollecting the events before an accident make it difficult for investigators to gather unambiguous accounts. It is, therefore, important that accident reports record any material differences between eye-witness statements if those differences affect their conclusions. For example, the trauma suffered by the crew in the Kegworth crash make it certain that we shall never be entirely sure about their observations of the engine monitoring system in the minutes before the accident (AAIB, 1990). Subsequent analysis of the report has argued that some of its conclusions would have to be revised if elements of the testimony were assumed to be correct (Johnson, 1997a). The important point here is not that investigators must resolve every ambiguity in an eye-witness account. In contrast, the readers of an accident report should be able to assess the consequences of those ambiguities for the conclusions of the accident report. For instance, the MIIU report never fully explains the importance of the "apparent ambiguity" between the Pilot, the Third Mate and the Lookout. If this difference had no impact on the conclusions of the report then it would have been better to omit it. As things stand, it is unclear how the reader should interpret the previous citation.

A number of further problems complicate the use of eye witness accounts to support the conclusions in an accident report. For instance, it is not practical to reproduce the testimony of every witness. Most readers of an accident report do not have the time or motivation to read through many hours of transcripts. Accident reports, therefore, only present those parts of an eye witness account that are thought to be relevant to a reader's understanding. This process of filtration, and then of interpretation, is a necessary part in any accident investigation. It does, however, create a number of problems. In particular, it can be difficult for readers to correctly interpret the analyst's intentions when they do include direct quotations from eye-witness testimonies. For example, the MIIU report contains the following extract:

"The Pilot surmised that the vessels were military and called "the vessels northbound to Heath Reef" on VHF channel 16. A vessel identifying itself as a warship replied stating that it would pass "red to red". The Pilot acknowledged the message, which he assumed came from the lead vessel, which he estimated at this time to be between 2 and 3 miles, nearly dead ahead". (page 12)

The verbatim citations are important because they provide implicit information about the analyst's view of events. The reference to "northbound" vessels shows that in the analyst's view, the Pilot clearly identified the intended recipients of his signal. "Red to red" shows that he proposed a port-side manoeuvre. Conversely, however, direct quotations can also be used to throw doubt upon a particular piece of evidence. The witness' words are used because the analyst does not support them. In this interpretation, the "northbound" vessels citation might be used to indicate the ambiguity of the phrase used by the Pilot. These radically different interpretations for the same citation creates considerable potential for confusion. It is difficult to be sure that the reader's interpretation of the report will be that intended by the analyst. Previous research has shown that the readers of an accident reports often fail to pick up the many implicit indications that analysts intend within such citations (Love and Johnson, 1997).

Further problems complicate the selective use of testimony within accident reports. The filtration of a more detailed conversation can strip out the context that would normally support the interpretation of a phrase or sentence. There is, therefore, a danger that selective citation will distort the witness' original evidence. Some accident investigation authorities are aware of these dangers. Sub-regulation 16(3) of the Australian Navigation (Maritime Casualty) Regulations requires that if part of a report relates to a person's affairs "to a material extent" then the Inspector must give that person a copy of the relevant part of that report. Sub-regulation 16(4) provides that such a person may also submit written comments on the report. Figure 1 illustrates this feedback loop that is an important strength of the Australian maritime reporting process.

Figure 1: Australian Maritime Regulations Introduce Eye Witness Feedback.

The MIIU report includes an appendix of submissions that provides eye witnesses with an opportunity to re-establish the context of their comments. For example, the report claimed that the River Embley did not clearly display three red lights to indicate that it was a deep draught vessel. In the appendix of submissions, the Master states that:

"the lights within the inner route (of the barrier reef) are not routinely shown and that at times, given the closeness of the rock and reefs to the route in certain parts of the Reef, all vessels are constrained at times" (page 25)

In his response, the Master sought to clarify the interpretation of his evidence. In particular, he emphasised the importance of alternative indications rather than simply mandating the obligatory use of the draught lights mentioned above:

"Perhaps a signal indicating constraint due to length or manoeuvrability would be more appropriate? Perhaps a conclusion indicating that a more appropriate signal in the Barrier Reef "may have provided a greater prompt" is more valuable to assist others to develop from this experience, not just comply with regulations" (page 33)

The previous citations indicate that an accident investigation forms part of an argument between the analyst who reconstructs the interactions during an accident and the reader of the report. In the case of the River Embley collision, the reader is more convinced that the investigators have produced a balanced interpretation of the eye witness accounts because those eye witnesses have been given a statutory right to respond. It is regrettable that this practice is not more widely followed. In most cases, readers have no guarantee that accident reports accurately construct the context of eye witness testimony.

Principle 1: unless eye witnesses have the opportunity to comment on an investigator's treatment of their evidence then readers can have little confidence that important contextual information has not been omitted or summarised.

2.2 'Black Boxes'

It is increasingly common for automated logging systems to be a required feature in safety critical applications. These 'black box' recorders provide reliable evidence about the effects of operator interaction during an accident. Figure 2 reproduces an excerpt from the River Embley's course recorder.

Figure 2: River Embley Course Recorder Tracer

Black boxes help to corroborate other sources of evidence. For instance, the Marine Incident Investigation Unit argued that:

"Comparison of the course recorder roll and engine room print out with the chart position and log book entries were consistent, within a minute in time, with the automatic remote records." (page 21)

Logging systems are fallible. There have been several cases of black box recorders being left running in the aftermath of an accident. The tapes then loop and record over any data that has been collected during an incident (AAIB, 1989). Logging systems also provide erroneous results if they suffer from interference from other instruments. Errors can arise if the recording tracks are not calibrated to the incoming signals or if the remote devices are not functioning correctly in the first place. Even in incidents were these sources of evidence are available, it may not be possible to corroborate all of the operator-system interaction in the lead up to an accident. For example, the MIIU report does not cite course recorder logs for the Fremantle as it does for the River Embley. This is significant because there is disagreement over the Fremantle's actions immediately before the collision. The first citation comes from the main body of the report and indicates a 30 second delay between the change of course. The second citation comes from the River Embley's Pilot in the appendix of submission. He estimates a minimum delay of 35 seconds before the collision:

"When the Commanding Officer arrived on the bridge and was briefed, between 1.5 and 2.5 minutes before the collision, Fremantle was on a course of 008š. The alteration of course by applying 20 degrees of starboard wheel was made within 30 seconds of the collision during which time Fremantle would have covered a distance of about 200m or 230m." (page 27).

"I have gone through a reconstruction of my movements and actions from the first sighting of Fremantle's red light and each time have come up with a minimum of 35 seconds. This was when Fremantle was well into her turn because I could see her red side light and part of her aft deck. The run must have started more than 35 seconds before the collision" (page 34).

A number of reasons might explain why the MIIU report does not refer to an automated log on the Fremantle. The first is that she was a Royal Australian Navy vessel. Referring to such a log within the report might, therefore, have disclosed operational information. However, the report already contains full information about the Fremantle's activities prior to the incident. The second possibility is that ships which are under 50 meters in length are exempt from the reporting constraints that otherwise apply within the Great Barrier Reef. Again this argument can be questioned. If the Fremantle had not been carrying logging equipment then the report should have investigated the need to carry such equipment within congested shipping lanes. The meta point here is that we simply cannot tell which of these hypotheses are correct. The absence of information about the Fremantle's recording equipment not only hinders any effective analysis of the crew's detailed actions prior to the collision but it also leads to doubts about the accuracy of future reports that must rely upon the personal logs of the individuals concerned.

Principle 2: unless analysts provide complete information about the available sources of automatic logging information then it is difficult for readers to determine whether information is being withheld or whether failures in logging provision ought to be addressed by an accident investigation.

2.3 Rules and Regulations

Previous paragraphs have focused upon direct evidence. This comes from the people and systems who were directly involved in the accident itself. Most reports also contain evidence from indirect sources. These provide evidence that is relevant to an accident but which does not directly describe the events leading to the incident itself. For example, rules and regulations provide evidence about the expected norms of operating behaviour. They are indirect because they are unlikely to describe the exact circumstances of a particular incident.

As with eye-witness testimony and black box data, accident investigators must filter and interpret indirect evidence. They must filter it because there are many indirect sources that might have some relevance for an accident. In the MIIU case study these include International and National Maritime Regulations as well as training and operating procedures for naval and merchant shipping in coastal waters. Analysts must interpret these sources because they, typically, refer to general requirements for a wide range of cases. These high-level constraints must be applied to the specific characteristics of the accident 'scenario'. Figure 3 illustrates the role that such sources play within an accident report.

Figure 3: The Role of Legislation within Accident Reports

The MIIU report again illustrates our argument. All emphasis in the following citations is that of the original report:

"The International Regulations for Preventing Collisions at Sea, 1972, as amended from time to time, apply to all vessels upon the high seas and in all waters connected therewith navigable by seagoing vessels.

Rule 2 clearly states:

Nothing in the rules shall exonerate any vessel or the owner, master or crew thereof, from the consequences of any neglect to comply with these Rules or of the neglect of any precaution which may be required by the ordinary practice of seamen, or by the special circumstances of the case." (page 22).

Accident investigators must interpret the events leading to an accident in the light of such general requirements. The River Embley report, therefore, explains how the International Regulations for Preventing Collisions at Sea relates to the accident:

"When River Embley and Fremantle were about 8 miles apart, River Embley was steering 179š and Fremantle and the patrol boats in company were steering 348š. Although the two vessels were converging at an angle of 11š and a speed of about 28.4 knots, both the radar plot and visual observations would have shown that the warships were crossing vessels within the meaning of the Regulations." (page 23).

As with the selective use of citations, the reader must extract the implicit links between the previous regulation and the Embley case study. It is unclear whether the investigator intended that the failure to identify the warships as crossing vessels actually constitutes a "neglect of any precaution which may be required by the ordinary practice of seamen". More importantly, we are not told which precautions were neglected. It is, therefore, difficult to understand why the Fremantle's crew were prevented from correctly using visual or system observations of the approaching ship. This has important consequences for a clear understanding of the human factors failures that led to the accident. It also has important consequences for any companies or regulatory authorities that are trying to learn from previous mistakes.

Principle 3: if analysts use indirect evidence, such as legislation, to make a point about the differences between normative behaviour and operating practices then they must also provide enough detail for readers to assess whether or not the analysts interpretation of those sources is valid within the particular context of an accident.

2.4 Expert Witnesses

The previous section referred to legislation as an indirect form of evidence because it must be interpreted in the context of a particular accident. This interpretation process is, typically, performed by expert witnesses. For example, the MIIU report uses Mankabady's (1991) work on International Shipping Law to explain that a requirement to maintain a proper lookout by sight and hearing, also "involves the intelligent interpretation of the data received by way of [these] various scientific instruments". Figure 4 provides an overview of the role of such experts within the accident reporting process.

Figure 4: Expert Testimony and Indirection within Accident Investigations.

Experts introduce an additional level of indirection between both eye-witness evidence and the output of automatic logging systems, and the readers of an accident report. Expert testimony is, in turn, re-interpreted by investigators who use their evidence to draught the conclusions of an accident report. As before, this process of interpretation is beneficial and necessary. There are, however, a number of important dangers. Accident reports frequently omit information about the methods that experts use to support their testimony. This creates particular problems for any human factors analysis of an accident. Different error-modelling techniques have been shown to produce very different results when applied to the same accident scenarios (Johnson, 1997a). It can also be difficult for readers to assess the reliability of expert witnesses. Accident reports seldom justify their selection of domain experts.

Principle 4: accident reports must provide their readers with evidence not only about an expert's findings but also with some information about the techniques that were used to support those findings.

3. INFERENCE AND REASONING

The previous section has identified a number of ways in which the presentation of evidence can weaken, rather than support, the findings in an accident report. This section focuses more narrowly upon the quality of the argument that is used to support those findings.

3.1 Direct Arguments

A number of common argumentation techniques are exploited in accident reports. Unfortunately, it can be difficult to strip aside the details of a particular incident to examine these common features. One means of doing this is to use the vocabulary of logic that stretches from Barwise and Perry's Situations and Attitudes (1983), through Russel to the Ancient Greeks. For example, the term modus ponens refers to a 'method of affirming'. This can be thought of as a technique that allows us to derive particular conclusions from a known set of facts. If we know A and we have a rule of the form 'if A then B', we can use these two pieces of knowledge to conclude B. Readers are often forced to employ such inferences in order to understand the findings that are presented in accident reports. For example, the conclusions of the Embley report state that:

"The reasons for HMAS Fremantle's actions...involve a complex chain of human factors, which include, but are not limited to:

In other words, an accident was likely to occur if the Fremantle's crew conducted incomplete passage and contingency planning and they were unaware of reef traffic and they lacked experience of traffic encounters in the reef and they made a decision to apply 20 degrees of starboard helm based on incomplete and scanty information. In order to establish this explanation, we need to ground it in the evidence that was gathered during the investigation. For example, the conclusion that the crew did not obtain enough information is supported by evidence on page 18 of the report. The commander was clearly unaware of the position of the Embley as he ordered the manoeuvre:

"The Commanding Officer asked what rudder angle had been ordered and the Fourth Officer told him 10 degrees, and the Commanding Officer advised him to increase the angle to 20 degrees. At this time he became aware of the voices on the VHF. Almost immediately the Commanding Officer saw a green light and became aware of a "great black wall". He immediately issued direct orders to the helmsman of "hard to starboard" and full astern" (page 18).

Other conclusions are less easy to support. For example, the inadequacy of the Fremantle's passage and contingency planning was criticised in the following terms:

"Fremantle was also following a plan in accordance with RAN (Royal Australian Navy) operating procedures. However, whatever the quality of the plan, it was predicated solely on the 2.4m draught of the patrol boat and did not identify the waters off Heath Reef as being restricted for deep draught vessels or make any contingency for meeting a vessel constrained by its draught in an area where over half the width of the marked two-way route is less than 15m." (page 29)

The difficulty for the reader is that there is not enough information about the characteristics of the planning that was performed to explicitly determine whether or not it was adequate. In particular, a human factors analysis would suggest that it might have been impossible to predict all of the contingencies that could arise during such a navigation (Suchman, 1987). It would be difficult to anticipate all of the places in the passage where they might encounter deep draught vessels. Given these criticisms it is important that companies and regulatory authorities be given some clear indication of the ways in which the Fremantle's planning fell short of that expected.

Principle 5: analysis must be presented at a level which supports the report's findings AND enables designers to improve future systems

3.2 Identifying Weaknesses

It is important that analysts question the quality of argument in their reports in the same way that they question the quality of evidence from first-hand testimony, automated systems and expert witnesses. This questioning attitude can help to predict, and therefore remove, some of the doubts that readers express about the arguments in accident reports (Love and Johnson, 1997). For example, the MIIU found that the Fremantle's crew lacked experience of encounters within the Great Barrier Reef (page 30, cited above). Readers have to remember the evidence for this conclusion that is presented on pages 8 and 16 of the report. It was the Fourth Officer who was in charge in the run-up to the collision and he was undergoing watch keeping. such significant details in order to understand the conclusion on page 30:

"It (the Fremantle) normally operates with a crew of 23, but on 13 March the crew numbered 24. This included the Commanding Officer, the Executive Officer, the Navigating Officer and the Fourth and Fifth Officers, both under watch keeping training." (page 8).

"The Commanding Officer remained on the bridge monitoring the Fourth Officer until 21:20 when the Patrol Boat was off Hay Island. The Fourth Officer was fixing the ship's position every 6 minutes. Satisfied that the Fourth Officer was in complete control of the situation the Commanding Officer went to his cabin, about three metres from a flight of eight steps that led from the main deck to the bridge." (page 16)

Modus tollens, or method of denying, represents one way in which readers can attack the argument that is presented in an accident report. Informally, if we have a rule which states that if A is true then B is true and B is not true, we can conclude that A is also not true. This follows because our original rule does not allow A to be true and B to be false. As with modus ponens, the complexity of this argument illustrates the power of informal reasoning that reader's intuitively perform. In terms of our case study, we have a rule that says that if the Fourth Officer was undergoing training and was in charge immediately before the collision then the crew lacked experience of encounters on the reef. Modus tollens leads us to attack the conclusion that the crew lacked experience in reef encounters. The MIIU report tells us little about the Commander, the Executive Officer and the Navigating Officers' previous background. If the crew did have some experience of reef encounters then modus tollens raises further questions about the MIIU's argument. For example, page 27 records that the Commander and not the Fourth Officer was in charge immediately before the collision. Reasoning techniques, such as modus tollens, therefore help to identify a range of concerns about the argumentation in accident reports. This, in turn, helps to strengthen the conclusions that are proposed by accident investigators. For example, the MIIU's findings would have been better supported if they had provided direct evidence about the crew's expertise in reef encounters. Such an approach has been adopted as best practice within aviation reports. Full details are, typically, provided about the background of crew members in UK AAIB reports (1989).

Principle 6: analysts must systematically consider the doubts that a reader might have about their conclusions. Where possible, they must also provide additional evidence to address those doubts.

3.3 Indirect Arguments

Modus ponens and modus tollens are direct proof techniques. In the former case, the analysts must seek evidence that directly supports particular conclusions. In the later case, the contradiction of a conclusion helps the analyst to challenge and re-interpret the available evidence. We have seen, however, that direct evidence may not always be available. Eye witnesses, 'black boxes' and expert testimonies frequently only provide partial information about the vents leading to failure. This forces accident investigators to exploit indirect forms of reasoning. Reductio ad absurdum is an example of one of these techniques. This proceeds by assuming the opposite of the thing that you want to prove. You then show that it is impossible or irrational to hold this assumption. By proving that it is incorrect to assume otherwise, you indirectly provide support for the thing that you want to establish. For example, it was argued that, although the lighting on the Embley did not fully conform to the International Collision Regulations, they did not materially contribute to the causes of the accident:

"The absence of the deep draught signals on the River Embley cannot be said to have directly contributed to the casualty. The patrol boats were advised that she was constrained by her draught and this was apparently acknowledged". (page 30).

To assume that the deep draught signals would have affected the course of the accident is nonsensical because, even without these signals, the patrol boats knew and acknowledged that the River Embley was constrained by her draught. This is a weaker form of argument than modus ponens. There is no firm evidence that the lack of appropriate lighting did affect the course of the accident. There is simply a counter argument which suggests that it did not. The rhetorical weakness of indirect reasoning has important consequences for accident reports. Eye witnesses seldom have access to the wealth of evidence that is submitted to an accident enquiry. As a result, their arguments are must often depend upon indirect reasoning. This makes their argument appear weak in comparison to the direct reasoning employed within the body of a report. For example, the Master of the River Embley rejected the finding that an Aldis lamp might have been used to warn the Fremantle. He argued that the use of an Aldis lamp would not have helped to avoid the collision. He is doing this by showing the significant drawbacks, or 'absurdities', of assuming that the Aldis lamp should have been used:

"As the risk of, or impending, collision had only been observed by either vessel crew immediately before impact, and the sound signals - whose use was close at hand - not by hurrying some 10 meters to the wing (lighting an Aldis light in the wheelhouse would destroy night vision, and be unacceptable both aboard and during an inquiry), were "completed at or just before the moment of collision", use of the Aldis lamp was inappropriate in those brief moments" (page 33).

Similar forms of argument were employed by the Pilot in response to the suggestion that the collision might have been avoided if the vessels had been informed of each others' presence by the reef reporting system. Here the Pilot is showing the absurdity of assuming that the vessels would have been informed by Reefrep:

"It is only very occasionally that a ship is advised of other ships in the Reef, other than those in the section that the ship is entering. Consequently, by the time the ship is halfway through the section, it has passed the reported ships and is then meeting unreported ships which had been in the next section." (page 34).

Such arguments are often dismissed as 'supposition' and yet this form of reasoning is just as valid as the more direct techniques employed elsewhere within an accident report. Investigators can call upon the wealth of material collected in the aftermath of an accident to directly support their arguments. Eye-witnesses are, typically, forced to rely upon partial recollections and indirect inferences to argue their case.

Principle 7: unless analysts are aware of the rhetorical differences between direct and indirect forms of argumentation then they may be tempted to dismiss important but unsubstantiated lines of reasoning that could be substantiated through further investigation.

4. LITERATE INVESTIGATIONS

The principles or guidelines that have been identified in previous sections are of little benefit unless analysts have tools that help them to meet such requirements. This section, therefore, presents concrete means of improving the evidence and analysis in accident reports. Literate investigation techniques extend ideas from literate programming in Software Engineering (Knuth, 1984) and contextual task analysis in Human-Computer Interaction (Cockton, Clarke, Gray and Johnson, 1996) to explicitly capture the dependencies that exist between evidence and argumentation in accident reports.

4.1 Literate Programming and Design Rationale

In 1984, Knuth proposed that automatic indexing techniques could help software engineers to navigate between the various components of complex programs. These links were intended to reflect the dependencies that exist between code and its documentation. Over a decade later, this approach was extended to the more general problems of maintaining contextual information about complex design tasks (Cockton, Clarke, Gray and Johnson, 1996). Literate development techniques explicitly represent the dependencies that exist between different design documents. For example, hypertext links can be used to connect the detailed description of a design feature to the contractual requirement that it satisfies (Johnson, 1996).

Figure 5: Literate Specification for the Warning Cancellation.

Figure 5 illustrates how Rank Xerox's Questions, Options and Criteria (QOC) notation can be used to support this approach (Buckingham Shum, 1995). This diagram is taken from an analysis of a reactor cooling system (Johnson, 1996). The question of how to cancel blow-back warnings is answered by the design option that is specified by automatically_remove_warning. This approach is justified by the criteria that the automatic cancellation of warnings reduces burdens on system operators. It is not supported by the criteria that this helps the operator to observe the warning. Literate development techniques link the elements of these diagrams to design documents. For example, the criteria in Figure 5 are directly taken from requirements documents. Issues of operator confidence and workload refer back to the high-level human factors objectives that were identified in an initial task description for control room operators. In a similar way, options are explicitly linked to the clauses in a specification document.

4.2 Using Literate Approaches to Represent Direct Argumentation

Literate development techniques explicitly present the links that exist between design documents. This approach can be extended to represent the links that exist between evidence and particular lines of argument in accident reports. For example, the options in Figure 5 correspond to different lines of argument that might lead to a conclusion. Criteria can be compared to the evidence that supports or weakens a particular argument. Figure 6 shows how this approach might be applied to the Embley case study. The MIIU report concluded that the Fremantle's crew made several human 'errors'. These mistakes included their failure to complete adequate contingency and passage planning. This analysis is supported by evidence that the crew failed to identify the waters off Heath Reef as being restricted for deep draught vessels, see page 29 of the report. The human errors also included a lack of awareness about the other traffic on the reef. This is supported by evidence that both the Fourth Officer and the Commander assumed that the River Embley was some 2.5 miles away when they were, in fact, much closer. This evidence is cited on page 18 of the report. The Fremantle's crew also lacked experience of encounters within the Great Barrier Reef. This analysis depends upon two related pieces of evidence. Firstly, that the Fourth office was on the bridge in the lead up to the collision and secondly that this officer was undergoing training in watch keeping. Finally, human factors problems led to the collisions because the decision to apply 20 degrees of starboard helm was based upon incomplete and scanty information. The Commander's surprise at the consequences of his decision, cited on page 18 of the report, provide evidence for this assertion.

Figure 6: Conclusion, Analysis, Evidence (CAE) Diagram for the Noordam Collision

An important benefit of this approach is that it provides a graphical overview of the many different arguments that are used in an accident report. It also explicitly represents the links that exist between pieces of evidence that are scattered over dozens of pages in a conventional report. For example, evidence about the crew's level of experience in Reef encounters is cited on pages 8 and 18 of the MIIU report. It can be difficult for readers to identify and remember these relationships in conventional text-based, documents. This in turn can prevent them from forming the implicit inferential chains that are a common feature of many accident reports (Johnson, 1997).

There are strong differences between CAE diagrams and other notations used to support accident analysis, such as Fault Trees (Love and Johnson, 1997). These formalisms are, typically, used to map out a timeline of events leading up to an accident. In contrast, CAE diagrams represent the analytic framework that is constructed from the evidence about those events. In this respect, our approach shares much in common with Ladkin, Gerdsmeier and Loer's WB graphs (1997).

All of the evidence in Figure 6 supports the MIIU conclusion. Previous sections have, however, argued that investigators must directly address evidence or testimony that challenges particular conclusions. This evidence can be represented in a CAE diagram in a similar manner to the way in which negative criteria are represented in QOC diagrams. In Figure 5, a dotted line was used to show that automatic cancellations of system warnings did not increase designer's confidence that operators had observed error messages. Figure 7 extends this technique to challenge the conclusion that the Fremantle's crew were unaware of other traffic on the reef. There is evidence to show that both the Commanding Officer and the Fourth Officer were aware of the presence of other traffic, even if they did not know about the exact location of the River Embley.

Figure 7: CAE Diagram Showing Contradictory Evidence and Alternate Lines of Analysis

This diagram illustrates how a line of argument can raise further questions for accident investigators. The argument that the Fremantle's crew were unaware of other traffic is supported by evidence that both the Fourth Officer and the Commanding Officer incorrectly thought that the River Embley was 2.5 miles away. This, in turn, raises the question 'why did they make this mistake?'. Unfortunately, this issue is not directly addressed within the report. In consequence, readers have to form their own suppositions. For instance, there may have been an insufficient review of the existing situation when command was passed from the Fourth Officer to the Commander. This supposition is supported by evidence that the First Officer relied upon their colleagues' assessment while he regained his night vision:

"The Fourth Officer briefed him (the Commander) as to the situation and was told that Fremantle had room to starboard. Unable to see anything and assuming that River Embley was close, the Commanding Officer told the Fourth Officer to make a greater alteration to starboard." (page 18).

CAE diagrams are not intended to replace the informal argumentation that is found in conventional accident reports. In contrast, they provide a road-map of the evidence and analysis that is often distributed throughout dozens, if not hundreds, of pages of prose. The discipline of constructing diagrams, such as Figure 7, also helps to identify the inconsistencies and omissions that have weakened previous reports (Johnson, 1997). They also encourage analysts to consider the evidence that supports particular lines of argument. For example, the previous diagram forced us to consider the reasons why several members of the Fremantle's crew failed to accurately locate the River Embley.

4.2 Using Literate Approaches to Represent Indirect Argumentation

Previous sections have illustrated the use of CAE diagrams to explicitly represent direct forms of argumentation. Links are shown between a line of analysis and the evidence that supports it. It is equally important to develop graphical representations of indirect reasoning techniques, such as reductio ad absurdum. This style of argument is more common in the verbatim comments of eye witnesses than in the investigator's reports. If this 'weaker' form of reasoning is not included within CAE diagrams then there is a danger that a critical source of argument and insight may be neglected within an accident report. For example, the Master of the River Embley objected to the finding that additional lights might have been used to avert the collision. Figure 8 shows how the CAE notation can be extended to represent his argument, cited on page 33 of the MIIU report.

Figure 8: CAE Diagram Representing Indirect Reasoning.

This diagram represents the argument that additional lighting signals would not have provided an important prompt for the Fremantle because the patrol boats already knew that the Embley was a deep draught vessel. The dotted line is used to indicate that the line of argument contradicts the conclusion in the accident report. No evidence is presented to directly support the Master's assertion. This lack of evidence illustrates the weakness of indirect forms of argument. It also indicates lines of enquiry that might be pursued by accident investigators. For example, the MIIU report contains several pieces of evidence that might be used to support the Master's assertion even though the Master never refers to that evidence within his written submission. Figure 9 shows how graphical representations of argument structures can strengthen the indirect arguments made within eye witness accounts of human 'error' and system 'failure'. The Master's claim that the patrol boats already knew about the deep draught of the River Embley is supported by the fact that the Fremantle's Fourth Officer acknowledged a channel 16 VHF message to that effect at some time after 22:00hrs.

Figure 9: Introducing Evidence into CAE Diagram to Strengthen Indirect Reasoning.

Unfortunately, Figure 9 does not indicate the source of either an argument or any associated evidence. Without this information it will be hard for readers to correctly interpret the reliability of any conclusion. For instance, the MIIU report does not state the source that provided information about the content of the VHF radio message. Previous sections have argued that the reliability of such evidence would be very different if it were supported by an automated logging system, or black box, than if it were supported only by the personal recollections of the Fourth Officer.

4.3 Using Literate Approaches to Ground Evidence

CAE diagrams can be annotated to indicate the source of evidence and analysis. For example, Figure 10 extends the previous diagram show the individual's who were responsible for each of the components in Figure 9. Sources are recorded for both evidence and analysis. The source for a particular line of analysis must be shown if readers are to identify alternative positions for and against a particular conclusion. This is important because inconsistent or inconclusive arguments would be explicitly represented as both positive and negative lines of analysis from the same source.

Figure 10: Introducing Information Sources into CAE Diagrams.

We do not know the source that provided evidence about the VHF radio message. This might have come from the Fourth Officer himself. Alternatively, evidence might have been provided by the Master who received the message. It might also have been provided by a radio monitoring station. Figure 10 explicitly represents this uncertainty by classifying the source as 'unknown'. If, however, the source had been identified then analysts might use the syntax of CAE diagrams to directly assess the strength of the argument that they represent. For example, suspicions would be raised if the same source provided both a line of argument and the evidence that supported the argument. In our example, this would have arisen if the Master's testimony had been the only source of evidence about the contents of the Fourth Officer's radio message. His argument that the Fremantle already knew that the Embley was a deep draught vessel would only have been confirmed by his recollections of the message. Similarly, more weight can be attached to lines of analysis that are supported by multiple sources of evidence. An example of this is shown in Figure 11.

Figure 11: Extending CAE Diagrams to Represent Corroborative Sources

The MIIU report contains remarkably little information about the sources that corroborate particular pieces of evidence. Figure 11 had to be pieced together from background information and from sources that were not cited in the main report. The absence of such detail prevents readers from assessing the credibility not only of the evidence itself but also of the analysis that depends upon it. This brings us back to the central argument in this paper. Readers must be able to assess the reliability of evidence and analysis if they are to correctly interpret the conclusions in accident reports. The MIIU Inspector's analysis in Figure 11 is strongly supported by independent evidence from a range of eye-witnesses and external records. It is an indictment of many existing accident reports, that readers often cannot distinguish such strong forms of argument from the unsupported inferences illustrated in Figure 10. In contrast, the introduction of CAE diagrams encourages analysts to explicitly consider the corroboration of evidence for system 'failure' and operator 'error'.

5. CONCLUSION AND FURTHER WORK

Accident reports are a primary mechanism by which designers can learn from the mistakes of the past. These documents analyse and explain the causes of both human 'error' and systems 'failure'. Unfortunately, a range of recent work has identified limitations and weaknesses in conventional reporting techniques (Johnson, 1997, Ladkin et al 1997). This paper has, therefore, proposed seven principles that are intended to improve the quality of evidence and argumentation in accident reports. These principles or guidelines are heuristic in the sense that they have emerged through discussions with commercial and regulatory bodies as well as through the more formal analysis that is presented in this paper:
  1. unless eye witnesses have the opportunity to comment on an investigator's treatment of their evidence then readers can have little confidence that important contextual information has not been omitted or summarised.
  2. unless analysts provide complete information about the available sources of automatic logging information then it is difficult for readers to determine whether information is being withheld or whether failures in logging provision ought to be addressed by an accident investigation.
  3. if analysts point to differences between normative behaviour and operating practices then it is extremely important that sufficient detail is provided so that readers can accurately identify those differences.
  4. accident reports must provide their readers with evidence not only about an expert's findings but also with some information about the expert's competence and the reliability of the techniques that were used to support their findings.
  5. analysis must be presented at a level which both supports the report's findings AND enables designers to improve future systems.
  6. analysts must systematically consider the doubts that a reader might have about their conclusions. Where possible, they must also provide additional evidence to address those doubts.
  7. unless analysts are aware of the rhetorical differences between direct and indirect forms of argumentation then they may be tempted to dismiss important but unsubstantiated lines of reasoning that could be substantiated through further investigation.
Guidelines are of little benefit unless analysts have tools that help them to meet these requirements. The closing sections of this paper have, therefore, argued that techniques from literate programming (Knuth, 1984) and contextual task analysis (Cockton et al 1996) can be extended to support the presentation of accident reports. In particular, elements of Rank Xerox's QOC notation can be translated into Conclusion, Analysis and Evidence diagrams. These graphical structures provide an overview of the arguments that are presented in accident reports: Much work remains to be done. We have already conducted a number of informal evaluations to assess the utility of CAE diagrams for accident investigators. More validation is required. This raises further research issues. Simple comprehension tests or qualitative measures are unlikely to give reliable results for tools that are intended to support designers and analysts in large organisations. Further problems arise because accident reports support a wide range of evolving tasks. Lawyers use them in litigation, governments use them to support legislation, designers use them to prepare safety cases. It is, therefore, important that we validate our approach within these different contexts of use. Although we have now completed two sustained case studies of literate reporting techniques in maritime accidents, further examples are needed before we can apply this technique to the formal investigation of a 'real' accident (Johnson, 1997).

Little has been said about the costs associated with literate investigation techniques. We have deliberately kept our graphical notation as simple as possible in order to minimise training overheads. Tool support is, however, necessary if this approach is to be a commercial success. We have recently developed such a system. This enables multiple users to cooperate during the construction and editing of CAE diagrams. Different teams of investigators can, therefore, simultaneously add and amend evidence about different aspects of an accident. Further work is, however, needed to determine what version and concurrency control techniques are necessary to support this form of Computer Supported Cooperative Work (CSCW).

Finally, CAE diagrams were developed from ideas in contextual task analysis. This approach provides means of integrating diverse design and implementation documents within a mutually supportive web of development information. Future work intends to extend this approach to include CAE diagrams. For example, links might be drawn from the conclusions of an accident report to the criteria in a QOC diagram. This would provide designers with an explicit means of justifying particular development decisions in terms of the "lessons of the past". Such a holistic approach would link the analytical techniques of accident analysis to the constructive approaches of software engineering and systems development. The constructive use of CAE diagrams to support design is discussed in Learning the Lessons of Human 'Error' and Systems 'Failure'.

ACKNOWLEDGEMENTS

Thanks are due to the Australian Department of Transport and Regional Development and to the Maritime Incident Investigation Unit. Their openness has greatly helped efforts to improve accident reporting. I would also like to acknowledge the advice and support provided by members of the Glasgow Accident Analysis Group and the Glasgow Interactive Systems Group. This work is supported by EPSRC grants GR/L27800 and GR/K55042.

REFERENCES