Chris Johnson and Bryan Mathers,

Department of Computing Science,

University of Glasgow

Glasgow G12 8QQ.

Scotland.

{johnson,mathersb}@dsc.gla.ac.uk

+44 141 330 6053 (tel.)

+44 141 330 4913 (fax)

Function allocation is a neglected and critically important aspect of Computer Aided Learning (CAL). The division of responsibilities between human trainers and computer applications can determine the successful acquisition of both technical knowledge and practical skills. Previous work in this area has focussed upon the human contribution to CAL but has not focussed on the particular problems of function allocation. For example, Laurillard and Taylor's (1994) analysis of CAL in the classroom concluded that:

Laurillard and Taylor focus on the supervisory contribution to learning. There are other reasons why human support is important. For instance, teachers and trainers often help students to interpret educational material within the context of a particular organisation. They emphasise and re-iterate concepts that are particularly relevant from the students' perspective. This is necessary when "off the shelf" packages cannot be tailored to the specific educational and training needs of particular users.

Human mediation is also important if CAL systems do not keep pace with the rate of change in an organisation. This applies to curriculum changes in a school syllabus. It also applies to commercial organisations whose working practices are revised after the introduction of new equipment or in response to regulation (Cohen, Candland and Lee, 1995). Most commercial training materials are out of date "as soon as they are published". In consequence, tutors and trainers must make their students aware of the inadequacies of CAL resources.

Finally, human intervention is important because the introduction of CAL technology has a profound and often unpredictable impact upon learning practices within organisations (Hewett and Tscheligi, 1995). In particular, the relative importance of data acquisition and information interpretation has been altered by technological innovation. Educators must, therefore, monitor the impact that new tools have upon their students if they are to set appropriate aims and objectives.

The previous section has argued that the support activities of human educators can determine the success or failure of CAL tools. This suggests that the development of education resources is better supported if designers explicitly considered the allocation of educational functions between a teacher and their computer systems. Unfortunately, few design techniques can be recruited to support this task. Most existing methods focus upon the support that particular media or narrative techniques provide for student learning styles. Relatively little research focuses on the pragmatic problems of allocating particular educational tasks to either automated systems or human tutors.

Figure 1: Laurillard's Conversational Model for Effective Education.

Figure 1 illustrates how both the student and teacher contribute to the learning process. The right-hand components of the diagram represent the iterative process by which students modify their view of the concepts that they are being taught. They adapt their descriptions (activities 4 and 9) and modify their actions (activities 8 and 10) as they learn from their tutor's feedback. The left-hand components of Laurillard's model describe the iterative process that informs the teacher's interaction with the student. The tutor modifies the way that they present key concepts in response to a student's initial attempt to express those concepts (activity 3). They also modify the tasks that they set in response to their student's initial attempts to fulfil those tasks (activities 11 and 12). This notion of dual contribution explains Laurillard's view that education is essentially a conversation in which both the student and the teacher are attempting to maximise their common ground about the topic in question.

Most of the tutor's tasks in Laurillard's model can be performed by either a human instructor or by an appropriate CAL tool. For example, stage five involves the tutor in establishing task objectives for the student. This could take the form of an essay title set by the teacher. Alternatively, it might be implemented as a computer generated construction task embedded within a virtual reality simulation. This paper, therefore, sets out to determine whether Laurillard's model can help designers to identify appropriate allocations of training tasks within a complex and dynamic organisation (Montgomery, 1997). There is considerable evidence that such domain characteristics help to determine appropriate function allocations between human trainers and CAL applications (Plowman, 1996). However, most of this evidence focuses on educational practices in Universities or in schools. This paper, therefore, looks beyond these conventional settings to examine the impact of function allocation within the training mechanisms of one of Scotland's regional Fire Brigades.

This paper focuses on the introduction of CAL techniques to support the training of fire fighters. In order to understand the reasons why CAL was needed, it is important to understand the existing training regime. The Fire Brigade is divided into a number of watches. Each watch will, typically, rotate between day and night shifts.

Figure 2: Fire fighter using Lukas Cutters during a Practical Exercise

If a watch is on night shift and practical training is no longer possible then a technical training activity will be selected. These, typically, involve lectures that last three-quarters of an hour and are delivered by a Training Officer. Between five and ten minutes are then spent in question and answer sessions. Each technical session covers one of seventy-three subjects that form part of a two-year cycle. Fifteen of these topics are considered to be of special importance and are, therefore, repeated annually.

Completion of these various tasks is recorded on the individual's training card. However, training activities are interrupted if the watch is called to an incident.

Senior officers viewed CAL as a cost-effective means of raising standards in both practical and technical training. There was a concern, however, that automated systems should be targeted to provide the greatest support in areas where existing training techniques were perceived to be weakest. A questionnaire was, therefore, issued to two fire stations within the brigade. A total of twenty-seven returns were received.

Attitudes towards computer instruction packages and human tutors are partially determined by the amount of experience that an individual has in their occupation (Lave and Wenger, 1991). This has important consequences for the introduction of CAL into organisations such as the Fire Brigade. In particular, highly experienced individuals will, typically, react more positively to human trainers who are perceived to have shared some of their experiences. Conversely, experienced individuals are more likely to distrust CAL tools whose programmers may be viewed with scepticism and distrust. Figure 3 presents the range of experience within the two stations that were studied. As can be seen, only one fire fighter had less than three years experience. The majority had served in the brigade for more than twelve years. Such findings emphasise the importance of function allocation. There is a danger that experienced fire fighters may become alienated if training tasks are simply transferred from human instructors to CAL systems. Designers must, therefore, carefully consider the way in which such systems are to be integrated into the existing training practices of human tutors.

The questionnaire went on to assess the fire fighters' attitudes towards existing training methods. At the start of the project, they did not have access to any CAL systems. Training consisted of lectures and videos as well as drill-based instruction. It was also recognised that many fire fighters viewed "real" incidents as an important means of acquiring new knowledge and reinforcing key skills. Figure 3 presents the fire fighter's assessment of their training mechanisms in terms of "Ease of Learning. From the diagram, it is apparent that the fire fighters believed "real" incidents to be the easiest means of learning about their occupation. This finding has important health and safety implications. Incidents should reinforce training gained by other means. They are clearly not a satisfactory delivery mechanism for basic instruction.

The fire fighters identified lectures and video presentations as their least popular training methods. This is interesting for two reasons. Firstly, lecture based training is the primary mechanism by which human tutors deliver technical material to the fire fighters. Secondly, video presentations are perhaps the closest approximations to CAL tools that many of the fire fighters have experienced. The results of Figure 3 indicate that neither resource is being effectively deployed.

Figure 4 presents the fire fighters' response to the perceived effectiveness of different training methods. As can be seen. There is a strong correlation between these results and the responses shown in Figure 3. Lectures and videos are relatively ineffective training mechanisms in comparison to incident based training.

Figures 4 and 5 provided few insights into those features that reduced the effectiveness of lecture based training. Similarly, they did not identify those attributes that increased the perceived effectiveness of "real" incidents. These, more detailed, perceptions were address by two further questions. Fire fighters were encouraged to write down their feelings about the best and worst aspects of fire training. Table 1 presents a summary of their responses.

a) What do you enjoy most about fire training?

Table 1: Categorisation of Responses to Open Questions About Training Methods

Table 1 a) re-iterates the importance of practical training. It reveals that activist learning styles characterise many people who are attracted by, or are recruited into, the Fire Brigade. The importance of new experiences also suggests that any revised function allocation between human and computer based training should continue to provide a varied menu of training experiences. Similarly, if more training functions are to be implemented within CAL systems then there is a clear concern that this should not jeopardise the benefits of team-based learning that seem to be a significant strength of existing practices.

Table 1 b) provides further requirements for any revised function allocation. The fire fighters' antipathy towards repetitive exercises reinforces the importance of new experiences in Table 1 a). The fire fighter's irritation with detailed training exercises is an interesting finding. It, perhaps, suggests a need to continually situate exercises within "realistic" scenarios. However, the limited scope of the survey provides insufficient evidence to reach firm conclusions about this.

The initial analysis indicated that there was scope for improving training practices within the Fire Brigade. Senior staff in this organisation viewed CAL as a primary means of achieving such improvements. There was an implicit view that the re-allocation of training activities between computer-based training systems and human instructors would alleviate some of the problems that were revealed by our survey. The questionnaire also suggested that any re-allocation of training functions towards CAL tools must:

1. exploit the positive features of incident-based learning.
2. avoid the problems of lecture-based training.

Unfortunately, the questionnaire did not provide detailed evidence about the exact ways in which function allocation might be used to achieve these objectives. Nor did it identify ways in which the intervention of human instructors and CAL systems might be integrated to support the technical training and practical expertise of the fire fighters.

Two practical problems must be addressed if Laurillard's conversational framework is to guide the allocation of training tasks between CAL systems and human instructors. The first centres on whether these models can be used to identify weaknesses in an existing allocation. The second is whether such models can then be used to identify a better allocation through the introduction of new CAL systems or, more controversially, through the provision of greater human support for training activities.

Laurillard's model was primarily intended to promote a conversational view of education in which tutors and students maximise their shared view of any topic. Montgomery (1997) has, however, shown that her framework can also be used as a practical design tool. Activity-implementation charts provide check-lists that analysts can use to note whether each of Laurillard's 12 stages are supported by either human tutors, CAL systems or some other mode of learning. . Montgomery focussed on University education. In contrast, Table 2 shows how activity-implementation charts can also be used to analyse technical training within the Fire Brigade. A blank in the Teaching Mode column indicates that a learning activity is not supported by the current allocation of educational tasks.

Table 2 shows that many of Laurillard's learning activities are not explicitly supported. In particular, the use of traditional lecturing techniques leaves little room for the task-based activities in Laurillard's conversational model. If Montgomery is to be believed then this indicates an important opportunity for increasing the intervention of either human tutors or CAL applications. Such opportunities illustrate a basic point about function allocation; it is only possible to derive an appropriate allocation of responsibilities between automated and manual systems if designers can identify necessary functionality in the first place.

An important insight from our application of Montgomery's techniques was that problems still exist even if training activities already support a particular stage in Laurillard's model. For example, Table 2 indicates that fire fighters have an opportunity to describe key concepts in either written or spoken form; each technical lecture is followed by an open session of questions and answers.. This is a key component in stage 2 of the conversational framework. However, initial observations revealed that some fire fighters failed to ask any questions after technical training sessions. This is an important problem. Laurillard's dialogue depends upon the tutor being able to gauge their students' understanding through the questions that they ask. It also relies upon students testing whether or not they have correctly understood the key concepts that are being taught. Low levels of student participation in conventional lectures short-circuit the dialogue that Lauillard views as being essential to effective education.

Table 2: Activity-Implementation Chart for Technical Training in the Fire Brigade

Table 3 extends our application of activity-implementation charts to analyse the practical training that is provided by the Fire Brigade. This shows that the stages of Laurillard's model are better supported for practical training than they are for technical training. This imbalance perhaps results from the perceived importance of practical skills within the organisation. However, it also confirms the senior officers' view that function re-allocation, through the increased use of CAL techniques, offers the greatest benefit to technical training activities.

Table 3: Activity-Implementation Chart for Practical Training in the Fire Brigade

Our analysis identified a number of concerns about the use of Laurillard's model to assess function allocation. In particular, Montgomery's approach focuses upon the role of human tutors and CAL applications within controlled environments, such as Universities or Schools. It does not consider the ways in which "real" events can be used to reinforce practical skills and technical knowledge. This is important because Section 2 demonstrated that fire fighters perceive incidents to be the most efficient and effective means of training. The importance of these learning opportunities is also recognised by the Fire Brigade. Experience gained during real incidents is logged in the training records of individual fire fighters. Future work might address this limitation of Laurillard's model by introducing additional rows into activity-implementation charts. There must be some mechanism by which tutors can refine their drills in the light of student performance not only in simulations but also in real-world incidents:

Table 4: Proposed Extension to Activity-Implementation Charts for Practical Training

There is a strong reliance upon human support for the acquisition of practical skills by fire fighters. Question and answer dialogues again provide their instructors with the primary means of assessing whether or not they have acquired the skills that are being demonstrated. This has important implications for revised function allocations. Video clips and destkopVR techniques can provide fire fighters with useful guidance about the performance of practical tasks. However, the introduction of these CAL resources must not jeopardise the feedback that is necessary if tutors are to be sure that fire fighters can perform the tasks that are demonstrated to them. Similarly, the introduction of CAL resources should not jeopardise the feedback that is necessary is fire fighters are to gauge whether or not they have successfully acquired the necessary practical skills.

Activity-implementation charts help to identify unsupported activities in Laurillard's conversation model of education. They can also be used to identify activities that would benefit from additional support. This section argues that, by identifying such activities, it is possible to target the re-allocation of training functions to maximise the effectiveness of both human instruction and CAL techniques.

Our initial survey suggested that the lecture-based presentation of technical material provided limited opportunities for fire fighters to interact with their instructors. This analysis was supported by the relatively weak support for stages 2 and 4 of Laurillard's model that was identified in Table 2. Many fire fighters did not play an active role in the question and answer sessions that were intended to reinforce technical material. A number of authors have argued that CAL tools can be used to address such concerns (Cohen, Candland and Lee, 1995). Fire fighters are likely to be less reticent in exploring problem areas with the assistance of a computer-based tool than they are in asking questions in front of their colleagues. CAL tools can also reduce the problems that arise when training sessions are interrupted by a call-out. Fire fighters can work through a package at their own speed and resume their interaction when other duties permit.

These claims were tested by the development of a CAL tool that provided an introduction to the equipment carried on a Heavy Rescue Vehicle (HRV). A central concern was to avoid the criticisms of lecture-based presentations that were revealed by the initial survey. The main design aim was, therefore, to maximise interaction with the subject material. The principle technical means of achieving this was through the application of desktopVR. In particular, the photo-realistic facilities of QuicktimeVR were used to construct a three-dimensional representation of the storage area inside the HRV. This approach provides high levels of support for activist learning styles; fire fighters can use their keyboard and mouse to move inside the vehicle itself (Johnson, 1998). Equipment can be located either through exploration using the desktopVR view, shown in the middle panel of Figure 5, or by selecting an item from the list on the right.

Figure 5: The Heavy Rescue Vehicle Training Package

The HRV package was intended to move some of the technical teaching duties away from the Training Officers and onto a CAL application. It was, therefore, necessary to provide more detailed information about each item of equipment on the vehicle. For example, fire fighters could have accessed the text from a conventional lecture through hypertext links in the QuicktimeVR model. This would have ignored many of the benefit from reallocating training activities to CAL systems. Alternatively, Figure 6 shows the different multimedia resources that were provided for the HRV package. Detailed textual material was presented in the form of technical notes. Images of the equipment "in action" were presented through video clips. Our initial survey indicated that such diversity was extremely important from the users' perspective. Fire fighters accessed the multimedia resources by selecting from an option list that was provided for each item of equipment. This enabled fire fighters to choose the information resource that best met their needs. They were not constrained by the pre-determined ordering of material in a lecture.

Figure 6: Multimedia Resources in the Heavy Rescue Vehicle Training Package

Two further features of the HRV package emerged as a direct result of our work into function allocation. The first of these concerned the presentation of particular items of equipment. Our need to improve upon the passive nature of traditional lectures persuaded us to support three-dimensional interaction with these objects. The aim was to increase the fire fighters interaction with the material in a CAL tool in a manner that is difficult, or impossible, to achieve given the constraints upon human instructors in conventional lectures. The intention was to enable fire fighters to directly manipulate the equipment so that they could observe significant details, such as power connections. Figure 7 shows the results of this approach for Lucas cutters. QuicktimeVR techniques were again used to enable fire fighters to manipulate three-dimensional views of the HRV equipment using a conventional keyboard and mouse.

Figure 7: Object Rotations in the Heavy Rescue Vehicle Training Package

The second contribution of function allocation was in the integration of video material into the HRV package. It has already been noted that both human instructors and video presentations were perceived as being less effective than direct involvement in incidents. In developing a CAL system to replace some aspects of human instruction, it was important to learn as much as possible from criticisms of the existing approaches. Video was, therefore, used to illustrate the practical application of rescue equipment within simulated rescues. This approach is shown in Figure 8.

The HRV package was intended to reduce the reticence that some fire fighters exhibited during question and answer sessions. This was identified as a significant problem because Laurelard's model stresses that learners must be provided with the opportunity to verify that they have correctly understood key concepts. CAL helps to address such concerns because fire fighters can use an automated system to check their understanding in their own time. They can also consult a CAL application without exposing any possible confusion to their colleagues or to a Training Officer.

Montgomery's (1997) Activity-Implementation charts can also be used to assess the impact of changes to an existing task allocation. For example, the HRV package supported stage 1 of Laurillard's model. It addressed the fire fighters' criticisms of conventional lectures by enabling them to interact more directly with technical training material . One side effect of this was to weaken support for stages 3 and 12. The introduction of a computer based system actually reduced the opportunity for the trainer to modify their material in response to the changing needs of their students and of the Fire Brigade. The HRV package was designed as a self-contained product and could not easily be updated by Training Officers. This is a significant weakness. If fire fighters reported usability problems or identified further informational needs then Training Officers were forced to provide ad hoc support through additional lectures or printed documentation. This reduced the benefits to be gained from allocating training tasks to a CAL tool.

A second package was, therefore, developed to provide Fire Brigade personnel with means of extending and maintaining their CAL resources. This approach is supported by Laurillard's conversational view of education because human instructors can tailor the presentation of material to support their students needs. Training Officers, rather than external designers, can direct the allocation of CAL tools to those areas that they consider to offer the best rewards. Unfortunately, this approach also creates a new set of design problems. The HRV package was tailored to support particular training activities. In contrast, the new user interface was to provide fire fighters with access to the wide range of material that Training Officers wanted to present. This included information about safe lifting procedures as well as details about the effective use of breathing apparatus and good operating practices for rescue equipment. A slide metaphor was chosen and Training Officers were provided with a tool to link-in text and multimedia resources. The result can be thought of as a highly simplified version of Microsoft's PowerPoint interface. Figure 9 illustrates the results of using of this tool to generate a training package about foam delivery systems.

The Slide Package again shows that Laurillard's model can be used to identify weaknesses and omissions in the existing division of labour between human instructors and computer-based training packages. The HRV package did not enable Training Officers to tailor the presentation of material in response to changing user needs. The Slide Package overcame these objections by enabling officers to create their own CAL resources. It is important to emphasise, however, that the use of such educational models does not replace iterative design techniques. For example, the decision to use a slide metaphor owes more to Lydia Plowman's (1996) study of narrative in multimedia than it does to Laurillard's conversational model of educational tasks:

Figure 9: The Self-Instruction Slide Package

In retrospect, the HRV and Slide Packages failed to address many of the fundamental concerns raised by Laurillard's model. The HRV package supported the presentation of material, embodied in stage 1 of the model. The Slide Package enabled Training Officers to update CAL material in stages 3 and 12. They did not, however, provide any support for stages 4-8 in table 3. These stages focus on the use of tasks and exercises to provide students with feedback about their understanding of key concepts. These stages also provide the tutor with feedback about the need to improve their delivery techniques. A third package was, therefore, developed to address the failings of previous applications. An important concern was to deliver a flexible and extensible assessment system that could be tailored by Fire Brigade staff in the same way that the slide package could be extended. We were also concerned to support a diverse range of comprehension tools to avoid the fire fighter's antipathy towards repetitive training exercises.

Further analysis also revealed that certain questions should be classified as "need to know". From the fire fighters' perspective, it was important that they could identify these questions as being critical to their understanding of the topic. From the instructors' perspective, these questions helped to assess both the fire fighter's grasp of the central topics but also the performance of their instructional systems in indicating the degree of importance associated with particular topics. This issue is especially important where training functions are being re-allocated away from human instructors towards CAL systems. Training Officers use a myriad of rhetorical techniques to emphasise the importance of key topics during a lecture: repetition, prosodic stress, visual movements and expressions. Many of these cues are entirely lost when training tasks are transferred to CAL systems.

Figure 10: A Photographic Multiple Choice Question

Self-assessment techniques, such as that shown in Figure 10, provide fire fighters with a mechanism by which they can gain feedback about their understanding of key topics. Our analysis of stages 7 and 12 in Laurillard's model also indicates that it is important for Training Officers to get feedback about the effectiveness of their training techniques. For this reason, the Comprehension Package provided logging facilities. Tutors can track the performance of individual fire fighters as well as groups of students. It is important to emphasise, however, that this system was intended to aid individual comprehension rather than provide a means of assessment. Our intention was to raise the level of understanding within the group rather than to achieve a normal distribution of marks around the mean. Even so, the development of this tool raised important social questions about the use of logging techniques. Brevity prevents a detailed analysis of issues that arise when assessment functions are re-allocated from human tutors to CAL tools. In passing, however, it is critical to note that any analysis of this topic must consider not only the fire fighters' attitudes but also the attitudes of their representatives within the various Trades Unions.

Stage 11 of Laurillard's model centres on the way in which a teacher modifies a task to address some need revealed by the learner's previous descriptions or actions. Such modifications can be automated through the application of user models to adapt questions in response to students' interaction. However, this approach could not easily have been adopted in this project. Firstly, the user model would have had to cover the vast range of technical and practical subjects that the Training Officers supported through the Slide Package. Second, it was difficult to envisage a user model that could be developed and maintained by the Training Officers as they observed successful or unsuccessful adaptations by the user model. The application of user models was, therefore, rejected in favour of simplifying the mechanisms by which questions were generated. Figure 11 illustrates the Training Officer's interface that was used to develop the questions that the Comprehension Package presented to the fire fighters.

Figure 11: Question Generation in the Comprehension Package

To summarise, the HRV package helped to transfer lecturing duties away from Training Officers towards an interactive CAL system. However, the changing requirements of the Fire Brigade forced the design of a more extensible tool. This resulted in the Slide Package. The need to reinforce key concepts and assess the fire fighters' understanding in turn led to the development of the Comprehension Tool. This illustrates how the iterative development of CAL tools profoundly affects the allocation of training responsibilities within complex organisations. These various packages reduced the need for lecture-based tuition but increased the need to monitor group performance and tailor CAL resources to their users' needs.

Previous sections have stressed that our application of education theory does not guarantee a successful reallocation of training functions. Laurillard's model indicated weaknesses in the existing practices but did not prescribe concrete means of avoiding those weaknesses. In consequence, there must be some means of assessing whether our introduction of presentation tools, such as the Slide Package, and self-assessment tools, such as the Comprehension Package, actually supports users' needs. This implies that designers must have some means of assessing the utility of particular function allocations. A number of different perspectives must be taken into account. For instance, any reduction in the tutor's load may be supported by Training Officers but rejected by the fire fighters. Further problems arise because changes in function allocation can have longitudinal effects that are difficult to predict and to measure. A move towards CAL tools is often perceived to threaten the long-term careers of instructors whose expertise may become devalued. Conversely, the introduction of CAL techniques can create new challenges for officers who must design and mainatin the content of these tools. The evaluation of function allocations is a research area in its own right. Given the pragmatic nature of this project, we needed some means of assessing whether or not our integration of CAL actually supported the fire fighters' understanding of technical material.

It was decided to focus on whether the Comprehension Tool helped to reinforce the fire fighters' understanding of particular topics. A matched subject design was adopted; each fire fighter was paired with another officer of equivalent rank. Each member of the pair was then randomly assigned to one of two groups. Both groups were given access to the foam resources that had been developed using the Slide Package. One group was then given the Comprehension Tool. They were encouraged to use it to identify any areas that they might not have understood during their interaction with the foam tool. The second group was given a pencil and paper test without any feedback about the accuracy of their responses. One week later both groups were re-tested using the Comprehension Tool. It was hypothesised that the group that had previous access to the self-assessment tool would achieve significantly higher scores than the group that had performed the pencil and paper test. This hypothesis would support the claim that fire fighters could use the Comprehension Tool to improve their grasp of the core material after using the Slide Package, as suggested by stages 6 and 8 in Laurillard's model.

A weakness in this experimental design is that learning effects might improve the results of the group that already had some experience with the Comprehension Tool. These effects were minimised by ensuring that both groups were entirely confident in the use of the tool before the second test began.

Tables 6 and 7 present the results obtained for the two groups involved in this evaluation.

Table 6: Results for the first group of Fire Fighters.

Table 7: Results for the second group of Fire Fighters.

A T-test failed to show any significant differences in the re-test scores between those who had access to the Comprehension Tool and those who sat the pencil and paper test. A number of reasons explain this finding. Firstly, the Comprehension Tool was used for the re-test. As mentioned before, this application was intended to reinforce key concepts rather than to achieve a normal distribution of marks. Secondly, there were some minor problems in applying ranking techniques to the two watches. The evaluation was forced to match the station officer in the second group with a sub-officer in the first group. Thirdly, problems arose from the context in which the evaluation was conducted. In the middle of the second session, one of the watches was called out to attend a fire. The results for the second group are, therefore, split between those who were tested one week after initial exposure to the foam package and those who were tested one week after that.

These minor concerns hide a deeper sense of unease about the evaluation of function allocation in complex organisations. The Slide Package and Comprehension Tool enabled the Fire Brigade to develop and maintain their own CAL resources. Training Officers, therefore, determined the extent to which automated tools supported their training activities. They controled the detailed allocation of training duties between human instructors and CAL systems. Unfortunately, we could provide them with little guidance about how to assess the impact that such allocations would have upon the knowledge, skills and "real world" performance of fire fighters.

Function allocation is a critically important aspect of computer aided learning. Human instructors help to reiterate the key concepts that are presented in CAL tools. They also help to ensure that students are made aware of the omission that arise when educational packages are not updated frequently enough to reflect changing training requirements. It is, therefore, important that designers consider the impact that new CAL tools can have upon the existing allocation of training activities. There is a danger that the introduction of automation will jeopardise the support activities that human instructors must provide if students are to be provided with relevant and up to date information (Hewitt and Tscheligi, 1995). This paper, therefore, argues that Laurillard's (1993) conversational model can be extended to analyse the allocation of training activities.

This work builds upon Montgomery's (1997) application of Laurillard's model to support the design of CAL tools. The innovative aspect of our approach has been to extend Montgomery's work on University education to support the training activities of a regional Fire Brigade. This has highlighted a number of strengths and weaknesses with existing techniques. An important benefit is that they provide designers with a taxonomy of educational tasks. Rather than focussing on the content of a CAL tool, Laurillard's model focuses on the stages of exposition and reinforcement that support skill acquisition and technical training. A weakness is that Laurillard focuses on the performance of tasks within training environments that are under the control of either a human instructor or a CAL simulation. It does not account for the ways in which "real world" experiences might contradict or reinforce the material presented within a computer package. This is a particularly important omission for the fire service where officers are regularly called upon to apply the knowledge that they gain in training sessions.

Laurillard's model is principally an analytical tool. It can be used to determine whether human instructors and CAL tools support key stages in knowledge acquisition and skill development. It provides limited support for the constructive design of new tools. This argument has been illustrated by the implementation of CAL systems that were intended to support the existing training allocation within the Fire Brigade. Our work focused on the technical training areas that were demonstrated to have the poorest coverage of the stages in Laurillard's model. The initial attempts produced an interactive training tool for a Heavy Rescue Vehicle. This was intended to avoid the pitfalls of conventional lectures by increasing user participation and interaction with the technical material. The principle means of achieving this was through the application of desktopVR techniques. The decision to exploit these techniques was based on feedback from the Fire Brigade rather than the insights gained from theoretical models of function allocation. However, Laurillard's approach did provide a valuable means of assessing the application of this new CAL tool. Her model emphasises the importance of changing educational resources in response to the tutor's observation of student performance. The HRV tool was developed as a stand-alone application. It could not be easily modified by Training Officers.

A second CAL application was developed to avoid the limitations of the HRV package. This tool was specifically developed to provide Fire Brigade instructors with means of tailoring the presentation of training resources. A slide-show metaphor was adopted. This provided a generic interface to a range of multimedia resources that could be linked in by the instructor. Again, Laurillard's model was used as a means of critically evaluating the new task allocation that was created by the introduction of this tool. A particular weakness was that although the CAL resources improved the exposition of technical material, they did little to help fire fighters determine whether or not they had correctly understood this material. This weakness corresponded to at least two stages of Laurillard's model that were not covered after the introduction of the Slide Package. We, therefore, developed a Comprehension Tool that could be used by fire fighters as a self-assessment package.

We were anxious to establish that the application of Laurillard's model resulted in a better function allocation than had existed before the introduction of our new CAL systems. This raised a host of methodological and practical problems. The methodological problems centred around the identification of a suitable evaluation technique that might demonstrate the utility of a function allocation. This is a research area in its own right. The value of any division of tasks is likely to be a highly subjective concept. In our case study, the fire fighters had a different perspective from the Training Officers. The Training Officers had a different perspective from the Senior Officers and so on (Lave and Wenger, 1991). In anticipation of the results of on-going research in this area, we decided to use a more focused empirical approach. This failed to identify any significant differences between groups that had access to our Comprehension Tool and those that did not. The results of this validation exercise were almost certainly affected by the practical problems of conducting such tests within complex organisations. Subjects could not be exactly ranked as the method required nor could we prevent offices from being called away to attend a fire during one of our evaluation sessions.

This paper has presented the practical experiences of using function allocation techniques within a complex organisation. It has demonstrated that Laurillard's model can be extended from University education to the more practical training problems of a Fire Brigade. Above all, however, we have identified the pressing need for appropriate evaluation techniques that might be used to assess the utility of changes in training allocations. During the later stages of this project, we became aware of the complex criteria that were being applied to assess the success or failure of our systems. Some people were interested in the financial benefits that might be derived from allocating training duties to CAL tools. Others saw this as an opportunity to provide Fire Fighters with more varied material and, in particular, to support activitist learning styles. Some individuals wanted to ensure that the Fire Brigade investigated "leading edge" technology. They were less worried about establishing the "effectiveness" of that technology. Finally, there was a concern that the introduction of new technology might distance fire fighters from the experience that human instructors could impart from their own first-hand experiences.

Alan Thompson helped with the design and implementation of some of the systems described in this report. Thanks are also due to the members of Glasgow Interactive Systems Group (GIST) who provided valuable advice and encouragement with this research.

A. Cohen, K. Candland and E. Lee, The Effect of a Teacher-Designed Assessment Tool on an Instructor's Cognitive Activity. In K. Nordby, P.H. Helmersen, D.J. Gilmore and S.A. Arnesen (eds.) Proceedings of Interact'95. 405-409. 1995.

T. Hewitt and M. Tscheligi, Advanced Interaction in University Based Education. . In K. Nordby, P.H. Helmersen, D.J. Gilmore and S.A. Arnesen (eds.) Proceedings of Interact'95. 423-430. 1995

C.W. Johnson, On the Use of Pseudo 3D Images in Human-Computer Interaction. In C.R. Roast, L. Nigay and H. Johnson (eds), Procedings of HCI'98, Springer Verlag, London, 1998.

D. Laurillard, Rethinking University Education: A Framework for the Effective Use of Educational Technology, Routledge, London, 1993.

D. Laurillard and J. Taylor, Designing the Stepping Stones: An evaluation of interactive media in the Classroom. Journal of Education Television, (20)3:169-184, 1994.

J. Lave and E. Wenger, Situated Learning: Legitimate Peripheral Participation. Cambridge University Press, Cambridge, 1991.

M. Montgomery, Developing a Laurillardian Design Method for CAL. In Proceedings of Ed-Media '97. Association for the Advancement of Computing in Education, 1997.

L. Plowman, Narrative, Linearity and Interactivity: Making Sense of Interactive Multimedia. British Journal of Education Technology, (27)2:92-105, 1996.