Evaluation and Multimedia Learning Objects: Towards a Human-Centred Approach
Tom McEwan and Sandra Cairncross
HCI Research Group, Napier University, Edinburgh
0131 455 2793 t.mcewan@napier.ac.uk
Abstract
If initial attempts to incorporate reusable multimedia into university teaching were characterised by the efforts of enthusiasts, the growth of the “Learning Object” economy in recent years requires a more strategic and systems-based approach. While all reusable learning materials have a value, both financial and educational, multimedia in particular is expensive to produce. Systematic production (based on ISO 14915 standards (2002a, 2002b, 2003)) and evaluation of fitness for purpose, are paramount. Human-centred design (HCD) methodologies, based on the ISO 13407 standard (1999), are now well-established for ensuring that investments in technology result in benefits, and in particular lay heavy emphasis on the frequent use of evaluation.
This paper reviews literature in educational multimedia production, interaction design, HCD, and pedagogy, and summarises ongoing attempts to standardise a definition of reusable learning objects (RLOs), with multimedia learning objects (MLOs) as a specific example.  We conclude that evaluation must be central to the production and deployment of MLOs. We report on experiences applying an existing framework for effective production of multimedia learning resources, and propose revisions to this framework to add effective evaluation mechanisms.
Keywords
Multimedia; learning objects; human-centred design; ISO13407; evaluation
1.	Introduction
Patterns of teaching and learning are changing, particularly in the post-school arena. There is a growing need to offer greater flexibility in “product delivery” to students. This is partly because of changes in student availability: demand for part-time study has increased and, increasingly, full-time students are spending less time on campus due to part-time work and other commitments. This has been driven, in the UK, by changes to student funding and a greater emphasis from the government, industry and education itself on continued professional development and life-long learning. In the face of the need to make teaching more efficient, class sizes have increased through rationalisation of subjects, while scheduled contact time has decreased, with students being expected to spend a greater proportion of their time in independent study outside the classroom. There also remains an emphasis on promoting more effective learning resulting from the Dearing report (NCIHE, 1997), which highlighted the need to improve the quality of learning and advocated a more learner-centred approach in which the various needs and circumstances of learners are recognised.
This has led to many universities and colleges adopting a blended approach to supporting teaching and learning, combining taught classes with independent study resources available on-line, and providing support through emails and discussion fora. Increasingly this is provided through the use of managed and virtual learning environments. As well as providing access to material to support self-study, information and communications technology (ICT) is also being used within the classroom. This ranges from displaying simple presentations to more complex animations and simulations. 
Resources are thus needed to support learning both within and outside the classroom. Many educators are looking towards reusable learning objects (RLOs) as one means of meeting this demand. Essentially, RLOs can be thought of as resources that can be re-used in the construction of knowledge by learners in a variety of contexts. For example, a set of PowerPoint™ slides on a key topic can be used by lecturers in different institutions or as a handout for further study. A video or an interactive media package produced to teach the Golden Ratio to graphic designers, might be useful in a Human Computer Interaction course. 
Learning objects are not new, but their provenance is vague and different definitions have emerged in recent years. Neither Laurillard (1993) nor Boyle (1997) use the term as such but relevant ideas, such as combining learning content with learning process, are present. Mayes (1995) introduces Tertiary Courseware – the idea that the trails left by both learners and teachers through educational material are themselves of value. Wiley (2000) credits Hodgins, chair of the IEEE LTSC (Learning Technology Standards Committee) with coining learning objects as a term at a meeting in 1994. L’Allier (1997) supplies a definition of learning objects, in a white-paper for an educational technology vendor. The MANTCHI project (Newman et al 1999), introduced the concept of the ATOM – “Autonomous Teaching Object in MANTCHI” – a reusable, maintainable set of resources for both learners and teachers typically covering ten hours activity in a single topic.   In a working paper for the Dublin Core metadata standard, Guenther (1999) uses the term “multimedia learning objects” (MLOs) as an example of an interactive resource. 
In 2002 the LTSC proposed standards for learning objects (eg IEEE 1484.12.1-2002), while metadata standards continue to evolve for all digital media. This has coincided with educators and learning technologists setting up domain-specific workshops and special interest groups to share reusable materials and approaches.  Additionally, the Advanced Distributed Learning (ADL) initiative, funded by the US Department of Defense, is evolving a standard for RLOs: the Sharable Content Object Reference Model, or SCORM (ADL Technical Team, 2001), based upon the input of many different organisations. Boyle & Cook (2001) and Boyle (2003) summarise recent developments in the field of learning objects and identify ongoing challenges. 
The picture that emerges is of large commercial interests driving a commoditised approach to learning, while educators try to pool resources to allow more time to support individual learners at a personal level. Meanwhile, in our experience at least, individual learners each year increase their expectations for the presentation quality of educational materials they receive. Additionally, students with impairments increasingly have legal backing (e.g. SENDA, 2001) to insist that materials are presented through media channels, and in formats, that meet their needs.
We have grown to see learning resources as more than simply one-off text or graphic-based representations of information. Plainly not every resource will be reused, but we would argue that every resource created by educators, in the course of their teaching, has the potential to be of value to other educators, students or even themselves in future, so we will use the terms, reusable learning objects and resources, interchangeably, while acknowledging that most resources would need additional work to become effective RLOs, not least to conform with emerging standards. 
With advances in network technology and in particular the increasingly availability of affordable broadband connections, multimedia can readily be delivered on-line, providing easy access to animations, video and audio, as well as text and graphics. Interaction can also be supported. This opens up many possibilities to educators, including the provision of the subset of RLOs often referred to as multimedia learning objects (MLOs), which we will define for this paper as “reusable interactive software containing rich media, which learners use in the construction of knowledge about a specific topic”. This neither requires, nor rules out, the mediation of a teacher during the learning process, but does imply a process of curriculum design to ensure that the MLO is relevant to the learners’ needs.
The potential benefits that multimedia can offer teaching and learning are outlined first in this paper, and barriers to realising that potential are identified. The need for taking a human-centred approach to the design and development of MLOs is established and factors which must be considered are explored, in particular the need to consider the context in which the MLO will be used. The centrality of evaluation throughout the lifecycle is highlighted and a design and evaluation framework is proposed to assure this. The paper concludes by exploring practical issues associated with using the framework.
2.	Realising the Potential of MLOs
The use of multimedia in supporting teaching and learning is not new. Cairncross and Mannion (2001) explore how the key elements of multiple media, user control over the delivery of information and interactivity can be used to enhance the teaching and learning process.  Multimedia has the potential to create high quality learning resources. A range of interlinked media elements can be used to convey a given message and the user can study through media channels and at a time and place convenient to them, taking as long or as little as they need. Since it is programmable, interactive applications can engage the user in activities, as well as reacting or responding to selections made by users. It can be used to create an integrated learning environment, combining explanation with illustrative examples, on-line assessment with feedback, thus providing opportunities to practice and experiment. 
Until recently, the introduction of new learning technologies in universities, such as interactive multimedia, has largely occurred on an ad hoc basis. Enthusiasts have pioneered its use in specific modules or subjects. However a more strategic approach is now emerging with the setting up of departmental and institutional units to promote and oversee the integration of new learning technologies into a range of courses or programmes. In the UK, government-funded programmes such as the Learning Technology Dissemination Initiative (LTDI) and Computers in Teaching Initiative (CTI - now the Learning, Teaching and Support Network (LTSN)) have also contributed to the take up of new learning technologies. These initiatives bring a need for a more systematic approach to the design, development and deployment of MLOs.
Research into how best to utilise multimedia elements is now well established and ongoing. Different bodies, including the International Organization for Standardization (ISO) establish guidelines. ISO’s three-part 14915 standard, “Software Ergonomics For Multimedia User Interfaces” is particularly relevant:
•	Part 1: Design principles and framework (ISO, 2002a);
•	Part 2: Multimedia navigation and control (ISO, 2003);
•	Part 3: Media selection and combination (ISO, 2002b).
Although such guidelines are available, and have been debated in draft form for many years, many designers and developers ignore these, choosing to use the latest tool or advance simply because they can, perhaps to impress peers rather than because it benefits the user. This has led to innumerable websites with pointless animations that annoy the users: multimedia has not always lived up to its promise. In the context of educational applications, such a focus on tools rather than users has resulted in designs which are detrimental to learning. Aldrich et al. (1998) report that learning applications often contain interactions that are largely gratuitous, doing little to support effective learning. Laurillard (1995) points out educational benefits will only be realised if the technological pull is counterbalanced by an equally powerful pedagogical pull. Rogers and Scaife (1998) point out many multimedia applications fail to live up the developers’ claims of providing enhanced learning environments. A different approach is needed, one which focuses on the needs of a learner, rather than being driven primarily by technological advances, yet, even when this is taken, Cairncross & Mannion (2000) found that learners did not always make full use of interactive features provided. 
This is not new: there have been recurrent moves over the years towards a more learner-centred approach. However with increased use and interest, it has become vital that such an approach is established and adopted on a wider scale. Laurillard (1993) first laid the foundations with a strong emphasis on focusing on what the student needs rather than the media capabilities of the materials, and provides advice on the design of learning activities. Boyle (1997) starts with the conceptual design of an application, in which decisions on educational approaches precede realisation, through the presentation design, of the multimedia interface. However this approach focuses on the design of the application itself and does not fully address how such interactive multimedia learning applications should be used to best effect, for example integration into the wider curriculum. 
It is not just learners’ needs which should be considered: some educators may feel threatened by MLOs as they can be seen to represent a challenge to more traditional approaches to teaching. Friesen (2003), making a case against learning objects because of engineering approaches in general and the American military-industrial complex in particular, criticises “pedagogical neutrality” (a central philosophy of the SCORM community) as an oxymoron. He suggests that although “object”, as derived from object-oriented approaches, is well-defined, “learning” is not, with no single agreed definition of how learning occurs. Friesen cites Cuban in passing comment on waves of failed (or obstructed) implementation of technology in the classroom, “Seldom did investigators try to adopt a teacher's perspective or appreciate the duality of continuity and change that marked both schools and classrooms” (Cuban, 1986; pp. 2, 6). 
This techno-scepticism should not be ignored, if only to avert the commonplace failure of systems implementers to take into account the needs of all users of the system. Rather, it should be seen as a challenge adequately to define the scope of the system, taking into account the totality of the existing systems. Activity theorists (eg Cole & Engeström 1993) would suggest that examining the praxis (prevailing informal and formal rules), community (ie other stakeholders), and division of labour is fundamental to arriving at a fuller picture. Turner & McEwan (2003) contrast the more holistic focus of Activity Theory (AT) with the limitations of Task Analysis approaches that concentrate on lower-level actions and operations (the latter is, perhaps, a manifestation of the reductionist approaches criticised by Friesen). In AT, a three-layer model is considered, where activities relate to the individual’s motives, actions to goals, and operations to more immediate conditions. Consideration of learning objects at all three levels leads, for example, to the realisation that the usability needs of providers of content are just as important as those of consumers of that content. Therefore a requirement might be that the creators of learning objects must be meaningfully acknowledged or rewarded for the reuse of their work, which was, of course, a central objective of Nelson (1974, 1990), in first developing the concepts of hypermedia. 
If we are to design effective interactive learning experiences then a human-centred approach to their design should be taken. Both pedagogical issues and human-computer interaction principles should be addressed, both in creation of MLOs and in their deployment in educational programmes. The needs of different stakeholders, especially those of learners and their teachers, or lecturers, should be considered and addressed. Learning materials would be improved if designers evaluate their use in real world learning contexts, and thereby gain insights into how real people actually learn from the material (or not), how and why real people create that material, and how that material must be shaped for different audiences. It is not just the product and its use that should be evaluated. The design process itself should also be reflected upon. 
Reflective practice should be integral to the work of a professional (Schön, 1991) and insights gained can be used to inform the design of other MLOs in related domains and, indeed, different areas of the curriculum. Ongoing evaluation can help the classroom teacher or lecturer come to a better understanding of how best to use new learning technologies with their students and could form the basis of practitioner-based research projects (Jarvis, 1999). Done collectively it can promote sharing of best practice. This is essential if learning design teams are to avoid repeating mistakes and to build on successes. There is a need to disseminate findings, both positive and negative, in this area. 
3.	Towards Learner-Centred Design 
If the above concerns are to be addressed, then design for effective learning with interactive multimedia must extend beyond the physical user interface. Norman (1988) points out that for any design to be successful in terms of developing usable and understandable products then that design must be based on the needs and interests of the users and be informed by an understanding of their limitations and capabilities. Dix et al. (2004) stress the need not just to consider the interface itself, but also who will be using the system, what they will be using it for and where they will be using it. The context in which the learning application will be used must also be considered. 
Lecturers and teachers wishing to use interactive multimedia applications to enhance the teaching and learning of their students need advice not only on what constitutes good learning design within an application but also on how to integrate interactive multimedia into the wider curriculum. Equally, developers of multimedia courseware should consider the context in which their applications will be used. They must also produce usable interfaces. User-centred design (also known as human-centred design), as advocated by human-computer interaction practitioners, (e.g. Dix et al., 2004; Preece et al., 1994) and standardised in ISO 13407 (1999) serve as a useful starting point towards meeting the needs of learners and developing effective applications. Carter (2002) urges the usage of both ISO 13407 and ISO 14915 within a framework to create multimedia appropriate for use in engineering education. Eklund & Lowe (2000) have also proposed the use of ISO 13407 - as a quality assurance methodology for technology-delivered education and training.
How, then, do we go about applying these standards to the design and development of MLOs. Moving towards a more human-centred and indeed learner-centred approach involves consideration of both human-computer interaction (HCI) principles and pedagogical issues. The over-riding objective is to produce systems that support users in their tasks in a manner that is safe, effective, efficient and enjoyable (Preece et al. 1994). Designing usable and understandable systems requires knowledge about:
•	who will use the system;
•	what it will be used for;
•	the context and environment in which it will be used;
•	what is technically and logically feasible.
In the context of educational design, this implies consideration of the learners and their teachers and lecturers, what they will be studying and where. Such a user-centred approach is consistent with a systems approach to course development. Ellington et al. (1993), describing their systems approach to course design in higher education, propose that, before designing and implementing a course of instruction, the educational requirements of the learners should be analysed first. Objectives and learning outcomes should be formulated. They also advocate evaluation of the course once it has been implemented as an ongoing refinement process. Ellington et al’s (1993) model locates development in the wider context of use, highlighting the need to start with learning requirements and the need for evaluation once the course has been delivered. 
Therefore, the process of developing learner-centred multimedia courseware can most usefully be based on both human-computer interaction and educational design principles. In the following section we investigate how the inter-relationship might be manifest in the resulting MLOs, which then allows us meaningfully to evaluate them. 
4.	Educational Design – Beyond the Interface
Prior to ISO 14915, a variety of HCI researchers and practitioners developed general guidelines for the design of multimedia interfaces (see e.g. Boyle 1997; Phillips, 1997 for overviews). These guidelines can be broadly separated into two categories: information display and information access. 
Information display is concerned with the selection and presentation of media elements; choosing which media element or elements should be used to convey a given piece of information, and how those elements should be combined and presented on screen.
Information access is concerned with the architecture of an application and how users navigate through the information space as well as playback of dynamic media elements, such as audio and video. Heller and Martin (1995) have developed a media taxonomy that provides a framework for designers to combine different media in a more systematic manner. Blattner and Dannenberg (1992) explore models, metaphors and paradigms for multimedia interfaces, examining composition and combination of media elements. As previously discussed, ISO 14915 for multimedia interfaces, is a useful source of general advice, with sections on media selection and combination (ISO 14915-3, 2002b), multimedia control and navigation (ISO 14915-2, 2003).
Guidelines for information display and access, in terms of general human-computer interaction principles, whilst necessary, are not in themselves sufficient; if we are to obtain “best value” from the investment in courseware creation, then design for effective learning with interactive multimedia must extend beyond the physical user interface. Wenger (1987), Laurillard (1993), and Boyle (1997) all stress the importance, and indeed the necessity, of considering pedagogical issues when developing computer-based educational applications. Nor is it enough to simply define educational objectives and learning outcomes. Effective instruction, whether implemented through traditional teaching methods or through interactive multimedia, depends primarily on an understanding of the learning process itself. Just as we needed to consider a more holistic approach to the development of learning materials in the previous section, we need to consider the totality of the processes by which different learners construct knowledge. Plainly these include factors such as confidence and motivation, which might need us to consider affective issues such as frustration and enjoyment. It’s significant that Vygotsky (1962, 1978) is a common source to the fields of both activity theory and pedagogy.
To exploit interactive multimedia to its full advantage the courseware designer needs to be aware of how learners actually learn and the different types of learning that exist. Attention must also be given to how best to incorporate multimedia components to enhance learning, which means examining the wider curriculum to establish how best to integrate the learning application into that curriculum (Cairncross and Mannion 2001). The context in which the (learning) application will be used must also be considered. This mirrors the increasing awareness of contextual factors in other application domains (Holtzblatt, 1999). Meaningful evaluation of the MLO requires consideration of each of these issues.
Interaction design is more than the design of the physical interface; it is also about supporting people as they work or learn and communicate with others (Preece et. al., 2002). Attention must also be given to social or contextual factors, which also affect whether learners actually use applications and learn from them. When developing multimedia courseware, it should be remembered that learners are not a homogenous group; different learners will approach and use applications in different ways, some more effectively than others. The same learner, depending on their motivation, can take different approaches at different times (Prosser and Trigwell 1999). A design that supports a learner with one particular approach to learning may be less suitable for another learner with a contrasting approach, or, indeed, for the same learner at a later date, or in a different context.
Accessibility must also be considered. Within the UK, recent legislation (SENDA 2001) requires educators to engage more deeply with these issues. Educators may not, even unconsciously, discriminate against a student with one or more impairments. This is a substantial challenge as few educators can be expected to have a detailed understanding of the impact these impairments have upon a learner’s ability to learn in a given context. Taking the example of a learner with a relatively rare impairment, for which the lecturer makes an imperfect effort to accommodate, based on limited knowledge of related, more common impairments. The resulting situation pleases neither party, resulting in a spiral of decreasing motivation. Further, what works for one person, might be detrimental to the needs of others: certain impairments can be handled by reducing the amount of text on screen at once, while others can best be handled by putting as much text on a single screen as is feasible. 
In an ideal environment the interface devices would ensure that no impairment becomes a disability or a handicap, in terms of the learner achieving the learning outcomes. In practical terms the available facilities may be incomplete or limited. By redundancy (presenting the same material in multiple media channels) and by following guidelines for presenting and tagging content, (for example standards such as SCORM (ADL Technical Team, 2001) and SMIL (W3C, 2001)), active discrimination against those with impairments is greatly reduced, although there will always remain residual issues to address. This goes beyond simple interface design, and while metadata standards will help, there are individual contexts to consider. Further work is clearly needed in this area if we are to move towards inclusive design. 
These rules and guidelines, although informing the design and implementation of the MLOs, also act as benchmarks that form the basis of an evaluation framework for MLOs. This framework enables designers to examine adherence to human computer interaction principles, the use of multimedia components and the pedagogical approach taken, and is described in section 6 below. First, however, we must consider challenges with evaluation itself.
5.	Evaluation Concerns
Evaluation of MLOs must amount to more than asking learners if they enjoyed using them, examining whether more students passed, or if lecturers managed to update their materials more frequently, though these are all important. Boyle (2003) suggests that more detailed evaluation should be ongoing. Evaluation should then take place at every stage of the lifecycle and consider as many viewpoints as possible, including teachers and lecturers as well as students. Eklund et al (2003) also urge evaluation throughout the process and cite an example of the results of one evaluation leading to adopting a more blended-learning approach. Bradley & Boyle (2003) also highlight the effectiveness of the blended-learning approach and provide preliminary results of some evaluation - feedback from learners about their experiences. 
The Star lifecycle (Hix & Hartson, 1993) can be helpful here, forming as it does a basis for human-centred approaches to evaluation, which take place throughout the lifecycle. The Star lifecycle is built on a series of steps (analysis, requirements, design, prototyping and implementation), each of which was followed by a relevant evaluation phase. Thus evaluation was placed at the centre of the star, with the other phases connected to it from around the periphery.
Consideration should be given to the total learning experience. It is interesting to note that a web search for the phrase “Total Learner Experience” in January 2004 led to only two hits, while “Total Learning Experience” led to 2,430. This emphasis is revealing; the latter phrase leaves the impression of learning experiences being dictated either by the vendor of the tools or by the provider of the material to be learnt. A concern is that this failure to engage with the learner as a human being, with a mixture of circumstances, motives, knowledge and aspiration, will repeat the failings of earlier educational technologies. The implied notion of learner as passive consumer of educational products is not helpful and goes against growing acceptance of more constructive approaches to learning, as advocated by, for example Biggs (1999).
Oliver and Conole (1998) give an overview of the issues involved in evaluating communications and information technologies and propose a framework to assist practitioners in selection of an appropriate evaluation methodology, data collection methods and analysis techniques. Consideration must also be given as to the extent which some of the traditional concerns of usability apply in the context of learning applications. Gilmore (1995), in the context of problem solving, found that ease of use, implemented through direct manipulation (as opposed to command-based) interfaces, did not lead to faster solution times. Rogers (1999) suggests that we should consider the impact on learning of exploiting computational power for number crunching or when creating graphical representations. Making learning easier may result in less learning taking place. Mayes and Fowler (1999) stress that educational software should encourage the learner to think, and to take an active role in the learning process. For them, a usable educational system should build upon the learner’s existing knowledge and support the learner in carrying out tasks.  However Mayes et al. (1990) point out that, when faced with poorly designed interfaces, valuable learning time is lost in having to spend time learning how to use the application. A balance has to be struck between designing applications that are easy to learn how to use, whilst ensuring that learners can learn from them. 
In the context of MLOs this evaluation should not be limited to the effectiveness of the human-computer interface; other factors must also be considered. Mayes & Fowler (1999) stress that consideration must be given to the prior knowledge of users and the educational setting in which the proposed application will be used, if effective learning applications are to be designed. 
Davies and Brailsford (1994) propose that courseware should be evaluated in terms of:
•	Usability, which measures the effectiveness of an interface and its functions;
•	Learnability, which measures the ease with which a product’s functions are learned and remembered;
•	Educability, which measures a product's educational effectiveness (note that this inverts the dictionary definition - where educability is the measure of a learner’s effectiveness); 
•	Utility, which measures the take-up and usefulness of a product in terms of productivity gains. 
In a similar vein, Milne and Heath (1997) suggest the following 
•	Academic accuracy, whether the material is “correct”;  
•	Usability, the ease at which staff and students can use the software; 
•	Learning effectiveness, the effectiveness with which the courseware supports the process of learning. 
Harvey (1996), in turn, reports that the following factors should be considered:
•	Subject content and the way in which the material is structured;
•	Usability or the level at which a student is able to work through the package without help, in terms of on-line feedback and support; 
•	Pedagogy and the quality of the approach adopted by the package and how it encourages quality in learning through, for example, the use of assessment ;
•	Layout and the stylistic presentation of the material within the package.
Evaluation should encompass usability, pedagogy and fitness for purpose, and this should be done in reference to the context in which the MLO is being used. 
6.	Design and Evaluation Framework for MLOs
The process model in Cairncross & Mannion (1999) proved to be a useful starting point for a design framework for interactive multimedia learning applications. This was based on an iterative waterfall model with pedagogical concerns being embedded at each stage. The model advocated that the needs of the user should be considered at each stage in the process and, where possible, typical users should be involved in the actual process itself. In particular, users should be involved in evaluation of the multimedia courseware. This has been used for different domains. Cairncross and Mannion (1999, 2000) used it to develop an application, on analogue to digital conversion, for computing and engineering students. It was also used by postgraduate students, to develop applications to support the teaching and learning of biological processes, and of constraint satisfaction problems. The usefulness of the model was reflected upon. One postgraduate student commented:
I felt that more was required during the [early] stages from the ‘users’ of the application and in particular the teachers of the subject. ….
In addition to the participation of a teacher at the initial stages and the use of low–tech materials for initial design ideas I found that, in these initial stages, I was also considering past problems with learning the topic and ways to improve matters with the multimedia application. ….  I think that it is important to pay attention to past difficulties in order to learn from them, to decide how best to reach the learning outcomes required.
In taking this approach the postgraduate student was influenced by Scaife et al.’s (1997) informant design framework in which developers work closely with teachers. This suggests that teachers should have a role in identifying areas where multimedia can add value. Learners too should be involved; they will likely offer a different perspective. It was found that, in practice, the boundaries between different stages, between the design and implementation stages, were blurred. It was also felt that a more flexible approach would be helpful, so that a previous stage can be repeated if necessary, without following the framework through to the end before repeating the stage. 
The model was revised in light of these findings and consideration given to the concerns about evaluation outlined above. This resulted in the design and evaluation framework shown below (Figure 1).
The design and evaluation framework proposes an iterative approach to development and recognises that there are different cycles of iteration, between stages and within stages. For example, problems encountered during the development phase may be resolved within that phase or require changes to be made to the original specification.
<figure 1 about here>

The framework locates the application in the setting in which will be used, stating that learning is the focus of the initial requirements gathering and specification phase. This includes not just what is to be learned (content) but the results of that learning (outcomes) and who will be using the application and where (context). Consideration should also be given to identifying areas were multimedia can add value. Design should be informed by learning considerations as well HCI. Issues here include media selection and presentation, access to information and support for learning activities.
A distinction is drawn between functionality testing, during implementation, and usability testing, during evaluation. A detailed evaluation of an application involves evaluating both the usability of the application, i.e. how easy it is to use and its educability (Davies & Brailsford, 1994). An educator must ask whether an application meets the defined learning objectives and ensure that it embodies sound educational principles. Once the first step evaluation is complete, and any modifications made, the application can then be integrated into the rest of the course. 
Evaluation should not end with the introduction of the interactive application: its effectiveness should be monitored as an on-going process. Whilst it is not always possible to simultaneously compare the performance of a group of students using the application with a group using traditional means, a comparison with previous cohorts can be carried out. Alternatively, students’ performance in assessments based on the material in the interactive application can be compared with that in assessments from other parts of the course.
As with the reflective approach to design described earlier, on-going evaluation by the classroom teacher or lecturer can help them come to a better understanding of how best to use new learning technologies with their students and could form the basis of practitioner-based research projects (Jarvis, 1999). 
The framework is a model. User involvement is encouraged throughout the design lifecycle and ideally this should involve learner as well as teachers. Input should be sought during requirements gathering. This can help identify areas where multimedia can add value, and can also help pinpoint problem areas. Users should be asked to comment on early designs and should be involved in the evaluation of the final product before it is released.
Active participation of users throughout the design process is recommended by others, for example Holtbatz’s (1999) contextual design and Scaife et al.’s (1997) informant design. However from our experience this is not always easy to achieve. Even when developers are working in-house it can be difficult to get detailed input due to other commitments on their time. Securing access to learners can also be problematic. They, too, have other demands on their time and volunteers may not always be typical of the wider population. Paying volunteers may encourage wider participation, but the act of payment may influence motivation. Whilst designers should strive to involve users, they should also recognise that it may not always be possible. In such instances they should then try to find out as much about users from other sources and try, where possible, to consider their perspective when developing the application.
7.	Conclusions – Framework in Action
This framework is based on a reflection on the experiences of developing and teaching with MLOs. Both the technical aspects of interface design and the social aspects of integration into the curriculum were underpinned by consideration of HCI principles and pedagogical theory. 
The design and evaluation framework is learner-centred, encompassing both the technical aspects of interface design and the social aspects of integration into the curriculum. As such the framework extends the scope of emerging approaches, which (although prompted by learning considerations) focus on the design of the application (for example, Boyle 1997). The framework also offers advice on how the key elements of interactive multimedia can best be exploited to enhance learning, thereby providing refinement to more general development models (for example, Ellington et al. 1993).
Key elements of the framework include:
•	locating design firmly in the context-of-use;
•	encouraging user participation in the design process;
•	highlighting learning considerations, as well as more general HCI principles.
Research has highlighted the importance of considering the social aspects of integration into the curriculum as well as technical aspects of interface design. Contextual factors can often be the major influence on whether interactive learning applications are effective or are even used (Cairncross and Mannion 2000). The importance of contextual factors is emphasised through highlighting the need for on-going evaluation following release and integration. Feedback so obtained can be used to inform changes to the requirements and design when it is being revised or up-dated. Even if no further versions are to be developed the design team should still seek feedback, reflect on that feedback and the actual process of design. 
Reflective practice should be integral to work of a designer and insights gained can be used to inform the design of other applications. Done collectively it can promote sharing of information. This is essential if design teams are to avoid repeating mistakes and to build on successes. 

ADL Technical Team, 2001. Sharable Content Object Reference Model (SCORM) Version 1.2  Advanced Distributed Learning. Available online at http://www.adlnet.org/ [3 Jan 2004]
Aldrich, F., Rogers, Y., & Scaife, M., 1998. Getting to grips with interactivity: helping teachers assess the educational value of CD-ROMs. British Journal of Educational Technology, 29 (4), 321-332.
Biggs, J., 1999. Teaching for quality at university. Buckingham: Open University Press.
Blattner, M.M., & Dannenberg, R.B., 1992. in M. M. Blattner, R. B. Dannenberg, eds. Multimedia interface design. Reading, MA:  Addison-Wesley.
Boyle, T., 1997. Design for multimedia learning. London: Prentice Hall.
Boyle, T. & Cook, J., 2001. Towards a pedagogically sound basis for learning object portability and re-use. In G. Kennedy, M. Keppell, C. McNaught & T. Petrovic (Eds.), Meeting at the Crossroads. Proceedings of the 18th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education. (pp. 101-109). Melbourne: Biomedical Multimedia Unit, The University of Melbourne. 
Boyle, T., 2003. Design principles for authoring dynamic, reusable learning objects. Australian Journal of Educational Technology, 19(1), 46-58. 
Bradley, C., & Boyle, T., 2003. The development and deployment of multimedia learning objects. Learning Objects Symposium 2003, at ED-MEDIA 2003, June 24, 2003, Honolulu, USA. Available online from: http://www.cs.kuleuven.ac.be/~erikd/PRES/2003/LO2003/Bradley.pdf [3 Jan 2004]
Cairncross, S., & Mannion, M., 2001. Interactive multimedia and learning: realising the benefits Innovations in Education and Teaching International, 38 (2), 56-64.
Cairncross, S., & Mannion, M., 1999. How multimedia functions in engineering education. IEE Engineering Science & Education Journal, 8 (3), 100-106.  
Cairncross, S., & Mannion, M., 2000. Multimedia and learning: patterns of interaction, In: McDonald, S., Waern, Y., and Cockton, G., eds. People and computers XIV – usability or else! Proceedings of HCI 2000. London, UK: Springer-Verlag, 375-388.
Carter, J., 2002. A framework for the development of multimedia systems for use in engineering education. In: Computers and Education 39 111-128.
Cole, M., & Engeström, Y., 1993. A cultural-historical approach to distributed cognition. In G. Salomon (ed.) Distributed Cognitions: Psychological and educational considerations. Cambridge: Cambridge University Press. 
Cuban, L., 1986. Teachers and machines the classroom use of technology since 1920. New York: Teachers College Press.
Davies, P., & Brailsford, T., 1994. Guidelines for multimedia courseware developers in higher education Volume 1: delivery, production and provision. University of Nottingham; UCoSDA.
Dix, A., Finlay, J., Abowd, G., & Beale, R., 2004. Human-computer interaction. 3rd ed. London: Prentice Hall.
Eklund, J., Kay, M., & Lynch, H.M., 2003.  e-learning: emerging issues and key trends. Commissioned paper by the Australian flexible learning framework policy and research program to inform the flexible learning advisory group’s planning for 2004 and beyond. [online] Available from: http://www.flexiblelearning.net.au/research/2003/elearning250903final.doc  [30 Nov 2003]
Eklund, J., & Lowe D., 2000. A quality assurance methodology for the production of technology delivered education and training. In G. Davies & C. Owen (Eds.) Proceedings of WebNet2000 – World Conference on the WWW and Internet. AACE. VA, USA. 162-169.  
Ellington, H., Percival, F., & Race, P., 1993. Handbook of educational technology 3rd edn. London: Kogan Page.
Friesen, N., 2003. Three Objections to Learning Objects. In R. McGreal (Ed.) Online education using learning objects. London: Taylor & Francis Books Ltd.
Gilmore, D., 1995. Interface design: have we got it wrong? In: K. Nordby, D. Gilmore, & S. Arnesen (Eds.) INTERACT'95: Proceedings of the fifth IFIP conference on human-computer interaction, Lillehammer, Norway June 25-29 1995. London: Chapman & Hall, 173-184.
Guenther, R., 1999. List of resource types (Dublin Core working draft). Available online at http://es.dublincore.org/documents/1999/08/05/resource-typelist/index.shtml [3 Jan 2004]
Harvey, J., 1996. Choosing courseware: some guidelines to first step evaluation. In: G. Stoner (Ed.) Implementing learning technology. Edinburgh: Learning Technology Dissemination Initiative, 35-40.
Heller, R.S., & Martin, C.D., 1995. A media taxonomy. IEEE Multimedia, 2 (4), 36-45.
Hix, D., and Hartson, H.R., 1993. Developing user interfaces: ensuring usability through product and process. J. Wiley and sons.
Holtzblatt, K., 1999. Customer centred design as discipline. In: M.A. Sasse & C. Johnson (Eds.) Human-computer interaction INTERACT 99: Proceedings of seventh IFIP conference on human-computer interaction, Edinburgh 30 August – 3 September 1999. Amsterdam, Netherlands: IOS Press, 3-17.
IEEE 1484.12.1, 2002. Standard for learning object metadata. [online] Available from:  http://ltsc.ieee.org/wg12/par1484-12-1.html [30 Nov 2003]
ISO 13407, 1999. Human-centred design processes for interactive systems. International Organization for Standardization.
ISO 14915-1, 2002a. Software ergonomics for multimedia user interfaces -- Part 1: Design principles and framework. International Organization for Standardization.
ISO 14915-2, 2003 Software ergonomics for multimedia user interfaces -- Part 2: Multimedia navigation and control. International Organization for Standardization.
ISO 14915-3, 2002b. Software ergonomics for multimedia user interfaces -- Part 3: Media selection and combination. International Organization for Standardization.
Jarvis, P., 1999. The practitioner-researcher: developing theory from practice. San Francisco CA: Jossey-Bass.
L’Allier J., 1997.  NETg's map to its products, their structures and core beliefs, NETg whitepaper [online] Available from:  http://www.netg.com/Upload/uk_Frame_Reference.pdf [30 Nov 2003]
Laurillard, D., 1993. Rethinking university teaching: a framework for the effective use of educational technology. London: Routledge.
Laurillard, D., 1995. Multimedia and the changing experience of the learner. British Journal of Educational Technology, 26 (3), 79-89.
Mayes, J. T. (1995). Learning technology and groundhog day. In: W. Strang, V. B. Simpson, & D. Slater (Eds.), Hypermedia at Work: Practice and Theory in Higher Education (pp. 28-37). Canterbury: University of Kent. 
Mayes, J.T. and Fowler, C.J.H., 1999. Learning technologies and usability: a framework for understanding courseware. Interacting with Computers, 11, 485-497.
Mayes, J.T., Kibby, M.R., & Anderson, A., 1990. Signposts for conceptual orientation: some requirements for learning from hypertext. In: R. McAleese & A. Green eds. Hypertext: state of the art. Oxford: Intellect, 121-129.
Milne, J., & Heath, S., 1997. Evaluation handbook for successful CAL courseware development. University of Aberdeen: MERTaL.
NCIHE (National Committee of Inquiry into Higher Education), 1997. Report of the national committee of inquiry into higher education: higher education in the learning society. Norwich: HMSO.
Nelson, T., 1974. Computer lib / Dream machines. South Bend IN: Aperture.
Nelson, T., 1990. Literary machines. Sausalito, CA: Mindful Press.
Newman, J., Mayes, T., Draper, S., Benyon, D., Gray, P., Kilgour, A., & McKinnon, L., 1999. Lessons from a multi-university remote tutoring community - MANTCHI. In T. Foster (Ed.) Proc 3rd int conference on the role of universities in future information society (RUFIS99): the virtual university. Northern Arizona University, Flagstaff, AZ, USA. LC5800.R845 1999. pp 103-109.
Norman, D. 1988. The psychology of everyday things. New York, NY: Basic Books.
Oliver, M. & Conole, G., 1998. Evaluating communication & information technologies: a toolkit for practitioners. LATID University of North London, London: BP ELT Report No 3.
Phillips, R., 1997. The developer’s handbook to interactive multimedia: a practical guide for educational applications. London: Kogan Page.
Preece, J., Rogers, Y., & Sharp, H., 2002. Interaction design: beyond human-computer interaction New York, NY: Wiley & Sons.
Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., & Caery, T., 1994. Human-Computer Interaction. Wokingham: Addison Wesley.
Prosser, M., & Trigwell, K., 1999. Understanding learning & teaching: the experience in higher education. Buckingham: Open University Press.
Rogers, Y., & Scaife, M., 1998 How can interactive multimedia facilitate learning? In: J. Lee (Ed.) Intelligence and multimodality in multimedia interfaces: research and applications. Menlo Park, CA: AAAI Press.
Rogers, Y., 1999. Commentary: what is different about interactive graphical representations? Learning and Instruction, 9, 419-425.
Scaife, M., Rogers, Y., Aldrich, F., & Davies, M., 1997. Designing for or designing with? Informant design for interactive learning environments. In: Human factors in computing systems, proceedings of CHI 97, Atlanta, Georgia, USA March 22-27 1997. New York, NY: ACM, 343-350.
Schön, D.A., 1991. The reflective practitioner: how professionals think in action. Aldershot: Ashgate Publishing.
SENDA, 2001. Special education needs and disability act (2001) [online] Available from: http://www.hmso.gov.uk/acts/acts2001/20010010.htm [30 Nov 2003]
Turner, P., & McEwan, T., 2003 Activity theory: another perspective on task analysis. In D. Diaper & N. Stanton (Eds.), The handbook of task analysis for human-computer interaction. New Jersey: Lawrence Erlbaum Associates.
Vygotsky, L.S., 1962. Thought and language. Cambridge, MA: MIT Press.
Vygotsky, L.S., 1978. Mind in society. The development of higher psychological processes. Cambridge MA: Harvard University Press,
Wenger, E., 1987. Artificial intelligence & tutoring systems. Los Altos, CA: Morgan Kaufmann. 
Wiley, D.A., 2000. Connecting learning objects to instructional design theory: a definition, a metaphor, and a taxonomy. In D. A. Wiley (Ed.), The instructional use of learning objects: online version. http://reusability.org/read/chapters/wiley.doc [30 Nov 2003]
W3C, 2001. Synchronised Multimedia Integration Language (SMIL 2.0). Available online at http://www.w3.org/TR/smil20/ [3 Jan 2004]