The Role Evaluation Has to Play:
Developmental Testing

Context:
This article, the second of three on the role of evaluation in course design, explains the part evaluation plays during development of the course.

Source:
Melton, R. F. 1997. "The Role Evaluation Has to Play." From R. F. Melton, Objectives, Competencies and Learning Outcomes: Developing Instructional Materials in Open and Distance Learning, pp. 109–112. London: Kogan Page.

Copyright:
Reproduced with permission. Non-exclusive World English language rights. ISBN 0749421738. Kogan Page Limited, Tel: 0171-278-0433, Fax: 0171-837-6348, website: www.kogan-page.co.uk.

Developmental testing

Once detailed materials have been developed, students-in addition to specialists-can be asked to study the materials and to provide feedback through the process of developmental testing. Within this process students are asked to study the materials under conditions that are as near as possible to those ultimately to be used in the presentation of the course, and are then asked to provide feedback on the strengths and weaknesses of the materials. A whole course might be subjected to developmental testing, and there are examples of this in my own institution where courses have been subjected to such testing during their first year of presentation and then remade during the subsequent year. However, this can be a time-consuming and expensive process.

An alternative approach is to accelerate the development of one or two units within the development process, and to subject these to developmental testing. Where such an approach is adopted it is logical to focus evaluation on strategies that are to be used in subsequent units rather than on details that are only relevant to the units being evaluated (see Figure 9.1). The logic behind such an approach is to highlight ways in which strategies might need to be modified before their use is extended to other units. For such an approach to be effective, careful consideration must be given to the scheduling of developmental testing within the whole process of course development.

Q1 Use of Media

In answering the questions below please reflect on the way in which you havemade use of the media over the full period of developmental testing

(a) Which of the materials listed below have you used to good effect in studying the prototype?
(Ring all that apply in column 1 of the table below)

(b) Which of the materials, if any, would you describe as being of little help?
(Ring all that apply in column 2 of the table)

(c) Do you wish to comment further on any of the above materials?
(If yes, ring the code, and comment below)        1

Figure 9.1 A question focusing on strategies adopted within a language course

The number of students taking part in the developmental testing of materials may vary from six to 60 or more, depending on the type of testing to be undertaken. Within the process described below I will assume that around 60 students have been asked to study the materials and to provide related feedback.

Once students have completed their study of the units to be tested, it makes sense to interview a limited number to obtain their reactions to the materials and the process they have been through. Interviewing six students on an individual basis can provide a great deal of information, while interviewing 12 students-six with above average entry qualifications and six with below average entry qualifications-can provide insights from two different perspectives. The interviews should be open-ended, encouraging students to freely express their opinions rather than constraining them to responding to points which we as interviewers might have preconceived to be important.

Whether the views expressed in the interviews are typical of the students involved in the developmental testing remains to be seen, and follow-up questionnaires will be needed to determine the extent to which this Is the case. The minimum number of students responding to questionnaires needs to be around 30 if reasonably reliable data are to be obtained. However, if separate data are required on students in the upper and lower groups then the number of students responding to questionnaires needs to be about 60. Every attempt should be made to minimize the time lapse between the interviews and the issuing of follow-up questionnaires.

Bearing in mind that the time allocated to developmental testing may be very limited, it is worth noting that, even before the results of the interviews are available, evaluators will already be aware of some issues they will want to monitor, and related tests and questionnaires may actually be prepared for this purpose prior to developmental testing getting under way. For example, within a behaviourist approach it may be taken for granted that evaluators will want to determine the extent to which stated objectives have been achieved, and they will need to develop tests for this purpose. They will also want to identify the extent to which related study problems make it difficult for students to achieve particular objectives, and questionnaires may also be designed in advance for this purpose. Such tests and questionnaires may be presented to students at about the same time as the interviews take place, ensuring that important data can be collected and analysed without delay.

The opportunity should also be taken to interview others involved in the presentation and administration of the materials, such as teachers and administrators, thus gaining insights from a variety of different perspectives. The advantage of such an approach is that each perspective contributes to a fuller understanding of the issues, and at the same time highlights obvious limitations in any one perspective.