Course Evaluation
If a primary aim of evaluating instruction is to improve the quality and effectiveness of the teaching and learning involved, there are a variety of procedures that can be adopted to inform the activities of those involved in providing and supporting instruction. In distance education the separation between teachers and students normally involves instruction or learning experiences being mediated through written or recorded materials. These 'published' materials will normally remain in substantially the same form for several (or many) cohorts of learners, although some modifications and amendments can usually be arranged. In turn, this separation entails the absence of direct verbal and non-verbal feedback from students that is so important in all teaching. The evaluation of any distance education teaching materials or course of instruction (whether or not a formal 'course' with clearly defined expectations of outcomes) may seek to provide information that can be used during the process of developing or pr
eparing materials or learning experiences -formative evaluation procedures, or information about how well the 'finished' instruction has worked in normal use -summative evaluation procedures (Scriven, 1967). In practice, it is often impossible to draw such a clear distinction, but it does provide a useful way of considering methods of evaluation.
Formative Evaluation
- Critical Commenting
Whether or not the teaching materials are being prepared by a course team, it shoiuld be possible to arrange for peer review of draft materials. At an informal level this can simply involve one or more colleagues reading, listening to or looking at draft materials and providing comments in terms of the suitability of content (e.g. is it factually correct and supported by examples or evidence, up-to-date, at an appropriate level for the intended students, etc.) and the style of presentation (e.g. does it adopt an engaging style, employ appropriate sequencing and strategies, clearly present information and/or arguments, etc.). The reactions of colleagues can be augmented by adopting the more formal procedure of inviting one or more experts in the field to act as Assessors to comment on the draft materials.
Where arrangements have been made for systematic critical commenting, with teachers or writers reviewing the materials prepared by all the others working on the same course or programme, there is the potential to improve not only individual teaching materials, but also the overall course of instruction. However, it is possible that a 'course team' approach can introduce new problems when writers come to revise their draft materials, as discussed in further detail below.
- Developmental Testing
As the name implies, Developmental Testing takes place during the preparation phase and involves trying out draft materials with students. The feedback obtained is used to guide and inform writers' revisions to the materials before they are committed to print or tape (Nathenson & Henderson, 1980). Such testing may range from a fairly informal student try-out of a single piece of teaching, to an elaborate procedure for testing draft materials for a whole course of instruction.
Students study the draft materials in the usual manner and may be asked to undertake any other requirements, for example submitting assignments, attending tutorial sessions, etc. and possibly sitting an examination upon completion of the course . Their comments on and reactions to the teaching can be collected by means of questionnaires and/or interviews and observations and are fed into the process of revising the course materials for 'final' presentation.
Experience of developmental testing at the Open University (Henderson, et al, 1983) indicates the strength of the procedure for the revision of materials within the overall structure of the course and that these can be of benefit to both course writers and students. It is, however, not particularly suitable for enabling major structural changes to be made to the course. In an attempt to allow for greater flexibility, a number of other procedures have been tried that are part formative and part summative (Henderson, et al, op. cit.). These involve collecting feedback from students and tutors on a short-term 'published' version of course materials, to inform revisions to be made for subsequent presentations.
- Revising Draft Materials in the light of Formative Evaluation
A number of procedures have been discussed for the collection of evaluative comments on the quality and effectiveness of teaching materials during their preparation or development. It should be recognised, however, that evaluation is rarely uncontroversial and that course writers may experience problems when trying to take account of evaluative information and data in the revision of their teaching. For example, it may be difficult to decide which source of information should be given greatest weight (especially if conflicting views have been expressed), comments on later parts of a course or programme may not be as extensive as those on earlier parts, etc. The actual strategies adopted by writers for the revision of draft materials will, in part, be determined by emotional factors, as the process is not simply a technical task Riley, 1984).
Summative Evaluation
No matter how well distance teaching materials have been prepared, it is not until students study and try to learn from them that it is possible to determine how good an educational experience results. It is all too easy to consider the 'product' of course development in distance education to be the materials that are delivered to students by one means or another, whereas it is really the interaction of learners with those materials and other resources,possibly including tutors and fellow students (Thorpe, 1979).Summative evaluation procedures are intended to provide information about a course or materials in use.
- Feedback from Tutors
In many situations, those who have been responsible for preparing distance education materials also provide tutorial support to their students. In these circumstances there exists a direct channel for students' reactions and comments to be made known to the course writer(s) and for any problems with the instructional arrangements to be discovered. However, many distance education tutors may not have been directly involved in the development of teaching materials for the course(s) they tutor. Where such tutors support a course by marking assignments, running tutorial sessions(face-to-face or by means of telecommunications), etc. mechanisms can be implemented to collect, on a systematic basis, evaluative comments from them on a range of issues, e.g. their reactions to the teaching materials, accounts of problems their students have encountered in their studies and assignments, etc. (Ryan, 1982). The experience of tutors in making the course work can provide particularly useful information for subsequent modi
fications to or adaptations of the teaching materials and instructional arrangements.
- Feedback from Students
The collection of feedback from students undertaking a course can provide information that may be of value to (i) students of the course in that or subsequent presentations, (ii) students of a replacement course and (iii) students taking other courses prepared by the same writer(s). It may be possible to implement some revisions during the presentation of a course as a result of students' comments, for example by providing a supplement to update information or to clarify a problem area. More frequently, the student feedback from one presentation of a course helps to determine revisions for subsequent presentations. After a course or programme of instruction has been presented in substantially the same form to many cohorts of students, feedback may be collected to inform decisions about remaking or replacing the course. Information gained by course writers about the success (or otherwise) of approaches and strategies employed in their distance teaching may prove to be of great value when they prepare furth
er courses.
Students may be invited to comment on their experiences of studying a particular course by means of questionnaires and/or interviews. Feedback may be sought on each unit of instruction or part of a course. Questionnaires are probably most widely used for collecting feedback from students and can include several different types of question. At the simplest level a questionnaire may just ask students to indicate what they thought were the 'worst' and the 'best ' aspects of the teaching and to specify any particular problems they encountered. At the other end of the scale, questions may be posed on all aspects of the course, possibly using multiple-choice response categories that can be coded for computer analysis. Questions may seek to obtain information about the following main areas of interest:
Extent of Utilisation. In distance education it is important to know exactly what is being studied and what use is being made of the teaching materials and other resources. Students may be asked to indicate which parts of the course or programme they have studied, which components they have used, how much time they have spent on their studies, etc. They may also be asked to report on any problems they have encountered in obtaining the course materials or in gaining access to resources.
When interpreting the responses to these questions, it must be remembered that students' answers are subjective due to their individual 'orientation' to their studies (Taylor, et al, 1981a). They may all have different reasons for studying a particular course or set of materials, have varying amounts of relevant prior knowledge and experience in the area of study and be undertaking their studies in a variety of financial, geographical, domestic and, possibly, occupational circumstances. Questionnaire responses should not be considered without reference to possible differences in such student characteristics. For example, a student may report having spent considerably less than the notional amount of study time on a particular unit of instruction, due to a high level of existing knowledge about that aspect of the course. Another student of the same unit may report having spent considerably longer than the notional study time, because of great personal interest in the topics and a willingness to explore them
at great length.
Overall View of the Teaching. Students may be asked to rate the teaching of a particular unit of instruction in terms of its interest, perceived relevance or usefulness, level of difficulty, etc. They may also be asked to rate individual components of a course (e.g. teaching text, audio-tape, etc) in terms of their relative usefulness.
Again, when it comes to interpreting the responses, account must be taken of the subjective nature of students' answers. The 'relevance', 'interest', 'difficulty', etc. of any unit of instruction will mean something different to each person studying it and an individual student's rating will largely be determined by his or her orientation. For example, a topic that is considered by one student to be highly relevant (perhaps for vocational reasons) , may be rated at the opposite extreme by another student with different orientation.
The analysis of responses on rating scales requires some caution. Very often 4-, 5- or 6-point rating scales are used (e.g. 1 = Very Interesting, 2 = Fairly Interesting, 3 = Of Little Interest, 4 = Of No Interest) . It is not valid to average the responses from a group of students to the same question (i.e. reporting an 'average rating' of 2.1 for a particular unit of instruction) , because the difference between points 1 and 2 on the scale is not the same as that between points 2 and 3, etc. (They are usually 'Ordinal Scales'). Responses should be reported in a form such as "x% of students found the unit 'Very Interesting' or 'Interesting'".
General Style of Presentation. Course writers may be keen to receive students' comments on the style of presentation, both in terms of layout, design, etc. and the coherence and clarity of the teaching. Perhaps more importantly, students could be asked to comment on the extent to which the teaching style or strategy had enabled them to become actively engaged in learning from the materials.
Here again, it is important to take account of students' orientations to their studies when interpreting responses. For example students with an 'instrumental' or 'extrinsic' orientation may prefer a didactic style of presentation, while other students may not.
Specific Content Issues. It is important to know how well the teaching has achieved its aims and objectives. To this end, information about students' problems with key concepts, ideas and relationships, etc. can be of great value to course writers when it is time for revisions to be made.
Students may be asked to indicate on a multiple-point scale how well they felt particular key points had been taught, or may simply be asked to report any specific problems they had encountered. Simply to know that difficulties exist may not be enough. It may be possible for students to suggest whether problems arise from the inherent difficulty of certain ideas, etc., or from the course writer(s) explication in the learning materials. Any indication of the nature of specific difficulties can be of great value when revisions are made to the teaching.
At the Open University extensive use has been made of questionnaires for the collection of feedback from students during course presentation (Lockwood, 1983; Ballard, 1983). The questionnaires used for courses in their initial presentation contain some items that are course specific and others that allow comparisons to be made between courses for certain aspects of the information, e.g. workload, use made of course components, etc. Depending upon the number of students taking a particular course, questionnaires may be sent to all or to just a sample of students. If a series of questionnaires is to be used for a course or programme of study (for example, one questionnaire for each part of a course), problems of diminishing response rates may be encountered, with fewer students providing feedback on later parts of the course. In such circumstances, it has been found that the decline in responses can be reduced by using 'panels' of respondents, i.e. sending the questionnaires for each part of the course to a
different sub-set of students, so that each individual receives only one form to be completed. An increase in responses to the questionnaires on later parts of the course will normally compensate for any reduction in the total number of respondents for each part.
Student Interviews. Questionnaires can be criticised as a method for collecting student feedback on the grounds that they elicit responses to questions that are of interest to the course writer(s) or evaluator(s) and not necessarily on issues that are considered important by students. The constraints Of questionnaire design can in part be overcome by including space for students to comment in an open-ended way about their reactions to and experience of studying a course. Interviews with students, conducted either face-to-face or by telephone, can provide a richness of response difficult to achieve by use of questionnaires alone. Depending on the type of information sought, a highly structured interview schedule may be utilised or just a short list of key areas to be covered in a relatively free-flowing discussion.
Although interviews may be more time-consuming to conduct and analyse than questionnaires and may elicit responses from a smaller number of students, in-depth interviews are particularly useful for gaining a view of courses from the students' perspective (Taylor, et al , 1983).
- Revising Courses in the light of Summative Evaluation
By a variety of means, a wide range of evaluative comments and information can be collected from students, tutors and, possibly, other interested parties. Each of the various methods described above has its strengths and weaknesses and for any particular course one or more of the procedures may be considered to be most appropriate. For example, a questionnaire survey of students may provide an indication of significant problem areas and lead to some follow-up interviews being undertaken with tutors and students in order to determine the nature of the problems and to elicit suggestions about how they can be overcome. However, before any procedures are initiated, it is important to establish the extent to which it is possible to make changes to the teaching in the light of any evaluation. To a large extent, this means considering what information can be acted upon (and is therefore worth collecting), and when it should be collected.
It may be very difficult to amend teaching materials in their 'published' form due to the physical format (e.g. as printed booklets, audio- or video-tape, etc.) or a lack of resources (human or otherwise) to implement changes. Even if evaluation indicated a major problem with the teaching materials, it would be difficult to do more than change the sequence of presentation, omit a part or make it optional, provide additional study guidance, etc. However, in such cases there is little point in conducting a thorough evaluation of all aspects of the course or programme of study - it would be better to concentrate on aspects that can be acted upon for future presentations, for example tutorial and assessment arrangements.
Timing the collection of evaluative information is similarly important. It is important to collect information and data at a time when it can be of most use. Very often, evaluation studies are undertaken during the initial presentation of a course in order to inform changes for future presentations. However, there may be reasons to believe that the first cohort of students is not typical of the students who are likely to study the course, for example they may be more highly motivated to take that particular course. In those circumstances, a second or subsequent cohort may be considered more appropriate. If changes are to be implemented during the current or in a subsequent presentation of a course, the evaluative information must be available in time to be acted upon. If amendments are to be made to printed or recorded materials, production time is necessary-if tutorial or assessment arrangements are to be revised, adequate notice is required. When deciding how and when to evaluate distance education cours
es, it is necessary to consider what changes are possible given the time and resources available.
|
