Evaluating Distance Learning Programs and Courses

What is the purpose of the evaluation?

1. Who are the stakeholders that need to know the outcomes of a program or course?
2. What needs to be known? What is the purpose of the evaluative data?
3. Why do these stakeholder need to know?
4. When do the stakeholders need to know-during and/or after the completion of a distance learning program or course?
5. How should the data be presented?
6. Is the evaluation design empirical or anecdotal?
7. How often do the stakeholders need the data?
8. How will the data be used?
   

Student learning outcomes?
Student attitudes before, during, and/or after a program/course?
Physical skills-fine and gross motor coordination?
Course/program administration?
Learning environment?
Method of course delivery?
Appropriate use of the delivery technology(ies)?
Instructor presentations?
Course logistics?
Marketing?
Reliability of the technology?
Availability of the instructor?
Degree of student familiarity with the hardware/software?
Instructor familiarity with the hardware/software?
Relevance of the course content?
Value of team projects?
Effectiveness of the support staff (hotline, etc.)?
Availability of library resources?

1. Formative evaluation is conducted formally and informally throughout a course/program to provide corrective feedback to the stakeholders that need the data. This can be accomplished through scored tests and quizzes, self-tests that are not scored, and one-minute evaluations given at the end of a class. The latter asks one pertinent question such as “ What was the most important thing that you learned in class today?”
2. Summative evaluation takes place at the end of a course or program. These data are used to re-design a course or program. This type of evaluation includes attitudes towards the course/program as well as learning outcomes. In addition, summative evaluation would also include administration of the program/course.
   Sample summative evaluation questions could include open ended constructed response questions such as:

• Identify the strengths and weaknesses of the course/program.
• Would you recommend this course/program to your colleagues or other students?
• What would you do differently?
• What would you add or eliminate?
• How relevant and useful was the content?
• What are some of the benefits that you gained during the course/program?
  

1. Questionnaires
2. Surveys
3. Interviews-structured and open ended
4. Computer data logs
5. Student personal diaries
6. Direct observations
7. Portfolios
8. Tests and exams
9. Self-tests
10. Pre/post test
11. Multiple-choice
12. True/False
13. Matching
14. Essay questions
15. Attitude surveys
16. Scenarios
17. Simulations
18. Games
19. Product assessment criteria

These criteria apply to a variety of delivery technology formats including audio , video, data, and print. Several major course components have been addressed.: Course design, administration, learning outcomes, use of technology, quality of presentations, learning environment, quality of the instructional materials, and selection of teaching strategies.

The criteria can be selected for use by students in the course, peer evaluation, or administrative evaluation. Select those criteria that you feel are most appropriate for your use. Change the wording to meet your needs.

There are a variety of rating scales that can be used for the different criteria. It may be useful to mix several different rating criteria on one instrument.

Rating criteria

1. The course is based on an appropriate systematic learning model.
2. All learning performance objectives (end-of-course and intermediate) are clearly specified and shared with the students.
3. All tests and quizzes are based on the learning performance objectives.
4. The cognitive level of learning (recall, comprehension, application, and critical thinking) are specified for each learning performance objective.
5. The learning performance objectives are sequenced from lowest to highest.
6. A variety of learning domains are used including cognitive, affective, and motor.
7. All teaching strategies are based on the learning performance objectives.
8. A variety of student involvement exercises and activities are used.
9. Cooperative learning is encouraged.
10. Team projects are used.
11. Team projects are evaluated both collectively and individual student contributions.
12. Presentations are limited to 10-12 minutes.
13. Instructor questions are based on the learning performance objectives.
14. A variety of presentation media are used.
15. The learning model is shared and discussed with the students.
16. A variety of delivery technologies were used appropriately.
    

1. Ease of admission
2. Ease of registration
3. Test security is formally maintained.
4. Adequate library materials are used and available
5. Course logistics (delivery and return of homework, instructional materials, delivery and return of special equipment) are adequate.
6. The site coordinator was helpful.
7. The role of the site coordinator was clearly understood.
8. The performance evaluation of the site coordinator was conducted by the instructor.
9. Student recruitment strategies were adequate.
10. The marketing strategies were effective.
11. The specific student audience had been adequately defined.
12. Instructors were formally trained for use of the technology.
13. Instructors were trained how to modify course for delivery at a distance.
14. Adequate secretarial assistance was available for the instructors.
15. Handouts were prepared for the instructors after design.
16. Adequate support staff were available for the instructors.
17. The instructors were regular college/university/institutional staff.
18. Instructors were hired as adjunct part time.
19. Students were allowed to evaluate instructor teaching in a confidential way.
20. For ITV courses, videotapes were available for review.
21. An toll-free line was available for call-ins.
22. Adequate handouts/study guides were used.

1. The course was based primarily on cognitive learning performance objectives.
2. Some affective learning performance objectives were specified.
3. The objectives were provided to the students with an explanation as to how they were used.
4. The end of course objectives contained specific circumstances under which they would be assessed and specific assessment criteria.
5. The learning performance objectives were rated as recall, comprehension, application, and critical thinking.
6. Each of the four levels above were explained to the students with samples provided.
7. The learning performance objectives were logically sequenced.

1. The delivery technology was reliable.
2. The instructor was proficient in the use of the technology.
3. Students were given adequate training in the software delivery system.
4. Students were provided adequate time to familiarize themselves with the delivery software.
5. The software program was reasonably easy to use.
6. A student hotline was provided for technical questions.
7. The student was provided with a minimum platform requirement prior to the course registration.
8. The number of field sites was reasonable.
9. Students were able to communicate with each other outside of the course.
10. There was a reasonable balance between the synchronous and asynchronous classes.

1. The instructor was easily understood.
2. The instructor used good non-verbal communication.
3. Good articulation.
4. Adequate pacing.
5. Handouts were correlated with the TV/computer screen.
6. Questions by the students were encouraged.
7. The instructor asked frequent questions.
8. After a question was asked by the instructor, adequate time (8-10 seconds) was provided to the student to respond.
9. Positive feedback.
10. Adequate student involvement in exercises and activities.
11. Good use of examples (for instance...) and non-examples (don’t confuse A with Z).
12. Effective use of visuals such as graphics, pictures, and video clips.
13. The learning performance objectives were always clear.
14. The instructor dress was appropriate.
15. Good use of movement.
16. Used a variety of instructional strategies.
17. Maximum 10- to 15-minute presentations without student involvement.
18. Responses to the student questions was adequate.
19. The instructor maintained reasonable eye contact between on-site students and students at the field sites.
20. The instructor was enthusiastic throughout the course.

1. The reception area was comfortable.
2. Lighting was adequate.
3. The reception area(s) was easily assessable.
4. The environment was appealing.
5. The primary learning environment was:
   • home
   • classroom on a campus
   • library on campus
   • library off-campus
   • public school
   • office
   • other
6. Students worked primarily
   • alone
   • with one other student
   • with a team of 4-9 students
   • combination
7. There was adequate parking.
8. Security was available.

1. Teaching strategies were matched to the learning performance objectives.
2. The rationale for the use of a teaching strategy was explained to the students.
3. There was appropriate on-site participation.
4. A variety of teaching strategies were utilized.
5. Overall the teaching strategies were interesting and useful.
6. Different teaching strategies included

• case studies in writing
• case studies on videotape
• role playing
• scenarios
• games
• simulations
• other

1. The class size was appropriate.
2. There was a reasonable balance between real and delayed time classes.
3. The course was totally asynchronous.
4. The instructor used
   • listservs
   • bulletin boards
   • chat rooms
   • audio conferencing
   • e-mail
   • voice-mail
5. Student e-mail messages were answered within 24 hours.
6. Navigation through the course was easy.
7. Students were able to communicate with each other.
8. Students received adequate feedback on assignments and projects.
9. Students worked primarily in teams.
10. The navigation icons were consistent through all of the web pages.
11. Graphics were effective.
12. Graphics and pictures were easily downloaded.
13. All computer conferencing dialogue was available to all students at any time.
14. The instructor provided some useful URLs.
15. There was good discussion among teams during project work.
16. The course syllabus was clear and directive.
17. There was adequate real-time interaction with the instructor.
18. Students were taught how to identify, access, and evaluate URLs.
19. WWW contextual assumptions

1. The student has access to the WWW on demand.
2. The student will explore web resources independently or as guided learning with specific guidance from the instructor.
3. Web-based instruction functions best in a constructionist environment.
4. The instructor is not the sole source of information and therefore becomes a guide on the side-a facilitator rather than disseminator of information.
5. The amount of time that a student has to learn something is variable unless for some reason it is specified.
6. Students must accept responsibility for their own learning.
7. Learning does not take place in a fixed location. It takes place at home, at work, in a library, as well as in a classroom.
8. The primary content resource shifts from a single text and teacher to a variety of information resources, multimedia as well as print, available worldwide.
9. Content is presented using a hypertext format with links to further levels of detail and elaboration that is under the control of the student.
10. Student identified resources (URLs) must be evaluated as to validity and reliability.