Kirkpatrick's Four Levels of Training Evaluation

The topic of training evaluation seems particularly relevant to those who would be educators, whether or not they would work in a corporate or an academic environment. I thought that those of you who have not been exposed to Kirkpatrick's taxonomy for training evaluation might find it interesting and potentially useful in considering the nature of learning and some of the goals of education.

Level 1 = Reactions
- A Level 1 evaluation merely guages a learner's reaction to a program.
- The focus at this level is on questions like, "Did you like it?", "Was it relevant?", "Would you recommend it to your colleague?"
- This is the least expensive, the most common, and also considered the weakest form of training program evaluation.
- According to Kirkpatrick, this is the minimally requisite level of training program evaluation.
- This level of evaluation has consequences for Level 2. Although a positive reaction does not guarantee learning, a negative reaction can significantly inhibit learning. If learners do not like the training, they will not spend time with it, will not advocate it, and may spread bad new about it.
- Issues such as equipment problems, and at a superficial level, instructor effectiveness (e.g, "charm" and aestheticism) can be captured in a Level 1 evaluation.
- If you are a course developer, gathering information like this can be essential in determining whether or not the course was appreciated (and thus, deemed desirable and valuable) by your external (or internal) client.
- Tools: "Happy sheets" or "smile sheets." Questionnaires, interviews, observations and focus groups are used less frequently, but are still quite viable.

Level 2 = Learning
- Evaluation at this level measures whether learning took place, i.e., change in K/S/A's (knowledge, skills, attitudes).
- Most Level 2 evaluations are administered at the end of a course and so have the limitation of not being able to assess retention over time.
- Thus, information about mastery of the content is also limited in a typical level 2 assessment.
- Tools: Pre- and post-tests. Empirical evidence of bona fide learning (vs. mere accessing of prior knowledge) requires both a pre-test and a post-test.

Level 3 = Transfer/Performance
- Level 3 evaluation assesses change in behavior as a result of training. For example, after taking a face-to-face leadership class, did people modify their leadership strategies in accord with the demands of the situation?
- For obvious reasons, level 3 evaluations are more difficult to perform and for some content areas, more difficult to reliably assess (e.g., ethics). Evidence of behavioral or attitudinal change may only be perceived by the learner over time.
- Tools: Observation and interview before and after, over time are required to assess change, relevance of change, and sustainability of change.

Level 4 = Business or Environmental Results

- Did the training program make a measurable difference in the organization's productivity or have an environmental impact? This level of evaluation measures the impact of a training program on business results, e.g., production, quality, sales, accident frequency.

- This is the most expensive and the least likely performed of Kirkpatrick's four levels of evaluation, but it can also be the most persuasive means of arguing the benefits of training to a corporate organization focused on profits.

- Aside from the need to access confidential financial information, due to the difficulty in linking business results directly to training, it is particularly difficult to interpret results in financial terms.

Level 5 (Not Kirkpatrick's, but mentioned by some) = ROI
- Sometimes classified as a type of Kirkpatrick Level 4 evaluation
- Level 5 evaluation assesses the impact of the training on the organization's bottom line--ROI (Return on Investment).
- There are so many factors that are required to accurately assess ROI of training that there are also a variety of methods with variable reliability.
- Tools: Various, e.g., ROI equations

Note: Kirkpatrick also classifies competence into four levels.


References

Kirkpatrick, D. L. (1996). Evaluating training programs:The four levels. San Francisco: Berrett-Koehler. Amazon.com

Kirkpatrick's Four Levels of Evaluation
San Diego State University College of Education
http://coe.sdsu.edu/eet/Articles/k4levels/index.htm

The New Kirkpatrick Four Level Evaluation Certification Program
Corporate University Enterprise, Inc.
http://www.cuenterprise.com/cue_evaluation.php

Symbolic frame.....Debi and Bala

http://garnet.acns.fsu.edu/~bg03/Class_assignment%5b1%5d.mht

April 5 Change Management Group Assignment

This is where each of the four groups should post its response to the class activity assigned at the end of the Change Management class on April 5.

Scenario: You are one of four new faculty members that have just been hired one week ago by a College of Information (COI). Yesterday, the university Provost directed the Dean of the COI to revamp and significantly expand the school’s extant distance learning program to include all of the face-to-face courses now offered in the new undergraduate program. All fourteen faculty (10 continuing faculty and 4 new faculty) will participate in the redesign and implementation. What is more, all fourteen faculty members will participate in teaching the 5-credit hour distance Information Science course required of all new distance IS students. Per the instructions of the Provost, this revised instructional program must be implemented within 30 days, at the beginning of the new semester.

The Provost has become enamored with full-featured Web conferencing systems (Webcons) as a means for alleviating the distancing effect inherent in distance education and as a strategy for positioning the new distance program as one of the most modern. She is committed to technology that will enable remote guest lectures, open question-and-answer sessions, small-group work, and collaborative writing--all of which can be conducted live, with participants able to see and hear each other. The planned full-featured Webcon will include live audio and video. You happen to know that Webcons now on the market are able to smoothly integrate live video and audio to create a higher-order virtual learning environment that approximates its face-to-face counterpart, in addition to permitting synchronous and asynchronous communication and having many of the functions of course management systems.

Typical barriers to widespread Webcon implementation include funding and human resource issues.

Though you are a new Ph.D., your interview, your background in multimedia and knowledge of Internet technologies, in addition to your considerable experience in project management and teaching at-a-distance, both as a T.A. and as an adjunct instructor, make you the most qualified faculty member (new or old) to successfully lead and expedite this project. Rather than starting your new faculty job a few months from now, as you had negotiated, the Dean has asked you to relocate immediately in order to take charge.

Assignment (30 minutes):
Using the frame that your group was assigned, develop a management plan. In doing so, answer the following questions.

1) What factors would you consider?

2) Specifically list as many barriers to this change as you can.

3) Identify relevant strategies that you would use to combat/accommodate each barrier and effectively manage these changes and this project?

Reference

Foreman, J., and Jenkins, R. (2005, April/May). Full–featured Web conferencing systems. Innovate: Journal of Online Education, 1(4).
http://www.innovateonline.info/index.php?view=article&id=45

Change Management Preclass Assignment

Please post your preclass assignment here.

Provide a brief paragraph describing an authentic example of a problem dealing with managing change at the personal or organizatoinal level, especially in your work or academic career. For instance, you might provide a description of a problem you have had in a meeting dealing with change. Ideally, the example you provide will be one in which you have been personally involved, but it can also be a story that you have heard about and know well. Obviously, you will not reveal the names of individuals or institutions. We will discuss the examples in class on April 5.

Thank you in advance for your help in preparing for our class.

Alison