Click here to skip navigation
An official website of the United States Government - whitehouse.gov

Sign In

Training and Development Policy Wiki

Page History: Training Program Evaluation


Compare Page Revisions




Page Revision: 10/11/2016 3:23:42 PM


Training Program Evaluation

EVENT ANNOUNCEMENT: 

OPM's FED Evalu-Con 2016 was held on Monday, September 26th, 2016!

Federal Employee Development Evaluation Conference (FED Evalu-Con) was hosted by the U.S. Office of Personnel Management (OPM), and showcased learning sessions on effective and innovative program evaluation practices and tools.  This event provided attendees with an opportunity to increase program evaluation knowledge and capacity. The event also facilitated networking between evaluators of training and other types of evaluation programs, making the Federal evaluator community more cohesive and collaborative.

Session slide presentations and materials are available for download by clicking the link next to the presenter's name and session title.

Event Agenda

7:30AM – 8:30AM

Check-in & Pick-up Conference Materials

Atrium

8:30AM – 8:45AM

Welcome & Opening Remarks
Cheryl Abram, U.S. Office of Personnel Management Introduction Evalu Con OPM.pptx

General Session
(1459/1460/1461)

8:45AM – 9:00AM

Karlease Kelly, U.S. Department of Agriculture (No slides)

General Session
(1459/1460/1461)

9:00AM – 9:15AM

Tammy Tippie, Performance Improvement Council (No slides)

General Session
(1459/1460/1461)

9:15AM – 10:00AM

Government Budget for Program Evaluation
Diana Epstein, Office of Management and Budget 

General Session
(1459/1460/1461)

10:00AM – 10:15AM

Break

 

10:15AM – 11:30AM

Leadership Panel: Making Evidence Based Decisions (No slides)
Kirstin Austin (Moderator), Department of Homeland Security;
Joellen Jarrett, Small Business Administration;
John Levock, Office of Management and Budget;
Reese Madsen, U.S. Department of Defense;
Cheryl Seminara, Federal Emergency Management Agency

General Session
(1459/1460/1461)

11:30AM – 12:30PM

Lunch

(On your own)

12:30PM – 1:30PM

Logic Models (Planning)
Yvonne Watson, U.S. Environmental Protection Agency (6a) Logic Models (Yvonne Watson).pptx

Room A (1459)

Development and Use of Indicators for Program Evaluation (Implementation)
Goldie MacDonald, Center for Disease Control and Prevention 

Room B (1460)

Making Meaningful Recommendations (Reporting)
Carimanda Baynard, U.S. Department of State (6c) Making Meaningful Recommendations (Carimanda Baynard) Updated.pptx

Room C (1461)

1:30PM – 2:30PM

Developing Evaluation Plans (Planning)
Jone Papinchock, Federal Deposit Insurance Corporation (7a) Developing Evaluation Plans (Jone Papinchock).ppt

Room A (1459)

Outcome Harvesting (Implementation)
Travis Mayo, U.S. Agency for International Development (7b) Outcome Harvest (Travis Mayo).pptxHow would YOU use an Outcome Harvest.docx

Room B (1460)

Data Visualization (Reporting)
Kendra Kintzi, U.S. Trade and Development Agency Fed EvaluCon_USTDA Data Visualization (Kendra Kintzi).pdfCopy of Mock Dashboard Data.xlsxPower Africa Infographic.pdf

Room C (1461)

2:30PM – 2:45PM

Break

 

2:45PM – 3:45PM

Obtaining Vendor Support (Planning)
Rebekah Sobel, United States Holocaust Memorial Museum

Room A (1459)

Writing the “Right” Evaluation Questions (Implementation)
Michael Coplen, U.S. Department of Transportation OPM Eval-Conf-Getting-Eval-Qs-Right-FINAL2b.pptx

Room B (1460)

Data Collection (Reporting)
Dean Olsen, U.S. Department of State (8c) Data Collection (Dean Olsen).pptx

Room C (1461)

3:45PM – 4:00PM

Closing Remarks
Cheryl Abram, U.S. Office of Personnel Management

Room A (1459)


Speaker bios will be posted shortly and be sure to check back here after the conference for copies of the presentations that were delivered.


Participants, please click here for detailed logistics information including driving or metro directions and contact information. 


 What Is Training Program Evaluation?

Training program evaluation is a continual and systematic process of assessing the value or potential value of a training program.  Results of the evaluation are used to guide decision-making around various components of the training (e.g. instructional design, delivery, results) and its overall continuation, modification, or elimination. 

To assist agencies in evaluating their training programs, OPM published the 2011 OPM Training Evaluation Field Guide. The Training Evaluation Field Guide is designed to assist agency training representatives (i.e., training managers and supervisors, training liaisons/coordinators, agency evaluators, instructional designers, training facilitators and others who have a significant role in training effectiveness) in evaluating training effectiveness and in demonstrating training value to stakeholders and decision makers. Data and information were gathered from fifteen federal agency representatives who volunteered their time to attend a one-day working meeting, participate in individual interviews and submit samples of their tools and case studies. This Field Guide reflects the input from the working group.  The OPM Training Evaluation Field Guide Supplement is an abbreviated version of the OPM Training Evaluation Field Guide.

SEMINAR MATERIALS for OPM Seminar on Evaluating Leadership Development Programs—Easing into Levels 3 & 4

View this quick video for more information on the training evaluation field guide and ways agencies can use evaluation and training data to inform decisions related to training investments.


Logic Models

Logic models are very easy tools to help you plan and evaluate your training program. A logic model provides a representation of a "theory of change" (if...then) that clearly aligns the training inputs and activities to the outputs and results.  Logic models are created in the planning phase and can be completed for programs, initiatives, individual courses, events and activities. There is an important distinction between training program evaluation and training evaluation. Training program evaluation examines the entire program including the training component, while training evaluation is assessment of the training course, process, or component within a program. For example, the Senior Executive Service Candidate Development Program (SESCDP) involves a recruitment, selection and certification process. These processes, in addition to the required 80 hours of training and 4-month developmental assignment, should also be consistently monitored and evaluated to determine progress and areas of improvement.  All processes within the program contribute to program success or failure, yet all processes are not "training". Knowledge, understanding and skill in program evaluation in necessary to effectively evaluate training and development programs. 

Contact us at SESDevelopment@opm.gov if you would like training on how to use the OPM Training Evaluation Field Guide and/or Logic Models in your agency.

For more detailed information on logic models, you can watch this video or take EPA's 
FREE Logic Modeling Course.


Regulations

Agencies are required to evaluate their training programs annually to determine how well such plans and programs contribute to mission accomplishment and meet organizational performance goals (5 CFR 410.202).  The law authorizes OPM to require Federal agencies to report training data (Example: Sample Agency Training Report FY2010).  An important part of the evaluation process involves consideration of training costs, and other elements not directly addressed in the typical evaluation.  Agencies must track and report accurate training data on all completed training events to OPM as prescribed by the Final Rule on Training Reporting Requirements, published on May 17, 2006, in the Federal Register, The Federal Workforce Flexibility Act of 2004 (P.L. 108-411).

Agencies can reference the Guide for Collection and Management of Training Information for detailed information on reporting training data.


 Other Evaluation Methods

The Training Evaluation Field Guide uses the Kirkpatrick Model of evaluation as the basis of instruction.  However, there are other viable, well researched and more appropriate models available to evaluate your training program, not just the training component of your program. Agencies should use the model that more closely meets their needs.

  • Daniel Stufflebeam's CIPP Model (Context, Input, Process, Product)The CIPP Model for evaluation is a comprehensive framework for guiding formative and summative evaluations of programs, projects, personnel, products, institutions, and systems. This model was introduced by Daniel Stufflebeam in 1966 to guide mandated evaluations of U.S. federally funded projects.http://srmo.sagepub.com/view/encyclopedia-of-evaluation/n82.xml
  • Robert Stake's Responsive Evaluation Model: Robert Stake (1975) coined the term responsive evaluation. Responsive evaluation distinguishes four generations in the historical development of evaluation: measurement, description, judgment and negotiation. ‘Measurement’ includes the collection of quantitative data.http://mailer.fsu.edu/~sullivan/SEA_Newsletter/Responsive_Evaluation.pdf
  • Robert Stake's Countenance Model: This model focuses on description and judgment. Stake wrote that greater emphasis should be placed on description, and that judgment was actually the collection of data. Stake wrote about connections in education between antecedents, transactions and contingencies (outcomes). He also noted connections between intentions and observations, which he called congruence. Stake developed matrices for the notation of data for the evaluation. Data is collected through these matrices.http://ged550.wikispaces.com/Robert+Stake's+Countenance+Model
  • Kaufman's Five Levels of Evaluation: Modeled after University of Wisconsin professor Donald Kirkpatrick's four level evaluation method, Roger Kaufman's theory applies five levels. It is designed to evaluate a program from the trainee's perspective and assess the possible impacts implementing a new training program may have on the client and society.
    http://www.ehow.com/info_8582553_kaufmans-five-levels-evaluation.html#ixzz2ePyjynYl 
  • PERT (Program Evaluation and Review Technique:The Program (or Project) Evaluation and Review Technique, commonly abbreviated PERT, is a statistical tool, used in project management that is designed to analyze and represent the tasks involved in completing a givenproject. First developed by the U.S. Navy in the 1950s, it is commonly used in conjunction with the critical path method (CPM).http://en.wikipedia.org/wiki/Program_Evaluation_and_Review_Technique
  • Michael Scriven's Goal-Free Evaluation Approach: This approach is premised on the assumption that an evaluation should establish the value of a program by examining what it is doing rather than what it is trying to do.http://www.click4it.org/index.php/Goal-Free_Evaluation
  • Illuminative Evaluation Model: An illuminative evaluation is a custom built research strategy which lacks in formal statements of objectives, avoids (but does not exclude) statistical procedures, employs subjective methods, and is primarily interested in the informing function of an evaluation, rather than the more usual inspectoral or grading functions of an evaluation.http://www.iisd.org/casl/CASLGuide/EvalModel.htm


Other Helpful Resources

Logic Model Template for Supervisory & Managerial Training:   This logic model provides agencies with a systematic and visual aid to identify a training program’s components: inputs, outputs, outcomes and evaluation methodologies, and illustrates the relationship among them.  The logic model has been prepopulated with Federal supervisory and managerial curriculum-specific examples for agencies to reference when creating their agency-specific logic model.  Once completed, the logic model will provide stakeholders with a roadmap that describes the sequence of related events.  Stakeholders will also have a clear understanding of the curriculum goals and evaluation. 

Note:  The Federal Managerial Curriculum Logic Model components have been color-coded to map to each of the stages included in the OPM Training Field Guide Supplement. 

Certifications: Both Kirkpatrick and Phillips offer training evaluation certificates although a certified evaluator is not necessary to evaluate the effectiveness of agency training. The 2011 OPM Training Evaluation Field Guide, books and/or training courses on performance measurement, program evaluation, and training evaluation should provide enough information to successfully evaluate your agency training.

Associations: The American Evaluation Association (AEA) is an international professional association of evaluators devoted to the application of many other forms of evaluation. Evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness. The AEA has approximately 5500 members representing all 50 states in the US as well as over 60 foreign countries.


FAQs

WHAT should you evaluate?

Training program evaluations can help the organization to reach many different goals during the life cycle of a training program.  One primary reason to evaluate is to determine if the benefits derived from the training justified the costs.

Some additional reasons include:

  • Examining the assumptions upon which an existing or proposed training course or program is based
  • Inquiring, up front, about the expected results
  • Assessing how much of the knowledge and skills learned during training transferred to on-the-job behaviors 
  • Collecting information about inputs, activities and outcomes.
  • Comparing it to some pre-set standards or targets.
  • Determining whether the results of the training contributed to the achievement of the organization’s goals
  • Reporting findings in a manner that facilitates their use and improves program effectiveness 

WHY should you evaluate?

Aside from the requirement in 5 CFR 410.202, agencies face very real demands to demonstrate training program efficiency, program effectiveness and public accountability. Use of evaluation data meets these demands in various ways:

Planning: To assess needs, set priorities, direct allocation of resources, and guide policy

Analysis of Course/Program Effectiveness or Quality: To determine achievement of objective, identify strengths and weaknesses of a program/course, determine the cost-effectiveness of a program/course, and assess causes of success or failure

Direct Decision-making: To improve effectiveness, identify and facilitate needed change, and continue, expand, or terminate a program/course

Maintain Accountability: To stakeholders, funding sources, and the general public

WHEN should you evaluate?

There are several basic questions to ask when deciding when to carry out an evaluation. If the answers to these questions are "Yes", this may be the time to evaluate.

  • Is the program/course important or significant enough to warrant evaluation?
  • Is there a legal requirement to carry out a program evaluation?
  • Will the results of the evaluation influence decision-making about the program/course?
  • Will the evaluation answer questions posed by your stakeholders or those interested in the evaluation?

Control Panel

Unexpected Error

There was an unexpected error when performing your action.

Your error has been logged and the appropriate people notified. You may close this message and try your command again, perhaps after refreshing the page. If you continue to experience issues, please notify the site administrator.

Working...