Appendix A-7: Office of Executive and Management Development (OEMD)

FY 2000 Performance Report
Goal & Performance Indicators Checklist

Forward | Back | Table of Contents | OPM Home Page

OPM STRATEGIC PLAN GOAL IV

DELIVER HIGH QUALITY, COST-EFFECTIVE HUMAN RESOURCES SERVICES TO FEDERAL AGENCIES, EMPLOYEES, ANNUITANTS, AND THE PUBLIC

OEMD Goal 1 -- M

By the end of FY 2000, OPM executive and management development programs and services are high-quality, timely, and customer-oriented and are relevant to individual and organizational performance improvement.

This program goal was established under Strategic Plan Goal IV to improve the performance of Government by developing the Government’s key executives and managers – a key human resource. To accomplish this, we design, develop, deliver and evaluate high quality executive and management leadership improvement interventions to federal agencies so that subject organizations can receive the developmental experiences necessary for them to maximize their efficiency and effectiveness. There are seven indicators to assess whether or not this goal is achieved. Of these, the third indicator was identified at the beginning of the year as most critical for achieving the goal.

We met this goal by: (1) Conducting two curriculum reviews and re-validating the curriculum against the Executive Core Qualifications (ECQs) and Competencies, (2) Completing the design of a new Leadership for Results program (Action Learning model), (3) Including a larger number of speakers from showcase Government organizations; examples of which were presentations from senior executives of the Postal Service, NASA and the FDA’s Food and Nutrition Service. (4) Implementing, delivering and assessing five diversity programs (180 participants), and (5) In recognition of the increasing value of technology as a leveraging factor in delivering value to its customers, implementing several initiatives which assisted us in giving increased value to our customers.

M Assess overall program quality and effectiveness through the use and analysis of end-of-course evaluations for training and development programs. Additional measures address how well courses improve participant’s individual and organizational effectiveness back in the agency.

We met this indicator by adopting the Kirkpatrick Model of Evaluation. The Kirkpatrick Model of Evaluation is the most universally accepted model for evaluating Human Resources and training programs because it allows for evaluations at the results level. Specially, the four levels of the model are:

Level 1: Reaction –
Were the participants pleased with the program?
Level 2: Learning –
What did the participants learn in the program?
Level 3: Behavior –
Did the participants change their behavior based on what was learned?
Level 4: Results –
Did the change in behavior positively affect the organization?

OEMD measures all programming at Level 1 of the Kirkpatrick’s Model using end-of-course evaluations (Five-point Likert Scale). In FY 2000, OEMD surpassed its estimate for participant ratings of core programs in program content, relevance and facilities. Our estimate was 4.5, but actual ratings averaged 4.6. Participant ratings for non-core programs also exceeded the standard of 4.4 with an average of 4.6. Additional measures at Levels 2-4 of Kirkpatrick’s Model addressed how well courses improved individual and organizational effectiveness back in the agency. These are described in indicators 2 and 5. OEMD conducted a total of five Level 3 and Level 4 Kirkpatrick Model evaluations in partnership with its customers. FEI’s Leadership for a Democratic society (LDS) evaluated two programs (Programs 254 and 257) at Levels 3-4. FEI’s Center for Executive Leadership (CEL) also conducted two Level 3-4 evaluations with the Bureau of the Census and the Treasury Department. Results of these studies showed desired changes in participant behavior as measured by pre- and post test scores. The Western Management Development Center (WMDC) completed an in-depth evaluation of financial return on investment (Level 3 and 4) in partnership with the Department of the Army. The direct return on investment of this training was calculated to be in excess of $478,000 in documented savings compared to a training expense of $100,000. A project plan for a Level 3 evaluation was completed by Eastern Management Development Center (EMDC) and will be implemented in FY 2001.

M In FY 2000, expand pilot program efforts to measure the amount of learning that takes place in courses and seminars (Level 2, Kirkpatrick Model of Evaluation). Data is based on participants’ self-rating of their knowledge of the subject matter covered in the target seminar before and after completion of the program.

In FY 2000, OEMD evaluated selected programs at Level 2 of the Kirkpatrick Model of Evaluation – what was learned in the program. Representative of the Level 2 results is Program 254, LDS. When 54 program participants were asked to rate the achievement of their personal learning goals (which they had identified at the beginning of the program), 69% responded that they had either fully or mostly accomplished their learning goals. Less than 3% either dropped the goal and had nothing to report or felt they had not learned anything. Additionally, at the end of every program, LDS program participants are asked to evaluate their achievement on FEI’s twelve learning outcomes. Five selected measures from Program 254 are as follows:

In FY 2000, the MDC’s collected data (self-reported) from participants to measure knowledge gained from seminars. Data were collected from over 1,000 participants in 46 WMDC and 3 EMDC seminars. Analysis of referenced data showed that the “average knowledge” gained by seminar participants as a result of their seminar participation was approximately 30%. Participating managers indicated their “before seminar knowledge” was average (2.8 on a 5-point scale), and their “after seminar knowledge” was very good to outstanding (4.3 on a 5-point scale). Similar data will also be collected in FY 2001 and will continue to provide an outstanding basis for evaluating seminar design and effectiveness.

M Ensure executive and managerial training and development programs and curriculum reflect current trends in Government, leadership competencies, and changes in policy direction. Use ongoing assessment of results for continuous review and update. [Critical indicator]

In FY 2000 OEMD conducted two curriculum reviews and re-validated its curriculum against the Executive Core Qualifications (ECQs) and Competencies. The ECQs and Competencies have been defined and validated by the Office of Executive Resources Management as necessary for success at the SES level. (See Executive Resources, Salaries and Expenses section of the Annual Plan). OEMD used the ECQs and competencies to develop a “ Leadership Journey” that integrates core program offerings at FEI and the MDCs and provides leaders with a development path from the supervisory level to SES certification. This “Journey” is a key part of our communication and marketing initiatives.

In FY 2000, FEI completed design of a new Leadership for Results program (Action Learning model), which uses state-of-the-art learning approaches to enable participants to work on significant agency issues as the vehicle for their training. This pilot test of the program began in August, 2000, with the Department of the Treasury and ends January, 2001. The program builds Return on Investment (ROI) into the design by having participants work in teams on significant agency business issues.

In FY 2000, OEMD established a Succession Planning Service with three Government groups (NOAA, DeCA and the CFO Fellows Program) to assist them in aligning their human resources and strategic goals. Services offered range from succession planning program development to assessment, coaching and mentoring training and services, to focused skill development.

In FY 2000, the MDCs improved the FY 1999 pilot and implemented, delivered and assessed five diversity programs (180 participants) that earned overall program evaluations of 4.4 on a 5.0 scale. The curriculum design was based on extensive review of policy initiatives, Government/private industry best practices and expert review of program design. The objective of the workshops was to communicate the business imperative of a diverse workforce as defined in the Report to the President’s Management Council (PMC) on Hispanic Employment in the Federal Government (submitted by the PMC Interagency Work Group).

OEMD, recognizing the increasing use of technology as a leveraging factor in delivering value to its customers, implemented several initiatives:

CEL partnered with the Defense Intelligence Agency/JIVA to provide five weeklong “Foundations for Virtual Teams” courses at the FEI. Each cohort represented intelligence analysts from across Government who are learning how to integrate leading edge virtual communication technology with virtual teamwork into the collaborative communities they are mandated to create.

A session introducing the foundations for virtual teams was offered by CEL as part of a larger team building for Defense Criminal Investigation Service (DCIS) in August 2000.

A weeklong program in working in virtual teams was custom designed and offered to DeCA in August 2000 by CEL.

A session introducing the collaborative process as the underlying foundation for building a knowledge management system was developed for the State Department OPAP Project by CEL.

In FY 2000, OEMD developed a plan to expand its Website capabilities to allow program participants to register electronically for programs, prepare pre-program materials, and complete course evaluations.

M The Federal Executive Institute’s on-going multi-year evaluation project is designed specifically to measure program quality and effectiveness in 10-12 different areas. Some of these areas regularly include: pre-program sponsor forms and participant objectives’ surveys which indicate expectations in advance of the program, on-site evaluations of every element of the program, and post-program feedback from alumni and agencies. Major initiatives in FY 2000 will include: 1) a continuing survey of sponsors and key agency officials to determine the effectiveness of the Institute’s programs; 2) revision of pre-program sponsor’s form to link objectives to action-planning and post-program follow-up with agency sponsors; 3) use of computer based simulations to allow high-level skills “practice” to measure improvement; and 40 post-program case studies of cross-functional groups that have been part of multi-phase training at the Institute (e.g., Regulatory agencies working on similar issues and measuring improvement over time).

In FY 2000, a different approach was used as a replacement for the multi-year evaluation project methodology. Although the multi-year approach outlined above was felt to be appropriate when begun in the mid-90’s, we have changed the thrust of the approach (as reflected in the FY 2001 Annual Performance Plan). In short, in FY 1999, we reassessed the progress and results of the multi-year evaluation project and determined that a more structured approach was required. This different approach is the Kirkpatrick Methodology and it addresses this indicator, i.e., measure program quality and effectiveness. The Kirkpatrick Model of Evaluation is described under the first indicator, “Assess overall program quality and effectiveness….”

M In FY 2000, survey data from recent graduates of Management Development Center (MDC) seminars will be used to develop information on how to increase the ability of MDC alumni to apply the knowledge they gained in ER programs back at their work place (Levels 3-4 of the Kirkpatrick Model of Evaluation). This information will also be used to improve the design of our programs.

EMDC completed an evaluation study to collect information from FY 2000 graduates of the Leadership Potential Seminar. This multi-method survey asked graduates to self-assess their behavior changes in the 11 course objectives as a direct result of attending the course. These course objectives reflected competency-based knowledges, skills and abilities that are considered vital for leadership within the workplace. Respondents indicated significantly successful behavior change. Ninety-three percent indicated more effective behaviors after the course. More specifically:

a. 92% of respondents indicated either “Completely” or “Generally Successful” in demonstrating “increased awareness of self and others.
b. 86% indicated “Completely or Generally Successful” in demonstrating behavior change in delivering and receiving feedback.
c. 85% indicated “Completely or Generally Successful” in demonstrating behavior change related to understanding the foundations of leadership, and that management and leadership are complementary but not interchangeable.

In FY 2000, a web-based survey was developed and delivered to MDC alumni in August 2000. Resulting information and correlation of data has not been completed due to resource constraints. It is expected that these data will be evaluated by mid-year 2001.

M In FY 2000, begin piloting a Federal analogue of the Corporate University Exchange to share benchmarking data on leading edge management development programs with agency counterparts.

In FY 2000, OEMD continued to do substantial benchmarking but by means other than the establishment of a federal analogue such as the Corporate University Exchange and often the benchmarking was done with other members of various consortia. Consequently, this indicator and the following one are combined in the FY 2001 Annual Performance Plan.

OEMD participated as a member of the American Productivity and Quality Center and was a sponsor of an ROI study conducted by the Center. The study benchmarked five organizations (three private sector, one Government and one quasi-government) that conduct ROI studies in executive and management training.

In FY 2000, OEMD, FEI/LDS, began sharing benchmarking data on leading edge management development programs with agency counterparts. Such sharing of data on the Action Learning Model/Program (Leadership for Results) took place with the Department of the Treasury, the Nuclear Regulatory Commission and the Department of Veterans Affairs. Additionally, FEI/CEL shared benchmarking data on the on-going research and subsequent management development programs being conducted by the Center for Creative Leadership (CCL) regarding the leadership skills necessary for executives working in international environments with the USAID. These exchanges enabled OEMD to access forward thinking practices/processes. This sharing of benchmarking data has led to improvements in the quality of programming and more nearly ensures that OEMD programs are using leading-edge management theories and practices.

In FY 2000, FEI hosted several international visitors who joined in discussions of both the FEI and the visitor Government programs. The purpose of these visits included potential collaboration, sharing best practices and student and faculty exchanges. International visitors included individuals or delegations from Korea, the United Kingdom, Hong Kong, and Malaysia.

M In FY 2000, seek membership in executive development consortiums (i.e., Government, academia, business) for the purpose of benchmarking our programs and identifying meaningful evaluation and measurement techniques used in other executive development programs.

EMDC continued its representation of OEMD on the National Academy of Public Administration’s (NAPA) Consortium on Government Performance to gather up-to-date information (e.g., case studies on successful implementation of GPRA principles) on the Government Performance and Results Act for inclusion in OEMD program offerings. Based on learnings shared through consortium members, EMDC made course modifications to its GPRA Seminar. Overall course evaluations for FY 2000 seminars were 4.6 on a 5.0 scale. Additionally, EMDC continued its representation on the executive program committee that is responsible for designing the 2000 Excellence in Government conference. The President’s Quality Award Program ceremony remains the centerpiece of the conference.

In FY 2000, OEMD participated in the Organization for Economic Cooperation and Development (OECD) Human Resources consortium of 25 countries that held a two-day meeting to share/benchmark leadership development practices. As a direct result of this participation, four countries expressed a keen interest in sending their senior executives to FEI. Additionally, OEMD participated in the Center for Creative Leadership’s (CCL) “Friends of the Center” Conference and Executive Forum where a number of private companies shared best practices.

OEMD Goal 2 -- M

By the end of FY 2000, participation in OPM’s executive and managerial training programs is increased and income and costs are balanced.

This program goal was established under Strategic Plan Goal IV in order to assist OPM in meeting its goal of delivering high quality cost effective human resources. In order to accomplish OEMD Goal 1 (improve individual and organizational performance), most effectively, we must continually improve, expand and grow our programs and products to meet client needs. This must be done in a fiscally responsible manner that provides the opportunity for development of new directions and methodologies while also keeping costs to consumers at a low/reasonable cost. As a revolving fund, it is necessary that we break-even over a reasonable period of time. In short, if we are to achieve Goal 1, our overall financial health must provide the margins needed to take advantage of investment opportunities while extending our market reach (growth in participation). There are four indicators to assess whether or not this goal is achieved. Of these, the first indicator was identified at the beginning of the year as most critical for achieving the goal. We met this goal by substantially exceeding both our FY 1999 performance and our FY 2000 estimate. OEMD conducted 328 sessions comprised of 10,164 participants, resulting in 87,890 participant training days.

In addition to the accomplishments described under each indicator, we developed and maintained a Balanced Scorecard that translated the OEMD mission and strategies into objectives and measures, organized into four different perspectives: financial, customer, internal business procedures and learning and growth. We achieved 85% of our measures. The use of this methodology focused our attention and resources on key assets essential to our ability to survive and prosper. The measures, which are monthly reviewed, updated -- and shared with all employees -- resulted in better and quicker decisions. We believe the achievement of 85% of our measures was significantly responsible for OEMD’s best year ever (in 32 years) and employees shared in that success via goal sharing.

M Outcome evaluation assesses the extent to which a program achieves its outcome-oriented objectives. It focuses on outputs and outcomes to judge program effectiveness, but may also assess program process to understand how outcomes are produced. The Kirkpatrick Model of Evaluation, described above, is used to assess overall program quality and effectiveness and how well courses improve participant’s individual and organizational effectiveness (outcome). The analysis of those evaluations is used to continually review and update the training and development programs and curriculum (process). [Critical indicator]

Finally, OEMD output is measured primarily as the net increases and decreases in participants, and participant training days.

In FY 2000, 10,164 managers and executives participated in OEMD residential and non-residential programs for an overall increase of 36% over FY 1999. The dramatic growth is due primarily to increases in agency demand for custom programs and consulting. Additionally, course evaluations were high; initial estimate for core programs was 4.5 but ratings averaged 4.6 on a 5.0 scale (Likert) and also exceeded the standard of 4.4 on a 5.0 scale with an average evaluation of 4.6 for non-core programs. Finally, a systematic approach to curriculum improvement was achieved via utilization of curriculum conference results, focus group reports and benchmark collaborations.

M OEMD’s financial strategies continue to support current and future program needs, as well as maintain a strong fiscal condition. Financial performance for each organization is measured as a function of annual income and cost in the context of a three-year financial plan which includes meeting the requirements to “break-even” and setting a 3-year investment horizon to ensure that ER is careful and thorough about providing full value to its customers.

At the end of FY 2000, OEMD’s cumulative retained earnings stand at approximately $2,500,000. This provides a sufficient reserve to ensure financial stability and a stable base for investment to meet current and future program needs.

In FY 2000, OEMD, in its continuing effort to provide for current and future program needs and give value to its customers, designed, developed and implemented its Leadership Learning System (LLS) of which Training Server is the core database (Phase I). We began pilots of on-line courses for each site for pre/post course materials and data collection, on-line registration information forms, virtual collaboration courses and computer-based simulations. Intended outcomes are to lower program costs, provide evidence of impact of our programs through state-of-the-art technology while at the same time increasing program quality.

M In FY 2000, financial performance standards for each organization will include actual performance within tolerances of initial estimate (+$250k,-$125k), sufficient cumulative retained earnings reserves to accommodate appropriate contingencies, and an investment strategy consistent with program requirements and break-even over a three-year period.

By design, in FY 2000 OEMD organizations exceeded performance tolerances on the plus side to achieve annual return earnings of +$439,000 and cumulative retained earnings of approximately $2,500,000. This was accomplished through revenue growth and the accompanying economies of scale and cost management. While OEMD is required to break even over a reasonable period of time, tolerances were exceeded to ensure a stable financial base of cumulative retained earnings including a sufficient contingency reserve. We believe that cumulative balance should remain between $2 and $3 million.