Performance Measurement

Appendices | Back | Table of Contents | OPM Home Page

Quality of Performance Data

Overall, the quality of the performance data and information used to describe our performance during FY 2000 is good. The performance information presented in this report, particularly as it relates to whether goals were met or not met, fairly represents our performance during the year. Our FY 2000 Performance Plan contained 117 annual goals, supported by 458 measures or indicators.

As reported in the Management Discussion and Analysis section of this report, we did not meet 10 of the 117 goals for FY 2000. This determination was based on our strict assessment of performance information that demonstrated that we had not met the performance levels we had set for these goals. Most of these 10 unmet goals were related to Strategic Goals I and IV, as four pertained to policy initiatives and another four pertained to the delivery of customer services.

We had no data to report for 27 of our FY 2000 performance indicators. Primarily, this was due to our decision to postpone our survey of agency Human Resource Specialists until the spring of 2001. We postponed the survey because of the low response rate we received in FY 1999 and feedback we had received from Specialists indicating that responding to it every year was too often. Information from this survey will be in the FY 2001 Performance Report. Because of the postponement, we did not have Specialists’ data for this report. However, the lack of this performance information did not significantly compromise our ability to determine whether performance goals were met, because of our use of multiple measures. Postponing the Specialists’ survey had the greatest impact on indicators not met for Strategic Goal I.

We lacked data in several other instances because of changes we made to our survey of HR Directors to align this measuring instrument with our new corporate measurement strategy. As is the case with the Specialists’ survey, not having this information did not significantly compromise our ability to assess performance regarding our performance goals.

We dropped 24 indicators during the year, 67% of them relating to Strategic Goal I. Most of these indicators supported performance goals that we dropped because of adjustments we made to our policy agenda during the year. Because these dropped goals were not being pursued, there was no need to expend resources to collect data for their related indicators. We dropped indicators for the other Strategic Goals primarily because we realized during the year that these indicators did not clearly relate to the respective goal and would not contribute to determining whether the goal was met.

We did not meet the performance targets described in 65 indicators, which represent 14% of the total number of indicators in our FY 2000 Performance Plan. Most of the indicators we did not meet were related to our policy leadership agenda under Strategic Goal I and the delivery of direct services to our customers under Strategic Goal IV. Missing the target on these 65 indicators resulted in our not achieving 10 of our performance goals for the fiscal year, with the biggest impact occurring in our delivery of services to our retirement program customers (retired employees and survivor annuitants).

While we were disappointed that we did not achieve these goals, we either made significant progress toward our desired outcomes during the year or fell only slightly short of the established target level for critical indicators. For instance, we did not meet our goal to establish a Governmentwide Career Intern program during FY 2000 (see Employment Service Goal 9 in Appendix A-2) because the executive order establishing the program was not issued until December 2000, after the end of the fiscal year. Also, we did not meet our goals to maintain at FY 1999 levels the delivery of services to our retirement program customers (Retirement and Insurance Goals 11-13, Appendix A-8) because we did not meet the target levels we set for processing claims, responding to customer inquiries, and processing customer service requests. However, customer satisfaction with these services remained at high levels and we have made improving these services a priority in our FY 2001 and FY 2002 performance plans.

Data Reliability. We recognize the need to strengthen the data used to support some of our performance indicators. A preliminary review conducted by our Inspector General of 116 of the 458 performance indicators used in this report indicates that 59% of the indicators are based on reliable information. The IG was unable to determine the reliability of 24% of the indicators reviewed and found that the remaining 17% were unreliable. We expect these figures to improve in the IG’s final report, as we responded to their recommendations and provided supporting documentation for certain indicators.

Low response rates to the HR Directors’ and HR Specialists’ surveys are a concern. In FY 1999, the response rate to the HR Directors’ survey was low enough to cause us to chose not to use the survey results in our performance report. In FY 2000, the response was improved but remained low enough to result in a margin of error of 9.9%. Thus, while we used the results for our performance in FY 2000, the performance information pertaining to these results is not as precise as we would prefer. As stated previously, concerns we had about the response rate to the FY 1999 HR Specialists’ survey prompted us to postpone repeating it until the spring of 2001. Fortunately while this survey had a 24% response rate when last administered in FY 1999, the size of the sample was large enough that this response rate supported a 95% confidence level and a 2% margin of error.

For many of our indicators we relied on secondary anecdotal information to determine whether we had met the target established in our performance plan. Many of these indicators were dependent on the HR Specialists’ survey and pertained to their satisfaction with our information sharing, technical assistance and other support services that we provide to them. Since we did not have the information from this survey, we used feedback collected through informal methods during conferences, briefings, and through correspondence. While this practice lacks the precision of reliable quantitative data, it provided a sense of their level of satisfaction with our support services. Further, the anecdotal evidence used in this report is being used by OPM program managers in carrying out their responsibilities.

We have already taken steps to make our performance data more reliable in future performance plans. For instance, we discontinued several indicators in our FY 2001 and FY 2002 Performance Plans that were not based on reliable data sources. We have established an agency level performance measurement work group that will institutionalize our surveys of HR Directors and Specialists to ensure that these data collection tools are administered on a regular schedule and provide relevant and reliable information. Also, we are improving our data validation and verification program to strengthen our data collection procedures and controls pertaining to performance data. Finally, we are implementing a new corporate measurement framework that represents a significant change in our approach to performance measurement.

Multiple Measurement Strategy. Our approach to performance measurement has been to use multiple indicators to assess progress regarding our goals. Under this approach, for most annual goals described in our performance plan we defined at least two performance indicators. Because much of the work we do involves developing and deploying human resources policy, most of our indicators have been descriptive. Also, many of our goals and indicators have been focused on the activities in which we are engaged as we develop and implement these policies. The descriptive and activity-based nature of these indicators reflects the challenges of assessing the results of policy development and deployment and in measuring the quality of the Federal workforce.

Our Corporate Measurement Framework

After developing five Annual Performance Plans (FY 1998 through FY 2002) and our first Annual Performance Report (FY 1999) under the Government Performance and Results Act, we realized that our performance measurement system needed to be improved. As we revised our Strategic Plan during the spring and summer of 2000, we identified the following weaknesses in the goals and indicators appearing in these documents:

• Many of our goals and indicators were activity-based, measuring only whether OPM performed certain activities, and did not measure the program results those activities were intended to bring about.
• Indicators were focused only on our annual performance goals, and did not provide information regarding whether we were making progress toward achieving our strategic goals.
• Indicators were not complete in that they did not cover all aspects of the goal. For instance, the indicators for some goals dealt only with customer satisfaction data, but did not include measures of efficiency, effectiveness, or cost.

The causes of some of these weaknesses are related to OPM’s performance planning and measurement approach. Prior to the development of OPM’s Strategic Plan dated September 2000, the approach for performance planning was to develop strategic goals, which were supported by a number of program-specific strategic objectives. These program objectives were then supported by numerous annual performance goals, which were evaluated by performance measures and indicators.

As a result, our FY 2000 performance plan contained 117 goals and 458 measures. The FY 2001 plan is an improvement, but still contains 93 goals and 375 measures. The large number of goals and measures makes our performance plans and reports long and unwieldy. For instance, the FY 2001 Performance Plan is 343 pages long and the FY 1999 Performance Report was 136 pages long, including fourteen appendices.

To strengthen our performance measurement system, we developed a corporate measurement framework as we revised our Strategic Plan in FY 2000 (see Federal Human Resources Management for the 21st Century, U.S. Office of Personnel Management Strategic Plan FY 2000-FY 2005). The corporate measurement framework reflects a change in our approach to performance planning so that our performance measures track progress toward our strategic goals and measure the long-term results we expect to achieve.

Under this framework, we identified the outcomes we seek for each of our Strategic Goals and developed broad measures that define these outcomes from multiple and balanced perspectives. We already have data to support many of these corporate measures and reported this information in the Management Discussion and Analysis section of this Report.

In addition, we began implementing the corporate measures in our FY 2002 performance plan and expect to complete this process as we prepare our performance plan for FY 2003. The implementation of this framework will include aligning our current performance data collection processes with the new framework and developing new data collection instruments where needed. As we collect data and determine baselines for the corporate measures, we expect to establish long-term targets that will appear in future Annual Performance and Strategic Plans.

Summary of Program Evaluation Results

We conduct targeted program evaluations, both internal and external, on specific program areas to measure our results and provide information for our strategic planning activities. Program evaluation activities are primarily conducted within the appropriate program offices and are supplemented by evaluations conducted by external contractors when necessary. In addition, our Office of Merit Systems Oversight and Effectiveness (OMSOE) conducts special studies on contemporary, crosscutting HRM issues. Our Office of the Inspector General conducts periodic evaluation studies and audits of our programs that provide an additional source of internal feedback. We also use reports produced by external organizations such as the General Accounting Office, the Merit Systems Protection Board, the National Academy of Public Administration and the groups that audit our trust funds. Examples of evaluation reports, their results, and the actions we took are described below.

• Competing for Federal Jobs: Job Search Experiences of New Hires” -- Merit Systems Protection Board (MSPB), February 2000. Major changes to the Government’s hiring process were made in the early 1990’s, including decentralizing the competitive examining process and providing a centralized automated system for all Federal vacancy announcements which could be accessed electronically (USAJOBS). MSPB notes that these changes appear to have had positive effects. However, MSPB recommended that agencies and OPM need to improve how vacancy announcements are posted on the Internet, i.e., they should be visually appealing, informative and easy to navigate. In response, OPM formed a working group which created a list of enhancements for vacancy announcements. The enhancements will be implemented during the redesign of USAJOBS. The new vacancy announcements will use the power of the Internet by creating hyperlinks to general information yielding a shorter, more specific and customizable announcement.
• The Role of Delegated Examining Units: Hiring New Employees in a Decentralized Civil Service -- MSPB (August 1999). This study examined how competitive hiring is being handled by agencies through delegated authority from OPM (i.e., via Delegated Examining Units). MSPB found positive opinions regarding the success of delegated examining. The report had several recommendations for Congress, OPM and departments/agencies. In response, among other actions, OPM made available to all Delegated Examining Units the assessment tools used for the Administrative Careers with America (ACWA) program.

In FY 2000, OPM’s Office of Merit Systems Oversight and Effectiveness (OMSOE) completed and distributed four special studies.

• The first, An Occupation in Transition, Part 3 – The HR Workforce: Meeting the Challenge of Change, is the final installment of a three-part study on the transition of the HR workforce. It is guiding the work of the Human Resources Management Council (HRMC) as that interagency group of HR leaders addresses the issues outlined in the report. Specifically, the HRMC established a subcommittee to focus solely on the HR profession, and this subcommittee is working closely on specific HR workforce improvement initiatives with OMSOE.
• The second study, The Three Rs: Lessons Learned from Recruitment, Retention and Relocation Incentives, describes the effects of recruitment, retention and relocation bonuses within agencies and is contributing valuable data to our total strategic compensation redesign efforts.
• Another study reviewed which authorities are being used by agencies to make employee work life more family friendly. Achieving a Balance: Meeting Work and Family Obligations describes optional workplace arrangements available to Federal employees. OPM’s Office of Work/Life Programs in the Office of Workforce Relations (OWR) has distributed this report to its interagency network of work/life coordinators It has also helped to reinforce OWR’s plan of action and commitment to future activities in the work/life arena.
• The fourth study, Incentive Awards -- The Changing Face of Performance Recognition, is a follow-up study to a 1998 report on incentive awards. OPM issued revised regulations in 1995 to give agencies more flexibility in tailoring award programs to their needs. The previous review showed that a number of agencies were on the verge of implementing changes based on the revised regulations. The 2000 report describes changes the agencies have made, and their assessment of the results of those changes. All agencies feel that the current regulations give them sufficient latitude to design the type of awards program that best fits their situation.
• We also conducted a study of benefit counseling as it is performed by agencies, Federal Benefits Counseling: Putting the Pieces Together. The study found that while most Federal employees are satisfied with the benefits information their agencies provide overall, they need more information in particular areas and they want pre-retirement seminars earlier in their careers than traditionally available. The study also showed that agency benefits officers believe they can obtain the information they need to adequately counsel employees, but they do not feel they have time to keep up with all the changes that are occurring in the benefits area. Finally, most benefit counselors use the OPM Website as their primary source for counseling support, and found it very useful.

We used the study’s findings to develop improvements in how we currently provide support to agency benefits officers, and also in conjunction with focus group studies we are conducting for our Retirement Systems Modernization Project, to develop the benefits counseling component of the modernized retirement system.

The Merit System Principles Questionnaire (MSPQ) is distributed to a nationwide random sample of Federal employees in the last quarter of the year, with results timed for use in our oversight reviews during the subsequent year. This survey is administered to a small but statistically valid sample of Federal employees. The entire questionnaire was rewritten in FY 1999 to solicit respondents’ overall impressions of how things are in their work units. Because of this, the FY 1999 results established a new baseline.

The MSPQ includes questions on the perception of merit principles and human resources management (HRM) effectiveness, including effective use of employees. It is our primary measurement tool for determining if the merit principles are being consistently honored and for assessing whether we are maintaining an effective oversight program. Several OPM offices use the results as surrogate measures of certain objectives and as indicators of needed improvements in policy.

Many of the quantitative indicators used in our FY 2000 Performance Plan are based on surveys of our customers and stakeholders. These surveys are valuable in assessing their satisfaction with our services, but also to research other aspects of our performance in making effective policy. Currently, we use three surveys for performance measurement purposes, as shown in the following table.

The HR Specialists Customer Satisfaction Survey is administered biennially and collects information from agency staff working in human resources management positions regarding the quality, usefulness, and effectiveness of the advice, assistance, training, informational materials, and other support we provide to these staff for the administration of HR programs across Government.

Client Satisfaction Survey of Retirement Program customers (i.e., annuitants). We have conducted annual customer satisfaction surveys of our Civil Service retirees and survivor annuitants since 1990. The purpose of the survey is to determine the level of client satisfaction with our services among retired employees and survivor annuitants. The areas covered by the survey include processing claims for retirement benefits, answering correspondence, handling telephone inquiries, processing of transactions for existing accounts, and the extent and quality of retirement counseling that the annuitants received from their former employing agencies. Only annuitants retired in the last two years were asked to complete the retirement counseling section.

For FY 2000, two surveys were conducted, with the first using a sample of transactions that occurred between October 1, 1999, and February 29, 2000; and the second covering the period from March 1, 2000, through August 31, 2000. Taken together, survey questionnaires were mailed to random samples of 1,227 annuitants and survivor annuitants. We received responses from 797 of these customers for a response rate of 65%. The findings presented here have a confidence level of 95% and a margin of error of 5%(+/-).

Consumer Assessment of Health Plans Study. The Consumer Assessment of Health Plans Study (CAHPS) is our primary vehicle for assessing health plan performance within the Federal Employees Health Benefits (FEHB) Program. This survey instrument is the health care industry’s most thoroughly tested and widely accepted tool for obtaining customer feedback regarding health plan performance. CAHPS-based results are published in our annual FEHB Open Season Guide for each carrier participating in the Program. Summary results were first used to assess overall customer satisfaction in our FY 1999 Performance Report and are being used in our Annual Performance Plans beginning with FY 2001.

The American Customer Satisfaction Index. The American Customer Satisfaction Index (ACSI) is an econometric indicator that was developed by the National Quality Research Center at the University of Michigan Business School and has been used in the private sector to measure customer satisfaction since 1994. It is based on modeling of customer evaluations of the quality of goods and services that are purchased in the United States.

We began participation in the ACSI in 1999 and have used it to validate and complement our own surveys. Those surveyed were Federal retirees and survivor annuitants. For 2000, OPM’s overall satisfaction score was 73, compared to the private sector score of 71.2. Specific results support those from our own surveys of retirement program customers. In FY 2000, annuitants seemed less satisfied with the timeliness of the claims process. And, although they find our information timely, clear and accurate, it is not as user friendly as they would like. Improving our claims processing times is a priority in our FY 2001 and FY 2002 performance plans. We will also contract for additional resources to respond to customer demand when it exceeds our ability to handle telephone calls. We will promote the use of our Website and its interactive services to allow customers more avenues to perform transactions and obtain information.

OPM Employee Survey Satisfaction. As part of its program evaluation, the Office of Human Resources and EEO administered an OPM-wide employee survey. Of those who responded, 78% said they were satisfied with their job overall. On a national survey, 66% of OPM employees indicated they were satisfied with their job, compared with 63% Governmentwide. When asked if OPM supports family/personal life responsibilities, 73% answered affirmatively, compared to 65% Governmentwide. On the OPM-wide survey, 76% of the respondents indicated OPM supports family/personal life responsibilities.