U.S. Office of Personnel Management FY 2000 Budget Justification/Performance Plan

VERIFICATION AND VALIDATION

A fundamental concept behind the Government Performance and Results Act is that measurement will become an integral part of program management and budget decisions. If measurement is to be used for such critical decision-making, it must be based on useful, reliable, valid data. Chosen measures must have meaning for stakeholders and for program managers. Simultaneously, any data collection must be practical and cost-effective. These are not easy standards to meet, particularly during the early stages of introducing a broad-based program of measurement.

At the U.S. Office of Personnel Management (OPM), we are fortunate in that we have a tradition of measurement in key program areas, and through these programs, direct access to top-quality measurement expertise. The Employment Service (ES), the Investigations Service (IS), the Office of Merit Systems Oversight and Effectiveness (OMSOE), and particularly the Retirement and Insurance Service (RIS) have developed and used extensive measurement information for many years. Data are used to track inputs, activities, outputs, and outcomes, including public customer satisfaction, and the data are accessible via internal management information systems. Other parts of OPM have less long-term measurement experience, but all introduced at least some measures and data management systems prior to the implementation of the Results Act.

Nevertheless, OPM, like other agencies, faces some significant challenges in assuring that the various measures used to support the Results Act and to improve internal program management are appropriate, reliable, valid, and cost-effective, and that systems are in place to ensure that data quality remains high. In addition, OPM, like other agencies, must take steps to ensure that measurement data are used effectively to improve program management.

OPM has a three-part strategy for approaching these challenges:

• Balanced Measurement: Using a balanced set of measures, from multiple sources, to assess program performance;

• Data Quality: Strengthening the quality of measurement data and systems used; and

• Use of Measurement Data: Ensuring that program managers understand how to use data effectively.

Each of these strategies is discussed below.

Balanced Measurement

OPM’s Strategic Plan includes five major goals -- four external and one internal -- which are all focused on the overarching outcome of achieving a high-quality, diverse Federal workforce, within the principles of a merit-based system. The five strategic goals define outcomes for policy leadership, oversight, technical assistance, and direct service delivery (external goals), and operating as a model employer (internal goal), to be reached by the end of FY 2002. The FY 2000 Annual Performance Plan sets incremental targets for progress toward the strategic goals.

Both the Strategic Plan and the Annual Performance Plan use the process of multi-measurement to assess our success in meeting the overarching goal of a high-quality, diverse workforce. Assessment of the quality of any workforce -- Federal or otherwise -- has been recognized as a significant measurement challenge. In 1992, a blue-ribbon panel of public and private sector executives, union officials and academicians, the Advisory Committee on Federal Workforce Quality, recommended that a broad definition of quality be adopted for measurement purposes. Their definition included measures of individual attributes of workers, organizational processes affecting workers, and outcomes at the individual, team, and organizational level, such as good customer service. The panel recognized that the cost and logistics of obtaining and maintaining many direct measures of workforce quality (such as assessments of the skill levels of individual employees) would be significant, and specifically included the use of employee and customer assessments as important surrogate measures of workforce quality.

Consequently, OPM is using a measurement approach which depends on a balanced set of intermediate indicators from multiple available data sources. Two key indicators are direct surveys to assess the perceptions of Federal employees about workforce quality and the fairness and equity of HRM processes, and user surveys -- from both public and internal customers -- to assess customer satisfaction with the content, process and outcomes of policies and services. The primary employee surveys used are the OPM Merit System Principles Questionnaire and the National Partnership for Reinventing Government (NPR) employee survey. The primary customer surveys used are OPM agency customer surveys (HR Directors and HR Specialists) and surveys of job-seekers, annuitants, and health insurance users. These surveys are described in more detail below.

Other direct measures include financial data, comparative data on the diversity of the Federal workforce, timeliness of operations, and program data. These data are available from OPM’s internal management information systems and systems maintained for Governmentwide use, like the Central Personnel Data File (CPDF), or from systems maintained by other agencies, like the Bureau of Labor Statistics and the Bureau of the Census.

Other key indicators are qualitative in nature, rather than quantitative. These are the results of externally conducted program evaluations, conducted by such entities as the Merit Systems Protection Board (MSPB), the General Accounting Office (GAO), and the National Academy of Public Administration (NAPA). These organizations conduct periodic studies of Federal HRM practices and outcomes, and their findings are an important measure of program success. OPM’s Office of the Inspector General also provides external evaluations, conducting audits and reviews of the earned benefits programs’ financial systems, work processes, and performance measures.

Finally, in some instances OPM relies on indirect, or surrogate, measures as indicators. One surrogate measure is the use of competency-based selection tools to hire and promote workers. The assumption underlying this measure is that recent research has identified the wide range of competencies needed by employees in today’s changing workplace, and the use of tools which are based on a full evaluation of these competencies will translate to a more highly-qualified workforce.

We will continually review our choices of program measures, and incorporate new or improved measures as needed. We will also work with any external providers of data to be sure we are using the best sources, and using the data appropriately.

Following is information on the two key surveys used as elements in OPM’s balanced measurement program.

• OPM Customer Satisfaction Survey. OPM introduced its first agency customer survey process during FY 1998. Two surveys were administered to assess how well we are meeting our responsibility to provide policy leadership, information, and assistance to agencies in conducting their HRM activities.

Each survey is targeted to the appropriate customers for the different kinds of services OPM provides. The first survey was mailed to 75 Human Resource Directors of Federal agencies and departments, including all members of both the IAG Executive Committee and the Small and Independent Agency Personnel Directors Group. This survey asked for evaluations of our policy leadership and our success in involving agencies as partners in the policy development process. The questions covered all HR policy areas in general, and also covered specific FY 1998 initiatives. The second survey was mailed to a nationwide sample of 4,850 HR specialists at all grade levels, and geographic locations. This survey asked for evaluations of our technical assistance and information sharing, including -- again -- specific FY 1998 initiatives, as well as new publications, CD-ROMs, and electronic methods of sharing information and guidance.

The results from these surveys are reported in each program’s section of the Annual Performance Plan. They will serve as the baseline for the improvements in customer satisfaction that are promised in our five-year Strategic Plan, and will be used to track performance over time and make continuous improvements in our products and services. As with any survey data, we are using the results as a starting point, and holding focus groups and conducting other follow-up to understand the areas of dissatisfaction and the improvements that should be made.

Our surveys also incorporate core customer satisfaction dimensions drawn from a research-based model developed by OPM psychologists. This model has been applied to customer satisfaction surveys used in other Federal agencies. The results on the core dimensions can be benchmarked against other Federal agencies, and also against data from similar surveys used in the private sector.

Although the return rates for our initial surveys this summer were lower than we had anticipated (57% for the HR Directors and 29% for the HR specialists), enough surveys were returned to give us confidence in using the data to establish baselines for improvement. We will take steps to improve the response rate next year, including more follow-up to survey recipients and administering the survey earlier in the summer.

• Merit Systems Principles Questionnaire. The Office of Merit Systems Oversight and Effectiveness administered its second annual Merit Systems Principles Questionnaire to a statistically valid, random sample of employees Governmentwide in the summer of 1998. The results of this questionnaire provide a broad indication of employee perceptions of agency accountability with merit system principles, which include merit staffing, fair treatment, integrity, efficiency, performance, training, protection from partisan political pressure and protection from reprisal for lawful disclosure of information. OMSOE uses the results in individual agency oversight visits, and to help identify needed improvements in Governmentwide HRM policies and procedures.

Because the questionnaire measures perceptions, it is necessary to examine the results further by discussing them with employees and looking at related hard data on human resources management. For example, employee perceptions regarding performance may be influenced by a small number of incidents rather than a large and pervasive problems. Nonetheless, they provide a useful marker, or indicator, that can track changes over time.

The 1998 questionnaire was administered to 1,961 supervisors (including managers and executives) and 1,943 non-supervisors Governmentwide. We had a response rate of 42%, which was an increase of 2% over the first year’s survey rate. We are confident that enough responses were received to accurately represent employee views.

Data Quality (Validation and Verification)

OPM will take the following steps to improve the quality of all of our measurement information: We will use our internal Results Act working group to establish standards for data quality, validity, and objectivity for all measures used in OPM programs. The working group will review the data that are currently being used in each program, share best practices, and develop a consistent set of standards for data validity and reliability. They will assist each other to make improvements where needed.

• We will review the quality standards followed by external data suppliers to ensure that we can have confidence in the data they provide.

• We will work with our Office of the Inspector General as they conduct their annual reviews of our Results Act implementation, including an evaluation of the goals and measures that we are using, and implement their recommendations.

• We will use only accepted professional accounting and tracking software to measure cost savings and improvements in quality and efficiency, and assess accuracy of data through quality control reviews.

• We will use accepted professional survey design, sampling techniques, and analysis procedures for internally developed customer satisfaction surveys, to ensure reliability and validity. We will work to improve response rates.

• We will use accepted qualitative analysis techniques to assess direct feedback, participant evaluation forms, and focus group results, to ensure objectivity.

• We will evaluate qualitative analyses of feedback against survey data to determine if the findings from the two assessment methods are consistent, and will follow up discrepancies with additional research, to include focus groups, telephone interviews, or on-site visits on specific issues.

• We will work to improve our data collection, to be sure we are using the most efficient and cost-effective processes. We will use automation to collect data whenever possible, including customer satisfaction data.

• We will work to improve our management information systems, to ensure that they provide managers with the information they need, as well as information for external reporting purposes. The RIS management information system (described in detail in the RIS program section of the Annual Plan) will serve as a good example for other OPM programs.

Use of Measurement Data

In addition to the challenges of assuring good measurement, it is important to ensure that managers understand how to interpret data, how to balance measurement data with other sources of information, and how to integrate measurement review and reporting into their program management.

To help ensure effective use of measurement in OPM, we will:

• Continue the process begun in FY 1998 to align individual performance plans for executives, managers, and employees with the achievement of Strategic and Annual Plan goals and objectives. This is a specific OPM Strategic and Annual Plan goal.

• Ensure that management and staff have the knowledge and skills they need to use measurement information. We will share widely useful guidance materials from other agencies and support necessary training of key staff.

• Communicate to all employees the content of the Strategic and Annual Plans, including the measures of success, so that employees are more aware of how their work helps to accomplish our strategic goals.

• To Budget Web Pages Home Page
• To OPM Web Site Index
• To OPM Home Page

Web Page Created 14 May 1999