II. PERFORMANCE EVALUATION SYSTEMS
For the purposes of this discussion, an evaluation is:
(1) an assessment of the effectiveness of ongoing and proposed programs in achieving agreed-upon goals and objectives and
(2) an identification of areas needing improvement through program modification (including the possible termination of ineffective programs), which
(3) takes into account the possible influence of external as well as internal organizational factors.
An evaluation can focus on the extent to which programs are implemented according to predetermined guidelines (process) or the extent to which a program produces change in the intended direction (impact).
Efficiency Versus Effectiveness
Questions of efficiency often are defined and answered in least-cost terms, with minimal consideration of priorities or of the relative worth of the programs pursued.
Increasing effectiveness often involves radical program adjustments--one reason why evaluations that focus on effectiveness may not be fully utilized.
Criterion of efficiency asserts that a choice among alternatives should be made in favor of the course of action that produces the largest result for a given application of resources. [1]
To guide this choice, it is necessary to determine appropriate levels of goal attainment or program adequacy (e.g., a minimum acceptable level of performance).
Cost-benefit analysis requires estimates of direct and indirect costs and of tangible and intangible benefits and focuses on issues of efficiency--the greatest benefits for the lowest cost.
Cost-effectiveness analysis relates incremental costs (expressed in monetary terms) to increments of effectiveness (often expressed in terms of the actual performance associated with program objectives).
Formative and Summative Evaluations
Formative evaluations provide the information necessary to design and/or modify service delivery systems.
Summative evaluations measure performance and program impacts.
Analysis of input variables can provide information necessary to identify more clearly why a program might or might not be successfully implemented.
Two kinds of intervening variables must be measured:
(1) program operation variables; and
(2) bridging variables, i.e., the intermediate steps selected to achieve program objectives.
The final products of the formative evaluation process should be:
(1) a service delivery plan, based on an understanding of the causal relations between the activities to be performed and the desired results;
(2) a set of goal statements, outlining a course of action in broad terms; and
(3) supporting objectives, which measure progress toward goal achievement.
Traditional Performance Measures
Efficiency measures relate a workload measure to a unit of input (e.g., cost).
Work standards measure of the amount of effort that specific tasks should require.
Utilization statistics provide another kind of performance measure--e.g., percentage of total capacity utilized, equipment downtime, etc.
Impact evaluation provides information relating to program consequences--intended or unintended, positive or negative.
Basic Approaches to Evaluation
Standard approaches for conducting an evaluation include: (1) before-and-after comparisons; (2) time-trend-data projections; (3) with-and-without comparisons; (4) comparisons of planned versus actual performance; and (5) controlled experimentation.
The first step is to identify the relevant objectives of the program or activities under evaluation and the corresponding evaluative criteria or effectiveness measures.
The final step should include a thorough search for other plausible explanations for the observed changes and, if any exist, an estimate of their effects on the data.
Rossi has identified a number of "competing processes" that may influence program effects. [2]
The outcome of any program is a function of net program effects and these confounding elements.
Before-and-after comparisons are the simplest and least costly evaluative approaches.
o Conditions in a given jurisdiction or target population are examined at two points in time--immediately before a program is introduced and at some appropriate time after its implementation.
o This approach is valid only in situations where comparisons are not likely to reflect short-term fluctuations and where program-related changes are clearly measurable.
Time-trend-data projections draw comparisons between actual post-program data and extrapolated data suggestive of conditions that would have prevailed without the program.
o Pre-program data are projected to the end of the evaluation period by means of standard statistical methods.
o Actual and projected estimates are compared to determine the amount of change resulting from the introduction of the program.
o Statistical projections may be relatively meaningless if data for prior years are unstable.
With-and-without comparisons examine a population to which a particular program has been applied and one or more "control groups" to which comparable programs have not been applied.
Although this approach controls for some important external factors, it generally is not a fully reliable measure of program effects.
After-the-fact comparisons involve rather straightforward procedures and yet are rarely used.
o A specific measurable objective is established prior to the initiation of the program.
o The actual performance (program outcomes) is then compared to this target.
o Appropriate, realistic objectives must be established as the basis for evaluation criteria. Targets may be overstated and, therefore, unattainable, or they be understated to make the program achievements look better.
Controlled experimentation is the most potent approach to evaluation; it also is the most difficult and costly to undertake. The basic steps are as follows:
(1) Identify relevant objectives and corresponding evaluation criteria
(2) Select target populations that have similar characteristics with respect to the likelihood of being effectively treated by the program.
(3) Assign target population to control and experimental groups in a scientifically random manner.
(4) Measure the pre-program performance of each group.
(5) Apply the program to the experimental group but not to the control group.
(6) Continuously monitor the operations of the experiment to determine if any actions occur that might distort the findings.
(7) Adjust any such deviant behavior, if appropriate and possible; if not, at least identify and estimate its impact on eventual findings
(8) Measure post-program performance of each group.
(9) Compare pre- and post-program changes in the evaluation criteria of the groups.
(10) Search for plausible alternative explanations.
The selection of an appropriate approach will depend on the timing of the evaluation, the costs involved and resources available, and the desired accuracy; some or all of the methods can be used in combination.
Applications of Evalutation Findings
As Rossi has observed: "Evaluations cannot influence decision-making processes unless those undertaking them recognize the need to orient their efforts toward maximizing the policy utility of their evaluation activities." [3]
Management audits involve an assessment of resource utilization practices, including the adequacy of management information systems, administrative procedures, and organizational structure.
Performance audit extends the focus of a management audit to include an examination of program result to determine if: (a) the desired benefits were achieved, (b) program objectives were met, and (c) alternatives were considered that might yield the desired results at a lower cost.
Sunset legislation provides for the following:
(1) Agencies and/or programs are assigned an mandatory termination date.
(2) An opportunity is given to justify the continued existence of the agency or program prior to termination.
(3) If the legislative body takes no formal action to reinstate the agency, the enterprise is concluded on that date.
(3) Reinstatement may leave the agency/program unchanged, whereas reconstruction may lead to significant modifications in the mandate and responsibilities.
(4) If reauthorized or reconstructed, the agency or program will again be subject to review and possible termination at the end of the next cycle.
If sunset laws are applied across-the-board, legislators are likely to take the safe route and allow the agencies/program continue.
Program reconstruction suggests a refining and re-targeting of programs (and policies) rather than setting totally new directions.
Programs are constantly adapted to emerging situation in order to avoid termination.
Faced with fiscal constraints, program modifications/reconstructions are becoming more viable.
PRODUCTIVITY/QUALITY MANAGEMENT
Concerns for the decline in the nation's annual rate of growth in productivity and the loss of competitive position have spawned a host of new initiatives to increase productivity.
Attempts have been made to integrate more traditional measurement techniques with new forms of participative management.
Current Models for Productivity Improvement
Models that seek to address the issues of performance, productivity, and quality tend to be hybrid systems.
o Under the comprehensive productivity measurement approach, efforts are made to incorporate measures of productivity into the organizational processes for goal setting and budgeting, going well beyond the simple control/accountability systems.
o The use of participative management approaches to focus correctional efforts usually results in a more integrated performance/productivity improvement effort that includes more sophisticated measurement devices.
o Major experiments are underway to develop methods by which to analyze and evaluate knowledge-based, white collar organizations in terms of resource and management requirements and service and production capacity.
o Team building involves a number of strategies designed to deal with intra- and inter-group competition and structural rigidities and unresponsiveness within an organization.
Current Models for Quality Improvement
The fundamental underlying concept of the Quality Circle is that small voluntary groups of key participants can discuss problems and plan for and implement actual solutions.
In Quality of Worklife (QWL) approaches, evaluation focuses on measurements of the quality of life in the work environment; employees participate in all phases of a decision-making process based on obtaining consensus among all sectors of the organization.
Productivity gainsharing has been adopted by both private and public organizations.
o Under this approach, a portion of the savings created by improvements in productivity is returned to the employees in the form of bonuses,
o Some critics have noted that these programs have relatively short life spans, especially if the participation of the workforce has not been firmly established.
Total Quality Management (TQM) is "a structural system for creating organization-wide participation in planning and implementing a continuous improvement process that exceeds the expectations of the customer. It is built on the assumption that 90 percent of problems are process, not employee, problems." [4]
o TQM advocate the education and retraining of employees to increase their "quality awareness" and to attitudes that manifest strict adherence to product and service specifications --that is "conformance to requirements."
o TQM approaches often are highly statistical, requiring extensive charting of reliability and defects rates to ensure that workers build in work quality rather than relying on later inspections for defects.
The Quality Improvement Process (QIP) builds on three basic elements:
(1) Problem-solving teams established at various levels within the organization;
(2) Formal mechanisms for the systematic identification and deployment of policy; and
(3) Application of Plan-Do-Check-Act procedures to involve workers at all levels in quality improvement on a day-by-day basis.
Perhaps the most important lesson to learn from the efforts in the 1980s to improve productivity and the quality of services is the fact that it is relatively easy to establish a productivity/quality improvement program. The hard part is to sustain such efforts.
Endnotes
[1] Herbert A. Simon and C.E. Ridley, Measuring Municipal Activities (Chicago: International City Managers' Association, 1938).
[2] Peter H. Rossi, Howard E. Freeman, and Sonia Wright, Evaluation: A Systematic Approach (Beverly Hill, Calif.: Sage Publications, 1979), pp. 172-175.
[3] Ibid., p. 283.
[4] L. Edwin Coate, "TQM on Campus: Implementing Total Quality Management in a University Setting," Business Officer (November 1990).