USING COST ANALYSIS IN EVALUATION

Meg Sewell and Mary Marczak

WHAT IS COST ANALYSIS?

Cost analysis (also called economic evaluation, cost allocation, efficiency assessment, cost-benefit analysis, or cost-effectiveness analysis by different authors) is currently a somewhat controversial set of methods in program evaluation. One reason for the controversy is that these terms cover a wide range of methods, but are often used interchangeably.

At the most basic level, cost allocation is simply part of good program budgeting and accounting practices, which allow managers to determine the true cost of providing a given unit of service (Kettner, Moroney, & Martin, 1990). At the most ambitious level, well-publicized cost-benefit studies of early intervention programs have claimed to show substantial long-term social gains for participants and cost savings for the public (Berreuta-Clement, Schweinhart, Barnett, et al., 1984). Because these studies have been widely cited and credited with convincing legislators to increase their support for early childhood programs, some practitioners advocate making more use of cost-benefit analysis in evaluating social programs (Barnett, 1988, 1993). Others have cautioned that good cost-benefit or cost-effectiveness studies are complex, require very sophisticated technical skills and training in methodology and in principles of economics, and should not be undertaken lightly (White, 1988). Whatever position you take in this controversy, it is a good idea for program evaluators to have some understanding of the concepts involved, because the cost and effort involved in producing change is a concern in most impact evaluations (Rossi & Freeman, 1993).

THREE TYPES OF COST ANALYSIS IN EVALUATION:

Cost allocation, cost-effectiveness analysis, and cost-benefit analysis represent a continuum of types of cost analysis which can have a place in program evaluation. They range from fairly simple program-level methods to highly technical and specialized methods. However, all have specialized and technical aspects. If you are not already familiar with these methods and the language used, you should plan to work with a consultant or read some more in-depth texts (see some suggested references at the end of this discussion) before deciding to attempt them.

COST ALLOCATION: Cost allocation is a simpler concept than either cost-benefit analysis or cost-effectiveness analysis. At the program or agency level, it basically means setting up budgeting and accounting systems in a way that allows program managers to determine a unit cost or cost per unit of service. This information is primarily a management tool. However, if the units measured are also outcomes of interest to evaluators, cost allocation provides some of the basic information needed to conduct more ambitious cost analyses such as cost-benefit analysis or cost-effectiveness analysis. For example, for evaluation purposes, you might want to know the average cost per child of providing an after-school tutoring program, including the costs of staff salaries, snacks, and other overhead costs.

Besides budget information, being able to determine unit costs means that you need to be collecting the right kind of information about clients and outcomes. In many agencies, the information recorded in service records is based on reporting requirements, which are not always in a form that is useful for evaluation. If staff in a prenatal clinic simply report the number of clients served by gender, for example, you might know only that 157 females were served in March. For an evaluation, however, you might want to be able to break down that number in different ways. For example, do young first-time mothers usually require more visits than older women? Do single mothers or women with several children miss more appointments? Is transportation to appointments more of a problem for women who live in rural areas? Are any client characteristics commonly related to important outcomes such as birth weight of the the baby? Deciding how to collect enough client and service data to give useful information, without overburdening staff with unnecessary paperwork requirements, requires a lot of planning. Larger agencies often hire experts to design data systems, which are called MIS or management-and-information-systems.

If you are working for an existing agency, your ability to separate out unit costs for services or outcomes may depend on the systems that are already in place for budgeting, accounting, and collecting service data. However, if you are in a position to influence these functions, or need to supplement an existing system, there are a number of texts that discuss the pros and cons of different ways of budgeting, accounting, and designing MIS or management-and-information-systems (see Kettner, Moroney & Martin, 1990).

COST-EFFECTIVENESS AND COST-BENEFIT STUDIES

Most often, cost-effectiveness and cost-benefit studies are conducted at a level that involves more than just a local program (such as an individual State Strengthening project). Sometimes they also involve following up over a long period of time, to look at the long-term impact of interventions. They are often used by policy analysts and legislators to make broad policy decisions, so they might look at a large federal program, or compare several smaller pilot programs that take different approaches to solving the same social problem. People often use the terms interchangeably, but there are important differences between them.

COST-EFFECTIVENESS ANALYSIS: Cost-effectiveness analysis assumes that a certain benefit or outcome is desired, and that there are several alternative ways to achieve it. The basic question asked is, "Which of these alternatives is the cheapest or most efficient way to get this benefit?" By definition, cost-effectiveness analysis is comparative, while cost-benefit analysis usually considers only one program at a time. Another important difference is that while cost-benefit analysis always compares the monetary costs and benefits of a program, cost-effectiveness studies often compare programs on the basis of some other common scale for measuring outcomes (eg., number of students who graduate from high school, infant mortality rate, test scores that meet a certain level, reports of child abuse). They address whether the unit cost is greater for one program or approach than another, which is often much easier to do, and more informative, than assigning a dollar value to the outcome (White, 1988).

COST-BENEFIT ANALYSIS: The basic questions asked in a cost-benefit analysis are, "Do the economic benefits of providing this service outweigh the economic costs" and "Is it worth doing at all"? One important tool of cost-benefit analysis is the benefit-to-costs ratio, which is the total monetary cost of the benefits or outcomes divided by the total monetary costs of obtaining them. Another tool for comparison in cost-benefit analysis is the net rate of return, which is basically total costs minus the total value of benefits.

The idea behind cost-benefit analysis is simple: if all inputs and outcomes of a proposed alternative can be reduced to a common unit of impact (namely dollars), they can be aggregated and compared. If people would be willing to pay dollars to have something, presumably it is a benefit; if they would pay to avoid it, it is a cost. In practice, however, assigning monetary values to inputs and outcomes in social programs is rarely so simple, and it is not always appropriate to do so (Weimer & Vining, 1992; Thompson, 1982; Zeckhauser, 1975).

An example will illustrate some of the differences between Cost-Effectiveness and Cost-Benefit studies, what they can tell you, and some of the issues that neither can effectively address:

"Suppose the drop-out rate in an inner-city high school is 50%. Prevention Program A enrolls 20 students, costs $20,000, and 15 of the 20 students (75%) graduate. Thus Program A resulted in 5 additional graduates at a cost of $20,000, or one additional graduate for every $4,000. Prevention Program B enrolls 20 students, costs $15,000, and 12 of the 20 students (60%) graduate. Thus Program B resulted in 2 additional graduates at a cost of $15,000, or one additional graduate for every $7,500 spent. Although Program B is cheaper ($15,000 compared to $20,000), Program A is more COST-EFFECTIVE ($4,000/each additional graduate, compared to $7,500/additional graduate). A COST-BENEFIT ANALYSIS in this situation, instead of comparing unit costs, would require estimating the dollar value of high school graduation (for example, by projecting the difference in lifetime earning capacity of graduates over drop-outs, and lifetime social service costs), and comparing the monetary value of producing more graduates to the monetary cost of providing the program in the first place. Neither method effectively addresses more intangible outcomes of graduation, such as increased self-esteem, or the value of a peer support system." (White, 1988, p. 430)

WHAT COST ANALYSES CAN TELL YOU:

• Cost analyses can provide estimates of what a program's costs and benefits are likely to be, before it is implemented. "Ex-ante" or "before the fact" cost analyses may have to be based on very rough estimates of costs and expected benefits. However, if a program is likely to be very expensive to implement, very difficult to "un-do" once it is in place, or very difficult to evaluate, even a rough estimate of efficiency may be quite valuable in the planning stages (Rossi & Freeman, 1993).



• Cost analyses may improve understanding of program operation, and tell what levels of intervention are most cost-effective. A careful cost analysis within a program might tell you, for example, that it doesn't so much matter whether you have a half-day program or a full-day preschool program for children, but that the teacher-to-child ratio does matter (that is, children benefit more from low ratios than they do from longer days). This information might influence decisions about how many teachers you need to hire, or how many classrooms you need, or how many children you can serve effectively.



• Cost analyses may reveal unexpected costs. A speech therapy program might unexpectedly find that it costs more to use paraprofessionals to work with children than professionals, because the paraprofessionals need more training and supervision, or work with fewer children at a time (White, 1988). Or, cutting the number of home visits allowed by caseworkers serving a large rural area (in order to save on mileage reimbursements) might have the unplanned result of higher long-distance phone bills, because the workers still feel a need to stay in close touch with their clients.

WHAT COST ANALYSES CANNOT TELL YOU:

• Whether or not the program is having a significant net effect on the desired outcomes. Unless you know for sure that the program is producing a benefit, it doesn't make sense to talk about the cost of producing that benefit (Rossi & Freeman, 1993). Cost analysis may be considered an extension of an impact or outcome evaluation, but it cannot take the place of one.



• Whether the least expensive alternative is always the best alternative. Often political or social values other than cost need to determine program and policy choices. When there are competing values or goals involved, cost analysis is often just one factor to be considered, and we need to have some other way of deciding which factors should take priority.

USING COST ANALYSES WITH THE STATE STRENGTHENING EVALUATION GUIDE:

If you are using the Five-Tiered Approach to Program Evaluation outlined in the State Strengthening Evaluation Guide, cost analyses can be used at several levels:

Tier 1 - Program Definition

At this stage, you will probably be using cost studies based on other people's experience in similar programs, since you are unlikely to have cost data of your own yet. This means that the estimates you use will only be approximations, and may not accurately reflect what your program's experience will be. However, "ex-ante" cost analyses, done in the planning stages before implementing a program, can potentially prevent some very costly mistakes. If you have access to cost-effectiveness studies of programs similar to the one you are considering, especially if they allow you to compare the relative costs and benefits of several different ways of delivering a service, before you have made substantial investments of time or money, some program design decisions may be easier. One common example in community-based programs is staffing (eg., deciding whether to use highly-trained professionals to deliver services, or to rely on less highly-paid paraprofessionals or volunteers). While many people assume that the paraprofessionals or volunteers are always less expensive, cost-effectiveness studies in some cases have found that the professionals may be less costly in the long run because they can see more clients, require less supervision time, or are more effective. Of course, costs also need to be weighed against other considerations, such as the fact that paraprofessionals recruited from the community served may more easily gain the trust of clients.

Tier 2 - Accountability

Clearly, fiscal accountability is one of the primary reasons for using any kind of cost analysis as part of your evaluation. Any responsible program should keep service statistics and financial records that are accurate and up-to-date enough to be able to determine some very basic information about unit costs, and funders usually require this. However, the minimal information routinely collected by programs for fiscal and reporting purposes is not always in a form that lends itself to evaluation uses. Often, unless advance planning has taken place, this data is too aggregated to reflect outcomes of interest to researchers and evaluators. At this stage (or earlier), careful consideration should be given to the kinds of client and cost data that will be needed later, so that it can be built into the accounting and record-keeping systems of the program (see Kettner, Moroney & Martin, 1990).

Tier 3 - Understanding and Refining

Like any other type of information gathered for evaluation purposes, the cost information collected in Tier 2 for accountability purposes provides programs with a basis for mid-course adjustments and program refinements, either at the end of a funding cycle, or in the course of implementation.

Tier 4 - Progress Toward Objectives

Using cost information in Tier 4 is closely tied to the program design issues of Tier 1, and the accountability issues of Tier 2. If appropriate program outcomes and indicators have been identified in Tier 1, and the appropriate unit cost information is included in the routine data that is collected as part of Tier 2, then the job of identifying progress toward objectives in Tier 4 becomes much easier.

Tier 5 - Program Impact

When it has been possible to conduct a full-scale cost-benefit analysis over a long period of time, and it shows significant long-term gains and cost savings in a particular population or problem area, the policy implications may be great. One of the best-known examples is the Perry Preschool Study (discussed earlier), which has been credited with persuading lawmakers to sustain or significantly increase their support for early intervention programs, including Head Start. It is widely believed that one reason that this study was so influential was the fact that it included a cost-benefit analysis. While monetary cost is not the only basis for policy decisions, it is usually a very salient one for voters and politicians.

ADVANTAGES OF USING COST ANALYSES:

• Promotes fiscal accountability in programs. Too often, program managers can't easily determine the cost of providing particular services or achieving certain outcomes, because they aren't systematically collecting the necessary data, either about clients or about costs. At the least, programs should be able to tell funders (or potential funders) that during a given time period, they provided a certain level of services to a specific number of clients, and at what cost (Jacobs, 1988).

• Helps set priorities when resources are limited. Program managers can use cost information in designing programs, and in budgeting and allocating funds to get the most out of their resources.



• Can be extremely powerful and persuasive to legislators, policy makers, and other funders. May help convince them to invest in particular kinds of programs. Some argue that this advantage of cost-benefit analysis may hold true even when it is not possible to assign monetary values to all program costs and outcomes; if the effect is strong enough, even a relatively incomplete cost-benefit analysis may be persuasive (Barnett, 1993).

DISADVANTAGES OF USING COST ANALYSES:

• Requires a great deal of technical skill and knowledge. A true cost-benefit analysis requires a solid grounding in economic theory and techniques, which is beyond the training of many evaluators. It may be necessary to hire a consultant if this type of analysis is desired.



• Critics feel that many cost analyses are overly simplistic, and suffer from serious conceptual and methodological inadequacies. There is a danger that an overly-simplistic cost-benefit analysis may set up an intervention to fail, by promoting expectations that are unrealistically high, and cannot really be achieved. This may result in political backlash which actually hurts future funding prospects instead of helping.

• There are no standard ways to assign dollar values to some qualitative goals, especially in social programs. For example, how do we value things like time, human lives saved, or quality of life?

• Market costs (what people actually pay for something) don't always reflect "real" social costs. For example, sometimes one person's cost is another person's benefit. Also, market costs don't necessarily reflect what economists call the "opportunity costs" of choosing to do one thing instead of another.
  

• Sometimes there are multiple competing goals, so we need to weight them or prioritize them in some way. If a program leads to improvement in one area, but more problems in another, is it still worth doing?

• Sometimes costs and monetary values are considered less important than other, more intangible values or program outcomes.
  
  
• The best-known cost-benefit studies have looked at long-term outcomes, but most program evaluations don't have the time or resources to conduct long-term follow-up studies.

HOW TO BUDGET & ALLOCATE COSTS FOR COST EFFECTIVENESS STUDIES:

The type of budgeting and accounting system your program or agency uses may well determine how much useful cost data is available for evaluating your program, or comparing it to others. Three major types of budgeting formats commonly used in social service programs will provide different types and amounts of information (Kettner, Moroney, & Martin, 1990).

The most common format is the Line-Item Budget format, which simply looks at revenues (money coming in from various sources, including grants, user fees or United Way funds) and expenditures (costs broken down into broad categories like salaries, rent, utilities, and postage), and tries to ensure that they balance. The main purpose of a line-item budget is financial control, and the categories are usually too broad to give much information about the cost of providing a particular service or obtaining a particular result.

The Functional Budget format starts with a line-item budget, and takes it a step further. It focuses on process, or the cost of providing a service. For example, with a Functional Budget, we could determine that it cost an adoption agency $45,000 to conduct 100 home studies (an activity which is a necessary part of the process of placing children in permanent homes).

The Program Budget, which also starts with a line-item budget, looks at the same information from the point of view of outcomes, or the cost of achieving a result. For example, if the 100 home studies resulted in actually placing 50 children in adoptive homes, the Program Budget would allow us to say that it cost the agency $45,000 to place 50 children, which is an outcome.

Another way to look at this is that functional budgets measure productivity, and program budgets measure the cost of achieving goals and objectives.

COMMON STEPS IN DEVELOPING PROGRAM & FUNCTIONAL BUDGETS (Kettner, et al., 1990):

1. Develop a line-item budget that shows all expenditures. This is the minimal level of budgeting and accounting that is required by many funders, such as the United Way. Some funders require a specific format, so that the categories used are standard across the programs that they fund.

2. Determine the agency's program structure. A distinct program is a set of activities or services designed to accomplish a specific set of agency goals and objectives. Many agencies have several different programs.

3. Identify all direct costs and indirect costs. Direct costs are those that benefit only one program (for example, salaries of staff who work only for one program, or supplies and equipment used only for that program). Indirect costs or "overhead" costs are those that benefit or are shared by more than one program (for example, several programs in an agency might share the same building, and be served by the same bookkeeping and secretarial staff, utilities, or janitorial services).

4. Assign direct costs to the appropriate program or project. This is usually fairly straightforward. If one county agent has full-time responsibility for operating your State Strengthening project, for example, then 100% of his or her salary and benefits would be assigned as an expense to that project in the budget. If a staff member spends 50% of his or her time on the State Strengthening project and 50% on another assignment, then half of that person's salary and benefits would be assigned to the State Strengthening project as a direct cost.

5. Allocate indirect costs to programs. Deciding how to divide up the indirect (shared) cost pool among several programs in the agency can be much more complicated and technical. The actual practice of allocating or dividing up the indirect costs is usually best left to an accountant. There are several methods for doing this, each with particular advantages and disadvantages (see Kettner, et al., 1990). Although cost allocation of indirect costs can be a time-consuming step, it is considered well worth doing because of the increased information it provides about the real costs of providing services.

6. Determine total program costs. The total cost of a particular program (such as your State Strengthening project) is the sum of the direct costs, and the portion of indirect costs that is allocated to that program.

Once we have this information about total program costs, then we can calculate unit costs. For a Functional Budget, this involves defining the units of service for each program (eg., hours of day care provided, meals delivered, home studies conducted), and calculating the cost per unit of service. In the adoption agency example above, the unit cost of conducting a home study would be $450 (total program cost divided by number of units of service provided). For a Program Budget, the final steps are determining the total cost of achieving the outcome objectives for the year, and calculating the cost per outcome. Using the adoption example again, we can say that the adoption agency described above successfully placed children in adoptive homes at a unit cost of $900/child (total program costs divided by the number of successful outcomes).

COST-EFFICIENCY & COST-EFFECTIVENESS STUDIES: From the point of view of program evaluation, both the Program and Functional Budgeting systems are more useful than a Line-Item Budget. Unit cost information allows for useful comparisons of the costs of delivering services and getting results. With this information, we can look at the unit cost of one adoption agency compared to another, to see whether one operates more efficiently. We can also compare the unit cost (per child) of adoptive placement to the unit cost (per child) of placement in foster care or residential treatment. This is basically what happens in a cost-effectiveness study.

In general, a cost-effectiveness study is more appropriate than a cost-benefit analysis when your goals or outcomes can't easily be quantified or monetized, or when there are multiple competing goals. As with budgeting and cost allocation, there are a variety of approaches to cost-effectiveness studies. The approach that is best for your purposes will depend on a number of factors. A good source of more detailed information about deciding what approach is most appropriate, and conducting the various types of cost-effectiveness studies, is Weimer & Vining (1992).

HOW TO CONDUCT A COST-BENEFIT ANALYSIS:

Cost-benefit analysis is by far the most complex and controversial of the three methods of costs analysis we have discussed. It should not be attempted by those who lack technical expertise in this area. However, for some purposes, it is also one of the most powerful methods. For those who decide to undertake a cost-benefit analysis in spite of the difficulties, Barnett (1993) outlines a nine step process. Various standard texts are recommended for more in-depth information (see below).

• Step 1: Define the Scope or Perspective of the Analysis - The first step is to describe the alternative(s) to be evaluated, and determine whose perspective will guide the evaluation. A narrow cost analysis might look only at the monetary costs and benefits to the individual participant or target of services, or to a particular funder or agency. A broader perspective might attempt to look at a wide range of costs and consequences (intended and unintended, direct and indirect) for society as a whole. A program that is not cost-effective from the perspective of a particular agency within its limited mission and budget may well be cost-effective from the perspective of society, because it saves expenses or prevents problems in other areas. Rossi and Freeman (1993) note that because different stakeholders may have different values and priorities, mixing different viewpoints is likely to result in "confused specifications and overlapping or double counting." Whether we like it or not, the perspective chosen for cost evaluation may have political implications. Therefore, while there are limitations to any one perspective, it is important for the evaluator to clearly state his or her position.



• Step 2: Conduct Cost Analysis - The next step is to identify and estimate the monetary value of all resources used in the intervention, not just the budgetary costs. Some costs, such as salaries of direct service staff, rental of office space, or program supplies, are obvious and simple to determine. Indirect costs of supervision and administration need to be included as well. Other resources and costs may go well beyond the items that are usually included in an agency budget. Sometimes "overhead" (like office space or supervision) is provided as an in-kind service by an existing agency, but since there are probably some additional demands made on the time of the agency staff, this should be figured into the "real" cost of the intervention (what would you have to pay if the time or space had not been donated)?. What about the value of the time of program volunteers (what would you have to pay them if they weren't volunteering their services)? Barnett argues that parent time is a frequently ignored cost factor in intervention programs for children; if a high level of parent involvement is required (for example, teaching the child, conducting physical therapy exercises, or attending many meetings and appointments), it may represent a cost savings to the agency, but a cost to the parent because there may be less time available for paid work, housework, spending time with other children in the family, or just the cuddling and nurturing aspects of being a parent.

• Step 3: Estimate Program Effects - This is where more traditional impact or outcome evaluation methods come in. As noted earlier, if we don't know that there is a significant beneficial effect of our program, there is little point in asking how much it costs to get the effect, or whether it is more cost-effective than another kind of program. Many texts on evaluation can assist you in designing a valid evaluation (Rossi & Freeman, 1993; State Strengthening Evaluation Guide, 1997). Often it is not possible to use a true experimental design in evaluating community-based programs, but there are a number of quasi-experimental designs available (Cook & Campbell, 1979). Also, don't forget that it is often possible to use existing data to estimate program effects, as well. If you are looking at an ongoing program, or one that is based on a national model (such as the Parents As Teachers program), check to see if formal evaluations have already been done elsewhere. You may also be able to get useful information from the program's service statistics, or from local, state, or federal census data [insert link here to Using Existing Data URL].



• Step 4: Estimate the Monetary Value of Outcomes - This is one of the most difficult and controversial aspects of conducting a cost-benefit analysis, and it may require the help of consultants. Some cost-savings are easier to estimate than others. For example, we may have data that the average cost of placing a child in residential treatment is $20,000 a year, so if we are able to prevent 20 children from being placed in residential treatment, the estimated savings is 20 X $20,000. However, other important outcomes may be much less obvious, and much harder to estimate.



• Step 5: Account for the Effects of Time - One of the trickiest and most technical aspects of cost-benefit analysis, especially for longitudinal studies that follow clients or outcomes over a period of years, is discounting of costs and calculating rates of return for alternative uses of the money (such as investing it). This includes taking into account the effects of inflation on the value of the dollar over time, or figuring the depreciation in the value of things like buildings and other capital equipment. Similar issues apply in estimating the value of benefits over a period of time. For example, if we want to look at the projected life-time earnings of a teenager who stays in school due to a drop-out prevention program compared to one who does not, we need to make projections about wages. If we want to look at whether the government will eventually recover its investment in the drop-out program through the taxes he or she will pay on the increased income, we need to make projections about future tax rates as well. These projections all require assumptions. Unless you or someone on the program staff has expertise in this area, it is strongly advised that you seek out a skilled consultant to help with this step.



• Step 6: Aggregate and Apply a Decision Rule - If you are looking at the costs and benefits on several outcomes (which is often the case), how will you decide which has priority? If a program for pregnant teenagers results in healthier babies (and lower hospital costs), but not in fewer repeat pregnancies, which outcome is more important?

• Step 7: Describe Distributional Consequences - This is related to choosing your perspective of analysis. It involves specifying who gains and who loses under different conditions (because in some cases, one party's benefit is another party's loss). This may be a highly controversial and political step in the process.



• Step 8: Conduct Sensitivity Analysis - This step involves identifying the assumptions behind your cost estimates, and considering how critical they are to your calculations. If one of your assumptions turns out not to be accurate, or if conditions change during the time of your study (for example, the minimum wage goes up, affecting salary costs), will that change your whole conclusion, or is the effect strong enough that there is some leeway?

• Step 9: Discuss the Qualitative Residual - Since there are almost always some things that can't be quantified or given monetary values, it is important that your report include some discussion of these issues. A frank description of some of these qualitative issues in your report can help round out your conclusions, and reduce the chances of your study being used inappropriately.

ANNOTATED BIBLIOGRAPHY

1. Barnett, W. S. (1993). The economic evaluation of home visiting programs. The Future of Children: Home Visiting, 3. Center for the Future of Children, the David and Lucile Packard Foundation, 93-112.

The Future of Children journal is available on-line. Some issues can be viewed or downloaded at the following World Wide Web site, and older hard copy issues can be ordered free of charge at: http://www.futureofchildren.org

Barnett is a strong advocate of more use of cost-benefit or cost-effectiveness analyses in evaluations of social programs. He has been involved in conducting several influential cost-benefit studies of home visiting and early intervention programs (including the well-known Perry Preschool Program study), and reviews several others. He argues that it is more feasible than most evaluators believe, and provides a relatively detailed discussion of steps and procedures.

2. Barnett, W.S. (1985). Benefit-cost analysis of the Perry Preschool program and its policy implications. Educational evaluation and policy analysis, 7. 333-342.

3. Berreuta-Clement, J. R., Schweinhart, L. J., Barnett, W. S. , et al. (1984). Changed lives: The effects of the Perry Preschool program on youths through age 19. Monographs of the High/Scope Educational Research Foundation, no. 8. Ypsilanti, MI: High/Scope Press.

One of the best-known and most influential longitudinal studies of the effects of early intervention programs. The influence of this study on funders and policy makers is often attributed to its use of a cost-benefit analysis to show that funding intensive programs for low-income preschoolers can be a good long-term investment for society.

4. Callor, S., Betts, S. C., Carter, R., & Marczak, M. (1997). State Strengthening Evaluation Guide. Tucson, AZ: USDA/CSREES & University of Arizona.

Program development and evaluation manual designed by the Evaluation Collaboration for use in State Strengthening Projects. Based on the five-tiered model of evaluation of Jacobs (1988). The manual may be viewed or downloaded from the World Wide Web at: ttp://ag.arizona.edu/sfcs/cyfernet/cyfar/evalgde.htm

5. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton-Mifflin Co.

This is a standard reference on quasiexperimental research designs, which are commonly used for evaluation of community-based programs.

6. Jacobs, F.H. (1988). The five-tiered approach to evaluation: Context and implementation. In H.B. Weiss, & F.H. Jacobs (Eds.), Evaluating family programs. New York: Aldyne de Gruyter.

This chapter describes a model of evaluation of community-based programs which served as the basis for the model used in the State Strengthening Evaluation Guide.

7. Kettner, P. M., Moroney, R. M., & Martin, L. L. (1990). Designing and managing programs: An effectiveness-based approach. Newbury Park, CA: Sage.

Useful discussion of the pros and cons of several budgeting systems (line-item, functional, and program budgeting) in terms of facilitating the calculation of unit costs. Emphasizes that cost allocation is time-consuming and requires careful choices about methodology. Concludes that despite the difficulties of setting up such a budgeting system, it is worth the time and effort for programs to be able to accurately determine true unit costs and program costs. Although the advantages are primarily discussed in terms of practical benefits for program managers, the ability to accurately allocate costs and value of services is also a prerequisite for cost-benefit analysis as a long-term evaluation strategy.

8. Mishan, E. J. (1988). Cost-benefit analysis (4th ed.). London: Unwin Hyman.

An economics textbook, and one of the standard references for cost-benefit analysis techniques. Although fairly technical and mathematically oriented, this book is intended to be accessible to serious non-economists who are willing to devote some time and effort to learning the theory and techniques. Includes a sometimes whimsical discussion of the problems of valuing intangibles, such as time.

9. Rossi, P. H., & Freeman, H. E. (1993). Evaluation: A systematic approach (5th ed.). Newbury Park: Sage Publications, pp. 363-401.

A standard evaluation text. Rossi and Freeman take the position that while "efficiency assessment" (their term for cost-benefit and cost-effectiveness analyses) is highly technical therefore not always feasible, all program evaluators should at least have some understanding of the basic concepts involved. This is because impact evaluations always implicitly involve issues of how much effort or cost is required to achieve a desired outcome.

10. Thompson, M. S. (1980). Benefit-cost analysis for program evaluation. Beverly Hills: Sage.

Another basic text, which elaborates on the quantitative aspects of benefit-cost (or cost-benefit) analysis. Includes in-depth discussions of issues of discounting and cost adjustments.

11. Weimer, D. L., & Vining, A. R. (1992). Policy analysis: Concepts and practice (2nd ed.). Englewood Cliffs, NJ: Prentice Hall.

Fairly technical discussion aimed at professional policy analysts; assumes a background in economics. Useful discussion and flow chart distinguishes five related approaches and when they are appropriate to use: Benefit-Cost Analysis, Qualitative Benefit-Cost Analysis, Modified Benefit-Cost Analysis, Cost-Effectiveness Analysis, and Multi-Goal Analysis. Appropriateness of each method is determined primarily by 1) whether the relevant impacts and inputs can be readily quantified and monetized, and 2) whether there are goals or benefits other than economic efficiency which need to be incorporated into the analysis.

12. White, K. R. (1988). Cost analyses in family support programs (pp. 429-443). In H. B. Weiss & F. H. Jacobs (Eds.), Evaluating family programs. New York: Aldyne de Gruyter.

Suggests use of general term "cost analyses". Basically takes a skeptical approach, noting that "present conceptualizations of cost analysis are overly simplistic, expectations are unrealistically high, and much of what is labeled as cost-effectiveness or cost-benefit research suffers from serious conceptual and methodological inadequacies" (p.429).

13. Zeckhauser, R. (1975). Procedures for valuing lives. Public Policy, 23(4), 419-464.

Serious discussion of philosophical, legal, and economic aspects of attempts to assign economic value to human lives that are saved, injured, or lost. More than most people may want to know, but a key ethical issue in cost-benefit analyses of many social and health related policies and programs.

Back to Alternative Methods website.