Evaluating energy conservation outreach programs: A case study in Minnesota

Erwryy Vol. 5, pp. I 169-l 177 Pergamon Press Ltd 19X0. Prmted in Great Britain

EVALUATING ENERGY CONSERVATION OUTREACH PROGRAMS: A CASE STUDY IN MINNESOTA? ERIC HIRST Energy Division, Oak Ridge National Laboratory, Oak Ridge, TN 37830, U.S.A. (Received 6 May 1980)

Abstract-The Minnesota Energy Agency and the Department of Administration sponsored workshops for boiler operators in 1979. The workshops, attended by engineers from state-owned facilities and from private buisnesses, dealt with improvements to fuel/air ratios for large gas and oil-fired boilers. The process and costs of developing and running these workshops are documented. Two different surveys of workshop attendees were conducted, one directly after the workshops and the second two months later. The first survey asked attendees about their reactions to the course (instructor, manual, and visuals) and their intentions to make the fuel/air ratio changes that were discussed during the workshops. The responses were remarkably positive, with 82% of the attendees rating the probability of making these changes as either excellent or good. The followup telephone survey sought to determine how many attendees had actually made these changes in the two months following the workshops. Again, the results were quite positive: 79% of the respondents said that they had made the adjustments to some or all of their boilers. Altogether 60% of the boilers operated by these respondents had been adjusted. If these responses accurately reflect the changes made because of the workshops, then the State of Minnesota saved 388,000 in fuel bills during 1979. This is three times the cost of running the two March workshops (including the value of the attendees’ time). This suggests that this workshop-and by implication, others tike it-are an effective use of State resources.

1. INTRODUCTION

During the past several years, governments at all levels have become active in the development and delivery of energy conservation outreach programs. These activities are designed to motivate and inform particular groups, thereby helping them to improve the efficiency with which they use energy. Examples of these activities include on-site and computerized home energy audits; publications dealing with methods to reduce energy use in homes, commercial buildings, and industrial processes; and development of educational activities such as workshops and courses. The U.S. Department of Energy estimates the federal energy conservation outreach budget at $270 million for fiscal year 1980.’ Both the size and importance of these outreach programs dictate careful efforts to evaluate them. Unfortunately, such evaluations are the exception rather than the rule. As a recent DOE report put it: “There has not been sufficient emphasis on program evaluation to determine the impact on client behavior and energy savings”.’ One particularly significant evaluation involves the Energy Extension Service pilot program.’ The pilot program involved ten states, which operated programs beginning in Fall 1977. DOE organized and conducted an evaluation of two programs in each of the ten pilot states. This paper presents our findings concerning the usefulness to participants and the consequent energy savings of a set of workshops run by the Minnesota state government. The workshops were sponsored by the Department of Administration (DOA) and the Minnesota Energy Agency (MEA); they dealt with improvements to fuel/air ratio for large gas and oil-fired boilers. Attendees at the two workshops held in March 1979 were boiler operators at facilities owned by the state (e.g. hospitals, universities, community tResearch sponsored by the Office of Conservation and Solar Energy, U.S. Department contract with the Union Carbide Corporation. EGY. 5’1?-* 1169

of Energy under

1170

ERIC HIRST

colleges, prisons). Attendees at the five workshops held in May 1979 were boiler operators from commercial and industrial establishments. Altogether, 280 people attended the seven workshops. At the end of each workshop, attendees were asked to complete a simple 21-question survey to determine their thoughts on the content of the course, the slides and transparencies used during the workshops, the manual prepared for the workshop, the instructor, their interest in attending other workshops related to boiler operation, and their intentions to make the conservation improvements discussed in the workshop. In addition to these multiple choice questions, attendees were asked to offer other comments concerning the workshops and other courses they might wish to attend. Of the 280 attendees, 242 (86%) completed the evaluation form. In May, about half the March workshop attendees were contacted by telephone. Those contacted were from 82% of the institutions represented at the two workshops. The major purpose of this followup telephone survey was to determine how many attendees had actually adjusted fuel/air ratios at how many boilers since attending the workshop. In addition, attendees were asked about problems in making these adjustments and their reactions to the workshop and the manual. As Table 1 shows, the 75 attendees at the March workshops came from a variety of state-owned facilities. The Department of Administration maintains a detailed data base that includes monthly fuel consumption records for 42 large state complexes (which account for about two-thirds of the energy consumed in all state-owned buildings). Approximately half of the total floorspace in these complexes was represented at the workshops. Annual fossil fuel use at the 19 complexes represented was 1.9 trillion Btu during the 1977/1978 fiscal year. The fossil fuel bill for these 19 complexes for this period was about $6 million. Table

I. Major

state-owned facilities and representation boiler operator workshops.

at the March

1979

In DOA data base Represented at workshops Number of million Number of million Number of facilities ft2 facilities' attendees ft2 Prisons Hospitals Community Colleges State Universities Capitol Complex

4 6 5 4 A 19

Totals

8 14 6 J?

1.4 3.5 0.4 7.0 - 0

6 .L

2.6 5.9 2.5 9.2 L 16

36

12.3

42

21.8

a

7 ;:

"An additional 14 facilities (39 attendees) were represented at the workshops. These include smaller buildings operated by other state agencies.

The next section deals briefly with the development and organization of the two workshops. In Section 3, we discuss the initial evaluation questionnaire and the results obtained from analysis of attendee responses. Section 4 concerns the followup telephone survey, conducted two months after the workshops. Section 5 contains additional evaluations of these workshops, using the detailed data base on fuel consumption maintained by DOA. In the last section, we present conclusions concerning the value of these workshops. 2.

DEVELOPMENT

OF

BOILER

WORKSHOPS

Careful control of fuel/air mixture3s4 (and therefore of excess air) can improve overall boiler efficiency by an average of 335%. These large potential energy savings were the initial impetus for staff in MEA to begin work, in August 1978, on development and organization of workshops to train boiler operators how to adjust fuel/air ratios. Agency staff felt that materials developed and experience gained in conducting workshops for state employees could be readily used to conduct workshops for boiler operators in the private sector (i.e. commercial and industrial buildings). At the same time, energy conser-

Evaluating energy conservation outreach programs

1171

vation staff in DOA were running a study on the use of wood pellets in boilers at a state university, Measurements taken on the university boilers during this study showed that the amount of combustion air was much too high. These developments led to discussions between MEA and DOA. DOA had an ongoing program to collect monthly fuel-consumption data from major state complexes (Table l), to fund detailed engineering audits of these complexes, and then to implement those energy conservation measures with an expected payback period of five years or less. Thus, DOA had good contacts with engineers and boiler operators at large stateowned facilities. DOA suggested that a local contractor be asked to prepare a manual and visuals for the proposed workshops and to conduct the workshops. This contractor was selected because of his previous and ongoing work with the Department to improve operations at the Capitol Complex power plant (which serves 1.6 x lo6 ft’ of state office buildings in St. Paul). A draft manual, “Techniques for Improving the Efficiency of Gas & Oil Fired Boilers,” was prepared by the contractor. During the Fall of 1978, the manual was reviewed and revised based on discussions among the contractor and MEA and DOA staff. In addition, a contract was signed with a writer to edit the manual to ensure that the final product was not only technically correct but also readable. Considerable time and attention were devoted to determining appropriate levels of sophistication and writing style so that the manual would both interest and inform boiler operators. The manual was intended to be a practical guide for boiler operators, a “how to do it” resource. Discussions also dealt with the number, type, and purpose of drawings, graphs, and examples throughout the text. Slides and viewgraphs were developed for the workshops. The slides were taken at three plants in Minnesota to show the physical arrangement, control devices and adjustments, and flames at actual boilers. The slides were taken by MEA staff with subsequent editing by the instructor and MEA staff. The viewgraphs were taken from illustrations in the boiler manual. In January 1979, DOA sent invitations to personnel at state facilities, including hospitals, universities, community colleges, prisons, and those operated by other state agencies. Two sets of letters were sent. One was from the Commissioner of Administration to his peers in other government agencies urging them to encourage participation in the workshops. The second was from the head of the DOA Office of Energy Conservation to the chief engineers and power plant operators at each institution. This dual mailing worked well; both workshops were filled to capacity. Two workshops were held in March and five were held in May. A total of 280 people attended the workshops. Each workshop included ten hours of classroom instruction and discussion, covering the following topics: introduction to combustion efficiency, major factors affecting boiler efficiency, basic mechanical adjustments, control theory and adjustments, and combustion control. Table 2 shows estimates of workshop costs. The Table includes two sets of figures: the first is based on actual expenses for the March 1979 workshops and includes many costs that would not apply to a continuing series. The second set estimates the cost for workshops assuming that the development work has already been completed (e.g. the May workshops).? The total cost to develop and conduct the two March 1979 workshops was $9200, equivalent to about $120/attendee. For additional workshops (for which only the marginal cost is relevant), the cost would be sharply reduced to about DO/attendee. MEA sponsored five similar workshops in May 1979. These workshops were held at Area Vocational Technical Institutes (AVTI) in the Twin Cities (two sessions), Mankato,

tTable 2 does not include the cost of attendees’ time to attend the workshops. The salary. fringe benefits, and overhead associated with this time amounts to roughly $150 per attendee per day.

1172

ERIC HIRST Table 2. Estimated costs to conduct boiler operator workshops March 1979 workshops Manuala g Slides and transp renciesa Conduct workshops OOA and MEA timeC Workshop sites$aff

$2,200 800

Totals Cost

per

attendeee

May 1979 workshops $

300

600 5,000 600

600 1,000 600

$9,200

$2,500

$

B

120

30

aContractor costs to write, edit, and print the manual and to prepare and develop the visuals. The estimated marginal cost assumes a 10% addition for improvements to the manual and visuals. b

Instructor's

fee of

$300/workshop:

'Staff time to design the program, select contractors, review contractor work, select a site to hold the workshops, invite attendees,and other activities related to the workshops. Approximately 200 hours of state staff time were involved with the March 1979 workshops. We assume an The marginal average cost (including all overhead items) of $25/hour. cost assumes that the manual and visuals are essentially complete; most of the time is then devoted to site selection, instructor selection, inviting atendees, and other mechanics associated with running the workshops d

Cost costs

eBased

of meals (two lunches) and coffee for attendees plus administrative borne by the Area Vocational Institute ($7.50 per attendee). on 75 attendees

for

two workshops

Duluth, and St. Cloud. Attendees, from the private sector, were charged $25 to help recover costs. These workshops used the same instructor, manual, and visuals as those used in the March 1979 workshops; however they were only one day long.

3. INITIAL

EVALUATION

At the end of each workshop, attendees were asked to complete a simple questionnaire to solicit their views on the workshop. The questionnaire (see Ref. 5) included 21 questions dealing with overall reaction to the workshop, slides and transparencies, manual, instructor, likely outcome of workshop, interest in attending other workshops, 1 and general comments. The Appendix of Ref. 5 shows tabulations of the responses to each question. Two questions asked about the visuals; more than 80% of the respondents rated the visual materials either excellent or good. Three questions were asked on the manual; 90% of the respondents rated the manual excellent or good. Finally, three questions asked about the instructor; more than 95% of the respondents rated the instructor excellent or good. These results show very positive reactions to the workshop as a whole and to its three major ingredients (manuals, visuals, and instructor). The instructor was the most highly rated of the three components and the visual materials the lowest rated. However, all three elements were highly rated. Two other questions dealt with the overall rating of the workshop, in terms of its coverage of important topics and the usefulness of information presented during the workshop. Eighty-seven per cent of the attendees rated the workshop excellent or good on these two measures. This 87% E/G rating is consistent with the individual ratings on the manual, visuals, and instructor. Two questions were designed to probe the effectiveness of the workshops in motivating boiler operators and in providing sufficient information to them so that they could make the recommended changes to improve boiler efficiency. A third question asked about the likelihood that they would make these changes. Almost all the respondents (94%) said the encouragement had been excellent or good and 78% said that the information content was excellent or good. Finally, 82% said that the “probability that [they]

1173

Evaluating energy conservation outreach programs

will make these changes” was excellent or good. This 820/, is remarkably high; even if only a small fraction of these respondents actually made the changes, the workshop would be cost-effective. The three questions discussed above, on information, encouragement and likelihood of change, represent the desired outcomes from the course. We examined the relationships between these (dependent) variables and the course materials (independent variable+-visuals, manual, instructor-to see which particular aspects of the course were most influential in determining outcomes. These relationships are presented in Table 3, in terms of Spearman’s rank correlation coefficient (r,).6 (The measure rs indicates the amount of correlation between two different variables. A high degree of positive correlation yields an I, close to + 1.0, a high degree of negative correlation yields an I, close to - 1.0, and values near 0 suggest very little correlation between the two variables). Table 3. Spearman rank correlation coefficients among workshop responses.

Encourage

Effectiveness of workshop: Make changes Information

Visual materials slides transparencies

O.lC? 0.22b

0.46a 0.54a

0.25a 0.40a

Manual content writing use as reference

0.24a 0.28a 0.14

0.33a 0.4aa 0.25b

0.37a 0.42a 0.38a

0.70a 0.40a 0.40a

0.2Ba 0.15 0.39a

0.06 0.30a

Instructor communication enthusiasm courtesy

aSignificant

at

manual

skills

5X level;

bsignificant

at 10"

level

The results of Table 3 show that each element of the course has a statistically significant influence on the outcomes of the course as measured by motivation, information, and probability of making changes. The instructor seems to be most influential with respect to motivation of workshop attendees (first column of Table 3). The visuals seem to be most influential with respect to delivery of sufficient information to make the changes, with the manual a close second. Finally, the manual appears to be the most important ingredient of the course to influence the probability of actually making the changes. The questionnaire also asked attendees to offer “other suggestions for improving the present workshop or for other training methods to improve efficiency of boiler operations”. Of the respondents, 40% wrote comments.’ The most frequently stated comment is that the workshop dealt primarily with large boilers. However, some of the attendees operate small facilities, for which much of the advice was irrelevant. A related comment is that they do not have the instrumentation needed to monitor excess air (0,, COa, or flue gas temperature monitors). Attendees were asked whether they would like to attend other workshops related to boiler operation and maintenance, specifically on preventive maintenance, water treatment, and soot removal. Of the respondents, 76% rated their interest in attending a preventive maintenance workshop excellent or good, 65% so rated water treatment, and 50% so rated soot removal. These responses, plus those listed voluntarily (Ref. 5) suggest that there is considerable interest on the part of these boiler operators for additional information to improve the performance of the systems they operate.

4. FOLLOWUP

EVALUATION

Results presented in the preceding section suggest that the boiler operator workshops were a success. However, this conclusion is based solely on attendee responses to a very

ERICHIRST

1174

simple questionnaire administered immediately after each workshop.’ It is unclear whether or not the manual and the information presented at the workshops will actually be used by the boiler operators to improve efficiency of boiler energy use. To obtain additional and more reliable feedback on the effects of these workshops, we contacted 30 March workshop attendees by telephone in May. A total of 27 interviews was completed with engineers from 27 of the 33 institutions represented at the workshops. Attendees from the other three institutions were on vacation or otherwise unreachable. Three interviews could not be completed because the respondents were too busy to talk with us. The questionnaire used in this followup survey is shown in Ref. 5. Basically, we asked whether or not fuel/air ratio adjustments had been made since attending the workshop and, if so, how many boilers had been adjusted, whether the workshop had been influential in deciding to make these adjustments, and whether any problems were encountered in making these changes. Other questions dealt with past efforts and likely future efforts to adjust fuel/air ratios, and more recent thoughts about the workshop and its components. Twenty-two of the 27 respondents (79%) said they made fuel/air ratio adjustments on some or all of their boilers; see Table 4. The 27 respondents are responsible for operation of 80 boilers; adjustments had been made on 48 (60%) of these. The 79% who actually made adjustments to some of their boilers is quite close to the 82% who said (at the end of the workshop in March) that they would make these changes. Table 4. Results of May 1979 followup telephone survey. Adjustment Made since workshops People contacted (total = 27)

22 yes 5 no

Boilers operated (total = 80)

48 yes 32 no

to fuel/air ratio Will be done later in 1979 10 yes 4 yes

Only 7 of the 22 respondents who made adjustments stated that they could see an. improvement in the efficiency of their boilers. Most said that they couldn’t tell (i.e. were unsure of the effect). Nine of the respondents reported having some problems in making these adjustments. Problems included operation of the boiler at low load, inadequate instrumentation, and smoking of the boiler. All 22 of the respondents who made these changes credited the workshop as a positive influence on their decision to make the adjustments. However, all 22 said that they had made such adjustments in the past; the most recent such adjustment (before the March workshops) was generally in fall 1978. Several respondents explained this apparent inconsistency by saying that, although they had known before the workshop about fuel/air ratios and had made such changes in the past, the workshop provided additional information and motivation to make such changes. Fourteen of the respondents, including four of the five who had not yet made fuel/air ratio adjustments, said that they planned to make such adjustments later this year.? The general intention was to make adjustments during the fall, when boilers are running near capacity. In response to our questions about the instructor, manual, and visual, the comments were as positive as they had been in the immediate evaluation. Twenty-four of the respondents said that they had used the manual since the workshop and had found it helpful. In many cases, copies had been made for other engineers at the institution who had not attended the workshop. ‘tone respondentsuggested that future workshops be held in the fall so that operators could make adjustments when the boilers were running at full load.

1175

Evaluating energy conservation outreach programs

The results obtained from the followup telephone survey can be used to estimate the benefits of conducting the March boiler operator workshops. Based on the fuel use figures maintained by DOA for some of the facilities represented at the workshops (see Table l), we assume that annual fossil fuel use in the boilers represented at the workshops amounts to 2.5 trillion Btu, with an annual cost to the State of $7.3 million. If fuel/air ratio adjustments were made to 60% of these boilers because of the workshops and if the effect of the workshops is felt for only one year, then the State will save $88,000 in fuel costs during 1979. This saving must be compared with the cost of the workshop, which amounts to $9000 (see Table 2) plus the value of the attendees’ time, which amounts to an additional $18,000.~ Thus, the benefit/cost ratio for these two workshops is about three.

5. ANALYSIS

OF

FUEL

CONSUMPTION

RECORDS

The Department of Administration collects monthly energy use and cost information from 42 large state-owned complexes; see Table 1.8 These data have been collected since January 1972. Because 19 of the 42 complexes were represented at the March workshops, a careful analysis of monthly fuel consumption before and after the workshops for the control (no attendees) and experimental groups may show a significant change in boiler fuel use after the workshop.9 If the workshops have an effect on boiler operations, the monthly fuel consumption data should reveal the kind of relationships shown schematically in Fig. 1. Here, the pre-workshop trends in fuel consumption are the same for both sets of complexes. However, the trend line for complexes that had operators at the workshops falls below the line for the non-attendee complexes beginning in March 1979. Unfortunately, we were unable to conduct such an analysis. In early 1979, DOA changed computer systems. Rather than use the large computer at the University of Minnesota, they decided to use a mini-computer in their offices. Thus, we were unable to obtain fuel consumption data for these large state-owned facilities for any time after December 1978. In addition, our analysis of the January, 1972 to December, 1978 fuel consumption data revealed many problems that would complicate use of this data to evaluate effectiveness of these workshops.8 The coefficient of variation for monthly fuel use (measured in Btu/ft’) across all complexes and months (3181 observations altogether) was 91%. Even tFifteen of the 75 attendees at the workshops were not boiler engineers and therefore not directly responsible for operation of these boilers.

MARCH

‘79

TIME

-W

Fig. I. Possible outcome of analysis of DOA data on fuel consumption complexes.

at 41 large state-owned

1176

ERIC HIRST

III

0 JAN

III

ill JUNE

II DEC

Fig. 2. Monthly energy use at a Minnesota community college shows large variations.

after adjusting for variations in heating degree days, the coefficient of variation was 79% (for Btu/ft2-HDD). The large variance in the data (see Fig. 2 for an example of the data from one complex) would make it extremely difficult to detect a saving of 2-3x. The large variance is due to many factors. These include: weather (cloud cover, wind direction and speed, humidity, heating and cooling degree days), occupancy (number of students, patients, or prisoners, and stall), special activities (swimming pool, gymnasium, laundry, cafeteria, computer), presence and type of air conditioning system, additions and demolitions at the facility, and adoption of energy conservation practices and measures. We constructed regression equations to explain variations in monthly fuel use as functions of these independent variables. ’ Our best equation had an R2 of 0.60. This implies that 40% of the variation in monthly fuel use is not explained by the factors included, a variation that is much larger than the energy savings expected from attendance at a boiler operator workshop.

6. CONCLUSIONS

This paper described the development and conduct of workshops held in March and May 1979 for boiler operators at state-owned and commercial facilities. The purpose of the workshops was to educate and motivate operators to adjust fuel/air ratios at their boilers to increase overall efficiency. A key ingredient in the success of these workshops was the instructor. The instructor chosen had considerable experience working on boilers similar in size and operation to those represented at the workshops. In addition, the instructor spent a great deal of time with boiler operators in similar institutions working on methods to improve efficiency. His practical experience and knowledge of the people and situations involved gave the workshops tremendous credibility and empathy that they would not have had if the instructor had experience that was less relevant. These judgements on the instructor are amply borne out by the responses in the surveys. Another important element in the success of these workshops was input from Energy Agency and DOA staff. DOA staff provided reviews of the material from the perspective of boiler operators--their interests, reading level, and knowledge. In addition, DOA had prior relationships with the chief engineers and physical plant supervisors at state-owned institutions that lent credibility to the invitations to attend the workshops. MEA staff provided engineering review of the workshop materials and editorial review. Staff in the Information & Education Activity of the Agency worked closely with the instructor in preparing the manual and in preparing the visuals used during the workshops.

Evaluating energy conservation outreach programs

1177

Thus, several inputs from State government were important: engineering, education, practical experience, and a prior working relationship with potential workshop attendees. The followup telephone survey showed that the manual developed for the workshop had been used after the workshop by almost all the respondents, suggesting that it met the needs and interests of these boiler engineers. Results obtained from a simple questionnaire completed by 242 attendees were quite favorable. The workshop visuals, manual, and instructor were all rated excellent or good by SO-95% of the attendees. In addition, 82% rated the probability of their implementing the recommended changes either excellent or good. A followup telephone survey generally confirmed the positive findings of the immediate evaluation. Of the 27 people with whom interviews were completed, 22 reported making changes in fuel/air ratios after attending the workshops. Looking at the responses in another way, adjustments had been made on 48 of the 80 boilers represented by the 27 attendees contacted. If these adjustments produce a 2% reduction in fuel use and fuel costs for a one-year period, the savings to the State of Minnesota is almost $90,000. This is roughly three times the cost of the workshop (including the value of the attendees’ time), yielding a simple benefit/cost ratio of three. The benefit/cost ratio of the May workshops is likely to be much higher because no startup costs were associated with these workshops. Our initial evaluation plan called for analysis of actual fuel consumption records from the facilities represented at the March workshops and from a group of similar facilities that were not represented at the workshops. This might have yielded an unobtrusive measure’ to use in evaluation, one that is particularly important for energy conservation programs. Unfortunately, we could not obtain data for 1979. Even if we had obtained 1979 data, we would probably have been unable to detect a statistically-significant reduction in fuel use. This is because the variation in monthly fuel consumption (both across time and across facilities) is so much larger than the energy savings likely from adjusting fuel/air ratios. Our participation in, and review of, other energy conservation program evaluations that involve analysis of fuel bills support these findings.lOp’ ’ Most outreach programs can be expected to have only a small effect on actual energy consumption, an effect much smaller than the normal month-to-month variation observed. Nevertheless, because of its crucial importance to evaluation of such programs, we recommexid that additional effort be devoted to collection and careful analysis of fuel consumption records.

REFERENCES of Energy, Comprehensive Program & Plan for Federal Energy Education, Extension & Information Actiuities, Washington, D.C., Jan. 1980. 2. U.S. Department of Energy, Energy Exrension Seroice Pilot Program Eualuation Report: The First Year. I. U.S. Department

Vol. I, II, and III, DOE/CS 2671, Washington, DC., Sept. 1979. 3. U.S. Department of Energy, “Combustion of Fossil Fuels and Industrial Wastes”, in Instructionsfor Energy Auditors, Vol. 1, Chap. 5 DOE/C.%0041/12, Washington, D.C., Sept. 1978. 4. S. Gardner, “Adjusting Fuel/Air Ratios”, Plant Engineering, Barrington, Illinois 14 Sept. 1979. 5. E. Hirst, Evaluation of Boiler Operator Workshops in Minnesota, Minnesota Energy Agency, St. Paul, Minnesota, June 1979. Also E. Hirst, Eunluation of Fiue Boiler Operator Workshops Held in Minnesota During May 1979, Minnesota Energy Agency, St. Paul, Minnesota, Sept. 1979. 6. George W. Snedecor and William G. Cochran, Stutisticnl Methods, 6th Edn. Iowa State University Press, Ames, Iowa (1967). 7. Carol H. Weiss, Eualuntion Research, Methods for Assessing Program Efictiueness. Prentice-Hall, Engelwood Cliffs, New Jersey (1972). 8. E. Hirst, C. Eastes, and R. Tyler, Analysis of Energy Use in Minnesota Stare-Owned Facilities, Minnesota Energy Agency, St. Paul, Minnesota, Dec. 1979. 9. E. J. Webb, D. T. Campbell, R. D. Schwartz, and L. Sechrest, Unobtrusiue Measures: Nonreactive Research in the Social Science. Rand McNally & Company, Chicago, Illinois, (1966). 10. A. K. Meier, “Measuring Residential Energy Conservation”, Changing Energy Use Futures (Edited by R. A. Fazzolare and C. B. Smith), Vol. I. Pergamon Press, Oxford (1979). Il. E. Hirst, R. Maier, and M. Patton, Evaluation of Telephone Energy Conseruarion Information Centers in Minnesota. Oak Ridge National Laboratory, Oak Ridge. Tennessee, April 1980.