• Welcome
• About Us
• National Retrospective Evaluation of WAP
• ARRA-Period Evaluation
• SEP Evaluation
• EECBG Program Evaluation
• Past Evaluations
• NEAT/MHEA
• Publications
• Relevant Links
• Contact Us

National Retrospective Evaluation of the WAP

• Evaluation Overview
• Surveys & Data Forms
• News Highlights
• Important Links

SEP National Evaluation Description

INTRODUCTION

The State Energy Program (SEP) was created by Congress in 1996, when the State Energy Conservation Program and the Institutional Conservation Program were consolidated.  SEP provides financial and technical assistance for a wide variety of energy efficiency and renewable energy activities undertaken by the states and territories. These include several mandatory activities, such as establishing lighting efficiency standards for public buildings, and a broad range of optional activities such as providing energy audits and building retrofit services, holding workshops and training sessions, delivering technical assistance, supporting loan and grant programs, and encouraging the adoption of alternative energy technologies.

The U.S. Department of Energy’s (DOE’s) effort to perform a rigorous independent evaluation of SEP employing primary data began in 2007.  First, a ‘white paper’ was prepared by a group of nationally-recognized program evaluators that presented a recommended study approach.  A scope of work was then prepared by Oak Ridge National Laboratory (ORNL) based on the white paper. In addition to supporting the federal government’s needs for such an evaluation, the scope was informed by input on the states’ needs and priorities provided by a Network Committee of State Energy Office directors and SEP managers.  That draft work scope was reviewed by a Peer Review Panel of evaluation experts, following DOE’s recommended peer review procedures, and modifications were made to reflect the Panel’s recommendations.  Subsequently, a competitive solicitation was issued by ORNL and a team of independent evaluators, led by KEMA, Inc., was selected to perform the evaluation as described in the scope of work and further elaborated in a detailed study plan.  The detailed study plan addresses the information needs of multiple audiences including DOE management, the Office of Management and Budget (OMB), Congress, the states, and those at all levels that actually run the program.

EVALUATION OVERVIEW

As designed, the SEP National Evaluation focuses on four key outcomes: (1) energy, cost, and demand savings; (2) job creation; (3) renewable energy generation; and (4) carbon reduction. Outcomes will be quantified for program year (PY) 2008, the most recent year before the large increase in SEP funding that occurred under the American Recovery and Reinvestment Act (ARRA), and for the ARRA period (PY 2009 – 2011).

A sample of 82 programmatic activities will be selected for in-depth study to represent those program types receiving the greatest amount of SEP funding during each of the two study periods (PY 2008 and ARRA).  The term “programmatic activities” is used here to refer to a related set of activities (e.g., multiple energy audits) performed in a single year under a common administrative framework.  Typically, the programmatic activities designed and carried out by the states with SEP support involve a number of actions (e.g., multiple retrofits performed or loans given). 

The SEP National Evaluation will focus on those broad program types that cumulatively account for approximately 80% of SEP funding in each study period.  For PY 2008, that includes Building Codes and Standards; Building Retrofits; Clean Energy Policy Support; Loans, Grants and Incentives; Renewable Energy Market Development; and Technical Assistance.   For the ARRA period, the targeted program areas are Building Codes and Standards; Building Retrofits; Loans, Grants and Incentives: and Renewable Energy Market Development.

In order to make the most efficient use of the resources available, different broad program types and their various subcategories will be studied at different levels of rigor, depending on their importance and the nature of the activities undertaken. Tables 1 and 2 show the number of programmatic activities of each broad program type and subcategory that will be sampled at various rigor levels for PY 2008 and the ARRA period. High rigor (H) approaches typically employ a control group to produce reliable findings regarding what would have occurred in the absence of the program. Appropriate data collection methods include on-site metering and monitoring, surveys of participants and non-participants, the use of monthly billing data, examination of generation records for renewable energy facilities, and other standard methods.  Modeling approaches can also be used when calibrated with metering or other facility-specific data.  Medium high (MH) rigor methods include the use of engineering based, statistically adjusted engineering based, and modeling based approaches that are generally not as reliable as those efforts that employ on-site measurement.  In selected cases, findings can be taken from the evaluation literature for similar efforts and adjusted to reflect the climate and use conditions that are associated with the specific activities under study.

Table 1: Sampling Targets by Broad Program Area, Subcategory, and Rigor Level (PY 2008)

Broad Program Area	Subcategory	Rigor Level1	Number of PAs2 Sampled
Building Codes and Standards	Building Code Development and Support	MH	1
Building Codes and Standards	Generalized Workshops and Demonstrations	MH	3
Building Codes and Standards	Targeted Training and/or Certification	MH	2
Building Codes and Standards	Technical Assistance to Building Owners	MH	1
Building Retrofits	Building Retrofits: Nonresidential	H	2
Building Retrofits	Building Retrofits: Residential	H	2
Building Retrofits	Generalized Workshops and Demonstrations	MH	5
Building Retrofits	Technical Assistance to Building Owners	MH	6
Clean Energy Policy Support	Policy and Market Studies; Legislative Support	MH	8
Loans, Grants and Incentives (excluding Retrofits)	Alternative Fuels, Ride Share and Traffic Optimization	MH	5
Loans, Grants and Incentives (excluding Retrofits)	Technical Assistance to Building Owners	MH	3
Loans, Grants and Incentives (Retrofits Only)	Building Retrofits: Nonresidential	H	4
Loans, Grants and Incentives (Retrofits Only)	Building Retrofits: Residential	H	2
Renewable Energy Market Development	Generalized Workshops and Demonstrations	MH	6
Technical Assistance	Technical Assistance to Building Owners	MH	3
	TOTALS	Total	53
		MH	43
		H	10

¹ H = High rigor; MH = Medium-high rigor

² PAs = Programmatic Activities

 Table 2: Sampling Targets by Broad Program Area, Subcategory and Rigor Level (ARRA)

Broad Program Area	Subcategory	Rigor Level1	Number of PAs2 Sampled
Building Codes and Standards	Building Codes and Standards: Codes	MH	2
Building Codes and Standards	Generalized Workshops and Demonstrations	MH	1
Building Codes and Standards	Targeted Training and/or Certification	MH	1
Building Retrofits	Building Retrofits: Nonresidential	H	6
Building Retrofits	Building Retrofits: Residential	H	2
Loans, Grants and Incentives (excluding Retrofits and Projects)	Renewable Energy Market Development: Manufacturing	MH	2
Loans, Grants and Incentives (Retrofits and Projects)	Building Retrofits: Nonresidential	H	5
Loans, Grants and Incentives (Retrofits and Projects)	Building Retrofits: Residential	H	1
Loans, Grants and Incentives (Retrofits and Projects)	Renewable Energy Market Development: Projects	MH	4
Renewable Energy Market Development	Renewable Energy Market Development: Manufacturing	MH	1
Renewable Energy Market Development	Renewable Energy Market Development: Projects	MH	4
	TOTALS	Total	29
		MH	15
		H	14

¹ H = High rigor; MH = Medium-high rigor
² PAs = Programmatic Activities

Because it is very common for states’ SEP efforts to receive funding from multiple sources, steps will be taken to separate the impacts attributable to SEP support from those resulting from other inputs.  This determination of attribution will be performed separately for each individual programmatic activity studied.  The multi-step attribution approach employed in this study will use logic models, model validation, and a variety of established techniques to quantify SEP-generated effects.

To augment the findings from this study, the results from other evaluations implemented by the states or utility companies can be used if the methods employed are suitably rigorous and the activities in question are included in the sample selected independently for the national evaluation.

Interim reports will be available starting in late 2011 on the results of individual studies of specific SEP programmatic activities. Summary reports on the outcomes of each broad program type will be made available as each one is completed, beginning in early 2012.A final report on all outcomes for both study periods is expected around the end of 2012.

---

Contact: Martin Schweitzer is the ORNL Principal Investigator for this project.