State Energy Program (SEP) National Evaluation

INTRODUCTION

The State Energy Program (SEP) was created by Congress in 1996, when the State Energy Conservation Program and the Institutional Conservation Program were consolidated.  SEP provides financial and technical assistance for a wide variety of energy efficiency and renewable energy activities undertaken by the states and territories. These include several mandatory activities, such as establishing lighting efficiency standards for public buildings, and a broad range of optional activities such as providing energy audits and building retrofit services, holding workshops and training sessions, delivering technical assistance, supporting loan and grant programs, and encouraging the adoption of alternative energy technologies.

solar panels
The U.S. Department of Energy’s (DOE’s) effort to perform a rigorous independent evaluation of SEP employing primary data began in 2007.  First, a ‘white paper’ was prepared by a group of nationally-recognized program evaluators that presented a recommended study approach.  A scope of work was then prepared by Oak Ridge National Laboratory (ORNL) based on the white paper. In addition to supporting the federal government’s needs for such an evaluation, the scope was informed by input on the states’ needs and priorities provided by a Network Committee of State Energy Office directors and SEP managers.  That draft work scope was reviewed by a Peer Review Panel of evaluation experts, following DOE’s recommended peer review procedures, and modifications were made to reflect the Panel’s recommendations.  Subsequently, a competitive solicitation was issued by ORNL and a team of independent evaluators, led by DNV KEMA, was selected to perform the evaluation as described in the scope of work and further elaborated in a detailed study plan.  The detailed study plan addresses the information needs of multiple audiences including DOE management, the Office of Management and Budget (OMB), Congress, the states, and those at all levels that actually run the program.  A set of Frequently Asked Questions provides a succinct description of key elements of the SEP National Evaluation.

EVALUATION OVERVIEW

As designed, the SEP National Evaluation focuses on four key outcomes: (1) energy, cost, and demand savings; (2) job creation; (3) renewable energy generation; and (4) carbon reduction. Outcomes will be quantified for program year (PY) 2008, the most recent year before the large increase in SEP funding that occurred under the American Recovery and Reinvestment Act (ARRA), and for the ARRA period (PY 2009 – 2011).

clean fuels

A sample of 82 programmatic activities has been selected for in-depth study to represent those program types receiving the greatest amount of SEP funding during each of the two study periods (PY 2008 and ARRA).  In addition to the 82 “primary” sample points, a number of “secondary” sample points were selected to allow for substitutions in case any of the programmatic activities in the primary sample could not be evaluated due to lack of available respondents and data. The term “programmatic activities” is used here to refer to a related set of activities performed in a single year under a common administrative framework.  Typically, the programmatic activities designed and carried out by the states with SEP support involve a number of actions (e.g., multiple retrofits performed or loans given). 

The SEP National Evaluation focuses on those broad program types that cumulatively account for approximately 80% of SEP funding in each study period.  For PY 2008, that includes Building Codes and Standards; Building Retrofits; Clean Energy Policy Support; Loans, Grants and Incentives; Renewable Energy Market Development; and Technical Assistance.   For the ARRA period, the targeted program areas are Building Codes and Standards; Building Retrofits; Loans, Grants and Incentives; and Renewable Energy Market Development.

 

wind turbines

In order to make the most efficient use of the resources available, different broad program types and their various subcategories are being studied at different levels of rigor, depending on their importance and the nature of the activities undertaken. Tables 1 and 2 show the number of programmatic activities of each broad program type and subcategory being studied at various rigor levels for PY 2008 and the ARRA period. Appropriate high rigor (H) data collection methods include on-site metering and monitoring, surveys, examination of generation records for renewable energy facilities, and other standard methods.  Modeling approaches can also be used when calibrated with metering or other facility-specific data.  Medium high (MH) rigor methods include the use of engineering based, statistically adjusted engineering based, and modeling based approaches that are generally not as reliable as those efforts that employ on-site measurement.  In selected cases, findings can be taken from the evaluation literature for similar efforts and adjusted to reflect the climate and use conditions that are associated with the specific activities under study.  A set of 26 data collection instruments, tailored to different types of programmatic activities and respondents, has been approved by OMB for use in this study.

Table 1: Number of Programmatic Activities in Study by Broad Program Area, Subcategory, and Rigor Level (PY 2008)

Broad Program Area

Subcategory

Rigor Level1

Number of PAs2 Sampled

Building Codes and Standards

Building Code Development and Support

MH

1

Building Codes and Standards

Generalized Workshops and Demonstrations

MH

3

Building Codes and Standards

Targeted Training and/or Certification

MH

2

Building Codes and Standards

Technical Assistance to Building Owners

MH

1

Building Retrofits

Building Retrofits: Nonresidential

H

2

Building Retrofits

Building Retrofits: Residential

H

2

Building Retrofits

Generalized Workshops and Demonstrations

MH

5

Building Retrofits

Technical Assistance to Building Owners

MH

6

Clean Energy Policy Support

Policy and Market Studies; Legislative Support

MH

8

Loans, Grants and Incentives (excluding Retrofits)

Alternative Fuels, Ride Share and Traffic Optimization

MH

5

Loans, Grants and Incentives (excluding Retrofits)

Technical Assistance to Building Owners

MH

3

Loans, Grants and Incentives (Retrofits Only)

Building Retrofits: Nonresidential

H

4

Loans, Grants and Incentives (Retrofits Only)

Building Retrofits: Residential

H

2

Renewable Energy Market Development

Generalized Workshops and Demonstrations

MH

6

Technical Assistance

Technical Assistance to Building Owners

MH

3

 

 

 

 

 

TOTALS

Total

53

 

 

MH

43

 

 

H

10

1 H = High rigor; MH = Medium-high rigor
2 PAs = Programmatic Activities

 Table 2: Number of Programmatic Activities in Study by Broad Program Area, Subcategory and Rigor Level (ARRA Period)

Broad Program Area

Subcategory

Rigor Level1

Number of PAs2 Sampled

Building Codes and Standards

Building Codes and Standards: Codes

MH

2

Building Codes and Standards

Generalized Workshops and Demonstrations

MH

1

Building Codes and Standards

Targeted Training and/or Certification

MH

1

Building Retrofits

Building Retrofits: Nonresidential

H

6

Building Retrofits

Building Retrofits: Residential

H

2

Loans, Grants and Incentives (excluding Retrofits and Projects)

Renewable Energy Market Development: Manufacturing

MH

2

Loans, Grants and Incentives (Retrofits and Projects)

Building Retrofits: Nonresidential

H

5

Loans, Grants and Incentives (Retrofits and Projects)

Building Retrofits: Residential

H

1

Loans, Grants and Incentives (Retrofits and Projects)

Renewable Energy Market Development: Projects

MH

4

Renewable Energy Market Development

Renewable Energy Market Development: Manufacturing

MH

1

Renewable Energy Market Development

Renewable Energy Market Development: Projects

MH

4

 

 

 

 

 

TOTALS

Total

29

 

 

MH

15

 

 

H

14

1 H = High rigor; MH = Medium-high rigor
2 PAs = Programmatic Activities

Because it is very common for states’ SEP efforts to receive funding from multiple sources, steps will be taken to separate the impacts attributable to SEP support from those resulting from other inputs.  This determination of attribution will be performed separately for each individual programmatic activity studied.  The multi-step attribution approach employed in this study will use logic models, model validation, and a variety of established techniques to quantify SEP-generated effects.

PV on houses

To augment the findings from this study, the results from other evaluations implemented by the states or utility companies can be used if the methods employed are suitably rigorous and the activities in question are included in the sample selected independently for the national evaluation.

Full-scale data collection for the SEP National Evaluation is scheduled to be completed in July, 2013, and data analysis for all 82 programmatic activities under study will be finished the following month.  All internal draft reports on broad program area findings are expected to be completed in November, 2013, and the job creation analysis will be done in January 2014.  A draft report on all findings will be ready for external review in February, 2014, with the final report expected in April of that year.

Contact: Martin Schweitzer is the ORNL Principal Investigator for this project.