General Studies and Presentations

This presentation provides an update of the current status and an overall assessment of EM&V efforts in Massachusetts. The presentation provides further detail on results from several recent studies including: Lighting Market Assessment Study, Lighting Market Lift Study, Efficient Neighborhoods + Study, Residential Code Compliance Study, Minisplit Participant Survey, Umbrella Marketing Study, C&I End-User General Population Study, C&I Customer Profile Study.

The Focus in 2014 has been on implementing the 2013-2014 EM&V plan delivered last October. There are 53 studies in progress (14 in the planning stage, 32 in implementation, and in final reporting.) Strengths of EM&V efforts have included the huge volume of work that is underway, bolstering of stakeholder outreach and engagement, and a collaborative framework that is working well.  One weakness is that study implementation and reporting delays are increasingly common. Ramped up PA staffing levels present expanded opportunities, while non-consensus on ex-post savings claimed by the PAs in 2013 Annual Reports may be a challenge going forward. Conclusions from specific study results are also including in the presentation.

This briefing presentation provides a status report of EM&V efforts in Massachusetts.

The PowerPoint lists EM&V studies recently completed or near completion in Massachusetts and some of the lessons learned from these studies. These studies include: Residential New Construction Net-to-Gross Study, Upstream Lighting Impact Evaluation, C&I Mid-Sized Customer study, Residential Hours of Use Study, and Umbrella Marketing End-User Tracking Study. The PowerPoint also describes some anticipated 2014 results bearing on program design and implementation.

In an effort to better document the strategic evaluation plan, and have an evaluation plan that is more aligned with the energy efficiency program planning period, the PAs and EEAC Consultants have put together the following document to describe the evaluation planning process and describe studies, both generally planned, and specifically in progress.

For the 2013-2015 program period, the PAs and the EEAC consultants have agreed to simplify the existing system of research areas, reducing the total number of areas to three: Residential, Non-Residential, and Cross-Cutting. The Residential (including low-income) area combines three different research areas from the  2010-2012 program period: Residential Products, Residential Retrofit, and Residential New Construction. The Non-Residential area combines two different areas from the 2010-2012 program period: Large C&I/New Construction, and Small C&I. The Cross-Cutting area remains largely the same as in 2010-2012. The plan also describes oversight and responsibility of the EM&V process, types of evaluation functions, regulatory considerations, available budget, assigned staff, a discussion of planning principles, stages of evaluation, research completed since 2010, research in progress, and near term priorities and planned research.

This PowerPoint presentation describes EM&V in Massachusetts in the past, present, and future.

The presentation discusses how EM&V, including impact evaluation, process evaluation, and market assessment,  is used in Massachusetts. It also describes the EM&V framework in Massachusetts, the EM&V planning process  to date, the current EM&V planning process, and what EM&V has accomplished since 2010. Additionally, the presentation provides a status report of studies in progress, indicating that approximately 40 studies have been underway in 2013. Specific non-residential and residential evaluations are discussed. Last, the presentation suggests EM&V priorities for the future.  These include continuing to build the market assessment component of the EM&V program, developing a top-down NTG research program, continuing to move toward market-based NTG methods, and strengthening and systematizing the collection of sales data. Planning process in-progress changes and future planning process changes are also listed.

This memo provides insights into post-secondary education programs administered around the country. The goal of this research was to identify post-secondary education programs that quantify energy savings and claim these savings in regulatory filings.

The majority of evaluation reports related to post-secondary energy education programs focus on Building Operator Certification (BOC) programs. Among BOC programs, kWh savings range from 0.04 kWh to 0.72 kWh per square foot. For those BOC programs that calculated it, gas savings ranged from less than 0.01 to 0.02 therms per square foot. Among educational programs that do not focus exclusively on building operators, program administrators determine measure-specific savings and annual savings due to actions taken as a result of course participation. In general, the studies reviewed found that a large percentage of savings associated with measure installations were removed when rebates were factored into the analysis.  Based on the post-secondary energy education programs presented in this memo, evaluators highlight several best practices.

The goal of this research effort was to identify K-12 energy education programs that quantify energy savings and claim these savings in regulatory filings. As part of this effort, Opinion Dynamics conducted a literature review of K-12 energy education evaluations around the country and identified six that quantified savings (two of which claimed these savings).

Overall, few K-12 education programs quantify energy savings and those that do, tend to give students take-home energy saving measures such as CFLs or make energy saving improvements to school buildings that have quantifiable energy savings (such as changing lighting controls). The authors identified six programs  outside MA that quantified energy savings, three of which claimed these savings as part of their portfolio. Two of the three that claim savings provide kits to students while the third  coordinates with the schools to conduct a school-based project that saves energy. The savings for the kit based programs ranged from approximately 300 kWh to 500 kWh per student. Only the California K-12 Energy Education program (CA E3), attempted to quantify savings from behavioral efforts, but savings from these efforts were not claimed.  Savings from this program were calculated by asking students about what actions they took in the home after receiving the curriculum. Savings ranged from 400 kWh per student to 1,400 kWh per student and were most often derived from students installing CFL bulbs, turning off lights when not in a room, and reducing appliance use in the home.

This report presents findings from  the evaluation team’s interviews with residential and C&I customers to understand the pre-campaign levels of awareness of Mass Save. The report also builds upon an interim evaluation of the 2010 Massachusetts Umbrella Mass Save Statewide Marketing Campaign, which focused on documenting the campaign’s organizational structure and initial strategy.

Baseline results suggest that over one-third (39%) of residential customers have seen or heard the term Mass Save, with awareness of the MassSave.com website being lower at 17%.  22% of all residential respondents reported being exposed to at least one source of information about Mass Save in the previous 6 months. Most (84%) of residential respondents had a favorable opinion of  Mass Save after it was described. In general, residential respondents had little knowledge of organizations promoting energy efficiency. Many residential respondents also associated Mass Save with utilities and energy efficiency service providers. Overall, commercial customers had slightly lower levels of Mass Save awareness.

The primary objective of the study was to assist the Massachusetts PAs in quantifying the net impacts of their commercial and industrial natural gas energy efficiency programs by estimating the extent of program free-ridership, participant “like” and “unlike” spillover, and nonparticipant “like” spill over. To accomplish this, telephone surveys were conducted with 2011 program participants in each of the PA’s C&I natural gas programs and with design professionals and equipment vendors involved in these installations.

The report includes statewide figures for the custom and prescriptive programs and different measure types. The statewide free-ridership rate is 30.5 percent, the participant spillover “like” rate is 8.5 percent, and the nonparticipant spillover rate is 0.7 percent, resulting in a statewide net-to-gross rate (NTGR) of 78.7 percent. For measure types combined across all PAs and programs, the insulation measure type has the lowest level of free-ridership (7.8 percent) while the controls measure type has the highest free-ridership rate (38.8 percent). Within the custom programs, insulation measures have the lowest free-ridership and also the largest attributable participant like spillover. Within prescriptive programs, water saving measures has the lowest free-ridership rate(14.0 percent) while controls have the highest like spillover rate (16.1 percent). The report also provides detailed results for each PA, which include free-ridership and spillover rates by measure and program type, along with corresponding error margins.

This report is a process review of the effectiveness of Community Based Partnerships (CBP) developed to educate and promote energy efficiency behaviors among target segments. The report draws on a review of the data made available to the evaluation team, interviews with stakeholders, and participant research.  It is designed to document the status of the efforts, report findings and provide guidance to stakeholders based on the research conducted, and facilitate additional discussion to gather more complete tracking data.

Based on a number of insights related to CBP implementation practices and process, the evaluators developed several recommendations for future efforts. These recommendations included determining the goals of each community-based effort upfront, being strategic with selection of communities, understanding the targeted population and barriers to achieving goals, establishing metrics before launching the effort and tracking metrics consistently, considering most efficient and cost-effective delivery structure that would align with the effort’s goals, and requiring that all costs and resources required for support be clearly documented and tracked.

The goal of this study was to provide a comprehensive set of statistically reliable non-energy impact (NEI) estimates across the range of C&I retrofit programs offered by the Massachusetts electric and gas PAs. To achieve this goal, the evaluation team conducted a large scale in-depth interview (IDI) effort with a sufficient sample to provide statistically significant NEI estimates across prescriptive and custom electric and gas measure groups. Data was collected on NEI types and dollar values as well as like and unlike spillover.

The evaluation team captured NEI information for 789 prescriptive and custom electric and gas measures. Positive NEIs or non-energy benefits were realized for 58% of measures, while 3% of measures resulted in negative NEIs. An additional 40% of measures reported no positive or negative NEIs. The analysis identified the presence of NEIs resulting from energy efficiency programs, providing statistically significant NEI estimates and found a strong and statistically significant correlation between NEIs and program savings for prescriptive electric, custom electric, and custom gas measures. They also found a statistically significant, though weaker, correlation between NEIs and savings for prescriptive gas measures. Evaluators examined differences in attribution rates between participants who realized NEIs and those who did not report NEIs; however, the analysis did not provide conclusive evidence that NEIs and attribution were correlated. Results suggested that Massachusetts energy efficiency programs did result in substantial unlike spillover. Between 10% and 25% of measures resulted in some type of energy efficiency measure being installed without program support.

This memo discusses Low-Income Non-Energy Impacts (NEIs) that were not included in the NMR study submitted to the PAs’ 2012 Mid-Term Modification filing.  It was decided to add or revise the value used for the following four NEIs in the Low-Income Programs: refrigerator recycling, lighting quality, price hedging, and economic development.  The group has also agreed to perform a study of health NEI study which will be incorporated into the programs once the research is complete.

The group determined that for refrigerator recycling, environmental benefits derived from properly collecting, destroying, or recycling the materials contained within refrigerators should be included in the Low-Income Programs for consistency. The total benefit is comprised of 3 parts: $1.06 for avoided landfill space, $1.25 for recycling of plastics and glass, and $170.22 for incineration of insulating foam. The group also agreed that lighting quality should be included as a benefit and that $56 per LI participant is a reliable estimate of lighting quality NEIs and will use that in lieu of the $3.00 and $3.50 per bulb and fixture, respectively. The group determined that there is a value in minimizing program participants’ exposure to price increases and pointed to a paper that found a hedge value of $0.76/MMBTU of gas and $0.005/kWh. The group determined that there is an economic development benefit with respect to low-income programs that should be factored into cost effectiveness screening, as long as it complies with applicable regulatory and legal precedent and calculated economic development values of $0.486 per therm and $0.04 per kWh.