Special & Cross-Sector Studies

Synopsis: This Phase 1 report presents findings from the Community Based Program (CBP) Design Effectiveness Study for the Massachusetts energy efficiency Program Administrators (PAs). This study had three overarching goals: (1) to identify and document the breadth of CBP designs and attributes nationally, (2) to explore the relative effectiveness of various community engagement strategies, and (3) to explore what factors help to explain why community engagement strategies are variably effective across contexts. Phase 1 addresses the first of these goals, comments on the second and third, and lays the groundwork for the development of a Phase 2 study to answer an additional question about the long-term effectiveness of CBPs.

Conclusion: Evaluators reviewed 25 CBPs tied to ten energy efficiency program administrators’ existing ratepayer-funded offerings over the past ten years. Among these, we documented a diversity of goals, approaches, activities, and outcomes. Evaluators also encountered numerous additional programs that fell slightly outside of this scope but which provide context to the world of energy efficiency CBPs. Notably, there are many local programs throughout the United States and elsewhere which draw on community attributes, community-based social marketing, or other CBP design elements but which do not entail significant partnership with ratepayer-funded programs. Reviews of those programs provide useful insight about how best to complete grassroots community campaigns, but we did not review them in detail given our focus on the utility-community nexus. Key findings related to the breadth of CBP designs and attributes, the relative effectiveness of various community engagement strategies, and what factors help to explain why community strategies have been variably effective across contexts are presented. Evaluators also provide several overarching recommendations to help foster excitement and engagement from local partners and implement a successful CBP.

Net-to-Gross Methodology Research

Synopsis: The goals of this study are to examine the general methods used in Massachusetts for estimating net savings and to compare the current methods to methods currently being used in other regions across the country or being considered for use in Massachusetts. Evaluators also document the strengths, weaknesses, and integration issues of the various net-to-gross (NTG) methods for different programs and markets in Massachusetts, including a general approach that can be used by evaluators to structure the combining of studies to determine best estimates of NTG. Two main activities were conducted as part of this study: 1. Review approaches to utilizing self-report information and potential adjustments to current practices in Massachusetts. 2. Develop a framework to integrate NTG or net savings results from diverse methods, including self-reports, market effects, and codes and standards evaluation approaches.

MA Top-Down Modeling Extended Methods Review

Synopsis: This memo presents the findings and recommendations from: (1) an extended literature review of top-down modeling methodologies applied in studies in both the energy and non-energy sectors, and (2) the insights and findings from in-depth interviews with nationally recognized experts in the field of econometrics, macroeconomics, and top-down modeling. Based upon the results of the literature review and expert interviews, we provide an assessment of the advantages and disadvantages of different theoretical methodologies and approaches, and provide recommendations for enhancements to future top-down efforts.

Conclusion: The literature review and interviews all support the concept that panel regressions are a fairly blunt tool that can be used to glean broad qualitative results, but may not provide definitive quantitative estimates. The literature and interviews suggest use of experimental design or natural experiments as being highly preferable as key concerns include the difficulty of determining the right model specification, and the limited explanatory variables available. Some of the experts interviewed, and the literature, emphasized the importance of incorporating available information on the structural drivers of the dependent variable and of the variables being controlled for–in our case, often physical relationships as well as savings estimates. Several suggested trying the analysis multiple ways and then explaining why one approach should be preferred. The specific methods that can be used for top-down analysis are nearly as broad as the field of econometrics itself. There is no instruction manual for procedures that can be followed. Most approaches that have been considered have some merit and may be the best for some circumstances. Nevertheless, there are some general guidelines that emerge for our top-down research, which are further discussed in the memo.

PY 2016 Mass Save Awareness Campaign Report

PY 2016 Mass Save Awareness Campaign Appendix A

Synopsis: In 2016, the Massachusetts energy efficiency Program Administrators (PAs) implemented the sixth year of a statewide marketing campaign, under the trademark of Mass Save®. This report presents findings from the November 2016 Mass Save statewide marketing assessment conducted by Opinion Dynamics. The purpose of this study is to document the 2016 campaign design and marketing strategy as they relate to assessing changes in key metrics over time and to provide updated measures of Mass Save awareness, knowledge, and
perceptions based on the 2016 campaign.

Conclusion: Key findings from the study include the following: The 2016 campaign was the most successful Mass Save awareness campaign since we began assessing brand awareness in 2012. There is additional potential to increase awareness of Mass Save same among Spanish-speaking respondents. A fair number of low-income respondents indicated being aware of Mass Save; however, there is still appreciable potential to increase awareness in this segment. Overall, the two additional population groups (low-income and Spanish-speaking) surveyed have healthy levels of awareness given that the campaign only recently started targeting those segments. In 2016, there were statistically significant increases in the level of familiarity with the brand. Awareness of the Mass Save website among residential customers has increased since December 2014. Campaign messaging was clear among residential and C&I customers. Mass Save messaging exposure increased among residential customers. Targeting a particular segment of the population (such as Spanish and Portuguese speakers and low income residents targeted in 2016) often takes multiple years to reach saturation. Consistency in messaging will be a successful means of increasing awareness and understanding of targeted populations. Mass Save should continue marketing through radio and retail, as well as increase marketing activities in digital channels.

2016

CCSI Circuit Rider Immediate Survey Memo

2015

Savings Evaluation Methodology for Codes and Standards Initiative

Comprehensive Review of Behavior and Education Programs

Synopsis: This report provides a comprehensive review of behavior and education (K-12) programs for consideration by the Massachusetts Program Administrators (PAs) and Energy Efficiency Advisory Council (EEAC) in future program planning. The evaluation team utilized in-depth and informational interviews in addition to a thorough review of secondary literature, focusing research efforts on recent changes in the landscape of behavior and education programs.

Conclusion: The review of behavior programs included (1) recent trends in residential behavior programs, (2) identifying program offerings implemented in small markets, including potential solutions to implementation barriers, and (3) small and medium commercial behavior programs. The review of education programs included both a review of K-12 programs, including kit-based and non-kit-based programs, as well as a review of savings calculation methodologies. The report summarizes key findings related to these topics. The report also identifies opportunities for consideration by the PAs and EEAC in future program planning.

2014 Massachusetts Statewide Marketing Campaign – Post-Campaign Report

Synopsis: During 2014, the Massachusetts-based energy efficiency Program Administrators (PAs) implemented the sixth year of a statewide marketing campaign, under the trademark of Mass Save®. This report presents findings from the Mass Save statewide marketing assessment conducted between December 5, 2014 and January 18, 2015. The purpose of this study was to document the 2014 campaign design and marketing strategy and to assess its impact on customer awareness of Mass Save, as well as its ability to deliver clear and recognizable messages. Evaluators reviewed all statewide marketing materials used in 2014. The evaluation team also fielded general population surveys of residential and C&I customers using a mixed-mode approach that included both telephone and web-based surveys.

Conclusion: Key findings from the study include: awareness of Mass Save has increased significantly since December 2013; customer awareness of MassSave.com and self-reported website usage increased in 2014; campaign messaging was clear and resonated with residential and commercial customers; self-reported exposure to Mass Save messaging increased significantly among residential and commercial customers; and depth of knowledge for program offerings is also increasing among residential and commercial customers.

Top-down Modeling Methods Study – Final Report

Synopsis: The work presented in this document is part of a multi-year initiative designed to assess the utility of top-down modeling as a viable technique for evaluating energy efficiency programs in Massachusetts. This document presents a summary of the Year 1 investigation into possible top-down methods for net impact evaluations, as a supplement to techniques currently used. Top-down models attempt to measure changes in energy consumption over time that are attributable to programmatic interventions by the utilities. The goal of this type of modeling is to isolate the effect of program activity from other natural changes and policy variables.

Conclusion: This Year 1 top-down research provided a number of key recommendations for conducting the next phase of pilot studies in Massachusetts. Evalutors summarized these recommendations as follows: continue refinement of the PA-Muni model to investigate the stability of models and possible changes to model specification that may reduce confidence intervals as outlined above; for the PA Data model, continue to collect data through the C&I database to extend the available data series to include five years of consumption and program tracking data, then continue collecting the necessary data going forward for future analysis; continue to refine the existing models to further explore approaches to weather normalization, industry segmentation, and inclusion of other key explanatory variables such as technology trends; and incorporate multiple lag periods of the program and consumption variables.

Synopsis: The objectives of this document were to identify sub-segments of the C&I lighting and controls market that are suitable for prospective or retrospective market effects studies and outline the recommended methods for conducting market effects studies of each of the identified markets. Based on previous experience, the evaluators identified criteria that any proposed market effects studies would need to meet in order to provide value to the PAs.

Conclusion: Applying the criteria developed for proposed market effects studies, the evaluators identified two potential studies: A retrospective assessment of the market effects of programs to support high efficiency linear fluorescent technology, with a primary focus on low-wattage T8 lamps, and a baseline (prospective) study of the commercial market for lighting controls. Section 2 of the report presents an analysis of the PAs’ program activity in regards to commercial lighting for the years 2011–2013. Section 3 presents the recommended methods for conducting a retrospective market effects study of the high-performance T8 market and a prospective study of the lighting controls market. Section 4 presents a preliminary set of market indicators for use in market effects assessments for programs that promote low-wattage T8s and commercial lighting controls.

Synopsis: The objective of this document was to outline appropriate methods for evaluating residential new construction programs’ effects on this market based on the range of methods available for measuring market effects. This includes methods for establishing qualitative evidence of the programs’ effects on markets and quantifying the effects, which incorporate spillover, as well as estimating net savings.

Conclusion: The body of this document outlines proposed methods for evaluating residential new construction programs’ effects on this market. The evaluators recommend two primary methodological components for the market effects studies proposed. The first is theory-based evaluation, described in Section 2. This qualitative approach identifies how program activities are expected to lead to market effects and measures the associated indicators periodically. The second component is quantifying market effects and the associated net energy savings. Appendix A of the document identifies and differentiates among the types and sources of savings that may stem from program efforts in the non-residential new construction market in Massachusetts. This document also identifies other studies that estimate net savings in Massachusetts, assesses whether or not these estimates overlap with a possible non-residential new construction market effects study, and, where there is overlap, recommends how it can be accounted for.

LED Market Effects – Baseline Characterization Final Draft

Synopsis: This report is the first draft report of a baseline characterization study of the market for LED lighting products in Massachusetts. This baseline characterization is the first phase of a two-phase study to determine the market effects resulting from LED lighting programs offered by the Program Administrators (PAs) in Massachusetts. The goal of this study was to develop a baseline of the current conditions of the market for LED products in Massachusetts and a selected comparison area to be used for future analysis of market effects resulting from PA-sponsored programs in Massachusetts. This study relied on a variety of primary data collection and analysis efforts conducted in Massachusetts and the comparison area.

Conclusion: Evaluators found that many aspects of the residential LED market are at a similar stage of development in both regions and the difference in measures of LED adoption between Massachusetts and the comparison areas is sufficiently small. Evaluators do not believe that they can link any observed differences in demographics to the relatively small differences in observed indicators of residential market development for LED lighting. For the non-residential markets, findings suggest that the Massachusetts market is more advanced than the comparison area as measured by vendor-reported sales and customer-reported installation of screw-in LED lamps, downlight fixtures, and outdoor fixtures. The differences in the reported LED share of equipment installed by contractors are in the same range. Differences in the portion of customers who report having at least one LED fixture installed are relatively small. However, 42 percent of Massachusetts customers report having at least one LED bulb installed versus only 13 percent in the comparison area.

Residential Lighting Cross-Sector Sales Research Memo

Synopsis: The goal of the cross‐sector lighting sales research was to review evaluations conducted in other jurisdictions on commercial purchases from lighting programs targeting the residential sector, and in turn to produce a recommended “placeholder” value on the proportion of these program lighting purchases that Massachusetts customers install in commercial settings. Secondary research into applicable studies formed the basis of the methodology.

Conclusion: The Evaluation Team recommends using a placeholder of 7% to be applied to the Massachusetts upstream lighting program sales as an indicator of those used in commercial settings. The 7% figure is the overall rounded average of all the studies considered in the review. All the studies under review included some form of bias, but by allowing all of the studies to have equal weight in the recommended average evaluators minimize the per‐study bias.

2014

Methods for Measuring Market Effects of Massachusetts Energy Efficiency Programs

Synopsis: This document is one of a series of work products addressing consistent methodologies for the measurement of market effects from cross-cutting energy efficiency programs run by the Massachusetts Program Administrators (PAs). It focuses specifically on the effects of three residential HVAC programs or subprograms and one commercial HVAC program on the markets for unitary HVAC equipment and controls and for residential Central Air Conditioning (CAC), gas heating, and ductless mini-split heat pumps. The objectives of this document are to outline appropriate methods for evaluating the programs’ effects on these markets based on the range of methods available for measuring market effects. This includes methods for establishing qualitative evidence of the programs’ effects on markets and quantifying the effects, which incorporate spillover, as well as estimating net savings.

Conclusion: The research presented in this docuement was conducted in stages. The body of this document, which was prepared in spring and early summer 2014, proposes that the PAs develop certain market effects data sources and described these sources. In summer and fall 2014, the study team conducted additional research to assess the prospects for the proposed data sources, as proposed in Section 3.3 of the main body of the document. The findings and recommendations from this additional research are reported in Appendix A

Cool Smart and Gas Networks Brand Assessment Final Report

Synopsis: This report presents findings from an integrated assessment of statewide energy efficiency brands in Massachusetts. Researchers conducted an assessment of contractor awareness and perceptions around two HVAC contractor-targeted brands – Cool SmartSM and GasNetworks® – and the state’s umbrella brand, Mass Save, which generally targets residential and commercial customers. To complement prior Mass Save studies that focused on end-use customers, this study aims to explore the relationships between the three brands and characterize their roles within the HVAC supply-side market. The study also has the goal of determining whether the presence of three co-existing brands contributes to any market confusion.

Conclusion: The study identified several key findings. Neither Cool Smart nor GasNetworks has a formal brand strategy. Marketing and outreach for the Cool Smart program has both a target audience and a distinct set of core messages while GasNetworks has relied on a much broader set of marketing messages to engage its target audience. Awareness of Cool Smart and GasNetworks is moderate to high while awareness of Mass Save® is almost universal. As expected, most contractors associate Cool Smart and GasNetworks with rebates for heating and cooling equipment, but associations with Mass Save are varied. There is no definitive relationship between awareness and attitudes toward Mass Save, GasNetworks, and Cool Smart. At present, there is no evidence to suggest that having coexisting brands is causing confusion in the market.

2013

2013 Statewide Marketing Campaign – Post Campaign Report

Synopsis: This report presents findings from the December 2013 Mass Save statewide marketing assessment conducted by Opinion Dynamics. The purpose of this study is to document the 2013 campaign design and strategy, as well as its impact on customer awareness of Mass Save, and its ability to deliver clear and recognizable messages. After the conclusion of the 2013 Statewide Marketing Campaign, the evaluation team conducted a review of all campaign marketing materials, as well as telephone surveys with a sample of residential and commercial customers in Massachusetts. The surveys were the fourth wave of a series of surveys conducted since February 2012.

Conclusion: Results of the study suggest that awareness of Mass Save continues to be moderate, though there was a significant increase among C&I customers. Awareness of the Mass Save website has increased significantly among both C&I and residential customers. Self-reported exposure to Mass Save messaging increased significantly among residential customers. Campaign messaging was clear and resonated with residential customers. Awareness of other statewide brands (GasNetworks and COOl SMART) remains low among both C&I and residential customers.

Northhampton Leading the Way and Power Pittsfield Initiatives – Evaluation Report

Synopsis: This report presents the results of the evaluation of the Powering Pittsfield and Northampton Leading the Way initiatives, also referred to as the Serrafix initiative. Both initiatives were a part of a larger EE2020 initiative led by Serrafix, a consulting group that offers energy efficiency, sustainability planning, and other services. In designing the initiative, Serrafix relied on two concepts to achieve success – “concierge service” and utilizing existing relationships with community leaders and business networks. The evaluation results presented in this report draw on a review of materials related to the initiatives, interviews with the initiatives’ stakeholders, and review and analysis of the tracking data made available to the Evaluation Team.

Conclusion: Despite extensive and multi-faceted marketing and outreach efforts, neither initiative achieved the set goals or deeper savings. Because of the difficulties reconciling participation data with energy savings, the evaluation team had difficulty accurately summarizing the initiatives’ impacts. Based on the analysis, however, Northampton leading the way initiative achieved 38% of the project goal, completing 24 out of the planned 63 projects. Powering Pittsfield initiative achieved 31% of the goals, completing 16% out of the planned 51 projects. The depth of savings analysis did not reveal either initiative achieving deeper savings. Evaluation research revealed that multiple factors might have prevented the initiatives from achieving their goals, including the following: Timing of ramp-up period; implementation status of the utility-administered energy efficiency programs upon which the two initiatives built; length of funding period; community selection; various ownership structures and split incentive barrier; stakeholder engagement; and planning and management challenges.

2013 Statewide Marketing Campaign – Pre-Campaign Snapshot Final Report

Synopsis: This report presents results from the pre-2013 marketing campaign statewide survey to document current levels of awareness of Mass Save against which to measure changes over time. The Team plans to conduct a post-campaign survey to gauge the immediate impact of the 2013 campaign and will use the data collected to further explore changes in awareness over time. Target audiences include both single- and multi-family residential customers, and C&I customers of all sizes.

Conclusion: Unaided awareness among residential customers remains moderate (36%) and has not changed since the post 2012 campaign survey. The percentage of residential customers, who consider themselves somewhat or very familiar with Mass Save, remains relatively low (19%). The top unaided Mass Save association is with energy assessments or audits (26%), followed by saving energy or environmental concerns in general (20%). However, when provided with a set of options, residential customers indicate that they most associate Mass Save with a “resource for energy efficiency information and programs” (43%). Just under half of residential (46%) customers aware of Mass Save identify utilities or energy efficiency service providers as sponsors. Customer awareness of energy efficiency programs in general is high. Almost three quarters of residential respondents (73%) know there are programs to help save energy in their home. Unaided awareness of Mass Save is moderate among C&I customers with 47% reporting that they have seen or heard the term before. This represents an increase since the last statewide marketing survey when awareness was 40%. Similar to residential respondents, the top un-aided Mass Save association among C&I customers is with energy assessments or audits (22%). Just over half of commercial customers (55%) aware of Mass Save identify utilities or energy efficiency service providers as sponsors. Half of C&I respondents (51%) know there are programs to help save energy in their business, and, as expected, owners are significantly more likely than those who lease their facility to know about programs (57% and 42%, respectively).

Cross-Cutting Behavioral Program Evaluation Final Integrated Report

Synopsis: This report serves as the third annual impact and process evaluation of the Massachusetts Behavior/Feedback Programs and Pilots. The purpose was to address a set of overarching researchable issues the PAs tasked the Evaluation Team to assess for all behavioral programs and pilots in Massachusetts. The researchable issues included determining energy savings impacts of these efforts, how they differ by program/pilot type, whether these programs/pilots lead to additional participation in other programs and the energy savings that are associated with cross-program participation.

Conclusion: The 2012 impacts for the NGRID and NSTAR behavioral programs range from 41 kWh to 258 kWh per household for the electric cohorts, and from 0.28 million British thermal units (MMBtus) to 1.90 MMBtus for the gas cohorts. Since their start in 2009, the NGRID and NSTAR behavioral programs have generated a total of 113,827 MWh and 710,255 MMBtus in energy savings. Since its start in 2010, the WMECo behavioral program has generated a total of 5,036 MWh in energy savings. The Massachusetts statewide goals are measured in lifetime benefits. In 2012, the NGRID, NSTAR, and WMECo behavior programs generated 65,582 megawatt hours (MWh) and 344,682 MMBtus in energy savings. As such, the 2012 program cycle savings achieved through the behavioral programs represent 3.7% of the total 2012 preliminary statewide lifetime electric savings goal and 2.4% of the total 2012 preliminary statewide gas lifetime savings goal. The 2012 program cycle savings achieved through the behavioral programs represent 23% of the total 2012 statewide annual electric savings goal and 31% of the total 2012 statewide annual gas savings goal in Massachusetts. Differences across PAs, described in more detail in the report, are largely a result of program investment, rather than a reflection of program effectiveness.

HVAC Replace on Failure Non-Energy Impacts Memo

Synopsis: This memo provides adjusted deemed Non-Energy Impacts (NEI) values that address the differences in NEIs for residential heating, cooling, and water heating equipment that is early replacement compared to replace on failure. NEIs considered include: thermal comfort, health benefits, property value increase, home durability, and noise reduction.

Conclusion: Evaluators took several steps to adjust Non-energy impact (NEI) estimates for residential heating, cooling, and water heating measures that replaced failed equipment. First, they used professional judgment and the NEI literature to attribute a portion of the NEIs to the measure’s “newness” and a portion to the measure being energy efficient. Second, using the attribution factors, they estimated the value of the portion of NEIs associated with the energy efficiency of the measure for systems that are replaced on failure. Then, using data from the current HEHE and Cool Smart evaluation, they determined the percentage of program participants that replaced failed systems and assigned the adjusted NEI values to these participants. Results for specific NEIs and measures are provided in tables and discussed in more detail in the memo.

2012 Massachusetts Statewide Marketing Campaign Evaluation Final Report

Synopsis: This report presents results from the post 2012 statewide umbrella marketing survey effort conducted by Opinion Dynamics. During 2012, the Massachusetts-based energy efficiency Program Administrators (PAs) implemented the fourth year of this marketing effort, under the trademark of Mass Save®. The goal of the research presented in this report is to document changes in awareness of Mass Save over time. To support the assessment of pre- and post-campaign awareness, Opinion Dynamics conducted telephone surveys with a sample of residential and commercial and industrial (C&I) target audiences.

Conclusion: Overall, the team found divergent results within the residential and C&I populations, with C&I customers showing greater changes in awareness and other metrics over time. These increases among businesses suggest that the ramp-up of Mass Save Statewide Campaign efforts targeted to C&I customers is beginning to have an effect. While awareness of Mass Save remains moderate among C&I customers, there has been a significant increase in Mass Save awareness among C&I customers over awareness prior to the 2012 campaign launch. There has also been a significant increase in the percentage of C&I customers who most associate Mass Save with a website and those who are aware of the site. Awareness of and familiarity with Mass Save have not changed significantly among residential PA customers since the pre-campaign survey nor since the 2010 baseline study. In contrast, there has been a significant increase in the percentage of residential respondents who mention Mass Save as an entity promoting energy efficiency after the 2012 campaign compared to before its launch.

Special & Cross-Sector Studies

2023

2022

2020

2019

2018

2017

2016

2015

2014

2013

Search by Tag

Search by Date Range

Stay Informed