DISCLAIMER

The contents of this report are the views of the contractor and do not necessarily reflect the views or policies of the U.S. Department of Housing and Urban Development or the U.S. Government.

Review of Energy

Performance Contracts in

Public Housing

Submitted by:

Anna E. Bower

Michael E. Canes

Stuart D. Funk

Jyothsna Prabhakaran

LMI

Amy Deora

CIVIS Analytics

Rob Hazelton

Dominion Due Diligence Group

Foreword

I am pleased to present this report on a national evaluation conducted by the U.S.

Department of Housing and Urban Development’s (HUD’s) Of ice of Policy Development and Research. The study assesses the ef ectiveness and value of HUD’s Energy Performance Contracting Program administered by the Of ice of Public and Indian Housing (PIH). Energy Performance Contracts (EPCs) are a tool HUD makes available to Public Housing Authorities (PHAs). Over three decades, EPCs have enabled 250 PHAs, which include approximately 250,000 units, to secure private-sector financing to upgrade the energy efficiency of their housing stock by using the energy (and water) savings to pay for the improvements.

The findings of this report have two main implications for more effective policy. First, the EPC program remains a valuable tool for PHAs that seek to maintain their properties within the public housing program. It is complemented by additional repositioning tools that HUD has made available through the Rental Assistance Demonstration (RAD) and more recently with reforms to Section 18 (Demolition and/or Disposition) and Section 22

(Voluntary Conversion to Tenant-Based Assistance). Second, the EPC program continues to be less utilized by smaller PHAs, and further attention wil be critical to enable these smaller PHAs to enhance their utility cost savings. HUD can support small PHAs by expanding the Moving to Work Program and improving access to utility partnerships, as described in the recently published PIH Notice 2018-20. Congress can also support small PHAs by passing the Economic Growth Act that wil eventually allow small and rural PHAs to optionally freeze their utility baselines.

The quantitative components of the evaluation found that PHAs having implemented EPCs reduced their electrical consumption more than twice as much as PHAs that have not. Researchers could not produce a similar analysis for other types of utilities, however, because of data-quality limitations in operating subsidy utility expense-level data. On an annual basis, HUD uses PHA-submitted data as well as Measurement & Verification (M&V) reports, which HUD requires to evaluate individual EPC performance and savings achievement to ensure proper provision of EPC incentives. These analyses provide substantial detail on individual EPC projects but do not allow for comparisons with other PHAs that did not pursue an EPC. Enhanced data collection policies and activities for utility expense-level data are needed to better measure the full benefits of EPCs when comparing EPC-using PHAs to non-EPC-using PHAs.

EPCs involve highly complex activities that have significant lead-time requirements and require partnerships with third parties. Safeguarding the public interest requires PIH’s Field Of ices and Energy Team to exercise due diligence when analyzing a potential EPC project to ensure that the PHA wil remain financially solvent after implementation.

This in-depth analysis may deter participation by some Energy Service Companies (ESCOs) or PHAs. The study found that an important reason for PHAs declining to execute EPCs is the time and technical dif iculty associated with the EPC process, despite significant training having been provided by HUD in the past. The EPC process is further burdensome for small PHAs that may lack the staff or expertise to fully manage the program. EPC participation across the public housing portfolio lags for small PHAs.

Nearly 50 percent of large and medium PHAs have used EPCs, but only 7.9 percent of all PHAs and only 3.7 percent of small PHAs have participated. EPC is HUD’s only ii

Foreword

program specifically focused on energy, although tools such as RAD allow for the leveraging of significant resources that many PHAs are using to achieve energy savings.

HUD is also aware that multiple other structural factors may constrain small PHAs from using EPCs. One such barrier is that ESCOs operate using a profit-driven business model, and the scale and complexity of EPCs for small PHAs may not provide them with sufficient profit potential or the ability to mitigate risk. Another factor is that third-party financiers may not find it advantageous to investigate and lend funds to small PHAs.

Other options for pursuing energy savings outside of EPCs may be more suitable for small PHAs.

The study also shows that the adoption of private-sector financing and conversion of rental assistance models through RAD has increased exponentially in recent years. Although RAD (and now, the Section 18 and Section 22 alternatives) of ers an ability to complete energy and utility efficiency improvements as part of broader project renovations, not al PHAs are interested in or are able to use these tools. HUD views EPCs as a mechanism that can continue to provide such PHAs with a means of tapping into private financing for sorely needed capital improvements. For this reason, the EPC program remains a valuable financing mechanism in HUD’s toolbox.

Overall, this evaluation il ustrates that energy conservation ef orts within HUD’s public housing stock will require policies that encourage and motivate PHAs to sustain partnerships with private capital financiers and the companies that deliver energy services—whether through EPCs, RAD, or other pathways. Through continued and new ef orts, more PHAs will be able to participate in these energy savings activities, resulting in bet er housing for the communities they serve.

Seth D. Appleton

Assistant Secretary for Policy Development and Research

U.S. Department of Housing and Urban Development

iii

Contents

Executive Summary ................................................................................................ ES-1

Background and Method ...................................................................................... ES-1

Principal Findings ................................................................................................ ES-1

Recommendations ............................................................................................... ES-3

Chapter 1 Introduction ...............................................................................................1-1

Chapter 2 Background ...............................................................................................2-1

Chapter 3 Survey Sample ..........................................................................................3-1

Research Population ..............................................................................................3-1

Sample Design and Process ...................................................................................3-1

Chapter 4 Survey Implementation .............................................................................4-1

Survey Response Rate ...........................................................................................4-1

Corrections to the Data ...........................................................................................4-3

Blank or Incorrect Responses ...........................................................................4-3

Discrepancies in Respondent Group Category Designations ............................4-3

Implications for Survey Response Analysis.............................................................4-5

Adjustment of Sample Sizes ...................................................................................4-5

Chapter 5 Survey Analysis ........................................................................................5-1

Statistical Weighting ...............................................................................................5-1

Statistical Hypotheses Testing—Comparing Group 1 with Group 3 ........................5-2

Development of Ratings and Averages .............................................................5-2

Calculation of Weighted Average Scores ..........................................................5-2

Hypotheses Testing ..........................................................................................5-4

Within Group 1 Dif erences—Units Subject to an EPC vs. Units Not Subject

to an EPC .........................................................................................................5-5

Survey Evidence Concerning Why PHAs Did Not Undertake EPCs ........................5-7

How to Improve HUD’s EPC Process .....................................................................5-8

Chapter 6 Telephone Follow-up Interviews ..............................................................6-1

Introduction .............................................................................................................6-1

Representation by Group and Size .........................................................................6-1

Survey Results .......................................................................................................6-1

Interviewee Recommendations ...............................................................................6-3

Chapter 7 Analysis of Utility, Financial, and Physical Conditions Data .................7-1

iv

Contents

Background ............................................................................................................7-1

Analytic Approach ...................................................................................................7-1

Analysis of UEL Data ........................................................................................7-1

Analysis of Long-Term Electricity Consumption Trends ....................................7-2

Analysis of Short-Term Electricity Consumption Trends ....................................7-6

Analysis of Financial and Physical Condition ....................................................7-7

Chapter 8 Relationship of the RAD Program to EPCs .............................................8-1

Background ............................................................................................................8-1

PHA Options ...........................................................................................................8-2

Gains in Energy Efficiency from EPCs and RAD .....................................................8-2

Implications.............................................................................................................8-3

Evidence .................................................................................................................8-4

Qualitative: Stakeholder Views..........................................................................8-4

Quantitative: EPC Approvals .............................................................................8-5

Chapter 9 Conclusions and Recommendations .......................................................9-1

Conclusions ............................................................................................................9-1

Recommendations ..................................................................................................9-2

Appendix A Responses to Survey Questions ......................................................... A-1

Appendix B Summary Notes from Survey Fol ow-up Telephone Interviews ........ B-1

Appendix C UEL Data Analysis ................................................................................ C-1

Appendix D Maps of Regions and of PHA Populations .......................................... D-1

Appendix E Abbreviations ........................................................................................ E-1

References ................................................................................................................. R-1

v

List of Exhibits

Exhibit 3-1. Mapping of HUD Regions to Census Regions ......................................3-1

Exhibit 3-2. Group Population Counts .....................................................................3-3

Exhibit 3-3. EPC Sampling Strategy .......................................................................3-4

Exhibit 3-4. Non-EPC Sampling Strategy ................................................................3-4

Exhibit 4-1. Full and Partial Response Rate ............................................................4-2

Exhibit 4-2. Completed Survey Response Rate ......................................................4-2

Exhibit 4-3. Survey Responses by Region ..............................................................4-3

Exhibit 4-4. PHA Classifies Itself as Group 1 Whereas HUD Information

Classifies It as Group 2 or 3 .................................................................4-4

Exhibit 4-5. HUD Data Classifies PHA as Group 1 Whereas PHA Classifies

Itself as Group 2 or 3 ...........................................................................4-4

Exhibit 4-6. HUD and PHA Disagree on Whether the PHA is in Group 2 or 3 .........4-4

Exhibit 4-7. Final Numbers of Survey Responses by Group and PHA Size .............4-6

Exhibit 5-1. Survey Questions Relating to Utility, Financial, and Physical

Condition .............................................................................................5-2

Exhibit 5-2. Ratings At ached to Answers to Utility-Related Questions ....................5-2

Exhibit 5-3. Weighted Averages, Standard Deviations, and Standard Errors ..........5-3

Exhibit 5-3. Weighted Averages, Standard Deviations, and Standard Errors ..........5-4

Exhibit 5-4. Group 1 and Group 3 Dif erences in Average Utility Scores

(* indicates statistical significance at the 95-percent level) ...................5-4

Exhibit 5-5. Differences Between Average Financial and Physical Ratings

(* indicates statistical significance at the 95-percent level) ...................5-5

Exhibit 5-6. Percentage of Group 1 Respondents Who Would Do

Another EPC ........................................................................................5-5

Exhibit 5-7. Survey Responses on Utility Performance at EPC vs. Non-EPC

Units ....................................................................................................5-6

Exhibit 5-8. Survey Responses on Financial Condition at EPC vs. Non-EPC

Unitsa ...................................................................................................5-6

Exhibit 5-9. Survey Responses on Physical Condition at EPC vs. Non-EPC

Unitsa ...................................................................................................5-6

Exhibit 5-10. Why PHAs Chose Not to Do an EPC .................................................5-7

Exhibit 5-11. Why PHAs Began but Did Not Complete the EPC Process ................5-8

Exhibit 5-12. Ways to Improve HUD’s EPC Process Suggested by PHAs ..............5-8

Exhibit 6-1. Breakdown of Interview Sample by Group and PHA Size.....................6-1

Exhibit 7-1. Example Data Set and Calculated Averages for FL041 ........................7-2

Exhibit 7-2. Summary of Changes in Electricity Consumption Between EPC

PHAs and Non-EPC PHAs ...................................................................7-6

vi

List of Exhibits

Exhibit 7-3. Pre-, Post-, and Non-EPC PASS, MASS, and FASS Scores ................7-9

Exhibit 7-4. Available Funds Over Time to EPC and Non-EPC PHAs ................... 7-10

Exhibit 8-1. Numbers of RAD Housing Units Authorized by Congress,

by Year ................................................................................................8-1

Exhibit 8-2. Numbers of PHAs Executing EPCs, Converting to RAD or

Doing Botha ..........................................................................................8-2

Exhibit 8-3. EPC Approvals by Year .......................................................................8-5

Exhibit C-1. Data Modifications Made .................................................................... C-2

Exhibit C-2. Results of Statistical Analysis ............................................................. C-4

vii

Executive Summary

Background and Method

The U.S. Department of Housing and Urban Development (HUD) seeks to assess the effectiveness and value of its Energy Performance Contract (EPC) program by comparing trends in the utility consumption and the financial and physical condition of public housing authorities (PHAs) that have implemented EPCs with those that have not.

It also wants to discover the extent to which implementation factors affect the rate at which EPCs are used by PHAs. HUD contracted with a team headed by LMI (the LMI team) to obtain the comparative and implementation assessments. 1

The study followed four lines of investigation.

1. Survey: An online survey was sent to a sample of several hundred PHAs. Using survey data, the team tested whether the PHAs that used EPCs outperformed those that did not with respect to reducing utilities and improving financial and physical condition.

2. Telephone interviews: The team interviewed 20 of the responding PHAs in more depth concerning their experiences with EPCs or improving their utility consumption without the use of this program.

3. Data analysis: Information from various HUD databases about PHA utility consumption and financial and physical condition was used to corroborate survey responses and statistically test the efficacy of EPCs in improving public housing utility consumption and financial and physical condition.

4. RAD impact: The interaction of HUD’s Rental Assistance Demonstration (RAD) program with the EPC program was assessed. The number of housing units in the RAD program and the number of EPC applications were compared to see to what extent the former may have replaced the latter.

Principal Findings

After analyzing the data obtained in the steps previously described, the LMI team reached the following conclusions:

EPC Savings

1. Statistical tests performed using HUD Utility Expense Level (UEL) data showed that PHAs that have implemented EPC projects have been able to cut their electricity consumption by more than twice as much as PHAs that have not (10.3 percent as compared with 4.4 percent). This conclusion is backed at a 95 percent level of statistical confidence.

2. Tests using UEL data for EPC users showed electricity consumption declined at a significantly greater rate in the years after an EPC was performed than in the years immediately before. This result held at a 95 percent level of 1 The LMI team includes LMI, which headed the team, CIVIS Analytics, and Dominion Due Diligence Group.

ES-1

statistical confidence for all such PHAs as well as for small and large ones, but only at a 90 percent level of confidence for medium-sized PHAs.

Survey Results

3. Analysis of PHA survey data indicates those that performed EPCs reduced their use of natural gas and water significantly more than non-EPC users, at a 95 percent level of statistical confidence. Survey data indicated that EPC-using PHAs reduced their use of electricity and fuel oil by more than non-EPC

PHAs as well, but those results were not statistically significant.

4. Survey data suggest that, during the past several years, those PHAs that have used EPCs have improved their financial condition somewhat more than those that have not. Surveyed EPC-using PHAs also indicated that the financial condition of units included within the EPC improved more than at their other units. These findings were not conclusive at a 95 percent level of statistical confidence, however. They also could not be confirmed through examination of administrative financial data.

5. PHAs that executed EPCs reported greater improvements in the physical conditions of their properties than PHAs that did not. The dif erence in reported improvements between the two samples was significant at a 95 percent level of statistical confidence.

6. Approximately 60 percent of PHAs that had executed at least one EPC

indicated they were favorably disposed toward doing another. Because at least some of the other 40 percent that had executed at least one could already have accomplished all the energy investments they considered cost-effective, it is likely that more than 60 percent found the EPC program useful.

7. According to survey evidence, about two-thirds of PHAs that had executed an EPC at some units but not others indicated that the EPC resulted in lower utility consumption and better physical and financial condition than occurred in their non-EPC units. These responses furnish a strong rationale for why most of the EPC-using respondents expressed wil ingness to do another.

RAD Impact

8. The RAD program is being used by a large and growing number of PHAs.

According to interview data, the RAD program offers more flexibility in the use of funds to address housing units’ physical needs and is easier to negotiate than an EPC. The annual number of EPC applications has declined over time as more PHAs have turned to RAD.

9. Few PHAs are likely to undertake both RAD and an EPC. Indeed, PHAs that have undertaken RAD often have bought out their EPC contracts.

EPC Challenges and Possible Responses

10. Of those PHAs that have never undertaken an EPC, the most important reason (39 percent) given was that such a project would not be cost-effective from their perspective. Other leading reasons were that RAD is a better alternative (20

percent) or the process is too complicated (18 percent).

ES-2

Executive Summary

11. Of those PHAs that started but did not complete an EPC, the main reason given was that they did not have sufficient staff to see the project through. Another reason was that HUD’s EPC process is too cumbersome.

12. Interviewed PHAs indicated that HUD’s EPC program should be among the options available. Interviewees suggested, however, that because many PHAs, particularly smaller ones, do not fully understand the program, do not have the staff to undertake an EPC, or do not know how to deal effectively with contractors, more technical support would be useful.

13. Survey respondents and interviewees identified several means by which HUD

could promote the use of EPCs. Such strategies include continued encouragement to undertake the program, direct subsidies, a streamlined application process, and greater technical assistance.

Recommendations

Based on these conclusions, the LMI team offers the following recommendations: 1. Because there is strong statistical evidence that EPCs have helped PHAs improve their units’ energy efficiency, the program appears to be accomplishing one of its principal purposes. If gains in energy efficiency remain an important HUD goal, then the EPC program should be maintained if not strengthened.

2. The program offers PHAs a means to tap into private capital markets. The RAD

program also does so, but not all PHAs are interested in RAD or able to take advantage of it. For these PHAs, it is important to maintain the EPC program as an option.

3. According to some interviewees, HUD is no longer emphasizing its EPC program to PHAs to the extent it once did, and a number of PHAs are skeptical of undertaking it. These interviewees assert that increased communication regarding the program, particularly with respect to lessons learned and how to deal with contractors, would lead more PHAs to consider the program. In addition, several interviewees believe that more financial and technical support, particularly directed toward smaller PHAs that seek to self-manage EPCs, likely would increase participation.

4. We did not compare utility savings achieved by RAD-using PHAs with those using only EPCs. A possibility exists that RAD-related savings might be greater because RAD transformations are often holistic rehabilitation or construction of new units in which synergies among energy conservation investments can be realized. The comparison should be made to determine whether the RAD

approach achieves as much if not more energy ef iciency than the EPC

approach.

5. We did not examine whether who paid utility bil s between a PHA and the tenants made a difference in the PHA choosing to participate in the EPC program. HUD’s Resident-Paid Utility Incentive is intended to deal with this issue, but the extent to which PHAs have taken advantage of the incentive is unclear. Even with the incentive, however, PHA payment of utilities may provide greater inducement to undertake an EPC than when tenants pay the bil s for individually metered utilities. This issue should be examined by HUD to determine whether PHAs with partial or full tenant utility payments are less inclined to undertake EPCs.

ES-3

Chapter 1

Introduction

This study assesses the effectiveness and value of the U.S. Department of Housing and Urban Development’s (HUD’s) Energy Performance Contract (EPC) program by surveying public housing authorities (PHAs) and comparing trends in the utility consumption and the financial and physical condition of those PHAs that have implemented EPCs with those that have not. It also seeks to discover the extent to which implementation factors affect the rate at which EPCs are used by PHAs. HUD contracted with a team headed by LMI (the LMI team) for the comparative and implementation assessments.

EPCs enable PHAs to secure private-sector financing to upgrade the energy efficiency of their facilities, utilizing the savings that result from these improvements to pay back the contractors. The availability of EPCs thus enables PHAs to invest more than would be possible through capital budgets, operating income, or HUD-provided subsidies alone. As a result, PHAs that make use of EPCs should be able to lower utility bil s and improve the physical condition of their housing units. If the energy efficiency investments prove cost-effective, they should improve these PHAs’ financial condition as well.

Several reasons exist, however, why a PHA may not use the EPC program to address energy efficiency investments. To gain approval from HUD to initiate an EPC, PHAs must go through several layers of review, requiring their staf to spend considerable time and effort. Further, not all prospective EPCs obtain positive returns, and there are other means whereby housing authorities might improve their utility, financial, and physical conditions, in particular, the Rental Assistance Demonstration (RAD) program, 1 or by obtaining grants from state or local governmental authorities. Even if no outside financing is obtained, a PHA might invest in energy efficiency simply because it is cost-effective to do so. Thus, it is not a foregone conclusion that PHAs that use EPCs wil improve their utility consumption and other conditions at a greater rate than PHAs that do not.

This study followed four lines of investigation. First, the LMI team sent an online survey to more than 400 PHAs selected through a disproportionate stratified sampling strategy.

One purpose was to obtain information from PHAs that have undertaken EPCs concerning their effectiveness in reducing utilities and in improving financial and physical conditions. The survey also obtained information from PHAs that have not undertaken EPCs regarding improvements over time in the same three metrics. This latter group was broken into two parts: PHAs that at one point started but then did not complete the EPC process and those that had never begun an EPC. Al survey respondents were asked their views on the implementation of HUD’s EPC program and how it might be improved. In all, 327 or more than 70 percent of those to whom the survey was sent responded.

1 RAD is a federal housing program administered by HUD that enables assisted rental housing to access private capital markets for purposes of development and rehabilitation. The program was initially authorized by Congress in 2012 as a demonstration program and since has expanded from a ceiling of 60,000 units to 455,000 units.

1-1

Second, the online survey was followed by a set of telephone interviews with a subset of the responding PHAs stratified by size and geographic area that went into more depth concerning their experiences with EPCs or improving their utility consumption without the use of this program. In all, 27 PHAs were selected out of which 20 telephone interviews were conducted.

Third, various HUD databases containing information about utility consumption and the financial and physical condition of PHAs were examined. The LMI team obtained access to HUD’s Real Estate Assessment Center (REAC) databases that included the Financial Management Assessment Subsystem (FASS) and Physical Assessment Subsystem (PASS), and also to several years of Utility Expense Level (UEL) data, as submitted by PHAs on form 52722. This information enabled the LMI team to corroborate survey responses and to statistically test the efficacy of EPCs in improving public housing utility consumption and financial and physical conditions.

A fourth approach looked to see whether PHAs prefer the RAD or EPC program as a means to finance improvements in upgrading the energy efficiency of their housing units.

RAD enables PHAs to finance a broader range of improvements than the EPC program and has dif erent implementation requirements. The number of housing units in the RAD

program and number of EPC approvals and applications were compared to see to what extent RAD may have replaced EPCs for energy efficiency upgrades of PHA portfolios.

The rest of this report is organized as follows:

• Chapter 2, a brief background on the public housing EPC program.

• Chapter 3, the sampling method used to gather information on the relative effectiveness of HUD’s EPC program.

• Chapter 4, how the survey was implemented.

• Chapter 5, survey results, including the effect of EPCs on utility consumption and what respondents reported with respect to EPC implementation.

• Chapter 6, results obtained from in-depth telephone conversations with a subset of PHA directors who responded to the survey.

• Chapter 7, analysis of HUD administrative data. These data were used to statistically test whether HUD’s EPC program affected PHA utility consumption as well as these entities’ financial and physical condition.

• Chapter 8, HUD’s RAD program and its impact on PHA choice with respect to financing various improvements to public housing.

• Chapter 9, conclusions from the study and recommendations.

• Appendix A, graphs showing the relative frequency of answers given to each of the survey questions.

• Appendix B, detailed notes on each of the 20 follow-up telephone interviews with PHAs. The notes summarize what each interviewee had to say, omitting only the identity of the individual PHA.

• Appendix C, discussion of methods used to deal with missing or improperly reported data.

1-2

Introduction

• Appendix D, maps of HUD vs. U.S. census regions, and of PHA and sample populations.

• Appendix E, list of abbreviations used in the study and their meanings.

1-3

Chapter 2

Background

Energy performance contracts (also called energy service performance contracts or ESPCs) were first authorized by Congress in 1986. They were initially called shared energy service contracts and were authorized by the Consolidated Omnibus Budget Reconciliation Act of 1985. 1 They were later identified as ESPCs in the Energy Policy Act of 1992. 2 Such contracts are primarily aimed at enabling federal agencies to enhance the energy efficiency of their working facilities. ESPCs supply federal agencies with a source of funding for energy-saving purposes beyond annual appropriations.

Under the act’s provisions, agencies can contract with private companies to furnish energy-saving equipment and the upfront money to finance it. Contractor investments, including a return on capital, are expected to be repaid from the resulting energy savings.

According to the Federal Energy Management Program (FEMP), nearly $5.2 bil ion has been invested within the ESPC program since its inception, resulting in cumulative energy cost savings of $12.3 bil ion. A wide variety of federal agencies have undertaken ESPC projects, including the General Services Administration, the Department of Defense, and the Department of Energy. FEMP guides and assists agencies in implementing ESPCs at their own facilities.

In HUD’s case, public housing properties are not regarded as federal buildings and therefore are not covered by FEMP regulation. Instead, energy performance contracts in public housing were separately authorized under Section 118(a) of the Housing and Community Development Act of 19873 and implemented through revised Performance Funding Systems regulation within the Code of Federal Regulations (CFR). 4

HUD has a systematic approach to its EPC program (HUD, 2018c). A public housing authority prepares a request for proposals, conducts an energy audit, and creates a contract package with the contractor that wil supply the energy investments. An initial review is conducted by a HUD field of ice, Energy Center, or contractor, and another level of review by a HUD panel, which examines the earlier review to be sure it is complete. The reviewers use technical and completeness review checklists to ensure that the project is technically sound and that it contains a complete set of required documents. Once approved, the project is authorized by a field Public Housing Director.

From the applicant’s perspective, the key steps are gaining approval of the application by the field office or Energy Center, and confirmation that the review was complete from HUD’s review panel.

Of the approximately 3,000 PHAs in the United States, nearly 90 percent are small or very small with under 500 units. Medium (500–1,250 units), large (1,250–6,600 units), and very large PHAs (over 6,600 units) constitute the remaining 10 percent. Among 1 Public Law (P.L. ) 99–272, 100 Stat. 82, also known as COBRA 85.

2 102nd Congress H.R.776.ENR, abbreviated as EPAct 92.

3 P.L. 100-242, 101 Stat. 1815.

4 24 CFR 990, which applies specifically to HUD.

2-1

PHAs, a few hundred have undertaken EPCs. Nearly 50 percent of medium to large PHAs have undertaken them, but only about 4 percent of the smaller ones have done so. These relative proportions were taken into account in stratifying the survey sample as described in the next chapter.

Since 2012, PHAs have had access to another program, the Rental Assistance Demonstration, to tap private capital to enhance energy efficiency (along with other improvements) in public housing. This program converts PHA-owned properties to rental assistance under Section 8 of the United States Housing Act of 1937, 5 which furnishes project-based rental assistance or project-based voucher assistance over long-term contracts, typically 20 years. RAD provides incentives for PHAs to invest in energy efficiency because utility costs are frozen at pre-RAD levels, analogous to the “frozen rolling base” in EPCs. HUD’s long-term financial commitments to subsidize tenant rents serve as a credit enhancement to attract long-term debt and equity from the sale of Low-Income Housing Tax Credits (LIHTCs) in order to attract capital to upgrade facilities and reduce backlogs of unfunded capital improvements. In Chapter 8, the effect of RAD on HUD’s EPC program is examined to see whether it significantly changed PHA preferences in obtaining private finance to upgrade housing units.

5 Enacted September 1, 1937 (P.L. 75-412).

2-2

Chapter 3

Survey Sample

Research Population

The chief method of gathering data on the relative performance of Energy Performance Contract-using and non-EPC-using public housing authorities with regards to utility, financial, and physical conditions was a survey of PHAs. The population under study consisted of three groups: Group 1—executed at least one EPC; Group 2—started the process but did not complete an EPC; and Group 3—never executed an EPC. For purposes of comparing relative performance, we mainly looked at Groups 1 and 3. There were 2,971 PHAs in all. Of those, 237 had implemented an EPC by fiscal year (FY) 2015

or earlier1 and 2,734 PHAs had not (we refer to this group as non-EPC).

Sample Design and Process

We categorized sizes of the PHAs as small (very small/small), medium, and large (large/very large). We categorized regions according to U.S. Census regions: South, West/Midwest, and Northeast. Because the West region has smaller subgroup counts, we combined the West and Midwest regions into a single category for purposes of getting a nationwide representation. We also compared the three census regions with the 10 HUD regions to ensure consistent mapping. Exhibit 3-1 shows the mapping of the 10 HUD regions to the census regions for each state. In appendix D we show the two regional breakdowns in map form. Appendix D also contains maps showing the distribution of the population of PHAs among the states and the number included in our sample in each state.

Exhibit 3-1. Mapping of HUD Regions to Census Regions

HUD region

Area name

Census region

New England

Connecticut

Northeast

New England

Maine

Northeast

New England

Massachusetts

Northeast

New England

New Hampshire

Northeast

New England

Rhode Island

Northeast

New England

Vermont

Northeast

New York/New Jersey

New Jersey

Northeast

New York/New Jersey

New York

Northeast

Mid Atlantic

Washington, DC

South

Mid Atlantic

Delaware

South

Mid Atlantic

Maryland

South

1 We chose FY 2015 as the cutoff because we wanted to compare performance after an EPC had been implemented with before implementation and needed at least a few years of post-EPC utility consumption to do so. Choosing FY 2015 as the cutoff meant that, at a minimum, we would have 3 years of post-EPC implementation data.

3-1

Exhibit 3-1. Mapping of HUD Regions to Census Regions

HUD region

Area name

Census region

Mid Atlantic

Pennsylvania

Northeast

Mid Atlantic

Virginia

South

Mid Atlantic

West Virginia

South

Southeast Caribbean

Puerto Rico

South

Southeast Caribbean

Virgin Islands

South

Southeast Caribbean

Alabama

South

Southeast Caribbean

Florida

South

Southeast Caribbean

Georgia

South

Southeast Caribbean

Kentucky

South

Southeast Caribbean

Mississippi

South

Southeast Caribbean

North Carolina

South

Southeast Caribbean

South Carolina

South

Southeast Caribbean

Tennessee

South

Midwest

Illinois

West/Midwest

Midwest

Indiana

West/Midwest

Midwest

Michigan

West/Midwest

Midwest

Minnesota

West/Midwest

Midwest

Ohio

West/Midwest

Midwest

Wisconsin

West/Midwest

Southwest

Arkansas

South

Southwest

Louisiana

South

Southwest

New Mexico

West/Midwest

Southwest

Oklahoma

South

Southwest

Texas

South

Great Plains

Iowa

West/Midwest

Great Plains

Kansas

West/Midwest

Great Plains

Missouri

West/Midwest

Great Plains

Nebraska

West/Midwest

Rocky Mountain

Colorado

West/Midwest

Rocky Mountain

Montana

West/Midwest

Rocky Mountain

North Dakota

West/Midwest

Rocky Mountain

South Dakota

West/Midwest

Rocky Mountain

Utah

West/Midwest

Rocky Mountain

Wyoming

West/Midwest

Pacific/Hawaii

Arizona

West/Midwest

Pacific/Hawaii

California

West/Midwest

Pacific/Hawaii

Hawai

West/Midwest

Pacific/Hawaii

Nevada

West/Midwest

Northwest/Alaska

Alaska

West/Midwest

Northwest/Alaska

Idaho

West/Midwest

3-2

Survey Sample

Exhibit 3-1. Mapping of HUD Regions to Census Regions

HUD region

Area name

Census region

Northwest/Alaska

Oregon

West/Midwest

Northwest/Alaska

Washington

West/Midwest

The population counts for the three size categories are shown in exhibit 3-2. The size distribution of the population is skewed, with fewer numbers of large PHAs than medium and small PHAs. Large PHAs, however, account for most of the units affected by EPCs.

For any moderately sized sample, an equal probability sample design would include extremely small numbers of larger PHAs. Therefore, to allow for separate analysis by size, we selected a fairly large sample size per size-band. This method was achieved by systematically increasing the sampling fraction as the size of the PHA increases.

Exhibit 3-2. Group Population Counts

Group 1

Group 2

Group 3

Smal

South

24

10

1,308

West/Midwest

33

15

961

Northeast

43

7

251

Subtotal

100

32

2,520

Medium

South

26

2

56

West/Midwest

21

5

35

Northeast

27

3

29

Sub-total

74

10

120

Large

South

15

7

19

West/Midwest

18

2

16

Northeast

30

0

8

Subtotal

63

9

43

Total

237

51

2,683

Group 1: For the EPC population, we employed a disproportionate stratified sampling strategy for the three size categories defined in exhibit 3-2. Specifically, we applied an incremental sampling fraction for each size category from small to large PHAs. Large PHAs were selected with certainty; that is, the sampling fraction was one, medium was 0.8 and small 0.6. In addition, the sample is stratified by region, although within each size category the sampling fraction is kept roughly constant and the sub-samples selected proportional to corresponding population counts (exhibit 3-3).

3-3

Exhibit 3-3. EPC Sampling Strategy

EPC population

EPC sample Sampling fraction

Smal

South

24

16

0.6

West/Midwest

33

21

0.6

Northeast

43

28

0.6

Subtotal

100

65

0.6

Medium

South

26

21

0.8

West/Midwest

21

17

0.8

Northeast

27

22

0.8

Sub-total

74

60

0.8

Large

South

15

15

1.0

West/Midwest

18

18

1.0

Northeast

30

30

1.0

Subtotal

63

63

1.0

Total

237

188

Group 2: Group 2 members (PHAs that had initiated but did not complete an EPC) were selected with certainty as there are only 51 such PHAs in the population.

Group 3: For the non-EPC population, we matched the subgroups to make this group as similar as possible to the EPC population. Large PHAs were selected with certainty.

There are only 43 large PHAs in Group 3 compared with 63 in Group 1. The sampling fraction for medium PHAs was 0.57 and that for small only 0.03, because there are thousands of small PHAs that have not conducted EPCs. We increased the small and medium size-bands in Group 3 to compensate for the smaller number of large PHAs as compared with Group 1 (exhibit 3-4).

Exhibit 3-4. Non-EPC Sampling Strategy

Non-EPC

Non-EPC

Sampling

population

sample

fraction

Smal

South

1,308

20

0.02

West/Midwest

961

25

0.03

Northeast

251

32

0.13

Subtotal

2,520

77

0.03

Medium

South

56

24

0.43

West/Midwest

35

20

0.57

Northeast

29

24

0.83

Subtotal

120

68

0.57

Large

South

19

19

1.00

West/Midwest

16

16

1.00

Northeast

8

8

1.00

Subtotal

43

43

1.00

Total

2,683

188

3-4

Survey Sample

We expected (and obtained) about a 70-percent response rate to our nationwide survey of PHAs. Our aim was to obtain Group 1 and Group 3 samples of about 132 responses each, suf icient to establish statistical significance if dif erences were found between the two groups in utility, financial, or physical condition performance.

3-5

Chapter 4

Survey Implementation1

Survey Response Rate

Following the Office of Management and Budget approval of the survey instrument, 2 it was sent to 427 public housing authorities’ contacts. As indicated in Chapter 3, these PHAs were divided into three groups, 188 to those that completed an Energy Performance Contract, 51 to those that had started but not finished an EPC, and 188 to those that had never undertaken an EPC. Initial communication to recipients took the form of a note from HUD staff asking for cooperation with the survey team. Following that note, the survey was sent out on August 24, 2018, and closed as of September 19, 2018 (4 weeks). The survey was followed by several reminders to complete it by a fixed date. These reminders were sent every few days beginning about a week after the survey was initially sent and ending a day or two before the survey period closed.

In some cases, the email address given for a PHA contact did not work and the survey bounced back. In many of those instances, however, an alternate email address was secured and the survey was sent to the alternate recipient. In six cases, however, no alternate address was obtained, the survey was treated as not sent, and six was subtracted from the total in calculating the response rates. The revised total, therefore, was 421.

Two types of responses to the survey were received: complete and partial. Although ongoing efforts were made to explain the survey to respondents and to encourage completion, not all survey responses had been completely fil ed out by the time the survey period ended. In all, 327 responses were received, of which 280 were fully completed.

Exhibit 4-1 shows the cumulative response rate over time of those that responded either partially or fully. Fourteen percent responded soon after the initial request and, during the next 4 weeks, the rate rose to 78 percent. This rate exceeded the rate expected by LMI in its initial formulation of a project plan (70 percent).

1 The survey questions and the relative frequencies of answers received for each are shown in appendix A.

2 Reginfo.gov, ICR Reference No: 201711-2528-001, https://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=201711-2528-001.

4-1

Exhibit 4-1. Full and Partial Response Rate

In exhibit 4-2, the response rate of only the 280 PHAs that fil ed the survey out completely is shown. The rate begins at around 13 percent and then rises over time, finally getting to 67 percent after about 4 weeks. Although the LMI team would have preferred that all responses were completely fil ed out, a 67 percent rate is still quite substantial.

Exhibit 4-2. Completed Survey Response Rate

4-2

Survey Implementation

The rate of response was separated into regions as well. The regional response rates are shown in exhibit 4-3. This table shows fairly uniform regional response rates for fully plus partially completed surveys, but some discrepancy among regions for only fully completed surveys.

Exhibit 4-3. Survey Responses by Region

Fully + partially

Fully + partially

Ful y completed

Ful y completed

completed surveys completed surveys

surveys only

surveys only

Region

(#)

(%)

(#)

(%)

Midwest

76

81.7

70

75.3

Northeast

114

74.5

94

61.4

South

105

78.9

92

69.2

West

32

76.2

24

57.1

Total

327

77.7

280

66.5

Corrections to the Data3

Blank or Incorrect Responses

Although the high response rate to the survey was gratifying, not all the responses were correctly fil ed out and a few had to be eliminated. For example, the 327 responses included 19 in which the respondent entered the survey website but failed to fil out any questions at all. We cannot be sure why this situation occurred, but, as there was no useful data from these 19 responses, they were eliminated from further consideration, leaving 308 or about a 73-percent response rate.

Five other respondents appeared to misunderstand what they were expected to do and ended up answering the wrong set of questions. In other words, they classified themselves in one fashion but answered a dif erent set of questions than that classification required. These five were therefore eliminated as well, leaving 303

successful responses.

Discrepancies in Respondent Group Category Designations

A fairly large number of respondents classified themselves differently than the LMI team had based on data provided by HUD. Exhibit 4-4, for example, shows that 21 PHAs categorized themselves as having executed an EPC whereas HUD data showed them as having started but not completed an EPC. In addition, 10 PHAs said they had executed an EPC whereas HUD data indicated they had not.

3 The steps taken to correct the survey data are discussed in more detail in appendix C.

4-3

Exhibit 4-4. PHA Classifies Itself as Group 1 Whereas HUD Information

Classifies the PHA as Group 2 or 3

Classification discrepancy

Number

PHA—Group 1, HUD data—Group 2

21

PHA—Group 1, HUD data—Group 3

10

Total

31

It is possible that project timing could explain some of this discrepancy. For example, if a PHA started an EPC, did not complete it at the time but later completed it, HUD

information might have classified it within Group 2 whereas the PHA would classify itself as Group 1. It is also possible that the PHA undertook some other, non-EPC energy efficiency project which it then mistakenly classified.

Exhibit 4-5 shows the opposite type of discrepancy, namely where HUD information indicates a PHA has done an EPC whereas the PHA designated itself as in Group 2 or Group 3. Some of this discrepancy might be explained by instances where there has been turnover in PHA management and the present management is unaware that an EPC was done previously.

Exhibit 4-5. HUD Data Classifies PHA as Group 1 Whereas PHA

Classifies Itself as Group 2 or 3

Classification discrepancy

Number

HUD—Group 1, PHA—Group 2

5

HUD—Group 1, PHA—Group 3

7

Total

12

In exhibit 4-6, HUD data and individual PHAs agree that no EPC was done, but, in 21

cases, HUD data indicate that the PHA never did an EPC whereas the PHA indicates it started but stopped the process. We cannot be sure what a PHA meant when it indicated it started the process; possibly some of the starts involved little more than an initial review that did not go very far. In two cases, HUD data indicated a PHA had started but not completed an EPC whereas the PHA said it never started one. This discrepancy could result from a PHA thinking it never looked seriously at an EPC

whereas HUD classified it as having started the process, or by personnel turnover at a PHA, with new leadership unaware that previous leadership had started the EPC

process.

Exhibit 4-6. HUD and PHA Disagree on Whether the PHA is in Group 2 or 3

Classification discrepancy

Number

HUD—Group 3, PHA—Group 2

21

HUD—Group 2, PHA—Group 3

2

Total

23

4-4

Survey Implementation

In all, more PHAs classified themselves as in Groups 1 or 2 than HUD data. Assuming most of these PHAs did, in fact, engage with HUD’s EPC process, more PHAs have been wil ing to do so than previously thought.

Implications for Survey Response Analysis

Because we wanted to see whether an EPC made a meaningful dif erence in energy performance, it was important that we make appropriate corrections to the survey data.

As already mentioned, six instances in which we were unable to secure correct email addresses were eliminated. We also eliminated the 19 responses in which the respondents entered the survey website but did not answer any questions, and five more where the respondents appeared to answer the wrong set of questions.

Concerning the discrepancies between how PHAs classified themselves and how the LMI team performed based on HUD-supplied data, we generally took the view that the PHA would have the best knowledge as to whether it considered or completed an EPC.

In most cases, therefore, we accepted the PHA’s classification and assumed it had answered the appropriate set of survey questions.

After detailed review with the Energy Center, however, we eliminated nine more responses from consideration as it appeared that the center had conclusive information that a PHA had, in fact, completed an EPC, whereas the PHA said it had not. Elimination of these nine PHAs reduced our sample further, to 294 PHAs.

Adjustment of Sample Sizes

Because we eliminated responses on various grounds, and in 58 cases accepted PHA classifications of whether they belonged to Groups 1, 2, or 3 as opposed to our original classification based on HUD data, we had to make adjustments to our group sample sizes. Once this modification was done, we ended with 160 responses from Group 1, 37

from Group 2, and 97 from Group 3. Group 1 and Group 3 samples were used to test whether the energy performance and financial and physical condition of PHAs that conducted EPCs were significantly dif erent from those that did not.

In exhibit 4-7, we show the three groups separated into small, medium, and large PHA subgroups. For Group 1, 54 percent of total responses, we received about the same number of useful responses from each of the three subgroups, whereas for Groups 2 (13

percent of total responses) and 3 (33 percent of total responses), we received successively more useful responses as PHA size declined. For statistical testing purposes, however, we did not distinguish among the size classes of the PHAs.

4-5

Exhibit 4-7. Final Numbers of Survey Responses by Group and PHA Size

4-6

Chapter 5

Survey Analysis

Statistical Weighting

Our approach to testing the efficacy of HUD’s Energy Performance Contract program consisted largely of comparing performance measures between the EPC and non-EPC

groups (Group 1 and Group 3), using summary statistics such as means/averages and proportions. In probability-based sample designs, however, the collected data must account for selection probabilities and be adjusted for dif erences in proportions among the dif erent strata or subgroups. We employed the following steps to weight the survey data collected and calibrate the data to population totals.

1. Base weights: Our sample design was stratified in such a way that the selection probabilities were not equal among the different strata. For example, small South public housing authorities in Group 1 had a sampling probability of 0.6 whereas small South PHAs in Group 3 had a selection probability of 0.013. To account for unequal selection probabilities, we calculate base weights by taking the inverse of selection probabilities.

Base weight h = 𝑁𝑁h,

𝑛𝑛h

Where Nh is the population count of stratum h and nh is the sample size in stratum h.

2. Adjust for non-responses: As noted in Chapter 4, not all PHAs to whom the survey was sent responded. We calculated response rate (RR) adjustment factors by taking the inverse of the response rate for each stratum as shown in the following formula:

RR weight h = 𝑛𝑛h,

𝑟𝑟h

Where nh is the sample size of stratum h and rh is the number of responses collected in stratum h.

3. Applying both weights together, we arrived at a combined weight for each stratum h using the formula:

Final weight h = base weight h x RR weight h.

This final weight calibrates all responses to known population counts and keeps the response proportions of each stratum the same as in the population.

The weighted data obtained using the steps described previously are only used for calculating weighted means, weighted standard deviations, and weighted standard errors when comparing Groups 1 and 3 and performing statistical hypotheses testing.

Other reports, frequencies, cross-tabulations, charts, and tables are presented using the raw, unweighted data.

5-1

Statistical Hypotheses Testing—Comparing Group 1 with

Group 3

The survey questions included in the statistical analyses for group comparisons are shown in exhibit 5-1. These inquiries include questions on utility, financial, and physical condition performance.

Exhibit 5-1. Survey Questions Relating to Utility, Financial, and

Physical Condition

Group 3: No energy

Group 1: EPC

Group 3: Alternative investments

investments

Q9–Q11. On average, by how much Q40–Q42. On average, by how

did utility consumption change due to much did utility consumption the EPCs you implemented?

change due to the utility

conservation investments you

made?

Q12–Q14. On average, by how

Q43–Q45. On average, by how

much did the following utility

much did utility expenses change

expenses change due to the EPCs

due to the utility conservation

you implemented?

investments you made?

Q21. Overal , how did the EPCs you Q47. How did the utility

Q49. Over the past 5 years,

executed affect the financial

conservation investments affect the would you say that the

condition of your PHA?

overal financial condition of your

financial condition of your PHA

PHA?

has shown: (measure of extent

of improvement)?

Q22. How did the EPCs affect the

Q48. How did the utility

Q50. Over the past 5 years,

overal physical condition of the

conservation investments affect the would you say that the

included properties?

overal physical condition of the

physical condition of the

properties in your PHA?

properties within your PHA

has shown: (measure of extent

of improvement)?

Development of Ratings and Averages

For each of the questions specified in exhibit 5-1, we needed to standardize the ratings across both of the groups to make meaningful comparisons. For that purpose, numeric ratings were created. These ratings are shown in exhibit 5-2. If a respondent indicated it experienced both an increase and a decrease of consumption or expenses for any of the utility sources, however, that response was removed from the data set and the test.

Exhibit 5-2. Ratings Attached to Answers to Utility-Related Questions

Calculation of Weighted Average Scores

After standardizing the rating scores as indicated, we compared ratings of Group 1

PHAs with Group 3 PHAs using weighted averages for consumption and expense for each utility (electricity, natural gas, fuel oil, and water) and for financial and physical 5-2

Survey Analysis

conditions. The weighted average ratings were calculated using the following generic formula:

𝑛𝑛

Weighted average = ∑ 𝑤𝑤(𝑖𝑖)∗𝑟𝑟𝑟𝑟𝑟𝑟𝑖𝑖𝑛𝑛𝑟𝑟(𝑖𝑖)

𝑖𝑖=1

,

∑𝑛𝑛 𝑤𝑤(𝑖𝑖)

𝑖𝑖=1

Where w(i) = the final weight obtained after the weighting process, rating(i) = the corresponding rating score calculated for the measure, and n = the number of responses.

We il ustrate with an example calculation for the weighted average of electricity consumption for Group 1. The related question for this example is Q9. On average, by how much did electricity consumption change due to the EPC(s) you implemented?

Here, n = number of EPC PHAs that responded to Q9 = 111.

Numerator = sum (weights ÷ electricity consumption rating) for 111 respondents = 375.63.

Denominator = sum of all weights of the 111 respondents = 173.16.

Weighted average = numerator/denominator = 375.63/173.16 = 2.1692.

Similarly, weighted averages were calculated for al the dif erent measures that were compared across Groups 1 and 3. The detailed averages, standard deviations, and standard errors are shown in exhibit 5-3. Among the weighted averages for utilities, a lower number means a better score; that is, a bigger reduction in utility use or payments.

For financial and physical condition, a bigger number indicates a better score; that is, a bigger improvement over time.

Exhibit 5-3. Weighted Averages, Standard Deviations, and Standard Errors

Number of

Weighted

Weighted std.

Std. error mean

Group responses (n)

average/mean

deviation

s/sqrt(n)

Electricity

1

111

2.1692

1.63753

0.15543

consumption

3

37

2.6243

5.46642

0.89867

Electricity

1

111

2.4410

1.87246

0.17773

expense

3

38

2.6286

5.36416

0.87018

Natural gas

1

105

2.0652

1.63492

0.15955

consumption

3

22

2.7579

5.07379

1.08174

Natural gas

1

105

2.2909

1.77449

0.17317

expense

3

21

2.8741

3.20220

0.69878

Fuel oil

1

15

2.2198

2.04196

0.52723

consumption

3

7

2.9948

4.24903

1.60598

Fuel oil expense

1

13

2.1455

2.04764

0.56791

3

7

3.1496

2.75227

1.04026

Water

1

114

1.7196

1.49936

0.14043

consumption

3

30

2.5306

5.15194

0.94061

Water expense

1

111

2.1115

1.89231

0.17961

3

29

2.5098

4.94338

0.91796

5-3

Exhibit 5-3. Weighted Averages, Standard Deviations, and Standard Errors

Number of

Weighted

Weighted std.

Std. error mean

Group responses (n)

average/mean

deviation

s/sqrt(n)

Financial

1

144

3.5529

1.13746

0.09479

condition

3

84

3.3707

4.51324

0.49243

Physical

1

148

4.0115

0.72607

0.5968

condition

3

84

3.7708

3.96467

0.43258

Hypotheses Testing

We tested whether there were statistically significant dif erences in performance between Groups 1 and 3 for each of the consumption, expense, physical, and financial measures. The null hypothesis in each case was that “there is no dif erence between the averages/means of a given measure between the EPC group and the non-EPC group.”

The test, therefore, was whether the null hypothesis was rejected, that is, whether there was a statistically significant dif erence between the group averages.

Exhibits 5-4 and 5-5 show the results of the hypothesis testing. The measures with an asterisk (*) indicate that the null hypothesis that there was no dif erence between the groups was rejected at a 95 percent confidence level, meaning that there was a difference and that the chances that this difference resulted from random factors are no more than 5 percent. 1

Exhibit 5-4. Group 1 and Group 3 Differences in Average Utility Scores

(* indicates statistical significance at the 95-percent level)

Exhibit 5-4 indicates that among the eight measures, only three showed statistically significant dif erences at the 95-percent level: natural gas consumption, natural gas expense, and water consumption. The other dif erences between weighted averages did not achieve that level of statistical confidence. The Group 1 respondents, however, scored lower in water expense, electricity consumption, electricity expenditures, fuel oil consumption, and fuel oil expense. This result means that the implementation of an EPC

1 More generally, results indicating statistical significance at a 95 percent (or 90 percent) confidence level means that we are 95 percent (90 percent) confident that there is a true dif erence in performance between EPC PHAs and non-EPC PHAs. If a result is not statistical y significant at a 95 percent (90 percent) level, then the chances that whatever dif erence occurred resulted from random factors is greater than 5 percent (10 percent).

5-4

Survey Analysis

made a positive difference in these utilities, although not one we could infer was statistically significant at the 95-percent level.

Exhibit 5-5. Differences Between Average Financial and Physical Ratings

(* indicates statistical significance at the 95-percent level)

4.01

Physical Conditions*

3.77

Group 1: EPC

3.55

Group 3: Non-EPC

Financial Conditions

3.37

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

Exhibit 5-5 shows average scores for financial and physical conditions. For those categories, a higher number represents a better outcome. Although the numbers indicate that the financial condition of those PHAs that executed an EPC improved by more than such condition for those that did not, the difference is not statistically significant. At the same time, there was a statistically significant dif erence in the improvement of the physical condition between those that executed an EPC and those that did not. This difference may have arisen from the investments in energy-efficient equipment, or from financial returns from the energy investments which then could be used for other upgrades, or from both.

Within Group 1 Differences—Units Subject to an EPC vs.

Units Not Subject to an EPC

We also examined Group 1 survey responses to see whether respondents thought there were dif erences in performance between units within PHAs that had implemented an EPC and those that had not. We first examined whether a Group 1 respondent, having completed at least one EPC, would consider doing another. The responses are shown in exhibit 5-6. Approximately 60 percent of PHAs that had completed at least one EPC

indicated they were favorably disposed toward doing another. Because at least some of the other 40 percent could already have accomplished all the energy investments they considered cost-effective, we consider this convincing evidence that PHAs that had gone through the EPC process at least once found it useful.

Exhibit 5-6. Percentage of Group 1 Respondents Who Would Do Another EPC

We then asked those that had done an EPC to compare utility, financial, and physical condition performance at those units that had an EPC with such performance at units 5-5

that had not. The questions and distributions of responses are shown in exhibits 5-7, 5-8, and 5-9.

Exhibit 5-7. Survey Responses on Utility Performance at EPC vs. Non-EPC Units

Σ = 69.49%

Exhibit 5-8. Survey Responses on Financial Condition at EPC vs. Non-EPC Unitsa

Σ = 67.86%

a The response options for this question included “Somewhat worse at the units subject to the EPC” but no respondent chose that answer.

Exhibit 5-9. Survey Responses on Physical Condition at EPC vs. Non-EPC Unitsa

Σ = 66.67%

a Response options for physical condition at EPC units vs. non-EPC included “a lot worse” and “somewhat worse” but no respondent chose either of those two answers.

Nearly 70 percent of the Group 1 respondents thought that utility performance and financial and physical conditions had improved more at their EPC units than the non-EPC units. These findings are consistent with our statistical tests of differences in performance between PHAs that executed EPCs and those that did not, and provide a positive rationale for why Group 1 respondents would express wil ingness to do another EPC.

5-6

Survey Analysis

Survey Evidence Concerning Why PHAs Did Not

Undertake EPCs

Group 3 and Group 2 PHAs were asked why they had not undertaken an EPC or why, if they had started the process, they did not complete it. Exhibit 5-10 shows the responses received from those that had never done an EPC.

Exhibit 5-10. Why PHAs Chose Not to Do an EPC

As shown in the figure, the principal reason given (39 percent) for not doing an EPC was a matter of economics; the EPC would not be cost-effective, at least from the PHA’s point of view. We were unable to tell from this answer whether it occurred because, despite HUD’s Resident-Paid Utilities Incentive program (RPU), the PHA would not receive enough of the savings, or whether it simply meant that an EPC would not pay for itself regardless of who paid the utilities. This question might be investigated further. 2

Another important reason was the option to enter the Rental Assistance Demonstration program instead. The scope and financial basis of RAD are dif erent from those of an EPC and appeared to appeal to a significant proportion of those answering this question.

Those that had started but not completed an EPC responded as shown in exhibit 5-11

when asked why they did so. The figure indicates that the combined reasons of insufficient staff, paperwork, and expense cover more than one-half of those that stopped during the process. These factors may be more significant mainly to smaller PHAs, for whom the management of an EPC might constitute a significant outlay.

2 HUD’s RPU is designed to overcome any disincentive to PHA investment in energy efficiency from residents paying part or all the utilities. Data supplied by the Energy Center suggests that around 40 percent of PHAs that undertake EPCs take advantage of the incentive program (40 of 105 during the 6-year period 2012–2017 with 3 of the 105 indeterminate), but gives no indication whether those that did not take advantage had resident paid utilities or not. Recent HUD-supplied data indicates, however, that, as of 2018, about 73 percent of households within one or another form of public housing receive utility allowances. This data suggests, although it does not prove, that some PHAs with tenant paid utilities are not taking advantage of the RPU program.

5-7

Exhibit 5-11. Why PHAs Began but Did Not Complete the EPC Process

How to Improve HUD’s EPC Process

An important aspect of this project was to examine the implementation of HUD’s EPC

process to find ways it could be improved. Al three groups of PHAs were asked to address this question directly. Respondents could check multiple answers if they chose, and therefore the totals are more than the total number of responses to the survey.

Exhibit 5-12 shows the distribution of responses.

Exhibit 5-12. Ways to Improve HUD’s EPC Process Suggested by PHAs

As seen in the figure, the principal answer given was to increase operating subsidies to PHAs undertaking an EPC. The fourth highest response also indicated a desire for direct payment, whereas the third asked for more assistance in kind (that is, technical help).

The second most prevalent answer asked HUD to simplify its EPC process, although we did not ask respondents to identify ways this simplification should be done and few offered specifics. A few of our telephone interview contacts indicated that one way PHAs overcome the administrative requirements is to let their contractor help because some of the contractors have experience with the EPC process. Contractors charge for this service, however, altering the economics of the project from what they otherwise would be.

5-8

Chapter 6

Telephone Follow-up Interviews

Introduction

To gain further insight into public housing authorities’ views on HUD’s Energy Performance Contract program, the team interviewed a number of the participants by telephone after the writ en survey was completed. Some 27 PHAs were selected to be interviewed, consisting of 9 from each group. The three groups were further divided into three each of small, medium, and large PHAs to sample opinions from as many different perspectives as possible within the limited sample.

Several efforts were made to contact the 27 PHAs to secure their participation in the follow-up telephone interviews. An initial note requesting participation was followed by two reminder notes plus a set of personal emails to PHA directors who had worked with one of the LMI team members. These efforts ultimately resulted in interviews with 20 of the 27 initially contacted, or a response rate of 74 percent.

Representation by Group and Size

The 20 interviewees consisted of 8 from Group 1, 6 from Group 2, and 6 from Group 3.

Thus, 40 percent had completed at least one EPC but the other 60 percent had not.

We interviewed representatives of eight small, five medium, and seven large PHAs. In exhibit 6-1, we show how the interview sample broke down between the two classifications (that is, by group and by size). Generally, within a total sample of 20, we were able to obtain a reasonable number of perspectives from each group and from each size category of PHA.

Exhibit 6-1. Breakdown of Interview Sample by Group and PHA Size

Group 1

Group 2

Group 3

Total

Smal

3

2

3

8

Medium

3

1

1

5

Large

2

3

2

7

Total

8

6

6

20

Survey Results

Although each PHA had something unique to offer concerning its experience (or non-experience) with HUD’s EPC program, a number of themes emerged from the interviews. These issues are briefly enumerated below.

1. A great deal of Rental Assistance Demonstration participation is taking place among PHAs whether or not they have previously completed an EPC.

6-1

RAD appears to have become most PHAs’ option of choice with regards to investment in facility upgrades. Several reasons were given, but the main one is that RAD enables a greater scope of investment in property upgrades. Because many PHAs see capital funding needs as reaching well beyond energy-related investments, this feature of RAD is especially attractive.

A second reason is that funding for RAD through the capital fund has been somewhat more stable than past capital funding. This consistency has increased the confidence of lenders and made RAD an attractive investment for them.

In addition, some PHAs see RAD as simpler to navigate than the EPC program, however, other PHAs said that the RAD program can be dif icult to negotiate.

2. The EPC and RAD programs are considered by PHAs to be alternatives.

Several PHAs remarked that they view EPCs and RAD as alternatives, not programs that should be pursued simultaneously. They see the programs as different means of raising private capital, with RAD the bet er choice if capital needs go beyond energy investment.

3. Energy savings are typically incorporated into any improvement project.

A number of PHAs remarked that they routinely look at potential energy savings from new investments whether or not they are EPC- or RAD-related. If they have not obtained funds from either of these programs, then they use some combination of operating and capital funds, state grants, and utility grants to make the needed energy investments.

4. If utilities are tenant paid, PHAs may not have the incentive to pursue the EPC

program.

In cases where utilities were entirely owner paid, PHAs often found that EPC or RAD energy investments would pay off. If utilities were partially or entirely tenant paid, however, PHAs were less likely to undertake EPC investments although, if stil made, the savings were popular with tenants. 1

5. In several cases, the payoff from an EPC came mainly from water savings.

In several cases where the PHA felt its EPC investment had clearly paid off, the savings had come mainly from reduced water bil s, not energy bil s. It could be that energy savings had already been wrung from the properties prior to the EPC, that tenants increased their electricity usage over time, or that water rates had risen more steeply in recent years, but, in any case, the people interviewed were satisfied with their EPCs because the water savings had made the investments cost-effective.

6. PHAs are generally open to any program or funding source as a means to improve their housing stock.

Several PHA directors said they are open to any source of funding that wil help them improve the physical condition of their properties. This funding includes EPCs, RAD, state and local funding sources, or utility programs that offer upfront capital. From this perspective, if an EPC offers the best deal to fund 1 As indicated in Note 2, Chapter 5, p. 5–7, HUD’s Resident-Paid Utilities Incentive (RPU) program is designed to maintain PHA incentive to invest in energy efficiency even if utilities are tenant paid. We were unable to tell from existing data, however, the extent to which PHAs with units where utilities are at least partly tenant paid take advantage of this program.

6-2

Telephone Follow-up Interviews

improvements, they are open to it, but if other programs, typically RAD, offer better possibilities, they wil consider those first.

Interviewee Recommendations

Because an important aim of this study was to identify ways in which HUD might be able to improve the implementation of its EPC program, interviewees were asked for their recommendations. Such recommendations already had been sought from the writ en survey, but the telephone follow-up interviews enabled more in-depth discussion. We emphasize that EPC program implementation recommendations were not necessarily uniform among PHAs; some recommended one thing, some another. The following, however, are the most common recommendations among the 20 PHAs we interviewed.

1. Better integrate and coordinate the RAD and EPC programs to help PHAs select the better program for their situation.

A number of PHAs felt that they could use more assistance in understanding the relative advantages of RAD vs. EPCs as well as the complexities of substituting one for the other or doing both simultaneously. Although they understood the basics of both programs, they were unsure how each best fit into their particular situation and recommended that HUD furnish more assistance in helping them navigate the choices. 2

2. Help PHAs decide if and how the RAD and EPC programs can be used together.

This recommendation follows from the first. PHAs are open to all sources of funding, and if an EPC plus RAD offers more funding at better terms in a particular instance, the PHA would want to take advantage. The complexities of doing both are difficult to work out, however, and PHAs would like help from HUD

in considering such an option.

3. Increase communication about the EPC program, sharing success stories, and creating a network of PHAs to share experiences and supply expertise.

A few PHAs felt that other PHAs either do not fully understand the EPC program or have heard negative stories about it that make them shy of initiating their own effort. Those holding such views recommend that HUD furnish more positive publicity for the program and also connect PHAs that have completed EPCs with those that have not for advice and guidance.

4. Train PHAs and field office staff to better understand the EPC program, including a. the procurement process and how to negotiate an EPC,

b. how to manage a contractor,

c. an explanation of how PHAs benefit from the savings, and

d. how to educate and engage tenants and maintenance staf upfront. 3

2 HUD offers PHAs a document entitled “The Rental Assistance Demonstration (RAD) and Energy Performance Contracts (EPCs)”

at https://www.hud.gov/sites/dfiles/PIH/documents/RAD_EPC_FAQs.pdf. It is unclear to what extent interviewees are aware of the document.

3 Although some interviewees offered this recommendation, HUD did in fact offer multiple 1-day and 2-day EPC training sessions to PHAs between 2013 and 2015. These training sessions were held around the United States. In addition, in 2018, HUD sponsored a number of 3-day training sessions for PHAs focused on how to evaluate EPC savings.

6-3

Some PHA directors felt that a number of others, mainly but not only smaller PHAs, were either unaware of how to conduct an EPC or had no idea how to work with the EPC contractor to obtain maximum benefits. They particularly stressed that an EPC contractor needs to be closely managed to be sure that the PHA understands what the contractor is doing and how the financial benefits are being shared.

5. Streamline the application process and supply more technical assistance, especially for smaller PHAs.

A number of PHAs offered this pair of recommendations. Streamlining and technical assistance were options from the writ en survey, however, so they might not have been offered as frequently had PHAs not had the ability to check them off. Stil , we questioned PHAs during the interviews concerning these recommendations and most said they stil would make them.

6. Increase PHA exposure to the EPC program.

This recommendation was offered by a few PHAs that felt that at one time HUD

had focused considerable attention on the EPC program but was no longer doing so. They felt that if HUD wants more PHAs to consider the program, it should educate them again on its merits and why they should seriously consider it.

Other recommendations were received from one or more PHAs, although we have summarized the most prevalent. In appendix B, however, we have included summaries of our notes from all 20 of the interviews, with only the identities of the PHAs omitted.

Other PHA suggestions and observations regarding the EPC program can be found there.

6-4

Chapter 7

Analysis of Utility, Financial, and Physical

Conditions Data

Background

Although the public housing authority survey was the principal means of col ecting data to address the central questions of this project, the study conducted other analyses using HUD administrative data. Specifically, the study gathered Utility Expense Level (UEL) data as well as HUD internal ratings of PHA financial and physical condition and PHA financial data to analyze whether PHAs that used Energy Performance Contracts performed better in these respects than PHAs that did not.

Analytic Approach

Analysis of UEL Data

HUD housing units consume various forms of energy, including electricity, natural gas, fuel oil, and, in a few cases, coal. We examined HUD’s energy consumption data and determined that although electricity data were sufficient for statistical analysis of PHA consumption trends, natural gas and other energy-related data were not. We reached the latter conclusion because there was less such data, the data sets were incomplete in many instances, and some of them were recorded using varying or even inapplicable metrics. 1 Hence, our analysis of HUD energy data from administrative sources is limited to electricity.

To measure the effectiveness of EPCs with respect to PHA electricity consumption, we used UEL data reported by PHAs and maintained by HUD. LMI selected a sample of 427 PHAs, consisting of 188 EPC PHAs and 239 non-EPC PHAs. Of this sample of 427, useful electricity data were available for only 321, including 146 PHAs that had done EPCs and 175 that had not. To assess the ef ectiveness of EPCs, we made two different types of comparisons:

• Long-term trends: We compared electricity consumption for all years prior to the execution of an EPC with consumption for all years after such execution. Where a PHA used multiple EPCs, we compared all years prior to the execution of the first EPC with consumption for years thereafter. For the non-EPC PHA group, we compared consumption before and after 2010, choosing that year because it was roughly the midpoint of our electricity consumption data set.

• Short-term effect: We compared utility consumption 2 years prior to deploying an EPC with consumption 2 years after an EPC. For the non-EPC group, we set the base year as 2010, and compared consumption 2 years before 2010 with 2 years after 2010.

1 These issues are described more fully in appendix C.

7-1

Analysis of Long-Term Electricity Consumption Trends

We took the following steps.

Step 1—Data cleansing: We initially processed and sanitized the available UEL data for each PHA. For the EPC PHAs, we required an EPC year (the year when an EPC was executed) for each PHA record. We observed inconsistencies in the data for when an EPC was executed; 76 of the 146 EPC PHAs did not report the year the EPC was initiated. Because we did not know when the EPC was begun, we eliminated these records and limited the sample to only 70 EPC PHAs.

Step 2—Calculating average consumption across all years: We calculated average utility consumption before and after the EPC was executed (or the 2010 base year for non-EPCs) for each individual PHA.

To il ustrate, consider FL041. This PHA reported utility consumption for three Asset Management Projects (AMPs). For all three, the first EPC year was 2007. The following steps were used to calculate the pre-EPC and post-EPC utility consumption averages: 1. We first calculated pre-EPC average consumption in years prior to 2007, specifically for 2004, 2005, and 2006. We then calculated average consumption for the years after 2007. These data covered the years 2008 to 2016.

2. We summed these consumption totals to derive totals for the PHA and then compared the pre-EPC average utility consumption with the post-EPC

consumption.

Exhibit 7-1 shows the calculations for the pre-EPC and post-EPC average utility consumption for FL041. The year 2007 is left blank because that was the year an EPC

was executed by this PHA.

Exhibit 7-1. Example Data Set and Calculated Averages for FL041

Electricity Consumption in Thousands of Kilowatt Hours (000 KWh) Pre- Post-Unit

2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 EPC EPC

FL041000001 2,371 2,573 2,902 N/A 2,339 2,217 2,403 2,335 2,221 2,248 2,308 2,339 2,448 2,615 2,318

FL041000002

453 478 500 N/A 1,415 1,356 1,462 1,434 1,356 1,439 1,799 1,950 1,651

477 1,540

FL041000003 1,720 1,595 1,644 N/A 1,120 1,009 1,067 944 915 886 933 1,058 1,141 1,653 1,008

Total PHA

4,544 4,646 5,046 N/A 4,874 4,582 4,932 4,713 4,492 4,573 5,040 5,347 5,240 4,745 4,866

FL041

We then compared average consumption levels for the entire population of EPC-using and non-EPC-using PHAs to assess the extent to which consumption dropped after EPCs were implemented or after the 2010 base year.

To do so, we used the fol owing formulae:

EPC_pre(all years) = ∑ 𝑈𝑈𝑈𝑈 (𝑂𝑂𝑂𝑂 𝑟𝑟𝑎𝑎𝑎𝑎 𝐸𝐸𝐸𝐸𝑈𝑈 𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃 𝑏𝑏𝑏𝑏𝑂𝑂𝑏𝑏𝑟𝑟𝑏𝑏 𝐸𝐸𝐸𝐸𝑈𝑈 𝑦𝑦𝑏𝑏𝑟𝑟𝑟𝑟)

𝑁𝑁𝐸𝐸𝐸𝐸𝑈𝑈−𝑝𝑝𝑟𝑟𝑏𝑏

7-2

Analysis of Utility, Financial, and Physical Conditions Data Non-EPC_pre(all years) = ∑ 𝑈𝑈𝑈𝑈 (𝑏𝑏𝑂𝑂 𝑟𝑟𝑎𝑎𝑎𝑎 𝑛𝑛𝑏𝑏𝑛𝑛−𝐸𝐸𝐸𝐸𝑈𝑈 𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃 𝑏𝑏𝑏𝑏𝑂𝑂𝑏𝑏𝑟𝑟𝑏𝑏 2010)

𝑁𝑁𝑁𝑁𝑏𝑏𝑛𝑛−𝐸𝐸𝐸𝐸𝑈𝑈𝑝𝑝𝑟𝑟𝑏𝑏

EPC_post(all years) = ∑ 𝑈𝑈𝑈𝑈 (𝑏𝑏𝑂𝑂 𝑟𝑟𝑎𝑎𝑎𝑎 𝐸𝐸𝐸𝐸𝑈𝑈 𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃 𝑟𝑟𝑂𝑂𝑟𝑟𝑏𝑏𝑟𝑟 𝐸𝐸𝐸𝐸𝑈𝑈 𝑦𝑦𝑏𝑏𝑟𝑟𝑟𝑟)

𝑁𝑁𝐸𝐸𝐸𝐸𝑈𝑈−𝑝𝑝𝑏𝑏𝑃𝑃𝑟𝑟

Non-EPC_post(all years) = ∑ 𝑈𝑈𝑈𝑈 (𝑏𝑏𝑂𝑂 𝑟𝑟𝑎𝑎𝑎𝑎 𝑛𝑛𝑏𝑏𝑛𝑛−𝐸𝐸𝐸𝐸𝑈𝑈 𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃 𝑟𝑟𝑂𝑂𝑟𝑟𝑏𝑏𝑟𝑟 2010)

𝑁𝑁𝑁𝑁𝑏𝑏𝑛𝑛−𝐸𝐸𝐸𝐸𝑈𝑈𝑝𝑝𝑏𝑏𝑃𝑃𝑟𝑟

Where,

NEPC-pre = number of years in the pre-EPC period;

NNon-EPCpre = number of years before 2010;

NEPC-post = number of years in the post-EPC period;

NNon-EPC-post = number of years after 2010 for non-EPC PHAs; and UC = utility consumption of a PHA.

Step 3—Dealing with outliers: After calculating average electricity consumptions for each PHA, we looked at the percentage change from before to after an EPC was executed.

This percentage change, however, could be strongly affected if there is inconsistent data for only a single year. Such inconsistencies occurred through various factors, such as deviations from a PHA’s previous consumption trend. Most of the inconsistencies were probably due to measurement or recording errors. To avoid results affected by such outliers, we sought to eliminate those that were the most extreme. To do so, we defined statistical outliers as deviating significantly from year-to-year percentage changes across all PHAs. A well-known approach to identify outliers is the 1.5 ÷ IQR rule, where IQR is known as the inter-quartile range. We calculated the upper and lower bounds of the percentage change data as shown here:

Upper bound = (1.5 ÷ IQR) + Q3

Lower bound = Q1–(1.5 ÷ IQR).

Where Q1 and Q3 are the first and third quartiles of the percentage change distribution across all PHAs, and IQR is the inter-quartile range (Q3–Q1). In other words, we eliminated data points that were more than 50 percent above or below the 25th and 75th percentiles of year-to-year changes displayed on average by the entirety of the population.

Using this method, we detected 12 outliers among EPC PHAs and 19 among non-EPC

PHAs. After excluding outliers, the final data set consisted of 58 EPC PHAs and 156

non-EPC PHAs.

Step 4—Paired t-test for within-group comparisons: A paired-samples t-test was conducted to compare long-term electricity consumption changes in Group 1 PHAs before and after an EPC was executed. In this test, we assumed the null hypotheses that “There is no dif erence between average electricity consumption for all years before and for all years after an EPC was executed.” We conducted this test at a 95 percent confidence level. We also conducted paired-samples t-tests for each of the individual size groups: small, medium, and large PHAs. For non-EPC PHAs, as explained previously, we conducted paired-sample t-tests to compare consumption during all years prior to 2010 and after 2010. The null hypothesis assumes that “There is no difference between average electricity consumption for all years before and after 2010.”

7-3

Findings for EPC PHAs

1. All PHAs (58): There was a significant dif erence when comparing pre-EPC

electricity consumption (mean = 5.8 mil ion kWh/yr., standard deviation [SD] =

10.9 million) with post-EPC consumption (mean = 5.07 mil ion kWh/yr., SD = 9.7

mil ion); t(57) = 2.764, p = 0.008. This test shows that overall among EPC PHAs, there was a statistically significant decrease in annual electricity consumption after an EPC was executed relative to before at a 95 percent confidence level.

2. Small PHAs (19): There was a significant dif erence when comparing pre-EPC

electricity consumption among small PHAs (mean = 0.96 million kWh/yr., SD =

0.81 million) with post-EPC consumption (mean = 0.86 mil ion kWh/yr., SD = 0.72

mil ion); t(18) = 2.225, p = 0.039. This test shows that for small EPC PHAs there was a statistically significant decrease in annual electricity consumption after an EPC was executed relative to before at a 95 percent confidence level.

3. Medium PHAs (19): There was no significant difference when comparing pre-EPC electricity consumption among medium PHAs before an EPC was executed (mean = 2.4 mil ion kWh/yr., SD = 1.7 million) with post-EPC consumption (mean

= 2.3 mil ion kWh/yr., SD = 1.6 mil ion); t(18) = 1.362, p = 0.190. This test shows that for medium-sized EPC PHAs, we did not observe a change in annual electricity consumption after an EPC was executed relative to before that was statistically significant at a 95 percent confidence level. We did, however, observe a significant decrease for this size group at a 90 percent confidence level.

4. Large PHAs (20): There was a significant dif erence when comparing pre-EPC

electricity consumption among large PHAs (mean = 13.6 mil ion kWh/yr., SD =

15.9 mil ion) with post-EPC consumption (mean = 11.7 mil ion kWh/yr., SD = 14.3

mil ion); t(19) = 2.64, p = 0.016. This test shows that for large EPC PHAs, there was a statistically significant decrease in annual electricity consumption after an EPC was executed relative to before at a 95 percent confidence level.

Interpretation of results: The results of these tests indicate that, at a 95 percent or 90

percent confidence level, there was a statistically significant reduction in PHA annual electricity consumption when comparing that consumption before and after an EPC was executed.

Findings for Non-EPC PHAs

1. All PHAs (156): There was a significant dif erence when comparing pre-2010

electricity consumption (mean = 3.3 mil ion kWh/yr., SD = 4.8 million) with post-2010 consumption (mean= 3.1 kWh/yr., SD = 4.4 mil ion); t(155) = 3.075, p =

0.002. This test shows that overall among non-EPC PHAs, there was a statistically significant decrease in annual electricity consumption after 2010

relative to before at a 95 percent confidence level.

2. Small PHAs (61): There was no significant dif erence when comparing pre-2010

electricity consumption (mean = 0.8 mil ion kWh/yr., SD = 0.9 million) with post-2010 consumption (mean= 0.84 kWh/yr., SD = 0.85 mil ion); t(60) = 1.658, p =

0.102. This test shows that among small non-EPC PHAs, there was no statistically significant change in annual electricity consumption after 2010

relative to before at a 95 percent confidence level.

7-4

Analysis of Utility, Financial, and Physical Conditions Data 3. Medium PHAs (60): There was a significant difference when comparing pre-2010

electricity consumption (mean = 2.7 mil ion kWh/yr., SD = 2.2 million) with post-2010 consumption (mean= 2.6 kWh/yr., SD = 2.1 mil ion); t(59) = 2.108, p =

0.039. This test shows that among medium non-EPC PHAs, there was a statistically significant decrease in annual electricity consumption after 2010

relative to before at a 95 percent confidence level.

4. Large PHAs (35): There was a significant dif erence when comparing pre-2010

electricity consumption (mean = 8.4 mil ion kWh/yr., SD = 7.7 mil ion) with post-2010 consumption (mean = 7.8 kWh/yr., SD = 6.8 mil ion); t(34) = 2.454, p =

0.019. This test shows that among large non-EPC PHAs, there was a statistically significant decrease in annual electricity consumption after 2010 relative to before at a 95 percent confidence level.

Conclusion: From the tests described previously, we can conclude that there were statistically significant long-term decreases in annual electricity consumption after an EPC was executed relative to before among small and large PHAs and overall among this group. A statistically significant decrease occurred in annual electricity consumption, however, among medium and large non-EPC PHAs after 2010 relative to before, and overall among such PHAs. Clearly, electricity consumption was decreasing among the various PHA populations during the time period under study.

Step 5—Independent samples t-test for between-group comparisons: An independent samples t-test was conducted to compare long-term electricity consumption trends between the EPC and non-EPC groups. In this test, we assumed the null hypothesis that

“There is no difference between the average change in electricity consumption before and after an EPC was executed for EPC PHAs and the average change in electricity consumption before and after 2010 for non-EPC PHAs.” We conducted this test at a 95

percent confidence level. We also conducted independent samples t-tests for each of the individual size groups: small, medium, and large PHAs.

Findings for Comparison of EPC vs. Non-EPC Electricity Consumption Comparing the two groups, there was a higher percentage decrease in annual electricity consumption among EPC than non-EPC PHAs (10.3 mean percent decrease for EPC

PHAs compared with a 4.4 mean percent decrease for non-EPC PHAs). We used an independent samples t-test to test whether this difference was statistically significant and found that it was. The outcome suggests that EPCs probably did play a role in reducing electricity consumption more rapidly for PHAs that used them during the period and particularly for small PHAs. They also may have helped medium- and large-sized EPC-using PHAs reduce their electricity consumption by more than comparably sized non-EPC users, but these dif erences were not statistically significant. The results are summarized in exhibit 7-2.

7-5

Exhibit 7-2. Summary of Changes in Electricity Consumption

Between EPC PHAs and Non-EPC PHAs

Smal PHAsa

Medium PHAs

Large PHAs

All PHAsa

Average change in electricity

−11.5%

−7.9%

−11.5%

−10.3%

consumption after an EPC

Average change in electricity

−2.8%

−4.9%

−6.3%

−4.4%

consumption with no EPC (2010

base year)

a Statistically significant difference at the 95-percent level.

Sensitivity test for EPC versus non-EPC comparison: Because the choice of 2010 as the before and after dividing year for non-EPC PHAs might have influenced the results obtained, we reran the analysis using 2009 or 2011 as the dividing year instead. This change made little difference in terms of the change in electricity consumption for the non-EPC PHAs (for example, still −6.0 percent for all PHAs using 2009 and −6.4 percent using 2011), and in both cases there was a statistically significant difference between the rate at which annual electricity consumption declined for EPC PHAs and non-EPC

PHAs. This result reinforces the conclusion that EPCs did have a meaningful effect on longer-term electricity consumption trends for PHAs that used them relative to PHAs that did not.

Analysis of Short-Term Electricity Consumption Trends

To analyze short-term trends of electricity consumption from using EPCs, we compared average annual consumption 2 years prior with 2 years post the year an EPC was executed. For non-EPC PHAs, we looked at 2 years prior to and 2 years post-2010. We repeated steps 1 to 4 as described in the previous section. No statistically significant difference was observed between 2 years prior-EPC consumption and 2 years post-EPC

consumption for all subgroups of PHAs at the 95 percent confidence level. For the non-EPC PHAs, no statistically significant dif erence in consumption before and after 2010

was observed across all size groups. We conclude that although electricity consumption was dropping on average during the longer period for all PHAs and dropped even more for EPC users, the differences are not sufficient during 2-year periods to draw conclusions from a statistical perspective.

Several limitations to the statistical findings should be noted.

1. We did not have data for all EPC years. In particular, we did not have UEL data for 2008 and 2009 but were able to fil in some of these missing data from past year figures in the 2010 data set. 2

2. Available UEL data may not have included all energy consumption occurring within a PHA. For example, tenant-paid utility data are excluded from UEL data although they generally are included in PHA utility allowance calculations.

3. A number of data quality issues were discovered, including gaps and inaccuracy.

For example, consumption totals sometimes seemed far out of line with the size of a PHA or with previous trends for the same PHA. We corrected the data as best we could, changing units that seemed unrealistic, checking other years’

2 The UEL data sets consist of the current year’s data as well as 3 years of prior data.

7-6

Analysis of Utility, Financial, and Physical Conditions Data reports to see whether a missing or unrealistic number had been included there, removing outliers, and so forth.

Analysis of Financial and Physical Condition

REAC Scoring Data

To measure the impact of EPCs on a PHA’s financial and physical condition, we obtained scoring data from the Real Estate Assessment Center (REAC) Online Systems (the database is labeled Public Housing Assessment System [PHAS], which includes the Physical Assessment Subsystem (PASS), the Financial Management Assessment Subsystem (FASS), and the Management Assessment Subsystem [MASS]), and a total score that includes these three scores plus a Capital Fund and a Resident score. We did not use the Capital Fund score nor the Resident score in our analysis but instead looked at each of the other three plus an average of the sum of these three scores over time.

The data covered several years of PASS, FASS, and MASS scores, enabling us to compare pre-EPC with post-EPC scores for EPC PHAs, and pre- and post-2010 scores for non-EPC users. The idea was that if the use of EPCs materially improved a PHA’s financial or physical condition relative to non-use of EPCs, this result might show up in the respective trends of FASS and PASS scores over time.

Step 1—Data considerations: Several data considerations and processing changes were made to compensate for data inconsistencies or missing data:

1. Some PHAs were missing individual subsystem scores; however, overall scores and other individual scores were available. For such cases, we inferred remaining individual subsystem scores by subtracting available scores from total PHAS scores.

2. The PHAS scoring method was not consistent across the years and was different for 2007 and prior and for 2011 and after. Data were not available for the period from 2008 to 2010.

Total points 2007 and prior: PASS—30 points; FASS—30 points;

MASS—30 points.

Total points 2011 and after: PASS—40 points; FASS—25 points;

MASS—25 points.

Due to the scoring inconsistencies, PHAS scores were standardized across all years using the percentage points they received in each year.

3. Resident score and Capital Fund score are 10 points each but, as stated previously, were not used for this analysis.

4. The analysis included 116 EPC PHAs and 2,729 non-EPC PHAs.

Step 2—Calculating average scores across all years: We calculated average total scores before and after an EPC was executed (or the base year 2010 for non-EPCs) for each individual PHA. We used the standardized percentage scores for all the analyses in this section. We calculated average standardized scores for the pre-EPC and post-EPC periods (or pre-base and post-base for non-EPCs) for each PHA using the formulae:

EPC_pre_PASS(all years) = ∑ 𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃 (𝑂𝑂𝑂𝑂 𝑟𝑟𝑎𝑎𝑎𝑎 𝐸𝐸𝐸𝐸𝑈𝑈 𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃 𝑏𝑏𝑏𝑏𝑂𝑂𝑏𝑏𝑟𝑟𝑏𝑏 𝐸𝐸𝐸𝐸𝑈𝑈 𝑦𝑦𝑏𝑏𝑟𝑟𝑟𝑟)

𝑁𝑁𝐸𝐸𝐸𝐸𝑈𝑈−𝑝𝑝𝑟𝑟𝑏𝑏

7-7

Non-EPC_pre_PASS(all years) = ∑ 𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃(𝑂𝑂𝑂𝑂 𝑟𝑟𝑎𝑎𝑎𝑎 𝑛𝑛𝑏𝑏𝑛𝑛−𝐸𝐸𝐸𝐸𝑈𝑈 𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃 𝑏𝑏𝑏𝑏𝑂𝑂𝑏𝑏𝑟𝑟𝑏𝑏 2010)

𝑁𝑁𝑁𝑁𝑏𝑏𝑛𝑛−𝐸𝐸𝐸𝐸𝑈𝑈𝑝𝑝𝑟𝑟𝑏𝑏

EPC_post_PASS(all years) = ∑𝐸𝐸 𝑃𝑃𝑃𝑃𝑃𝑃 (𝑂𝑂𝑂𝑂 𝑟𝑟𝑎𝑎𝑎𝑎 𝐸𝐸𝐸𝐸𝑈𝑈 𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃 𝑟𝑟𝑂𝑂𝑟𝑟𝑏𝑏𝑟𝑟 𝐸𝐸𝐸𝐸𝑈𝑈 𝑦𝑦𝑏𝑏𝑟𝑟𝑟𝑟)

𝑁𝑁𝐸𝐸𝐸𝐸𝑈𝑈−𝑝𝑝𝑏𝑏𝑃𝑃𝑟𝑟

Non-EPC_post_PASS(all years) = ∑ 𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃 (𝑂𝑂𝑂𝑂 𝑟𝑟𝑎𝑎𝑎𝑎 𝑛𝑛𝑏𝑏𝑛𝑛−𝐸𝐸𝐸𝐸𝑈𝑈 𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃 𝑟𝑟𝑂𝑂𝑟𝑟𝑏𝑏𝑟𝑟 2010)

𝑁𝑁𝑁𝑁𝑏𝑏𝑛𝑛−𝐸𝐸𝐸𝐸𝑈𝑈𝑝𝑝𝑏𝑏𝑃𝑃𝑟𝑟

Where,

NEPC-pre = number of al years in the pre-EPC period;

NNon-EPCpre = number of all years before 2010;

NEPC-post = number of years in the post-EPC period;

NNon-EPC-post = number of years after 2010 for non-EPC PHAs; and PASS = standardized PASS score of a PHA.

The average FASS, MASS, and overall PHA scores for all EPC and non-EPC PHAs were calculated in a similar manner, using the same formulae for pre- and post-comparison periods.

Step 3—Detecting outliers: After calculating average scores for each PHA, we looked at the percentage change of average total scores before and after an EPC was executed.

To eliminate inconsistencies from the data, we found statistical outliers from the percentage change across all PHAs; that is, data points that lie outside the overall pattern in a distribution. As described previously, a well-known approach for finding outliers is the 1.5 ÷ IQR rule, where IQR is known as the inter-quartile range. We followed the same procedures described in Step 3 (p. 7-3) to detect and eliminate outliers from the data set.

Step 4—Paired T-test: A paired-samples t-test was conducted to compare total scores in EPC PHAs during all years before and after an EPC was executed. In this test, we assumed the null hypothesis that “There is no difference between average total (PASS, MASS, FASS) scores for all years before and for all years after an EPC was executed.”

We conducted this test at a 95 percent confidence level. We also conducted paired-samples t-tests for each of the individual size groups: small, medium, and large. For non-EPC PHAs, we conducted paired-sample t-tests to compare total scores during all years prior to 2010 and al years after 2010. The null hypothesis proposes that “There is no difference between average total scores for all years before and after 2010.”

Findings

Exhibit 7-3 shows results that were observed in all the paired-samples t-tests conducted across PHAs at the overall level. The PASS scores showed a significant increase in pre and post when an EPC was executed. The non-EPC PASS scores, however, also showed a significant increase in pre- and post-2010. Management scores (MASS) for both groups declined significantly during the respective periods. 3 Financial scores (FASS) showed a significant increase in the non-EPC group but not in the EPC group.

Overall, there was no score improvement in the EPC group and a slight increase in the non-EPC group. From these data, we are unable to conclude that the use of EPCs 3 MASS scores are importantly affected by tenant occupancy rates. Other factors include Tenant Accounts Receivable and Accounts Payable. We did not investigate why MASS scores declined as they were not central to our investigation.

7-8

Analysis of Utility, Financial, and Physical Conditions Data improved a PHA’s physical or financial condition relative to PHAs that did not utilize them.

Exhibit 7-3. Pre-, Post-, and Non-EPC PASS, MASS, and FASS Scores

EPC PHAs

PASS (n = 112)

MASS (n= 114)

FASS (n = 111)

Overall score (n = 111)

Pre-

Post- Difference Pre- Post- Difference Pre- Post- Difference Pre-Post- Difference

mean mean

mean mean

mean mean

mean mean

0.80

0.83

4.010%a

0.88

0.83

−5.803%a 0.84 0.83

−1.402%

0.84

0.83

−0.596%

Non-EPC PHAs

PASS (n = 2,645)

MASS (n = 2,626)

FASS (n = 2,431)

Overall score (n = 2,611)

Pre-

Post- Difference Pre- Post- Difference Pre- Post- Difference Pre-Post- Difference

mean mean

mean mean

mean mean

mean mean

0.84

0.87

3.636%a

0.91

0.83

−8.776%a 0.84 0.92

9.084%a

0.86

0.87

0.233%a

a Statistically significant at the 95-percent level.

Financial Data: LMI also studied financial data extracted from the HUD Financial Data system. This system yielded the following information for each PHA:

• Total revenue

• Total expenses

• Net investment in capital assets

If EPCs have materially improved PHAs’ finances relative to non-use of EPCs, this improvement might show up in revenue less expense data, or in net investment data.

Data were available in two tabulations: before 2013 and after 2013. Data from both tabulations were integrated for all years.

Analyses: Prior to the analyses, the data were divided into two groups of PHAs, EPC

and non-EPC, and further by size: small, medium, and large. For each of the subgroups, we calculated:

Available funds = total revenue–total expenses.

We then plotted net available funds across PHAs during the years to observe whether trends emerged. Exhibit 7-4 shows available funds during the years since 2002, by PHA size class. For all the subgroups, we noticed decreases in available funds in the last few years.

7-9

Exhibit 7-4. Available Funds Over Time to EPC and Non-EPC PHAs

Data source: HUD Financial Data System.

7-10

Analysis of Utility, Financial, and Physical Conditions Data Further detailed analyses could be explored to compare available funds pre and post the year of EPC execution for EPC PHAs with pre- and post-2010 for the non-EPC PHAs.

We could use a similar paired-sample t-test to compare means in the pre- with post-time periods to determine if the effect of an EPC was significant or not. Our reading of PHA financial data, however, was that many factors other than EPCs heavily influence both available funds and net investment in capital assets, and we had lit le confidence that tests involving EPCs alone would yield meaningful results. If any conclusion could be drawn from exhibit 7-4, it is that the financial situation of all PHAs, whether they did EPCs or not and regardless of size, deteriorated during the latter years under study. The effects on HUD’s budget of federal budget sequestration occurring in 2013 and a few years after that probably were important reasons why this decline occurred.

7-11

Chapter 8

Relationship of the RAD Program to EPCs

Background

Since 2012, public housing authorities have been able to use an alternative source of private investment capital to upgrade units under their jurisdiction. As briefly explained in Chapter 2, the Rental Assistance Demonstration program converts PHA-owned properties to rental assistance under Section 8 of the United States Housing Act of 1937, which supplies project-based rental assistance or project-based voucher assistance over long-term contracts, typically 20 years. 1 These long-term commitments to subsidize tenant rents serve as a credit enhancement to attract long-term debt and equity from the sale of Low-Income Housing Tax Credits (LIHTCs) in order to attract capital to upgrade facilities and reduce backlogs of unfunded capital improvements. 2

Although Congress initially limited the RAD program to 60,000 units, in order to keep pace with growing demand, Congress has expanded RAD by more than sevenfold, to 455,000 units, as documented in exhibit 8-1.

Exhibit 8-1. Numbers of RAD Housing Units Authorized by Congress, by Year

Time

Enabling legislation

Ceiling on RAD units

Summer 2012

Further Continuing Appropriations Act, 2012

60,000

December 2014

Consolidated Appropriations Act

185,000

December 2016

Consolidated Appropriations Act

225,000

April 2018

FY18 Omnibus Spending Bil

455,000

As of October 2018, PHAs have generated over $7.6 bilion towards capital improvements across 123,000 converted units. An additional 150,000 units are in the process of converting (unpublished PIH data). Further, as of the beginning of 2018, around 300,000 units of the 1.1 mil ion publicly owned housing units were in the process of converting to RAD (HUD, 2018a). Of these housing units, about 57 percent were converting to Project-Based Vouchers and 43 percent to Project-Based Rental Assistance (HUD, 2018b). HUD’s expectation is that the number wil reach 600,000 by 2021 should the cap be further relaxed (HUD, 2018b). Given the rate at which Congress has expanded the program to date and its obvious desire to at ract private capital to upgrade the nation’s subsidized housing stock, the 455,000-ceiling likely will not be a permanent cap.

1 See for example https://www.hud.gov/RAD/program-details.

2 Two additional alternatives for accessing private capital for property improvements are not within the scope of this study. The Streamlined Voluntary Conversion program offers small PHAs with less than 250 units the ability to convert public housing units to assistance with Section 8 tenant-protection vouchers rather than project-based Section 8. Demolition and disposition authority under Section 18 also enables PHAs to reposition public housing for financial sustainability and access private capital.

8-1

PHA Options

Because the unit cap was greatly increased in 2018, many more PHAs wil be able to access public and private capital through the RAD program. Even as it is, more PHAs have undertaken RAD conversions than Energy Performance Contracts, however, a number have done both or undertook an EPC but later converted the properties to RAD

and bought out the EPC contract. Exhibit 8-2 provides information on the numbers of PHAs that have done one or the other or both. Of those shown as doing both, however, 60 had completed their EPCs by 2011, whereas the RAD program was only enacted in 2012. Thus, at most only 33 were doing both at the same time.

Exhibit 8-2. Numbers of PHAs Executing EPCs, Converting to RAD or Doing

Botha

PHAs that have done

or are doing EPCs

PHAs engaging in RAD

PHAs that have done both

260

476

93

a Data sources: Number of PHAs doing EPCs and the year of completion from data supplied by HUD in 2017. The database contains 315 separate EPCs but in 55 cases a single PHA had done more than one. RAD data obtained from HUD at http://www.radresouce.net/. The number doing both was obtained by comparing the PHA names in the two sets of data.

For a PHA deciding whether to convert to RAD, an ongoing EPC project complicates matters because of the debt already incurred and because it means having to manage two contracts at a given property simultaneously. Indeed, the very presence of ongoing EPC-related debt can discourage financiers of RAD projects from getting involved with a particular PHA in the first place.

The PHA might deal with this issue by refinancing the EPC debt as part of the RAD

conversion. That simplifies matters for managing projects and erases the debt, but may involve costly negotiations over terms of the buyout with the EPC contractor.

Given the present ceiling on the number of RAD units, some PHAs may seek to use EPCs for units that are not contemplated for RAD conversion in the near future. This situation is more likely to be the case for PHAs that on principle wish to retain their properties within the public housing program or where RAD rents would be inadequate to support the financing for holistic property improvements. The EPC program offers such PHAs an alternative private financing option.

Gains in Energy Efficiency from EPCs and RAD

An EPC contractor is repaid from savings in energy costs, which motivates contractors to make energy savings investments that are cost-effective during a set time horizon.

These investments may not be the most energy-efficient technology available because the contractor may view such technology as relatively untried or risky in some other manner, and unlikely to achieve an acceptable savings-to-investment ratio (SIR). Thus, some potential energy savings may be left on the table. Further, although cash-poor yet creditworthy customers are good clients for EPCs, these contracts are focused on energy-consuming systems and tend not to achieve synergistic energy savings that might be obtained with comprehensively retrofitted or rehabilitated properties.

8-2

Relationship of the RAD Program to EPCs

Under the RAD program, a PHA obtains investment in a property against both inherent land value and a stream of future payments to be derived from rental subsidies provided by HUD. Under RAD, PHAs have been able to achieve leverage ratios of as much as 9:1

(HUD, 2016). That is, for every $1 contributed by public housing funds, RAD is leveraging up to $9 from sources such as LIHTC, private mortgage debt, accrued land equity, grants, and other funding.

The RAD-enabled investments may include no energy efficiency investments, some energy efficiency investments, or practically all that are available. Thus, a RAD project may accomplish more or less energy efficiency than an EPC. A PHA and its consultants do not necessarily have the energy expertise of EPC contractors, and therefore might not identify as many promising efficiency candidates. Most of the significant capital improvement and housing replacement projects under RAD, however, have leveraged LIHTC markets. The LIHTC program does not dictate efficiency standards, but most states provide preferences or requirements within their Qualified Al ocation Plans (QAPs) to obtain the credits. A PHA thus often wil have to meet minimum energy and water efficiency standards established in the QAP if it chooses to make use of LIHTC tax incentives in attracting private capital.

HUD has provided both the EPC and RAD programs with incentive mechanisms to overcome situations in which residents pay part or all the utilities. Additionally, RAD

rents with project-paid utilities are set in a way comparable to the frozen rolling base provision of the EPC program. Such similarities suggest that the energy savings potential of the two programs does not dif er greatly.

Importantly, RAD transformations are often holistic rehabilitations or construction of new units. These changes enable such projects to focus on energy consumption over an entire building system, in which synergies among energy conservation investments can be realized. This achievement implies that RAD investments could achieve more energy efficiency than EPCs simply because they can realize available synergies, but the matter of which program actually achieves more is an empirical question.

Implications

The previous discussion suggests the following:

• Due to the demand for and success of the RAD program thus far, it is likely that this program wil continue and expand during the next few years.

• If PHAs near or reach the present 455,000 unit limit, Congress likely will increase it further.

• Because they would incur contractor debt and because they would not want to manage two separate rehab contracts rather than only one, PHAs contemplating whether to use an EPC wil be discouraged from doing so if they believe they wil be able to undertake RAD. This perception implies that the number of EPC

applications wil be reduced from what it was in the past.

• There wil , however, be a niche for EPC projects, for PHAs that on principle wish to retain their properties within the public housing program or where RAD rents would be inadequate to support the financing for holistic property improvements.

8-3

• EPC contractors wil invest in efficiency to the point where expected gains are equal to costs but wil leave further efficiency gains on the table if they believe the incremental costs exceed the incremental savings. RAD projects may contain any level of energy efficiency investment depending on the peculiarities of the units involved and the financial leveraging choices made by the local PHA.

Evidence

Qualitative: Stakeholder Views

Conference Participant Feedback

One member of the LMI team attended a RAD conference in Washington, D.C., in 2018.

This individual discussed the relationship of RAD to EPCs with other conference participants such as PHA officials, consultants, and financiers. Below is a summary of the feedback received:

• EPCs can impose complications for RAD financing. Therefore, if a PHA is considering RAD, it should not do an EPC at this point.

• Mature EPCs are easier to resolve in RAD. EPCs with remaining large debt, however, wil be subject to processing and underwriting requirements.

• If a PHA cannot use RAD because of the unit cap or the nature of its housing units, an EPC can be a good option for upgrading energy-related equipment.

• Most EPCs are based on SIRs, which can result in specifying energy system components that are above code minimum but not necessarily the most efficient equipment available because expected savings from more efficient equipment are not expected to cover incremental costs. Were HUD to require PHAs to comply with more efficient LIHTC QAP thresholds or with green building certification programs accepted by HUD, more energy efficiency could be achieved.

• The use of less efficient equipment in an EPC than that specified in QAP

requirements may necessitate early equipment replacement if the sponsoring PHAs later seek to use RAD and LIHTCs to access private capital. Such early retirement is in many cases economically wasteful and could be avoided by requiring PHA compliance with LIHTC efficiency thresholds for future EPCs.

• A recent article, “The Intersection of EPC and RAD: A Roadmap for PHAs,”

(Bordenave et al., 2016) concluded that converting a project through RAD after it has done an EPC is better than undertaking an EPC alone. The study also concluded that, in 80 percent of tested scenarios, it is better to undertake a new energy project via RAD than through an EPC.

Survey Conclusions

A section of the LMI team’s online survey asked PHAs that had not done EPCs to give reasons why they chose not to do so. One of the main answers was that other options, such as RAD, were a superior choice for them. The most prevalent answer given to this question, however, was that an EPC was not cost-effective from their perspective. In such cases, RAD stil may have been an alternative because its economic basis is different from that of EPCs, but it was not the main reason PHAs did not use an EPC.

8-4

Relationship of the RAD Program to EPCs

Follow-up Telephone Interviews with Survey Participants

One of the main conclusions drawn from our telephone interviews with 20 PHAs is that many, if not most, use RAD as their vehicle of choice to tap into private funds to upgrade their housing units. In several cases, the PHAs interviewed were in the process of switching from the EPC program to RAD, buying out their EPC contractor in the process, and in others, they had compared RAD with EPCs and decided they would go with the former.

The main reason given was the broader funding scope of RAD. PHA directors looking to upgrade more than their energy equipment find RAD is better suited to their needs than EPCs, whose investment objectives are more limited. Some PHAs also indicated that RAD is easier to use because paperwork requirements are less. One interviewee stated that from their perspective, HUD was no longer emphasizing the EPC program so that RAD now implicitly seems to be HUD’s outside funding vehicle of choice.

Quantitative: EPC Approvals

Exhibit 8-3 shows the number of EPCs approved by HUD in each fiscal year between 2011 and 2018. 3

Exhibit 8-3. EPC Approvals by Year

Fiscal year

EPC approvals

2011

19

2012

20

2013

26

2014

7

2015

21

2016

12

2017

14

2018

10

Although one cannot draw strong conclusions from a limited set of years, it seems evident that the number of EPC approvals has been decreasing, with the 2016–2018

numbers only a little more than one-half those of 2011–2013. Although there could be many reasons for such a decline, it is consistent with the proposition that RAD has impacted the level of EPC activity.

3 For years 2011 through 2016, the Energy Center provided EPC project completions by year. For those years, we assumed the project applications were approved in the year before the projects were completed. For 2017 and 2018, the Energy Center provided us actual numbers of HUD approvals.

8-5

Chapter 9

Conclusions and Recommendations

Conclusions

This study has focused on two basic questions asked by HUD.

1. Are the utility, financial, and physical conditions of public housing authorities that have executed an Energy Performance Contract better relative to PHAs that have not executed an EPC?

2. How much of an impact do the implementation factors of existing EPCs have on the program’s adoption rate?

To address these questions, the LMI team used four dif erent methods of acquiring data.

These approaches included an online survey, follow-up telephone interviews, examination of administrative data, and assessment of the effect of the Rental Assistance Demonstration on the EPC program. From the results obtained, the LMI team reached the following conclusions:

1. Statistical tests performed using HUD Utility Expense Level data showed that PHAs that have implemented EPC projects have been able to cut their electricity consumption by more than twice as much as PHAs that have not (10.3 percent as compared with 4.4 percent). This conclusion is backed at a 95-percent level of statistical confidence.

2. Tests using UEL data for EPC users showed electricity consumption declined at a significantly greater rate in the years after an EPC was performed than in the years immediately before. This result held at a 95 percent level of statistical confidence for all such PHAs as well as for small and large ones, but only at a 90-percent level of confidence for medium-sized PHAs.

3. Analysis of PHA survey data indicates those that performed EPCs reduced their use of natural gas and water more than non-EPC users, at a 95-percent level of statistical confidence. Survey data indicated that EPC-using PHAs reduced their use of electricity and fuel oil by more than non-EPC PHAs as well, but those results were not statistically significant.

4. Survey data suggest that, during the past several years, those PHAs that have used EPCs have improved their financial condition by somewhat more than those that have not. Surveyed EPC-using PHAs also indicated that the financial condition of units included within the EPC improved more than their other units.

These findings were not conclusive at a 95-percent level of statistical confidence, however. They also could not be confirmed through examination of administrative financial data.

5. PHAs that executed EPCs reported greater improvements in the physical conditions of their properties than PHAs that did not. The dif erence in reported improvements between the two samples was significant at a 95-percent level of statistical confidence.

9-1

6. Approximately 60 percent of PHAs that had done at least one EPC indicated they were favorably disposed toward doing another. Because at least some of the other 40 percent that had done at least one could already have accomplished all the energy investments they considered cost-effective, it is likely that more than 60 percent found the EPC program useful

7. According to survey evidence, about two-thirds of PHAs that had done an EPC at some units but not others indicated that the EPC resulted in lower utility consumption and better physical and financial conditions than occurred in their non-EPC units. These responses furnish a strong rationale for why most of the EPC-using respondents expressed a wil ingness to do another.

8. The RAD program is being used by a large and growing number of PHAs.

According to interview data, the RAD program offers a wider application of monies to housing units’ physical needs and is easier to negotiate than an EPC.

The annual number of EPC approvals has declined over time as more PHAs have turned to RAD.

9. Few PHAs are likely to undertake both RAD and an EPC. Indeed, PHAs that have undertaken RAD often have bought out their EPC contracts.

10. Of those PHAs that have never undertaken an EPC, the most important reason given was that such a project would not be cost-effective from their perspective.

Other leading reasons were that the process is too complicated or that RAD is a better option.

11. Of those PHAs that started but did not complete an EPC, the main reason given was that they did not have sufficient staff to see the project through. Another reason was that HUD’s EPC process is too cumbersome.

12. Interviewed PHAs indicated that HUD’s EPC program should be among the options available. Interviewees suggested, however, that because many PHAs, particularly smaller ones, do not fully understand the program, do not have the staff to undertake one, or do not know how to deal effectively with contractors, more technical support would be useful.

13. Survey respondents and interviewees identified several means by which HUD

could promote the use of EPCs. These methods include continued

encouragement to PHAs to undertake the program, direct subsidies, a streamlined application process, and greater technical assistance.

Recommendations

The LMI team offers the following recommendations:

1. Because there is strong statistical evidence that EPCs have helped PHAs to improve their units’ energy efficiency, the program appears to be accomplishing one of its principal purposes. If gains in energy efficiency remain an important HUD goal, the EPC program should be maintained if not strengthened.

2. The program offers PHAs a means to tap into private capital markets. The RAD

program also does so, but not all PHAs are interested in RAD or able to take advantage of it. For these PHAs, it is important to maintain the EPC program as an option.

3. According to some interviewees, HUD is no longer emphasizing its EPC program to PHAs to the extent it once did, and a number of PHAs are skeptical of 9-2

Conclusions and Recommendations

undertaking it. These interviewees assert that increased communication regarding the program, particularly with respect to lessons learned and how to deal with contractors, would lead more PHAs to consider the program. In addition, several interviewees believe that more financial and technical support, particularly directed toward smaller PHAs that seek to self-manage EPCs, likely would increase participation.

4. We did not compare utility savings achieved by RAD-using PHAs with those using only EPCs. A possibility exists that RAD-related savings might be greater because RAD transformations are often holistic rehabilitation or construction of new units in which synergies among energy conservation investments can be realized. The comparison should be made to determine whether the RAD

approach achieves as much if not more energy ef iciency than the EPC

approach.

5. We did not examine whether who paid utility bil s between a PHA and the tenants made a difference in the PHA choosing to do an EPC. HUD’s Resident-Paid Utilities Incentive program (RPU) is intended to deal with this issue, but the extent to which PHAs have taken advantage of the incentive is unclear. Even with the incentive, however, PHA payment of utilities may provide greater inducement to undertake an EPC than when tenants pay the bil s for individually metered utilities. This issue should be examined by HUD to determine whether PHAs with partial or full tenant utility payments are less inclined to undertake EPCs.

9-3

Appendix A

Responses to Survey Questions

This appendix presents the results of the online survey. For each question, the relative frequency of responses is shown in graphical form. Where applicable, responses for Group 1 respondents are shown in blue, those for Group 2 in yellow, and those for Group 3 in red. Most questions in the survey were directed at one or another of these three groups although a few were directed at all of them. The color coding is meant to enable the reader to see at a glance which of the three responded to that particular question. In each case, the number of the question and the number responding is shown. Certain questions are grouped together and listed more than once because their responses were used to compare Groups 1 and 3 with one another.

A-1

A-2

Responses to Survey Questions

A-3

A-4

Responses to Survey Questions

A-5

A-6

Responses to Survey Questions

A-7

A-8

Responses to Survey Questions

A-9

A-10

Responses to Survey Questions

A-11

A-12

Responses to Survey Questions

A-13

A-14

Responses to Survey Questions

A-15

A-16

Responses to Survey Questions

A-17

A-18

Responses to Survey Questions

A-19

A-20

Responses to Survey Questions

A-21

A-22

Appendix B

Summary Notes from Survey Follow-up

Telephone Interviews

Group 1: Have Done an EPC

Smal PHAs

Small 1

• Public housing authority conducted two Energy Performance Contracts that resulted in enough savings to cover their costs.

The first EPC was self-managed and the second was done in conjunction with several other PHAs through an energy service contractor.

The EPCs have resulted in the improved financial condition of the PHA.

PHA does not need more EPCs. The only remaining improvements would have a very low payback.

• PHA has also done other energy conservation work using programs through its utility company.

Completed weatherization of units at no direct cost to the PHA and was able to remove that from the EPC project.

• PHA has converted to the Rental Assistance Demonstration, which is viewed as a financial and administrative change that does not affect the day-to-day operations.

The first EPC was paid off before converting to RAD, and the second was bought out through the RAD process. Buying out the EPC went smoothly and did not present any issues.

• Recommendations for improving the EPC program:

Provide training to the tenants. Tenant behavior has a big impact on how successful the EPC can be.

Education or assistance for the PHA with the procurement process.



When procuring an energy service contractor (ESCo), PHAs need to know how to get the best value for the work being conducted by the contractor.



ESCos may have preferred bidders and there could be opportunities to save cost by requiring the ESCo to expand the bidder pool.



Opportunities can be found through thorough questioning of the ESCo bidding process and the PHA being well informed. The EPC wil be paid off faster if expenditures are limited from the beginning.

B-1



Those administering the EPC at the local level need to question and be aware of exactly what the ESCo is offering and realize they are also trying to make a profit.

Account for distribution cost in savings calculations because utility distribution costs can increase faster than commodity costs.

PHAs need to be very dedicated to make sure everything in the EPC process is done as efficiently as possible.

Small 2

• The EPC process was started in 2005 and was completed through an ESCo.

Savings were enough to cover the cost of debt with some extra left over.

Had some issues after the EPC installation with the expected level of continued tenant education.



Education of tenants is important because they drive the level of consumption.



Continuous tenant education was included in the EPC contract but did not meet expectations.

• Believe that the EPC program has been more or less taken over by the RAD

program.

Unsure if the two could be used together. The financing and details would have to be discussed very early in the project to figure out if they would work together.

• Recommendations for improving the EPC program:

Tenant education because tenants need to be aware of what the EPC is doing for them, especially if they pay utilities.



Education on how to conserve energy and be more efficient.



PHA used an education awareness team that employed young people

from the community to help educate tenants.



PHA included an Energy Fair in the EPC contract to put on an

educational event for the tenants after the EPC installation to help familiarize tenants with new products.

Small 3

• The EPC was conducted in 2000 as a 12-year contract.

Believe EPC went smoothly and did not feel that being a small PHA presented any additional challenges with completing the EPC.

Energy and water savings exceeded expectations.

Had issues with the Energy Information and Performance Center system (online platform) when that was implemented.

View having an EPC as a potential hindrance when updating and transferring the portfolio to RAD.

B-2

Summary Notes from Survey Follow-up Telephone Interviews

• PHA did not extend the existing EPC and would not consider another EPC

because they have converted to RAD.

Able to do more wholesale improvements under the RAD program.

Have done modernization under RAD which has included energy savings.

Believe that PHAs can choose RAD or an EPC but not both.

• Recommendations for improving the EPC program:

The interviewee did not directly work the process of obtaining an EPC but believes it could be more streamlined.

More technical expertise could be provided.



Field office could not provide the assistance needed when the EPC was executed.

The EPC and RAD programs need to be more coordinated. The programs come out of different offices in HUD but they are not communicating among themselves.

Medium PHAs

Medium 1

• The PHA has an EPC but is currently converting to RAD and working to buy out the EPC.

Unlikely to do another EPC because of the RAD conversion.

Had considered another EPC in the past but struggled to find enough savings to make it worthwhile because the PHA is newer than other PHAs (1970s).

Under RAD, the PHA will be able to do more deep energy efficiency projects.



RAD wil fast track about 10 years of deep renovations, including cosmetic and exterior in addition to energy savings upgrades; these renovations would not be possible under an EPC.

• Recommendations for improving the EPC program:

Many PHAs do not want to do an EPC because of lack of understanding of the program.

HUD needs to provide a more holistic and organic approach to explain the importance of EPCs and dispel the myth that EPCs are a scam.



EPCs are important because RAD is not working for a lot of authorities and there are a lot of opportunities for savings through an EPC.



Suggests using social media or podcasts to broadcast success stories of the EPC program.



PHAs fear that tenants wil complain about disruptions during the implementation of an EPC and that wil lead to disciplinary action from HUD.

Provide incentives for tenants to recompense above-average disruptions.

B-3

Medium 2

• The EPC was completed in 2012 but the director who executed it is no longer at PHA.

Feel like they are stil paying for improvements that they are not get ing much benefit from anymore.

Tenants get most of the benefits of the savings from the EPC.

Would consider an EPC again if it would improve housing for the tenants.

• Considered RAD but could not get funding through the state to convert.

• Recommendations for improving the EPC program:

PHAs need to be sure that when implementing an EPC they are getting the most value for the contract and a guarantee that improvements made will have a longer life than the length of the contract.

Large PHAs

Large 1

• Completed one EPC and currently undergoing a second EPC which is larger in scale and value.

The first EPC was as conservative as possible to reduce risk.

Water cost savings greatly exceeded expectations because of large increases in water rates, enabling the PHA to expand the EPC.

Al utilities are owner paid so the PHA received all the benefit of the savings.

The EPC is self-managed.

• RAD was considered and led to a pause in the second EPC project.

EPC expansion was limited to only a couple of projects before they were converted to RAD.

For others, private equity that would be invested was greater under the EPC

than under the RAD program.

RAD is complicated and how it integrates with EPCs is a further complication.

Also, it can take a long time to achieve RAD conversion (at least 5 years).

EPCs were structured so that they could be paid out under RAD if that decision is made in the future.

• Recommendations for improving the EPC program:

Improve understanding of how the savings are passed through to the PHA from HUD.



Many PHAs think that with the utility savings, HUD will provide less money for utilities, meaning the PHA does not get the benefit of the savings while stil having to pay off the EPC.

PHAs need to understand how to capture operating and maintenance (O&M) savings through an EPC—O&M savings are not always included in the EPC.

B-4

Summary Notes from Survey Follow-up Telephone Interviews PHAs also think HUD wil take away unneeded O&M funds resulting from an EPC.

Large 2

• The EPC was completed by 2008 and covered most of the units in the PHA.

Electricity is tenant-paid whereas water is owner-paid, with most of the EPC

savings coming from water and the PHA getting most of the benefit from the savings.

• The PHA started a second EPC but did not finish it because of converting to RAD.

Many of the improvements needed were not energy-related.

Saw RAD as a stable funding option moving forward (with unknown future of capital funding).

Paid off the EPC contractor for some of the RAD properties while holding off on converting others to RAD because energy savings would be the main component.

If the PHA was not doing RAD, it would probably do another EPC but the upgrades needed now do not have energy savings associated with them.

• Recommendations for improving the EPC program:

Address the stigma that an EPC is taking on debt. Leadership at many PHAs wil refuse to look at the EPC program because it does not want to take on debt.

The EPC program is no longer on the forefront of communications. It seems as if the program has come and gone.

Guide PHAs on how to weigh the options between RAD and an EPC.



RAD is not a good fit for many authorities and HUD needs to

communicate to PHAs that if they are not doing RAD, they should

consider an EPC.

Streamline the process for small PHAs—have a simpler process if an EPC

applies to less than a certain number of units.

Train the field office staff so that they understand what an EPC is when PHAs ask questions.



In some cases, field offices are unable to promote the EPC program because of lack of understanding.

Large 3

• Completed an EPC that covers about 99 percent of the housing portfolio.

Did not have any real issues with the project set up or implementation; partnered with other PHAs in a self-managed EPC.

Getting residents and maintenance staff behind the project took some effort.



Tenants complained about low-flow fixtures.

B-5



Some of the equipment was complex and required training for

maintenance staff.

Used the Energy Center for the second phase of the EPC and thought it provided a good resource and was very responsive.

Made further capital improvements from the savings of the EPC and freeing up funds to do other improvements that do not have as much utility saving payback.

• The PHA considered RAD but decided it was not a good fit; may reconsider in the next year.

EPC and RAD could be done together but would require more research into the programs.

• Recommendations for improving the EPC program:

HUD needs to provide more clear guidance between the RAD and EPC

program and the potential benefit of each; programs need to be better integrated.



Pressuring PHAs to use RAD or an EPC can put them in analysis

paralysis and they end up not taking advantage of either program.



The RAD program causes PHAs to have second thoughts about the EPC

program.

Have the subsidy from the EPC go directly to the bank or lender instead of through the PHA; this approach would help get more lenders on board with the EPC program.

Engage residents and maintenance staff early in the EPC planning process.

Group 2: Started but Did Not Complete an EPC

Smal PHAs

Small 1

• Did not finish the EPC; decided to go with RAD. Al changes that would have been included in the EPC were incorporated into the RAD project.

The EPC would not have been able to address all the changes needed, only small changes.

RAD provided more economic benefits.

Utilities are tenant paid and tenants love the renovations and utility savings.

• Could not provide any recommendations for improving the EPC program.

Small 2

• Did not finish the EPC because it could not address the full needs of the project and was able to convert to RAD to accomplish the tasks planned for in the EPC

along with other infrastructure needs that would not have been covered by the EPC.

B-6

Summary Notes from Survey Follow-up Telephone Interviews

The EPC process was a lot of paperwork and more of a repair than addressing the full needs of the buildings.

Did not see the value of completing an EPC for a small PHA.

• Strong preference for the RAD program.

Much broader and able to bring housing to a higher standard immediately (eliminated $80 mil ion worth of deferred maintenance).

Addressed all needs, not only energy.

• Recommendations for improving the EPC program:

HUD would need to provide a large subsidy for completing an EPC. The RAD

program is a much better option.

Small 3

• The PHA started the EPC process in 2009/2010 working with an ESCo and does not know why the EPC was not completed.

• The PHA was slightly likely to try an EPC again but the RAD program is definitely something it is going to pursue (starting to look into it).

EPCs are a challenge for a small PHA with lit le staff and funding.

• Recommendations for improving the EPC program:

Need more direct support from HUD, to the point of HUD coming in and executing the EPC. As a new executive director, little was known about the EPC program and the program was not understood.

Medium PHAs—Medium 1

• Did not finish the EPC because the economics were not there because utilities are tenant-paid. The PHA would not have benefited from energy savings. If the utilities were owner-paid, it would have made a huge difference.

• The PHA is currently looking into RAD and would plan to achieve energy savings through that program.

• Could not provide any recommendations for improving the EPC program.

Large PHAs—Large 1

• Did not complete the EPC because two of the larger projects included in the EPC

were being identified for redevelopment in the transformation plan.

Considered using RAD to execute the transformation plan but determined that the program was not right for these projects.

Did not have anything against the EPC program but it did not make sense considering RAD was being pursued for the remaining projects included in the EPC.

B-7

The EPC would have only covered water conservation, now either project is going to be redeveloped or RAD improvements have covered both water and energy improvements.

Felt that the PHA was bearing most of the risk and the ESCo was not.

• Recommendations for improving the EPC program:

Provide more training and education about EPCs to the field offices.

Received a lot of information about EPCs from the ESCo although the field office did not have a good sense of how an EPC worked.

Provide guidance on how to manage a third party (ESCo).



What is a good deal, what should be expected, what is standard, and so on.

Group 3: Never Did an EPC

Smal PHAs

Small 1

• The PHA looked into an EPC but decided against it because the PHA had enough available cash from capital funding to implement needed energy-saving measures.

Installed water meters and low-flow showerheads/aerators, replaced toilets as needed (estimate 80–90 percent low flow now), capped outside spigots, changed to LED lighting as needed, high-efficiency windows, and so on, and units are well insulated.

Have 198 units and do not have any additional energy conservation needs that would warrant an EPC.



Lack of return on investment.



The administrative burden would be too high for a small PHA; more effective to implement energy conservation measures on their own than to complete all the paperwork required for an EPC.

• PHA converted to RAD last year, closing in November 2017.

Capital fund alone was enough to do what has been done for energy but not enough to do all other desired improvements at the PHA.

Al owed them to get out from under unfunded mandates of public housing.

After the first year, expect to already have excess funds for other improvements.

• Recommendations for improving the EPC program:

Al ow for direct payments so that PHA can act as its own contractor and use direct funding to do all the units at once. Would prefer hiring temporary employees or using housing staff to carry out an EPC because of bad experiences working with contractors (for example, using less than spec equipment).

The administrative burden would be too high for a small PHA.

B-8

Summary Notes from Survey Follow-up Telephone Interviews

• Looking into the EPC program helped the PHA realize the energy improvements they could make and at least opened eyes to what they could do to save energy for them and their tenants.

Small 2

• Did not know about the EPC program or if anyone had looked into it in the past.

Have completed other energy investments (new heat pumps, insulation, windows, and so on) but have not done any analysis of resulting savings.

Units are in good condition, have been managed well, and money spent wisely.

• Looked into the RAD program but prefer to keep units as public housing.

• Does not have any problems with the programs coming out of HUD.

Medium PHAs

Medium 1

• PHA looked into the EPC program about 10 years ago but decided against it because of adequate funding through normal capital funds.

Energy savings investments incorporated into normal capital projects.

Not opposed to the EPC program but have not looked into it more recently.

• PHA had attempted the RAD program but the deal fell through due to contaminated soil on the property.

Not considering RAD again because capital funding has been enough (even increasing).

Very disappointed by the amount of time spent on RAD that ultimately was not successful.

• Recommendations for improving the EPC program:

Simplify the process and provide more assistance. The PHA’s administrative staff is barebones and do not have time to review the process and consider whether it is worth the investment.

The Energy Center should do as much upfront work for the PHA as possible so the PHA does not have to take on the full administrative burden; anything to reduce the administrative burden for small and medium PHAs would help.

Medium 2

• Unsure if the PHA has looked into the EPC program in the past because there has been complete management level turnover within the last year.

The only experience with EPCs was from talking to other PHAs that were buying out their EPCs prior to converting to RAD; does not know much more about the EPC program.

PHA is looking to convert to RAD and redevelop properties within the next 5 years so would not consider doing an EPC.

B-9

Experience is that people do not understand EPCs if they do not already have institutional knowledge of the program at the PHA.

• Recommendations for improving the EPC program:

Provide more communication, education, and training about the EPC

program. PHAs are always looking for ways to improve their housing and for new sources of funding.

Medium 3

• The PHA has not looked into an EPC because it does not have the staff to review the process; EPCs have been on the radar for a long time but the PHA does not have the resources to research.

Has completed other energy projects with the local energy company (grant type program) to improve insulation.

Incorporates energy efficiency into any project that is conducted through a normal budget.

• Working toward RAD for about 25 percent of units—has been an attractive option because units are very old and have high maintenance costs.

• Recommendations for improving the EPC program:

Providing an operating subsidy would help get an EPC up and running.

Starting an EPC is time-consuming and the PHA would need to take someone away from their regular responsibilities.

Large PHAs

Large 1

• Looked into the EPC program and is currently interested in figuring out how to incorporate an EPC as a component of RAD work being done.

Anticipates the RAD program to take a long time but could do an EPC in projects scheduled further out.

Going to try to combine RAD tax credit with an EPC for a building that only needs renovations (not complete teardown and rebuild).

• Recommendations for improving the EPC program:

Improve communication and exposure of the EPC program; many PHAs have heard about the program but do not understand the “how.”

Develop and implement training and tools/provide technical assistance to help PHAs take on a project as sophisticated as an EPC.

Provide help to smaller PHAs that do not have the resources to do anything other than manage their properties (that is, like looking into programs such as EPC and RAD).

B-10

Summary Notes from Survey Follow-up Telephone Interviews Large 2

• Considered an EPC in 2008 and completed training. Evaluated capacity of staff and determined that the PHA did not have the capacity to take on the project.

Talked to other PHAs and found that—

some were disappointed by the monetary benefit of the program, and

in-house maintenance staff had a hard time understanding new equipment installed during an EPC.

• The PHA has completed energy conservation projects through normal operating funds or other grants/subsidy money.

Includes energy improvements when doing any project.

Used grant funding to do bathroom improvements.

Have to consider costs and potential savings when incorporating energy efficiency into projects (for example, in replacing water heaters, the PHA went with a less efficient model because savings from more efficiency would have never generated the payback needed to justify the additional cost).

• Working to convert to RAD—have the application submit ed.

Currently only for part of the PHA, but if it goes well, wil probably go RAD for all units.

It is hard for a PHA to manage more than one investment program at once (that is, RAD and EPC).

• Recommendations for improving the EPC program:

Provide technical expertise—it is hard finding and hiring qualified personnel or contractors to fulfil needs; PHAs could use direct help with building system-level technical expertise to be able to oversee an ESCo.

B-11

Appendix C

UEL Data Analysis

To supplement our survey analysis, Utility Expense Level data were reviewed to determine if there is evidence of decreased utility consumption when comparing before and after an Energy Performance Contract EPC implementation at a public housing authority. Utility consumption was also evaluated for non-EPC PHAs, comparing consumption before and after an established base year. To undertake this analysis, a good deal of data cleansing was necessary. This appendix explains the data cleaning and analysis process as well as data limitations and potential ways to include more of it.

Source Data and Cleaning

UEL data reports were available for funding years 2008, 2011, 2012, 2013, 2014, 2015, 2016, and 2017. Each UEL data report contains actual annual consumption data for various utilities (energy, water, etc.) at the Asset Management Project level.

Consumption data reflects the year prior (months 7/1/20XX through 6/30/20XX) to the funding year (that is, the funding year 2008 UEL data contains consumption data for 7/1/2006 through 6/20/2007) as well as the 3 years prior.

UEL data review was limited to data for PHAs that were included in the EPC survey sample. Sample PHA UEL records for each available year were combined into one file and data was scrubbed for inclusion in the analysis. Data cleaning steps included—

• review of utility type and unit of consumption. Where possible, units or utility type were corrected.

• standardization of unit and utility type nomenclature.

• exclusion of flat-rate consumption.

• exclusion of duplicates.

• exclusion of AMPs without data records for each available year of data. Because data was rolled up to the PHA level, AMPs with only some years of data had to be excluded because they would have skewed utility consumption (that is, showing an increase/decrease because of their addition or removal in a given year and not because of actual changes in consumption).

Based on our assessment of the UEL data, we determined that only electricity data could be included in the statistical analysis. Exhibit C-1 provides an explanation.

C-1

Exhibit C-1. Data Modifications Made

Utility type in UEL data

Data review explanation

Electricity

Determined useable records via data cleaning steps.

Gas

Data quality issues noted in addition to data cleaning rules:

• Inconsistent units (100 cubic feet; 1,000 cubic feet; cubic feet; flat rate, gallons, kWh, mbtu, other/unspecified, decatherm, lbs,

therms) for AMPs year over year.

• Unable to determine the correct reporting unit. Excluded AMP

data if unit changed year over year.

• Data that had all 8 years of records recorded in the same unit of measure but with extremely wide ranges of year-over-year

consumption. For example, if consumption increased by an order

of magnitude, we took it to mean that the reporting unit in at least one of the years likely was incorrect.

• Unable to determine whether records were accurately reporting natural gas in appropriate units without additional information

about the building(s). (That is, gross square feet to compare with the Department of Energy’s Commercial Building Energy

Consumption Survey).

Only 36 EPC PHAs had sufficient data available for analysis after excluding outliers and PHAs without EPC year.

Fuel Oil

Inconsistent units (flat rate, gal ons, Therms, other/unspecified) and limited PHAs with fuel oil records (50 PHAs total before excluding based on data cleaning steps).

Propane

Only 15 PHAs total with propane records, many AMPs without

records for each year.

Lighting listed in UEL data as

Majority of data reported as a flat rate. A few records that were not Security Lights, Other (Common

reported as flat rate included a very limited number of AMPs (less Area Lighting), Other (Street

than 15 AMPs) that had sporadic annual records (for example, 1–2

Lights), Other (Site Lighting), Other years of data for each AMP).

(Street Lighting), Other (Outdoor

Lighting), Other (Outdoor Lights)

Diesel listed as Other (Diesel) and 3 records: 1 year for one AMP and 2 years for another AMP.

Other (Diesel Fuel)

Steam

87 records, unable to determine unit (other, 1,000 lbs, lbs, 100 cubic feet, MWH, kW, flat rate, BTU)

Coal

Records for only one AMP.

Other (Solar)

Unknown utility type. Few (23) records with varying unit—kWh,

Therms, flat rate, cubic feet.

Other (Stove Gas)

Records for only one PHA.

Water Records

Sewerage and Water (water only, if Unable to compare water records because of varied reporting at the sewer reported separately)

AMP level as water/sewer combined, water only, or sewer only year to year. Unable to determine if consumption reflected water use, water use plus sewer, or just sewer in UEL records due to data

quality.

Water and Sewer (if combined)

Same as above.

Water (if reported separately)

Same as above.

Sewer (if reported separately)

Same as above.

Sewer

Same as above

C-2

UEL Data Analysis

Exhibit C-1. Data Modifications Made

Utility type in UEL data

Data review explanation

Other/Fees listed as Other (Waste Flat rate utilities.

Water Treatment), Other (Storm

Drainage Fee), Other (Storm

Drainage), Other (Stormwater),

Other (Stormwater Fees), Other

(Sprinklers), Other (Stormwater

Utility)

Other/Unknown Utility Type Records

0

Unknown utility type.

Other (specify)

Unknown utility type.

Other

Unknown utility type.

Select a Utility Type

Unknown utility type.

Other (Fire Protection), Other (Fire Flat rate utility.

Service), Other (Fire Line)

Other (Boiler treatment)

Flat rate utility.

Other (FLAT RATE)

Unknown utility type/flat rate utility.

44

Unknown utility type.

Other (Landfil User Fee )

Flat rate utility.

Electricity Data Analysis

After excluding unusable records from the analysis, electricity UEL records were expanded to include additional years based on the three previous years of available data for each funding year record. Consumption for years 2004, 2005, 2006, 2008, and 2009

was able to be added for AMPs where the base years were listed as non-frozen.

Electricity consumption at the PHA level was analyzed for two groups: PHAs that have performed a PHA (survey sample “Group 1”) and PHAs that have not completed an EPC

(survey sample “Group 2” and “Group 3”). Electricity consumption data were summed to the PHA level and average consumption calculated before and after the EPC execution or a fixed-base year for non-EPC PHAs. For the non-EPC PHA analysis, 2010 was chosen as the fixed base year. Where a PHA had completed more than one EPC, only the first execution year was used in the analysis.

The analysis was performed as follows:

• Electricity consumption for all years before and after the EPC/base year was averaged at the AMP level.

• Average AMP level consumption was summed to the PHA level.

• The percent change in consumption was calculated for each PHA.

• Outliers were determined using interquartile ranges of the percent change.

• Final results show the percent change in electricity consumption after an EPC/base year for all PHAs and broken down by size (small, medium, large) by taking the average of the percent change in consumption at each PHA.

C-3

• Levels of statistical significance were calculated for dif erences in reductions in electricity consumption between EPC and non-EPC PHAs. These levels were reported in the text in Chapter 7, exhibit 7-2, and are also shown in exhibit C-1.

Exhibit C-2. Results of Statistical Analysis

C-4

UEL Data Analysis

Data Limitations

Many limitations and uncertainties exist with the UEL data analysis. For this reason, UEL

data was supplementary to the survey data collected from PHAs. Known limitations include:

• unknown sizes (gross square footage) of building(s)/AMP year over year;

• differing data availability for before/after an EPC or variable base year (for example, EPCs conducted in 2005 wil have limited data before the EPC.

Likewise, EPCs conducted in 2004 or prior wil have no data before the EPC and could not be included in the analysis);

• limited utility types analyzed based on data cleaning rules;

• missing EPC implementation year in EPC data, limiting the number of EPC PHAs that could be included;

• unknown if UEL data accounts for all utility consumption (that is, some utilities may be fully or partially tenant paid); and

• unknown which or how many AMPs affected by EPC implementation.

Further Analysis

Because the main source of data for this project was intended to be a broad survey of PHAs, we were limited in how much UEL data we were able to access, cleanse, and analyze. Further efforts to work with UEL data might yield a larger sample and a more robust analysis. These ef orts could include:

• refine data cleaning rules to be able to include more records and utility types (natural gas, fuel oil, water, and so on) in analysis;

• continue to fil data gaps further with data about 3 previous years provided in each UEL report;

• couple analysis with other HUD utility data that may be available (for example, utility allowance calculations from form HUD-52667 and PHA’s per unit month project expense level calculations on form HUD-52723);

• include more PHAs that have conducted an EPC by working with PHAs and the HUD Energy Center as well as using survey data collected to fil in EPC year (current data gap for many EPC PHAs);

• further analyze the impact of multiple EPCs conducted at a PHA;

• expand UEL analysis to PHAs not included in the survey sample;

• evaluate how outliers are excluded. For example, outliers were excluded at the PHA level, further analysis may be possible if outliers are instead excluded at the AMP level; and

• if available, obtain gross square footage data about PHAs to perform analysis based on energy use intensity.

C-5

Appendix D

Maps of Regions and of PHA Populations

D-1

Note: There was no PHA selected from Alaska in the sample.

D-2

Appendix E

Abbreviations

AMP

Asset Management Project

CFR

Code of Federal Regulations

EPC

energy performance contract

ESCo

energy service contractor

ESPC

energy service performance contract

FASS

Financial Assessment Subsystem

FEMP

Federal Energy Management Program

HUD

U.S. Department of Housing and Urban Development

IQR

inter-quartile range

LIHTC

low-income housing tax credit

MASS

Management Assessment Subsystem

O&M

operating and maintenance

PASS

Physical Assessment Subsystem

PHA

public housing authority

PHAS

Public Housing Assessment System

P.L.

Public Law

QAP

Qualified Al ocation Plan

RAD

Rental Assistance Demonstration

REAC

Real Estate Assessment Center

RPU

Residents-Paid Utility Incentive program

RR

response rate

SD

standard deviation

SIR

savings-to-investment ratio

UEL

Utility Expense Level

E-1

References

Bordenave, J., R. Santangelo, S. Morgan, and M. Nail. 2016. The Intersection of EPC

and RAD: A Roadmap for PHAS. Journal of Housing & Community Development 73 (2): 27–34. https:/ ssfengineers.com/wp-content/uploads/2016/04/JHCDMarApr16-3-28-16-

Pivotal-Point-Apartments-EHA-ARC.pdf.

U.S. Department of Housing and Urban Development (HUD). 2018a. Rental Assistance Demonstration. Washington, DC: HUD.

https:/ www.hud.gov/sites/dfiles/Housing/documents/RAD_100000_Homes_20180813.pdf.

———. 2018b. RAD Talk. Washington, DC: HUD. RAD in Review.

https:/ www.hud.gov/sites/dfiles/Housing/documents/RAD_Jan2018_Newsletter_013118.pdf.

———. 2018c. Notice PIH 2018-20—Partnering with Utility Companies on Energy Performance Contracts. Washington, DC: HUD.

https:/ www.hud.gov/sites/dfiles/PIH/documents/PIH_2018-20_UPP_Notice.pdf.

———. 2016. Interim Report—Evaluation of HUD’s Rental Assistance Demonstration.

Washington, DC: HUD’s Of ice of Recapitalization: iii.

https:/ www.huduser.gov/portal/sites/default/files/pdf/RAD-InterimRpt.pdf.

R-1

UEL Data Analysis

C-3

Document Outline

Review of Energy Performance Contracts in Public Housing

Foreword

Contents

List of Exhibits

Executive Summary

Background and Method

Principal Findings

Recommendations

Chapter 1 Introduction

Chapter 2 Background

Chapter 3 Survey Sample

Research Population

Sample Design and Process

Chapter 4 Survey Implementation11 The survey questions and the relative frequencies of answers received for each are shown in appendix A.

Survey Response Rate

Corrections to the Data33 The steps taken to correct the survey data are discussed in more detail in appendix C.

Blank or Incorrect Responses

Discrepancies in Respondent Group Category Designations

Implications for Survey Response Analysis

Adjustment of Sample Sizes

Chapter 5 Survey Analysis

Statistical Weighting

Statistical Hypotheses Testing—Comparing Group 1 with Group 3

Development of Ratings and Averages

Calculation of Weighted Average Scores

Hypotheses Testing

Within Group 1 Differences—Units Subject to an EPC vs. Units Not Subject to an EPC

Survey Evidence Concerning Why PHAs Did Not Undertake EPCs

How to Improve HUD’s EPC Process

Chapter 6 Telephone Follow-up Interviews

Introduction

Representation by Group and Size

Survey Results

Interviewee Recommendations

Chapter 7 Analysis of Utility, Financial, and Physical Conditions Data

Background

Analytic Approach

Analysis of UEL Data

Analysis of Long-Term Electricity Consumption Trends

Analysis of Short-Term Electricity Consumption Trends

Analysis of Financial and Physical Condition

Chapter 8 Relationship of the RAD Program to EPCs

Background

PHA Options

Gains in Energy Efficiency from EPCs and RAD

Implications

Evidence

Qualitative: Stakeholder Views

Quantitative: EPC Approvals

Chapter 9 Conclusions and Recommendations

Conclusions

Recommendations

Appendix A Responses to Survey Questions

Appendix B Summary Notes from Survey Follow-up Telephone Interviews

Group 1: Have Done an EPC

Small PHAs

Medium PHAs

Large PHAs

Group 2: Started but Did Not Complete an EPC

Small PHAs

Medium PHAs—Medium 1

Large PHAs—Large 1

Group 3: Never Did an EPC

Small PHAs

Medium PHAs

Large PHAs

Appendix C UEL Data Analysis

Source Data and Cleaning

Electricity Data Analysis

Data Limitations

Further Analysis

Appendix D Maps of Regions and of PHA Populations

Appendix E Abbreviations

References