Nearing the Finish Line:

5-Year Findings From the Family

Self-Sufficiency Evaluation

U.S. Department of Housing and Urban Development | Office of Policy Development and Research

DISCLAIMER The contents of this report are the views of the contractor and do not necessarily reflect the views or policies of the U.S. Department of Housing and Urban

Development or the U.S. Government.

NEARING THE FINISH LINE:

5-YEAR FINDINGS FROM THE

FAMILY SELF-SUFFICIENCY EVALUATION

Stephen Freedman

Nandita Verma

Betsy Tessler

Barbara Fink

MDRC

9/12/2021

Foreword

The Family Self-Sufficiency (FSS) program, established in the Cranston-Gonzalez

National Affordable Housing Act of 1990, is one of the largest efforts by HUD to improve self-sufficiency among those receiving federal rental assistance. The voluntary program focuses on helping participants access employment, financial literacy, education and training assistance, and the broader range of services within their communities. The FSS program also includes a work-based financial incentive to help build program participants’ assets. FSS coordinators set goals with participants—generally over a 5-year period—and any increases in participants’ rent caused by increased earnings are held in escrow, where they accrue interest until they are returned to the participants upon graduation. To graduate, participants must be employed, not receiving public assistance, and have met other individually set self-sufficiency goals, such as attaining an educational or occupational credential, attending financial literacy workshops, or addressing credit-score issues. Other research has examined aspects of the FSS model, including using employment-focused services and motivating participants using financial incentives, but this research represents the most extensive and rigorous evaluation to date.

HUD commissioned this national impact evaluation of the FSS program in 2012. The evaluation uses a randomized controlled trial to compare over a period of 5 years the outcomes of families eligible to participate in FSS to a control group, whose members were ineligible for program participation. This report, the third in a series, updates findings on the program’s effects from a previously published 3-year assessment and focuses on longer-term outcomes over 5 years after participants enrolled in this study, which is the standard term of the FSS Contract of Participation in the FSS program. The report focuses on the 2,556 study participants (the eligible

“impact sample”) who were 18 to 61 years of age at study enrollment.

Based on the first FSS report in this series, “Early Findings from the Family Self-Sufficiency Program Evaluation,” financial literacy was the top priority for program participants, and most people in the study signed up for FSS with the goals of building savings, improving credit scores, or learning money management. The work-based financial incentive (i.e., escrow account) was not the primary driver for program participation, indicating that people were interested in the suite of financial literacy services referred to them by FSS, rather than one specific tool to build economic mobility. This is important because it demonstrated that FSS program enrollees were likely confident in their employment-related skills and interested in how to improve their financial situation regardless of increased income. These goals were, indeed, reflected in the services engaged by the participants. Within the first 36 months of followup, significantly more participants in the FSS group engaged in financial counseling services than did the control group.

In addition, higher levels of participation in education and training activities were accompanied by only a small increase above the control group in earning an occupational credential or license.

This 5-year report demonstrates that FSS is successful at helping HUD-assisted residents save money, and some subpopulations of the FSS sample showed positive effects on credit scores or use of credit.

FSS graduates realized a substantial disbursement from their escrow savings accounts, averaging nearly $10,000, with 12 percent of graduates receiving more than $20,000. Within the study group, about 23 percent of FSS participants were still enrolled in the program after 5 years with some escrow accrued. These participants showed an average escrow balance of about $7,200, with approximately one-fourth of this group having accrued more than $10,000. Households commonly use these resources to start a new business, repair credit, buy a home, or pay for education.

ii

FSS programs also led to positive effects on attitudinal and perceived well-being outcomes.

FSS group members were more likely than control group respondents to report that they had improved their financial situation in the past year and were better able to plan for their future finances. Three subgroups of the FSS sample—2-year college graduates or higher, those working part-time hours at random assignment, and those having no recorded income at random assignment—experienced statistically significant increases on one or more outcomes based on credit scores or use of credit when compared to the control group.

HUD acknowledges that FSS would benefit from program improvements. Over the 5-year followup period, there was no evidence that the FSS program increased employment or earnings for participants when compared to the control group. This evaluation, until recently, took place during a long economic expansion. It documents high levels of employment, which led to increased average annual earnings for both groups. Against this backdrop of high labor force participation and increased earnings, FSS-type employment services seemed to add little to what individuals could do on their own. Also, employers make their own decisions regarding hiring, promotions, and pay. Thus, it is hard for a light-touch and referral-based program like FSS to control broader labor market trends. As expected, 41 percent of FSS participants that were enrolled in the Housing Choice Voucher program had barriers to employment, such as physical health and access to affordable childcare. It might be possible for FSS programs to improve employment and income outcomes for participants with a focus on removing these particular barriers.

The 5-year report also highlights other areas for improvement for FSS program grantees.

While after 5 years, approximately 47 percent of program participants either graduated or remained in the program using an extension, more than 53 percent ended participation in FSS prior to graduating. More than 40 percent of the FSS group members who exited the FSS program within 5 years forfeited their escrow accruals. Additionally, the average credit scores of the overall FSS

treatment group were below 600, in the suboptimal range, and scores had not improved compared with the control group scores. Both groups were determined to have incurred high amounts of debt, most commonly automobile or educational loans. This was combined with an increase in the use of Alternative Financial Services (e.g., payday loans) among the FSS group members. Although the study finds positive impacts for specific subgroups within the FSS treatment group regarding improved credit scores, the FSS program was unable to demonstrate positive impacts for the entire treatment group. These results, coupled with the stated goal of program participants that they want FSS to help them improve their financial security, illustrate that FSS grantees should implement program improvements that address credit outcomes and remove unnecessary barriers to graduation.

The final report, slated for release in 2022, will provide more conclusive evidence about FSS participants and how they benefit from the program. With 6 to 7 years of followup, the final report will not only allow the evaluation to track the remaining FSS participants through their extended contract period and examine circumstances after FSS for those who graduated—how they are faring, whether they continue to receive housing subsidies, and how they use their escrow—it will also support an analysis of how families fared during the COVID-19 pandemic.

The evaluation of implemented policy is a vital step of the policy process, with feedback shaping future initiatives and understandings of effective methods to increase self-sufficiency.

iii

In May 2022, HUD published a final rule on the FSS program that addresses some of the barriers identified in this report, as well as concerns raised by program administrators and staff. 1

Among its changes, the rule permits any adult household member to enroll in the program and eliminates regulations identified as potential barriers to graduation. The rule also eliminates a cap on savings that had been in place for only higher-income families. The changes and feedback from this evaluation assist HUD in program improvement via evidenced-based decisionmaking. HUD

will continue the FSS program after incorporation of feedback in improvements and expand as declared in the August 2022 agenda for economic justice and asset building. 2

Solomon J. Greene

Principal Deputy Assistant Secretary for Policy Development and Research

U.S. Department of Housing and Urban Development

1 Streamlining and Implementation of Economic Growth, Regulatory Relief, and Consumer Protection Act Changes to Family Self-Sufficiency (FSS) Program, 87 Fed. Reg. 30,020 (May 17, 2022), available at

https://www.federalregister.gov/documents/2022/05/17/2022-09528/streamlining-and-implementation-of-economic-

growth-regulatory-relief-and-consumer-protection-act

2 U.S. Department of Housing and Urban Development, Bridging the Wealth Gap: An Agenda for Economic Justice

and Asset Building for Renters (August 2, 2022), available at

https://www.hud.gov/sites/dfiles/PIH/documents/Bridging_Wealth_Gap.pdf

iv

Contents

Executive Summary ..................................................................................................................................... ix

The Family Self-Sufficiency Program .................................................................................................... x

Study Design and Sample Characteristics ............................................................................................. xi

Key Findings ........................................................................................................................................ xii

Program Implementation ............................................................................................................... xii

Enrollment, Graduation, and Escrow Disbursements .................................................................... xv

Employment and Earnings ............................................................................................................ xix

Credit Use and Financial Well-Being ........................................................................................... xxi

Housing Subsidy Receipt ............................................................................................................ xxiv

Variation Across Sites in Program Impacts ................................................................................ xxiv

Next Steps........................................................................................................................................... xxv

Chapter 1. Introduction ................................................................................................................................. 1

Housing Assistance and Employment Context ...................................................................................... 3

The Family Self-Sufficiency Program .................................................................................................... 4

Core Features of the FSS Program ................................................................................................... 5

How Might the FSS Components Help Participants Advance? ....................................................... 8

The National Family Self-Sufficiency Impact Evaluation ................................................................... 11

The Study Sample ................................................................................................................................ 14

Structure of This Report ....................................................................................................................... 18

Chapter 2. Program Implementation Update and Reflections .................................................................... 19

Data Sources and Followup Period ...................................................................................................... 19

Program Practice and Policy Updates .................................................................................................. 20

Staffing and Caseloads ................................................................................................................... 20

Interim Escrow Disbursements ...................................................................................................... 20

Requirements for Graduation ......................................................................................................... 21

A New FSS Performance Measurement System .................................................................................. 22

Case Management in the Final Years of the FSS Contract .................................................................. 23

Final Push in Years 4 and 5 ........................................................................................................... 23

Working with Participants Not Likely to Graduate ....................................................................... 25

Balancing Accountability and Self-Reliance ................................................................................. 25

Graduation ..................................................................................................................................... 26

Service Referrals and the Role of the Program Coordinating Committee ..................................... 27

Staff Reflections ................................................................................................................................... 28

Escrow Incentive ............................................................................................................................ 29

Interim Disbursements ................................................................................................................... 31

Who Benefits from FSS ................................................................................................................. 32

Promising Practices ........................................................................................................................ 32

Making the FSS Program Mandatory or Keeping it Voluntary ..................................................... 35

Program Length ............................................................................................................................. 35

Conclusions .......................................................................................................................................... 35

Chapter 3. FSS Enrollment Status, Escrow Accruals, and Graduations ..................................................... 37

How Family Self-Sufficiency Escrow Works ...................................................................................... 38

FSS Enrollment Status, Graduations, and Escrow Accrual and Disbursement .................................... 39

Variation in Graduation and Escrow Accrual for Selected Subgroups .......................................... 42

Variation in Graduation Rates by Housing Agency ....................................................................... 43

Who Appears to Be Benefiting the Most from Enrollment in Family Self-Sufficiency? .................. 46

v

Conclusions .......................................................................................................................................... 56

Chapter 4. Impacts on Employment and Earnings ...................................................................................... 58

Data Sources and Methods ................................................................................................................... 58

Program Impacts on Employment and Earnings .................................................................................. 59

Employment and Earnings Trends for Control Group Members ................................................... 59

Impacts on Employment and Earnings ................................................................................................. 62

Conclusions .......................................................................................................................................... 65

Chapter 5. Impacts on Credit Use, Housing Costs, Receipt of Housing Subsidies,

and Financial Well-Being ..................................................................................................................... 66

Experian and Clarity Credit Data ......................................................................................................... 67

Impacts on Credit Scores ...................................................................................................................... 69

Impacts on the Use of Traditional Financial Services and Sources of Credit ................................ 74

Impacts on the Use of Alternative Financial Services and Sources of Credit................................ 75

Impacts on the Use of Credit (Traditional and AFS) and Incidence of Credit Problems .............. 76

Impacts on Housing Status and Rent Subsidies ............................................................................. 80

Impacts on Homeownership .......................................................................................................... 83

Conclusions .......................................................................................................................................... 84

Chapter 6. Variation in Program Impacts ................................................................................................... 85

Program Impacts by Subgroups: Who Benefits in the Years 3-5? ....................................................... 86

Subgroups Based on Participant Baseline Characteristics ............................................................. 87

Employment and Earnings ...................................................................................................... 87

Credit Scores and Use of Credit .............................................................................................. 93

Variation in Housing Outcomes .............................................................................................. 94

Subgroups Based on Program Implementation Features ............................................................... 95

Program Clusters Based on Program Focus ................................................................................... 96

Program Clusters Based on Site Monitoring and Engagement Practices .................................... 102

Conclusion .......................................................................................................................................... 104

Chapter 7. Summary and Next Steps in the Evaluation ............................................................................ 105

What the 5-Year Results Tell Us ........................................................................................................ 105

The Evaluation Going Forward .......................................................................................................... 110

Appendix A ............................................................................................................................................... 111

Appendix B ............................................................................................................................................... 116

Appendix C ............................................................................................................................................... 118

Appendix D ............................................................................................................................................... 132

Appendix E ............................................................................................................................................... 140

References ................................................................................................................................................. 184

vi

List of Exhibits

Exhibit ES.1: Family Self-Sufficiency Graduation and Enrollment Status in Months 12 and 36 and

in the Last Month of Followup ....................................................................................... xvii

Exhibit ES.2: Average Quarterly Employment Rate and Annual Earnings, Family Self-Sufficiency

Impact Sample .................................................................................................................. xx

Exhibit ES.3: Impacts on Use of Financial Services in 2019, Family Self-Sufficiency Impact

Sample ............................................................................................................................ xxii

Exhibit 1.1.

Core Components of the HUD Family Self-Sufficiency Framework ................................. 7

Exhibit 1.2:

Simplified Schematic of the Family Self-Sufficiency Program Theory of Change ............ 10

Exhibit 1.3:

Data Sources for the Family Self-Sufficiency Study ........................................................ 12

Exhibit 1.4:

Public Housing Agencies Participating in the National Family Self-Sufficiency

Evaluation ......................................................................................................................... 13

Exhibit 1.5:

Baseline Characteristics of Households in the Family Self-Sufficiency Impact

Sample .............................................................................................................................. 14

Exhibit 1.6:

Baseline Characteristics of Heads of Household in the Family Self-Sufficiency

Impact Sample .................................................................................................................. 16

Exhibit 3.1:

Family-Self Sufficiency Enrollment Status at the End of Year 5 and in the Last

Month of Followup ........................................................................................................... 40

Exhibit 3.2:

Family Self-Sufficiency Graduation and Enrollment Status in Months 12 and 36 and

in the Last Month of Followup ......................................................................................... 41

Exhibit 3.3:

First Month of Escrow Accrual and Cumulative Percentage of Family Self-

Sufficiency Group Members Who Ever Accrued Escrow Credits, by Month of

Followup ........................................................................................................................... 42

Exhibit 3.4:

Indicators of Family Self-Sufficiency Escrow Account Accrual for Selected

Subgroups ......................................................................................................................... 44

Exhibit 3.5:

Indicators of Family Self-Sufficiency Graduations and Escrow Credits Accrual in

Months 1 to 72 by Housing Agency, Family Self-Sufficiency Service-Use Sample ....... 45

Exhibit 3.6:

Baseline Characteristics of the Family Self-Sufficiency Group, by HCV and Family

Self-Sufficiency Program Status and Escrow Accrual Status ........................................... 48

Exhibit 3.7:

Change in Reported Annual Head-of-Household Earnings and Total Household

Earnings, Between Month of Random Assignment and the Last Month of Followup,

by HCV and Family Self-Sufficiency Enrollment Status and Escrow Balance ................ 53

Exhibit 4.1:

How to Read the Impact Tables in This Report ................................................................ 60

Exhibit X:

Impacts on Employment, Family Self-Sufficiency 18-Month Survey Respondent

Sample .............................................................................................................................. 60

Exhibit 4.2:

Quarterly Employment Rate Among Control Group Members by Self-Reported

Employment Status at Random Assignment, Family Self-Sufficiency Impact Sample ... 61

Exhibit 4.3:

Average Quarterly Earnings Among Control Group Members, by Self-Reported

Employment Status at Random Assignment, Family Self-Sufficiency Impact Sample ... 63

Exhibit 4.4:

Impacts on Employment and Earnings in Years 1 Through 5, Family Self-Sufficiency

Impact Sample .................................................................................................................. 64

Exhibit 5.1:

VantageScore Ranges ....................................................................................................... 68

Exhibit 5.2:

Family Self-Sufficiency Group Members’ Average Experian Vantage 3.0 Credit

Scores, 2012–19, Family Self-Sufficiency Impact Sample (FSS Group Only) ................ 69

vii

Exhibit 5.3:

Changes in FSS Group Members’ Average Experian Vantage 3.0 Credit Scores,

2012–19, Family-Self Sufficiency Impact Sample (FSS Group Only)............................. 70

Exhibit 5.4:

Changes in Family Self-Sufficiency Group Members’ Average Clarity Clear Early

Risk Credit Scores, 2014–19............................................................................................. 71

Exhibit 5.5:

Changes in the Distribution of Clarity Clear Early Risk Credit Scores, 2014–19,

Family Self-Sufficiency Impact Sample (FSS Group Only) ............................................ 72

Exhibit 5.6:

Impacts on Credit Scores After 5 to 6 Years of Followup, Family Self-Sufficiency

Impact Sample .................................................................................................................. 73

Exhibit 5.7:

Changes in Debt Balance Amount from 2014 to 2019, by Type of Credit, Family

Self-Sufficiency Group Members in the Family Self-Sufficiency Impact Sample ........... 74

Exhibit 5.8:

Impacts on Use of Traditional and Alternative Financial Services, Family Self-

Sufficiency Impact Sample ............................................................................................... 76

Exhibit 5.9:

Impacts on Incidence of Credit Problems, Family Self-Sufficiency Impact Sample ................ 78

Exhibit 5.10: Change in Debt-to-Income Ratio from 2014 to 2019, by Research Group, Family

Self-Sufficiency Impact Sample ....................................................................................... 81

Exhibit 5.11: Impacts on Shelter Costs and Housing Subsidies, Years 1 Through 5 Family Self-

Sufficiency Impact Sample ............................................................................................... 82

Exhibit 6.1A: Impacts on Selected Outcomes by Employment Status at Random Assignment ................ 88

Exhibit 6.1B: Impacts on Selected Outcome by Educational Attainment at Random Assignment ................ 89

Exhibit 6.1C: Impacts on Selected Outcomes by Income at Random Assignment ................................. 90

Exhibit 6.1D: Impacts on Selected Outcomes by Disability Status at Random Assignment .................. 91

Exhibit 6.1E: Impacts on Selected Outcomes by Percentage of Rent and Utilities Expenses Paid by

Household at Random Assignment ................................................................................... 92

Exhibit 6.2:

Program Clusters: Data and Definitions ........................................................................... 96

Exhibit 6.3A: Impacts on Selected Outcomes by Program Emphasis on Job Search and

Postemployment Services ................................................................................................. 98

Exhibit 6.3B: Impacts on Selected Outcomes by Program Emphasis on Education and Training

Services ............................................................................................................................. 99

Exhibit 6.3C: Impacts on Selected Outcomes by Program Emphasis on Financial Services ................ 100

Exhibit 6.3D: Impacts on Selected Outcomes by Program Emphasis on Monitoring and

Engagement .................................................................................................................... 101

viii

Executive Summary

Policymakers have long sought to improve the labor market outcomes and address the barriers to work faced by households receiving federal rental assistance. The U.S. Department of Housing and Urban Development (HUD) Family Self-Sufficiency (FSS) program blends housing assistance with employment-focused services and a financial incentive to help improve the economic well-being of households receiving federal rental assistance. A voluntary program, FSS is HUD’s main strategy for helping households receiving federal housing choice vouchers (HCVs) to increase their earnings and make progress toward economic self-sufficiency. Until recently, limited evidence was available about the FSS program’s effectiveness in achieving such outcomes. To fill that gap, HUD

commissioned a national impact evaluation in 2012 and selected MDRC to lead it. This report follows two earlier reports from the FSS evaluation and presents longer-term findings (Verma et al., 2021; Verma et al., 2019).

Through annual grants totaling approximately $80 million, HUD funds about 700 public housing agencies (PHAs) to operate FSS programs across the country. The funding mostly covers modest resources for the programs to hire service coordinators, who work with FSS participants, typically over a 5-year term, to set goals that will help them make progress toward self-sufficiency and refer them to a broad range of services in the community. To encourage participants to go to work, increase their earnings, or both, the program includes as its centerpiece a work-based financial incentive—an escrow account—which acts as a long-term savings account.

Like others receiving housing subsidies, FSS participants face rent increases when their earnings increase (typically 30 percent of additional earnings). In the FSS program, a housing agency credits a family’s escrow account with an amount based on their rent increase. The money accrued in the escrow account is disbursed to participants when they graduate from the program, usually after 5 years. To graduate, participants must be employed at that time and must achieve other self-sufficiency goals, such as attaining an educational or occupational credential, attending financial literacy workshops, or addressing credit-score issues. As an additional requirement, FSS participants and the members of their household may not receive cash assistance for the 12 months leading up to graduation. Graduates from the FSS program receive all funds deposited in their escrow accounts, plus interest, and can use that money for any purpose.

The FSS evaluation uses a randomized controlled trial to test the program’s effects. It compares the outcomes of program-eligible individuals who were randomly assigned to an FSS group, and a control group, whose members were not eligible to participate in the program, for a period of 5 years.

Differences between the two groups’ average outcomes represent the program’s effects or impacts.

Statistically significant differences between groups indicate with a strong degree of confidence that the impacts can be attributed to FSS rather than to chance.

Eighteen housing agencies in seven states agreed to participate in this evaluation and enrolled 2,656

voucher recipients in the study between October 2013 and December 2014. The evaluation focuses on the 2,556 study participants (the eligible “impact sample”) who were 18 to 61 years of age at study enrollment. This report describes outcomes and the program’s impacts on them as measured 5 years ix

after participants enrolled in this study. Taking a longer-term perspective, this report looks at the progress FSS participants made toward building economic security (increasing earnings, improving financial well-being, and receiving their housing subsidies, for example). Is there evidence that, over a 5-year period, participants in FSS are on a different economic and financial trajectory than their counterparts who do not receive FSS services or the escrow incentives? Do they graduate and earn escrow disbursements? To help frame and interpret these longer-term results, the report includes an update on how program staff members engage with FSS participants in the later years of their program enrollment, whether special engagement strategies are used to keep participants on track for graduating and earning their escrow disbursements, and how, if at all, housing agencies are beginning to examine their programs’ practices in light of HUD’s new performance measurement system, introduced in 2018.

The followup period covered in this report (2013 to 2019) ended before the start of the COVID-19

pandemic, which began its sweep across the nation in March 2020. The FSS programs in this study, along with their housing agencies, had to change their regular operations when the pandemic began, and most shifted online to engage with program participants. Although none of the service changes or disruptions affect the outcomes examined in this report, they occur during the followup period for this evaluation. The final report for this study, due in mid-2022, will take into account the pandemic and its implications for the program’s effects on participants’ work outcomes, graduation and escrow receipt, financial well-being, and receipt of housing and other government subsidies.

The Family Self-Sufficiency Program

The FSS program was established in 1990 by Section 554 of the Cranston-Gonzalez National Affordable Housing Act. HUD regulations set the basic framework for the program, but local housing agencies can design how they implement service provision—an element of flexibility offered by the federal framework. The escrow component, on the other hand, strictly follows HUD’s specifications for calculating credits. Although all adults in FSS households are encouraged to seek employment, only the household head—the voucher holder—must meet that requirement to graduate from the program. 3

At program enrollment, participants sign a Contract of Participation (COP) and complete an Individual Training and Services Plan (ITSP). The typical FSS contract can last up to 5 years, during which participants are expected to achieve all agreed-upon goals. Each FSS operator is required to organize a Program Coordinating Committee (PCC) comprising service providers in the community. Through these committees, FSS operators can learn about services and resources in their communities, and service providers can become invested in the success of the program by providing services to FSS

participants. In addition, some housing agencies host FSS services performed by committee members.

3 Amendments to the FSS statute enacted in 2018 (but not yet in effect) will make several changes to the FSS program.

Among the changes is allowing for more flexibility for families to meet graduation requirements through the employment of any adult household member, not solely the head of household. For a list of all changes forthcoming, see Center on Budget and Policy Priorities, 2020.3

x

In recent years, HUD has produced a comprehensive resource guide on the FSS program, offering housing agencies practical, hands-on tips for its operation. 4 In 2018, HUD announced a new performance measurement system for assessing programs that receive FSS funding. The system evaluates FSS programs on the basis of the number of participants served and on participants’ average earnings gains and graduation rates. Both developments reflect HUD’s efforts to strengthen the program’s performance.

The evaluation captures how the programs operated on the ground during the followup period. In some cases, the local program implementation practices, as described by PHA administrators and case managers, differed from HUD’s latest thinking about program best practices and, in some instances, program requirements. In addition, HUD’s Final FSS Rule, which went into effect in November 2022, addresses some of the concerns raised and recommendations articulated by program administrators and staff. 5

Study Design and Sample Characteristics

This evaluation posits that the program is expected to produce short- and long-term effects in two ways: (1) by increasing participants’ access to ongoing case management and coordination services that help make them more employable or improve their financial security, and (2) by offering a long-term escrow savings account for participants to start or maintain employment and increase their earnings over time. Through these mechanisms, it is hoped that participants will eventually earn enough to pay for housing without a subsidy and thereby free up housing subsidies to serve other households.

For a variety of reasons, however, positive effects may not materialize. The potential to earn escrow disbursement represents a distant and uncertain reward that may not motivate participants to increase their earnings in the short run, and housing voucher rules mean that higher earnings result in higher rent. Also, FSS participants face various barriers that may limit their chances of finding new employment or increasing their earnings, especially if the FSS referrals and services do not help address those barriers. In addition, members of the control group may have access to similar employment-related services in their communities, in which case the outcomes between the program and control group could display little contrast.

Alternatively, positive impacts may not be realized in the shorter term but may be realized after 5 years or longer, after participants complete an education or training goal they set for themselves as part of the FSS program, or when they may see themselves “nearing the finish line” and feel a strong incentive to find a new job, maintain employment, or work extra hours to graduate and receive a large escrow disbursement.

4 See HUD, 2017a.

5 Streamlining and Implementation of Economic Growth, Regulatory Relief, and Consumer Protection Act Changes to Family Self-Sufficiency (FSS) Program, 87 Fed. Reg. 30,020 (May 17, 2022), available at

https://www.federalregister.gov/documents/2022/05/17/2022-09528/streamlining-and-implementation-of-economic-

growth-regulatory-relief-and-consumer-protection-act

xi

The FSS evaluation is designed to build rigorous evidence about the operations and the effectiveness of the program. It draws on a combination of qualitative and quantitative data collected across 18 study sites to assess how the FSS program improves participants’ financial well-being. The evaluation examines the effects for the study sample overall and for certain subgroups—including those who were not working when they joined the study and those who had completed little education at that time. The evaluation also explores whether some program features are particularly effective at boosting participation or improving outcomes. Those program features include emphases on job searches, education and training, and financial management workshops and counseling, as well as specific implementation approaches, such as having small or large caseloads or maintaining more or less frequent contact with participants.

To select the study sites, MDRC examined HUD data, identified candidate housing agencies, followed up with program administrators, and visited candidate sites. Ultimately, 18 housing agencies in seven states agreed to participate in the evaluation: California, Florida, Maryland, Missouri, New Jersey, Ohio, and Texas. These agencies represent a wide range of contexts within which FSS programs operate. There are large and small housing agencies and FSS programs in large and small cities and suburban settings. Working with each agency, MDRC developed study recruitment and enrollment processes; agencies did not have to increase their FSS enrollments for the evaluation, but they did have to double the number of households who signed up for a chance to participate in the FSS program because one-half were assigned to the control group.

Study participants were enrolled over 15 months, from October 2013 through December 2014. The participants in the study share broadly similar demographics to the national FSS population. The sample is predominantly (91 percent) female, with an average age of 39 years at random assignment.

More than 76 percent of study participants in the sample had a child under age 18 at home, typically age 12 or younger. About 14 percent did not have high school diplomas or equivalency certificates.

More than one-half (56 percent) were working at the time of study enrollment. A high percentage (about 70 percent) were receiving Supplemental Nutrition Assistance Program (SNAP) benefits, or food stamps, implying that a large proportion had earnings equal to 130 percent of the federal poverty level or less (the SNAP benefit eligibility cutoff). Reflecting national patterns, only a small proportion received cash assistance (fewer than one-sixth), and about one-third of the sample reported having received housing vouchers for 10 years or more. The opportunity to receive financial education and management services was a big draw for sample members; job-related services were also an important feature but to a lesser extent.

Key Findings

Program Implementation

As described in earlier reports on this evaluation, the FSS framework provides housing agencies with considerable leeway in setting local policies, case management or coordination approaches, and staffing arrangements that influence how the programs work with participants to build assets and advance toward self-sufficiency. In addition to variations in program operations across housing xii

agencies, HUD policies and guidance provide important context for understanding how programs support participants.

• For the most part, program policies and practices across the FSS study agencies remained largely unchanged in the later years of followup (2019 and 2020), with a few notable exceptions.

Of the FSS programs in the evaluation that participated in the 2020 round of implementation data collection, all but one allow participants to receive interim disbursements of the funds in their escrow accounts—a slight increase since the previous round of data collection, completed in 2018.

Beginning in 2018, most of the FSS programs in the study eliminated any additional employment requirements beyond those set by HUD—for example, specifying a rate of pay or number of months that the participant had to be employed before graduation. However, even where the additional requirements have been eliminated, FSS staff still encourage and support participants to reach for that higher bar—it is just not codified in the official plans (or Individual Training and Services Plans) they develop with their case managers.

In November 2018, HUD announced a new performance measurement system for rating FSS

programs, using a “composite score.” 6 As of early 2020, housing agencies in this study had not yet made significant changes to their program operations as a result of the new system. Staff members, however, raised concerns that to implement changes in an attempt to increase their composite scores, they might have to shift the focus of FSS to getting participants employed as quickly as possible. As a result, the program might not be able to pay as much attention as it would prefer in helping participants find higher paying jobs or get the education and skills needed to obtain better jobs.

• Agencies’ approaches to case management in participants’ final years of FSS remained as varied as they were earlier in participants’ time in the program.

Housing agencies did not make significant changes in their expectations regarding how often staff members were expected to reach out to participants. As participants in the FSS study programs approached Year 5, most programs continued to mandate contact at least quarterly, and all except one still required at least one annual in-person meeting with each participant. The FSS programs in the evaluation do not hold additional meetings specifically to review goals or to prepare for graduation.

Although the official expectations for contact did not change, many staff members reported higher engagement toward the end. In the last year or so, they said they made a final push to keep participants on track, and the process of reviewing goals and adjusting plans intensified. If participants were not on track to complete all their goals by the end of the contract period, staff members talked to them, reminded them of the terms and conditions of the contract, and explained their options, which could include changing their goals (within the time frame allowed by the housing agency) or receiving an extension. Participants who were on track might complete all of their goals and graduate early at some 6 The new FSS Performance Measurement System includes three parts: changes in earnings from the time participants join FSS, graduation rates, and participation rates. See HUD, 2018.

xiii

programs, whereas at others, they might add new goals and continue in the program for the full 5 years (and earn additional escrow).

• FSS staff members consider the escrow incentive to be integral to the program and value its motivational influence. The preference among some staff members for participants to receive large lump-sum payments at graduation may help explain why interim disbursements are rarely used.

All staff members appreciated that the FSS program gave participants the opportunity to build assets, but they differed in their views on the extent of the escrow account’s importance. Whatever their opinion of escrow’s influence, FSS staff members believed that achieving educational, employment, and financial services goals does more than escrow itself to help individuals gain confidence and move toward self-sufficiency over the long term.

In theory, FSS program staff members believed that rewarding incremental progress by granting interim (pregraduation) escrow disbursements would help keep participants engaged and remove barriers to meeting goals. These payments can help participants overcome financial barriers such as lack of reliable transportation or education expenses. Despite these advantages, many staff members wanted participants to learn to budget and save and build up a large escrow balance, and they felt that providing interim disbursements did not encourage self-sufficiency. As a result, interim disbursements were not routinely publicized or discussed during check-in meetings and were rarely used.

• Most staff members value the Program Coordinating Committee and consider it critical to the program, yet participants have some needs that it is not able to meet.

Nearly all study sites have a Program Coordinating Committee (PCC), a collaborative group of service providers. In larger urban areas, some PCCs are countywide, serving multiple housing agencies. Some housing agencies invite all their primary service referral providers to join their PCCs; in others, the PCC members include a subset of referral partners. Across the board, FSS staff members said that the main value of the PCC was that it enabled them to stay connected to, and network with, local service providers, which in turn helped ensure that FSS participants had access to the services they needed.

FSS staff members said they were generally satisfied with the organizations represented on the PCCs, how they interacted with the FSS program, and the range of services they provided to FSS participants; they did not recommend any significant changes. Staff from several FSS programs noted some types of services that participants needed but were not available from their PCCs, most notably transportation, childcare assistance, and mental health services. Some also mentioned that having access to discretionary funds to support such needs of participants would also be helpful.

• Staff members identified a range of promising practices that they thought worked best to keep participants engaged and making progress toward their goals.

Staff offered a range of practices that they associated with—or thought could lead to—stronger outcomes for participants, including more frequent in-person contact, setting and monitoring shorter-term goals (including breaking up long-term goals into shorter steps), developing a strong rapport xiv

between case managers and participants, enabling case managers to focus solely on FSS and not requiring them to perform non-FSS (or other housing voucher program-related functions), developing valuable service referrals, and using interim escrow disbursements more generously to help participants meet their goals. Whereas some staff members thought FSS should be a mandatory program, others felt strongly that it should not. Likewise, staff members were divided about whether FSS could be shortened to fewer than 5 years or whether 5 years was the right length for this program.

Enrollment, Graduation, and Escrow Disbursements

For a 5-year program such as FSS, several outcomes are of interest. The two previous reports produced by this evaluation, covering the first 3 years of followup, presented a broad range of program participation outcomes, including the types of FSS-related activities participants engaged in, the contact they maintained with program staff members, how long they stayed enrolled in FSS, and (for those who left FSS) whether they graduated and received escrow disbursements. These reports have shown high rates of participation in various types of FSS-related activities, including education and training, financial management services, and employment services. Within the first 36 months of followup, FSS realized moderate-level increases (between 6 and 14 percentage points) above control group levels in activities related to job search, homeownership preparation, postemployment services, and education and training. FSS had a much larger effect (greater than 20 percentage points) on the use of financial counseling services. Higher levels of participation in education and training activities were accompanied by a small increase above the control group level in earning an occupational credential or license, but it did not appear to have increased the attainment of academic credentials (such as conferral of a degree or a diploma). Program graduation rates also hovered around 4 percent through month 36, roughly midway through the entire 6- to 7-year followup period for this evaluation.

This report updates those earlier analyses, focusing exclusively on graduation, a formal marker of success in the program, and escrow outcomes.

• So far, about 17 percent of the FSS participants have achieved their program goals and graduated from the program; another 23 percent remain enrolled in FSS with some escrow accrued.

Typically, FSS participants can take up to 5 years to graduate from the program. Under certain circumstances, the program may extend a participant’s FSS contract by another 2 years. 7 Exhibit ES.1

presents FSS enrollment, graduation rates, and escrow accrual statuses at three points of followup: Month 12, Month 36, and the last month of followup, which ranges from 60 to 75 months, depending on study enrollment. By Month 12, the end of the first year after enrolling in FSS, about 86 percent of the FSS group was officially active in the program, with about one-fourth beginning to build an escrow balance. By the last month of followup, more than one-half of the study group (or about 53 percent) had ended their participation in FSS for reasons other than meeting its graduation requirements. HUD

and housing agency data include limited information about exit reasons but show that a large proportion of FSS program exits involve participants leaving the voucher program or moving to another housing 7 In response to the COVID-19 pandemic and associated economic downturn, HUD recently gave housing agencies the approval for giving PHAs authority to extend FSS contracts by a third year.

xv

agency, being terminated from FSS, or leaving the FSS program voluntarily. As documented in earlier reports, most programs have policies that allow them to terminate FSS participants who consistently miss appointments and are not responsive to case managers’ repeated attempts to reach them.

xvi

Exhibit ES.1: Family Self-Sufficiency Graduation and Enrollment Status in Months 12 and 36 and in the Last Month of Followup

100

0.3

4.0

16.9

90

23.7

80

39.8

23.4

70

60

)%

6.5

tage ( 50

62.7

22.9

Percen 40

30

53.2

20

33.2

10

13.3

0

Month 12

Month 36

Last Month of Followup

Month of Followup

Exited

Enrolled with no escrow balance

Enrolled with escrow balance

Graduated

Notes: The FSS service-use sample includes all, or a randomly selected subsample, of housing choice voucher heads of household who were randomly assigned to the FSS group between October 18, 2013, and December 22, 2014, in 12 public housing agencies and were age 18 to 61 at the time of random assignment. The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Rounding may cause slight discrepancies in calculating sums.

Sources: MDRC calculations using housing authority administrative data and U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data.

xvii

About 60 percent of FSS participants accrued some escrow credit over their FSS program tenure; most began building escrow balances in the first 2 years after enrolling in the program, after which the chances of earning any escrow began to deteriorate (only about 7 percent of those who accrued any escrow began seeing credits in Year 3 or later). By the end of the followup period, close to 17 percent of FSS participants had graduated, most of whom had earned an escrow disbursement. This number could grow as high as 40 percent. About 23 percent continued to be enrolled in the program, with some escrow accrued, and could graduate in the next 1 to 3 years. Nearly 7 percent have remained enrolled in the program without accruing escrow.

• Many FSS graduates realized a substantial escrow disbursement, averaging nearly $10,000 per recipient.

On average, the FSS graduates earned $9,651 in escrow disbursements, with close to 13 percent receiving more than $20,000, a significant one-time payment for these families. The likelihood of graduating from FSS and earning an escrow disbursement can be influenced by several factors, including a participant’s employment status and earnings at program enrollment. Those not working at the time of program entry could potentially benefit the most from the FSS escrow because all their future earnings would be included in the calculation of escrow credits. Nonworking participants could also face the most severe barriers to finding and maintaining employment, which is required for graduation and earning the escrow. By contrast, FSS participants who are working full time or have relatively high earnings at program enrollment may have the best prospects of increasing their earnings by finding better jobs or advancing with their current employers. They may also be the most likely to maintain their employment after they start accruing escrow credits. They may experience smaller increases in their earnings, however, and accrue only a small amount of escrow.

The findings in this report show that FSS group members who reported that they were not working at study entry were the least likely, so far, to graduate from FSS, compared with subgroups that reported higher household earnings at study entry. FSS participants who reported the highest household earnings (more than $20,000 a year) at study entry have, on average, accrued relatively small amounts of escrow and received relatively small disbursements compared with other subgroups. The graduation rate for FSS group members with 2-year college degrees or higher (29 percent) exceeds the rate for the other educational subgroups, as does the percentage of them who have graduated and received more than $5,000 in their escrow disbursements (17 percent).

• Most FSS group members who remained enrolled in the program had positive escrow balances at the end of the followup period.

Most participants who were still enrolled in the program at the end of the followup period had positive escrow balances and, therefore, strong financial incentives to complete their goals. Nonetheless, few participants, if any, would be expected to start accruing escrow (based on patterns through Year 5).

Those who remained enrolled in FSS at the end of this report’s followup period showed escrow balances of about $7,200, on average, with 24 percent of this group having accrued more than $10,000.

The evaluation will continue to track graduations and disbursements through the middle of 2021, but xviii

some of those still enrolled in FSS may receive additional extensions, which the housing authorities are authorized to offer because of COVID-19, and they may still be enrolled when this evaluation ends.

Regardless of the final graduation rate, this study shows that the FSS program eventually provides a large lump sum payment for graduates, often exceeding the maximum amount that a household with a low or moderate income could receive as an earned income credit on its federal tax return. However, a relatively small proportion of FSS enrollees reach the point of graduating and earning escrow disbursements. Some may benefit in other concrete ways, such as from case management services and referrals—for example, by enrolling in an educational or training activity, learning to budget and manage their family finances more efficiently, increasing their savings, or reducing their debt.

Nonetheless, it might be unreasonable to expect any tangible benefits of enrollment for FSS group members who left the FSS program while continuing to receive housing voucher assistance, and this is perhaps true for the small group of current FSS enrollees with no escrow accrual after 5 years in the program.

Employment and Earnings

Although the FSS program is designed to provide services and referrals to help participants address a variety of difficulties, including childcare needs, mental and physical health issues, and transportation challenges, the case management component of the program is in service of the ultimate goals of the program: to support participants’ efforts to find employment, remain employed, or increase their earnings. The escrow incentive promotes the same goals; therefore, an important test of the FSS model is an assessment of its ability to meet those goals as compared with the control group.

This evaluation relies on the National Directory of New Hires quarterly wage records to examine the program’s effects on work outcomes. Typically, FSS participants are allowed to take up to 5 years to achieve their program goals, and given their highly personalized goals and pathways to attain them, positive effects on work outcomes may occur in the later years of program enrollment.

• Over a 5-year followup period, the FSS and control groups experienced comparable quarterly employment levels and average earnings. There is no evidence that FSS

produced overall improvement in labor-market outcomes for household heads.

Quarterly wage data show high levels of employment for FSS and control group members, although most members of both groups experienced at least one quarter of joblessness. More than 85 percent of both groups worked for pay at some point during the 5 years of followup. On average, about 64 percent were employed in any given quarter. Members of both groups averaged a bit more than $75,000 in total earnings during the followup period—or about $15,000 per year (the averages include zeroes for individuals who had no earnings). As shown in exhibit ES.2, there are only small, not statistically significant differences in employment or earnings outcomes between the two study groups.

These outcomes were also examined for subgroups of participants defined based on their work and education, for example (not shown in exhibit ES.2, which presents the results for the overall sample).

xix

Exhibit ES.2: Average Quarterly Employment Rate and Annual Earnings, Family Self-Sufficiency Impact Sample

FSS

Control

Difference

Outcomes

Group

Group

(Impact)

P-Value

Employment (%)

Average quarterly employment rate

Year 1

61.6

61.4

0.2

0.887

Year 2

64.0

63.7

0.3

0.847

Year 3

64.9

63.9

1.0

0.502

Year 4

64.8

64.2

0.6

0.672

Year 5

64.9

64.4

0.5

0.732

Years 1 through 5

64.0

63.5

0.5

0.651

Earnings ($)

Total earnings

Year 1

11,967

12,144

-177

0.596

Year 2

14,178

14,044

134

0.760

Year 3

15,509

15,307

202

0.688

Year 4

16,526

16,346

180

0.742

Year 5

17,748

17,490

258

0.662

Years 1 through 5

75,929

75,332

598

0.767

Sample size (total = 2,548)

1,282

1,266

Notes: The FSS impact sample includes housing choice voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Average quarterly employment rate is calculated as total quarters with employment divided by total quarters of followup, expressed as a percentage. Estimates were regression adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between research groups. The p-value indicates the likelihood that the difference between the FSS group and control group arose by chance. Statistical significance levels are indicated as * = 10 percent; ** = 5 percent; and *** = 1 percent.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires The most consistent pattern of positive effects on employment and earnings outcomes was recorded for FSS group members with 2-year college degrees or higher at random assignment, but none of the differences between the program and control group were statistically significant.

xx

Credit Use and Financial Well-Being

This report also includes a comprehensive analysis of the FSS program’s effects on financial security outcomes, as captured by credit-use behavior and credit scores. Using data from Experian®, the nation’s largest credit bureau, the evaluation examines both traditional financial transactions (store and credit card transactions, auto and student loans) and alternative financial services, or high-risk loans, such as payday loans, as recorded by Clarity Services, Inc., a subsidiary of Experian. 8 Many FSS programs encourage participants to attend workshops or meet with financial counselors to build financial management skills such as budgeting, managing personal and household finances, building savings, managing debt, and “cleaning up” credit history. These efforts could help participants build financial behaviors that could be reflected in their credit-use patterns and scores. FSS also aims to increase household earnings, both for enrolled participants and other adults in the household (escrow accruals are based on household earnings), possibly changing household disposable income and affecting credit outcomes.

Experian’s VantageScore, a credit-scoring model that uses scores ranging from 300 to 850, was used to analyze traditional transactions. 9 Clarity and VantageScore ranges are aligned and are generally grouped in five categories: Deep Subprime (300–499), Subprime (500–600), Near Prime (601–660), Prime (661–780), and Excellent (781–850).

• Between 2012 and 2019, VantageScores and Clarity scores for most FSS group members were below 600, or in the subprime range. The FSS program did not lead to increases above the control group outcome in average credit scores.

In 2013 and 2014, the period corresponding to study enrollment, most FSS group members had relatively low (subprime) credit scores. VantageScores averaged about 555, increasing by nearly 20

points during the 5 years of followup: The proportion of FSS group members with prime VantageScores increased from about 8 percent in 2013 to 16 percent 6 years later, whereas the proportion of FSS group members with subprime scores (below 601) decreased by a similar amount (not shown).

In general, those with lower VantageScores appeared to use high-risk, alternative financial services loans more often than those with higher scores; therefore, more of them tended to have Clarity scores (as measured by Clarity’s Clear Early Risk Score™). In 2019, about 85 percent of FSS group members with deep subprime (below 500) VantageScores had a Clarity score, compared with about 30 percent of FSS group members with prime (above 660) VantageScores (exhibit ES.3).

8 According to Clarity, the greater availability of online short-term installment lending—which more than tripled in loan volume between 2015 and 2019—has led to a sharp increase in overall alternative borrowing nationwide. See Clarity Experian, 2020.

9 VantageScores are available for a larger number of financial services customers with low or moderate incomes. The other option is Experian’s FICO scores. Review of these data showed that about 85 to 90 percent of impact sample members had a FICO score as of the end of followup in December 2019, whereas 96 percent of impact sample members had a VantageScore. See Experian, 2020.

xxi

Exhibit ES.3: Impacts on Use of Financial Services in 2019, Family Self-Sufficiency Impact Sample

FSS

Control Difference

Outcome

Group

Group

(Impact)

P-Value

VantageScore (%)

No score

3.8

3.9

-0.1

0.919

Deep subprime (300–499)

15.3

16.8

-1.5

0.295

Subprime (500–600)

50.1

48.8

1.3

0.522

Near prime (601–660)

15.3

15.3

0.0

0.991

Prime/excellent (661–850)

15.6

15.3

0.3

0.798

Financial service use (%)

No financial services

23.8

26.9

-3.1 *

0.060

Traditional financial services only

63.9

60.6

3.3 *

0.081

Alternative financial services only

1.8

1.8

0.0

0.977

Both traditional and alternative financial services

10.5

10.7

-0.2

0.861

Total balance ($)

All financial services

18,992

18,241

751

0.482

Traditional financial services

18,725

17,979

746

0.484

Revolving credit

1,892

1,969

-78

0.633

Installment credit

16,266

15,870

396

0.691

Alternative financial services

270

263

7

0.865

Single-payment credit

34

34

0

0.998

Installment credit

44

42

2

0.887

Other credit

192

187

5

0.884

Notes: The FSS impact sample includes housing choice voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Sample sizes for specific outcomes may vary because of missing values. Estimates were regression adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. No specific weights were applied to responses to adjust for differences in sample size by housing authority. Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between the FSS group and the control group for continuous variables and selected outcomes expressed as proportions.

In addition, a chi-square test for categorical variables was run to determine whether there is a difference in the distribution of related outcomes by research group. The p-value indicates the likelihood that the difference between the FSS group and the control group arose by chance. Statistical significance levels are indicated as: *** = 1 percent;

** = 5 percent; * = 10 percent.

Sources: MDRC calculations using Experian Vantage 3.0 credit scores and Clarity Clear Early Risk scores The use of alternative financial services also increased over time, suggesting an increase in the use of nontraditional credit products. On average, control group members recorded similar patterns as their program group counterparts on their Vantage and Clarity credit scores.

xxii

• Both study groups incurred high levels of debt, often in automobile and education loans.

According to Vantage data, the typical FSS group member carried a balance of about $9,000 in nonhousing-related debt at study entry. Over the followup period, FSS group members’ average debt more than doubled, reaching an average of nearly $19,000 in non-housing-related debt (see exhibit ES.3, which focuses on 2019, the final year of followup for this report). Installment debt, especially automobile and student loans, accounted for the largest proportion of their non-housing-related debt.

Control group members also accrued non-housing-related debt during 2015 to 2019, but FSS group members took on more in total (about $750 more, a difference that is not statistically significant). As a result, FSS group members averaged larger monthly payments, reaching nearly $400 per month by December 2019 (not shown). During 3 of the followup years (2016, 2017, and 2019), FSS group members paid about $30 (or about 12 percent) more, on average, per month than did control group members for loans from traditional financial services lenders. 10 A small proportion of FSS group members (22 percent) had high-risk, alternative financial services loans between 2015 and 2019. The incidence of alternative borrowing increased somewhat over time—from 7 percent of FSS group members in 2015 to 12 percent 4 years later. Although alternative financial services borrowers were a relatively small proportion of the full FSS group, they were relatively steady users of such credit. They averaged about one such loan per year and incurred about $3,000 (or $600 annually) in alternative financial services debt during the followup period. Control group members showed an almost identical pattern of alternative financial services use, resulting in only scattered and small impacts on measures of alternative financial services use.

• About 6 of 10 members from both groups experienced at least one relatively serious credit problem during the final year of the followup period.

In addition to helping participants “clean up” problems in their credit reports, FSS programs typically counsel them to limit their debt and avoid high-risk lending products, such as those offered by alternative financial services lenders. In 2019, the last year of the followup, almost no members of either research group relied solely on alternative financial services credit, and only a modest proportion (11 percent) of members of each research group combined traditional and alternative financial services credit use. The FSS program led to a small (3 percentage-point and statistically significant) increase over the control group average in the use of traditional lending sources without alternative financial services.

Credit users may also demonstrate financial stress if they keep borrowing until their total balances (the total amount they have borrowed) reach or approach the limits of their available credit. Financial services corporations and lenders assess this potential credit problem in real terms by calculating a customer’s ratio of total balance to total available credit. They often define potential borrowers as being financially at risk if they have a total balance that equals or exceeds 75 percent of their available 10 See Verma et al. (2019). FSS group respondents to the FSS 36-Month Survey (administered in 2017) reported incurring about $14,000 in non-housing-related debt, a somewhat smaller amount than the average for 2017 recorded by Experian. According to survey responses, average debt levels for FSS group members exceeded the control group average by about $1,600, a somewhat larger amount than recorded by Experian.

xxiii

credit. 11 A relatively large proportion of members of each research group (about 40 percent) came close to “maxing out” their available credit during each of these years. Over time, the problem worsened somewhat for each group, with the average balance-to-credit ratio in each group reaching 43

percent in 2019. According to credit data, the FSS program did not affect the percentage of participants who had credit balances close to the maximum.

A perhaps more serious credit problem involves late or forgone payments. About one in six FSS and control group members delayed or missed a payment on the loan balances they owed. In 2019, about 8 percent of the FSS group missed or delayed an alternative financial services loan payment, as did a similar percentage of the control group.

Housing Subsidy Receipt

The FSS program does not require families to give up their housing assistance once they graduate.

They may be required to exit the voucher program if their income exceeds the maximum the voucher program allows for housing subsidy eligibility. Further, unlike other alternative rent policies that allow families receiving federal rental subsidies to keep their extra earnings between regular recertifications, FSS participants are required to report changes in household earnings to the housing agencies. (This same mechanism also allows the housing agencies to credit their escrow accounts for rent increases they may have experienced because of higher earnings.) According to HUD administrative data, nearly 70 percent of the members of each research group continued to receive housing subsidies at the end of the 5-year followup period. The groups also received roughly similar average levels of housing subsidies. In the absence of employment and earnings effects in the first 5 years, these results are not surprising.

Variation Across Sites in Program Impacts

An important element of this evaluation is to examine how program effects vary across types of FSS

programs. These effects could vary across housing agencies because of differences in how FSS

administrators design program policies and set expectations for program participants. Most housing agencies operate relatively small FSS programs, making it difficult, however, to conduct an independent assessment of the effects of any single program. 12 This report explores patterns of variation for clusters of programs, as did the last report and as the evaluation will continue to do through its end.

To examine housing agency-level variation in participation and impacts, agencies with similar implementation features were grouped into a series of “site clusters,” and the magnitude of employment and earnings impacts were estimated for each cluster. For example, the data used to group housing agencies into clusters of low, medium, and high emphasis on monitoring and engagement are based on three program implementation features and practices at study launch: caseload sizes, expectations about program contact, and the program’s focus on short-term goals. FSS programs classified as having a high 11 Experian routinely calculates this measure for several credit sources.

12 The national evaluation includes 18 housing agencies and clusters them on the basis of various dimensions of program practice to assess variation in outcomes and impacts.

xxiv

emphasis on monitoring and engagement have smaller caseloads, expect participants to maintain more frequent contact with FSS coordinators, and focus on having participants establish short-term goals. 13

So far, only one test shows statistically significant variation in impacts among site clusters. Participants affiliated with PHAs with a strong emphasis on monitoring and engagement averaged nearly $13,000

(or 15 percent) less in total earnings over 5 years compared with the control group. The negative impact for PHAs with a high emphasis on monitoring and engagement possibly reflects the tendency of these programs to encourage FSS group members to participate in education and training-related activities, which may decrease their hours or weeks of employment. If so, this negative impact could diminish or turn positive after FSS group members complete their participation. In fact, differences in employment and earnings outcomes were smaller in Year 5 than in previous years and were no longer statistically significant. Moreover, on some credit score outcomes (average score in 2019 and average increase in scores over time), FSS group members in FSS programs with high monitoring and engagement averaged higher scores than their counterparts in the control group. The longer-term data will be important to analyze and interpret this pattern more thoroughly.

At present, the researchers can only conjecture as to what implementation features are associated with the patterns of impacts on employment and earnings. Additional tests of variation in impacts by PHAs will be examined in future years as FSS group members reach the end of their FSS contracts.

Next Steps

This report captures participants’ experiences in Years 4 and 5, the period leading up to their potential graduation from the program. This far into followup, there is limited evidence that the FSS program encourages higher rates of economic mobility or financial security: Both the FSS group and the control group appear to experience comparable trajectories. However, about one-third of the FSS group members continue to be enrolled at the end of the followup period, and new patterns of outcomes and effects may emerge in the year of followup remaining in the evaluation. The final report, slated for release in 2022, will provide more conclusive evidence about FSS participants and how they benefit from the program. With 6 to 7 years of followup, the final report will not only allow the evaluation to track the remaining FSS participants through their extended contract period and examine circumstances after FSS for those who graduated—how they are faring, whether they continue to receive housing subsidies, and how they use their escrow—it will also support an analysis of how families fared during the COVID-19 pandemic.

13 Similar tests were run for housing agencies that varied in the types of services that they emphasized during the goal-setting stage (job search and postemployment support, education and training, or financial counseling and workshops) and also for housing agencies with similar performance levels, as measured by HUD for an earlier (pre-evaluation) cohort of FSS participants.

xxv

Chapter 1. Introduction

The U.S. Department of Housing and Urban Development (HUD) Family Self-Sufficiency (FSS) program is designed to support the economic mobility efforts of low-income families receiving federal rental assistance in the form of public housing or housing choice vouchers (HCVs—also known as

“Section 8,” after Section 8 of the Housing Act of 1937). 14 In the FSS approach, the provision of the rental subsidy is augmented by referrals to services and a long-term escrow account to encourage work and increase earnings. Since the program’s inception in the 1990s, limited evidence has been available about its effectiveness. To build such evidence, HUD commissioned an impact evaluation of the FSS

program in 2012. This report, the third in a series, updates findings on the program’s effects, focusing on longer-term outcomes over 5 years of followup, which is generally the official term of the FSS

program. It also includes, for the first time, a detailed analysis of the program’s effects on study participants’ financial security and credit outcomes.

About 700 public housing agencies (PHAs) receive annual grants from HUD to operate FSS

programs. 15 The funding includes modest resources to hire FSS coordinators (or case managers or coaches) to work with participants to set “self-sufficiency” goals and to refer them to services in the broader community. At enrollment, participants sign a Contract of Participation (COP) and complete an Individual Training and Services Plan (ITSP). The typical FSS contract can last up to 5 years, with extensions possible, during which participants are expected to achieve all agreed-upon goals.

Participants are also offered an escrow account, a work-based financial incentive designed to encourage them to go to work, increase their earnings, and build long-term savings in an interest-bearing account, which the housing agency maintains. Like others receiving housing assistance, FSS

participants see their rents increase when their earnings increase, but the housing agency credits their escrow account with an amount based on their rent increase. In this way, participants accrue escrow balances, which they receive when they graduate from the program—that is, when they are employed and have met all goals outlined in their COP and ITSP; nongraduates forfeit their escrow accruals. 16

The national evaluation uses a randomized controlled trial to test whether FSS achieves its intended effects and puts families on a track to self-sufficiency. Randomized controlled trials employ an experimental design that compares the outcomes of a program group whose members are eligible to participate in the intervention with those of a control group whose members are not eligible to participate in the intervention. Random assignment of study participants to either a program group or a control group is designed to ensure that the individuals in the program and control groups are similar at the start of the study. 17 Differences between the program and control groups’ outcomes reflect the 14 Project Self-Sufficiency, Operation Bootstrap, and the Jobs Plus program are some examples of efforts that use the housing subsidy platform to support work. Other reforms built into the housing subsidy rent policies, similar to the ones being tested as part of HUD’s Rent Reform Demonstration, also have a core objective of “making work pay”

(see Riccio, Deitch, and Verma, 2017).

15 This report uses the terms public housing agencies and public housing authorities, both PHAs, interchangeably.

16 In the HCV program, escrow forfeiture becomes part of housing assistance payment (HAP) equity and is restricted to HAP activities.

17 Randomization does not affect study participants’ use of resources that are available in the general community.

1

program’s impacts. Statistically significant differences indicate that the impacts can be attributed with a high degree of confidence to the intervention rather than to chance.

The evaluation focuses on FSS programs implemented by non-Moving to Work (MTW) housing agencies. At the time of study design, MDRC and HUD agreed to exclude the MTW housing agencies, which have the administrative flexibility to modify their FSS programs without legislative or regulatory changes, and FSS programs for public housing residents, which serve considerably fewer participants. Eighteen non-MTW housing agencies operating FSS programs for tenant-based housing choice voucher recipients agreed to participate in this evaluation and together enrolled 2,656 study participants. 18

The first report from this evaluation introduced the study design and detailed the findings over the initial 18 to 24 months of followup (Verma et al., 2019). It documented FSS implementation strategies across the 18 sites, participants’ engagement in services and activities, and program impacts on labor force outcomes and receipt of government benefits. The first report showed that housing agencies have substantial discretion over FSS program implementation, leading to broad variation in how case management services are delivered—a theme that continues to surface in evaluation reports. The early data also showed that enrollment in the program increased participants’ engagement in a range of employment-related services relative to the control group. Building on this, a second report, covering 36 months, examined whether bigger differences between the program and control groups begin to emerge over a longer-term followup period (Verma et al., 2021). The report looked at a broad range of outcomes, including participants’ employment, earnings, and material, financial, and economic well-being. The report documented few notable program effects in the interim period.

In 2018, HUD extended the national FSS evaluation through 2021, allowing 6 to 7 years of followup for the study sample. With the exception of participants who receive an extension, most FSS

participants in the study are expected to reach their 5-year FSS contract terms in the followup period covered by this report. Focusing on 60 months of followup, this report provides an update on program implementation and program outcomes described in previous reports. In addition, with HUD’s support, MDRC acquired credit data for study participants, allowing for a comprehensive and unusually detailed analysis of program effects on the credit behavior of voucher holders enrolled in the FSS evaluation.

The analysis period covered in this third report—2012 to 2019—ends before the start of the COVID-19 pandemic, which hit the nation in March 2020. The FSS programs in this study and their housing agencies had to make dramatic changes in how they delivered services, shifting to online engagement with program participants. 19 Although none of the service changes or disruptions affect the outcomes examined in this report, they occur during the longer-term followup period for this evaluation. The final report, due in 2022, will take into account the pandemic and its implications for the program’s effects on participants’ work outcomes, graduation and escrow receipt, financial well-being, and 18 In fiscal year 2014, funding streams for HCV and public housing FSS programs were merged, and housing authorities could submit one application for their annual grants. This consolidation of funding streams also meant that PHAs could use the funding to serve both public housing and housing choice voucher FSS programs, if applicable.

19 During the field research period for this report, the MDRC team also followed up with staff at 17 of the 18 sites to get an update on their program operations and adaptations because of COVID-19.

2

receipt of housing and other government subsidies. No further qualitative data collection is planned for the remainder of the evaluation, but a brief survey, which will be fielded to capture the longer-term outcomes and postexit circumstances of former FSS participants in the study, has been adapted to collect information on how families fared in the face of the economic shocks caused by the pandemic.

Housing Assistance and Employment Context

Low-income renters receive federal housing subsidies in three main forms: HCV, project-based rental assistance, and public housing assistance. The HCV program, the nation’s largest rental assistance program, provides subsidies to slightly more than 2 million households, allowing them to rent homes in a neighborhood of their choice if the housing meets HUD inspection standards and the landlord is willing to accept housing vouchers. 20 Households contribute 30 percent of their monthly income to their rent (minus adjustments to defray childcare expenses or for other reasons), and the HCV program covers the remaining rent and utilities expenses, up to the local payment standard.

Eligibility for housing vouchers is limited to households with income below 80 percent of the median income for the metropolitan area or county in which they choose to live. However, the program prioritizes extremely low-income households by reserving at least 75 percent of available vouchers each year for households with income at or below 30 percent of the area median income. 21 There are no time limits on how long households may receive such subsidies as long as they remain eligible on the basis of their household income. However, once household income exceeds the limit of 80 percent of the area median income for 6 consecutive months, eligibility for the subsidy ends.

As with any means-tested program, the provision of a government benefit has the potential to affect the recipients’ work efforts. Some analysts have argued that housing subsidies not only improve access to decent housing but may also promote work. 22 This view holds that the housing stability that comes from rent subsidies may enable recipients to focus on employment or build human capital and that when housing assistance takes the form of vouchers, households are able to move to better quality neighborhoods that offer better prospects for their families. 23 This view, however, is challenged by evidence that suggests that although many households benefit in selected ways, housing assistance alone may not, on average, improve employment outcomes (Jacob and Ludwig, 2008; Mills et al., 20 Housing vouchers became part of U.S. housing policy in the 1970s. See Schwartz (2006) for additional background information on the HCV program.

21 Under current rules, the escrow deposit calculation varies depending on the family’s income level, with different rules for families below 50 percent and between 50 and 80 percent of the area median income levels. These rules make it difficult for those with higher incomes to earn escrow. Among the changes planned (not yet enacted) is the stipulation that families with income above 50 percent (up to 80 percent) will accrue escrow using the same calculation as other families.

22 See Sard and Waller (2002) for one discussion on this perspective. Nonelderly and nondisabled households account for only about 49 percent of all voucher-assisted households (Center on Budget and Policy Priorities, 2017), and their employment and earnings trajectories have been a long-standing policy concern.

23 Recent research has also shown positive long-term effects for young children of the original Moving to Opportunity demonstration and, in a separate nationwide study, the benefits that low-income children experience in adulthood when they move to high-opportunity neighborhoods as children. See Chetty and Hendren, 2017a, 2017b.

3

2006; Shroder, 2010). 24 In this case, voucher holders may feel less pressure to work when their housing expenses are subsidized, and their remaining income is adequate to sustain their family without the cost of seeking work (because of transportation expenses, for example) or finding adequate childcare while working. However, as mentioned previously, the HCV program’s rent rules could also discourage work. Voucher holders must pay 30 percent of their earnings for rent until they are no longer eligible for this subsidy. Thus, their participation in the housing choice voucher program subjects them to an implicit “tax” on additional earnings that could negatively affect their inclination to work (Popkin et al., 2000, 2010; Popkin, Cunningham, and Burt, 2005). The FSS program’s interest-bearing escrow account, designed to encourage work, is intended to address this issue. 25

Given the potential employment advantages that voucher receipt may offer and the potential work disincentives inherent (because higher earnings can result in higher rent or reduced benefits), researchers and policymakers have questioned the expected effects of FSS and other similar programs.

The Family Self-Sufficiency Program

In 1990, Section 554 of the Cranston-Gonzalez National Affordable Housing Act established the FSS

program against a backdrop of policy discussions about persistent poverty among beneficiaries of government programs. Although operated by about 700 housing agencies around the country, the FSS

program itself reaches a fairly small fraction of all voucher families—partly a reflection of the funding appropriated by Congress to operate this program. 26 Early research has also shown that, at least from the perspective of program staff members, voucher recipients may not fully understand the terms of the FSS program and believe that they risk losing their housing assistance and other public assistance benefits by not fulfilling their FSS contracts (Rohe and Kleit, 1999). These program operators also believe that issues such as caregiving responsibilities, potential lack of motivation, transportation barriers, and distrust of social programs could be possible enrollment deterrents (Rohe, 1995; Rohe and Kleit, 1999). MDRC’s site recruitment discussions with program staff evoked similar reactions.

In the most recent round of annual grants, HUD made available about $80 million in funding for the FSS program (HUD, 2019). The FSS grants, which offer support for coordinator positions, include no provisions for program management and administrative costs. 27 Grants issued in fiscal year 2018 show that the size of the programs funded can range from as few as 15 participants in the smallest program 24 For example, the findings from the Welfare-to-Work program conducted in the early 2000s found that having and using a voucher reduced employment rates and earnings amounts in the first year or two after random assignment, but the small negative effect of vouchers disappeared over time, and vouchers had no significant effect overall on employment and earnings during 3.5 years of followup. The most rigorous evidence from the United States suggests a loss of 10 to 20 cents in earnings per dollar of assistance (see Shroder, 2010).

25 Changes to rent policies could also incentivize work and increase earnings for HCV participants, as is being tested as part of HUD’s Rent Reform Demonstration (see Riccio, Deitch, and Verma, 2017).

26 HUD makes funding available for FSS programs through annual grants, but such funding is limited to the amount that Congress appropriates.

27 HUD funds the FSS programs through the FSS Notice of Funding Availability (NOFA). Housing authorities apply for this funding on an annual basis.

4

to more than 1,000 in the largest. 28 Thus, although FSS is one of HUD’s main employment-focused initiatives, particularly for voucher holders, it remains a small program at the federal and local levels.

Housing agencies operating FSS programs are required to prepare an FSS Action Plan and have it approved by HUD. The Action Plan is expected to detail program parameters—for instance, size of the program and the population served, types of services that will be offered, and program rules and policies. Once HUD approves the plan, housing agencies have a fair amount of flexibility regarding how they structure service delivery or implement their programs. In 2017, HUD published its first comprehensive resource guide for FSS operators. Without enforcing a particular service framework, the guide provides practical, hands-on tips for operating the FSS program (HUD, 2017b). 29 As documented in this report, the evaluation captures how the programs operated on the ground during the followup period. In some cases, the local practices, as described by PHA administrators and case managers, differed from HUD’s latest thinking about program best practices and, in some instances, program requirements. In addition, HUD’s Final FSS Rule, which went into effect in November 2022, addresses some of the concerns raised and recommendations articulated by program administrators and staff.

In 2018, HUD announced a new performance measurement system to assess programs receiving HUD

funding for FSS programs (HUD, 2018). The performance score is a composite based on three measures: the extent to which the earnings of FSS participants increase over time after joining the FSS

program, the FSS graduation rate, and the portion of expected participants served. 30 HUD plans to use the performance metric to identify high- and low-performing FSS programs, which could inform its understanding of best practices and delivery of technical assistance. 31 When implemented, this performance measurement system would add a new monitoring context for FSS programs nationwide. 32 Interviews with FSS program staff members, discussed later in this report, provide some early front-line reactions to the potential implications of this new assessment system on site-level practice.

Core Features of the FSS Program

Guided by statutory requirements and HUD regulations, the FSS program is anchored around two core components: coordination of support services and an escrow savings account (see exhibit 1.1). Except for the escrow account, local housing agencies can decide how to structure their case management and case coordination services—an element of flexibility offered by the program.

28 MDRC analysis of HUD FSS grant awards included in the NOFA.

29 In 2019, all FSS applicants are required to have a representative from their agency complete an online training (see HUD, 2019).

30 The three measures are weighted as follows: earnings (50 percent), graduation rate (30 percent), and participation rate (20 percent).

31 Toward these goals, at least once per year, HUD will analyze data collected through the Public Housing Information Center to calculate performance scores for each FSS program that received an FSS coordinator grant in one or more of the past three fiscal year NOFA competitions.

32 As of early 2023, however, HUD has yet to use this system to set funding levels or for other administrative purposes due to a congressional prohibition.

5

Participation in the FSS program is voluntary. Housing agencies promote the program through various communication channels, including flyers, program brochures, newsletters, websites, and scheduled briefing sessions. More informal channels, such as referrals from friends and relatives, also help spread the word about FSS programs. Once enrolled, program services are designed to help participants make progress toward their FSS goals. Although all adults in FSS households are encouraged to seek employment, only the household head—typically the voucher holder—is required to meet the employment requirements of the FSS contract to graduate and collect the escrow funds. 33 Participants attain their goals, graduate from the FSS program, and access the escrow savings their households have accrued, usually within 5 years.

Housing agencies place no restrictions on participants’ use of escrow funds, but they report that households most commonly use these resources to start a new business, repair credit, buy a home, or pay for education. 34 Some programs also consider interim disbursements, or partial payments before graduation, as long as participants use the funds to meet approved expenses related to their self-sufficiency goals. Tuition, car purchase, credit repair, uniforms, tools, homeownership, or business startup are examples of expenditures that can be approved.

33 Amendments to the FSS statute enacted in 2018 (but not yet in effect) will allow for more flexibility for families to meet graduation requirements through the employment of any adult household member, not solely the head of household.

34 The 42-month survey conducted as part of the Work Rewards demonstration sheds some light on the desired uses of the escrow. Administered before the escrow funds had been disbursed to graduates, the Work Rewards survey showed that about one-third of the respondents indicated that they would save their escrow money for an emergency.

Other uses included saving for children’s future educational expenses, paying for basic necessities, and buying a house. See Verma et al., 2017.

6

Exhibit 1.1. Core Components of the HUD Family Self-Sufficiency Framework

FSS = Family Self-Sufficiency.

There is little published data on FSS graduations or escrow disbursements. A HUD Report to Congress, focusing on the period of July 2017 to June 2018, indicated that 47 percent of graduates during this period had escrow savings averaging approximately $7,700 (HUD, 2020). The Opportunity NYC‒

Work Rewards demonstration (hereafter, Work Rewards), the first randomized controlled trial of an FSS program, followed about 1,600 FSS participants over 6 years and showed that about 45 of the FSS

participants graduated and received escrow disbursements. Households in the FSS-only group received an average of about $3,800 (the FSS-plus-incentives group, which received FSS and two additional special workforce incentives, received nearly $700 more in escrow disbursements on average than the FSS-only group). Escrow disbursements in the bottom quartile of payments averaged less than $1,000, and the top quartile averaged more than $15,000 (Verma et al., 2017).

Graduation from the FSS program is an official measure of program success. The individual enrolled in the program, typically the head of household, must complete all the agreed-upon goals and activities listed in the ITSP, including the employment requirement. If the head of household is unable to meet the employment requirement, and there are other earners in the household, the family is not eligible to receive any accrued escrow at the time of graduation, which is a potentially limiting aspect of the escrow component for households with multiple adults. This receipt requirement applies to all 7

members of the household, and no member of the household may receive Temporary Assistance for Needy Families (TANF) cash assistance for the 12 months leading up to graduation. As discussed in chapter 2, some FSS programs require participants to work toward additional goals to graduate. 35 It is possible, though, for participants to graduate from the program and not receive any escrow funds. This outcome could happen for various reasons, including not having had the earned income increases that are necessary to trigger escrow accumulation. 36

HUD also requires all FSS programs to form a Program Coordinating Committee (PCC). With referrals central to the FSS service delivery model, the intent of the PCC is to create a mechanism by which the service providers in the community can become invested in the success of the FSS program. The PCC, which operates as a collaborative, is intended to provide both guidance to the FSS program and direct services to the program’s clients. The PCC usually comprises service providers that accept FSS

referrals.

How Might the FSS Components Help Participants Advance?

Exhibit 1.2 offers a simplified schematic to identify the two main mechanisms by which the FSS

program might help participants advance: (1) increasing the payoff through case management and referral services and (2) incentivizing work through the escrow account.

Mechanism 1. Increase the Payoff Through Case Management: Although FSS programs may vary in their service delivery approaches, they all include some dimension of goal setting, needs assessment, and referrals to services that may help participants overcome barriers to work. Typically, FSS

coordinators (or case managers) work with participants (and sometimes other household members) to identify goals the participant will aim to achieve during the 5 years of program participation. During this process, they discuss the types of supports participants might need to advance toward their FSS

goals, such as the following:

• Securing childcare to make balancing work and home life commitments more feasible.

• Engaging in and completing education and training to improve employment prospects and create pathways for advancement.

• Finding and maintaining stable employment.

• Establishing, repairing, or improving the participant’s credit score to increase employment prospects and decrease reliance on high-cost alternative credit sources, such as pawn, automobile titles, and payday loans.

Progress along each of the above pathways would make it easier and more remunerative to work.

Furthermore, some of these pathways, such as credit score improvement, may also help participants manage their financial resources and thus improve material hardship irrespective of the program’s impact on employment and earnings.

35 This practice is not consistent with HUD regulations or guidance, and it is further discussed in chapter 2.

36 In Work Rewards, 30 percent of FSS participants who graduated did not receive an escrow disbursement. A recent HUD analysis indicates that 63 percent of participants earn some escrow while in the program, and 47 percent of graduates earn some amount of escrow (see HUD, 2020).

8

Improving outcomes, such as education or credit, for example, may be difficult with a program that offers less intense case management or support (as in a “light-touch” service coordination approach, which may be less structured, require infrequent contact with staff, and lack a strong monitoring and engagement focus). In general, outcomes will depend on several factors, including the strength of the service providers (and the service providers’ models) in the local community, the case management model (including the type and frequency of followup), and the capacity and willingness of the participant to follow through on a course of action. This willingness is directly targeted in other interventions that apply a more behavioral science-informed coaching approach (Guare and Dawson, 2016), but it is somewhat weakly targeted in FSS programs, given the fairly far-off possible reward of the escrow disbursement. 37 Because some of these factors vary among the housing agencies in this study, the evaluation will explore how program practices affect participant outcomes, if at all.

37 These efforts, which focus on “executive skills”—or roughly, the capacity to plan, manage, and cope—attempt to achieve larger impacts than are typically achieved with conventional case management. MDRC’s MyGoals Demonstration is testing the effects of executive skills-informed coaching and incentives on work outcomes.

9

Exhibit 1.2: Simplified Schematic of the Family Self-Sufficiency Program Theory of Change FSS

Interim Outcomes

Long-Term Outcomes

Initial and ongoing

coordinaton of—and

Increased participation

Increased financial security

referral to—range of

in education and

services:

training/supportive

services/workshops/

Improved material well-

being

Employment,

classes

education and training,

Reduced reliance on

financial literacy/

government assistance

management,

homeownership,

health-related services,

social services,

supportive services

Increased

employment and

earnings

Incentive to increase

Escrow accrual

earnings through the

Escrow received at

long-term savings escrow

graduation

account

Mechanism 2. Incentivizing Work by Building Escrow: As described, the escrow account is designed, in part, to counteract the disincentive effect of the implicit “tax” built into housing voucher rent rules. Specifically, 30 percent of a recipient’s earnings must be contributed to rent, so 30 percent of any earnings gains are diverted to increased rent payment. This rent policy could discourage additional work by decreasing the marginal gain for any added hour of work done. It may also discourage any work effort among those participants who are not working by reducing the effective wage rate below the “reservation wage,” which is the wage rate necessary to induce potential workers to enter or reenter the labor market.

The degree to which this tax on wages discourages work—or efforts to work harder or find a better job—is not well established. Consequently, it is difficult to estimate the potential impact of an FSS

program and, more specifically, the impact of the escrow component that is intended to cancel out that disincentive.

Two additional aspects are to be considered: the effectiveness of escrow as an incentive and the factors that may constrain participants’ responses to it. First, escrow represents an incentive to work, but it cannot be earned until participants fulfill graduation requirements, so it is a distant and uncertain reward. It may not, therefore, effectively (or completely) counteract any disincentive effect of the HCV

rent rules because those costs are immediate and certain. The current structure of the escrow account was the impetus for testing more immediate, work-related cash incentives alongside the typical FSS

program escrow incentive as part of the Work Rewards demonstration.

10

Second, FSS participants face various barriers that may limit their employment prospects and increase the cost of work (beyond the contours of the local job market and proximity to work). Like most low-income households, participants may receive multiple means-tested benefits, including Supplemental Nutrition Assistance Program (SNAP) and TANF benefits. These benefits work similarly to HCV rent rules, and uncertainty or fear about decreases in or loss of these benefits could discourage work efforts.

Poor educational attainment, criminal history, and poor or no credit history may limit the types of jobs participants can qualify for and obtain, thus reducing the payoff from work or increased hours of work.

Likewise, family obligations and responsibilities (such as taking care of children or loved ones with disabilities or who are sick), being sick or having disabilities, and the need to secure employment-related transportation may further discourage work or additional hours by increasing the costs associated with employment (by reducing the effective wage, potentially below zero). For some, a cost-benefit calculation of minimum wage with uncertain hours may conclude that, at least in the short run, not working is a better choice for the family. Thus, in isolation, housing voucher rules may constitute only a small part of the decision to not work or to not work more hours. For this reason, the previously described support services may be important components.

The National Family Self-Sufficiency Impact Evaluation

Until recently, questions about the FSS program’s effectiveness had not been investigated using methods that would support unambiguous causal inferences. 38 Drawing on this conceptual framework in exhibit 1.2, the evaluation, which relies on a randomized controlled trial, assesses how FSS affects program participants' core economic mobility outcomes. In addition to looking at effects for the full sample, pooled across all study sites, the evaluation also examines the effects for certain subgroups to better understand what works best for whom. For example, the program may have larger effects for participants who are not employed at study enrollment because it is often easier for individuals to advance to higher wage jobs once they are already employed than to get a job in the first place. Given referrals to supports and services, the FSS program may also have different effects depending on a participant’s barriers to work or preparation for work. Based on program theory, prior evidence, or policy interest in a given subgroup, the evaluation focuses on subgroups defined by participant characteristics at enrollment: work status and educational attainment. 39 In addition, given the variation in program implementation practices across sites, the subgroup analysis also considers program impacts for participants who are exposed to different program engagement and implementation strategies or program “types.”

38 Work Rewards was the first to use a random assignment design to test the effects of an FSS program (Nuñez, Verma, and Yang, 2015; Verma et al., 2012, 2017). Results from that study, which focused on a single site, showed that the program was effective in enrolling participants in education and training activities or linking them to financial literacy programs, but there were few gains in the range of economic and material outcomes tracked for the sample. A small quasi-experimental analysis of the Compass Working Capital FSS programs in Lynn and Cambridge, Massachusetts, showed that the Compass FSS programs were associated with an average gain in annual household earnings of $6,305

between the fourth quarter of 2010 and the first quarter of 2016 (Geyer et al., 2017).

39 These subgroups were identified during the design stage. Additional subgroups examined are mainly considered exploratory analyses.

11

The evaluation relies on a combination of qualitative and quantitative data, and it considers questions related to both program implementation and impacts. Exhibit 1.3 lists the data sources central to this report and the followup period covered by each.

Exhibit 1.3: Data Sources for the Family Self-Sufficiency Study

Data

Data Period

Length of Followup

Baseline characteristics

October 2013–December 2014

At random assignment

Wage records

April 2013–December 2019

60 months (20 quarters)

FSS graduation and escrow data

October 2013–March 2020

63 months

Housing subsidy data

October 2013–December 2019

60 months

Experian and Clarity credit data

December 2012–December 2019 60 months

FSS staff interviews

March 2020–June 2020

Program Years 4 and 5

Eighteen housing authorities in seven states—California, Florida, Maryland, Missouri, New Jersey, Ohio, and Texas—were selected to participate in the FSS study (see exhibit 1.4). These sites broadly represent the contexts within which FSS programs operate: small, mid-sized, and large FSS programs and small, mid-sized, and large voucher programs. 40 Although no data were available to distinguish typical or higher quality FSS programs during site recruitment, MDRC and HUD sought to include a broad range of sites, including ones with different program sizes, staff caseloads, and case management or coordination practices. 41 Site-specific enrollments, including program and control groups, ranged from 50 to 350, reflecting varying enrollment targets based on the sizes of the existing programs. 42

40 See Verma et al. (2019) for site recruitment details. Site selection considered various factors, such as program size, the possibility of building clusters of sites within states, regional and local diversity, and varying program approaches.

MDRC examined HUD data from 2010 to 2012, creating a list of potential sites; conducted phone followup with about 60 program administrators; visited 27 sites; and ultimately negotiated agreements with 18 sites.

41 HUD’s performance management system ranking for FSS programs was not available until 2018. The ranking system classifies 20 percent of FSS programs as high performing, 60 percent as standard, 10 percent as low, and 10

percent as troubled.

42 Seven PHAs agreed to enroll fewer than 100 study participants, five agreed to enroll between 100 and 200, and another six agreed to enroll between 200 and 350. PHAs operating larger FSS and HCV programs agreed to larger samples.

12

Exhibit 1.4: Public Housing Agencies Participating in the National Family Self-Sufficiency Evaluation MARYLAND

FLORIDA

13

The Study Sample

From October 2013 to December 2014, the 18 public housing agencies participating in the evaluation enrolled and randomly assigned 2,656 households. This figure includes a small number of households that later withdrew voluntarily from the study, or that program staff members determined to have been ineligible for FSS at their time of random assignment and removed from the study, as well as households headed by individuals 62 years or older who are not the focus of the main impact analysis.

Excluding those individuals reduced the sample to 2,556. These 2,556 study participants compose the sample for the entire impact analysis in this report (or the “impact sample” in exhibits).

Exhibits 1.5 and 1.6 present sample characteristics from the baseline survey that participants completed at study enrollment. As shown, the sample is largely composed of households with children, and 76

percent of households included a minor child. Nearly 34 percent of participant households included another adult. Although labor market outcomes for other adult household members are not analyzed here, their earnings affect household subsidies and contribute to household escrow accruals; other adults may also benefit directly or indirectly from FSS case management. Approximately 70 percent of study households reported receiving SNAP benefits; 16 percent reported receiving TANF benefits.

The FSS program is designed to help participants move off cash assistance, such as TANF, and reduce reliance on public assistance in general. 43 Slightly more than one-half of the sample (54.2 percent) reported having received Section 8 housing assistance for 6 years or less. About 31 percent reported having received Section 8 housing assistance for 10 years or more.

Exhibit 1.5: Baseline Characteristics of Households in the Family Self-Sufficiency Impact Sample

Characteristic

Impact Sample

Average number of household membersa

3.2

Average number of adults in householda

1.5

Households with more than 1 adult (%)

33.7

Average number of children in household

1.8

Number of children in household (%)

0

23.8

1

22.7

2

24.7

3 or more

28.8

For households with children, age of youngest child (%)

0–2 years

20.8

3–5 years

20.4

6–12 years

41.3

13–17 years

17.5

Primary language spoken at home is English (%)

92.2

43 The Work Rewards data suggest that more families may enter and exit the TANF system over the followup period.

14

Characteristic

Impact Sample

Receives TANF (%)

15.8

Receives food stamps/SNAP (%)

69.6

Length of time receiving Section 8 housing choice voucher (%)

Less than 1 year

5.0

1–3 years

27.6

4–6 years

21.6

7–9 years

15.2

10 years or more

30.6

Total annual household income (%)

$0

4.5

$1–$4,999

17.0

$5,000–$9,999

18.7

$10,000–$19,999

31.9

$20,000–$29,999

19.3

$30,000 or more

8.5

Payment for rent and utilities (%)

$0

1.9

$1–$199

15.0

$200–$399

24.3

$400–$599

21.3

$600–$799

15.1

$800 or more

22.4

During the past 12 months, household experienced at least one financial hardship (%) 59.0

Not able to buy prescription drug

13.3

Not able to buy food

28.9

Not able to pay telephone bill

28.2

Not able to pay rent

18.5

Not able to pay utility bill

43.4

Sample size

2,556

aMaximum response option for number of adults in a household is four.

Notes: The FSS impact sample includes housing choice voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Sample sizes for specific measures may vary because of missing values. Rounding may cause slight discrepancies in calculating sums. Detail may sum to more than total for questions that allow more than one response.

Source: MDRC calculations from Baseline Information Form data

15

Exhibit 1.6: Baseline Characteristics of Heads of Household in the Family Self-Sufficiency Impact Sample

Characteristic

Impact Sample

Sample member characteristics

Female (%)

90.6

Average age (years)

39

Marital status (%)

Married, living with spouse

7.7

Married, not living with spouse

6.8

Cohabitating

1.4

Single, widowed, or divorced

84.0

Race/ethnicity (%)

Black, non-Hispanic/Latino

73.3

Hispanic/Latino

15.8

White, non-Hispanic/Latino

6.7

Other

4.2

Education

Highest degree or diploma earned (%)

GED certificate

3.0

High school diploma

10.6

Some college or received technical/trade license

55.0

Associate’s or 2-year college degree

10.8

4-year college or graduate degree

6.5

None of the above

14.0

Has trade license or training certificate (%)

47.0

Employment status

Currently employed (%)

56.2

Regular job

48.4

Self-employed

4.2

Temporary or seasonal job

3.5

Currently working 35 hours or more per week (%)

30.5

Average hours worked per week

18.3

Average weekly earnings ($)

213

Barriers to employment

Has any problem that limits work (%)

41.2

Physical health

18.8

Emotional or mental health

7.6

Childcare access or cost

17.8

Need to care for disabled household member

7.3

Previously convicted of a felony

6.3

Limited English-speaking ability (%)

3.8

Does not have access to transportation for employment (%)

16

Characteristic

Impact Sample

No access to public transportation

17.8

No access to an automobile

18.2

FSS program

Heard of escrow before random assignment (%)

44.0

Interest in FSS services related to (%)

Job-related services

70.5

Social services

32.4

Financial services

95.5

Sample size

2,556

Notes: The FSS impact sample includes housing choice voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Sample sizes for specific measures may vary because of missing values. Rounding may cause slight discrepancies in calculating sums. Detail may sum to more than total for questions that allow more than one response.

Source: MDRC calculations from Baseline Information Form data

The sample is predominantly female (90.6 percent), with an average age of 39 years at study enrollment (see exhibit 1.6). About 41 percent of the sample reported some barrier to employment. Physical health (18.8 percent) and access to affordable childcare (17.8 percent) represented the most common difficulties. Approximately 56 percent of study participants were working at the time of study enrollment (with about 30.5 percent working full time). Work status and earnings are primary outcomes of interest for this study because FSS programs are designed to enable and encourage (more remunerative) work and because employment is one of the requirements for program graduation and access to accumulated escrow.

Overall, study households and heads of household are broadly similar to those in the FSS national population, with some notable differences. 44 Sample members are somewhat more likely to have no children present (23.8 percent in the study sample have no children versus 17.6 percent in the national FSS population). Study households are less likely to report no income (4.5 percent versus the national FSS figure of 6.5 percent) but are also less likely to report income of $30,000 or more (8.5 percent versus 13.9 percent). Sample members also report higher levels of TANF and SNAP benefit receipt than the averages for the national FSS population (15.8 percent versus 10.0 percent for TANF; 69.6

percent versus 37.5 percent for SNAP). Study sites tend to run larger housing voucher and FSS

programs and spend more on rent and utilities per participant than the national population of housing agencies that run FSS programs, a consequence of the need to select sites that would allow for sample recruitment within the required 1-year window.

44 See Verma et al. (2019). To assess whether individuals and households in the study were broadly similar to their site- and national-level counterparts, MDRC compared sample members with the broader FSS population in the study sites and with the national population of FSS participants and housing voucher holders. The team also compared the study’s participating PHAs with all PHAs operating FSS programs.

17

Structure of This Report

The report is organized around seven chapters, with chapters 2, 3, 4, and 6 updating interim findings.

Chapter 2 focuses on program strategies to keep participants engaged over the longer term, especially as they approach graduation. Chapter 3 shares the latest results on graduation and escrow disbursements and describes the characteristics of the participants who graduate and receive escrow.

Chapter 4 updates the analysis of the program’s effects on key labor market outcomes (employment and earnings)—assessments based on National Directory of New Hires wage records comparing average outcomes for FSS and control group members over the 5-year followup period. Chapter 5

introduces the credit analysis—a new source of data and analysis for this study—and describes credit use patterns among study participants and the FSS program’s effects, if any. Chapter 6 investigates the variation in impacts and outcomes for subgroups defined by baseline individual characteristics and program features. The final chapter summarizes the essential takeaways from this longer-term assessment and describes future work on this evaluation.

18

Chapter 2. Program Implementation Update and Reflections

As described in chapter 1, Family Self-Sufficiency (FSS) programs incorporate case management and goal setting, referrals to services, and escrow accounts. The FSS framework provides public housing agencies (PHAs) with considerable leeway in setting local policies, approaches, and staffing arrangements that influence how the programs work with FSS participants to build assets and work toward self-sufficiency. In addition to variations in program operations across PHAs, changes in HUD

policies and guidance provide an important context for understanding how these programs work to support participants.

A previous report, which focused on FSS policies and practices about midway through the program’s 5 years, provided an overview of how the programs had changed since the study began, how they were working with participants who were a little more than midway through the 5-year program, and programs’ strategies for keeping participants engaged. This chapter highlights major changes reported by staff in their FSS program policies and practices since the last update and focuses on case management approaches, especially as participants enter the last year or 2 of their 5-year contracts.

The chapter also takes a closer look at the Program Coordinating Committees (PCCs), a central component of the FSS framework, and their role in supporting the service needs of FSS program participants. The chapter also describes staff members’ understanding of HUD’s proposed performance measurement system and their reflections on their programs’ successes, challenges, and best practices.

Data Sources and Followup Period

For the final round of interviews with program staff, the MDRC evaluation team used structured interview protocols, which covered topics including updates on policy and practice, case management in participants’ final years of FSS, service referrals and the role of the PCC, and staff member reflections on their FSS programs. Between March and June 2020, the evaluation team conducted these interviews with program staff members at 17 of the 18 FSS study sites that continued to participate in the long-term evaluation. 45 The first of these interviews, held in person in mid-March 2020, occurred just as the COVID-19 pandemic was accelerating across the United States and immediately before many states began to implement stay-at-home orders. The remaining visits to FSS study sites were canceled as a result, and all remaining interviews were conducted by phone or video calls. Program supervisors across all 17 participating FSS sites were interviewed, as well as up to two case managers from each of eight of the sites. This final round of interviews occurred shortly after the last FSS study participants were expected to have completed their 5-year contracts. 46

45 The Housing Authority of the City of Los Angeles, one of the original FSS study sites, declined to participate in HUD’s extension of the evaluation.

46 The last enrolled sample member had a December 22, 2019 contract end date. As noted later in this report, some FSS study participants that were expected to complete their 5-year contracts in this period may have received an extension.

19

Program Practice and Policy Updates

For the most part, program policies and practices across the FSS study sites remained largely unchanged in the last few years. 47 A few notable exceptions are discussed in this section.

Staffing and Caseloads

Staffing configurations remained relatively stable since the 2018 research interviews, although some sites experienced considerable staff turnover: eight sites had the same number of case managers in 2020; four sites added one or two new case managers; 48 one site was funded for three fewer case managers; and four sites had unfilled positions and were therefore short staffed, with higher caseloads.

One of these sites characterized their caseloads as including roughly 25 percent of individuals who would likely make progress toward their goals even without FSS, 25 percent who would likely not make progress no matter what the case managers do, and 50 percent “in the middle,” who could really be helped by the program. These case managers tended to focus most of their energy on the middle 50

percent, especially given their large caseloads. Other sites described similar caseload characteristics.

Caseload sizes ranged from 27 at the smallest site to roughly 150 at the largest. 49 FSS staff members continued to have the same mix of functions they reported in the previous round of research, with some programs using FSS-funded staff solely to work on FSS-related functions and other programs using FSS staff to also handle HCV functions for FSS participants. 50

Interim Escrow Disbursements

FSS programs may allow participants to receive a portion of their escrow funds before graduation to help the family accomplish its goals. All but one of the FSS programs in the national evaluation that participated in the 2020 round of data collection allow participants to receive a portion of the funds in their escrow account early (for example, prior to graduation). This represents a slight increase in the number of programs allowing interim disbursements. In 2018, three programs did not allow early escrow withdrawals. Program policies vary in terms of how many times a participant can request an interim disbursement and what percentage of their escrow balance they can request; most programs limit the request to 25 percent or 50 percent of the balance. Some FSS programs use the disbursement as a last resort only when other available funds for goal-related issues such as car repair, books, and uniforms are exhausted. Since 2018, several programs have increased the limits on the maximum 47 See appendix A.

48 HUD funds the FSS programs each year through the annual FSS Notice of Funding Availability (NOFA). Funding is limited to the amount appropriated by Congress. Housing agencies must apply for this funding on an annual basis.

Staff positions are funded according to program size. For example, the 2019 NOFA stated, “As in prior years, eligibility for funding will be based on the number of FSS program participants in your entire FSS program during the target period, according to a formula that requires 15–24 families to support one part-time position, 25 families to support one full-time position, and an additional 50 families to support each additional position beyond the first full-time position (75 families for two full-time positions, 125 families for three full-time positions, etc.).” Furthermore, the NOFA states, “While this NOFA requires at least 25 FSS participants to qualify for one full-time FSS Coordinator position, grantees are encouraged to serve at least 50 FSS participants per full-time coordinator position.” (See HUD, 2019.)

49 This number does not include instances in which unfilled positions effectively increased caseloads.

50 A couple of these sites split the responsibilities, with some staff handling only FSS case management functions and the other(s) handling HCV functions.

20

amounts, number of withdrawals, or both that participants can receive. One program relaxed its policy after losing funding used for stipends and incentive payments by removing its previous limit of one withdrawal. However, as discussed later in this report, few participants actually received interim disbursements despite the more generous policies adopted in recent years.

Requirements for Graduation

Interviews with FSS staff members revealed a tension many felt between their wanting participants to graduate and wanting to prepare them to be financially self-sufficient. As noted in the FSS Year 3

evaluation report, most FSS programs in the study had added specific employment-related graduation requirements. In 2018, two sites accepted any employment to satisfy graduation requirements. The rest, according to program staff members, set the bar higher. Some sites specified the hours of weekly employment required to graduate (such as 30 or 32 hours per week), and some required 6 or 12 months of continuous employment to graduate. Ten sites specified a rate of pay (for example, wages that are

“sustainable”), and two required people who were employed at enrollment to increase their income. The rationale for adding graduation requirements, beyond HUD’s specifications that a participant had to be employed and receive no public assistance for 12 months at the time of graduation, was that staff members wanted participants to be able to support their families financially, or at least be close to doing so, at the time of graduation.

Since 2018, the majority of sites have eliminated additional employment requirements beyond those set by HUD. 51 Four sites continue to use extra requirements for determining graduation eligibility, which are included in their HUD-approved Action Plans. Even where the additional requirements have been eliminated, FSS staff members encourage participants to reach for that higher bar—it is just not codified in their Individual Training and Services Plans (ITSPs). Among the sites that do not set additional employment requirements across the board, some count any employment as sufficient for meeting graduation requirements; others review each case before allowing the person to graduate. At one such site, staff members required participants to obtain “decent” jobs that they defined as having an opportunity for growth. Staff at some sites expressed an understanding that extra requirements could limit the number of graduations, but they also recognized that if participants are to become financially stable and independent, then they need to earn a certain amount.

All sites continue to allow participants to change their goals within different time periods before the end of the contract. This change is typically allowed until 6 to 12 months before the contract ends, but at two sites, not later than 2.5 to 3 years after starting the program. Many sites required the participant’s field of employment at the time of graduation to match what was written on the ITSP— if it was written on the ITSP. To prevent this stipulation from becoming a barrier to graduating, many sites left the field of employment unspecified on the ITSP. Other staff members indicated that FSS case managers 51 See Code of Federal Regulations, 1996. “Determination of suitable employment. A determination of suitable employment shall be made by the PHA based on the skills, education, and job training of the individual that has been designated the head of the FSS family, and based on the available job opportunities within the jurisdiction served by the PHA.” The FY19 NOFA adds that “PHAs shall not require or define a certain number of hours, tenure of employment or rate of pay as ‘suitable’ for all FSS participants to adhere to.”

21

discussed employment goals and career interests at each regular FSS check-in and indicated to participants the need to change the ITSP as their employment goals and situations changed.

A New FSS Performance Measurement System

One change since the last round of site visits, which could eventually affect FSS program practices, was HUD’s introduction of a new program performance measurement system. In November 2018, HUD released a new method for rating FSS programs, using a “composite score.” 52 HUD expects to use the new system to help identify best practices and the need for technical assistance. Guidance provided by HUD in April 2019 stated, “In the future, HUD will likely consider the FSS performance score of an FSS program in determining FSS funding awards.”

In the 2020 interviews, the evaluation team discussed HUD’s new performance system and asked supervisors whether there were shifts (or an expectation about forthcoming changes) in program policies or case management practices to achieve favorable ratings on HUD’s new composite measure.

Understanding these changes also helps the evaluation explain how, if at all, the federal ranking system changed the program experience for study participants. As of early 2020, FSS programs had not yet made significant changes to their operations as a result of the new system, but there were some adjustments in policies and practices. The biggest change occurred at one site that ended its requirement that participants earn enough to become “zero HAP” (that is, to no longer receive any housing assistance payments) as a condition of graduation. Staff members acknowledged that their new policy aligns with most other FSS programs by allowing someone to graduate who will still be receiving HCV assistance. Other FSS programs also made changes to their graduation policies, and although in some cases the purpose for the change was to align these policies with HUD guidelines, as discussed previously, the changes also served to increase graduation rates. 53 By relaxing the requirement that participants have a full-time job or work a certain number of hours per week, more participants would qualify to graduate. In addition, to boost programs’ participation rates, a few sites planned to increase enrollment—one of the factors in the composite score.

The emphasis on earnings increases in the composite score has led a few programs to shift the focus of their work with participants. Staff members in these sites reported that they would need to place less emphasis on education goals and move toward more of a “work-first” approach, helping participants find a job quickly. Staff at one site said they would shift their focus away from homeownership and instead concentrate on getting participants into jobs and graduating them as quickly as possible. Staff members raised concerns that the changes meant to increase sites’ composite scores could do a disservice to participants in the longer term. If the focus of FSS programs were to shift to getting participants employed as quickly as possible, staff members would not be able to pay as much attention to helping participants find higher paying jobs or get the education and skills needed to obtain better 52 The new FSS Performance Measurement System includes three parts: changes in earnings from the time participants join an FSS program, graduation rates, and participation rates (see HUD, 2018).

53 In some cases, these changes were made when sites learned that such policies were not allowed by HUD, not necessarily as a response to the new performance system.

22

jobs. This shift in focus would also mean that job quality and whether or not jobs come with benefits (e.g., paid sick leave) would not be taken into account.

One site instituted different rules for FSS participants to count income increases between annual recertifications for FSS participants but not for other voucher holders. This policy was an effort to capture earnings increases for FSS participants more quickly. 54

So far, not all housing agencies have made or are planning to make changes in response to the new measures. Many noted they are “already doing what they should be doing” in terms of helping participants find employment and raise their earnings. Staff members noted they already focused on finding jobs for the unemployed, such as increasing contact with case managers for this group and requiring the unemployed to attend a monthly job club. A few said that changes might still be forthcoming.

The new performance system was introduced toward the end of the FSS contract period for the study participants. Efforts to boost enrollment and focus on employment came too late to influence the experience for FSS participants in the period covered in this report. Over the long term, more participants may graduate in the sites that brought their graduation requirements in compliance with program rules.

Case Management in the Final Years of the FSS Contract

For the final round of field research, examining case management practices as participants neared the end of their time in the FSS program was of particular interest. As noted in earlier reports from this evaluation, there are differences across sites in the scope, time horizon, and specificity of goals as they are laid out on the ITSP; the specific strategies staff members use to keep participants engaged and motivated over the 5 years; how frequently case managers meet and talk with participants; and the strength of the connections and relationships with service providers in the community. This section looks at case management practices at the study sites, focusing on Years 4 and 5.

Final Push in Years 4 and 5

The frequency with which staff members were expected to reach out to participants did not change significantly as participants approached Year 5. 55 Most continued to mandate quarterly contact at a minimum, and all except one still required at least one annual in-person meeting. 56

The FSS programs in the evaluation did not hold additional meetings specifically to review goals or to prepare participants for graduation. The annual meetings that occurred about 1 year before the end of 54 In 2018, most sites were counting income increases between recertifications, whereas a handful of sites waited until the next annual recertification. Two sites that waited until the next annual recertification did allow FSS participants to request that it take effect in order to earn escrow.

55 Although contact expectations had not changed, caseload sizes had changed. About one-third of the sites experienced a decline in caseloads, one-third experienced an increase, and the remaining were about the same.

56 As in previous interviews, supervisors and case managers at nearly all sites said line staff contacted unemployed participants more often than employed participants, with some sites requiring monthly contact for unemployed participants. Communication with employed participants is less likely to occur in person (because they do not want participants to miss work to attend an in-person meeting).

23

the contract were often used to remind participants that they were getting close to the end of their contract and to urge them to complete any of their remaining goals.

Also, about 1 year before the end of the contract, a small number of sites sent official notification of the contract end date by mail. These letters put in writing the date by which all goals must be completed for participants to graduate and earn their escrow.

At the regularly scheduled annual or quarterly meetings, case managers continued to review participants’ goals and assess whether they were “on track” to graduate. In the two sites that updated participants’ goals annually, case managers focused on the final set of goals. Throughout the last year, FSS staff members made a final push to keep participants on track, and the process of reviewing goals and adjusting plans intensified. At some sites, this process began in the fourth year. As one staff member said:

[At the] start of Year 4, I remind them, “You’ve got a couple of years left. Here’s what you’ve accomplished.” We want to take full advantage of the last 2 years.

The timing of the final push varied by the housing agencies’ rules for how close to the contract end date changes could be made to the ITSP, whether graduation before Year 5 was encouraged or expected, and individual participants’ progress in completing their goals. At one site, where participants must be employed for 12 months before the end of the contract, case managers said they take stock in the middle of the fourth year to figure out how the remaining goals can be achieved in the next year and a half. If participants at these sites are not employed at the start of the fifth year, it is apparent that they will not meet the graduation requirements; in that case, case managers will discuss with participants their status in terms of graduation and may suggest that they apply for an extension.

One case manager reported:

Fifth year is usually just waiting. As soon as they’ve done their training and start employment in their field of training—I set a clock and tell them they have 1 year, and I’ll graduate them.

Although official contact expectations did not change throughout the program, numerous staff members reported higher engagement toward the end. Staff members from several sites noted that they try to keep in touch more frequently during a participant’s final year in FSS. One case manager requested participants to maintain monthly contact in the last year (an increase from the expected quarterly contact frequency), an expectation specified on the ITSP.

Increased contacts in the final year were not spread uniformly across the caseload. Some staff members reported spending more time with those individuals they thought were likely to graduate. To some extent, according to staff members, this is because the participants call more often, wanting to make sure they are not missing anything or wanting reassurance. The paperwork and documentation requirements also led to additional contact.

Other staff members said the opposite—that they spend less time with the individuals who are likely to graduate because they are more independent. These staff members reported that those not on track took more time as they continued working with them to remove barriers. A staff member who focused more on participants who did not seem to be on track to graduate said:

24

I don’t want them to see it as a failure or another failure. So, I still encourage them and try to keep them focused on what they can do with the time left.

Working with Participants Not Likely to Graduate

If participants were not on track to complete all goals by the end of the contract period, case managers talked to them, reminded them of the terms and conditions of the contract, and explained options. To the extent allowed by housing agency policy, FSS programs continued helping participants meet the graduation requirements by allowing participants to update their goals on the ITSP, usually up until 6

months or 1 year before their expected FSS program graduation date. 57 FSS staff members also discussed the possibility of an extension, and they explained that the individual must have extenuating circumstances and that granting extensions is not guaranteed. The last option is to close the case or ask if the participant wants to leave the program voluntarily.

In discussing reasons that clients failed to make progress or left the FSS program, case managers lamented that when difficulties arose in their lives, participants withdrew instead of asking for help: Instead of calling us for help, they give up on the program until they’re in a better mental state.

To motivate clients, sometimes we have more personal conversations of what’s going on in their lives.

Some FSS programs terminate inactive participants. The make-up of the caseload reaching the final year of the program will vary depending on the extent to which the site terminates for lack of contact or encourages participants to voluntarily withdraw if they are not making progress toward their goals.

Usually, when they get to Year 4, there’s a pretty good chance that they’ll graduate.

Balancing Accountability and Self-Reliance

Staff members’ responses to questions about how communication with participants has changed over time reveal how staff members work to balance participants’ feelings of accountability toward their case managers with behaviors that demonstrate self-reliance. Throughout the 5 years of the program, staff members from sites with lower contact expectations stressed how they allow participants to drive their experiences in the program and that they put the emphasis on self-reliance. Staff members who were expected to make more frequent contact focused on their role in creating accountability for participants. At sites with monthly contact expectations, where case managers and participants develop strong relationships, staff members describe contacts intentionally changing in content or decreasing in frequency to allow and account for participants’ increasing independence. The type of case management fluctuates and flows differently over the course of the program. Participants may start out needing serious intensive case management, and by Year 3, case managers can step back—even if they continue to check in regularly.

57 One FSS program required all participants who did not have a high school diploma to earn a high school General Educational Development (GED) credential in order to graduate. They did not allow participants to drop this goal. In contrast, staff members at another program noted that participants could change their educational goals toward the end because “the main thing is that they are working.”

25

We want to give you the resources for you to go out there and feel you can do it and need us less each time.

The contact with employed participants differed and was more along the lines of signaling staff members’ availability to help if needed. For example, “Hey, how are you doing? Just checking to see if you need anything?”

Graduation

Some participants complete all their goals and graduate early, and others take the entire 5 years or may need an extension. Much of the difference in timing is based on individual participants and how quickly they progress and complete their goals, but some reflected a difference in approach across the FSS

programs. For example, staff members at a few sites said they allowed or even encouraged participants to set new goals and keep earning escrow. The others were more likely to encourage early graduation. In one site that indicated most of its graduates complete the program before 5 years, the supervisor noted: If a client completes all goals, we graduate them. We don’t just encourage them, we graduate them. If they completed their goals, they’re free of cash assistance and they’re employed, then they will be scheduled to graduate at that time.

The process for identifying who is ready for graduation can vary. Sometimes, case managers see that a participant has met all goals and will prepare the participant for graduation. In some cases, the participants know, and they bring it up first. Roughly 2 months before the contract end date, the case manager begins the final paperwork and verification, such as proof of employment (such as paystubs) and documentation that all goals have been achieved. A few supervisors reported spending considerable time reviewing verification and paperwork for graduation.

Only one site set an annual goal for the number of graduates per case manager: each case manager was expected to have at least 10 participants graduate each year. This goal was a minimum requirement, and usually the number was higher (caseloads averaged about 80–90). Staff members at a few sites mentioned holding a graduation ceremony to celebrate the graduates.

We have a huge graduation. It’s very nice. It’s done with our other graduations of the agency. So they’re recognized. Like last year, the mayor of [city] came and did the keynote. We’ve had congressmen—we have a huge graduation. …There’s music, there’s food, it’s catered, it’s dress nice, it’s just a very formal kind of graduation. And for a lot of our families, they’ve never had a graduation of any kind. So they do testimonials, we invite HUD officials to come and other officials, so it’s a big deal, the graduation. Yeah, they know about it. They hear about it. Usually the whole neighborhood knows about it because right after graduation we get a huge jump of people wanting to join the program. So we’ve got a lot of people calling to get on the waitlist.

Although this section has focused on case management practices and policies, the services provided through case management in FSS programs largely depend on referrals. The next section discusses service referrals—a key component of the FSS model.

26

Service Referrals and the Role of the Program Coordinating Committee

In previous rounds of staff interviews, the Program Coordinating Committee (PCC) and provision of services were often discussed. In this latest round of interviews, the team delved more deeply into the PCC, its role in FSS programs, and staff members’ reflections on its value.

Referrals to services in the community are central to the FSS program model. Nearly all study sites have a PCC (a collaborative group of service providers), a requirement of the program. In larger urban areas, some PCCs are countywide, serving multiple housing agencies. Some housing agencies invite all of their primary service referral providers to join the PCC; in others, the PCC members include a subset of referral partners.

HUD encourages sites to think broadly about the PCCs’ roles, including aligning and coordinating services, contributing to the FSS Action Plan and helping guide its implementation, sharing best practices, and joint fundraising or advocacy activities (HUD, 2017a). Among the study sites, PCCs typically met several times a year (these meetings included FSS program staff members) and focused on service coordination by providing updates about community programs and services. Across the board, the primary value attributed to the PCC was that it enabled FSS staff members to stay connected to, and network with, service providers.

As far as using the PCC in other ways, one site said they brought participant issues to the PCC for guidance “maybe once or twice in the last 3 years.” Another housing agency has PCC members evaluate FSS program applicants for its scholarship fund and revolving loan fund opportunities.

Personal connections at the service providers appear to help participants move more quickly through the provider’s intake process and, in some cases, can expedite service delivery. Some FSS staff members mentioned that these types of relationships help them when, for example, the workforce agency has job listings or someone else on the PCC lets them know of a new program or opportunity.

When we need to access a resource, it’s really nice to have that contact person. So if we have somebody at Catholic Charities or we have somebody that we know, they may send us an email and say, “Hey, we have utility assistance now, we have money, the money came in, send your clients now.” Or I have somebody who called and they don’t have food, be able to know who to call. And yeah, they can come and pick up a bag of groceries tomorrow. Or have somebody that I can call and they can say we can squeeze them into this appointment. …And you can streamline some of the processes that clients don’t like to go through when they’re at a new agency. You can prepare them for what to expect.

Most staff members value the PCC and consider it critical to the program. Only one supervisor was not quite as positive, saying the PCC was not necessarily essential for participants to succeed because case managers could refer participants to non-PCC organizations for services.

FSS staff members appear to be satisfied with the organizations represented on the PCCs and the way they interacted with the FSS program, and they did not recommend any significant changes. A few wanted more diversity in the types of partners or ways to get partners more engaged in the collaborative effort.

27

Right now, the way it is, is we have our meetings and different representatives from each agency; they come and usually they just disseminate a lot of information about some of the services that they already have and some upcoming services or workshops or anything like that. I would like to see more interaction as far as how they’re connecting with our participants.

So we’re looking into trying to have … the meetings be geared more towards the [case managers] bringing different situations that the participants have or different participants say and just say, hey, I have this participant who is in need of X, Y, Z. And then just bringing those issues to the PCC meetings to have the members, they—oh well, here is a viable resource for them. And then maybe spend less time just disseminating information.

One site indicated that their PCC included a number of organizations connected to homeownership, but with the increased focus on employment as a result of the new performance metrics, they were looking for more employment and financial literacy services. At most sites, workforce development agencies and other employment services were longtime members of the PCC or were among the agencies receiving the most referrals from FSS programs. Case managers commonly referred participants to other services, including financial coaching/budgeting, credit, preemployment training, education and workforce training, help with utilities, social service assistance, and homebuyer education programs.

Childcare and transportation were the primary gaps in services mentioned by staff members at multiple sites. 58 PCC partners that can help in these areas were considered particularly valuable.

One of our partners is [childcare organization]. So whenever there’s funding available for childcare, then we know if they’re on the list—right now we’re doing January 2020, so they can apply now, and how quickly they’re going through the list or if funding has been stopped. So it’s very beneficial for us. Every agency that we have serves a purpose. Even with transportation.

You know, Catholic Charities has a little bus service, and we’re updated on all of that.

Although the PCCs often facilitated participants’ access to services, the quality of those services and the degree to which they met participants’ needs were largely outside the purview of the FSS program.

If they became aware of any issues with services, FSS staff members said they reached out to the PCC

and the agency to get their perspectives and see if the issues could be addressed. In cases in which the issues continued, one recourse was to stop referring participants to the agency.

Staff Reflections

As part of the final round of interviews with FSS program staff, the evaluation team asked staff to reflect on various aspects of the program, including the features of the model, the types of participants who seem to benefit most and least from the FSS program, and promising practices in their FSS

program. These reflections can be instructive as HUD and other FSS programs consider policies and guidance that might be helpful to program staff members.

58 Homebuyer education classes and dental services were other service gaps mentioned in interviews.

28

Escrow Incentive

Supervisors and case managers reflected on the value of the escrow incentive, how they promote it, and how it could become a more effective incentive. From their perspective, escrow is a primary reason that many voucher holders join the FSS program. Beyond drawing people to the program, FSS staff members considered escrow integral to the program and valued its motivational influence. Although all staff members appreciated the opportunity for participants to build assets, they differed as to the extent of its importance. To some, escrow was credited with increasing employment and goal completion because they believe “escrow is the number-one motivator”; “it’s what they are working for.”

FSS without escrow—it would be just like regular Section 8. Why would you join? Just for the resources and supportive services? There are agencies in the community that are helping people.

To others, escrow was important but not the main motivating factor for participants’ engagement: “If that were the case, we’d have more graduates and fewer terminations.” Some staff members pointed to cases in which participants had a lot of money in escrow yet remained unmotivated and did not progress toward their goals. Finally, some staff members believed that “motivation is more powerful when it comes from something more personal”; that is, individuals’ motivations for action come from the satisfaction of moving forward in terms of employment or other goals rather than just accumulating a pot of money.

Some don’t want to know exactly how much in escrow they have. They’re working for the satisfaction of self-accomplishment. It [escrow] is a big incentive, but they don’t necessarily focus on that.

Beyond sending an annual escrow statement, FSS staff members have considerable latitude in what they say about escrow and how often they talk about it. A number of case managers said they look up an individual’s escrow balance before each check-in meeting. One case manager said that although they may not mention the actual amount, they work the fact that the participant has escrow into the conversation. Others, however, said they only bring up the topic when participants have questions or ask for their balance. Case managers acknowledged that not all participants were equally motivated by escrow, and they adjusted their communication based on how individual participants reacted to the escrow incentive.

I’ve seen where it goes both ways, where people don’t care about the escrow at all, and then I’ve seen where people have stayed in FSS because—strictly because of the escrow… they have pushed harder towards their goals once they see how much they have in escrow, or they’ve kept on to a job because they see—you know, I’ve been able to convince them to keep onto a job because they see how much they have in escrow.

A few staff members mentioned that, although they typically do not use escrow as a motivational tool, escrow played a bigger role toward the end or when a participant was thinking of dropping out.

If I can use escrow as a motivational tool in the final year—I do. I don’t really harp on “you have this much, you need to do it so you can get this” until the last year, and I see they might not make it. If there’s a chance—whatever I can do to motivate them at that point in time.

29

Isolating and assessing the influence of the escrow incentive is difficult. Staff members noticed that its motivational value increased as people started accumulating funds, and their balances grew. The incentive seemed to provide additional encouragement for those already making progress (versus causing participants to find new employment to begin earning escrow).

And when they start to get these statements yearly, it becomes a very like oh, it says I’ve got $2,000.00 … on it. Oh, my gosh. I said yeah, just imagine how much you’re going to get when you keep working and keep going. And I’m like, and it’s not taxed and it’s your money. We don’t count it, and da-da-da-da-da. Really? It’s my money? I can do whatever? Yes, you can.

The fact that the escrow account does not change participants’ current financial realities weakens its influence because it is distant, and one is not guaranteed to receive it.

It’s too far of a goal, takes too long to accomplish, so then they don’t—they say I don’t need it, I can survive, it’s not worth the headache.

The message staff members sent—and, importantly, that participants heard—about whether the escrow funds “belong” to the participant was murky. According to staff members, some participants worried there was a catch, or they did not really believe the government would give them the money.

They have the thought that, “This is not my money anyway, so I’m not gonna even get my hopes up about getting it. This is not my money.” That’s how they feel.

On the other hand, some participants feel the escrow is theirs and become upset that they cannot use it to pay bills.

They’ve (the case managers) gotten calls that say, “I can’t pay my rent this month—can I go into my escrow?” A lot of them (participants) feel that the escrow is their money…and then they feel, “If that’s not what the escrow is for, then the program isn’t doing anything for me.”

One staff member noted that participants have what she called a “day-to-day mentality,” meaning that they were only thinking about what was happening at that moment. For many participants, the reality was that when they earned more, the extra money was taken away in lost benefits and increased rent, without giving them a chance to catch up on bills. One of the questions about the value of escrow as an incentive was whether participants understood the rules around escrow. Despite staff members’

repeated explanations and reminders, there may have been a disconnect in terms of understanding that they would only get the escrow funds upon graduation from the program.

No matter how many times we bring it up and send them letters, I think there are some who don’t understand.

There are strategies for making the escrow incentive feel more “real” to participants. One such strategy was to provide interim disbursements (discussed below), and a second was to invite current FSS

participants to the graduation ceremony to watch people getting their checks.

When pressed, the majority of interviewees said it would be better to run a service-oriented FSS

program than an escrow-oriented program. FSS staff members believe that although escrow may be a 30

valuable motivator, achieving educational, employment, and financial services goals does more to help individuals gain confidence and direction and move them toward self-sufficiency over the long term.

Interim Disbursements

As noted previously, FSS programs may allow participants to receive a portion of their escrow funds before graduation to help the family accomplish its goals. Based on data analyzed for this evaluation, interim disbursements are rare; in 2018, less than 4 percent of study participants received one.

Interviews in 2020 explored how staff members viewed and used interim disbursements and aimed to uncover why this aspect of FSS programs was not used more fully.

In theory, FSS program staff members believed that granting interim disbursements would help keep participants engaged by removing barriers to meeting goals. Staff members from sites that did provide interim disbursements said they saw a direct relationship between participants receiving one and successfully graduating, especially when the disbursement was used to gain or maintain employment—

such as by purchasing or repairing a car that enabled a participant to get to work. One site recently relaxed its disbursement policy by allowing participants to use escrow disbursements for car repair upon receiving guidance from HUD that this was an allowable use.

Beyond bringing participants closer to achieving a goal, disbursements make the escrow benefit seem

“real” to the participant. According to one staff member, it’s “as if a light bulb goes off and they can see that that escrow is there to be utilized as a means to help move them forward.”

I think it’s really effective. Especially when they’re able to request funds and be able to utilize those funds, what we’ve disbursed to them, to apply it towards something in reality. And they see like okay, I do this, then this happens, kind of like that whole rewards system. And that gets very effective. Because otherwise, where else would they get this money from?

Not all staff members, though, held that interim disbursements led to successful outcomes. One case manager shared the experience of a participant who had the opposite reaction to receiving an interim disbursement; that participant’s attitude was:

Let me just get 50 percent of what I have in my escrow account. Then I’m just gonna not show up or not do anything.

Despite positive views about the benefits of interim disbursements, the final round of interviews confirmed the earlier finding that few participants received one. At many sites, staff members said there were fewer than five per year; they were more frequent (estimated at around 10–20 per year) in a few other, not necessarily larger, sites. A variety of factors contributed to this pattern. 59 One was that this program feature was not routinely publicized or talked about during check-in meetings. Case managers expected participants to remember from their FSS orientation years earlier and request a disbursement if they needed one, or case managers brought it up when participants spoke about specific issues (such as car trouble making it difficult to get to work). Another factor was a preference for participants to hold onto their escrow and accrue as much as possible. Before allowing a disbursement, staff members worked 59 We do not have information about the number of disbursements that were requested and/or denied.

31

with participants to find alternative sources of funds to meet their needs, using escrow only as a last resort. A few sites used their own scholarships or stipends before turning to escrow.

So, they (case managers) do kind of walk through: is this really a need, is this something that you could take care of yourself without doing your withdrawal to ensure that if you actually did need a disbursement for some higher level reason, you still had access to it for that year?

Although some staff members did encourage the use of interim escrow disbursements so that participants could more easily meet their goals and graduate, other staff members expressed that using the escrow in this way did not encourage self-sufficiency because participants will not always have a pot of funds (like the escrow) to dip into for emergencies. Those staff members felt that FSS

participants needed to learn how to save funds independently and not rely on interim escrow disbursements.

Who Benefits from FSS?

Staff members were asked about the types of participants that benefited from the FSS program. FSS

programs do not have a set schedule or prescribed path, so participants have to work independently and stay committed to their goals. Staff members believe that those who put forth more effort, who really want to move toward self-sufficiency (ascribed to a participant who makes progress), benefit the most from FSS programs. A few staff members found that those who were already on the path toward self-sufficiency when they joined the program (for example, they were employed at the time they started the FSS program) or had already taken steps (such as enrolled in education or training) were the most likely to succeed.

Other staff members remarked that those who needed and wanted education, who needed more coaching, and who would not have made progress on their own, benefited the most from FSS programs.

Some staff members focused on escrow, reporting that those with no earned income or lower starting incomes have the most to gain through the escrow component of the program.

Staff members also reflected on which participants did not benefit from FSS programs. Their responses mainly touched on those for whom employment was either not possible or hard to achieve, such as the disabled and those with a criminal background. Lack of support and encouragement from one’s family was another factor cited:

If you have everybody who’s constantly against you and you don’t have any support, it doesn’t, you know, referring all you want (won’t help). Unless you’re someone who’s motivated and able to overcome these obstacles, it’s gonna be hard.

Promising Practices

Staff members reflected on the program practices that they thought worked best to keep participants engaged and making progress toward their goals, offering some recommendations to strengthen the FSS program. These reflections and recommendations span the FSS programs’ core components and reflect a range of experiences and observations across FSS study sites.

32

Frequent and in-person contact. Staff members reflected on the importance of more frequent contact.

One case manager noted, “The basic fundamental thing we do is communication” as a reason for believing that their monthly contact is the practice they think works best and could be replicated elsewhere. Staff members at another site referred to frequent face-to-face contact as a means to ensure open lines of communication, and another mentioned conducting home visits. To replicate this, programs could require more of a hands-on approach, with set expectations for frequent contact.

Set short-term goals. Staff members at several sites made a direct connection between how they set and monitor goals and participants’ confidence and progress. One strategy involved setting shorter-term goals and having participants set new goals every year. These staff members did not allow goals to keep rolling over from one year to another without seeing progress. Fragmenting long-term goals into smaller steps on the ITSP was another strategy. Case managers said not to assume that “someone knows how to fill in the blanks,” so being specific with how to accomplish each step was essential.

HUD could suggest that FSS programs guide participants to set short-term goals—perhaps a set of goals for each year.

Build rapport between staff members and participants. Several supervisors mentioned the commitment of their staff members, how well they worked as a team, and their ability to develop rapport with participants as their program’s best feature. The importance of developing relationships so that participants open up and tell case managers about things going on in their lives was mentioned by staff members at several sites.

I’ve also learned not to give up really quick because you might just be dealing with them at a bad time. It might be a bad month. It might be a bad quarter. It might even be a bad year. So, I’ve learned that although they have to do the requirements, you still have to be patient and try to figure out a way to…you have to try to see it through their eyes as best you can and take yourself to where they are and try to work from there and come up with resources, come up with ideas. Just be encouraging. And then if it doesn’t work, it doesn’t work. But at least you’ve done all you could do to reach them.

Dedicated staffing. Staff members at several sites made a pitch for separating FSS case management functions from HCV responsibilities. Another staffing-related recommendation was that case managers should have a social service background. Additional funding for salaries for FSS staff supervisors and for professional development for case managers60 was also proposed.

Develop valuable service referrals. Several staff members highlighted strong connections with community resources and the knowledge gained through their PCCs about available services and programs as programmatic strengths. A few sites mentioned the benefit of requiring participants to attend classes and workshops, and one site that runs a multisession training series to educate participants about financial literacy topics views that as an important piece of their program. Staff members flagged the need for funding to offer onsite training to increase access to services.

60 Specific areas of training included how to help participants develop a career path, keeping them motivated, trauma and loss/grief counseling, starting a small business, and how to refer participants to mental health services or therapy without offending them.

33

Modify escrow policies. Staff at two sites mentioned interim escrow disbursements as an aspect of the program that worked well. Both of these sites had recently made changes, although the changes were very different. One site became more lenient in allowing disbursements to help participants get ahead.

The other set new rules restricting the amount that could be disbursed, which they believed would help participants to “be good stewards of their money, teaching them how to save and making sure they have an asset at the end.” Several staff members recommended that HUD change the escrow calculation by removing the income limit so those with higher incomes can still earn escrow. One staff member suggested creating a financial incentive to motivate participants to get a high school equivalency diploma or training that would open doors to better employment. This could be a separate incentive or a method for earning an interim escrow disbursement.

Other recommendations addressed the performance measures and new policies staff wanted to see.

Ideas included the following:

Use more current data to create performance measures. The performance measures were new, and supervisors were still trying to understand how their programs would be affected and whether their work helping FSS participants to gain employment and get better jobs would be accurately reflected in the measure. Some of the concerns were about timing; staff members wanted new jobs that participants obtained after the last recertification to be included. Others felt that the expectation that FSS participants will be able to achieve significant earnings increases is not realistic for many of the participants they are currently serving –such as FSS participants who receive disability benefits or experience a severe barrier to employment or, at the other extreme, FSS participants who were already working full-time hours when they entered the program.

Allow successful families to continue in FSS programs. Staff members reported having participants who had just gotten started on some of their nonemployment-related goals, and then a new job caused them to earn out of the program before they could make progress. Homebuyers, too, could benefit from FSS in important ways, such as by accruing savings that they could use when they graduate (creating a nest egg in case they need to make repairs to their homes). HUD could consider eliminating policies that require participants to leave the FSS program, such as when the total tenant payment (TTP) is higher than the fair market rent or when a participant purchases a home. 61

Remove common stumbling blocks. Extra financial help to remove barriers and support participants was viewed as going a long way toward increasing success in FSS programs. It is easy for an emergency, unexpected bill, or other obstacle to create undue stress and financial strain on participants, which could cause them to disengage from the program. Staff members suggested that HUD consider proposing a general change to subsidized housing regulations to allow programs to delay rent hikes for a couple of months after an earnings increase to give participants a chance to catch up on their bills.

Staff members also encouraged the creation of a mechanism for childcare agencies to give FSS families preference; staff also advocated for a change to FSS policies to allow people who become disabled 61 The Final FSS Rule eliminates these policies.

34

during their tenure in FSS and cannot meet the work requirement to graduate and receive the funds in their escrow accounts. 62

Making the FSS Program Mandatory or Keeping it Voluntary

Staff members at several sites recommended making the FSS program mandatory for nonelderly, nondisabled HCV residents. In particular, many would support making the FSS program mandatory if additional resources were available (because it would be expensive to add the staff that a mandatory program would require). A few staff members disliked the idea because they felt that forcing participants to join an FSS program would not yield the same results because not all residents are ready to become self-sufficient, and case managers would not want participants to feel forced to meet with them. One case manager referred back to her days working in a welfare-to-work program and noted that the people who did not want to participate overwhelmed her ability to work with motivated ones.

Program Length

Staff members also reflected on the 5-year term of the FSS program. Supervisors and case managers from most sites were happy with the current length of the program. A longer program allows more people to graduate and to earn more escrow. If the allotted time were shortened, goals would have to be “smaller” (for example, individuals would no longer be able to pursue a bachelor’s degree), and participants would not make as much progress toward self-sufficiency. Further, with a shorter program, participants who wanted to try something different, who lose their jobs, or who encounter other unexpected obstacles with their children or their physical or mental health would struggle even more to succeed. Many participants experience traumatic events that can take time to heal.

Other staff members supported making FSS a 3-year program (and with the possibility of a 2-year extension to 5 years). A shorter length of time would motivate participants to sustain higher engagement throughout the program and allow more people to benefit over time.

Conclusions

This chapter helped explain FSS program implementation and case management in Years 4 and 5 and provided context for interpreting graduation outcomes and program impacts described in the remainder of the report. Staff reflections also provided important insights for program improvements. Of note, the chapter described how sites have come into compliance with program rules on graduation requirements in response to an increased awareness of HUD guidelines about not setting additional blanket employment requirements, as well as the new composite score measure. Contact expectations remained constant over time and officially continue to the end of the FSS contract. In practice, case managers use the final year or two as a final push to urge participants to complete all goals so they are eligible for graduation, and many staff members try to increase contact toward the end of the program.

The PCC benefited FSS staff members by providing opportunities to network with and develop personal connections with community service providers. Escrow is valued by staff as a way to motivate 62 Another change reflected in the Final FSS Rule.

35

participants to stay engaged in the program and increase employment. Obstacles to completing goals can, in some cases, be remedied through interim escrow disbursements, but they were rarely used by study sites.

The following chapters explore longer-term outcomes and effects of the FSS program. The next chapter (chapter 3) explores findings on graduations and escrow accruals and disbursements, drawing on administrative data.

36

Chapter 3. FSS Enrollment Status, Escrow Accruals, and

Graduations

As discussed previously, the escrow account, a core feature of the Family Self-Sufficiency (FSS) program, is intended to encourage families to increase their earnings, reduce their reliance on government cash assistance programs, and build savings. Under the Housing Choice Voucher (HCV) program, most families pay 30 percent of their adjusted monthly income, known as the total tenant payment (TTP), for their rent and utility expenses, and the public housing agencies (PHAs) subsidize the rest. As a result, in most instances when a household’s income increases, its TTP also increases. In the FSS program, when the tenant pays the increased TTP, the housing agency credits the family’s escrow account on the basis of the increase in earned income (HUD, 2017a). 63 Upon graduation from the FSS program, the escrow balance account, with accrued interest, is disbursed to the FSS participant (the head of household) with no restrictions on the use of the money. Under certain circumstances, FSS

allows participants to access their escrow funds earlier than graduation—that is, receive an interim disbursement—for approved purposes related to their self-sufficiency goals, such as paying for emergency car repairs to prevent job loss.

The first report produced as part of this evaluation, which focused on the first 18 months of followup, showed that 38 percent of the FSS group had received an escrow credit during that followup period, and 35 percent of the FSS group maintained a positive balance in Month 18. By the end of 3 years of followup, the period covered by the second report, 52 percent of the FSS group had received at least one escrow credit, and 40 percent of FSS group members had maintained a positive balance. Consistent with the program’s design, less than 5 percent of FSS group members had fulfilled all of their program goals within 3 years of enrolling in the program. Most needed to continue accessing FSS services for at least another 2 years before qualifying for graduation and disbursement of the money in their escrow account.

This chapter extends that analysis and examines escrow credits and balances during the first 5 years of followup and, where data are available, into the first months of Year 6. The followup period covers the end of the 5-year enrollment period, as specified in the FSS Contract of Participation (COP), and for many FSS group members includes a few extra months. During these additional months in Year 6, PHAs often verified that participants had met all of their goals, completed the paperwork necessary for a disbursement check, and arranged for a graduation ceremony for the participant. The analysis first presents findings on graduations and escrow accruals and disbursements for the whole FSS group, and then it explores how these outcomes varied for selected subgroups. Next, the chapter considers whether the patterns of graduation and escrow accrual and disbursement varied by housing agency and, if so, whether these differences appear to be related to variations in how particular housing agencies implement the FSS program.

The findings show the following:

63 The amount of escrow credited to the account depends on the household’s income level; those with the lowest incomes are credited an amount equal to the rental increase, and those with higher incomes are credited with a percentage of the increase.

37

• Over the 5- to 6-year followup period, about 17 percent of the FSS group graduated from the program. Most FSS graduates received a substantial final disbursement from their FSS escrow account. Disbursements averaged nearly $10,000 per recipient.

• Over the same followup period, another 45 percent of FSS group members received at least one escrow credit, although about one-half of this group forfeited their escrow when they exited the FSS program without graduating. By the end of the followup period, nearly one-fourth of the FSS group had continued their enrollment and maintained a positive balance in their escrow accounts. They averaged about $7,000 in their accounts during the last month of followup.

• On average, at their time of random assignment, FSS graduates were younger, more likely to have attained a postsecondary degree, more likely to have been employed, and less likely to have reported a physical or mental health problem that made it difficult to find or maintain employment compared with other groups.

• FSS graduates experienced a much larger increase over time in estimated annual earnings—as reported to their housing agencies—compared with other FSS group members. They also were most likely to be living with one or more household members who were working for pay at the end of the followup period.

• Housing agencies differed substantially—from about 8 to 31 percent—in the proportion of FSS

group members who had graduated by the end of followup. Housing agencies with the strongest emphasis on monitoring and engagement rank highest in graduation rates, whereas housing agencies with less emphasis on monitoring and engagement had a higher incidence of current enrollment with a positive escrow balance.

How Family Self-Sufficiency Escrow Works

Escrow calculations can be complicated (HUD, 2016). When the head of household enrolls in the FSS

program and completes a COP, the FSS case manager records the participant’s “baseline” earnings from the most recent housing voucher recertification meeting. The earnings noted on the COP serve as the standard for calculating escrow credits in future months. Unlike other rent policies designed to encourage work, throughout the FSS contract period, FSS participants continue to pay their TTP for rent and utilities according to the same rules as other housing voucher holders. The housing agency maintains a single, interest-bearing depository account and records the balance for each individual in a separate ledger. When the head of household reports an increase in earned income to the housing agency (and the increase is verified), the FSS participant’s out-of-pocket payment for rent and utilities increases, but under FSS program rules, the housing agency issues an escrow credit for the amount of the increase attributable to an increase in earned income and deposits the money in the FSS

participant’s escrow account.

The likelihood of accruing escrow can vary due to a number of factors. FSS participants’ employment status at program enrollment, level of educational attainment, and other characteristics and life experiences can often affect their chances of increasing earnings over time, which, in turn, affects whether—and how quickly—their escrow balances increase. In particular, the relationship between a participant’s relative advantages in the labor market and escrow accrual may be complex. At the extremes, FSS participants who are not working at the time of program entry could potentially benefit 38

the most from the escrow account because all their future earnings would be included in the calculation of escrow credits. Unemployed adults, however, may also face the most severe barriers to finding and maintaining employment, which is required to graduate as well as to accrue escrow. By contrast, FSS

participants who enter the program while working full time or receiving relatively high earnings may have the best prospects of increasing their earnings by finding a better job, increasing their hours, or advancing with their current employer (Verma et al., 2017). 64 They may also be most likely to maintain their employment after they start accruing credits. The increase in their earnings, however, may be relatively small compared with their current earnings and lead to only a small amount being credited to their escrow account each month.

Housing agency practices and messages can also affect the incidence of escrow credits and the rate of increase of escrow account balances. HUD FSS program rules require housing agencies to issue an annual escrow account statement to program participants. In addition, at most sites, the possibility of accumulating escrow is used to motivate participants throughout the contract period to stay engaged in the program and meet their goals to graduate and thus receive the amount accrued in their escrow accounts. Staff members reported that interest in accruing escrow credits also gets stronger among participants with a positive balance as they get closer to graduation and the funds seem more attainable.

FSS Enrollment Status, Graduations, and Escrow Accrual and

Disbursement

Following HUD guidance, housing agencies compare current earnings (projected into an annual total) with baseline earnings recorded in the COP to determine the amount of escrow to credit to a participant’s account (HUD, 2017b: 92–93). 65 For the study sample, exhibits 3.1, 3.2, 3.3, and appendix exhibit B.1 summarize the changes in FSS group members’ enrollment status and escrow accrual over 5 to 6 years. By the end of the followup period, about 7 out of 10 FSS group members had exited the FSS program, with 17 percent graduating and 53 percent exiting without graduation.

In the previous report, covering the first 3 years of followup, relatively few FSS group members (4

percent) graduated from the program. Typically, these FSS graduates had found relatively high-paying jobs or increased their earnings above the maximum allowed for receiving an HCV subsidy, and some had accrued no escrow credits and graduated with no disbursement. By contrast, at the end of the followup period, most graduations followed the expected pattern, with FSS group members receiving official acknowledgment that they had attained their FSS goals (including employment and no cash assistance receipt) and disbursement of the funds accrued in their escrow accounts. Most FSS graduates 64 In the Work Rewards FSS study, those working at study entry were more likely to meet the graduation requirements than those who were not working. About 31 percent of the FSS-only households in the subgroup who were not working at the time of random assignment graduated from FSS programs, whereas 55 percent of the working subgroup graduated. The working subgroup also collected more escrow savings on average than did the nonworking subgroup.

Those in the FSS-only group (who received only the FSS program and made up one of three different programs groups in the study) who were working at random assignment received an average disbursement of more than $2,000; those who were not working at random assignment received an average disbursement of $1,000.

65 In reality, and as HUD’s FSS resource guide describes, the housing agencies are required to deposit all escrow funds for all FSS participants into a single, interest-bearing depository account and to account for these funds through a subsidiary ledger that records the balance of each FSS participant’s individual account (within the single account).

39

realized a substantial disbursement, averaging close to $10,000 per recipient. Nearly two-thirds of recipients of an escrow disbursement obtained $5,000 or more at graduation.

Exhibit 3.1: Family-Self Sufficiency Enrollment Status at the End of Year 5 and in the Last Month of Followup

Outcome

End of Year 5

Last Month

of Followup

FSS enrollment status (%)

Graduated

10.6

16.9

Received escrow disbursement

8.4

14.2

No escrow disbursement

2.2

2.7

Still enrolled

44.1

29.9

With escrow balancea

29.8

23.4

Without escrow balance

14.2

6.5

Exited, did not graduate

45.4

53.2

Accrued escrow credits

18.8

22.0

Did not accrue escrow credits

26.6

31.3

Among graduates who received escrow

disbursement:

Average total disbursement ($)

7,947

9,651

Total disbursement (%)

$1–$1,000

10.2

7.7

$1,001–$2,000

8.3

7.7

$2,001–$5,000

22.2

20.3

$5,001–$10,000

30.6

29.1

$10,001–$20,000

23.2

22.5

$20,001 or more

5.6

12.6

Among participants enrolled with an escrow

balance:

Average escrow balance ($)

6,339

7,220

Total balance (%)

$1–$1,000

17.5

11.8

$1,001–$2,000

15.7

18.9

$2,001–$5,000

20.5

20.1

$5,001–$10,000

25.5

24.9

$10,001–$20,000

16.3

16.0

$20,001 or more

4.5

8.3

Sample size

1,285

1,285

a“Still enrolled with escrow balance” includes anyone who had a positive escrow balance during the last 6 months of followup (July 2019 to December 2019).

Sources: PHA administrative data and U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data

40

Exhibit 3.2: Family Self-Sufficiency Graduation and Enrollment Status in Months 12 and 36 and in the Last Month of Followup

100

0.3

4.0

16.9

90

23.7

80

39.8

23.4

70

60

)

6.5

%

tage ( 50

62.7

22.9

Percen 40

30

53.2

20

33.2

10

13.3

0

Month 12

Month 36

Last Month of Followup

Month of Followup

Exited

Enrolled with no escrow balance

Enrolled with escrow balance

Graduated

Notes: The FSS service-use sample includes all or a randomly selected subsample of housing choice voucher heads of household who were randomly assigned to the FSS group between October 18, 2013, and December 22, 2014, in 12 public housing agencies and were age 18 to 61 at the time of random assignment. FSS impact sample includes housing choice voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Rounding may cause slight discrepancies in calculating sums.

Sources: MDRC calculations using housing authority administrative data and U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data 41

Most FSS group members who remained enrolled in the program had accrued escrow credits for 1

month or more and were maintaining a positive balance in their escrow account at the end of followup.

In exhibit 3.3, the line nearer the x-axis displays the year of followup when each FSS group member began accruing escrow credits, whereas the top line displays the cumulative percentage of FSS group members who ever accrued at least $1 in escrow credits. As the exhibit shows, nearly 60 percent of FSS group members (graduates and nongraduates combined) accrued escrow for 1 month or more during the followup period. Most FSS group members (44 percent out of 59 percent) began accruing escrow during Year 1 or Year 2 after random assignment, and relatively few thereafter (15 percent out of 59 percent). Many current enrollees could also look forward to receiving a substantial disbursement if they remained in the program and attained their goals. As appendix exhibit B.1 shows, the typical FSS group member with a positive balance in their escrow account had already accrued more than $7,000 in credits and could conceivably accrue additional escrow dollars before graduating.

Exhibit 3.3: First Month of Escrow Accrual and Cumulative Percentage of Family Self-Sufficiency Group Members Who Ever Accrued Escrow Credits, by Month of Followup Notes: The FSS service-use sample includes all or a randomly selected subsample of housing choice voucher heads of household who were randomly assigned to the FSS group between October 18, 2013, and December 22, 2014, in 12 public housing agencies and were age 18 to 61 at the time of random assignment. The FSS impact sample includes housing choice voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Rounding may cause slight discrepancies in calculating sums.

Sources: MDRC calculations using housing authority administrative data and U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data

Variation in Graduation and Escrow Accrual for Selected Subgroups

This section explores whether most FSS group members experienced similar patterns of escrow accrual and disbursement during the followup period or whether the incidence of escrow accrual or the amount accrued varied by subgroup. Evidence from the recently completed Work Rewards evaluation in New 42

York City, the only other study to have followed study participants for 6 years, found that FSS

participants who were working at the time of random assignment were more likely to regularly earn escrow credits and successfully graduate from the program than participants who were not working.

The households in the nonworking subgroup appeared to have accrued escrow credits at higher rates earlier in the program but were less likely to graduate from the FSS program than those in the working subgroup (Verma et al., 2017). The national FSS evaluation also examines escrow accruals and disbursements for the same subgroups.

Exhibit 3.4 displays important indicators of escrow accrual for selected subgroups, using data on baseline earnings collected from COP forms, housing agency administrative records, and responses to the Baseline Information Form (BIF). For these indicators, graduation, and especially graduation with an escrow disbursement exceeding $5,000, may be seen as the most favorable outcome, and having a positive current balance as a potentially positive outcome, depending on whether the FSS group member completes the program. In an important way, the findings resemble those from the Work Rewards evaluation in that they highlight challenges to graduation experienced by members of subgroups with greater barriers to employment. More specifically, as exhibit 3.4 shows, FSS group members who reported on the BIF that they were not working at random assignment or reported to their PHA around their time of random assignment that they had no household earnings recorded the lowest incidence, so far, of graduating from FSS programs among related subgroups.

In previous reports, the subgroup analysis compared the incidence of maintaining a positive balance, and subgroups with greater advantages in the labor market did not record the most positive results. For this analysis, which focuses more on variation in graduation rates, the incidence of graduation among FSS group members with the highest reported earnings at random assignment and FSS group members who reported working full-time hours at random assignment did not differ much from the rates for subgroups with “middle range” earnings or with part-time employment. However, FSS group members with more than $20,000 in reported earnings at random assignment have, on average, accrued relatively small amounts of escrow and received relatively small disbursements compared with other subgroups.

In addition, as exhibit 3.4 shows, the graduation rate for FSS group members with a 2-year college degree or higher (29 percent) greatly exceeds the rate for the other educational subgroups, as does their incidence of graduating and receiving more than $5,000 in their escrow disbursement (17 percent). By contrast—by a small margin—the subgroup with no degree or educational credential had the highest rate of continuing enrollment with a positive balance.

Variation in Graduation Rates by Housing Agency

Exhibits 3.4 (bottom panel) and 3.5 summarize the variation in measures of escrow accrual and disbursement among the 18 housing agencies. The differences among housing agencies were substantial. For example, for the three housing agencies with the highest graduation rates by the end of followup, an average of 31 percent of FSS group members had graduated, whereas the three housing agencies with the lowest incidence averaged only 8 percent (see exhibit 3.5). Similarly, about one in five FSS group members received a disbursement greater than $5,000 in the three housing agencies ranked first to third, compared with 4 percent for the three lowest ranking housing agencies.

43

Exhibit 3.4: Indicators of Family Self-Sufficiency Escrow Account Accrual for Selected Subgroups

Currently Enrolled

with Escrow

Graduated with Escrow

Sample

Balance Greater

Graduated

Disbursement Greater

Subgroup and Outcome

Size

Than $0 (%) from FSS (%)

than $5,000 (%)

Annual household earnings at FSS

enrollment

No earnings

552

25.5

15.9

11.6

$1–$10,000

165

26.7

23.0

11.5

$10,001–$20,000

208

26.9

19.2

10.6

$20,001 or more

280

19.3

17.9

3.9

Employment status at

random assignment

Not employed

568

22.5

12.3

9.0

Employed 1–34 hours per

week

336

25.6

21.4

10.7

Employed 35 hours per week

or more

372

22.0

19.4

7.8

Highest level of educational

attainment at random

assignment

No degree or credential

259

27.4

8.5

3.5

High school diploma or

equivalency certificate

320

23.1

15.3

7.5

Some college

470

22.6

16.4

9.4

2-year college degree or higher

236

20.8

29.2

17.0

PHA emphasis on monitoring and

engagement

Low

472

34.3

10.2

5.9

Medium

551

18.9

20.9

9.3

High

262

13.0

20.6

14.5

Notes: The FSS impact sample includes housing choice voucher heads of household who were randomly assigned between October 18, 2013, and December 22, 2014, and were age 18 to 61 at the time of random assignment. Sample sizes for related subgroups may not sum to total impact sample because of missing values. The FSS monitoring and engagement composite score incorporates three components: average caseload size, expected number of contacts per year, and the proportion of FSS group members with a Year 1 goal. A Z-score for each component was calculated using the site value and the mean of all 18 sites. The Z-scores were summed to create the composite value.

Sources: MDRC calculations from baseline data, Contract of Participation forms, and housing authority administrative data

44

Exhibit 3.5: Indicators of Family Self-Sufficiency Graduations and Escrow Credits Accrual in Months 1 to 72 by Housing Agency, Family Self-Sufficiency Service-Use Sample

Graduated

Escrow

Escrow

with Graduated or

Balance

Balance

Graduated Disbursement

Currently

Greater

Greater than

from FSS

Greater than Enrolled with

Outcome

than $0 (%)

$5,000 (%)

(%)

$5,000 (%)

Balance (%)

Average value for the three highest ranking housing

authorities

37.5

13.5

30.8

20.3

50.0

Median value for all housing authorities

19.4

4.1

23.5

9.3

42.1

Average value for the three lowest ranking housing authorities

4.2

0.0

7.6

3.7

30.5

Correlation coefficients (housing authority averages)

Average FSS caseload sizea

-0.59

-0.19

0.66

0.61

-0.12

Number of expected contacts

-0.47

-0.14

0.38

0.39

-0.27

Proportion of FSS group with a Year 1 goal

-0.45

-0.11

0.19

0.35

-0.46

Proportion of FSS group with a job search or postemployment

goal

0.21

0.01

-0.28

0.04

-0.02

Proportion of FSS group with an education or training goal

0.16

0.08

-0.04

0.19

0.21

Proportion of FSS group with a financial security goal

0.22

-0.10

-0.01

0.05

0.33

Emphasis on monitoring and engagement (composite score)b

-0.61

-0.18

0.49

0.54

-0.34

Total number of housing authorities = 18.

Notes: The FSS service-use sample includes all or a randomly selected subsample of housing choice voucher heads of household who were randomly assigned to the FSS group between October 18, 2013, and December 22, 2014, and were age 18 to 61 at the time of random assignment.

aAverage caseload sizes were multiplied by -1 to test whether small caseload sizes are positively correlated with greater incidence of escrow accrual and higher positive balances.

bThe FSS monitoring and engagement composite score incorporates three components: average caseload size, expected number of contacts per year, and proportion of FSS group members with a Year 1 goal. A Z-score for each component was calculated using the site value and the mean of all 18 sites. The Z-scores were summed to create the composite value.

Sources: MDRC calculations from housing authority administrative data and information provided by FSS administrators and case managers 45

Measures displayed in the bottom panel of exhibit 3.5 present results of simple tests of association (correlation coefficients) among the indicators of housing agency implementation features (see appendix exhibits A.1 and A.2 for details) and housing agency averages for escrow credit accrual.

Correlation coefficients with values close to +1.000 suggest a strong positive association between the program implementation feature and the accumulation of escrow credits. For example, in exhibit 3.5, the coefficient at the bottom of the middle column, “Graduated from FSS,” shows the result of testing whether FSS group members in housing authorities that strongly emphasized monitoring and engagement also tended to have relatively high graduation rates among the 18 PHAs. Alternatively, values close to -1.000 suggest that housing agency program implementation features that were intended to increase service use were associated with low rates of escrow credit accrual. Finally, coefficients of between -0.399 and +0.399 show little or no association between the implementation feature and escrow accrual outcome.

As exhibit 3.5 shows, the variation in how PHAs have implemented the FSS program appears to be at least moderately correlated with the variation in how far FSS group members had progressed toward graduation. The strongest correlations concern caseload size. PHAs with smaller caseloads tended to have higher graduation rates, whereas PHAs with larger caseloads tended to have a higher incidence of a less positive milestone: continued enrollment in an FSS program with a positive escrow balance.

Exhibits 3.4 and 3.5 show a similar finding for the more general measure of emphasis on monitoring and engagement. PHAs with the strongest emphasis on monitoring and engagement tended to have the highest graduation rates and the highest incidence of graduating with a disbursement that exceeded $5,000, whereas PHAs with the weakest emphasis on monitoring and engagement tended to have the highest rates of ongoing enrollment in FSS with a positive escrow balance. It should be noted that these findings could change over time, depending on how many still-enrolled FSS group members eventually achieve their goals and graduate from the program.

Who Appears to Be Benefiting the Most from Enrollment in Family

Self-Sufficiency?

This section extends the analysis of subgroup variation in escrow accrual and explores the characteristics and experiences of participants who appear most likely to benefit from their access to FSS services and the escrow incentive. 66 For this analysis, FSS group members are divided into five groups on the basis of their enrollment status within the HCV and FSS programs during the last month of followup and the balance in their escrow accounts, if any, at that time. FSS group members who graduated from the program (irrespective of disbursement amount) are considered to have benefited the most from their engagement in the program.

66 A later report will include findings of a more comprehensive analysis that will include additional followup on earning and disbursement of escrow credits and will use formal predictive analytics methodology.

46

Other groups that could also benefit include FSS group members who maintained a positive balance in the last month of followup. 67 The remaining three groups—FSS group members who exited from HCV and FSS, FSS group members who left the FSS program but continued to receive HCV subsidies, and FSS group members who remained enrolled in FSS but have a $0 balance in their escrow account—

are considered to be the least likely to benefit from their enrollment in the program. 68

Exhibit 3.6 displays selected baseline characteristics of members of the five groups. In several respects, the biggest contrast shown is between FSS graduates and FSS group members who left the program without graduating and continued to receive a housing choice voucher. FSS group members now enrolled in HCV only appear to have entered FSS programs at random assignment, with the most serious barriers to employment of any group. On average, they had the highest proportion of FSS group members with no educational credentials and the lowest proportion who entered the study with a 2-year college degree or higher. By a wide margin, they reported having the lowest employment rate at random assignment of any group and the highest incidence of having a physical or mental health problem that made it difficult to find and keep a job. (They also had the highest incidence of receiving SSI or SSDI disability benefits.) On average, the HCV-only group was also the oldest and the poorest of the five groups. By contrast, FSS graduates were the most likely of all groups to include recipients of a 2-year college degree or higher and reported the highest employment rate of any group at random assignment. (It should be noted that greater incidence of part-time employment accounts for most of the overall difference in employment rates for FSS graduates compared with all other groups.) FSS

graduates were also younger, on average, than the other groups and most likely to have children in their household. Possibly also contributing to their success in the program, FSS graduates were most likely to report at random assignment that they had heard of the FSS program’s escrow incentive.

67 At least some HCV households leave assistance because their household income increases above the maximum level for receipt of a housing subsidy. This group also includes most families that graduate from the FSS program before 6 years (see Smith et al., 2015). Using survey and administrative data, the authors compared the characteristics and experiences of participants in the Moving to Opportunity (MTO) program evaluation who left housing assistance with those of participants who continued to receive assistance. The authors of the study of MTO participants found that about 35 percent of study participants left housing assistance during the followup period. Among “leavers,” about 52 percent left for positive reasons, such as increased earnings or homeownership, whereas 48 percent left for negative reasons, such as eviction or violation of program rules. Leavers for positive reasons reported greater financial security and better housing conditions than respondents who remained on assistance, but those who left for negative reasons fared about the same as those still on assistance.

68 It is acknowledged that some FSS group members in these outcome groups could also have benefited from their receipt of FSS-related services—for example, by attaining an academic degree or occupational credential or by improving their credit score or ability to budget or save.

47

Exhibit 3.6: Baseline Characteristics of the Family Self-Sufficiency Group, by HCV and Family Self-Sufficiency Program Status and Escrow Accrual Status

Enrolled in Housing Choice Voucher Program

Enrolled

in FSS

Not Enrolled in

Not

with

Enrolled in FSS

Housing Choice

Enrolled

Balance

with Balance of

Graduated

Characteristic

Voucher Program

in FSS

of $0

$1 or More

from FSS

Total

Household characteristics

Average number of household membersa

3.3

3.1

3.3

3.3

3.3

3.2

Households with more than 1 adult (%)

35.7

35.3

39.0

33.9

35.2

35.3

Average number of children in household

1.8

1.6

1.8

1.9

1.8

1.8 *

Number of children (%)

***

No children

23.7

32.7

20.2

20.3

13.8

22.8

1 child

21.8

21.8

27.4

22.7

32.7

24.2

2 children

26.0

19.8

19.0

24.0

27.6

24.1

3 children or more

28.6

25.7

33.3

33.0

25.8

28.9

For households with children, age of youngest

child (%)

0–5 years

38.9

39.3

37.9

42.4

42.7

40.5

6–12 years

42.9

42.2

51.5

41.2

38.4

42.1

13–17 years

18.2

18.5

10.6

16.4

18.9

17.4

Primary language spoken at home is English (%)

93.9

89.8

89.2

90.6

93.5

91.9

Receives TANF (%)

13.7

18.4

12.0

22.6

12.0

16.3 ***

Receives food stamps/SNAP (%)

65.8

71.2

71.4

77.7

72.8

71.2 **

(continued)

48

Exhibit 3.6 (continued)

Enrolled in Housing Choice Voucher Program

Enrolled in

Not Enrolled in

Not Enrolled in FSS

FSS with

Housing Choice

Enrolled with Balance of

Balance of

Graduated

Characteristic

Voucher Program

in FSS

$0 $1 or More

from FSS

Total

Length of time receiving Section 8 Housing

Choice Voucher (%)

Less than 4 years

33.6

32.8

38.6

30.2

29.2

32.2

4–6.99 years

24.3

18.4

12.0

23.4

23.1

21.9

7 years or more

42.1

48.8

49.4

46.4

47.7

45.9

Total annual household income (%)

**

Less than $10,000

40.3

49.6

39.0

39.4

36.7

41.3

$10,000–$19,999

29.6

31.1

28.0

36.3

36.3

32.5

$20,000 or more

30.1

19.3

32.9

24.3

27.0

26.2

Payment for rent and utilities (%)

Less than $400

40.7

44.9

44.6

42.1

41.4

42.2

$400–$599

19.4

21.5

16.9

24.9

22.3

21.4

$600 or more

40.0

33.6

38.6

33.0

36.3

36.3

During the past 12 months, household experienced

at least one financial hardship (%)

57.6

59.5

54.8

55.7

59.4

57.7

Not able to buy prescription drug

12.5

14.1

13.3

11.1

15.3

13.0

Not able to buy food

23.8

27.7

21.7

27.9

28.2

26.2

Not able to pay telephone bill

28.0

23.4

30.1

30.3

33.3

28.7

Not able to pay rent

19.1

17.6

14.5

17.5

18.1

17.9

Not able to pay utility bill

42.8

45.3

41.0

44.4

42.6

43.5

(continued)

49

Exhibit 3.6 (continued)

Enrolled in Housing Choice Voucher Program

Enrolled in

Not Enrolled in

Not Enrolled in FSS

FSS with

Housing Choice

Enrolled with Balance of

Balance of

Graduated

Characteristic

Voucher Program

in FSS

$0 $1 or More

from FSS

Total

Sample member characteristics

Female (%)

89.7

87.5

89.3

95.3

94.5

91.4 ***

Age (%)

***

19–34 years

38.2

26.5

39.3

34.7

41.9

35.7

35–44 years

37.0

31.1

31.0

39.7

40.1

36.6

45–61 years

24.8

42.4

29.8

25.7

18.0

27.7

Average age (years)

38.5

42.2

38.9

38.8

37.1

39.1 ***

Married and living with spouse or cohabitating

(%)

10.1

10.5

11.9

8.0

10.1

9.8

Race/ethnicity (%)

Black, non-Hispanic/Latino

73.5

74.3

64.3

70.0

73.3

72.2

Hispanic/Latino

15.7

13.2

17.9

16.7

15.7

15.6

Other

10.8

12.5

17.9

13.3

11.1

12.2

Education (%)

***

No high school diploma or GED

13.6

17.1

13.1

15.7

6.5

13.6

High school diploma or GED

12.9

19.1

10.7

12.0

14.4

14.1

Some college or received technical/trade

license

57.4

52.1

52.4

55.9

47.0

53.9

2-year degree or higher

16.0

11.7

23.8

16.4

32.1

18.4

Has trade license or training certificate

50.6

42.4

48.8

47.3

44.7

47.1

(continued)

50

Exhibit 3.6 (continued)

Enrolled in Housing Choice Voucher Program

Enrolled in

Not Enrolled in

Not Enrolled in FSS

FSS with

Housing Choice

Enrolled with Balance of

Balance of

Graduated

Characteristic

Voucher Program

in FSS

$0 $1 or More

from FSS

Total

Employment status

Currently employed (%)

55.5

44.9

53.6

56.9

67.4

55.6 ***

Current employment type (%)

***

Regular job

49.9

35.2

47.6

49.2

56.5

47.7

Self-employed

2.3

5.5

3.6

3.4

2.8

3.4

Temporary or seasonal job

3.3

4.3

2.4

4.4

7.9

4.5

Currently working 1–34 hours per week (%)

24.1

22.7

20.2

29.0

33.5

26.3 **

Currently working 35 hours or more per week (%)

31.4

21.9

33.3

27.6

33.5

29.1 **

Average hours worked per week

18.4

14.1

18.3

17.9

21.2

17.9 ***

Average weekly earnings ($)

222

159

207

190

248

205 ***

Barriers to employment (%)

Has any problem that limits work

41.7

52.9

42.9

34.7

36.9

41.6 ***

Physical, emotional, or mental health

21.6

34.5

14.3

16.1

11.7

20.8 ***

Childcare access or cost

17.6

17.9

20.2

17.9

20.9

18.5

Other

15.3

19.3

12.0

10.4

12.0

14.2 **

Receives SSI or SSDI benefits

15.2

23.7

11.9

8.0

7.8

13.8 ***

Limited English-speaking ability

2.8

4.7

6.0

4.3

1.9

3.6

No access to public transportation

82.8

85.4

73.8

83.1

87.0

83.5 *

No access to an automobile

80.6

72.5

84.5

84.6

88.8

81.5 ***

FSS program (%)

Heard of escrow before random assignment

45.8

42.2

34.5

39.0

52.8

43.9 ***

(continued)

51

Exhibit 3.6 (continued)

Enrolled in Housing Choice Voucher Program

Enrolled in

Not Enrolled in

Not Enrolled in FSS

FSS with

Housing Choice

Enrolled with Balance of

Balance of

Graduated

Characteristic

Voucher Program

in FSS

$0 $1 or More

from FSS

Total

Interest in FSS services related to

Job-related services

67.7

69.6

72.6

75.3

69.0

70.4

Social services

31.4

38.1

28.6

29.7

31.5

32.2

Financial services

97.0

94.6

95.2

94.3

94.9

95.4

Sample size

427

257

84

300

217 1,285

Notes: The FSS impact sample includes housing choice voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Sample sizes for specific measures may vary because of missing values. Rounding may cause slight discrepancies in calculating sums. Detail may sum to more than total for questions that allow more than one response.

a Maximum response option for number of adults in a household is four.

Sources: MDRC calculations from Baseline Information Form data and housing agency administrative data Exhibit 3.7 shows the average change in employment and earnings over 5 to 6 years for each of the five groups, using data reported to HUD’s Inventory Management System/PIH (Office of Public and Indian Housing) Information Center (IMS/PIC) system. For these outcomes, FSS

graduates clearly stand out.

52

Exhibit 3.7: Change in Reported Annual Head-of-Household Earnings and Total Household Earnings, Between Month of Random Assignment and the Last Month of Followup, by HCV and Family Self-Sufficiency Enrollment Status and Escrow Balance

Enrolled in Housing Choice Voucher Program

Enrolled

Not Enrolled in

Not

in FSS with Enrolled in FSS

Housing Choice

Enrolled

Balance with Balance of

Graduated

Outcome

Voucher Program

in FSS

of $0

$1 or More

from FSS

Total

Head of household

Average earnings in month 1 ($)

10,873

6,498

10,715

8,424

10,010

9,280

***

Average increase in estimated

annual earnings ($)

4,237

1,419

1,870

6,531

15,462

5,904

***

Change in earnings (%)

***

Decrease

20.8

24.0

27.5

20.5

10.3

20.1

No change

32.5

47.2

36.3

23.6

10.8

30.0

Increase

46.7

28.9

36.3

55.8

78.8

49.9

Head of household employment (%)

***

In month 1 and in current or most

recent month

43.8

19.9

37.5

37.7

50.7

38.3

In month 1 only

11.0

17.9

15.0

11.6

7.4

12.2

In current or most recent month only

17.5

15.0

11.3

27.1

31.0

21.1

No employment in either month

27.8

47.2

36.3

23.6

10.8

28.4

Other household members

Average earnings in month 1 ($)

1,673

1,456

2,121

1,668

1,813

1,682

Average increase in estimated

annual earnings ($)

2,375

1,967

1,022

4,052

5,916

3,205

***

(continued)

53

Exhibit 3.7 (continued)

Enrolled in Housing Choice Voucher Program

Enrolled

Not Enrolled in

Not

in FSS with Enrolled in FSS

Housing Choice

Enrolled

Balance with Balance of

Graduated

Outcome

Voucher Program

in FSS

of $0

$1 or More

from FSS

Total

Change in earnings (%)

***

Decrease

4.3

5.3

8.6

6.5

4.2

5.3

No change

80.4

80.9

77.8

70.3

67.8

75.8

Increase

15.3

13.8

13.6

23.2

28.0

18.9

Other household members employment (%)

***

In month 1 and in current or most

recent month

5.0

4.5

7.4

7.2

7.9

6.1

In month 1 only

3.1

4.9

6.2

4.8

3.7

4.2

In current or most recent month only

12.0

9.8

8.6

17.7

20.6

14.1

No employment in either month

79.9

80.9

77.8

70.3

67.8

75.6

Total household

Average earnings in month 1 ($)

12,546

7,981

12,836

10,092

11,824

10,970

***

Average increase in estimated

annual earnings ($)

6,199

2,702

2,830

10,474

22,077

8,965

***

Change in earnings (%)

***

Decrease

30.6

26.4

35.1

21.6

8.1

23.1

No change

27.4

37.2

27.3

16.1

5.6

22.6

Increase

42.0

36.4

37.7

62.3

86.3

54.3

(continued)

54

Exhibit 3.7 (continued)

Enrolled in Housing Choice Voucher Program

Enrolled

Not Enrolled in

Not

in FSS with Enrolled in FSS

Housing Choice

Enrolled

Balance with Balance of Graduated from

Outcome

Voucher Program

in FSS

of $0

$1 or More

FSS

Total

Total household employment (%)

***

In month 1 and in current or most recent

month

36.3

27.2

40.3

45.2

56.3

40.5

In month 1 only

18.5

17.6

19.5

11.0

3.1

13.3

In current or most recent month only

17.8

18.0

13.0

27.7

35.0

23.6

No employment in either month

27.4

37.2

27.3

16.1

5.6

22.6

Sample size

418

246

80

292

203

1,239

Notes: The FSS impact sample includes HCV households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Calculations of change in annual earnings use the most recent available estimate reported to HUD, which could occur before or after month 36 of followup, depending on each study participant’s date of random assignment, HCV program status, and, if applicable, date of exit from the HCV program. Rounding may cause slight discrepancies in calculating sums and differences. Sample sizes for specific outcomes may vary because of missing values. A chi-square test was performed on subgroup differences in frequency distributions, and an F-test was performed on subgroup differences in means. Statistical significance levels are indicated as * = 10 percent; ** = 5 percent; *** = 1 percent.

Sources: MDRC calculations using housing authority administrative data and U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data

55

On average, FSS graduates increased their reported estimated annual earnings by more than $15,000, more than twice the increase of any other group. Nearly 80 percent of FSS graduates experienced an increase in earnings during the followup period, with about 40 percent of this increase accounted for by FSS group members who transitioned from joblessness to employment during the followup period. 69 On average, FSS graduates also experienced the largest increase in reported earnings from other household members, primarily due to one or more other household members starting employment during the followup period. Including earnings from all household members, FSS graduates increased their total household earnings by an average of $22,000 during the followup period, more than double the increase of any other group.

To a lesser extent, FSS group members who remained enrolled in the program and maintained a positive escrow balance also experienced an increase in earnings over time—averaging about $6,500

per FSS group member, and a somewhat smaller increase (of $4,000) over time in earnings from other household members. These averages exceeded the increases for all other groups except FSS graduates.

By contrast, FSS group members who left the program while continuing to receive HCV housing assistance recorded the smallest growth in earnings over time and the highest incidence of joblessness at their time of random assignment and also during the last month of followup.

On average, FSS group members who exited from both FSS and HCV also did better financially over time but by a smaller amount compared with FSS graduates or current enrollees who accrued escrow.

FSS group members who left all types of HUD assistance increased their estimated annual earnings by an average of about $4,200 and household earnings by about $6,200 during the followup period.

Conclusions

By the winter of 2019–2020 (just before the onset of the COVID-19 pandemic), implementation of the FSS program had reached a critical stage. Most participants who remained in the program had a positive balance in their escrow accounts and, therefore, a strong financial incentive to complete their goals, although few, if any, additional participants would be expected to start accruing escrow.

Although it is impossible to predict exactly how many additional FSS group members will graduate, one may assume that most FSS group members with at least $1,000 in escrow would eventually graduate. If so, the graduation rate would likely end up at close to 30 percent of the FSS group. This average would place the 18 housing agencies in the FSS evaluation above the combined average of 24

percent for the approximately 700 PHAs with graduation rates published by HUD in November 2018

but well below HUD’s current threshold (of 38 percent) for the highest ranking PHAs. 70

69 As exhibit 3.7 shows, 31.0 percent of FSS graduates became employed during the followup period, whereas 78.8

percent of graduates increased their earnings. Accordingly, (31.0 percent / 78.8 percent) x 100.0 = 39.4 percent.

70 See HUD, 2018. The average of 24 percent can be derived by dividing the national total of FSS graduates by the total of FSS enrollees eligible for graduation—also, coincidentally, by calculating the median graduation rate among the PHAs. See also Verma et al., 2017, table 3.1, p. 41, which shows a graduation rate of 43 percent for the Work Rewards FSS-Only group in New York City. Most FSS-Only group members (27.3 percent/43.1 percent) graduated during Year 6 of followup. See also Geyer et al., 2017. The evaluation of the FSS programs in Cambridge and Lynn, Massachusetts ended before most FSS participants had reached the end of their FSS contract.

56

However, as discussed previously, the COVID-19 pandemic and economic downturn have hindered the remaining participants’ normal progression through the program by causing high unemployment and preventing FSS group members from completing their service- or education-related goals. 71 The evaluation will continue to track graduations and disbursements, at least through the end of 2020, but one may assume that more FSS group members will have received 2-year extensions of their COP and may still be enrolled at the end of followup for the evaluation. If so, the graduation rate will likely be closer to the 2018 national average by the end of followup for the final report in the evaluation.

Regardless of the final graduation rate, the FSS program eventually provides a substantial one-time payment, often exceeding the maximum amount that a household with a low or moderate income could receive as an earned income credit on their federal tax return. However, a large majority of enrollees (possibly around 70 to 75 percent) do not receive any escrow disbursement. Some may benefit from receiving FSS services and case management—for example, by attaining an educational or occupational credential or by learning to manage their family finances more efficiently, increasing their savings, or reducing their debt. Nonetheless, it may be difficult to discover any tangible benefits of enrollment for FSS group members who left the program while continuing to receive HCV assistance, and perhaps the same is true for the small group of current FSS enrollees with no escrow accrual after 5 years in the program.

In the chapters that follow and in later reports, the question of “who benefits from FSS” will be examined through additional descriptive analyses of the financial well-being of FSS group members and through comparisons of financial outcomes for FSS group members with outcomes for control group members.

71 In response to the COVID-19 pandemic and associated economic downturn, HUD gave housing agencies the authority to extend FSS contracts to a maximum of 8 years.

57

Chapter 4. Impacts on Employment and Earnings

As described in previous chapters, the Family Self-Sufficiency program provides referrals to supportive services, financial counseling, and case management services that are meant to encourage participants to find a steady job or, if they are already working, to increase their earnings. In addition, the FSS escrow account serves as both an asset-building instrument and a work incentive.

This chapter focuses on the program’s effects on employment and earnings during the first 5 years (60

months) after program enrollment, recognizing that participants still enrolled in the program could be pursuing a wide range of goals during this period. It builds on the 3-year results in the previous report on this evaluation. The earlier data revealed high levels of employment for both study groups but no notable differences in earnings or employment outcomes for the FSS and control groups. This longer-term analysis uses more recent data from administrative records to determine whether the FSS program group experienced a higher employment rate or earned more on average than members of the control group. Chapter 4 focuses on the full sample, and chapter 6 examines variations in the effects on employment earnings by subgroup.

Overall, the analysis of employment and earnings outcomes, using National Directory of New Hires (NDNH) quarterly wage data for the full impact sample, reveals high levels of employment for both study groups and no statistically significant impacts on employment and earnings during the 5-year followup period or in Year 5.

Data Sources and Methods

Estimates of FSS program effects (or impacts) on earnings and employment use federal NDNH

quarterly wage data. 72 NDNH data provide information on quarterly earnings and are available for the impact sample for 2 quarters before and 20 quarters after the quarter of random assignment. 73 NDNH

records provide data on employment and earnings in all work covered by unemployment insurance, including across state lines (for those who commute into another state for work or who moved to a different state after random assignment), and on federal employment not captured in state unemployment insurance records. The records do not cover earnings from self-employment, some agricultural work, and informal jobs. Other research suggests that administrative data may miss relatively more employment for low-income populations than for higher-income groups, given the former group’s greater prevalence of work in informal jobs (Abraham et al., 2009). NDNH records also do not provide information about the hours worked during a quarter or week or on the characteristics of jobs held, such as hourly wage rates, benefits, and schedules.

Exhibit 4.1 includes an explanation of how to read the impact tables in this report. The study design uses random assignment to create the FSS and control groups; therefore, effects or impacts of the FSS

72 For a description of the variables included in the presented models and for analyses of the sensitivity of results to outliers and to different data-weighting approaches, see appendix C. In two previous reports, additional analyses on earnings and employment used responses to 18-month and 36-month followup surveys. An upcoming report will use responses to a Long-Term Followup Survey administered 6 to 7 years after random assignment.

73 Employment recorded during the quarter of random assignment may have occurred before the study participant’s date of random assignment. Accordingly, the analysis excludes this quarter from the followup period.

58

program can be calculated as the difference in average outcomes between the research groups.

Differences that are statistically significant (indicated by asterisks in the exhibits) are considered to be true program effects and not the result of chance. 74 Chapter 4 presents the effects of the program for the FSS impact sample, which excludes voucher holders age 62 or older at the time of random assignment. Chapter 6 presents impacts for subgroups included in the analyses of escrow accrual and graduations from FSS programs. The essential research question for subgroup analysis is whether the differences in impacts across subgroups are statistically significant. (In chapter 6, subgroup differences that are statistically significant are noted with daggers in the exhibits.)

Program Impacts on Employment and Earnings

Employment and Earnings Trends for Control Group Members

Exhibits 4.2 and 4.3 and appendix exhibit C.1 display quarterly employment and earnings trends for control group members, calculated with NDNH data. These averages, which reflect what would have happened in the absence of the program, provide the basis of comparison for estimating the impacts of the FSS program on employment and earnings. As shown by the solid line in exhibit 4.2, employment levels for control group members overall increased slightly over time, from about 58 percent in the second quarter before random assignment to around 64 percent in quarter 21 (with quarter 1

representing the quarter of random assignment). Control group members who reported on the Baseline Information Form (BIF) that they were not working (see the dotted line) experienced the biggest increase during the followup period, of about 16 percentage points, whereas employment rates remained relatively static for control group members who reported working part-time hours and declined slightly for control group members who reported having full-time employment.

74 An exception to this statement concerns a situation in which only one comparison among a series of related comparisons shows a statistically significant difference between the research groups—for example, if FSS group members averaged higher earnings than control group members during only one quarter of followup. In this situation, less credence would be given to this single impact estimate, even if the difference were statistically significant.

59

Exhibit 4.1: How to Read the Impact Tables in This Report

In the context of this evaluation, an “impact” is a measure of how much the intervention—Family Self-Sufficiency (FSS)—changed outcomes for program participants. The group outcome for the intervention is compared with that of the control group. The top row of the excerpted table below, for example, shows that 26 percent of the FSS group was working part time at the time of the 18-month survey, compared with 29 percent of the control group.

Because participants were assigned randomly to either the program group or the control group, the effects of the program can be estimated by the difference in outcomes between the two groups. The

“Difference” column in the table excerpt shows the differences between the two research groups’

outcomes—that is, the program’s estimated impacts on the outcomes. For example, the estimated program impact of the FSS program on the number of individuals working part time can be calculated by subtracting 29 percent from 26 percent, yielding a decrease, or estimated impact, of 3

percentage points.

The p-value shows the probability that this difference, or impact, arose by chance. In the table excerpt below, the difference between the program and control groups in current part-time employment has a 16.6-percent probability of occurring due to chance rather than as a result of the FSS-only program. In contrast, the difference on the measure current full-time employment has a 4.5-percent probability of having arisen by chance. For this evaluation, only differences that have a 10-percent probability or less of occurring by chance are considered “statistically significant” and therefore represent true program effects. The number of asterisks indicates whether the impact is statistically significant at the 1-percent (***), 5-percent (**), or 10-percent (*) level, meaning that there is only a 1-, 5-, or 10-percent probability, respectively, that the impact arose by chance.

Impacts on Employment, Family Self-Sufficiency 18-Month Survey

Respondent Sample

FSS

Control

Difference

Outcome

Group

Group

(Impact)

P-Value

Currently employed (%)

Works part-time hours

26.0

29.0

-3.0

0.166

Works full-time hours

41.5

37.1

4.3

0.045 **

60

Exhibit 4.2: Quarterly Employment Rate Among Control Group Members by Self-Reported Employment Status at Random Assignment, Family Self-Sufficiency Impact Sample

Note: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment.

Source: MDRC calculations using baseline data and quarterly wage data from the National Directory of New Hires.

61

As exhibit 4.3 shows, on average, control group members earned more over time—from $2,683 per quarter (including zeros for control group members without employment) in the second quarter before random assignment to $4,494 in quarter 21 after random assignment. This increase resulted from some control group members entering employment and other control group members increasing their hours or weeks of employment or earning more on the job. The average quarterly earnings also increased for control group members in all three subgroups on the basis of members’ self-reported employment at random assignment. Once again, control group members in the subgroup that reported no employment at random assignment experienced the biggest increase during the followup period.

Impacts on Employment and Earnings

Although the FSS program is designed to provide services and referrals to address a variety of difficulties faced by participants, including lack of adequate childcare, mental and physical health issues, and transportation challenges, this component is in service to the ultimate goal of the program: to move people to work and build economic self-sufficiency. The escrow incentive is designed with the same goal and is thus attached to increases in earnings. Therefore, an important test of the FSS

model is assessing its ability to increase employment (a condition for graduation and earning the accrued escrow), employment stability, and earnings.

Over the 5-year followup period, both FSS and control groups experienced comparable employment rates, and there is no evidence that FSS has led to impacts in employment covered by NDNH.

The bottom half of exhibit 4.4 shows program impacts on yearly earnings, as measured by the NDNH

data. The control group averaged $12,144 over the first year of followup. This figure rose to $17,490

in the fifth year of followup. Over this 5-year followup period, control members averaged about $75,000 in earnings, which translates into yearly earnings of nearly $15,000. 75 FSS group members earned, on average, about 1 percent more over 5 years, but the difference was not close to statistical significance. Members of both groups also averaged about the same number of quarters of employment during the followup period and were equally likely to earn at least $25,000 per year. 76

75 Using quasi-experimental methodology and a matched comparison group, an analysis of the Compass Working Capital FSS program, administered by the nonprofit agency for the PHAs in Lynn and Cambridge, Massachusetts, shows that the program produced employment and earnings impacts for participants. The study, which used income data available to HUD, found that the Compass FSS program was associated with an average gain in annual household earnings of $6,305 between the fourth quarter of 2010 and the first quarter of 2016 (Geyer et al., 2017).

76 The top panel of appendix exhibit C.1 compares estimated annual earnings for the FSS and control groups, using HUD IMS/PIC data. It shows a higher average for the FSS group, a difference of more than $1,100. However, the comparison is nonexperimental in that it includes FSS and control group members who still received HCV assistance at the end of Year 5. Moreover, it may be reasonably asserted that FSS group members’ access to an escrow account increases their incentive to report increases in earnings soon after they occur.

62

Exhibit 4.3: Average Quarterly Earnings Among Control Group Members, by Self-Reported Employment Status at Random Assignment, Family Self-Sufficiency Impact Sample

Note: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment.

Source: MDRC calculations using baseline data and quarterly wage data from the National Directory of New Hires.

63

Exhibit 4.4: Impacts on Employment and Earnings in Years 1 Through 5, Family Self-Sufficiency Impact Sample

FSS

Control

Difference

Outcomes

Group

Group

(Impact)

P-Value

Employment (%)

Ever employed

Year 1

73.4

72.5

0.9

0.496

Year 2

74.8

72.8

1.9

0.182

Year 3

74.2

72.5

1.7

0.257

Year 4

75.0

72.1

2.9 *

0.061

Year 5

73.5

74.1

-0.6

0.691

Years 1 through 5

86.9

85.4

1.5

0.193

Average quarterly employment rate

Year 1

61.6

61.4

0.2

0.887

Year 2

64.0

63.7

0.3

0.847

Year 3

64.9

63.9

1.0

0.502

Year 4

64.8

64.2

0.6

0.672

Year 5

64.9

64.4

0.5

0.732

Years 1 through 5

64.0

63.5

0.5

0.651

Employed in all quarters

Year 1

47.1

47.9

-0.9

0.571

Year 2

51.1

51.9

-0.8

0.634

Year 3

53.3

53.8

-0.5

0.784

Year 4

52.9

53.7

-0.8

0.652

Year 5

54.6

52.9

1.6

0.352

Years 1 through 5

24.8

27.3

-2.5

0.109

Earnings ($)

Total earnings

Year 1

11,967

12,144

-177

0.596

Year 2

14,178

14,044

134

0.760

Year 3

15,509

15,307

202

0.688

Year 4

16,526

16,346

180

0.742

Year 5

17,748

17,490

258

0.662

Years 1 through 5

75,929

75,332

598

0.767

Average annual earnings (%)

0.462

$0

13.1

14.6

-1.5

$1–$10,000

31.9

32.4

-0.5

$10,001–$25,000

31.7

29.3

2.4

Greater than $25,000

23.4

23.7

-0.4

Sample size (total = 2,548)

1,282

1,266

Notes: The FSS impact sample includes housing choice voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Average quarterly employment rate is calculated as total quarters with employment divided by total quarters of followup, expressed as a percentage. Estimates were regression adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between research groups. The p-64

value indicates the likelihood that the difference between the FSS group and the control group arose by chance.

Statistical significance levels are indicated as * = 10 percent; ** = 5 percent; and *** = 1 percent.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires Conclusions

For the full sample, the NDNH data reveal no impacts on quarterly earnings or employment. Given that a relatively large proportion of FSS group members have accrued escrow dollars and are relatively close to graduation, they have a strong financial incentive to maintain employment (if possible, during the COVID-19 pandemic) to be eligible to graduate. If so, FSS group members still enrolled in the program (about 30 percent) could conceivably start to earn more than the control group in Years 6 or 7. Chapter 6 examines the program’s effects on subgroups and whether study participants, defined by baseline characteristics such as their work and educational status or the type of FSS program they were enrolled in, respond differently to the services and escrow incentive offered by FSS. FSS group members may improve their financial well-being by reducing their debt or maintaining debt with lower interest, using traditional financial services businesses, and forgoing use of high-interest alternative financial services businesses, such as payday loan establishments. Chapter 5 explores these issues for the full impact sample, and chapter 6 continues the analysis for selected subgroups.

65

Chapter 5. Impacts on Credit Use, Housing Costs, Receipt of

Housing Subsidies, and Financial Well-Being

Chapter 5 analyzes whether the Family Self-Sufficiency (FSS) program led to positive effects on a series of financial security outcomes. The analysis uses credit scores and financial transactions data from Experian, the nation’s largest credit bureau, as the primary source for calculating financial and material well-being outcomes and estimating program impacts. Experian data include transactions involving traditional financial services, such as store and credit cards, auto loans, and student loans.

Experian also provided data on the use of alternative financial services (AFS), such as payday loans, recorded by Clarity Services, Inc., a subsidiary of Experian. Additional measures of financial well-being were calculated using HUD administrative data on participant housing expenditures and rent subsidies. 77

The FSS program seeks to promote housing choice voucher (HCV) program households’ self-sufficiency and financial security in several ways. First, FSS programs offer participants access to services and an escrow account that promote new employment, employment stability, and earnings growth. As discussed in chapter 4, after 5 years, FSS heads of household experienced similar incidences of employment and earnings on average compared with their counterparts in the control group. Nonetheless, the program could still improve FSS group households’ financial situation in two other ways. Once a head of household enrolls in an FSS program, other adult household members covered by the housing voucher can receive FSS services (either way, their earnings are counted toward the issuance of escrow credits). In this way, FSS could potentially increase participants’ household earnings and income beyond what they would have received in the absence of the program without directly affecting the head of household’s employment and earnings. In addition, as discussed in chapter 2, many FSS programs strongly encourage participants to attend financial management workshops or meet with counselors to receive instruction in managing personal and household finances or qualifying to purchase a home. As advocates for financial empowerment services often attest, these activities can lead to tangible financial gains, even without increases in income. 78 For example, participants in financial security activities can learn to (1) increase savings; (2) reduce debt; (3) increase credit scores; (4) forgo high-cost, nontraditional lending sources; and (5) avoid financial hardship.

Participants in financial security or homeownership preparation activities could also benefit in less tangible ways—for example, by reducing stress and experiencing a greater sense of control over life decisions and more optimism for the future. 79

77 See Verma et al., 2021, Chapter 6, for an analysis of FSS program impacts on indicators of financial well-being, using responses to a 36-month survey. A later report will include a similar analysis after 6 or 7 years of followup, using responses to a long-term followup survey. The report will also analyze additional followup for credit score and financial transactions data. See also Geyer et al. (2017), the Abt Associates’ evaluation of the Compass Working Capital FSS programs in Cambridge and Lynn, Massachusetts, which found that FSS program enrollees averaged higher Experian FICO credit scores and incurred less derogatory debt than members of a comparison group.

78 See, for example, Abbi (2012); Collins and Gjertson (2013); Lopez-Fernandini (2012); and McKernan, Ratcliffe, and Vinopal (2009).

79 In this chapter, the terms material well-being, financial well-being, and financial security are used interchangeably.

66

The main findings from this analysis are as follows:

• Average Experian Vantage credit scores for both groups increased by about 20 points during the followup period. Most often, this increase did not place the participant on a higher level, which would be associated with easier access to credit.

• According to credit data, during the followup period, the FSS program did not lead to increases above the control group in average credit scores.

• Over a 5- to 6-year followup period, both the FSS and control groups incurred higher levels of debt over time, often for automobile and education loans. During 3 of these years, including 2019, FSS group members paid about $30 (or about 12 percent) more, on average, per month than control group members for these loans from traditional financial services lenders.

• About 6 out of 10 members from both groups experienced at least one relatively serious credit problem, such as forgoing payments or borrowing close to the limit of available credit, during the final year of followup.

• The FSS program had almost no statistically significant effects on credit use or the incidence of credit problems.

• Over 5 years, FSS group members paid, on average, about 5 percent more out of pocket for rent and utilities than did control group members. For about 10 percent of FSS group members who graduated from the program (and 17 percent, when all available followup is included), disbursements from their escrow account made up for these additional expenses.

• Not surprisingly, at the end of the followup period, the incidence of homeownership was very low (about 5 percent) for both research groups.

Experian and Clarity Credit Data

For the analysis, Experian provided Vantage 3.0 (hereafter, Vantage) credit scores and associated financial transactions data. VantageScores were chosen for this analysis over Experian’s better known FICO scores because they include a larger number of financial services customers with low or moderate incomes. 80 Experian stores Vantage data in monthly archives, containing current “snapshots” of customers’ credit scores, as well as financial transaction indicators that cover part or all of the previous 12 months. For this analysis, MDRC collected data from eight monthly archives, spaced 12 months apart, starting in December 2012 and ending in December 2019. The final month corresponds to the end of Year 5 of followup (month 60) for participants who entered the study during December 2014, the final month of random assignment, and to early in Year 7 (month 75) for study participants who entered the study during October 2013, the first month of random assignment. The chapter treats data for December 2012 as pre-random assignment history; December 2013 and 2014 data as occurring 80 Experian also provided two types of FICO scores for the impact sample. Review of this information showed that about 85 to 90 percent of impact sample members had a FICO score as of the end of followup in December 2019, whereas 96 percent of impact sample members had a VantageScore. According to DeNicola (2019), VantageScores factor in recurring payments, such as utilities and rent, as well as the typical loan products used to calculate FICO

credit scores, such as credit card and mortgage payments, allowing individuals with less complete credit histories to be scored. In addition, the Consumer Financial Protection Bureau (2012) cites a report by the Federal Reserve that found that VantageScores are highly correlated with the more commonly used FICO credit scores.

67

during the “random assignment period”; and data from December 2015 through December 2019 as post-random assignment followup outcomes.

VantageScores vary from 300 to 850. According to Experian and Clarity, scores may be grouped into the ranges or “levels” shown in exhibit 5.1.

Exhibit 5.1: VantageScore Ranges

Credit Score

Rating

Percentage of U.S.

Customers (%)

Impact

300–499

Deep Subprime

5

Applicants will not likely be approved for credit.

500–600

Subprime

21

Applicants may be approved for some credit,

although rates may be unfavorable and have

conditions such as larger downpayment amounts.

601–660

Near Prime

13

Applicants may be approved for credit but likely

not at competitive rates.

661–780

Prime: Good

38

Applicants are likely to be approved for credit at

competitive rates.

781–850

Prime: Excellent

23

Applicants are most likely to receive the best rates

and most favorable terms on credit accounts.

Note: Experian also refers to these ratings as “Very Poor,” “Poor,” “Fair,” “Good,” and “Excellent.”

Sources: “What is a Good Credit Score?” Experian website: https://www.experian.com/blogs/ask-experian/credit-

education/score-basics/what-is-a-good-credit-score/; Clarity Experian (2020), table 3, p. 16

The analysis that follows will use these range values but also combine categories into Subprime (deep subprime and subprime), Near Prime, and Prime (both good and excellent), as warranted, for simplicity.

Clarity uses the name Clear Early Risk Score™ (hereafter, Clarity scores) for its credit scoring metric.

Clarity has calibrated its scores to the same range as Vantage—from 300 to 850—with equivalent range values. Clarity uses a different series of criteria for calculating scores, compared with Vantage and FICO, recognizing that the scores are intended primarily for use by alternative financial services lenders to evaluate the credit risk of potential borrowers, many of whom are in financial distress when applying for a loan and need immediate access to short-term credit. In addition, Clarity created its Clear Early Risk Score to provide credit ratings for financial services customers who lacked sufficient credit history or had too many credit problems to qualify for a Vantage or FICO score. By contrast, Clarity usually forgoes calculating a score for customers who never use AFS. For that reason, both Clarity and VantageScores are needed to obtain a relatively complete picture of each study participant’s credit status.

The impact analysis for credit scores compares average Vantage and Clarity scores and range values for FSS and control group members at the end of followup in December 2019 and compares trends in credit score values by research group. Next, the chapter compares FSS and control group members’

levels of debt from traditional and AFS sources of credit—in December 2019 and over time. The chapter also examines whether FSS group members experienced a decrease below the level of the 68

control group in the incidence of credit problems, for example, from extremely late or forgone payments or from carrying a total balance close to the study participant’s maximum credit limit.

Experian financial transactions data also include mortgages and home equity loans. The analysis will use this information to compare rates of homeownership for FSS and control group members.

Impacts on Credit Scores

The analysis of the FSS program’s impacts on credit scores begins with a short summary of year-by-year trends in credit scores for FSS group members. As exhibit 5.2 shows, the typical FSS group member received Subprime VantageScores (below 600) from 2012 through 2019. FSS group members’ scores improved somewhat over time, averaging 555 during the random assignment years (2013 and 2014) and increasing by nearly 20 points during the 5 years of followup. During the followup years (2015–2019), the proportion of FSS group members with Prime VantageScores (above 660) increased fairly steadily, from about 8 percent in 2013 to 16 percent 6 years later, whereas the proportion of FSS group members with Subprime scores (below 601) decreased by a similar amount (see exhibit 5.3).

Exhibit 5.2: Family Self-Sufficiency Group Members’ Average Experian Vantage 3.0

Credit Scores, 2012–19, Family Self-Sufficiency Impact Sample (FSS Group Only) Pre-random

Random

Followup period

assignment

assignment period

Sources: MDRC calculations using Experian Vantage 3.0 credit scores and Clarity Clear Early Risk scores data 69

Exhibit 5.3: Changes in FSS Group Members’ Average Experian Vantage 3.0 Credit Scores, 2012–19, Family-Self Sufficiency Impact Sample (FSS Group Only)

100.0

8.0

8.3

9.0

11.2

12.7

13.0

15.0

15.9

90.0

11.1

12.2

14.4

80.0

16.1

14.0

15.3

15.9

15.3

70.0

60.0

%)

56.0

56.1

tage ( 50.0

54.1

51.3

49.8

50.0

cen

46.6

49.8

Per 40.0

30.0

20.0

21.1

19.8

19.2

18.4

19.8

17.2

18.4

15.2

10.0

3.9

3.6

3.4

3.0

3.7

4.4

4.1

3.8

0.0

2012

2013

2014

2015

2016

2017

2018

2019

Pre-random

Random assignment

assignment period

period

Followup period

No score

Deep Subprime

Subprime

Near Prime

Prime

Sources: MDRC calculations using Experian Vantage 3.0 credit scores Clarity Clear Early Risk scores data As suggested by the overall trend in average VantageScores, FSS group members tended to make modest improvements over time to their scores and range levels, but their scores often moved up or down, sometimes dramatically, year by year. As appendix exhibit D.1 shows, from the start of the random assignment period (in 2013) to the end of the followup period (in 2019), a little more than one-half of FSS group members experienced an increase in their VantageScores of at least 10 points, whereas a little more than one-third of FSS group members experienced a decrease in their score of 10 points or more.

The largest percentage of FSS group members who improved their scores (11 percent) moved from the Subprime (below 601) level to the Near Prime (601–660) level, and only one-fourth of FSS group members had a Prime (above 660) level score in any monthly snapshot file from the followup period.

70

Trends in Clarity scores were less positive (see exhibit 5.4). In general, customers with lower VantageScores tended to use AFS loans more often than did customers with higher scores and therefore had a higher incidence of having a Clarity score. For example, in 2019, about 85 percent of FSS group members with Deep Subprime (below 500) VantageScores had a Clarity score, compared with about 30 percent of FSS group members with Prime (above 660) VantageScores (not shown in exhibits).

Nonetheless, use of AFS increased over time. As exhibit 5.5 shows, in 2014 (the earliest year with Clarity credit score data), nearly one-half of FSS group members had no Clarity score, implying that they were not using AFS products. In 2019, the proportion with no Clarity score was about 14

percentage points lower, suggesting an increase in the use of nontraditional credit products. 81 Also, at no time in the followup period did the proportion of FSS group members with Prime Clarity scores (above 660) exceed 5 percent, an outcome related to the lesser use of AFS by FSS group members with higher VantageScores. 82

Exhibit 5.4: Changes in Family Self-Sufficiency Group Members’ Average Clarity Clear Early Risk Credit Scores, 2014–19

850

800

750

700

re 650

co 600

3.0 Se 550

tagan 500

529

537

538

541

542

544

V

450

400

350

300

2014

2015

2016

2017

2018

2019

Random assignment

period

Followup period

Sources: MDRC calculations using Experian Vantage 3.0 credit scores; Clarity Clear Early Risk scores data 81 This pattern follows a national trend, as discussed in Clarity Experian (2020), p.16.

82 As exhibit 5.4 shows, FSS group members with a Clarity score averaged a score of 544 in 2019, about 30 points below the average VantageScore for the FSS group.

71

Exhibit 5.5: Changes in the Distribution of Clarity Clear Early Risk Credit Scores, 2014–

19, Family Self-Sufficiency Impact Sample (FSS Group Only)

100.0

0.9

2.2

3.1

3.4

4.1

4.7

4.1

6.2

6.6

6.6

8.1

7.0

90.0

80.0

31.1

34.6

34.1

36.7

70.0

34.0

35.6

60.0

)%

15.1

e (tag 50.0

16.2

17.5

cen

16.1

17.9

17.3

Per 40.0

30.0

48.9

20.0

40.7

38.7

37.1

35.9

35.3

10.0

0.0

2014

2015

2016

2017

2018

2019

Random assignment

period

Followup period

No score

Deep Subprime

Subprime

Near Prime

Prime

Source: MDRC calculations using Clarity Clear Early Risk scores data.

On average, control group members recorded similar patterns of Vantage and Clarity credit scores as FSS group members. As exhibit 5.6 shows, at the end of the followup period (in 2019), control group members averaged a VantageScore of 571 and a Clarity score of 539, nearly identical to the averages for the FSS group. Members of both research groups were equally likely to increase their scores over time and equally likely to move into the Prime score range by the end of followup.

72

Exhibit 5.6: Impacts on Credit Scores After 5 to 6 Years of Followup, Family Self-Sufficiency Impact Sample

FSS

Control Difference

Outcome

Group

Group

(Impact)

P-Value

Has credit scores in 2019 (%)

0.966

No scores

3.5

3.5

-0.1

0.943

Experian Vantage 3.0 score only

31.4

33.2

-1.8

0.294

Clarity Clear Early Risk score only

0.3

0.3

0.0

0.913

Vantage and Clarity scores

64.9

62.9

1.9

0.283

Average Vantage 3.0 score

574

571

2

0.505

Vantage 3.0 score (%)

0.810

No score

3.8

3.9

-0.1

0.919

Deep Subprime

15.3

16.8

-1.5

0.295

Subprime

50.1

48.8

1.3

0.522

Near Prime

15.3

15.3

0.0

0.991

Prime

15.6

15.3

0.3

0.798

Average Clarity score

544

539

5

0.159

Clarity score (%)

*

0.099

No score

34.9

36.8

-1.9

0.289

Deep Subprime

17.4

19.0

-1.7

0.268

Subprime

36.0

32.1

3.9 **

0.039

Near Prime

7.0

8.8

-1.8

0.103

Prime

4.8

3.4

1.4 *

0.065

Sample size total = 2,548

1,282

1,266

Notes: The FSS impact sample includes housing choice voucher heads of household who were randomly assigned between October 18, 2013, and December 22, 2014, and were age 18 to 61 at the time of random assignment. Sample sizes for specific outcomes may vary because of missing values. Estimates were regression adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment.

No special weights were applied to responses to adjust for differences in sample size by housing authority. Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between the FSS group and the control group for continuous variables and selected outcomes expressed as proportions. In addition, a chi-square test for categorical variables was run to determine whether there is a difference in the distribution of related outcomes by research group. The p-value indicates the likelihood that the difference between the FSS group and the control group arose by chance. Statistical significance levels are indicated as * = 10

percent; ** = 5 percent; *** = 1 percent.

Sources: MDRC calculations using Experian Vantage 3.0 credit scores and Clarity Clear Early Risk credit scores.

73

Impacts on the Use of Traditional Financial Services and Sources of Credit

According to Vantage credit data covering traditional financial services (and summarized in exhibit 5.7), during the random assignment year of 2014, the typical FSS group member carried a balance of around $9,000 in nonhousing-related debt. Over the next 5 years of followup, FSS group members’

average debt level nearly doubled, reaching an average of close to $19,000 in nonhousing-related debt.

As exhibits 5.7 and 5.8 show, the largest proportion of FSS group members’ nonhousing-related debt was for installment debt, which includes student debt, auto loans, and store financing for furniture, appliances, or other large purchases. (As appendix exhibit D.2 shows, automobile loans and student debt accounted for almost all of FSS group members’ total balances for installment debt from traditional credit sources during each year of followup.)83

Exhibit 5.7: Changes in Debt Balance Amount from 2014 to 2019, by Type of Credit, Family Self-Sufficiency Group Members in the Family Self-Sufficiency Impact Sample Notes: The FSS Impact sample includes housing choice voucher heads of household who were randomly assigned between October 18, 2013, and December 22, 2014, and were age 18 to 61 at the time of random assignment. Sample sizes for specific outcomes may vary because of missing values. Estimates were regression adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment.

No special weights were applied to responses to adjust for differences in sample size by housing authority. Rounding may cause slight discrepancies in calculating sums and differences.

83 Appendix exhibit D.2 shows that average student debt increased during the followup, which suggests that some FSS

group members were self-financing their attendance at degree programs or occupational skills training.

74

Sources: MDRC calculations using Experian Vantage 3.0 scores data and Clarity Early Risk scores data Control group members also incurred increasing amounts of nonhousing-related debt from 2015

through 2019, mostly through greater levels of installment debt. In 2019, they averaged about $750

less in debt than the FSS group, although the difference was not statistically significant. By taking on higher levels of debt, FSS group members also averaged larger monthly payments over time, reaching nearly $400 per month by December 2019. As appendix exhibit D.2 shows, in 2019, the FSS program led to a statistically significant $30 (or 12-percent) increase over the control group level in payments to installment credit accounts with traditional financial services lenders. Increases above the control group in average payments were also recorded in 2016 and 2017. 84

Impacts on the Use of Alternative Financial Services and Sources of Credit

In this section, the analysis considers whether FSS group members used AFS credit, such as payday and auto title loans, less often than their counterparts in the control group. It would be expected that FSS

group members’ greater access to financial management workshops and counseling would reinforce their reluctance to take out AFS short-term, high-interest loans, which often carry substantial risks.

According to research by Clarity on recent trends in AFS use, the greater availability of online short-term installment lending—more than tripling in loan volume between 2015 and 2019—has led to a sharp increase in overall AFS borrowing nationwide. 85 AFS installment loans, including loans issued from storefront businesses, typically carry higher balances than AFS short-term, single-payment loans (such as “payday loans”) and often require repayment over an average of 7 to 12 months. Over time, the characteristics of AFS borrowers have become more diverse—including more borrowers with resources to repay loans of $2,000 or more. For example, according to Clarity, in 2019, about one-third of AFS borrowers nationwide had VantageScores in the Near Prime or Prime level (601 or higher), compared with about one-fourth 4 years earlier (Clarity Experian, 2020).

For this analysis, Experian made available Clarity data on AFS use in a series of annual snapshot files, covering December 2014 through December 2019. The 2014 data represent indicators of AFS use around the time of random assignment, whereas data from 2015 through 2019 contain AFS outcomes during the followup period. Clarity data categorize AFS loans according to whether they originate online or in a storefront business and also whether borrowers must repay the entire balance in a single payment or in installments over time. 86

84 See Verma et al. (2021). FSS group respondents to the FSS 36-Month Survey (administered in 2017) reported incurring about $14,000 in nonhousing-related debt, a somewhat smaller amount than the average for 2017 recorded by Experian. According to survey responses, average debt levels for FSS group members exceeded the control group average by about $1,600, a somewhat larger amount than recorded by Experian.

85 See Clarity Experian (2020). The report covers changes in lending patterns from 2015 through 2019, before the onset of the COVID-19 pandemic and economic downturn. A later report in this evaluation will analyze changes in credit use during 2020.

86 Some Clarity data do not include these categories and are categorized as “other” in exhibits and text. “Other” AFS

loans tend to have larger balances than installment and single-payment loans and probably include mostly installment loans.

75

As appendix exhibit D.3 shows, a fairly small proportion of FSS group members (22 percent) took out an AFS loan during the followup years (2015 through 2019). The incidence of AFS borrowing increased somewhat over time—from 7 percent of FSS group members in 2015 to 12 percent in 2019.

AFS borrowers, although a relatively small proportion of the entire FSS group, were relatively steady users of AFS credit. They averaged about one AFS loan per year and incurred about $3,000 (or $600

annually) in AFS debt during the followup period. 87

During the followup years, control group members showed an almost identical pattern of AFS use, resulting in only scattered and small impacts on measures of AFS use.

Impacts on the Use of Credit (Traditional and AFS) and Incidence of Credit

Problems

Exhibits 5.8 and 5.9 summarize FSS and control group members’ total use of credit from traditional and AFS sources and incidences of credit problems during the followup period. As discussed above and in previous reports, FSS programs often provide participants with access to financial management workshops and counseling. Along with assistance on “cleaning up” problems in credit reports, FSS

programs typically counsel participants to limit their debt and avoid high-risk lending products, such as those offered by AFS lenders. As exhibit 5.8 shows, in 2019, the last year of followup, almost no members of either research group relied solely on AFS credit, and only a modest proportion (11

percent) of members of each research group combined traditional and AFS credit use. The FSS

program led to a small (3-percentage-point) and statistically significant increase over the control group average in the use of traditional lending sources. As discussed previously, debt levels approximately doubled for both research groups during the followup years, mainly because of increases in traditional credit use (most often, for automobile and student loans).

Exhibit 5.8: Impacts on Use of Traditional and Alternative Financial Services, Family Self-Sufficiency Impact Sample

Control

Difference

Outcome

FSS Group

Group

(Impact)

P-Value

Financial service use (%)

2014

No financial services

37.0

38.5

-1.5

0.411

Traditional financial services only

59.8

58.2

1.6

0.385

Alternative financial services only

1.1

0.5

0.6 *

0.098

Both traditional and alternative

2.1

2.8

-0.7

0.250

financial services

2019

No financial services

23.8

26.9

-3.1 *

0.060

Traditional financial services only

63.9

60.6

3.3 *

0.081

Alternative financial services only

1.8

1.8

0.0

0.977

87 The calculation is $702 divided by 0.221 = $3,176.

76

Control

Difference

Outcome

FSS Group

Group

(Impact)

P-Value

Both traditional and alternative

10.5

10.7

-0.2

0.861

financial services

Total balance ($)

2014

All financial services

9,489

9,153

336

0.601

Traditional financial services

9,478

9,140

337

0.600

Revolving credit

847

750

97

0.299

Installment credit

7,919

8,112

-193

0.738

Alternative financial services

11

13

-1

0.607

Single-payment credit

10

9

1

0.644

Installment credit

0

1

-1 **

0.040

Other credit

1

3

-1

0.323

2019

All financial services

18,992

18,241

751

0.482

Traditional financial services

18,725

17,979

746

0.484

Revolving credit

1,892

1,969

-78

0.633

Installment credit

16,266

15,870

396

0.691

Alternative financial services

270

263

7

0.865

Single-payment credit

34

34

0

0.998

Installment credit

44

42

2

0.887

Other credit

192

187

5

0.884

Total monthly payment to all financial

services ($)

2014

211

206

5

0.645

2019

441

409

33

0.126

Average change in total balance

between 2014 and 2019

Change in total debt (%)

Decrease

25.2

26.1

-0.9

0.606

Little to no change

16.3

16.7

-0.4

0.767

Increase

58.5

57.1

1.3

0.505

Average change in debt-to-income ratio

between 2014 and 2019

Change in debt-to-income ratio (%)

Decrease

25.0

25.6

-0.6

0.743

Little to no change

29.4

28.9

0.4

0.815

Increase

44.9

44.3

0.6

0.787

Sample size total = 2,548

1,282

1,266

77

Notes: The FSS impact sample includes housing choice voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Sample sizes for specific outcomes may vary because of missing values. Estimates were regression adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. No specific weights were applied to responses to adjust for differences in sample size by housing authority. Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between the FSS group and the control group for continuous variables and selected outcomes expressed as proportions.

In addition, a chi-square test for categorical variables was run to determine whether there is a difference in the distribution of related outcomes by research group. The p-value indicates the likelihood that the difference between the FSS group and the control group arose by chance. Statistical significance levels are indicated as *** = 1 percent;

** = 5 percent; * = 10 percent.

Source: MDRC calculations using Experian and Clarity Early Risk scores data

Exhibit 5.9: Impacts on Incidence of Credit Problems, Family Self-Sufficiency Impact Sample FSS Control Difference

Outcome (%)

Group

Group (Impact)

P-Value

2014

Any credit problem

51.5

52.9

-1.4

0.461

High debt-to-income ratio

12.8

11.4

1.4

0.281

Traditional financial services

Total balance greater than or equal to 75 percent

38.3

37.9

0.4

0.836

of total credit

Revolving credit

14.3

14.0

0.3

0.817

Installment credit

33.8

34.3

-0.4

0.802

Any debt 90 days or more past scheduled

15.5

15.9

-0.4

0.775

repayment date

Revolving credit

3.0

4.3

-1.3 * 0.094

Installment credit

17.4

18.6

-1.2

0.421

Alternative financial services

Incurred a late payment, loan collection, or charge off

2.3

2.2

0.0

0.947

2019

Any credit problem

63.4

61.2

2.3

0.226

High debt-to-income ratio

19.3

18.8

0.5

0.738

Traditional financial services

Total balance greater than or equal to 75 percent

43.0

43.3

-0.3

0.891

of total credit

Revolving credit

20.8

21.4

-0.6

0.722

Installment credit

41.9

40.2

1.6

0.376

Any debt 90 days or more past scheduled

16.6

17.1

-0.5

0.743

repayment date

Revolving credit

3.2

3.1

0.1

0.916

Installment credit

18.4

18.1

0.3

0.844

Alternative financial services

Incurred a late payment, loan collection, or charge off

7.6

7.4

0.2

0.866

Sample size total = 2,548

1,282

1,266

78

Notes: The FSS impact sample includes housing choice voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Sample sizes for specific outcomes may vary because of missing values. Estimates were regression adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members.

No specific weights were applied to responses to adjust for differences in sample size by housing authority.

Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between the FSS group and the control group for continuous variables and selected outcomes expressed as proportions. In addition, a chi-square test for categorical variables was run to determine whether there is a difference in the distribution of related outcomes by research group. The p-value indicates the likelihood that the difference between the FSS group and the control group arose by chance. Statistical significance levels are indicated as *** = 1 percent; ** = 5 percent; * = 10 percent.

Source: MDRC calculations using Experian and Clarity credit data

Increases in debt do not necessarily portend greater financial stress. Assuming greater debt, especially for education or training or for more reliable transportation, can lead to better employment and higher income. One metric used by lenders for gauging potential borrowers’ relative financial health is called the “debt-to-income” (DTI) ratio, which compares a customer’s total monthly loan repayments to his or her total pre-tax monthly income (Folger, 2021). According to online financial advice documents from the U.S. Consumer Financial Protection Bureau (2019), Experian, and others, lenders consider a DTI ratio of under 36 percent as an indicator of having a good financial condition and a DTI ratio above 43 percent as evidence of financial distress. 88 For this analysis, study participants’ DTI ratio was estimated with Vantage and Clarity data on monthly loan payments and HUD IMS/PIC data on head-of-household income. 89 As exhibit 5.8 shows, as members of both research groups increased their debt levels, they also incurred greater uncertainty in their financial condition over time. More than 40

percent of the members of each research group increased their DTI ratio between 2014 (in the random assignment period) and 2019 (at the end of the followup period), whereas about one-fourth of FSS and control group members decreased their DTI ratio. Nonetheless, exhibits 5.9 and 5.10 show that relatively few members of each research group (about one in five) incurred a high and problematic level (above 43 percent) in this measure. 90

Credit users may also demonstrate financial stress if they keep borrowing until their total balance reaches or approaches the limit of their available credit. Financial services corporations and lenders operationalize this potential credit problem by calculating a customer’s ratio of total balance to total available credit. They often define a potential borrower as being financially at risk if he or she has a total balance that equals or exceeds 75 percent of his or her available credit. 91 Exhibit 5.9 shows a version of the balance-to-credit ratio for FSS and control group members, as measured with data from 88 See Akin (2020); Investopedia (2020); U.S. CFPB (2019). According to these sources, lenders will typically reject a mortgage application for any applicant with a DTI above 43 percent.

89 See appendix D.4 for additional details on measure construction.

90 Not shown, an additional 5 percent of FSS group members had no recorded head-of-household income in their most recent HCV eligibility reexamination, according to IMS/PIC data, making it impossible to estimate a DTI ratio. These FSS group members would also be considered to have an at-risk financial condition.

91 Experian routinely calculates this measure for several credit sources.

79

2014 and 2019. 92 As exhibit 5.9 shows, a relatively large proportion of members of each research group (about 40 percent) came close to “maxing out” their available credit during each of these years. Over time, the problem worsened somewhat for each group. By 2019, 43 percent of the members of each research group had borrowed 75 percent or more of their available credit. According to credit data, the FSS program did not affect the incidence of having a credit balance close to the maximum credit limit.

A perhaps more serious type of credit problem involves late or forgone payments for outstanding debt.

Exhibit 5.9 shows that for traditional credit sources, about one in six FSS and control group members delayed or forwent repayment of the balance they owed for 90 or more days. In addition, in 2019, about 8 percent of FSS group members and 7 percent of control group members did not repay an AFS loan or delayed repayment.

In total, about 60 percent of FSS group members experienced at least one of these types of credit problems during 2019—a similar proportion to that of the control group.

Impacts on Housing Status and Rent Subsidies

The FSS program implements a 5-year strategy to increase voucher holders’ household income through better employment and, possibly, additional household members’ employment, leading to graduation and the disbursement of escrow dollars. In the longer term (beyond graduation), FSS is intended to facilitate voucher holders’ transition to homeownership or unsubsidized housing. During the 5-year contract period, the effects of FSS on housing status and subsidies are more difficult to predict. FSS

group members should be expected to have a greater incentive than control group members to remain enrolled in the HCV program to accrue escrow credits and qualify for escrow disbursement—and possibly to maintain their access to FSS-related services and case management. Among study participants who remained in the HCV program, like the control group, FSS group members would also be expected to pay more out of pocket for rent and utilities as their earnings increase. Nevertheless, other factors, such as the lack of available low- and moderate-cost rental housing in each locality, household composition, and household income levels, likely affect study participants’ housing choices, expenditures, and subsidy receipt.

92 See appendix D.4 for additional details on measure construction.

80

Exhibit 5.10: Change in Debt-to-Income Ratio from 2014 to 2019, by Research Group, Family Self-Sufficiency Impact Sample

Notes: Debt-to-income ratio is calculated as estimated monthly payment to creditors divided by estimated monthly income expressed as a percentage. Individuals with debt-to-income ratios of 44 percent or higher or that have no income are often considered by creditors to be ineligible for additional credit.

Sources: MDRC calculations using baseline data and Experian and Clarity credit data Exhibit 5.11 displays important summary measures of both groups’ cumulative housing costs and rent subsidies during Years 1 through 5 and in Month 60 (at the end of Year 5), calculated for the impact sample from HUD administrative data. As exhibit 5.11 shows, a nearly equal proportion of FSS group and control group households (about 69 percent) remained enrolled in the HCV program in Month 60.

During Month 60, FSS group members still in the HCV program averaged slightly higher out-of-pocket expenses (of $17) for rent and utilities. Concomitantly, FSS group members averaged lower amounts of housing subsidies. Over 5 years, the difference between the research groups in total out-of-pocket (family share) housing expenses was more pronounced, with FSS group members averaging nearly 81

$1,200 (or 5 percent) above the control group total, a statistically significant difference. This impact, however, overstates the effect of FSS on housing expenses for the portion of the FSS group who receive an FSS escrow disbursement at graduation.

Exhibit 5.11: Impacts on Shelter Costs and Housing Subsidies, Years 1 Through 5 Family Self-Sufficiency Impact Sample

Difference

Outcome

FSS Group Control Group

(Impact)

P-Value

Enrolled in HCV program in Month 60 (%)

68.8

69.3

-0.5

0.778

Gross rent (contract rent + utilities allowance)

Year 1

15,134

15,027

107

0.418

Year 2

14,339

14,117

222

0.269

Year 3

13,597

13,441

156

0.539

Year 4

13,070

12,893

177

0.553

Year 5

12,640

12,587

53

0.874

Years 1–5

68,856

68,084

772

0.465

Family share

Year 1

5,252

5,134

119

0.220

Year 2

5,306

5,110

196

0.126

Year 3

5,181

4,816

365 **

0.014

Year 4

4,919

4,667

252

0.116

Year 5

4,837

4,615

222

0.204

Years 1–5

25,694

24,521

1,173 **

0.034

Housing subsidy

Total housing subsidy ($)

Year 1

9,792

9,758

34

0.780

Year 2

8,965

8,894

72

0.662

Year 3

8,342

8,533

-191

0.325

Year 4

8,270

8,204

66

0.766

Year 5

7,851

8,045

-194

0.415

Years 1–5

43,347

43,631

-284

0.724

If HCV received in Month 60 ($)

Average gross rent in Month 60

1,537

1,545

-7

Average family share in Month 60

591

573

17

Average housing subsidy in Month 60

960

979

-19

Sample size total = 2,548

1,281

1,267

Notes: The FSS impact sample includes housing choice voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Rent and subsidy calculations used data from each household’s housing choice voucher annual and interim eligibility reexaminations. Recorded amounts were copied to successive months until a new eligibility reexamination took place or the household left housing assistance. Cumulative totals for former HCV households cover their months of eligibility following random assignment. Estimates were regression adjusted using ordinary 82

least squares, controlling for pre-random assignment characteristics of sample members. For each dollar amount outcome, values above the 99th percentile were considered outliers and dropped from the calculations. As a result of this procedure, adjusted mean values for total family share and subsidies detail do not sum to total rent plus utility allowance. A two-tailed t-test was applied to differences between the FSS group and the control group for continuous variables and selected outcomes expressed as proportions. In addition, a chi-square test for categorical variables was run to determine whether there is a difference in the distribution of related outcomes by research group. The p-value indicates the likelihood that the difference between the FSS group and the control group arose by chance. Statistical significance levels are indicated as * = 10 percent; ** = 5 percent; and *** = 1 percent. Results displayed in italics are nonexperimental. No tests of statistical significance were performed on differences between research groups in means or proportions. Sample sizes for specific outcomes may vary because of missing values. Rounding may cause slight discrepancies in calculating sums and differences.

Source: U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data

Impacts on Homeownership

As discussed above and in previous reports, many FSS programs encourage participants to aim for homeownership and support prospective homeowners with financial empowerment and

homeownership preparation workshops and individual counseling. Graduates from the FSS program may use the money disbursed from their escrow account to help purchase a home (or for other purposes). Nongraduates could also conceivably save enough from their earnings or other funds to make a downpayment on a house, although, as discussed in chapter 3, FSS group members who exited the program without graduating tended to have smaller increases in their income over time than did FSS group members who graduated from the program.

According to Vantage data, as of December 2019, only 5 percent of FSS group members had taken out a mortgage (and would be assumed to own a home), compared with 4 percent of the control group. A later report will analyze trends in graduations, disbursements, and takeup of mortgages and home equity loans during an additional year of followup. Most likely, the proportion of homeowners among FSS group members will increase over time as more FSS group members graduate from the program.

It should also be recognized that the COVID-19 pandemic and associated economic downturn may thwart some FSS group members’ aspirations to buy a home by requiring them to use all or most of their escrow disbursements to pay for basic household expenses.

83

Conclusions

In the future, FSS group members’ ability to achieve greater financial well-being, compared with the control group, will largely depend on how many FSS group members graduate from the program and receive their escrow dollars. A one-time infusion of $5,000 or more can make a big difference to a household’s finances, irrespective of whether the household uses the money to make a large purchase; fund the education of a family member; reduce debt; or weather reductions in hours worked, layoffs, and furloughs during the COVID-19 pandemic. For the full impact sample, patterns of credit use and changes over time in credit scores did not provide strong evidence of the positive effects of financial workshops and individual counseling offered by FSS programs, possibly because of the relatively short-term nature of this assistance. Nonetheless, as with the absence of impacts on employment, earnings, and credit scores for the full sample, results for the full impact sample may result from a combination of positive effects for some subgroups or clusters of PHAs with similar program approaches and negative effects for others. The next chapter explores these issues. It should also be recognized that financial well-being has a psychological component. As discussed in the previous report, using responses to a 36-month survey, a larger proportion of FSS group members, compared with the control group, provided a relatively positive assessment of their financial circumstances and their ability to handle problems. The next report will explore some of these issues again on the basis of responses to a long-term survey.

84

Chapter 6. Variation in Program Impacts

Findings on program impacts for the entire impact sample, presented in chapters 3 to 5, may mask positive or negative effects for certain groups that may have had different exposure to the Family Self-Sufficiency program or may have responded differently to FSS services and financial incentives. 93 Chapter 6 first examines the FSS program’s longer-term impacts for subgroups, defined by characteristics of study participants, recorded at baseline. Next, the chapter considers whether programs with similar program implementation orientations and practices led to more positive (or negative) effects on earnings, housing subsidies, and indicators of financial security compared with other programs.

These analyses draw on housing agency program records, quarterly wage data from the National Directory of New Hires (NDNH), HUD data, and data on credit scores and credit outcomes from Experian. As in chapters 4 and 5, for each subgroup, the analysis considers whether differences in average outcomes between the FSS group and the control group are statistically significant at the 10-percent level or less. In addition, the exhibits in chapter 6 present the results of an H-statistic, which show the likelihood that the observed variation in program effects among related subgroups is statistically significant and did not occur by chance.

For subgroups defined by individual baseline characteristics, the main findings are:

• There are isolated differences in impacts by subgroup on measures of employment and earnings and financial well-being, but there is no consistent pattern of variation to suggest that the program is more effective for selected subgroups of respondents.

• The most consistent pattern of positive effects on employment and earnings outcomes was recorded for FSS group members with a 2-year college degree or higher at random assignment, but each of these differences with the control group was above the 10-percent level of statistical significance.

• FSS group members in three subgroups (2-year college graduates or higher, working part-time hours at random assignment, and having no recorded income at random assignment)

experienced statistically significant increases above the control group on one or more outcomes based on credit scores or use of credit.

• On average, at the end of Year 5, FSS group members in most subgroups were paying a larger share of their housing costs out of pocket and received smaller subsidies than their counterparts in the control group.

For FSS program clusters with similar implementation features, the main findings are:

• With one exception, there are isolated differences in impacts when study participants are grouped into “site clusters” on the basis of similar implementation features for FSS programs, but there is no consistent pattern of variation.

93 Throughout this chapter, the terms housing agency, public housing agency ( PHA), site, and FSS program will be used interchangeably.

85

• FSS programs with the strongest emphasis on financial services and goal attainment did not lead to consistently better outcomes for FSS group members, compared with the control group, on outcomes based on credit scores or use of credit.

• Over 5 years of followup, FSS programs with a strong emphasis on monitoring and engagement averaged earnings decreases relative to the control group of about $18,000 (or 17 percent) per study participant. By Year 5, however, the differences in average earnings between research groups were smaller (10 percent) and no longer statistically significant. In addition, by the end of followup, FSS group members in programs with the strongest emphasis on monitoring and engagement had, on average, received more financial benefits than had control group members from a combination of higher average housing subsidies over time and average total disbursements from their escrow accounts.

Program Impacts by Subgroups: Who Benefits in Years 3 Through 5?

The FSS program offers three potential financial benefits to participants, compared with what they would likely realize on their own initiative: (1) greater access to employment preparation services and education and training, which could lead to gains in employment, employment stability, and earnings; (2) greater access to financial management workshops and counseling, which could lead to increased savings, lower debt, and increased ability to weather financial shocks; and (3) an escrow account, which could lead to the receipt of a one-time disbursement worth thousands of dollars at graduation (and possibly employment and earnings gains in the years before FSS graduation). As discussed in previous reports, FSS group members tend to engage in FSS services during the first 2 years of followup; they then either settle into ongoing employment as their primary means of pursuing their FSS goals or they exit the program.

During the early years of followup, FSS group members’ use of services (particularly longer-term education and training) could carry opportunity costs (for example, from cutting back on employment to attend postsecondary education or occupational skills training) or actual costs from self-financing participation in these activities through student loans or other means. These patterns of service use and employment raise the possibility that FSS program effects would change from negative or near zero to positive during the months leading up to graduation, when FSS group members would most likely be employed, and continue through the time when FSS graduates receive and begin to use their escrow disbursement. Furthermore, it would be reasonable to expect that subgroups, such as FSS group members with a 2-year college degree or higher at random assignment, with the greatest likelihood of graduating and receiving an escrow disbursement, would also have the best chance of experiencing other positive effects by the end of the followup period. These financial gains could come from better employment and earnings outcomes compared with the control group, from better credit-related outcomes, or from the net value of the escrow disbursement. For these reasons, the analysis will focus on Year 5 outcomes or the last month of followup for this report. 94

94 The term net value is used because recipients of an escrow disbursement may receive lower housing subsidies (because of higher earnings) while accruing escrow.

86

Subgroups Based on Participant Baseline Characteristics

Exhibits 6.1A through 6.1E and appendix exhibits E.1 through E.5 show FSS program impacts on selected outcomes for subgroups based on their characteristics around the time of random assignment: employment status, highest educational attainment, estimated annual household income, receipt of Supplemental Security Income or Social Security Disability Insurance (SSI/SSDI) benefits, reported barriers to employment, and percentage of out-of-pocket rent and utilities expenses during the month of random assignment. 95 Of importance to the FSS evaluation is examining whether the FSS program’s combination of services and financial incentives leads to increases above the control group more consistently among subgroups with greater disadvantages in the labor market or subgroups with fewer disadvantages. 96 Each exhibit illustrates the impacts for selected outcomes for a particular subgroup.

Employment and Earnings

For employment and earnings measures from NDNH data, mean values for FSS and control group members varied substantially by subgroup. However, the FSS program led to only small and not statistically significant differences (or impacts) on these measures when average values for each research group are compared. For example, in any given quarter during Years 1 through 5, about 82

percent of control group members who reported working full-time hours at baseline were employed, according to NDNH quarterly wage data—nearly twice the rate for control group members with no reported employment at baseline (see appendix exhibit E.1). Over 5 years, FSS and control group members recorded nearly identical average quarterly employment rates for all employment subgroups.

Similarly, control group members with full-time employment at baseline averaged almost $125,000 in total earnings over 5 years, more than three times the average for control group members with no employment at baseline. Moreover, the proportion of control group members in the full-time employment subgroup who averaged more than $25,000 per year in earnings (49 percent) was nearly six times larger than the proportion for control group members with no employment at baseline. For 95 Employment status, highest educational attainment, having a barrier to employment, and SSI/SSDI benefits receipt are self-reported; estimated household income and out-of-pocket rent and utilities expenses were collected from HUD

administrative data. Appendix exhibits E.1 through E.5 include regression-adjusted mean values for FSS and control group members, as well as FSS control group differences (impacts) in outcomes. These tables also include results for two additional subgroups: having or not having at least one barrier to employment (self-reported) and percentage of household income used to pay for rent and utilities (family share, from HUD administrative data). This subgroup analysis should be considered as exploratory. The final report will present a more formal analysis that will include outcome data covering additional years of followup and a machine learning framework that includes cross-validation to help reduce the number of tests and help validate any patterns seen.

96 The most disadvantaged subgroups include those who are not employed at study entry, have no educational degree or credential, have no annual household income or income up to $10,000, reported one or more barriers to employment, are receiving SSI or SSDI benefits, and are paying up to 25 percent of their housing expenses out of pocket (the last measure is related to the amount of household income). The moderately disadvantaged subgroups include those who are employed part time, have a high school diploma or equivalent or some college, have an annual household income of $10,001 to $20,000, and are paying between 25.01 and 50.00 percent of their housing expenses out of pocket. The least disadvantaged subgroups include those who are employed full time, have a 2-year college degree or higher, have an annual household income of more than $20,000, and are paying more than 50 percent of their household expenses out of pocket. The remaining subgroups may be considered “mixed,” meaning that they include study participants with a broad range of characteristics that could affect their future employment or earnings (for example “not receiving SSI/SSDI” and “not having a barrier to employment”).

87

all of these measures, averages for the FSS group are slightly smaller, but the differences are not statistically significant.

Exhibit 6.1A: Impacts on Selected Outcomes by Employment Status at Random Assignment 6.0

$2,000

4.8

$1,500

4.0

$1,134

2.6**

$1,000

)

2.0

$733

1.8

2.0

1.5

$553

$346

0.6

$500

$172

$121

$129

oints (%e p 0.0

$0

tag

Total

-0.5

-0.4

earnings in

Any credit

-$500

cen -2.0 -1.3

Average credit

Year 5

-$490

Per

Average

score in 2019

problem in

quarterly

2019

Total credit

-$1,000

employment

balance in

-4.0

rate in Year

2019

-$1,278

-$1,500

5

Total HAP in Years 1 to 5

-6.0

Not employed

Employed part-time

Employed full-time

-$2,000

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Estimates were regression-adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment. No special weights were applied to responses to adjust for differences in sample size by housing authority. Average quarterly employment rate is calculated as total quarters with employment divided by total quarters of followup, expressed as a percentage. The average credit score is expressed in “percentage change” format and calculated as (impact on average credit score / control group average) x 100. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; *** = 1 percent. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups. Statistical significance levels are indicated as: † = 10 percent; †† = 5 percent; and ††† = 1 percent.

Sources: MDRC calculations from baseline data, U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data, quarterly wage data from the National Directory of New Hires, and Experian and Clarity credit data.

88

Exhibit 6.1B: Impacts on Selected Outcome by Educational Attainment at Random Assignment 5.0

4.7

$3,000

$2,503

4.0

3.5

3.2 **

$1,948

$2,000

3.0

2.42.3

$1,329

2.0

$911

)

1.3

$1,000

1.0

$710$529

1.0

$257

oints (%

0.0

-$329

e p 0.0

$0

tag

$275

cen -1.0

-0.7

-0.5

-$389

Total credit

Per

Average

-1.1

quarterly

-1.4

Total earnings

-$834

balance in

-$1,000

-2.0

Any credit

employme

Average credit problem in 2019

in Year 5

nt in Year

score in 2019 †

-3.0

5

-$2,000

No degree

Some college

-4.0

High school degree/GED

2-year degree or higher

-$2,566

-5.0

Total HAP in Years 1 to 5

-$3,000

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Estimates were regression-adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment. No special weights were applied to responses to adjust for differences in sample size by housing authority. Average quarterly employment rate is calculated as total quarters with employment divided by total quarters of followup, expressed as a percentage. The average credit score is expressed in “percentage change” format and calculated as (impact on average credit score / control group average) x 100. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; *** = 1 percent. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups. Statistical significance levels are indicated as: † = 10 percent; †† = 5 percent; and ††† = 1 percent.

Sources: MDRC calculations from baseline data, U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data, quarterly wage data from the National Directory of New Hires, and Experian and Clarity credit data.

89

Exhibit 6.1C: Impacts on Selected Outcomes by Income at Random Assignment

10.0

$2,500

8.0**

$1,945

8.0

$2,000

$1,384

6.0

$1,148

$1,500

) 4.0

2.8

$1,000

2.2

2.21.9

2.0

$315

0.7

$309

$500

0.5

0.7

oints (%e p 0.0

$0

tag

Average

-0.5-0.4

-0.3

Total

-$188

-2.0

quarterly

Any credit

earnings in

-$290

-$237

-$500

cen

employment Average

problem in

Year 5

Per -4.0 rate in Year credit score

2019

-$832

-$1,000

5

in 2019

-$995

-$974

-6.0

Total HAP in

Total credit

-$1,500

Years 1 to 5

balance in

-8.0

2019

-$2,000

-10.0

No income

$1 to $10,000

$10,001 to $20,000

Greater than $20,000

-$2,500

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Estimates were regression-adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment. No special weights were applied to responses to adjust for differences in sample size by housing authority. Average quarterly employment rate is calculated as total quarters with employment divided by total quarters of followup, expressed as a percentage. The average credit score is expressed in “percentage change” format and calculated as (impact on average credit score / control group average) x 100. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; *** = 1 percent. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups. Statistical significance levels are indicated as: † = 10 percent; †† = 5 percent; and ††† = 1 percent.

Sources: MDRC calculations from baseline data, U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data, quarterly wage data from the National Directory of New Hires, and Experian and Clarity credit data.

90

Exhibit 6.1D: Impacts on Selected Outcomes by Disability Status at Random Assignment 5.0

$3,538 $4,000

3.9

4.0

$3,000

3.0

2.5

$2,000

) 2.0

1.5

1.0 0.5

$1,000

0.3

$153

oints (%

$35

$159

e p 0.0

$0

Average credit

Any credit

Total earnings in

-$11

tag -1.0

score in 2019

problem in

Year 5

Total

-$1,000

cen

credit

Per -2.0

2019

-$1,356

balance in

-2.1

-$2,000

-3.0

2019

Average

Total HAP in

quarterly

Years 1 to 5

-$3,000

-4.0

employment

rate in Year

-5.0

Did not receive SSI/SSDI

Received SSI/SSDI

-$4,000

5

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Estimates were regression-adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment. No special weights were applied to responses to adjust for differences in sample size by housing authority. Average quarterly employment rate is calculated as total quarters with employment divided by total quarters of followup, expressed as a percentage. The average credit score is expressed in “percentage change” format and calculated as (impact on average credit score / control group average) x 100. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; *** = 1 percent. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups. Statistical significance levels are indicated as: † = 10 percent; †† = 5 percent; and ††† = 1 percent.

Sources: MDRC calculations from baseline data, U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data, quarterly wage data from the National Directory of New Hires, and Experian and Clarity credit data.

91

Exhibit 6.1E: Impacts on Selected Outcomes by Percentage of Rent and Utilities Expenses Paid by Household at Random Assignment

9.0

$2,500

Notes: The FSS impact sample includes Housing Choi

7.9ce V

** oucher households that were randomly assigne

$2, d be

065 tween

Octobe

7 r

.0 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of rando $ m

2,

000

assignment. Estimates were regression-adjusted using ordinary least squares, controlling for differences in sample $1,500

mem

) b5er

.0 characteristics recorded at the time of random assignment. No special weights were ap $899 plied to responses to

adjust f

3 or

.0 differences in sample size by housing authority. Average quarterl

$1,000

$557y employment rate is calculated as total

quarters with e1m

.3 pl1oy

.4 ment divided by t

0.9 otal qua

1 r

.1 te

1 rs

.4 of follow-up, expressed as a percentage. Statistical significance

0.5

$277$303

$500

levoints (%els ar

1.0 e indicated as: * = 10 p

0 er

.1 cent; ** = 5 percent; *** = 1 percent. The H-statistic test was used to test for e p

$0

statis-tic

1. a

0 lly

- s

0 ig

.1 nificant differences in impact es

A tim

ny a

cr te

edsi a

t cross differen

T t s

otaul bgroups. Statistical significance le

$1, ve

059 ls are

tag

Average

-$62

indicated as: † = 10 percent; †† =

credit 5 pe

score rcent;

pr and †

oblem †† =

i

1 p

n 2019 ercent. a

eaT

r h

ni e i

ng m

s i p

n act is expressed in “percentage chan-ge”

$5

00

cen

Average

forma

- t a

3.0 nd calqcuul

ar a

t t

ereld a

y s (impaict on a

n 2019 verage credit score / control grou

Y p a

ear ve

5 rage) x 100.

-$1,000

Per

employment

-$941

-5.0

-$1,137

rate in Year 5

Total HAP in

-$1,500

Years 1 to 5

Total credit

-7.0

balance in

-$2,000

0 - 25

25.01 - 50

More than 50

2019

-9.0

-$2,500

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Estimates were regression-adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment. No special weights were applied to responses to adjust for differences in sample size by housing authority. Average quarterly employment rate is calculated as total quarters with employment divided by total quarters of followup, expressed as a percentage. The average credit score is expressed in “percentage change” format and calculated as (impact on average credit score / control group average) x 100. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; *** = 1 percent. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups. Statistical significance levels are indicated as: † = 10 percent; †† = 5 percent; and ††† = 1 percent.

Sources: MDRC calculations from baseline data, U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data, quarterly wage data from the National Directory of New Hires, and Experian and Clarity credit data.

These patterns of impacts are repeated for the other subgroups. Results for Year 5 show similar patterns of average values for employment subgroup members and little or no difference in averages by research group.

One possible exception to these patterns is study participants who entered the evaluation with a 2-year college degree or higher (as discussed in chapter 3, FSS group members in this subgroup averaged the highest graduation rate and received the highest average disbursement amount). FSS group members with a postsecondary degree showed greater employment stability and higher earnings in Year 5, compared with the control group, but the differences were slightly above the 10-percent level of statistical significance, probably because of the small number of study participants in this group.

92

Credit Scores and Use of Credit

Exhibits 6.1A through 6.1E and appendix exhibit E.2 summarize the variation by subgroup in the FSS

program’s effects on credit scores. Once again, FSS group members with a 2-year college degree or higher experience the most consistently positive effects. In 2019, at the end of the followup period, FSS group members with a postsecondary degree averaged a higher VantageScore than the control group by 19 points (about 3 percent) and a higher average improvement in their credit scores since random assignment compared with the control group—a difference of 14 points. As of 2019, FSS

group members in the graduate subgroup were the only subgroup members with an average score that placed them collectively in the Near Subprime rank. That year, the graduate subgroup also had the largest proportion of FSS group members with Prime VantageScores (of 661 or higher), 26 percent, which exceeded the control group mean by about 7 percentage points.

FSS group members with part-time employment at random assignment also experienced positive effects on credit scores compared with the control group. In 2019, they received an average VantageScore that exceeded the control group average by 15 points (3 percent). They also had a larger proportion of research group members in the Prime rank by 8 percentage points above the control group level.

A third subgroup that recorded gains above the control group on some credit score measures was FSS

group members with no recorded earnings at random assignment. In 2019, control group members in that subgroup had the lowest average credit score (544) and the smallest proportion of research group members in the Prime rank (6 percent) among all subgroups. FSS group members with no recorded income exceeded these averages for the control group, although the difference between research groups in average credit scores was not statistically significant (p=0.155). 97 Finally, as appendix exhibit E.2

shows, a small proportion of FSS group members with no recorded income at random assignment (11

percent) entered the evaluation with a Subprime VantageScore and increased their score to Prime by the end of the followup period. In contrast, almost no control group member in the no-income subgroup improved his or her score as much.

Appendix exhibit E.2 also shows some scattered negative effects on credit scores. One noteworthy finding concerns members of the highest income subgroup at random assignment. By a margin of 9

percentage points, a smaller proportion of FSS group members had only a VantageScore, an indirect indicator of forgoing use of high-interest alternative financial services. 98

As exhibits 6.1A to 6.1E and appendix exhibit E.3 show, subgroups differed considerably in average total debt from traditional and AFS lenders that study participants were carrying at the end of followup.

Variation in debt levels was especially large among the educational attainment subgroups, a difference 97 There is considerable overlap in membership in the no-income subgroup and the subgroup that paid out of pocket for 25 percent or less of total housing costs at random assignment. Not surprisingly, FSS group members in the latter subgroup recorded a small (4-percentage-point) increase above the control group level in the incidence of having a Prime VantageScore in 2019.

98 Study participants in the highest income subgroup also tended to pay out-of-pocket for more than one-half of their total housing expenses. For that reason, a negative effect of similar magnitude was recorded for the FSS group for this outcome.

93

of more than $30,000 between study participants with no educational credentials and study participants with a 2-year college degree or higher. More generally, as appendix exhibit E.3 shows, members of less disadvantaged subgroups tended to accumulate higher levels of debt than study participants with lower incomes or greater barriers to employment. For some subgroups, the FSS control group difference in average total debt appears to be relatively large. For example, among recipients of SSI or SSDI benefits, FSS group members averaged more than $3,500 in total debt above the control group average. Nevertheless, none of the differences between research groups in debt levels are statistically significant.

Surprisingly, subgroups included in this analysis showed relatively similar patterns of borrowing from AFS lenders. For example, among control group members, the incidence of ever incurring AFS debt during followup ranged from 18 percent (among control group members with no educational credentials and control group members with no recorded income at random assignment) to 27 percent (among control members with full-time employment). FSS group members showed a relatively similar pattern of AFS borrowing, leading to no statistically significant differences in the incidence of AFS

borrowing.

Next, the analysis considers whether the incidence of credit problems varied by subgroup and, more importantly, whether, for certain subgroups, the FSS program reduced the likelihood of incurring a credit problem below the level for the control group. As discussed in the previous chapter, having higher debt levels does not necessarily lead to a greater incidence of credit problems. For the FSS

Impact Sample, however, this appears to be the case. For example, control group members with full-time employment at random assignment averaged more than one and one-half times the amount of debt of control group members with no employment ($23,000 compared with $14,000; see appendix exhibit E.3). In addition, about two-thirds of control group members with full-time employment at random assignment incurred at least one credit problem, more than 10 percentage points higher than control group members without employment. Among control group members, every subgroup except one (no degree or educational credentials) had a majority of members with at least one recorded credit problem in 2019. That year, among all subgroups, the largest proportion of control group members with at least one credit problem had a credit balance close to the maximum amount of available credit.

FSS group members in two subgroups, those recorded as having more than $20,000 in income at random assignment and those paying more than one-half of their expenses for rent and utilities out of pocket at random assignment, had a statistically significant difference in the likelihood of experiencing a credit problem at the end of the followup period. In both instances, the incidence for FSS group members exceeded the control group level by 8 percentage points.

Variation in Housing Outcomes

Appendix exhibit E.5 displays research group means and differences (impacts) for rent and housing subsidy outcomes for subgroups defined with baseline data. Several outcomes presented in the table are nonexperimental because they exclude FSS and control group members who exited the HCV

program before the end of Year 5. Among control group members, enrollment levels in HCV

(experimental comparison) show a fair amount of variation, ranging from 54 percent (for control group members who paid more than one-half of their housing expenses out of pocket) to about three-fourths 94

(for multiple subgroups). Among most subgroups, a similar proportion of FSS group members continued their enrollment in the HCV program, and the differences are not statistically significant.

A relatively consistent pattern appears when comparing rent and subsidy outcomes for ongoing HCV

enrollees (nonexperimental). On average, at the end of Year 5, FSS group members in most subgroups were paying a larger share of their housing costs out of pocket and received smaller subsidies than their counterparts in the control group. These nonexperimental differences (higher expenditures and lower subsidies) between research groups were particularly large among study participants with a 2-year college degree or higher at random assignment.

The analysis of impacts on housing expenditures and subsidies also compares cumulative totals over 5

years for each research group. These comparisons are experimental, in which study participants receive zeroes for months when they are no longer enrolled in HCV. As appendix exhibit E.5 shows, over 5

years of followup, FSS group members in most subgroups averaged higher total housing expenses that they paid out of pocket, compared with the control group, although the difference was statistically significant for only three subgroups (FSS group members with some college credits or with a 2-year degree or higher at random assignment and FSS group members who did not receive SSI or SSDI disability benefits). For most subgroups, FSS group members also averaged less in total housing subsidies compared with the control group, although none of these differences were statistically significant. In contrast, escrow disbursements, most of which were or will be paid during Year 6 or later, may counteract and supersede reductions in housing subsidies from higher earnings over time.

The magnitude of the financial effect of escrow disbursements varies by subgroup, as discussed in chapter 3, but exceeded $2,000 per FSS group member among those with a 2-year degree or higher at random assignment.

Subgroups Based on Program Implementation Features

This section analyzes whether the impacts of FSS on employment, earnings, and other outcomes varied by implementation features adopted by different housing agencies. As a first step, the analysis tests whether impacts on important outcomes differed by PHA for any reason. To address this issue, the authors ran statistical tests for 5-year employment and earnings outcomes using quarterly wage data from NDNH to determine whether any differences in impacts by PHA could be found and whether this variation was unlikely to have occurred by chance. Test results showed that impacts varied by PHA overall, although sample sizes for most housing agencies are too small to reliably estimate the magnitude of site-specific impacts. Nonetheless, results from these initial statistical tests and results of additional testing of impacts by PHA that control for sample member characteristics strongly suggest that in the first 5 years of followup, some PHAs have positive impacts on employment and earnings, some PHAs have effects close to zero, and some PHAs have negative impacts. 99 The final report will draw on the complete time series available for this evaluation (6 to 7 years of data) to explore this pattern and examine the factors driving the cross-site variation in impacts and draw more conclusive inferences.

99 See Bloom et al. (2017) for a description of the statistical test. Test results showed that variation in the overall pattern of impacts by PHA is statistically significant at the 1-percent level, meaning that it was unlikely that all PHAs had the same impacts—negative, positive, or close to zero.

95

Next, the analysis considers whether PHAs with similar implementation orientation and practices have impacts of a similar magnitude and direction (see exhibit 6.2 for a description of the program clusters examined in chapter 6). Housing agencies may vary in impacts for reasons unrelated to differences in program implementation. For example, variation in impacts by PHA could result from differences in the characteristics of the research samples—as when certain PHAs have an unusually large proportion of subgroups who experienced positive or negative impacts on crucial outcomes. In theory, this caveat could apply to FSS, although, as discussed previously, the analysis based on 5 years of followup has not identified subgroups with consistently positive or negative impacts. 100 PHAs may also vary in impacts because of differences in local labor or housing markets that may affect employment opportunities or employment choices for FSS and control group members in unique ways (this issue will be explored in the final report and with additional followup). Grouping together PHAs on the basis of their implementation features, however, helps alleviate this measurement issue by combining participants from different regions of the United States; from small-, medium-, and large-sized cities and from suburban areas; and from high-growth and low-growth labor and housing markets.

Program Clusters Based on Program Focus

PHAs could vary in impacts because of differences in how FSS administrators and case managers work with participants to set their individual goals and training plans and whether these individual self-sufficiency road maps emphasize work, education, and training or focus on other aspects of financial security and management. For this analysis, the FSS programs are grouped into site clusters based on information recorded by case managers in FSS group members’ Individual Training and Services Plans (ITSPs) soon after random assignment (for most, recorded the same day as or shortly after enrollment), interviews with FSS program administrators and case managers, and documents collected on site.

Exhibits 6.3A through 6.3C and appendix exhibits E.6 through E.10 display the impacts on employment, earnings, and other outcomes for site clusters on the basis of the relative emphasis of each FSS program’s (1) job search and postemployment services focus, (2) education and training focus, and (3) financial services focus.

Exhibit 6.2: Program Clusters: Data and Definitions

Site Cluster Measure

Source Data

Component Measure(s) per PHA

Program Emphasis and Orientation

Emphasis on job search Individual Training and

Proportion of initial ITSPs that listed participation in

and postemployment

Services Plans (ITSPs)

job search, self-employment preparation, or

services

postemployment services as a goal or service.

Emphasis on education Individual Training and

Proportion of initial ITSPs that listed participation in

and training

Services Plans (ITSPs)

education or training as a goal or service.

100 Results from additional tests using conditional impacts and with additional covariates that record the interaction between subgroup characteristics and membership in the FSS group suggest that variations in impacts by PHA are only partly explained by variations in the baseline characteristics of each PHA. An additional (informal) test involving the creation of site clusters of PHAs suggested that impacts could be positive for one or more subgroups in one cluster of PHAs and negative in another.

96

Site Cluster Measure

Source Data

Component Measure(s) per PHA

Program Emphasis and Orientation

Emphasis on financial

Individual Training and

Proportion of initial ITSPs that listed participation in

services

Services Plans (ITSPs)

financial counseling or workshops as a goal or service.

Emphasis on monitoring Interviews with FSS

1. Average FSS caseload size.

and engagement

administrators and case

2. Expected number of communications with an FSS

managers; Individual Training

case manager per year.

and Services Plans (ITSPs)

3. Proportion of ITSPs that included at least one goal

to be completed in Year 1.

FSS = Family Self-Sufficiency. PHA = public housing agency.

Exhibits 6.3A, 6.3B, and 6.3C and appendix exhibit E.6 show the impacts of having a low, medium, or high emphasis on each type of program service. 101 For each test, the FSS program’s impacts were estimated on average quarterly employment and average total earnings over 5 years and also, in Year 5, as measured with NDNH quarterly wage data. These exhibits show only scattered effects on quarterly employment or earnings with statistical significance: an increase in average total earnings over 5 years for PHAs with medium emphasis on provision of job search and postemployment services and an increase in employment stability in Year 5 for PHAs with the least emphasis on education and training.

Exhibits 6.3A through 6.3D and appendix exhibits E.7 through E.9 continue the analysis of the variation of FSS program effects by service approach—this time on outcomes related to credit scores, credit use, and credit problems. Of particular interest are the results for PHAs with the strongest emphasis on providing financial services and attaining financial goals, as these interventions would, at least in theory, be expected to improve credit-related outcomes. So far, however, the results show only a modest association between program approach and the incidence of positive effects on credit-related outcomes.

101 Results for the “medium” clusters of PHAs may be the most difficult to interpret because PHAs could have a

“medium” level of emphasis for different reasons—for example, because they target services to certain subgroups or because they give participants more leeway to define their goals and choose their services.

97

Exhibit 6.3A: Impacts on Selected Outcomes by Program Emphasis on Job Search and Postemployment Services

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Estimates were regression-adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment. No special weights were applied to responses to adjust for differences in sample size by housing authority. Average quarterly employment rate is calculated as total quarters with employment divided by total quarters of followup, expressed as a percentage. The average credit score is expressed in “percentage change” format and calculated as (impact on average credit score / control group average) x 100. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; *** = 1 percent. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups. Statistical significance levels are indicated as: † = 10 percent; †† = 5 percent; and ††† = 1 percent.

Sources: MDRC calculations from baseline data, U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data, quarterly wage data from the National Directory of New Hires, and Experian and Clarity credit data.

98

Exhibit 6.3B: Impacts on Selected Outcomes by Program Emphasis on Education and Training Services

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Estimates were regression-adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment. No special weights were applied to responses to adjust for differences in sample size by housing authority. Average quarterly employment rate is calculated as total quarters with employment divided by total quarters of followup, expressed as a percentage. The average credit score is expressed in “percentage change” format and calculated as (impact on average credit score / control group average) x 100. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; *** = 1 percent. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups. Statistical significance levels are indicated as: † = 10 percent; †† = 5 percent; and ††† = 1 percent.

Sources: MDRC calculations from baseline data, U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data, quarterly wage data from the National Directory of New Hires, and Experian and Clarity credit data.

99

Exhibit 6.3C: Impacts on Selected Outcomes by Program Emphasis on Financial Services Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Estimates were regression-adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment. No special weights were applied to responses to adjust for differences in sample size by housing authority. Average quarterly employment rate is calculated as total quarters with employment divided by total quarters of followup, expressed as a percentage. The average credit score is expressed in “percentage change” format and calculated as (impact on average credit score / control group average) x 100. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; *** = 1 percent. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups. Statistical significance levels are indicated as: † = 10 percent; †† = 5 percent; and ††† = 1 percent.

Sources: MDRC calculations from baseline data, U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data, quarterly wage data from the National Directory of New Hires, and Experian and Clarity credit data.

100

Exhibit 6.3D: Impacts on Selected Outcomes by Program Emphasis on Monitoring and Engagement

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Estimates were regression-adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment. No special weights were applied to responses to adjust for differences in sample size by housing authority. Average quarterly employment rate is calculated as total quarters with employment divided by total quarters of follow-up, expressed as a percentage. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; *** = 1 percent. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups. Statistical significance levels are indicated as: † = 10 percent; †† = 5 percent; and ††† = 1 percent. The impact is expressed in “percentage change”

format and calculated as (impact on average credit score / control group average) x 100.

Source: MDRC calculations from baseline data, U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data, quarterly wage data from the National Directory of New Hires, and Experian and Clarity credit data.

For example, FSS programs with the strongest emphasis on financial services and goal attainment did not increase average credit scores above the control group level and did not affect total debt or the amount of increase in debt levels over time. More positively, the absence of adverse effects on credit outcomes contrasts with results for FSS programs with medium emphasis on financial services and goals. For example, FSS group members in PHAs with medium emphasis accumulated, on average, more debt than did control group members and were also more likely to experience at least one credit problem. However, FSS group members in PHAs with the least emphasis on financial services and goals also fared better compared with their counterparts in the medium emphasis PHAs.

Variation in program approach appears to be only marginally related to differences by research group in housing-related outcomes. Over 5 years, in two clusters of PHAs (FSS programs with high emphasis on financial services and programs with medium emphasis on job search and postemployment services), the FSS group average exceeded by a statistically significant amount the average for the control group in total housing costs paid out of pocket, and no statistically significant differences were found for total housing subsidies. Finally, for reasons that are unclear, a larger proportion of FSS group 101

members in programs with a high emphasis on education and training had left HCV assistance by the end of Year 5 than their counterparts in the control group.

Program Clusters Based on Site Monitoring and Engagement Practices

Another test of variation in program implementation features, also presented in Verma et al. (2019; 2021), concerns the measure of how strongly administrators and case managers in PHAs emphasize monitoring and engagement with FSS group members. As discussed in previous chapters, the data used to group PHAs into low-emphasis, medium-emphasis, and high-emphasis site clusters are based on the housing agencies’ FSS program implementation features and practices at study launch—caseload sizes, expectations about the frequency of contacts, and focus on establishing short-term goals. Sites classified as having a high or strong emphasis on monitoring and engagement have smaller caseload sizes, expect FSS group members and case managers to have more frequent contacts, and focus on establishing short-term goals. As discussed in previous reports, PHAs that ranked high on this measure tended to have relatively high participation rates for the FSS group, although differences in participation with the control group were not especially large. Moreover, as discussed previously (see exhibit 3.5), housing agencies with stronger emphasis on monitoring and engagement—especially those with relatively small caseload sizes—tended to have higher graduation rates after 5 years of followup compared with other housing agencies. However, some previous studies that analyzed links between program implementation practices and impacts on employment and earnings found that programs that ranked high on the studies’ version of a monitoring and engagement indicator did not lead to statistically significant increases in employment and earnings (Bloom, Hill, and Riccio, 2001: 40–42).

Appendix exhibit E.6 shows impact results on employment and earnings outcomes for PHAs in the monitoring and engagement clusters. Between 61 percent and 65 percent of control group members were employed in any given quarter during Years 1 through 5. Average total earnings for control group members ranged from about $71,000 to $88,000, with the highest average earned by control group members in housing agencies with the strongest emphasis on monitoring and engagement. This variation in earnings is also reflected in the differences among the clusters of housing agencies in the proportion of control group members who were employed during every quarter of followup and the proportion of control group members who averaged more than $25,000 per year in earnings.

Over 5 years, for outcomes calculated with NDNH quarterly wage data, the FSS program did not affect employment or earnings levels in housing agencies with a low or medium emphasis on monitoring and engagement. In addition, on average, FSS group members in all three types of programs earned about the same amount (about $75,000) during Years 1 through 5. FSS programs with the strongest emphasis on monitoring and engagement, however, led to a relatively large and statistically significant decrease in total earnings (of nearly $13,000, or about 17 percent) compared with the control group. Similarly, by a margin of about 6 percentage points, a larger proportion of control group members than FSS group members earned more than $25,000 per year. Moreover, nearly one-third of control group members worked for pay during all 20 quarters of followup, compared with about 20 percent of FSS group members. Not shown in any exhibit, the FSS programs with a high emphasis on monitoring and 102

engagement led to an increase of similar magnitude in the proportion of study participants with low annual earnings (up to $10,000).

This pattern of earnings differences suggests that some FSS group members were working part time or working intermittently, whereas control group members were more likely to work full-time hours or in jobs with more weeks or months of employment. This pattern is consistent with a finding for programs having significant upfront opportunity costs, in which participants delay or forgo employment or cut back on employment to facilitate their service use. To be successful, such programs must eventually help participants find employment at relatively well-paying jobs. If so, over time, participants’ earnings make up for their initial opportunity costs. As of Year 5, the pattern of impacts on employment and earnings suggests that FSS group members in programs with a strong emphasis on monitoring and engagement may be approaching the turning point when effects are no longer negative relative to the control group.

As appendix exhibit E.6 shows, during Year 5, the differences between research groups in average quarterly employment, average total earnings, and incidence of employment during all four quarters were smaller than in previous years and no longer statistically significant.

Additional findings based on credit data show that FSS programs with a strong emphasis on monitoring and engagement realized gains relative to the control group. In 2019, the last year of followup, FSS group members in high-emphasis PHAs had VantageScores that exceeded the scores for control group members by an average of 16 points. FSS group members in these PHAs also increased their scores over time by an average of 30 points, compared with a 10-point increase for control group members. FSS group members in programs with low or medium emphasis on monitoring and engagement did not realize gains in credit scores relative to control group members. For the most part, FSS programs with a strong emphasis on monitoring and engagement did not lead to better or worse outcomes in measures of use of credit or incidence of credit problems relative to the control group more often than FSS programs with less emphasis. One exception to this finding was for the measure of having any type of credit problem in 2019. For this outcome, FSS group members in the low-emphasis programs recorded a modest increase (of 6 percentage points) in the incidence of having at least one credit problem, whereas no difference between the research groups was found for the medium- and high-emphasis programs (see appendix exhibit E.9). Results for debt-to-income ratio (a component of the measure of any credit problem) were less positive for housing agencies with a strong emphasis on monitoring and engagement, with the FSS

group recording a 6-percentage-point increase above the control group level in the incidence of having a high level of debt repayments relative to monthly income.

FSS group members from programs with a strong emphasis on monitoring and engagement appear to have fared relatively well compared with the control group in measures of out-of-pocket housing expenditures and receipt of housing subsidies—or subsidies plus the value of escrow disbursements.

As appendix exhibit E.10 shows, FSS group members in the housing agencies with low or medium emphasis paid an average of more than $1,000 per household in housing expenses above the control group during Years 1 through 5, whereas both research groups in the high-emphasis PHAs paid about the same amount. 102 Moreover, FSS group members in PHAs with a strong emphasis on monitoring 102 The difference between research groups for PHAs with medium emphasis on monitoring and engagement was just above the 10-percent level of statistical significance (p-value = 0.108).

103

and engagement received several thousand dollars more on average in housing subsidies plus escrow disbursements than their counterparts in the control group. 103

Conclusion

In several respects, the subgroup findings after 5 or more years of followup are consistent with the findings from the previous report that analyzed outcomes after 3 years. So far, FSS group members in multiple subgroups did not experience gains above the control group in employment and earnings measures calculated with NDNH administrative data, but they did show isolated positive effects on other measures of use of credit and financial well-being based on other data sources. In addition, FSS

programs with the strongest emphasis on monitoring and engagement continued to show negative impacts on NDNH-recorded earnings. However, some evidence suggests that more positive effects on financial well-being may be seen after more FSS group members graduate from the program and receive their escrow disbursement (as discussed previously, graduations may be delayed or forgone because of the COVID-19 pandemic and economic downturn). The final report, based on an additional 12 months of followup for data sources used in this analysis and with responses to a long-term followup survey, will explore these issues further.

103 Not shown, combining average escrow disbursements with average total housing subsidies yields an increase for the FSS group of about $4,500 (statistically significant).

104

Chapter 7. Summary and Next Steps in the Evaluation

This report updates and expands upon MDRC’s interim reports examining the implementation and effects of the HUD Family Self-Sufficiency (FSS) program for housing choice voucher (HCV) recipients. These reports are part of the national evaluation of the FSS program sponsored by HUD.

FSS programs offer participants referrals to a broad array of services, plus access to a long-term escrow account, designed to work together to promote earnings growth and financial self-sufficiency over the 5-year program period. At program enrollment, participants work with case managers to set goals and action steps to achieve them. Most FSS participants commit to at least one employment-related goal, along with other goals related to financial security, education, or homeownership. In this report, the evaluation covers the full 5 years of followup (and partway through Year 6 for some outcomes), tracing longer-term program outcomes and effects. Importantly, the new results capture participants’ experiences in Years 4 and 5, the period leading up to their potential graduation from the program. This far into followup, there is no clear evidence that the FSS program encourages higher rates of economic mobility or financial security: both groups appear to experience comparable trajectories. About one-third of the FSS group members continue to be enrolled at the end of the followup period—and could qualify for additional extensions because of the pandemic—and new patterns of outcomes and effects may emerge in the remaining followup period covered by the evaluation. Nevertheless, the results offer some important insights and observations about the pattern of outcomes achieved by FSS participants in the 5 years following program enrollment. These insights and observations, along with the next steps for the evaluation, are the focus of this chapter.

What the 5-Year Results Tell Us

Case management strategies in Years 4 and 5 resemble efforts in the years leading up to the final push.

Contact expectations for staff and how often they were expected to reach out to clients did not change significantly as participants approached Year 5. At the regularly scheduled annual or quarterly meetings, case managers typically reviewed clients’ goals to assess whether they were “on track” to graduate. A small number of sites sent official notification of the contract end date by mail. These letters put in writing the date by which all goals must be completed for participants to graduate and earn their escrow.

Graduating from an FSS program, a marker of program success, is an outcome achieved by a small minority of FSS participants. At the end of Year 3, midway through the 5-year program, about 4 percent of FSS group members had graduated. This rate climbed to about 17 percent by the end of the followup period covered in this report: 14.2 percent graduated with an escrow disbursement, and the remaining 2.7 percent did not. Escrow disbursements were substantial for the graduates who had accrued escrow, averaging nearly $10,000 per recipient. Close to one-fourth of the individuals assigned to the FSS

program group continue to be enrolled in the FSS program and have accrued sizable escrow balances (approximately $7,000 in the last month of followup); some or all of these individuals may successfully graduate from an FSS program and earn an escrow disbursement, an outcome to be tracked by the ongoing evaluation. The graduation rate could end up between 25 and 30 percent, placing the 18

105

housing agencies in the FSS evaluation above the combined average of 24 percent for the approximately 700 housing agencies with graduation rates published by HUD. 104

A larger number of FSS participants accrue escrow credits, some with sizable balances, but only a small fraction graduate with an escrow disbursement. By the end of Year 1, nearly one-fourth of the FSS group had accrued some escrow. This rate climbed to about 52 percent by the end of Year 3 and remained under 60 percent by the end of the followup period. Relatively few additional FSS group members started accruing escrow after Year 3, suggesting that the probability of beginning to accrue escrow credits deteriorates about midway through the program. Further, a little more than 40 percent of the FSS group members who exited the FSS program during the followup period forfeited their escrow accruals because they ended their enrollment in the FSS program for reasons other than graduation. Thus, although the escrow account serves as an important draw for participants, the possibility of realizing the benefit of this savings instrument is quite diminished for most participants who enroll in the program (the benefits accrue to a smaller group).

FSS group members who left the program without graduating and continued to receive housing vouchers appear to have enrolled in FSS programs with more serious barriers to employment of any group: they had the lowest employment rate at random assignment of any group and the highest incidence of having a physical or mental health problem that made it difficult to find and keep a job.

They also had the highest incidence of receiving SSI or SSDI disability benefits. Enrollment in the FSS

program is open to individuals with a wide range of advantages and disadvantages in the labor market, but the program offers unequal chances of attaining the most tangible forms of program participation success—graduation and escrow disbursement. For different reasons, FSS group members at each extreme on the relative advantage continuum are less likely to benefit than FSS group members who are more in the middle. Increasing the proportion of FSS participants that could tangibly benefit from the program would require a program framework that calibrates success on the basis of starting levels or relies on targeting.

Several program staff members, when interviewed in 2020, stated that they viewed the escrow account as a draw for participants but did not necessarily see it motivating participants enough to engage in services, attain goals other than employment, and get them to the finish line. As some staff members indicated, their programs would see more graduations if accruing escrow were the sole requirement.

They pointed to cases in which participants accrued significant escrow balances yet could not achieve their program goals to graduate from the program. The administrative data also confirmed instances of forfeitures of substantial amounts of accrued escrow. Having program staff members promote interim disbursements to support participants’ pursuit of goals, as discussed below, might be a way to make the escrow benefit feel more tangible for participants. HUD may also want to consider alternative escrow models, especially ones structured around more short-term financial incentives, as in the case of the Work Rewards demonstration, or build the financial incentive into the rent policy, as in the case 104As noted in chapter 2, recent HUD guidance on determining “suitable employment” led FSS programs to eliminate additional employment requirements. This change, combined with the new performance measurement system, are expected to lead to increased graduation rates in the future.

106

of HUD’s Jobs Plus program for public housing residents or the Rent Reform Demonstration for voucher holders.

Interim disbursements are rarely used. Although the escrow account is designed as a distant savings incentive, this policy tool includes a short-term benefit feature, which theoretically could allow participants to draw on their accrued savings to support the pursuit of their FSS goals. The administrative data gathered for the evaluation show extremely low levels of interim disbursements, however. At the end of Year 3, less than 1 percent of the FSS participants had received an interim escrow disbursement. This estimate will be updated for the final report, but the proportion of participants receiving interim disbursements is expected to remain relatively low on the basis of the pattern documented so far; unclear from available data, though, is whether a higher number of interim disbursements are requested but not approved by program staff (this information is not captured in PHA or HUD IMS/PIC administrative data). In 2020, staff members in all but one of the sites reported that participants were permitted to receive interim disbursements. Despite generally positive views about the possible benefits of providing interim disbursements, they confirmed the earlier finding that few participants receive one. For the most part, this program feature is not routinely publicized or discussed during check-in meetings.

Further, staff members’ perspectives, such as their views that participants should accrue as much escrow as possible or look elsewhere to fund their needs, also affect the potential instrumental value of allowing interim disbursements. Providing staff members with additional guidance on the merits of short- and long-term uses of escrow balances might enable them to have conversations with participants about their escrow balances and how they can be used while the participants are building their escrow balances. For instance, the average balances for automobile and education installment loans in Vantage data imply that program staff could have encouraged participants’ greater use of interim disbursements for such purposes.

Employment and earnings trajectories are similar for FSS and control group members. This evaluation, until recently, took place during a long economic expansion. It documents high levels of employment for both study groups. Against this background of high labor force participation, the evaluation tests the effectiveness of the FSS program on increasing participants’ employment and earnings.

On that front, so far, there is no evidence that program group members may have worked more or earned more than their counterparts in the control group. In the first year of followup, both research groups’

employment rate was around 73 percent. A pattern of comparable employment rates and earnings is evident for each of the 5 years of followup, with program and control group members showing fairly high levels of attachment to the labor force (quarterly employment rates are somewhat lower for both groups, reflecting employment churning). As participants neared graduation, the point at which they could claim their escrow accruals, the FSS group’s employment and earnings trajectories did not diverge from those of their control group counterparts. The subgroup analysis, focusing on baseline employment and education status, also does not point to clear differences in impacts for these groups.

Overall, average annual earnings increased over time for both the FSS group and the control group, suggesting that FSS-type services add little to what individuals can do on their own or through job search and employment-focused services available to them in the community.

107

Although an additional year of employment and earnings outcomes remains to be examined for study participants, this overall pattern of labor market effects could lead to the conclusion that stronger (or different) approaches are needed to generate bigger and more transformative effects—such as interventions that can help with advancement (for those employed) and that can help participants with varying levels of barriers to employment to take significant steps toward self-sufficiency. The FSS

program includes several attractive features—it gives participants at least 5 years to work toward their program goals and helps them build savings—but improvements in how the core components of the model are delivered might be necessary to help participants make progress toward their goals and to help them advance. Part of this effort could involve considering alternate case management or coaching strategies. A new generation of employment-focused interventions, including MDRC’s MyGoals for Employment Success, combine personalized and structured goal setting and coaching with an explicit focus on participants’ executive skills. 105 The Compass Working Capital model, which is being implemented by some FSS programs in Massachusetts, focuses on helping participants build financial management skills and goals. In this program, coaches handle financial skill building directly instead of referring participants to community agencies, as is common for FSS programs in this evaluation. 106

Access to additional (and flexible) grant funding from HUD could also allow programs to offer tuition relief, scholarships, stipends, and other support services, which FSS staff members believe is critical to help participants make progress.

Could impacts on some financial security outcomes occur without increases in earnings or household income? A self-sufficiency program, especially one with a focus on financial education and management, should be able to demonstrate improvements in other aspects of families’ financial well-being. At the 3-year mark, and as documented in the interim report, survey data indicated that FSS

programs were effective in increasing participants’ connections to mainstream financial institutions, participants were more likely to be aware of their credit history, and they reported higher credit scores, compared with the control group. FSS group members, on average, however, incurred greater debt (it is unclear what is driving the higher debt levels). Consistent with programs that emphasize the use of financial security and homeownership preparation counseling and workshops, FSS programs led to positive effects on attitudinal and perceived well-being outcomes. FSS group members were more likely than control group respondents to report that they had improved their financial situation in the past year and were better able to plan for the future.

The detailed analysis of credit data presented in this report, the first of its kind for a national sample of FSS participants, allows the evaluation to examine some of the same financial security outcomes explored in prior survey efforts. Most importantly, this report considers whether FSS improves credit 105 See Castells and Riccio (2020). MyGoals, designed by MDRC, is part of an evaluation funded by the U.S.

Department of Health and Human Services (HHS), which tests different employment coaching programs for Temporary Assistance for Needy Families (TANF) and housing-assisted populations. The Kansas GOALs program for SNAP recipients also offers an example of a “next-generation” job-readiness/job-search program.

106 Using a matched comparison design and PHA data sources, the study of the Compass approach showed positive earnings and reduced TANF receipt for Compass participants than for their matched peers. Participants also achieved positive credit and debt outcomes that exceeded benchmarks. See Abt Associates evaluation:

https://www.abtassociates.com/sites/default/files/files/Insights/reports/2017/Compass%20FSS%20Evaluation%20R

eport_09082017_0.pdf.

108

scores above control group levels, which can positively affect other financial outcomes, including access to traditional credit. Despite the FSS program’s general emphasis on financial education and management, the longitudinal data show that most study participants have credit scores in the Subprime category, and the program did not lead to increases above the control group in average credit scores.

The two exceptions are the subgroups with higher graduation rates—those with 2-year degrees or more and part-time employment—which had modest gains in credit scores above the control group’s levels.

Both study groups also incurred high (and increasing) levels of debt, with automobile and education loans accounting for the largest proportion of their non-housing-related debt, which was incurred to support their education and employment needs. FSS group participants were repaying their debt with higher installment payments, which could be a function of debt management strategies participants were exposed to through their participation in financial education activities, yet programs may need to resort to other strategies—including stronger employment interventions—to help FSS participants improve their credit scores and access to traditional credit. Further, special attention may be needed to assess the types of financial management services participants are receiving and whether these services—workshops or one-on-one counseling—are robust enough to make a difference. The finding that the FSS program led to a small and statistically significant increase over the control group average in the use of traditional lending sources without alternative financial services is encouraging.

So far, effects of FSS on housing costs and subsidies are small. Five years out, roughly the same proportion of FSS and control group households remained enrolled in the HCV program. Moreover, cumulatively over 5 years (and including the entire impact sample), FSS group members paid, on average, about 5 percent more out-of-pocket for subsidized housing (family share) than control group members (a statistically significant difference). However, the difference between research groups in average total housing subsidies is close to $0 and not statistically significant. Non-experimental comparisons involving study participants in each research group who remained enrolled in the HCV

program at the end of year 5 show the FSS group as incurring a small increase in monthly family share and a small decrease in monthly housing subsidies, compared with control group members. These findings suggest that the main effects of FSS on housing costs and subsidies, if any, are yet to come. They are likely contingent on the proportion of FSS group members who are still enrolled in FSS and HCV at the end of year 5 who go on to graduate from the FSS and on the proportion of graduates who decide to move to unsubsidized housing or, possibly, to use their escrow disbursement to help pay for housing expenses out-of-pocket. The evaluation will continue to examine the long-term interaction between employment and housing outcomes—whether those who recently graduated and earned sizable escrow disbursements continue to receive housing assistance or move to private market housing without a subsidy or use their escrow disbursement to purchase a home; in this way, their exits from the voucher program could enable PHAs to assist new households. An additional year of housing subsidy data, along with the longer-term followup survey, will also allow the evaluation to examine families’ housing circumstances roughly 6 to 7 years after study entry.

109

The Evaluation Going Forward

The FSS evaluation is slated to end in 2022, providing 6 to 7 years of followup for the study sample.

Not only will this length of followup allow the evaluation to track the remaining FSS participants through their extended contract period and examine post-FSS circumstances for those who graduated (how they are faring financially, whether they continue to receive housing subsidies, and how they use their escrow), it will also support an analysis of how families fared in the aftermath of the COVID-19

pandemic. The FSS programs in this study made dramatic changes in their daily operations, and most shifted online to engage with program participants. So far, a small minority of FSS group members have received an escrow disbursement, which could improve their financial well-being and help them cope within the COVID-19 context.

The final report will take into account the pandemic and its implications for the program’s effects on participants’ work outcomes, graduation and escrow receipt, financial well-being, and receipt of housing and other government subsidies. No further qualitative data collection is planned for the final report, but a brief survey to be fielded in 2021 to capture the longer-term outcomes and post-exit circumstances of current and former FSS participants in the study has been adapted to gather information on how families coped with the economic shocks caused by the pandemic. The final report will also provide more conclusive evidence about who benefits and whether new patterns of outcomes and impacts are revealed for particular subgroups of participants or subsets of programs.

110

Appendix A

111

Appendix Exhibit A.1: Staffing, Escrow, and Graduation Policies, 2020

Case Management Staffing

Escrow

Graduation Requirementsa

At Least One

Interim

Housing

Number

HCV

Homeownership

Annual In-

Escrow

Withdrawal Employment

Employment Employment Limitations

Agency

with FSS Responsibilities Responsibilities Person Meeting Disbursement

Limits

Hours

Earnings

Stability

on Revising

Caseload

Expected

Permitted

Goals

Housing

5 or more

No

Yes

Yes

Yes

Up to 50% of 30 hours per

--

6 months

1 year (final

Agency #1

the balance

week

goal); 6

months

(interim

goals)

Housing

1–4

Yes

No

Yes

Yes

Up to 25% of

--

--

--

6 months

Agency #2

the balance

Housing

1–4

No

No

Yes

Yes

Up to 50% of

--

--

--

--

Agency #3

the balance

Housing

1–4

No

Yes

Yes

No

N/A

--

--

--

6 months

Agency #4

Housing

5 or more

Yes

No

No

Yes

None

--

--

--

--

Agency #5

Housing

5 or more

Yes

Yes

No

Yes

Up to 50% of

--

--

--

1 year

Agency #6

the balance

Housing

1–4

No

Yes

Yes

Yes

Up to 50% of

30 hours per Earnings must

--

--

Agency #7

the balance

week

be deemed

(exceptions “reasonable”

made on case- by Housing

by-case basis) Agency staff

Housing

1–4

Yes

No

Yes

No

N/A

--

--

--

2 years but

Agency #8

reviewed on a

case-by-case

basis as

requested

Housing

1–4

No

Yes

Yes

Yes

For enrollees

--

--

--

6 months

Agency #9

after March 1,

2018, a

maximum of

$5,000 during

FSS contract

Housing

5 or more

Yes

No

Yes

Yes

Up to 25% of the 32 hours per

--

12 months

12 months

Agency #10

balance

week

(continued)

112

Appendix Exhibit A.1 (continued)

Case Management Staffing

Escrow

Graduation Requirementsa

At Least One

Annual In-

Interim Escrow

Housing Number

HCV

Homeownership

Employment

Person

Disbursement

Withdrawal

Employment

Employment

Limitations on

Agency with FSS Responsibilities Responsibilities

Limits

Hours

Earnings

Stability

Caseload

Meeting

Permitted

Revising Goals

Expected

Housing

1–4

No

Yes

Yes

Yes

Cannot

30 hours

A “decent”

12

2.5 years

Agency

withdraw until

per week job (such as.

months

#11

after 12 months

a job with

of accruing

opportunity

escrow and

for growth)

maintaining

employment

Housing

1–4

Yes

Yes

Yes

Yes

Less than 100%

--

More than

--

--

Agency

of the balance

minimum

#12

wage

Housing

1–4

No

No

Yes

Yes

Up to 30% of

--

--

--

3 months

Agency

the balance

#13

Housing

1–4

No

Yes

Yes

Yes

Enrolled for at

30 hours

--

6 months

--

Agency

least 1 year;

per week

#14

Up to 25% of

the balance

(annually); car

repairs require

20%

contribution by

participant

Housing

1–4

Yes

Yes

Yes

Yes

Up to two

--

--

--

--

Agency

disbursements

#15

within a 12-

month period;

Up to 30% of

the balance

(exceptions for

home purchase

and education)

(continued)

113

Appendix Exhibit A.1

(continued)

Case Management Staffing

Escrow

Graduation Requirementsa

At Least One

Annual In-

Interim Escrow

Housing Number

HCV

Homeownership

Employment

Person

Disbursement

Withdrawal

Employment

Employment

Limitations on

Agency with FSS Responsibilities Responsibilities

Limits

Hours

Earnings

Stability

Caseload

Meeting

Permitted

Revising Goals

Expected

Housing

5 or more

No

Yes

Yes

Yes

Up to 50% of

--

--

--

2 months

Agency #16

the balance

Housing

1–4

Yes

Yes

Yes

Yes

Up to 50% of the 30 hours

--

--

6 months

Agency #17

balance

per week

Notes: FSS = Family Self-Sufficiency. HCV = Housing Choice Vouchers. TANF = Temporary Assistance for Needy Families. aUnless otherwise noted, all requirements are as of graduation. No receipt of TANF benefits for 12 months is a requirement at all sites. bSome sites have different employment requirements for disabled and/or elderly clients. These are not included in the exhibit.

Source: Information collected during MDRC interviews with FSS administrators and case managers in Quarter 4, 2015, and in Quarter 2, 2018 and Quarter 2, 2020

114

Exhibit A.2: Methodology for Creating Site Clusters

Site clusters were created from a variety of source data, described in exhibit 6.2. Some site clusters were created from a single measure, whereas others were created from two or three measures. Measures were created using the following procedure:

• Calculate site-level mean values for each source measure.

• Calculate a cross-site mean (“mean of means”) for each source measure.

• Calculate the cross-site standard deviation for each source measure.

• Calculate site-level z-scores for each source measure. For each site (PHA), subtract the site-level mean from the cross-site mean. Then, divide the difference by the cross-site standard deviation.

• For site clusters created from two or three source measures, sum the z-scores and then divide by the number of component measures to create an average composite score.

• Group sites with similar scores into high, medium, or low categories. Sites with an average z-score value above 0.5 were grouped in the high category, whereas sites with an average z-score below -0.5 were grouped in the low category. The remaining sites with average z-scores of between -0.5 and +0.5 were grouped in the medium sites.

115

Appendix B

116

Exhibit B.1: Family Self-Sufficiency and HCV Program Enrollment Status Following Random Assignment Family Self-Sufficiency Impact Sample (FSS Group Only)

99.8

100

93.2

FSS Program

HCV Program

90

94.0

82.8

86.2

80

73.5

70

66.1

72.1

59.3

) 60

56.1

%

60.9

tage ( 50

51.8

Percen 40

42.3

30

29.9

20

10

0

1

12

24

36

48

60

Last Month of

Follow-up

Followup Month

Notes: Month 1 is the month of random assignment. For most FSS group members, followup ended between months 61 and 72 after random assignment.

Sources: U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH

Information Center (PIC) data and Housing authority administrative data

117

Appendix C

118

Exhibit C.1: Impacts on Employment and Earnings by Quarter, FSS Impact Sample FSS

Control Difference

Outcomes

Group

Group

(Impact)

P-Value

Quarterly Employment (%)

Not employed at random assignment

Quarter 2

25.9

30.2

– 4.3 *

0.079

Quarter 3

33.9

33.2

0.7

0.788

Quarter 4

38.4

35.6

2.7

0.289

Quarter 5

40.8

36.8

4.0

0.126

Quarter 6

42.9

40.2

2.6

0.311

Quarter 7

45.5

44.9

0.5

0.848

Quarter 8

45.0

43.7

1.3

0.628

Quarter 9

44.0

42.8

1.2

0.649

Quarter 10

45.3

45.5

– 0.2

0.932

Quarter 11

46.0

46.0

– 0.1

0.981

Quarter 12

48.0

47.8

0.2

0.939

Quarter 13

47.9

47.1

0.9

0.755

Quarter 14

47.8

47.2

0.5

0.848

Quarter 15

49.6

48.2

1.3

0.623

Quarter 16

47.3

47.9

– 0.6

0.838

Quarter 17

48.3

48.6

– 0.3

0.910

Quarter 18

47.8

49.9

– 2.0

0.461

Quarter 19

46.4

49.2

– 2.7

0.327

Quarter 20

46.8

48.8

– 2.1

0.454

Quarter 21

48.5

46.9

1.6

0.573

Employed part-time (1–34 hours) at random assignment

Quarter 2

81.8

80.8

1.1

0.688

Quarter 3

78.4

81.2

– 2.8

0.344

Quarter 4

79.7

77.6

2.2

0.489

Quarter 5

81.2

77.0

4.2

0.170

Quarter 6

77.3

74.1

3.2

0.326

Quarter 7

78.2

75.4

2.9

0.376

Quarter 8

75.9

77.5

– 1.6

0.622

Quarter 9

77.3

75.7

1.6

0.621

Quarter 10

77.1

74.1

3.0

0.374

Quarter 11

78.8

73.5

5.3

0.103

Quarter 12

75.7

74.9

0.9

0.789

Quarter 13

74.5

71.1

3.4

0.320

Quarter 14

73.4

72.3

1.1

0.744

(continued)

119

Exhibit C.1 (continued)

FSS

Control Difference

Outcomes

Group

Group

(Impact)

P-Value

Employed part-time (1–34 hours) at random assignment

Quarter 15

72.5

71.9

0.6

0.865

Quarter 16

73.1

74.2

– 1.1

0.741

Quarter 17

73.4

73.2

0.2

0.943

Quarter 18

74.6

73.5

1.0

0.765

Quarter 19

75.3

75.6

– 0.3

0.920

Quarter 20

77.1

73.1

4.0

0.237

Quarter 21

77.3

73.9

3.4

0.312

Employed full-time (35 hours or more) at random assignment

Quarter 2

87.7

88.2

– 0.5

0.795

Quarter 3

84.9

86.6

– 1.8

0.412

Quarter 4

84.1

86.3

– 2.2

0.320

Quarter 5

82.5

84.8

– 2.3

0.345

Quarter 6

81.6

84.0

– 2.4

0.351

Quarter 7

82.0

84.9

– 2.9

0.254

Quarter 8

80.7

84.3

– 3.7

0.155

Quarter 9

78.7

84.4

– 5.6 **

0.035

Quarter 10

80.4

83.3

– 2.9

0.271

Quarter 11

81.2

81.8

– 0.6

0.824

Quarter 12

81.6

80.0

1.6

0.568

Quarter 13

82.0

80.4

1.6

0.566

Quarter 14

81.9

80.9

1.0

0.698

Quarter 15

82.3

81.8

0.6

0.828

Quarter 16

80.7

81.3

– 0.6

0.820

Quarter 17

81.1

79.4

1.8

0.527

Quarter 18

82.6

78.3

4.2

0.116

Quarter 19

80.6

77.7

2.9

0.318

Quarter 20

81.7

79.4

2.3

0.407

Quarter 21

79.2

81.4

– 2.3

0.414

Total Impact Sample

Quarter 2

58.8

60.8

– 2.0

0.154

Quarter 3

60.7

61.7

– 1.0

0.510

Quarter 4

62.9

61.7

1.2

0.421

Quarter 5

63.9

61.5

2.4

0.122

Quarter 6

63.4

62.1

1.3

0.410

Quarter 7

65.1

64.7

0.4

0.799

Quarter 8

63.9

64.5

– 0.5

0.734

(continued)

120

Exhibit C.1 (continued)

FSS

Control Difference

Outcomes

Group

Group

(Impact)

P-Value

Quarter 9

63.4

63.5

– 0.1

0.947

Quarter 10

64.2

64.1

0.2

0.921

Quarter 11

65.1

63.8

1.3

0.439

Quarter 12

65.2

64.5

0.7

0.686

Quarter 13

65.1

63.3

1.8

0.292

Quarter 14

64.8

63.7

1.0

0.547

Quarter 15

65.4

64.4

1.0

0.541

Quarter 16

64.2

64.5

– 0.4

0.829

Quarter 17

64.9

64.0

0.8

0.624

Quarter 18

65.2

64.5

0.7

0.682

Quarter 19

64.1

64.5

– 0.4

0.822

Quarter 20

65.1

64.3

0.8

0.643

Quarter 21

65.2

64.2

0.9

0.574

Total Earnings ($)

Not employed at random assignment

Quarter 2

610

730

– 120

0.189

Quarter 3

1,026

1,022

4

0.973

Quarter 4

1,195

1,205

– 10

0.936

Quarter 5

1,374

1,310

64

0.637

Quarter 6

1,585

1,544

41

0.780

Quarter 7

1,910

1,807

103

0.542

Quarter 8

1,970

1,885

85

0.615

Quarter 9

2,028

1,755

273

0.106

Quarter 10

2,183

2,039

144

0.447

Quarter 11

2,288

2,222

66

0.728

Quarter 12

2,386

2,325

61

0.759

Quarter 13

2,372

2,282

90

0.635

Quarter 14

2,404

2,424

– 20

0.921

Quarter 15

2,516

2,502

15

0.942

Quarter 16

2,529

2,513

17

0.938

Quarter 17

2,579

2,570

8

0.970

Quarter 18

2,623

2,690

– 67

0.763

Quarter 19

2,646

2,610

36

0.870

Quarter 20

2,713

2,745

– 32

0.891

Quarter 21

2,878

2,643

235

0.303

(continued)

121

Exhibit C.1 (continued)

FSS

Control

Difference

Outcomes

Group

Group

(Impact)

P-Value

Employed part-time (1–34 hours) at random assignment

Quarter 2

3,167

3,186

– 19

0.914

Quarter 3

3,241

3,133

108

0.575

Quarter 4

3,307

3,149

158

0.454

Quarter 5

3,571

3,314

257

0.253

Quarter 6

3,707

3,345

362

0.144

Quarter 7

3,512

3,689

– 177

0.473

Quarter 8

3,769

3,904

– 136

0.610

Quarter 9

3,915

3,922

– 7

0.981

Quarter 10

4,079

3,824

255

0.359

Quarter 11

4,054

3,884

170

0.541

Quarter 12

4,143

3,947

196

0.477

Quarter 13

4,106

3,804

301

0.288

Quarter 14

4,196

4,060

136

0.650

Quarter 15

4,213

4,248

– 35

0.908

Quarter 16

4,325

4,186

139

0.652

Quarter 17

4,590

4,073

518 *

0.093

Quarter 18

4,545

4,356

190

0.563

Quarter 19

4,595

4,594

1

0.998

Quarter 20

4,658

4,763

– 105

0.766

Quarter 21

4,861

4,601

260

0.439

Employed full-time (35 hours or more) at random assignment

Quarter 2

5,639

5,696

– 57

0.768

Quarter 3

5,656

5,765

– 109

0.599

Quarter 4

5,497

5,774

– 277

0.233

Quarter 5

5,338

5,805

– 467 *

0.066

Quarter 6

5,736

5,755

– 20

0.940

Quarter 7

5,854

6,074

– 220

0.401

Quarter 8

5,637

5,936

– 300

0.268

Quarter 9

5,885

6,004

– 119

0.676

Quarter 10

5,963

6,038

– 75

0.800

Quarter 11

5,929

6,065

– 136

0.641

Quarter 12

5,963

6,215

– 251

0.426

Quarter 13

5,963

6,262

– 299

0.331

Quarter 14

6,065

6,275

– 209

0.507

Quarter 15

6,364

6,439

– 76

0.810

Quarter 16

6,303

6,459

– 155

0.639

Quarter 17

6,510

6,253

256

0.420

Quarter 18

6,718

6,474

244

0.465

(continued)

122

Exhibit C.1 (continued)

FSS

Control

Difference

Outcomes

Group

Group

(Impact)

P-Value

Quarter 19

6,732

6,656

76

0.824

Quarter 20

6,700

6,757

– 57

0.868

Quarter 21

6,757

6,899

– 142

0.682

Total Impact Sample

Quarter 2

2,781

2,870

– 90

0.285

Quarter 3

2,983

3,015

– 32

0.737

Quarter 4

3,041

3,090

– 49

0.636

Quarter 5

3,162

3,169

– 6

0.955

Quarter 6

3,399

3,271

128

0.275

Quarter 7

3,520

3,579

– 59

0.632

Quarter 8

3,556

3,618

– 63

0.621

Quarter 9

3,704

3,576

128

0.330

Quarter 10

3,821

3,704

118

0.392

Quarter 11

3,846

3,813

32

0.813

Quarter 12

3,937

3,907

30

0.832

Quarter 13

3,905

3,883

22

0.879

Quarter 14

3,987

3,999

– 12

0.935

Quarter 15

4,123

4,136

– 13

0.930

Quarter 16

4,132

4,140

– 9

0.956

Quarter 17

4,285

4,071

214

0.162

Quarter 18

4,349

4,268

81

0.610

Quarter 19

4,392

4,334

58

0.716

Quarter 20

4,443

4,453

– 10

0.950

Quarter 21

4,564

4,436

129

0.434

Sample size (total = 2,548) 1,282 1,266

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment.

Es timates were regression-adjusted using ordinary l eas t s

quares, controlling for pre-ra ndom assignm

ent

characteristics

of sample members. Rounding may cause slight dis cre pa

ncies in calculating sums a

nd differences.

A tw o-tailed t-test

w as applied to differences between research groups . T h

e p-value indicates the likelih ood that the di ffer ence between

the FSS group and control group arose by chance. Statistical significance levels are indicated as: * = 10 percent; ** =

5 percent; and *** = 1 percent.

S

ource: MDRC calculations using baseline data and q

uart erly wage data from the Nat ional Directory of

New Hires.

123

Exhibit C.2: Regression Coefficients for Estimated Impacts on Total Earnings

in Years 1 to 5, Family Self-Sufficiency Impact Sample

Parameter Estimate

P-Value

Intercept

14,612

0.109

Assigned to FSS Group (impact)

598

0.767

Covariates

Sample member characteristics

Female

6,482

0.089

Age 18–34

18,523

<.0001

Age 35–44

14,598

<.0001

Married or cohabitating

2,550

0.517

Black

3,004

0.268

1 child

4,152

0.196

2 children

7,644

0.025

3 or more children

4,244

0.248

Has a child age 5 or younger

– 597

0.820

Education

High school diploma or GED

1,805

0.566

Some college

4,294

0.154

2-year college degree or higher

19,395

<.0001

Has trade license or training certificate

540

0.797

Public assistance

Received SNAP/food stamps

– 512

0.852

Received SSI or SSDI

– 12,172

0.000

Received TANF

238

0.943

Received Housing Choice Voucher less than 4 years

404

0.871

Received Housing Choice Voucher 4–7 years

5,435

0.044

Hardship and barriers to employment

Has any barrier to employment

– 10,080

<.0001

Reported 1 hardship in the year before random assignment

– 2,442

0.384

Reported 2 hardships in the year before random assignment

– 2,588

0.401

Reported 3 or more hardships in year before random assignment

– 4,248

0.128

(continued)

124

Exhibit C.2 (continued)

Parameter Estimate

P-Value

Employment

Currently employed

9,446

0.005

Currently employed full-time

5,444

0.079

Employed 1–6 months in the year before random assignment

9,550

0.006

Employed 7–11 months in the year before random assignment

2,507

0.554

Employed 12 months in the year before random assignment

– 920

0.826

Earnings

Employed in the quarter before random assignment

– 223

0.950

Employed in the second quarter before random assignment

– 2,770

0.423

Total earnings in the 2 quarters before random assignment

6

<.0001

Total earnings squared in the 2 quarters before random assignment

0

0.763

Enrollment

Randomly assigned in quarter 4, 2013

– 8,310

0.190

Randomly assigned in quarter 1, 2014

– 2,245

0.444

Randomly assigned in quarter 2, 2014

2,163

0.425

Housing Authority of the City of Alameda

11,928

0.093

Orange County Housing Authority

2,816

0.704

Housing Authority of the City of Riverside

– 622

0.929

Housing Authority of the City of Deerfield Beach

– 3,458

0.727

Housing Authority of the City of Ft. Lauderdale

– 8,353

0.292

Baltimore County Housing Office

– 9,606

0.223

Housing Opportunities Commission of Montgomery County

1,507

0.844

Housing Authority of Kansas City

– 2,101

0.790

Jersey City Housing Authority

– 3,587

0.681

Akron Metropolitan Housing Authority

– 15,596

0.097

Columbus Metropolitan Housing Authority

– 472

0.958

Lucas Metropolitan Housing Authority

– 8,687

0.378

Youngstown Metropolitan Housing Authority

– 5,129

0.586

Dallas Housing Authority

1,578

0.834

Fort Worth Housing Authority

– 9,724

0.270

Houston Housing Authority

398

0.957

Tarrant County Housing Assistance Office

– 13,426

0.094

Enrolled in FSS for help with employment

– 2,463

0.312

Medical coverage

Public medical insurance

– 7,185

0.021

Private medical insurance

5,057

0.166

(continued)

125

Exhibit C.2 (continued)

Parameter Estimate

P-Value

Savings

Had checking or savings account

7,563

0.002

Had savings between $1–$500

– 2,749

0.260

Had savings greater than $500

748

0.843

Debt

$1–$1,000

– 1,439

0.704

$1,001–$5,000

3,723

0.275

$5,001–$10,000

5,703

0.123

$10,001–$20,000

4,974

0.147

$20,001 or greater

11,466

0.001

R-square

0.506

Sample size

2,548

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between

Oct ober 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of ra ndom

assignment. Estimates were regression-adjusted using ordinary least squares, controlling for pre-ra ndom assignment ch

aracteristics of sample members. No special weights were applied to responses to adjust for differences in samp le size by site.

S ou rces: MDRC calculations using baseline data and quarterly wage data from the National Directory of New Hir es 126

Exhibit C.3: Unadjusted and Adjusted Impacts on Employment and Earnings in Years 1 to 5, Family Self-Sufficiency Impact Sample

Adjusted Impacts

Unadjusted Impacts

FSS

Control

Difference

FSS

Control

Difference

Outcomes

Group

Group

(Impact)

P-Value

Group

Group

(Impact)

P-Value

Ever employed (%)

86.9

85.4

1.5

0.193

86.7

85.7

1.0

0.483

Average quarterly employment rate

(%)

64.0

63.5

0.5

0.651

63.6

63.9

– 0.3

0.827

Total earnings ($)

75,929

75,332

598

0.767

74,569

76,709

– 2,140

0.447

Sample size (total=2,548)

1,282

1,266

1,282

1,266

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Average quarterly employment rate is calculated as: total quarters with employment divide

d by total quarters of follow

up, expressed

as a percentage. R egr ession-adjuste d es timates use d ordinary le ast squares, contr olli

ng for pre-

random assignment

characteristics o f sample m embers. No special weights

were applied to a djus t for differ ences in sam ple size by housi ng authority.

Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between research groups. The p-value indicates the likelihood that the difference between the FSS group and control group arose by chance. Statistical significance levels are indicated as:

* = 10 percent; **

= 5 percent; and * ** = 1 per

cent.

Source: MDRC calcu lations using qu arterly wag e data from

the National Direc tory of New Hire s

127

Exhibit C.4: Total Earnings in Years 1 to 5 by Level of Exclusion for Outlier Values, Family Self-Sufficiency Impact Sample

Sample

Standard

95th

99th

Outcome

Size

Mean

Deviation

Percentile

Percentile

Maximum

Earnings ($)

Include all values

2,548

76,299

72,490

215,619

278,566

534,445

Reset values above $25,000 to $0

2,548

75,632

70,963

212,361

274,734

371,134

Exclude top 1 percent

2,523

73,359

67,447

203,064

253,193

278,566

Exclude top 5 percent

2,421

66,541

59,686

178,984

204,803

215,619

Notes: The FSS Impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment.

S ource: MDRC calculations usin g quart

erly wage data fro m the National Directory of New Hir es

128

Exhibit C.5: Impacts on Total Earnings in Years 1 to 5 by Level of Exclusion for Outlier Values, Family Self-Sufficiency Impact Sample

FSS

Control

Difference

Outcome

Group

Group

(Impact)

P-Value

Earnings ($)

Include all values

76,339

76,257

82

0.968

Reset values above $25,000 to $0

75,929

75,332

598

0.767

Exclude top 1 percent

74,239

72,459

1,780

0.361

Exclude top 5 percent

67,153

65,918

1,235

0.500

Sample size (total = 2,548)

1,282

1,266

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, a nd with a head of household a ge 18 t

o 61 at the time of random assignment. Sample sizes for specific outcomes may vary bec ause of exclusion of outliers and missing values. Estimates wer

e regressio n-adjusted using ordina ry least squares, controlli

ng for pre-random assignment charact erist ics of sample members. No

special weights were applied to responses to adjust for differences in sample size by site. Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between research groups. The p-value indicates the likelihood that the difference between the FSS group and control group arose by chan ce. Statis tical significance levels

are indicated as: * = 10 per cent; ** = 5 percent; and *** = 1 p er

cent.

Source: MDRC calculations usin g quart

erly wage data from th

e National Directory of New Hires

129

Exhibit C.6: Impacts on Employment and Earnings During Years 1 to 5 by Weighting Strategy, Family Self-Sufficiency Impact Sample

FSS Control Difference

Outcomes

Group Group

(Impact)

P-Value

Unweighted

Ever employed (%)

86.9

85.4

1.5

0.193

Average quarterly employment rate (%)

64.0

63.5

0.5

0.651

Total earnings ($)

75,929 75,332

598

0.767

Equal weighting

Ever employed (%)

87.4

85.0

2.3 **

0.038

Average quarterly employment rate (%)

64.1

63.5

0.7

0.553

Total earnings ($)

75,328 76,046

– 717

0.708

Weighting by total FSS householdsa

Ever employed (%)

86.8

85.6

1.3

0.267

Average quarterly employment rate (%)

63.9

63.6

0.3

0.783

Total earnings ($)

75,620 75,749

– 129

0.948

Sample size (total=2,548)

1,282

1,266

Notes: The FSS Impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Average quarterly employment rate is calculated as: total quarters with employment divided by total quarters of follow-up, expressed as a percentage. Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between research groups. The p-value indicates the likelihood that the difference between the FSS group and control group arose by chance. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent. aTotal FSS caseload includes 5,686 households enrolled in FSS as of December 31, 2014. The total includes FSS group members and FSS

participants who enrolled in the program before the start of random assignment, enrolled in the program after the end of random assignment, or withdrew from the research sample.

Sources: MDRC calculations using quarterly wage data from the National Directory of New Hires and December 2014 data from the U.S. Department of Housing and Urban Development (HUD) Public and Indian Housing Information Center (PIC) Data

130

Exhibit C.7: Impacts on Changes in Reported Estimated Gross Annual

Head-of-Household Earnings, Family Self-Sufficiency Impact Sample

FSS

Control

Difference

Outcome

Group

Group

(Impact)

P-Value

Months 1 and 61, if still enrolled in HCV program

Average change ($)

5,229

4,096

1,133

Change in earnings (%)

Increase

46.6

45.6

1.0

No change

30.2

31.4

– 1.2

Decrease

23.2

23.0

0.3

Had earnings (%)

Months 1 and 61

34.1

34.0

0.0

Month 1 only

15.0

13.5

1.5

Month 61 only

20.8

21.2

– 0.4

No reported earnings

30.2

31.3

– 1.1

Month 1 and current or most recent month of enrollment in HCV program

Average change ($)

5,646

5,100

546

0.333

Change in earnings (%)

0.303

Increase

49.8

47.1

2.7

No change

29.6

31.9

– 2.3

Decrease

20.6

21.0

– 0.4

Had earnings (%)

*

0.061

Month 1 and current or

most recent month

39.4

38.5

0.9

Month 1 only

12.2

11.9

0.3

Current or most recent

month only

20.5

19.6

0.9

No reported earnings

27.9

30.0

– 2.1

Sample size (total = 2,466)

1,239

1,226

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment.

Income calculations used data from each household's most recent Housing Choice Voucher eligibility reexamination that took place between Months 1 through 61 after their date of random assignment. For these calculations, households with no reported income or who exited or became ineligible from the Housing Choice Voucher program were excluded from the calculations. Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. For each dollar amount outcome, values above the 99th percentile were considered as outliers and dropped from the calculations. The p-value indicates the likelihood that the difference between the FSS

group and control group arose by chance. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent. Sample sizes for specific outcomes may vary because of missing values. Rounding may cause slight discrepancies in calculating sums and differences.

Source: MDRC calculations using U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data

131

Appendix D

132

Exhibit D.1: Impacts on Indicators of Change Over Time in Credit Scores, 2013–19, Family Self-Sufficiency Impact Sample

FSS Control Difference

Outcome

Group

Group (Impact) P-Value

Average number of years with credit scores (2015–19)

No scores

0.16

0.17

– 0.01

0.774

Experian Vantage 3.0 score only

1.69

1.76

– 0.07

0.361

Clarity Clear Early Risk score only

0.03

0.02

0.01

0.557

Vantage and Clarity scores

3.13

3.05

0.07

0.346

Vantage 3.0 score

Pre-Random assignment year

2012

549

548

1

0.725

Random assignment years

2013

553

550

3

0.330

2014

557

556

1

0.718

Followup years

2015

562

559

2

0.422

2016

562

563

– 1

0.634

2017

568

568

– 1

0.804

2018

569

569

0

0.949

2019

574

571

2

0.505

Years with Vantage 3.0 score (2015–19)

No score

0.18

0.19

0.00

0.913

Deep Subprime

0.90

0.90

0.00

0.971

Subprime

2.48

2.52

– 0.04

0.532

Near Prime

0.76

0.73

0.04

0.411

Prime

0.67

0.66

0.01

0.867

1 year or more with prime credit score (%)

25.0

23.5

1.5

0.331

Highest score (2015–19) (%)

626

624

2

0.382

Change in score from 2013 to 2019

20

21

– 2

0.617

Change in score from 2013 to 2019 (%)

– 500 – – 51 points

19.8

17.9

1.9

0.244

– 50 – – 10 points

16.3

16.2

0.2

0.919

– 9 – 9 points (little change)

11.8

11.0

0.8

0.529

*

10 - 49 points

16.9

20.6

– 3.8 *

0.019

50 - 99 points

18.7

17.5

1.2

0.450

100 points or more

16.6

16.9

– 0.3

0.847

(continued)

133

Exhibit D.1 (continued)

FSS

Control Difference

Outcome

Group

Group

(Impact) P-Value

Change in score level from 2013 to 2019 (%)

Increase

Deep Subprime/Subprime to Prime (300–600 to 661–850)

6.9

6.7

0.2

0.850

Near Prime to Prime (601–660 to 661–850)

3.1

3.2

– 0.2

0.835

Deep Subprime/Subprime to Near Prime (300–600 to 601–660)

10.7

12.4

– 1.7

0.188

No change in score level

71.3

69.4

1.9

0.313

Decrease

Prime to Near Prime (661–850 to 601–660)

1.1

1.1

0.1

0.910

Near Prime to Deep Subprime/Subprime (601–660 to 300–600)

5.5

5.8

– 0.4

0.699

Prime to Deep Subprime/Subprime (661–850 to 300–600)

1.5

1.3

0.2

0.747

Clarity score

Average number of years with Clarity score

(2015–19)

3.15

3.08

0.08

0.314

1 year or more with Clarity score (%)

75.5

74.0

1.6

0.338

Average score (impact sample members with a Clarity score)

Pre-Random assignment year

2012

n/a

n/a

n/a

Random assignment years

2013

n/a

n/a

n/a

2014

529

526

3

Followup years

2015

536

534

3

2016

538

536

1

2017

541

539

2

2018

542

542

0

2019

544

539

5

Sample size (total = 2,548)

1,282 1,266

(continued)

134

Exhibit D.1 (continued)

N otes: The FSS impact sample includes Housing Choice Voucher heads of household who were randomly assigned between October 18, 2013, and December 22, 2014, were age 18 to 61 at the time of random assignment. Sample sizes f

or specific

outcomes may var

y because of mi ssing values. Estimates were reg ression-adj ust ed using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment. No special weights were applied to responses to adjust for differences in sample size by housing authority. Rounding may cause slight

discrepancies in cal

culating sums an d differences.

A two-ta

iled t-test

was applied to differences between the

F SS group a nd the control gro

up for continuous variables and s elected o utcomes e xpressed as pr oportions. In addition, a ch i-square t est for categorical var iables was run to determin

e whether t here is a d ifference i n th e distribution of related

outcomes by research group. The p-value indicates the likelihood that the difference between the FSS group and control group arose by chance. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; *** = 1 percent.

S

ource: MD RC calculations usi

ng Experian Va

ntage 3.0 credit

scores a

nd Clarit

y Clear Earl

y Ri sk scores

135

Exhibit D.2: Impacts on Total Credit and Average Monthly Payment, 2013–19, Family Self-Sufficiency Impact Sample

FSS

Control Difference

Outcome ($)

Group

Group

(Impact)

P-Value

Average total balance

Total non-housing related credit-accounts

2013

8,202

7,456

747

0.169

2014

9,478

9,140

337

0.600

2015

12,465

11,971

494

0.508

2016

15,481

13,272

2,209 ***

0.008

2017

16,649

15,538

1,110

0.229

2018

17,881

17,173

708

0.481

2019

18,725

17,979

746

0.484

Revolving credit accounts

2013

642

613

30

0.722

2014

847

750

97

0.299

2015

1,133

1,018

115

0.313

2016

1,446

1,252

193

0.140

2017

1,633

1,498

135

0.414

2018

1,754

1,743

12

0.944

2019

1,892

1,969

– 78

0.633

Installment credit accounts

2013

6,888

6,541

347

0.460

2014

7,919

8,112

– 193

0.738

2015

10,678

10,755

– 77

0.912

2016

13,457

11,804

1,653 **

0.032

2017

14,426

13,717

709

0.399

2018

15,574

15,124

450

0.629

2019

16,266

15,870

396

0.691

Average total installment balance by type

Auto loans

2013

3,074

2,970

104

0.687

2014

3,401

3,408

– 7

0.980

2015

4,563

4,577

– 14

0.968

2016

5,470

4,990

480

0.185

2017

5,992

5,256

737 *

0.054

2018

6,098

6,015

83

0.846

2019

6,590

6,052

538

0.218

Student loans

2013

3,658

3,403

256

0.532

2014

4,277

4,522

– 244

0.628

2015

5,749

5,979

– 230

0.696

2016

7,572

6,557

1,015

0.124

2017

8,033

8,145

– 112

0.877

2018

9,060

8,707

353

0.658

2019

9,176

9,148

29

0.973

(continued)

136

Exhibit D.2 ( continued)

FSS

Control Difference

Outcome ($)

Group

Group

(Impact)

P-Value

Personal installment loans

2013

155

168

– 12

0.724

2014

241

182

58

0.190

2015

366

199

167 ***

0.003

2016

415

256

159 ***

0.010

2017

400

316

84

0.198

2018

416

402

14

0.836

2019

500

670

– 171

0.230

Average monthly payments

Total non-housing-related credit accounts

2013

169

171

– 2

0.859

2014

201

197

4

0.696

2015

252

243

9

0.448

2016

300

270

30 **

0.022

2017

333

294

39 ***

0.010

2018

359

336

23

0.190

2019

384

352

32 *

0.052

Revolving credit accounts

2013

30

28

1

0.621

2014

40

36

5

0.190

2015

53

46

7 *

0.090

2016

65

59

6

0.221

2017

76

66

10

0.123

2018

79

76

3

0.649

2019

85

88

– 3

0.646

Installment credit accounts

2013

136

136

0

0.991

2014

156

154

2

0.826

2015

196

192

4

0.715

2016

233

206

26 **

0.016

2017

252

224

27 **

0.024

2018

265

249

16

0.225

2019

286

256

30 **

0.028

Sample size (total = 2,548)

1,282

1,266

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Sample sizes may vary because of missing values. Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. No specific weights were applied to responses to adjust for differences in sample size by housing authority. Rounding may cause slight discrepancies in sums and differences. A two-tailed t-test was applied to differences between the FSS group and control group for continuous variables and selected outcomes expressed as proportions. In addition, a chi-square test for categorical variables was run to determine whether there is a difference in the distribution of related outcomes by research group. The p-value indicates the likelihood that the difference between the FSS group and the control group by chance. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10

percent. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups.

Source: MDRC calculations using Experian Vantage 3.0 scores data

137

Exhibit D.3: Impacts on Use of Alternative Financial Services, Family Self-Sufficiency Impact Sample

FSS

Control Difference

Outcome

Group

Group

(Impact)

P-Value

Ever used AFS (%)

Total AFS

2014

3.2

3.3

– 0.1

0.850

2015

6.8

6.1

0.7

0.450

2016

9.2

7.7

1.5

0.170

2017

8.1

8.7

– 0.5

0.630

2018

10.5

9.8

0.8

0.516

2019

12.3

12.5

– 0.2

0.879

AFS Single Payment loans

2014

2.9

2.4

0.5

0.458

2015

4.7

4.3

0.4

0.614

2016

6.0

5.2

0.8

0.383

2017

5.3

5.5

– 0.2

0.820

2018

5.2

4.4

0.8

0.344

2019

3.7

3.9

– 0.2

0.778

AFS Installment loans

2014

0.0

0.4

– 0.4 **

0.031

2015

0.9

0.9

0.0

0.933

2016

1.8

1.3

0.5

0.286

2017

1.9

2.2

– 0.3

0.568

2018

3.3

3.2

0.0

0.957

2019

4.0

3.8

0.2

0.847

Total AFS balance ($)

Total AFS

2014

11

13

– 1

0.607

2015

63

58

4

0.742

2016

103

88

15

0.373

2017

82

100

– 19

0.268

2018

186

175

11

0.737

2019

270

263

7

0.865

AFS Single Payment loans

2014

10

9

1

0.644

2015

36

37

0

0.962

2016

53

52

1

0.931

2017

46

50

– 4

0.721

2018

39

46

– 7

0.621

2019

34

34

0

0.998

(continued)

138

Exhibit D.3 ( continued)

FSS

Control Difference

Outcome

Group

Group

(Impact)

P-Value

AFS Installment loans

2014

0

1

– 1 **

0.040

2015

6

4

2

0.503

2016

12

12

0

0.999

2017

11

18

– 7

0.183

2018

27

30

– 3

0.679

2019

44

42

2

0.887

Sample size (total = 2,548)

1,282

1,266

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Rounding may cause slight discrepancies in sums and differences. Sample sizes may vary because of missing values. Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. A two-tailed t-test was applied to differences between the FSS group and control group for continuous variables and selected outcomes expressed as proportions. The p-value indicates the likelihood that the difference between the FSS group and the control group by chance. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

Source: MDRC calculations using Experian and Clarity credit data

139

Appendix E

140

Exhibit E.1: Impacts on Selected Employment and Earnings Outcomes in Years 1 to 5, by Selected Baseline Characteristics, Family Self-Sufficiency Impact Sample

Sample Sizes:

FSS

Control

Difference

FSS

Control

Outcome

Group

Group

(Impact) P-Value Group

Group

In Years 1 to 5

Average quarterly employment rate (%)

Employment status

Not employed

44.3

44.0

0.3

0.885

575

551

Employed Part-Time

76.6

75.0

1.6

0.475 335

313

Employed Full-Time

81.9

82.5

-0.6

0.740 372

402

Educational attainment

No degree or credential

58.9

55.3

3.7

0.188 257

253

High school degree or GED

61.0

61.5

– 0.5

0.828 319

297

Some college

66.7

66.7

0.0

0.990 470

512

2-year college degree or higher

69.0

67.8

1.1

0.669 236

204

Total household income

$1–$10,000

52.1

53.4

– 1.3

0.582 318

320

$10,001–$20,000

59.4

57.5

1.9

0.366 438

401

More than $20,000

75.3

73.7

1.6

0.313 526

545

Reported barrier to employment

Yes

51.5

50.6

1.0

0.599 534

517

No

72.4

73.0

– 0.6

0.663 748

749

Disability status

Received SSI/SSDI

38.6

36.9

1.6

0.563 177

188

Did not receive SSI/SSDI

68.1

68.1

0.0

0.989 1,105

1,078

Percentage of rent and utilities expenses paid by

household

0–25

56.1

57.1

– 1.0

0.568 583

550

25.01–50

64.3

62.5

1.8

0.355 419

410

More than 50

79.6

76.8

2.8

0.174 280

306

Percentage of household income used for rent

and utilities

0–30

65.0

63.9

1.1

0.438 832

816

More than 30

62.5

62.4

0.1

0.953 450

450

Total earnings ($)

Employment status

Not employed

41,816 40,823 993

0.715 575

551

Employed Part-Time

80,557 77,984 2,573

0.535 335

313

Employed Full-Time

121,210 123,602

– 2,392

0.578 372

402

(continued)

141

Exhibit E.1 (continued)

Sample Sizes:

FSS

Control

Difference

FSS

Control

Outcome

Group

Group

(Impact) P-Value Group

Group

Educational attainment

No degree or credential

58,783

56,157

2,626

0.486

257

253

High school degree or GED

69,764

67,898

1,866

0.631

319

297

Some college

79,686

80,411

– 725

0.831

470

512

2-year college degree or higher

98,193

94,016

4,177

0.503

236

204

Total household income

$1–$10,000

48,920 50,542

– 1,622

0.681 318

320

$10,001–$20,000

57,951 56,287

1,664

0.602 438

401

More than $20,000

107,391 103,743

3,648

0.296 526

545

Reported barrier to employment

Yes

51,501 53,933

– 2,432

0.409 534

517

No

92,960 90,510

2,450

0.384 748

749

Disability status

Received SSI/SSDI

34,137 31,468

2,669

0.484 177

188

Did not receive SSI/SSDI

82,617 82,988

– 371

0.871 1,105

1,078

Percentage of rent and utilities expenses paid by household

0–25

54,359 55,196

– 837

0.775 583

550

25.01–50

73,799 73,425

374

0.916 419

410

More than 50

121,130 116,733

4,397

0.366 280

306

Percentage of household income used for rent and utilities

0–30

77,278 75,849

1,429

0.568 832

816

More than 30

74,492 73,338

1,154

0.740 450

450

Average annual earnings greater than $25,000 (%)

Employment status

Not employed

8.2

8.4

– 0.2

0.887 575

551

Employed Part-Time

22.8

20.4

2.4

0.420 335

313

Employed Full-Time

45.7

49.0

– 3.3

0.308 372

402

Educational attainment

No degree or credential

13.9

13.2

0.7

0.789

257

253

High school degree or GED

21.2

20.7

0.5

0.853

319

297

Some college

24.3

25.6

– 1.3

0.584

470

512

2-year college degree or higher

37.1

34.0

3.1

0.449

236

204

Total household income

$1–$10,000

10.0

11.9

– 1.9

0.413

318

320

$10,001–$20,000

12.4

12.8

– 0.4

0.859

438

401

More than $20,000

40.5

38.7

1.8

0.472

526

545

(continued)

142

Exhibit E.1 (continued)

Sample Sizes:

FSS

Control

Difference

FSS

Control

Outcome

Group

Group

(Impact) P-Value Group

Group

Reported barrier to employment

Yes

13.7

16.2

– 2.5

0.199

534

517

No

30.1

29.1

1.0

0.610

748

749

Disability status

Received SSI/SSDI

8.6

6.8

1.8

0.469

177

188

Did not receive SSI/SSDI

25.7

26.7

– 1.1

0.501

1,105

1,078

Percentage of rent and utilities expenses paid

by household

0–25

10.9

13.0

– 2.1

0.246

583

550

25.01–50

22.2

23.1

– 0.9

0.715

419

410

More than 50

49.3

45.4

3.8

0.303

280

306

Percentage of household income used for rent

and utilities

0–30

22.9

23.9

– 1.0

0.565

832

816

More than 30

24.6

23.0

1.6

0.514

450

450

In Year 5 Average quarterly employment rate (%)

Employment status

Not employed

47.4

48.7

– 1.3

0.586

575

551

Employed Part-Time

76.1

74.0

2.0

0.493

335

313

Employed Full-Time

81.0

79.2

1.8

0.463

372

402

Educational attainment

No degree or credential

57.6

56.4

1.3

0.727

257

253

High school degree or GED

61.6

62.2

– 0.7

0.828

319

297

Some college

67.3

67.3

0.0

0.996

470

512

2-year college degree or higher

73.5

68.9

4.7

0.169

236

204

Total household income

$1–$10,000

55.0

54.4

0.7

0.834

318

320

$10,001–$20,000

60.9

58.7

2.2

0.440

438

401

More than $20,000

74.6

74.1

0.5

0.828

526

545

Reported barrier to employment

Yes

52.7

52.7

0.0

0.992

534

517

No

73.1

73.0

0.1

0.940

748

749

(continued)

143

Exhibit E.1 (continued)

Sample Sizes:

FSS

Control

Difference

FSS

Control

Outcome

Group

Group

(Impact) P-Value Group

Group

Disability status

Received SSI/SSDI

39.2

41.3

– 2.1

0.609

177

188

Did not receive SSI/SSDI

69.0

68.5

0.5

0.748 1,105

1,078

Percentage of rent and utilities expenses paid

by household

0– 25

58.6

58.7

– 0.1

0.981

583

550

25.01–50

64.6

63.3

1.3

0.617

419

410

More than 50

78.0

76.6

1.4

0.628

280

306

Percentage of household income used for

rent and utilities

0–30

65.7

65.8

– 0.1

0.970

832

816

More than 30

64.0

61.2

2.7

0.283

450

450

Total earnings ($)

Employment status

Not employed

10,860 10,688 172

0.835

575

551

Employed Part-Time

18,660 18,314 346

0.778

335

313

Employed Full-Time

26,908 26,787

121

0.920

372

402

Educational attainment

No degree or credential

13,192

12,482

710

0.505

257

253

High school degree or GED

16,581

16,053

529

0.648

319

297

Some college

18,467

18,856

– 389

0.706

470

512

2-year college degree or higher

23,787

21,284

2,503

0.146

236

204

Total household income

$1–$10,000

12,481 12,166

315

0.780

318

320

$10,001–$20,000

14,694 13,547

1,148

0.265

438

401

More than $20,000

23,653 23,345

309

0.752

526

545

Reported barrier to employment

Yes

12,317 13,028

– 712

0.417

534

517

No

21,567 20,627

940

0.248

748

749

Disability status

Received SSI/SSDI

8,313 8,278

35

0.978

177

188

Did not receive SSI/SSDI

19,254 19,102

153

0.817

1,105

1,078

(continued)

144

Exhibit E.1 (continued)

Sample Sizes:

FSS

Control

Difference

FSS

Control

Outcome

Group

Group

(Impact) P-Value

Group

Group

Percentage of rent and utilities expenses paid

by household

0–25

13,844 13,566

277

0.755

583

550

25.01–50

17,094 16,791

303

0.766

419

410

More than 50

26,427 25,870

557

0.685

280

306

Percentage of household income used for rent

and utilities

0–30

18,024 17,917

107

0.886

832

816

More than 30

17,570 16,382

1,188

0.228

450

450

Employed in all four quarters (%)

Employment status

Not employed

35.8

35.6

0.3

0.915

575

551

Employed Part-Time

65.8

62.1

3.7

0.331

335

313

Employed Full-Time

72.3

70.7

1.6

0.611

372

402

Educational attainment

No degree or credential

48.9

46.4

2.5

0.558

257

253

High school degree or GED

52.6

50.9

1.7

0.626

319

297

Some college

55.7

55.3

0.4

0.891

470

512

2-year college degree or higher

62.7

56.4

6.2

0.150

236

204

Total household income

$1–$10,000

44.5

40.4

4.1

0.266

318

320

$10,001–$20,000

49.6

45.4

4.2

0.199

438

401

More than $20,000

65.2

65.6

– 0.4

0.876

526

545

Reported barrier to employment

Yes

42.7

40.7

2.0

0.481

534

517

No

62.7

61.8

1.0

0.680

748

749

Disability status

Received SSI/SSDI

29.8

29.9

– 0.1

0.975

177

188

Did not receive SSI/SSDI

58.5

57.0

1.5

0.424

1,105

1,078

Percentage of rent and utilities expenses paid

by household

0–25

46.6

44.6

2.0

0.493

583

550

25.01–50

54.4

51.9

2.5

0.425

419

410

More than 50

70.3

70.3

– 0.1

0.986

280

306

(continued)

145

Exhibit E.1 (continued)

Sample Sizes:

FSS

Control

Difference

FSS Control

Outcome

Group

Group

(Impact) P-Value

Group

Group

Percentage of household income used for rent

and utilities

0–30

54.9

54.1

0.8 0.734

832

816

More than 30

54.4

50.5

3.9 0.184

450

450

Sample size (total = 2,548)

1,282

1,266

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Average quarterly employment rate is calculated as: total quarters with employment divided by total quarters of follow-up, expressed as a percentage. Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between research groups. The p-value indicates the likelihood that the difference between the FSS group and control group arose by chance. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; and

*** = 1 percent. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups. Statistical significance levels are indicated as: † = 10 percent; †† = 5 percent; and ††† = 1 percent.

Sources: MDRC calculations using baseline data, U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data, and quarterly wage data from the National Directory of New Hires 146

Exhibit E.2: Impacts on Selected Credit Score Outcomes by Selected Baseline

Characteristics, Family Self-Sufficiency Impact Sample

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome

Group

Group

(Impact)

P-Value

Group

Group

Average Experian Vantage 3.0 credit score

in 2019

Employment status

Not employed

561

564

– 3

0.546

545

524

Employed Part-Time

584

569

15 **

0.033

322

297

Employed Full-Time

585

581

3

0.581

366

397

Educational attainment

No degree or credential

567

575

– 8

0.324 †

246

237

High school degree or GED

574

568

6

0.413 †

303

281

Some college

562

565

– 3

0.593 †

455

501

2-year college degree or higher

605

586

19 **

0.022 †

229

199

Total household income

No income

559

544

15

0.155

134

120

$1–$10,000

558

560

– 3

0.592

424

410

$10,001–$20,000

578

581

– 2

0.690

344

340

More than $20,000

591

588

3

0.650

331

348

Reported barrier to employment

Yes

569

571

– 1

0.774

507

499

No

577

572

5

0.276

726

719

Disability status

Received SSI/SSDI

576

568

9

0.308

165

178

Did not receive SSI/SSDI

573

572

1

0.682

1068

1040

Rent burdena

Lower

574

574

0

0.958

568

569

Higher

573

568

5

0.274

576

581

Percentage of rent and utilities

expenses paid by household

0–25

563

560

3

0.545

561

525

25.01–50

579

578

1

0.898

400

392

More than 50

589

583

5

0.475

272

301

(continued)

147

Exhibit E.2 (continued)

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome

Group

Group

(Impact) P-Value Group

Group

Has Vantage 3.0 credit score of 661 or higher in 2019

(Prime) (%)

Employment status

Not employed

11.6

11.7

– 0.1

0.961 †† 575

551

**

Employed Part-Time

22.1

14.4

7.8 *

0.008 †† 335

313

Employed Full-Time

17.1

19.8

– 2.7

0.311 †† 372

402

Educational attainment

No degree or credential

15.0

16.7

– 1.7

0.579

257

253

High school degree or GED

15.8

12.3

3.4

0.216

319

297

Some college

11.5

13.7

– 2.2

0.286

470

512

2-year college degree or higher

26.1

19.3

6.8 *

0.095

236

204

Total household income

No income

12.8

6.3

6.5 *

0.091

139

129

$1– $10,000

11.9

11.2

0.7

0.761

449

426

$10,001–$20,000

17.6

17.0

0.6

0.827

353

354

More than $20,000

19.2

22.0

– 2.9

0.328

341

357

Reported barrier to employment

Yes

15.2

15.1

0.0

0.986

534

517

No

15.7

15.5

0.2

0.912

748

749

Disability status

Received SSI/SSDI

11.0

13.6

– 2.6

0.482

177

188

Did not receive SSI/SSDI

16.2

15.7

0.5

0.718

1105

1078

Rent burdena

Lower

16.2

15.9

0.3

0.878

591

593

Higher

14.8

13.6

1.2

0.540

599

604

Percentage of rent and utilities expenses

paid by household

0–25

14.2

10.6

3.6 *

0.061

583

550

25.01–50

16.5

18.2

– 1.7

0.506

419

410

More than 50

17.6

19.2

– 1.5

0.632

280

306

(continued)

148

Exhibit E.2 (continued)

Sample Sizes:

FSS

Control Difference

FSS Control

Outcome

Group

Group

(Impact)

P-Value Group

Group

Has Vantage 3.0 score only, no Clarity Clear

Early Risk score, in 2019 (%)

Employment status

Not employed

32.5

33.1

– 0.7

0.811

575

551

Employed Part-Time

33.1

31.9

1.2

0.734

335

313

Employed Full-Time

29.0

33.6

– 4.6

0.149

372

402

Educational attainment

No degree or credential

35.6

36.5

– 0.9

0.833

257

253

High school degree or GED

33.7

35.5

– 1.8

0.631

319

297

Some college

27.5

29.1

– 1.6

0.566

470

512

2-year college degree or higher

31.1

36.6

– 5.4

0.228

236

204

Total household income

No income

33.4

30.0

3.4

0.607 †

139

129

$1– $10,000

31.3

32.7

– 1.4

0.652 †

449

426

$10,001–$20,000

32.0

29.7

2.2

0.516 †

353

354

More than $20,000

29.7

38.8

– 9.1 ***

0.007 †

341

357

Reported barrier to employment

Yes

31.7

33.5

– 1.8

0.527

534

517

No

31.1

33.2

– 2.1

0.367

748

749

Disability status

Received SSI/SSDI

26.5

29.9

– 3.4

0.494

177

188

Did not receive SSI/SSDI

32.2

33.8

– 1.6

0.400

1105

1078

Rent burdena

Lower

34.4

36.5

– 2.1

0.417

591

593

Higher

27.7

30.0

– 2.2

0.389

599

604

Percentage of rent and utilities expenses

paid by household

0–25

33.9

33.5

0.4

0.890 †

583

550

25.01–50

31.0

30.0

1.0

0.742 †

419

410

More than 50

27.2

36.6

– 9.5 **

0.013 †

280

306

(continued)

149

Exhibit E.2 (continued)

Sample Sizes:

FSS Control Difference

FSS Control

Outcome

Group Group

(Impact)

P-Value Group Group

Average change in Vantage 3.0 scores, 2013 to 2019

Employment status

Not employed

18

23

– 4

0.415

533

511

Employed Part-Time

23

13

10

0.135

314

294

Employed Full-Time

19

25

– 6

0.383

361

396

Educational attainment

No degree or credential

17

20

– 3

0.706

238

231

High school degree or GED

23

26

– 3

0.676

295

276

Some college

19

24

– 5

0.397

450

496

2-year college degree or higher

22

8

14 *

0.083

225

198

Total household income

No income

26

15

11

0.383

132

118

$1–$10,000

11

24

– 13 **

0.037

410

402

$10,001–$20,000

26

22

4

0.504

342

334

More than $20,000

20

22

– 1

0.840

324

347

Reported barrier to employment

Yes

21

24

– 2

0.674

491

489

No

19

19

– 1

0.905

717

712

Disability status

Received SSI/SSDI

27

18

9

0.416

159

172

Did not receive SSI/SSDI

19

22

– 3

0.484

1049

1029

Rent burdena

Lower

15

22

– 8

0.138

553

560

Higher

24

19

6

0.255

567

573

Percentage of rent and utilities expenses

paid by household

0–25

18

20

– 2

0.707

546

515

25.01–50

19

22

– 3

0.660

394

386

More than 50

24

21

3

0.694

268

300

(continued)

150

Exhibit E.2 (continued)

Sample Sizes:

FSS Control Difference

FSS Control

Outcome

Group

Group

(Impact)

P-Value

Group

Group

Increased Vantage 3.0 score from 600 or lower (Deep Subprime/Subprime) to 661 or higher (Prime), 2013–19 (%)

Employment status

Not employed

7.0

7.4

– 0.4

0.802 533 511

Employed Part-Time

8.7

4.7

4.0 *

0.062 314 294

Employed Full-Time

5.8

6.8

– 1.0

0.577 361 396

Educational attainment

No degree or credential

7.4

4.1

3.2

0.173 238 231

High school degree or GED

7.1

6.6

0.5

0.819 295 276

Some college

6.6

6.7

– 0.1

0.978 450 496

2-year college degree or higher

7.7

8.9

– 1.2

0.684 225 198

Total household income

No income

11.2

1.9

9.3 ***

0.009 †

132

118

$1– $10,000

6.2

7.9

– 1.7

0.385 †

410

402

$10,001–$20,000

6.4

6.0

0.5

0.815 †

342

334

More than $20,000

7.0

7.3

– 0.3

0.872 †

324

347

Reported barrier to employment

Yes

8.4

6.5

1.8

0.293

491

489

No

6.1

6.7

– 0.6

0.650

717

712

Disability status

Received SSI/SSDI

8.5

6.7

1.8

0.596

159

172

Did not receive SSI/SSDI

6.8

6.6

0.2

0.827

1049

1029

Rent burdena

Lower

5.4

6.7

– 1.3

0.379

553

560

Higher

8.1

6.3

1.7

0.274

567

573

Percentage of rent and utilities expenses

paid by household

0–25

7.2

5.6

1.6

0.308

546

515

25.01–50

7.7

7.1

0.6

0.749

394

386

More than 50

5.7

7.6

– 1.9

0.384

268

300

Sample size (total=2548)

1,282

1,266

(continued)

151

Exhibit E.2 ( continued)

Notes: The FSS Impact sample includes Housing Choice Voucher heads of household who were randomly assigned between October 18, 2013, and December 22, 2014, and were age 18 to 61 at the time of random assignment. Sample si zes for specific outcomes may vary becaus e of mis sing valu es. Estimat

es wer e regression-a djust ed using

ordinary least

sq uares, controlling for differences in sam

ple mem ber char

acteristics r

ecord

ed at the tim

e of r andom ass ignment. No

s

pecial weights were applied to responses t o adjust for differ ences in sa mple s ize by housi ng aut hority. R ounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between the F SS group and the control group for contin uous var iables an d selected o utcom es expresse d as pr oportions . In addition,

a chi-square test for categorical variables w as run to determin e whether

there is a difference in the distribu

tion of related

outcomes by research group. The p-value i ndicates t he likeli

hood that the diffe rence betwee n the FSS grou p and control

group arose by chance. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; *** = 1 percent.

The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups.

St atistical significance levels are indicated as: †

= 10 per

cent; †† = 5

percen t; and †††

= 1 p

ercent. aBa sed on HUD

H ousing Choice Voucher program regulati

ons, ho

useholds

described as havi

ng a "lower r ent

burden" di d not have to

pa y more than required out-of-pocket e

xpenses f

or rent an d utilities i n m

onth 1 (the m

onth

of random assignment)

because their gross rent was less than or equal to the area payment standard (representing the maximum housing subsidy allowed) and because the household was paying less than or equal to 30 percent of their adjusted monthly income for re nt and utilities (representing the expected percenta ge of ho

usehold inco me t

o be paid for re nt an d utilities ). In contrast,

households described as having a "higher

rent bur den" pai

d more than requi red out-of-

pocket expenses for rent and

utilities because their gross rent was higher than the area payment standard and because the household was paying more than 30 percent of their adjusted monthly income for rent and utilities. Not shown in table: Results for 130 respondents (

7 percent) who had a combination of "lowe r" and " higher" r ent burden i n mo

nth 1.

So urces: MDRC calculations using baselin e data a nd Experi an Vantag

e 3.0 s cores data an d Cl arity Cl

ear Early Risk

s

cores data

152

Exhibit E.3: Impacts on Selected Credit Outcomes, by Selected Baseline Characteristics, Family Self-Sufficiency Impact Sample

Sample Sizes:

FSS Control

Difference

FSS Control

Outcome

Group

Group

(Impact) P-Value

Group

Group

Total balance (traditional and alternative financial services) in 2019 ($)

Employment status

Not employed

15,339

14,205

1,134

0.447 561 534

Employed Part-Time

19,227

19,716

– 490

0.827 329 306

Employed Full-Time

23,645

23,091

553

0.799 371 402

Educational attainment

No degree or credential

9,160

7,831

1,329

0.375 251 244

High school degree or GED

12,012

11,755

257

0.861 309 286

Some college

17,811

17,537

275

0.862 467 508

2-year college degree or higher

42,217

40,269

1,948

0.665 234 204

Total household income

No income

16,232

16,468

– 237

0.954

136

125

$1–$10,000

16,211

17,185

– 974

0.568

436

416

$10,001– $20,000

18,364

16,980

1,384

0.483

351

347

More than $20,000

23,803

21,859

1,945

0.426

338

354

Reported barrier to employment

Yes

17,244

16,423

821

0.640

524

510

No

20,196

19,547

648

0.640

737

732

Disability status

Received SSI/SSDI

17,118

13,580

3,538

0.247

174

183

Did not receive SSI/SSDI

19,165

19,176

– 11

0.993

1087

1059

Rent burdena

Lower

17,184

17,917

– 733

0.632

584

579

Higher

20,647

18,487

2,160

0.198

586

595

Percentage of rent and utilities expenses

paid by household

0–25

15,970

17,108

– 1,137

0.450

572

537

25.01–50

19,391

17,326

2,065

0.290

410

400

More than 50

23,501

22,442

1,059

0.687

279

305

(continued)

153

Exhibit E.3 (continued)

Sample Sizes:

FSS Control

Difference

FSS

Control

Outcome

Group

Group

(Impact) P-Value

Group

Group

Change in total balance since 2014 ($)

Employment status

Not employed

8,466

7,270

1,196

0.445 554 527

Employed Part-Time

10,228

10,532

– 304

0.898 324 301

Employed Full-Time

10,484

11,140

– 656

0.757 370 401

Educational attainment

No degree or credential

3,941

3,910

31

0.981 245 240

High school degree or GED

5,740

5,207

533

0.715 305 281

Some college

8,340

9,489

– 1,149

0.485 466 505

2-year college degree or higher

24,482

19,092

5,389

0.246 232 203

Total household income

No income

10,192

8,510

1,682

0.669

135

124

$1–$10,000

8,845

9,885

– 1,040

0.565

429

410

$10,001–$20,000

8,831

9,370

– 539

0.796

349

342

More than $20,000

10,685

9,086

1,599

0.503

335

353

Reported barrier to employment

Yes

8,327

8,012

315

0.852

517

503

No

10,340

10,274

66

0.964

731

726

Disability status

Received SSI/SSDI

7,415

7,259

156

0.959

173

178

Did not receive SSI/SSDI

9,878

9,666

212

0.858

1075

1051

Rent burdena

Lower

7,746

9,596

– 1,850

0.238 †

576

574

Higher

10,930

9,108

1,822

0.278 †

582

587

Percentage of rent and utilities expenses

paid by household

0–25

9,113

9,369

– 256

0.872

564

529

25.01–50

9,683

9,346

337

0.867

407

396

More than 50

9,811

9,528

283

0.911

277

304

(continued)

154

Exhibit E.3 (continued)

Sample Sizes:

FSS Control

Difference

FSS Control

Outcome

Group

Group

(Impact) P-Value Group

Group

Ever used alternative financial services in

2015–19 (%)

Employment status

Not employed

18.0

19.6

– 1.6 0.502

556

531

Employed Part-Time

21.9

21.6

0.3 0.930

327

304

Employed Full-Time

26.8

27.3

– 0.5 0.872

366

399

Educational attainment

No degree or credential

21.4

17.5

3.9 0.289

249

243

High school degree or GED

20.6

18.9

1.7 0.605

306

285

Some college

22.9

26.2

– 3.3 0.241

462

505

2-year college degree or higher

22.7

22.1

0.6 0.886

232

201

Total household income

No income

22.0

17.7

4.3 0.464

136

125

$1–$10,000

18.4

20.1

– 1.7 0.545

432

414

$10,001–$20,000

23.0

23.1

– 0.1 0.966

346

344

More than $20,000

26.4

24.4

2.0 0.537

335

351

Reported barrier to employment

Yes

21.5

19.8

1.7 0.498

516

506

No

22.6

23.5

– 0.9 0.685

733

728

Disability status

Received SSI/SSDI

19.3

25.9

– 6.7 0.168

168

180

Did not receive SSI/SSDI

22.4

21.6

0.8 0.660

1081

1054

Rent burdena

Lower

21.0

18.9

2.1 0.349 †

579

576

Higher

22.5

25.7

– 3.1 0.211 †

580

590

Percentage of rent and utilities expenses

paid by household

0–25

17.8

18.8

– 0.9 0.698

570

535

25.01–50

22.1

24.5

– 2.4 0.424

403

396

More than 50

29.5

25.9

3.6 0.339

276

303

Sample size(total=2548)

1,282

1,266

(c ontinued)

155

Exhibit E.3 (continued)

Notes: The FSS Impact sample includes Housing Choice Voucher heads of household who were randomly assigned bet ween October 18, 2013, and December 22, 2014, and were age

18 to 61 at the ti me

of random as si

gnment. Sam ple

si

zes for specific outcomes may vary becaus e of missin g values. E stimates were r egr ession-adjust e d using ordina ry

least squares, controlling for differences in sample member characteristics recorded at the time of random assignment.

No special weights were applied to responses to adjust for differences in sample size by housing authority. Rounding m

ay cause slight discrepancies in calculating sums an

d differen

ces. A two-tailed t-t est was appl ied t o differen

ces

bet ween the FSS group and the control gr oup for

continuous variables an

d sel ected outco mes expressed as

proportions. In addition, a chi-square test for categorical variables was run to determine whether there is a difference in the distribution of related outcomes by research group. The p-value indicates the likelihood that the difference bet ween the FSS group and control group ar ose by ch ance. Statis tical significan ce l evels are ind icat ed as: *

= 10

p

ercent; ** = 5 percent; *** = 1 percent. Th

e H-statis

tic test was u sed to test for s ta

tistically sign ifica nt differenc es

in

impact estimates across different subgroups . Statisti

cal signifi

cance levels are in di cated as: †

= 10 p ercent; ††

= 5

pe rcent; and ††† = 1 percent. aBased on HUD Housi

ng Choice V

oucher program r e

gulations, hous ehol ds descri bed

as having a "lower rent burden" did not have to pay more than required out-of-pocket expenses for rent and utilities i

n month 1 (the month of random assignme

nt) because their gross rent was less t ha

n or equal t

o the

area paym ent

sta ndard (representing the maximum housi

ng subsidy a llowed) a

nd because the

hous ehold was

payi

ng less tha

n or

e

qual to 30 percent of their adjusted monthl

y income f or rent a

nd utilities (repres enti ng the expec te

d percentage of

household income to be paid for rent and utilities). In contrast, households described as having a "higher rent burden"

paid more than required out-of-

pocket expenses for rent and utilities because their gross rent was higher than the area payment standard and because the household was paying more than 30 percent of their adjusted monthly income for rent and utilities. Not shown in table: Results for 130 respondents (7 percent) who had a combination of "lower" and

"higher" rent burden in month 1.

Sources: MDRC calculations using baseline data and Experian and Clarity credit data 156

Exhibit E.4: Impacts on Incidence of Credit Problems in 2019, by Selected Baseline Characteristics, Family Self-Sufficiency Impact Sample

Sample Sizes:

FSS Control Difference

FSS Control

Outcome (%)

Group

Group

(Impact)

P-Value

Group

Group

Experienced any recorded

credit problem

Employment status

Not employed

57.0

55.5

1.5

0.614

561

534

Employed Part-Time

63.4

63.9

– 0.4

0.908

329

306

Employed Full-Time

72.2

67.5

4.8

0.148

371

402

Educational attainment

No degree or credential

50.6

47.1

3.5

0.464

251

244

High school degree or GED

57.1

58.2

– 1.1

0.784

309

286

Some college

67.5

65.1

2.4

0.416

467

508

2-year college degree or higher

76.0

73.7

2.3

0.582

234

204

Total household income

No income

59.7

57.5

2.2

0.753

136

125

$1–$10,000

60.3

58.4

1.9

0.577

436

416

$10,001–$20,000

62.2

62.5

– 0.3

0.933

351

347

*

More than $20,000

71.3

63.3

8.0 *

0.026

338

354

Reported barrier to employment

Yes

60.9

58.3

2.6

0.390

524

510

No

64.9

63.5

1.5

0.545

737

732

Disability status

Received SSI/SSDI

59.7

55.8

3.9

0.483

174

183

Did not receive SSI/SSDI

64.3

61.8

2.5

0.217

1087 1059

Rent burdena

Lower

62.2

59.6

2.6

0.346

584

579

Higher

65.3

63.3

2.1

0.453

586

595

Percentage of rent and utilities

expenses paid by household

0–25

60.5

59.4

1.1

0.712

572

537

25.01–50

62.3

60.9

1.4

0.675

410

400

*

More than 50

71.8

63.9

7.9 *

0.042

279

305

(continued)

157

Exhibit E.4 (continued)

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome (%)

Group Group

(Impact)

P-Value

Group

Group

High debt-to-income ratio

Employment status

Not employed

19.1

18.0

1.0

0.671

559 535

Employed Part-Time

22.1

19.9

2.2

0.504

328 305

Employed Full-Time

17.7

18.5

– 0.8

0.779

371 401

Educational attainment

No degree or credential

16.1

13.2

2.9

0.398

249 243

High school degree or GED

15.4

18.0

– 2.6

0.411

309 287

Some college

19.2

18.8

0.5

0.857

467 507

2-year college degree or higher

27.5

27.5

0.0

0.999

233 204

Total household income

No income

24.2

11.5

12.7 **

0.027 ††

132

123

$1–$10,000

17.6

21.7

– 4.2

0.137 ††

436

416

$10,001–$20,000

19.2

21.7

– 2.6

0.425 ††

351

348

More than $20,000

19.7

15.0

4.7

0.113 ††

339

354

Reported barrier to employment

Yes

18.9

17.9

1.0

0.693

523

509

No

19.7

19.3

0.4

0.862

735

732

Disability status

Received SSI/SSDI

21.4

21.4

0.0

0.993

173

182

Did not receive SSI/SSDI

18.8

18.5

0.2

0.887

1085

1059

Rent burdena

Lower

17.3

18.7

– 1.4

0.549

585

580

Higher

22.1

17.8

4.3 *

0.072

586

596

Percentage of rent and utilities expenses

paid by household

0–25

19.6

18.8

0.8

0.732

569

538

25.01–50

20.1

19.9

0.3

0.930

411

399

More than 50

18.0

16.8

1.1

0.731

278

304

(continued)

158

Exhibit E.4 (continued)

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome (%)

Group Group

(Impact)

P-Value

Group

Group

Total balance (traditional financial services) greater than or equal to 75 percent of total credit Employment status

Not employed

38.5

40.2

– 1.7

0.544

575

551

Employed Part-Time

43.2

42.2

1.0

0.789

335

313

Employed Full-Time

48.5

49.4

– 1.0

0.789

372

402

Educational attainment

No degree or credential

28.6

28.3

0.2

0.955

257

253

High school degree or GED

38.5

36.8

1.7

0.669

319

297

Some college

46.5

48.9

– 2.4

0.458

470

512

2-year college degree or higher

56.1

59.2

– 3.1

0.500

236

204

Total household income

No income

44.6

41.9

2.8

0.688

139

129

$1–$10,000

44.0

42.8

1.2

0.718

449

426

$10,001–$20,000

39.6

42.2

– 2.6

0.469

353

354

More than $20,000

45.5

44.5

1.1

0.772

341

357

Reported barrier to employment

Yes

41.1

41.7

– 0.6

0.840

534

517

No

44.1

44.6

– 0.5

0.827

748

749

Disability status

Received SSI/SSDI

37.2

36.8

0.5

0.931

177

188

Did not receive SSI/SSDI

44.2

44.2

0.0

0.993

1105

1078

Rent burdena

Lower

43.1

41.2

2.0

0.479

591

593

Higher

42.7

46.0

– 3.3

0.240

599

604

Percentage of rent and utilities expenses

paid by household

0– 25

43.0

42.8

0.1

0.962

583

550

25.01–50

39.0

40.4

– 1.4

0.667

419

410

More than 50

49.2

47.8

1.5

0.727

280

306

(continued)

159

Exhibit E.4 (continued)

Sample Sizes:

FSS

Control

Difference

FSS

Control

Outcome (%)

Group

Group

(Impact)

P-Value

Group

Group

Any traditional financial services balance 90 days or more past scheduled

repayment date

Employment status

Not employed

15.6

14.5

1.2

0.595

561

534

Employed Part-Time

15.5

18.9

– 3.4

0.276

329

306

Employed Full-Time

19.2

18.8

0.4

0.889

371

402

Educational attainment

No degree or credential

11.0

10.9

0.1

0.980

251

244

High school degree or GED

12.3

15.8

– 3.5

0.234

309

286

Some college

21.4

18.6

2.8

0.284

467

508

2-year college degree or

higher

21.2

19.8

1.4

0.731

234

204

Total household income

*

No income

10.8

21.1

– 10.4 *

0.044

136

125

$1–$10,000

17.8

15.8

2.0

0.445

436

416

$10,001–$20,000

15.9

14.5

1.4

0.631

351

347

More than $20,000

18.9

18.9

0.0

0.999

338

354

Reported barrier to employment

Yes

13.7

15.3

– 1.6

0.478

524

510

No

18.6

18.3

0.3

0.895

737

732

Disability status

Received SSI/SSDI

13.9

13.0

1.0

0.801

174

183

Did not receive SSI/SSDI

17.1

17.7

– 0.7

0.687

1087

1059

Rent burdena

Lower

15.1

17.8

– 2.7

0.222

584

579

Higher

16.8

16.5

0.3

0.896

586

595

Percentage of rent and utilities expenses paid by household

0–25

15.6

17.3

– 1.7

0.460

572

537

25.01–50

18.1

14.7

3.4

0.202

410

400

More than 50

17.5

18.7

– 1.2

0.715

279

305

Incurred late payment, loan collection, or charge off for alternative

financial service

Employment status

Not employed

6.3

5.1

1.2

0.396

556

531

Employed Part-Time

6.4

9.2

– 2.7

0.221

327

304

Employed Full-Time

11.0

8.7

2.2

0.317

366

399

Educational attainment

No degree or credential

6.1

7.0

– 0.9

0.709 †

249

243

High school degree or GED

8.3

4.5

3.8 *

0.073 †

306

285

Some college

6.9

10.3

– 3.4 *

0.063 †

462

505

2-year college degree or

higher

8.0

6.7

1.4

0.615 †

232

201

(continued)

160

Exhibit E.4 (continued)

Sample Sizes:

FSS Control

Difference

FSS

Control

Outcome (%)

Group

Group

(Impact)

P-Value

Group

Group

Total household income

No income

9.7

6.3

3.4

0.385

136

125

$1–$10,000

5.8

6.8

– 1.0

0.549

432

414

$10,001–$20,000

7.9

5.7

2.1

0.276

346

344

More than

$20,000

9.7

9.3

0.4

0.857

335

351

Reported barrier to

employment

Yes

8.0

4.9

3.1 *

0.053 ††

516

506

No

7.4

9.0

– 1.6

0.264 ††

733

728

Disability status

Received

SSI/SSDI

6.0

7.7

– 1.7

0.579

168

180

Did not receive

108

SSI/SSDI

7.7

7.5

0.2

0.875

1

1054

Rent burdena

Lower

7.1

7.8

– 0.8

0.622

579

576

Higher

8.0

7.5

0.5

0.754

580

590

Percentage of rent and

utilities expenses

paid by household

0–25

7.1

6.1

1.0

0.529 ††

570

535

25.01–50

5.1

9.0

– 3.9 **

0.037 ††

403

396

More than 50

12.0

7.9

4.1

0.107 ††

276

303

Sample size (total=2548)

1,282

1,266

Notes: The FSS Impact sample includes Housing Choice Voucher heads of household who were randomly assigned between October 18, 2013, and December 22, 2014, and were age 18 to 61 at the time of random assignment. Sample sizes for specific outcomes may vary because of missing values. Estimates were regression-adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment. No speci

al weights were ap plied to r

esponses to adjust for d ifferen

ces in sample s ize b y housi

ng authority. Rounding may

caus

e slight discrepanci es in calcu lating sum s and differences. A two-tailed t-test was appli ed to differ en ces b et ween

the FSS group and the control group for continuous variables and selected outcomes expressed as proportions. In additi on, a chi-square t

est for cat

egorical vari ables was run to determine whether there is a diff erence in t h e dist ribu tion

of rela ted outcomes

by research gr oup. The p-value indicates the likelihood that the differenc e betwee n the FS S

group

and co ntrol group arose by chance . Statistical significance levels are indicated as: * = 10 per cent; ** = 5 per cen t; * ** =

1 percent. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups. Statistical significance levels are indicated as: † = 10 percent; †† = 5 percent; and ††† = 1 percent. aBased on HU D Housing Choi ce Voucher program r egulations, households described as having a "l ower rent

bur

den" di

d not

have t o pay more tha

n require

d out-of-

pocket expenses for rent and utilities in mont

h 1 (the m

ont

h of ra

ndom

assi

gnment) because t

heir gross re nt was les s than or equal to the area payment standard (re presenti ng the m axi mum

housing subsidy allowed) and because the household was paying less than or equal to 30 percent of their adjusted monthly income for rent and utilities (representing the expected percentage of household income to be paid for rent and utilitie s). In contrast,

households described as having a "higher rent burden" paid more tha n required out-of -po cket

expe

nses for rent and utilities be cause their gross rent was higher than the area payment s tandard a nd beca use the

household was paying more than 30 percent of their adjusted monthly income for rent and utilities. Not shown in table: Results for 130 respondents (7 percent) who had a combination of "lower" and "higher" rent burden in month 1.

Sources: MDRC calculations using baseline data and Experian and Clarity credit data 161

Appendix Exhibit E.5: Impacts on Selected Indicators of Rent and Subsidies in Years 1

to 5, by Selected Baseline Characteristics, Family Self-Sufficiency Impact Sample

Sample Sizes:

FSS Control

Difference

FSS

Control

Outcome

Group

Group

(Impact)

P-Value

Group

Group

Enrolled in HCV

program in month 60

(%)

Employment status

Not employed

73.3

73.4

– 0.1

0.977

572

550

Employed part-time

71.4

73.8

– 2.4

0.506

336

312

Employed full-time

61.0

59.1

1.9

0.589

373

405

Educational attainment

No degree or

credential

71.9

72.8

– 0.9

0.826

257

256

High school degree

or GED

69.9

71.4

– 1.5

0.681

319

297

Some college

69.6

68.6

0.9

0.752

470

510

2-year

college

degree or higher

62.2

64.1

– 1.9

0.700

235

204

Total household income

No income

62.8

72.0

– 9.2

0.204

115

108

$1–$10,000

74.9

73.1

1.9

0.548

433

410

$10,001–$20,000

72.7

74.0

– 1.3

0.708

342

352

More than $20,000

59.6

61.5

– 1.9

0.590

391

397

Reported barrier to

employment

Yes

72.7

70.1

2.6

0.365

532

516

No

66.0

69.0

– 3.0

0.201

749

751

Disability status

Received SSI/SSDI

70.1

74.2

– 4.1

0.427

176

187

Did not receive

SSI/SSDI

68.9

68.2

0.7

0.728

1105

1080

Percentage of rent and utilities expenses paid

by household

0–25

74.1

75.4

– 1.3

0.630

582

550

25.01–50

69.7

71.6

– 1.9

0.551

419

408

More than 50

58.1

53.9

4.2

0.299

280

307

Rent burdena

Lower

69.0

73.0

– 4.0

0.134

585

587

Higher

69.4

65.8

3.6

0.184

579

584

Average gross rent in month 60,

if received HCV ($)

Employment status

Not employed

1,514

1,499

15

421

402

Employed part-time

1,554

1,556

– 2

237

233

Employed full-time

1,564

1,608

– 44

229

238

(continued)

162

Appendix Exhibit E.5 (continued)

Sample Sizes:

FSS Control

Difference

FSS

Control

Outcome

Group

Group

(Impact)

P-Value

Group

Group

Educational attainment

No degree or

credential

1,572

1,636

– 64

183

188

High school degree

or GED

1,491

1,480

11

225

210

Some college

1,510

1,516

– 5

332

345

2-year college

degree or higher

1,611

1,609

2

147

130

Total household income

No income

1,323

1,384

– 61

74

76

$1–$10,000

1,433

1,477

– 43

325

299

$10,001–$20,000

1,533

1,544

– 11

250

259

More than $20,000

1,736

1,697

39

238

239

Reported barrier to

employment

Yes

1,476

1,485

– 10

388

360

No

1,582

1,589

– 8

499

513

Disability status

Received SSI/SSDI

1,448

1,377

70

121

141

Did not receive

SSI/SSDI

1,556

1,572

– 16

766

732

Percentage of rent and utilities

expenses paid by household

0–25

1,561

1,594

– 34

430

416

25.01–50

1,525

1,497

28

293

291

More than 50

1,504

1,497

7

164

164

Rent burdena

Lower

1,483

1,456

27

405

427

Higher

1,541

1,577

– 36

403

383

Average family share in month 60, if received HCV ($)

Employment status

Not employed

492

463

29

421

402

Employed part-time

587

562

25

237

233

Employed full-time

783

764

19

229

238

Educational attainment

No degree or

credential

564

600

– 36

183

188

High school degree

or GED

611

577

34

225

210

Some college

551

536

16

332

345

2-year college

degree or higher

727

578

149

147

130

Total household income

No income

462

415

47

74

76

$1–$10,000

428

438

– 10

325

299

$10,001–$20,000

600

553

47

250

259

More than $20,000

831

827

3

238

239

(continued)

163

Appendix Exhibit E.5 (continued)

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome

Group

Group

(Impact)

P-Value

Group

Group

Reported barrier to

employment

Yes

506

515

– 9

388

360

No

655

615

39

499

513

Disability status

Received SSI/SSDI

491

496

– 5

121

141

Did not receive

SSI/SSDI

609

585

24

766

732

Percentage of rent and utilities expenses paid

by household

0–25

484

475

9

430

416

25.01–50

619

598

21

293

291

More than 50

807

794

14

164

164

Rent burdena

Lower

546

523

23

405

427

Higher

616

590

26

403

383

Average housing subsidy in month 60, if received HCV ($)

Employment status

Not employed

1,023

1,043

– 19

421

402

Employed part-time

982

1,001

– 18

237

233

Employed full-time

810

860

– 50

229

238

Educational attainment

No degree or

credential

1,019

1,036

– 17

183

188

High school degree

or GED

880

910

– 30

225

210

Some college

976

994

– 18

332

345

2-year college

degree or higher

923

1,025

– 102

147

130

Total household income

No income

880

964

– 84

74

76

$1–$10,000

1,009

1,045

– 36

325

299

$10,001–$20,000

945

994

– 49

250

259

More than $20,000

932

887

45

238

239

Reported barrier to

employment

Yes

983

980

3

388

360

No

938

982

– 44

499

513

Disability status

Received SSI/SSDI

961

900

61

121

141

Did not receive

SSI/SSDI

961

993

– 31

766

732

Percentage of rent and utilities expenses paid

by household

0–25

1,082

1,123

– 41

430

416

25.01–50

916

907

8

293

291

More than 50

735

725

10

164

164

(continued)

164

Appendix Exhibit E.5 (continued)

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome

Group

Group

(Impact)

P-Value

Group

Group

Rent burdena

Lower

944

941

3

405

427

Higher

942

995

– 53

403

383

Average total family share in

years 1 to 5 ($)

Employment status

Not employed

20,527 19,591

936

0.210

562

539

Employed part-time

26,063 25,492

572

0.621

329

305

Employed full-time

32,775 30,986

1,789

0.125

365

396

Educational attainment

No degree or

credential

25,174 25,084

90

0.948

252

250

High school degree

or GED

25,036 24,233

803

0.473

316

290

Some college

25,266 23,663

1,603 *

0.075

457

498

2-year college

degree or higher

28,695 25,574

3,122 **

0.038

231

202

Total household income

No income

16,726 15,991

735

0.669

111

104

$1–$10,000

17,682 17,084

598

0.465

427

403

$10,001–$20,000

26,429 25,203

1,226

0.200

340

349

More than $20,000

35,799 34,923

875

0.482

378

384

Reported barrier to

employment

Yes

23,657 22,594

1,063

0.208

526

506

No

27,058 25,954

1,104

0.142

730

734

Disability status

Received SSI/SSDI

23,560 23,880

– 320

0.815

173

186

Did not receive

SSI/SSDI

26,006 24,665

1,341 **

0.029

1083

1054

Percentage of rent and utilities expenses paid

by household

0–25

19,063 18,439

624

0.387

572

539

25.01–50

28,810 28,068

741

0.433

417

404

More than 50

33,920 31,865

2,055

0.152

267

295

Rent burdena

Lower

23,415 22,339

1,076

0.156

578

578

Higher

27,505 26,265

1,240

0.149

571

576

Average total housing subsidy in

years 1 to 5 ($)

Employment status

Not employed

48,955 48,826

129

0.918

564

542

Employed part-time

44,639 45,917

– 1,278

0.434

331

304

Employed full-time

34,471 33,737

733

0.633

363

392

(continued)

165

Appendix Exhibit E.5 (continued)

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome

Group

Group

(Impact)

P-Value

Group

Group

Educational attainment

No degree or

credential

45,604 48,170

– 2,566

0.189

254

246

High school degree

or GED

42,300 42,629

– 329

0.830

315

290

Some college

42,617 43,451

– 834

0.535

459

500

2-year college

degree or higher

42,416 41,505

911

0.679

230

202

Total household income

No income

41,786 41,973

– 188

0.952

112

106

$1–$10,000

48,915 49,206

– 290

0.840

430

403

$10,001–$20,000

45,063 46,058

– 995

0.524

336

347

More than $20,000

35,581 36,413

– 832

0.601

380

382

Reported barrier to

employment

Yes

44,895 44,289

606

0.640

524

506

No

41,952 43,468

– 1,516

0.153

734

732

Disability status

Received SSI/SSDI

40,254 41,610

– 1,356

0.516

174

186

Did not receive

SSI/SSDI

43,993 43,834

159

0.857

1084

1052

Percentage of rent and utilities expenses paid

by household

0–25

52,678 52,740

– 62

0.961

571

539

25.01–50

41,887 42,828

– 941

0.507

417

402

More than 50

27,494 26,594

899

0.568

270

295

Rent burdena

Lower

42,746 43,603

– 857

0.467

578

581

Higher

43,243 43,291

– 48

0.969

573

573

Sample size

(total =2,548)

1,281

1,267

(continued)

166

Appendix Exhibit E.5 (continued)

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment.

Sample sizes for specific outcomes may vary because of missing values. Estimates were regression-adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment. No special weights were applied to responses to adjust for differences in sample size by housing authority. Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied to differences between the FSS group and the control group for continuous variables and selected outcomes expressed as proportions. In addition, a chi-square test for categorical variables was run to determine whether there is a difference in the distribution of related outcomes by research group. The p-value indicates the likelihood that the difference between the FSS group and control group arose by chance. Statistical significance levels are indicated as: * = 10 percent; ** = 5 percent; *** = 1 percent. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups.

Statistical significance levels are indicated as: † = 10 percent; †† = 5 percent; and ††† = 1 percent. aBased on HUD Housing Choice Voucher program regulations, households described as having a "lower rent burden" did not have to pay more than required out-of-pocket expenses for rent and utilities in month 1 (the month of random assignment) because their gross rent was less than or equal to the area payment standard (representing the maximum housing subsidy allowed) and because the household was paying less than or equal to 30 percent of their adjusted monthly income for rent and utilities (representing the expected percentage of household income to be paid for rent and utilities). In contrast, households described as having a "higher rent burden" paid more than required out-of-pocket expenses for rent and utilities because their gross rent was higher than the area payment standard and because the household was paying more than 30 percent of their adjusted monthly income for rent and utilities. Not shown in table: Results for 160 respondents (6 percent) who had a combination of "lower" and "higher" rent burden in month 1. Results displayed in italics are non-experimental. No tests of statistical significance were performed on differences between research groups in means or proportions.

Sources: MDRC calculations using baseline data, U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data, and responses to the FSS 36-Month Survey 167

Exhibit E.6 Impacts on Employment and Earnings in Years 1 to 5, by Program Approach, Family Self-Sufficiency Impact Sample

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome

Group Group

(Impact)

P-Value

Group

Group

In Years 1 to 5 Average quarterly

employment rate (%)

Emphasis on job search and post-

employment services

Low

65.7

64.6

1.0

0.642

286

289

Medium

62.3

60.7

1.6

0.482

351

352

High

64.3

64.5

– 0.2

0.911

645

625

Emphasis on education and training

Low

64.6

61.8

2.7

0.186 †

386

392

Medium

63.4

65.9

– 2.5

0.131 †

578

568

High

64.4

61.4

3.0

0.204 †

318

306

Emphasis on financial services

Low

59.3

59.4

– 0.1

0.979

239

235

Medium

63.5

64.2

– 0.7

0.739

420

419

High

66.2

64.6

1.6

0.318

623

612

Program emphasis on monitoring and engagement

Low

61.9

61.0

0.9

0.656

469

468

Medium

66.8

65.3

1.4

0.384

551

547

High

61.9

64.4

– 2.5

0.329

262

251

Total earnings ($)

Emphasis on job search and post-employment

services

Low

72,944 74,201

– 1,257

0.754

286

289

Medium

76,811 70,118

6,693 *

0.086

351

352

High

76,504 79,069

– 2,565

0.395

645

625

Emphasis on education and training

Low

73,721 69,566

4,156

0.244

386

392

Medium

74,931 78,964

– 4,033

0.182

578

568

High

79,727 76,699

3,028

0.500

318

306

Emphasis on financial services

Low

62,399 63,511

– 1,112

0.789

239

235

Medium

76,611 79,497

– 2,886

0.432

420

419

High

80,490 77,193

3,297

0.274

623

612

(continued)

168

Exhibit E.6 (continued)

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome

Group Group (Impact)

P-Value

Group

Group

Program emphasis on monitoring and engagement

74,3

††

Low

27 70,762

3,565

0.297 †

469

468

76,9

††

Medium

39 73,770

3,169

0.291 †

551

547

75,5

**

††

High

46 88,435 – 12,889 *

0.008 †

262

251

Average annual earnings greater than

$25,000 (%)

Emphasis on job search and post-

employment services

Low

22.0

24.6

– 2.5

0.423

286

289

Medium

22.9

21.8

1.1

0.668

351

352

High

23.8

24.9

– 1.1

0.573

645

625

Emphasis on education and training

Low

22.1

19.3

2.9

0.251

386

392

Medium

22.7

25.3

– 2.6

0.222

578

568

High

25.9

26.7

– 0.7

0.803

318

306

Emphasis on financial services

Low

16.5

15.5

1.0

0.723

239

235

Medium

24.5

25.6

– 1.1

0.664

420

419

High

24.9

26.0

– 1.1

0.611

623

612

Program emphasis on monitoring and

engagement

Low

23.9

22.0

1.8

0.425

469

468

Medium

22.2

22.1

0.1

0.970

551

547

High

24.6

30.9

– 6.3 **

0.046

262

251

In Year 5 Average quarterly

employment rate (%)

Emphasis on job search and post-

employment services

Low

66.9

66.6

0.3

0.916

286

289

Medium

62.5

59.6

2.8

0.342

351

352

High

65.2

66.1

– 0.9

0.678

645

625

Emphasis on education and training

Low

66.3

62.8

3.5

0.195

386

392

Medium

64.2

66.5

– 2.2

0.316

578

568

High

64.4

62.5

1.9

0.551

318

306

(continued)

169

Exhibit E.6 (continued)

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome

Group Group

(Impact)

P-Value

Group

Group

Emphasis on financial services

Low

62.3

62.4

– 0.1

0.978

239

235

Medium

65.3

63.6

1.6

0.537

420

419

High

65.4

65.8

– 0.4

0.850

623

612

Program emphasis on monitoring and engagement

Low

60.8

61.3

– 0.5

0.844

469

468

Medium

68.1

66.9

1.2

0.586

551

547

High

65.9

64.2

1.7

0.610

262

251

Total earnings ($)

Emphasis on job search and post-employment

services

Low

17,210 17,438

– 228

0.848

286

289

Medium

17,242 15,730

1,513

0.178

351

352

High

18,195 18,574

– 379

0.667

645

625

Emphasis on education and training

Low

17,466 16,413

1,053

0.312

386

392

Medium

17,606 18,672

– 1,066

0.239

578

568

High

18,279 16,746

1,533

0.222

318

306

Emphasis on financial services

Low

15,408 16,711

– 1,303

0.312

239

235

Medium

18,297 18,281

16

0.988

420

419

High

18,167 17,357

809

0.340

623

612

Program emphasis on monitoring and engagement

Low

17,095 16,379

716

0.474

469

468

Medium

18,060 17,401

660

0.453

551

547

High

18,072 19,952

– 1,880

0.193

262

251

Employed during all four quarters (%)

Emphasis on job search and post-employment services

Low

59.2

56.0

3.2

0.372

286

289

Medium

50.3

47.5

2.8

0.431

351

352

High

54.5

55.0

– 0.5

0.858

645

625

Emphasis on education and training

Low

58.7

51.9

6.8 **

0.033 ††

386

392

Medium

51.5

54.8

– 3.2

0.229 ††

578

568

High

55.2

50.8

4.4

0.225 ††

318

306

(continued)

170

Exhibit E.6 (continued)

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome

Group Group (Impact)

P-Value

Group

Group

Emphasis on financial services

Low

53.4

52.1

1.3

0.741

239

235

Medium

55.5

52.7

2.8

0.367

420

419

High

54.4

53.4

1.0

0.687

623

612

Program emphasis on monitoring and engagement

Low

51.8

51.3

0.5

0.857

469

468

Medium

57.4

53.8

3.6

0.186

551

547

High

53.6

54.0

– 0.3

0.937

262

251

Sample size (

total = 2,548 )

1,282

1,266

Notes: The FSS impact sample includes Housing Choice V oucher h ouseholds that wer e rand omly as signed bet ween October 18, 2013, and December 22, 2014, and with a head of household age 18 to 61 at the time of random assignment. Average quarterly employment rate is calculated as: total quarters with employment divided by total quarters

of follow-up, expressed as a percent

age. Esti

mates wer e regression -adj

usted usin

g ordi

nary leas t squares,

controlli

ng for pre-random assignment c

haracteris tics of

sample me mbers . Roundi ng

may ca use slight

discrepan

cies in calculating sums and differen ces. A t wo-tailed t-test was

applied to differ ences betw

een research

groups. The p-value indicates the likelihood that the difference between the FSS group and control group arose by chance. St atistical significance levels are indi cated as: * = 10 p

ercent; **

= 5 per

cent; and ***

= 1 per

cent. The H-

statistic te st was used to test for statistically s

ignifican t differen ces in impac t es

timates acr oss dif ferent s ubgroups.

Statistica

l significance levels are indicated as: † = 10 percent;

†† = 5 perce nt; a

nd ††† =

1 perce nt. "Co nfirmatory

comparisons" use qualitative and quantitative data collected around the time of random assignment outcomes or data collected for FSS program participants who enrolled in FSS before the start of the evaluation and are not included

in the FSS impact sample.

Sources:

MDRC calculations using baselin e data, data colle cted from I ndivi dual Trai ning a nd Ser vices Plan forms, inf ormation provided by FSS administr ators a

nd case ma nagers, a

nd quart erly wage data from t

he National

Directory of New Hires

171

Exhibit E.7: Impacts on Credit Scores in 2019, by Program Approach, Family

S elf-Sufficiency Impact Sample

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome

Group Group

(Impact) P-Value Group

Group

Average Experian Vantage 3.0 credit score in 2019

Emphasis on job search and post-employment

services

Low

553

555

– 3

0.662

273

279

Medium

583

585

– 2

0.801

334

335

High

577

572

5

0.337

626

604

Emphasis on education and

training

Low

571

570

1

0.895

368

377

Medium

567

561

7

0.154

556

546

High

587

594

– 7

0.339

309

295

Emphasis on financial

services

Low

553

563

– 10

0.175 †

232

226

Medium

578

566

12 **

0.038 †

395

402

High

578

580

– 1

0.768 †

606

590

Program emphasis on

monitoring and engagement

Low

586

585

1

0.859 †

451

449

Medium

558

563

– 5

0.298 †

530

527

High

582

566

16 **

0.035 †

252

242

Has Vantage 3.0 credit score of 661 or higher

in 2019 (Prime) (%)

Emphasis on job search and post-

employment services

Low

8.9

10.6

– 1.8

0.489

286

289

Medium

17.3

17.4

0.0

0.992

351

352

High

17.4

16.4

1.0

0.597

645

625

Emphasis on education and training

Low

14.4

12.9

1.5

0.537

386

392

Medium

13.7

12.5

1.2

0.518

578

568

High

20.8

23.2

– 2.4

0.439

318

306

Emphasis on financial services

Low

9.0

14.7

– 5.6

*

0.056 †

239

235

Medium

16.0

12.6

3.3

0.153 †

420

419

High

17.6

17.6

0.0

0.987 †

623

612

Program emphasis on

monitoring and engagement

Low

20.7

18.8

1.9

0.450

469

468

Medium

9.9

12.5

– 2.7

0.152

551

547

High

18.1

15.0

3.1

0.315

262

251

(continued)

172

Exhibit E.7 (continued)

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome

Group Group (Impact) P-Value

Group

Group

H

as Vantage 3.0 score only, no Clarity Clear Early Risk score, in 2019 (%)

Emphasis on job search and post-employment services

Low

29.5

28.3

1.2

0.753

286

289

Medium

37.5

39.3

– 1.7

0.635

351

352

High

29.2

31.8

– 2.6

0.273

645

625

Emphasis on education and training

Low

30.9

29.0

1.9

0.568

386

392

Medium

25.7

29.8

– 4.1

0.115

578

568

High

42.2

45.0

– 2.8

0.470

318

306

Emphasis on financial services

Low

24.7

27.2

– 2.5

0.542

239

235

Medium

31.2

33.7

– 2.5

0.421

420

419

High

33.2

36.2

– 3.0

0.245

623

612

Program emphasis on monitoring and engagement

Low

37.8

41.4

– 3.7

0.227

469

468

Medium

25.4

27.1

– 1.7

0.525

551

547

High

33.1

30.8

2.3

0.568

262

251

Average change in Vantage 3.0 scores, 2013 to 2019

Emphasis on job search and post-employment services

Low

10

22

– 12

0.113

268

274

Medium

22

25

– 3

0.625

326

330

High

22

20

2

0.749

614

597

Emphasis on education and training

Low

7

21

– 13 **

0.037 †

363

369

Medium

28

22

6

0.256 †

550

540

High

18

24

– 6

0.369 †

295

292

Emphasis on financial services

Low

11

23

– 12

0.156

232

221

Medium

24

18

6

0.339

388

399

High

20

24

– 4

0.396

588

581

Program emphasis on monitoring and engagement

Low

22

23

– 1

0.899 †††

437 444

Medium

12

25

– 13 **

0.011 †††

524

516

High

30

10

20 **

0.014 †††

247

241

(continued)

173

Exhibit E.7 (continued)

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome

Group Group (Impact)

P-Value

Group

Group

Increased Vantage 3.0 score from 600 or lower (Deep Subprime/Subprime)

to 661 or higher (Prime), 2013–19 (%)

Emphasis on job search and post-employment services

Low

3.5

7.9

– 4.4 *

0.035 † 268 274

Medium

6.3

7.4

– 1.1

0.576 † 326 330

High

8.5

6.0

2.4

0.110 † 614 597

Emphasis on education and

training

Low

5.1

6.6

– 1.5

0.400 363 369

Medium

8.3

7.1

1.3

0.439 550 540

High

5.7

6.9

– 1.2

0.572 295 292

Emphasis on financial

services

Low

5.5

8.8

– 3.3

0.193 232 221

Medium

7.0

6.0

1.0

0.586 388 399

High

7.1

6.7

0.4

0.798 588 581

Program emphasis on monitoring and engagement

Low

7.8

6.3

1.5

0.398 437 444

Medium

5.5

7.6

– 2.2

0.161 524 516

High

8.5

5.4

3.0

0.212 247 241

Sample size (total=2548)

1,282

1,266

Notes: The FSS Impact sample includes Housing Choice Voucher heads of household who were randomly assigned between Oc tober 18, 2013, and December 22, 2014, and were age 18 to 61 a t the time of ra ndom assi gnment. Sample siz es for specific

outcomes may vary because of missing values. Estimates were regression-adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the time of random assignment. No special weights were applied to responses to adjust for diff erences in sample size

by housi

ng authority. R

ounding may caus e slight di screpancies in

cal culating sums and differenc es. A two-tailed t-test was applied t

o differences be tween the FS S gr oup and the control group

for continuous variables and selected outcomes expressed as proportions. In addition, a chi-square test for categorical variables was run to determine whether there is a difference in the distribution of related outcomes by research group. The p-value indicates the likeliho

od that the difference bet

ween the F SS group a nd

control grou p ar ose by cha

nce. Statistical

sig nificance levels are indicat

ed as: * = 10 percent; **

= 5 percent ; *** = 1 p er

cent. The H-sta tis

tic test was

used to test for

sta tistically significant differen ces in impact estimates ac ross differ ent subgrou ps. Statistical s ignif icance leve ls are indicated

as: † = 10 percent; †† = 5 percent; and ††† = 1 percent. aBased on HUD Housing Choice Voucher program regulations, households described as having a "lower rent burden" did not have to pay more than required out-of-pocket expenses for re

nt and utilities in month 1 (t he month of random assi gnment)

because their gr oss rent was

less than or eq ual to the area

pa

yment standard (representin g the maximum housing s ubsidy all owed) a

nd bec ause the hous ehol d was payi ng less than or

e

qual to 30 percent of their adj usted monthly income for rent and ut ilities (repr ese nting the expe cte d percenta ge of household

income to be paid for rent and utilities). In contrast, households described as having a "higher rent burden" paid more than re

quired out-of-pocket expense s for rent and utilities bec ause their gross rent was higher than t he a rea payme

nt standard and

bec ause the household was pa

ying more than 30 perce

nt of their a djusted m

onthl y income for re

nt and utiliti es. Not shown

in t able: Results for 130 respo ndents (7 percent) w

ho had a combi nation of "l ow er" and "hi

gher" rent burde n in month.

Sources: MDRC calculations using baseline data and Experian Vantage 3.0 credit scores and Clarity Clear Early Risk scores data

174

Exhibit E.8: Impacts on Incidence of Credit Outcomes in 2019, by Program Approach, Family Self-Sufficiency Impact Sample

Sample Sizes:

FSS Control

Difference

FSS Control

Outcome

Group

Group

(Impact)

P-Value

Group

Group

Total balance (traditional and alternative financial services) in 2019 ($)

Emphasis on job search and post-employment services

Low

20,260

18,799

1,461

0.534

279

286

Medium

18,370

18,109

262

0.897

344

342

High

19,082

17,733

1,349

0.379

638

614

Emphasis on education and

training

Low

20,869

19,908

962

0.628

377

286

Medium

18,151

16,541

1,610

0.311

569

559

High

18,188

19,362

– 1,175

0.612

315

301

Emphasis on financial services

Low

17,422

18,909

– 1,487

0.550

235

232

Medium

20,241

16,816

3,425 *

0.075

408

408

High

19,117

18,587

530

0.735

618

602

Program emphasis on monitoring and

engagement

Low

15,479

15,294

185

0.902

463

457

Medium

20,986

20,012

974

0.574

542

539

High

21,013

19,949

1,064

0.708

256

246

Change in total balance since

2014 ($)

Emphasis on job search and post-employment services

Low

11,349

10,803

546

0.816

276

282

Medium

9,192

9,688

– 497

0.807

339

341

High

9,290

8,041

1,249

0.428

633

606

Emphasis on education and

training

Low

9,736

9,476

260

0.902

373

376

Medium

9,649

8,667

982

0.556

567

553

High

9,456

9,936

– 481

0.818

308

300

Emphasis on financial services

Low

8,171

9,138

– 968

0.716

235

228

Medium

10,999

7,741

3,258 *

0.096

403

405

High

9,660

9,880

– 220

0.888

610

596

Program emphasis on monitoring and

engagement

Low

7,498

7,926

– 428

0.796

455

452

Medium

11,275

11,024

251

0.889

539

531

High

10,100

7,565

2,536

0.317

254

246

(continued)

175

Exhibit E.8 (continued)

Sample Sizes:

FSS

Control

Difference

FSS Control

Outcome

Group

Group

(Impact) P-Value Group Group

Ever used alternative financial services in 2015–19 (%)

Emphasis on job search and post-employment services

Low

26.4

24.9

1.5

0.702

277

285

Medium

16.3

17.3

– 1.0

0.724

340

340

High

23.2

23.5

– 0.3

0.908

632

609

Emphasis on education and training

Low

26.9

30.6

– 3.8

0.258

373

381

Medium

23.9

23.1

0.8

0.756

562

555

High

12.3

9.9

2.4

0.371

314

298

Emphasis on financial services

Low

33.2

31.0

2.2

0.622

234

230

Medium

22.8

24.5

– 1.7

0.568

403

408

High

17.7

16.6

1.0

0.633

612

596

Program emphasis on monitoring and engagement

Low

19.4

20.6

– 1.2

0.646

459

456

Medium

24.5

24.7

– 0.2

0.950

536

532

High

21.8

19.0

2.8

0.457

254

246

Sample size(total=2548)

1,282

1,266

Notes: The FSS Impact sample includes Housing Choice Voucher heads of household who were randomly assigned between October 18, 2013, and December 22, 2

014, and were age 18 t

o 61 at the time of ran dom assig nm ent. Sam

ple sizes for specific

o utcomes may vary because of missing

values. Estim ates were r

egression-adjust e

d using ordi

nar

y least s

quares, controlling for

differences in sample member characteristics recorded at the time of random assignment. No special weights were applied to responses to adjust for differences in sample size by housing authority. Rounding may cause slight discrepancies in calculating s ums and differences. A two-tailed t-test was applied t o difference s between the F SS group a nd the control g roup for continuous

v ariables and selected outcomes expres sed as propor tions. In a

ddition, a chi-sq uar e test for

cat egorical v ariables was run to

determine whether there is a difference in the distribution of related outcomes by research group. The p-value indicates the likelihood that the difference between the FSS group and control group arose by chance. Statistical significance levels are i ndicated as: * = 10 percent; ** = 5 per cent; *** = 1 percent. T he H-statistic tes

t was used

to te st for sta tistically significant

d ifferen

ces in impact estimates across dif ferent subgr oups. Statisti cal significan

ce l

evels are ind i

cated as: † = 10 percent; †† = 5

p ercent; and ††† = 1 percent.

Sources: MDRC calculations using baseline data and Experian and Clarity credit data 176

Exhibit E.9: Impacts on Incidence of Credit Problems in 2019, by Program Approach, Family Self-Sufficiency Impact Sample

Sample Sizes:

FSS Control Difference

FSS Control

Outcome (%)

Group

Group

(Impact)

P-Value

Group

Group

Experienced any recorded credit

problem

Emphasis on job search and post-employment services

Low

65.3

60.1

5.2

0.190

279

286

Medium

63.4

59.4

4.0

0.285

344

342

High

62.7

62.5

0.2

0.932

638

614

Emphasis on education and training

Low

66.6

63.4

3.2

0.337

377

286

Medium

64.3

61.2

3.1

0.275

569

559

High

58.5

57.7

0.9

0.827

315

301

Emphasis on financial services

Low

62.4

60.5

2.0

0.662

235

232

*

Medium

67.0

59.0

8.0 *

0.015

408

408

High

61.7

62.6

– 0.9

0.749

618

602

Program emphasis on monitoring and

engagement

Low

60.2

54.5

5.7 *

0.076

463

457

Medium

66.7

65.9

0.8

0.783

542

539

High

62.5

62.6

0.0

0.997

256

246

High debt-to-income ratio

Emphasis on job search and post-employment services

Low

19.3

19.7

– 0.4

0.909

275

285

Medium

19.5

18.0

1.5

0.629

344

343

High

19.5

18.5

0.9

0.684

639

613

Emphasis on education and training

Low

20.0

23.4

– 3.4

0.260

376

383

Medium

18.5

17.0

1.5

0.528

569

558

High

19.2

17.0

2.3

0.485

313

300

Emphasis on financial services

Low

16.2

20.4

– 4.2

0.248

234

232

Medium

20.4

16.9

3.5

0.205

408

408

High

19.8

19.5

0.4

0.871

616

601

Program emphasis on monitoring and

engagement

Low

18.7

17.6

1.1

0.680

463

458

Medium

19.6

21.9

– 2.3

0.346

538

538

High

20.2

14.0

6.2 *

0.077

257

245

(continued)

177

Exhibit E.9 (continued)

Sample Sizes:

FSS Control

Difference

FSS

Control

Outcome (%)

Group

Group

(Impact)

P-Value

Group

Group

Total balance (traditional financial services)

greater than or equal to 75 percent of total credit

Emphasis on job search and post-employment services

Low

51.4

43.6

7.8 *

0.052 ††

286

289

Medium

41.2

40.1

1.1

0.767 ††

351

352

High

40.2

45.0

– 4.8 *

0.073 ††

645

625

Emphasis on education and training

Low

46.2

45.6

0.5

0.881

386

392

Medium

42.2

44.4

– 2.1

0.457

578

568

High

41.6

37.2

4.5

0.249

318

306

Emphasis on financial services

Low

42.4

45.4

– 3.0

0.499

239

235

Medium

45.1

40.5

4.6

0.164

420

419

High

42.7

43.4

– 0.7

0.803

623

612

Program emphasis on monitoring and

engagement

Low

38.5

38.1

0.5

0.883

469

468

Medium

47.7

48.3

– 0.6

0.841

551

547

High

42.2

40.8

1.4

0.757

262

251

Any traditional financial services balance

90 days or more past scheduled repayment date

Emphasis on job search and post-employment services

Low

20.5

17.7

2.8

0.400

279

286

Medium

16.1

13.4

2.7

0.336

344

342

High

15.9

18.1

– 2.2

0.296

638

614

Emphasis on education and training

Low

19.6

17.8

1.8

0.528

377

382

Medium

15.2

18.3

– 3.1

0.165

569

559

High

15.4

13.8

1.7

0.577

315

301

Emphasis on financial services

Low

16.5

16.9

– 0.5

0.897

235

232

Medium

19.1

18.1

1.0

0.711

408

408

High

14.7

16.6

– 1.9

0.369

618

602

Program emphasis on monitoring and

engagement

Low

16.3

15.2

1.0

0.675

463

457

Medium

16.3

19.2

– 3.0

0.209

542

539

High

18.2

15.2

3.0

0.382

256

246

(continued)

178

Exhibit E.9 (continued)

Sample

Sizes:

FSS

Control

Difference

FSS Control

Outcome

Group

Group

(Impact)

P-Value Group Group

Incurred late payment, loan collection, or charge off for alternative financial service

Emphasis on job search and post-employment services

Low

8.2

6.8

1.4

0.538 †

277

285

Medium

7.8

4.0

3.7 **

0.043 †

340

340

High

7.5

9.3

– 1.8

0.253 †

632

609

Emphasis on education and training

Low

9.2

9.9

– 0.6

0.776

373

381

Medium

8.4

8.3

0.1

0.951

562

555

High

3.7

3.2

0.5

0.749

314

298

Emphasis on financial services

Low

8.0

12.8

– 4.8 *

0.099

234

230

Medium

8.1

7.0

1.1

0.545

403

408

High

6.9

5.8

1.2

0.421

612

596

Program emphasis on monitoring and

engagement

Low

7.9

7.2

0.7

0.714

459

456

Medium

7.6

7.6

– 0.1

0.964

536

532

High

7.1

7.3

– 0.2

0.943

254

246

Sample size(total=2548)

1,282

1,266

Notes: The FSS Impact sample includes Housing Choice Voucher heads of household who were randomly assigned between October 18, 2013, and Decem

ber 22, 2014, and were a

ge 18 to 61 at the time

of random

assi gnment. Sample

sizes for

specific outcomes may vary b

ecause of m issing valu

es. Estimates wer e regr ession-adj uste d using

ordinary

least squ ares, controlling for differen ces in s

ample memb er characteristi

cs reco rded at th e

time of

random

assignment. No special weights were applied to responses to adjust for differences in sample size by housing authority. Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test was applied t

o differences between the FSS

group and t he control

group for contin

uous va riables a

nd sel ecte

d outcomes

expresse

d as proportions. In addition, a chi-squar

e test for cat

egorical variables was r un to deter min e wheth er there

is a difference in the distribution of related outcomes by research group. The p-value indicates the likelihood that the difference between the FSS group and control group arose by chance. Statistical significance levels are indicated as:

* =

10 percent; ** = 5 percent; *** =

1 percent. The H-sta

tistic test was use d to

test for sta tis

tically sig nificant

differen

ces in impact estimates across

different su bgroups. St atistical signifi

cance l evels are i ndi cated as: † = 10

percent; † † = 5 percent; and ††† = 1 per cent.

Sources: MDRC calculations using baseline data and Experian and Clarity credit data 179

Exhibit E.10: Impacts on Selected Indicators of Rent and Subsidies in Years 1 to 5, by elected Program Implementation Features, Family Self-Sufficiency Impact Sample

Sample Size

FSS Control Difference

FSS

Control

Outcome

Group

Group

(Impact) P-Value Group

Group

Enrolled in HCV program in month 60 (%)

Emphasis on job search and post-employment services

Low

65.2

67.0

– 1.8

0.656

282

287

Medium

78.3

79.6

– 1.3

0.671

351

352

High

65.5

64.5

1.0

0.699

648

628

Emphasis on education and training

Low

68.4

68.6

– 0.2

0.949

385

392

Medium

67.3

66.2

1.1

0.689

577

567

High

70.6

77.6

– 7.0 *

0.050

319

308

Emphasis on financial services

Low

56.1

58.1

– 2.1

0.660

238

235

Medium

75.8

73.6

2.2

0.465

419

418

High

68.9

70.9

– 2.0

0.437

624

614

Emphasis on monitoring and engagement

Low

72.8

71.3

1.6

0.596

472

472

Medium

64.1

64.9

– 0.8

0.789

547

545

High

72.0

75.0

– 3.1

0.441

262

250

Average gross rent in month 60, if received HCV ($)

Emphasis on job search and post-employment services

Low

1,248

1,195

53

185

191

Medium

1,592

1,590

1

275

280

High

1,633

1,674

– 41

427

402

Emphasis on education and training

Low

1,299

1,310

– 12

267

265

Medium

1,547

1,582

– 35

393

371

High

1,796

1,753

43

227

237

Emphasis on financial services

Low

1,146

1,180

– 33

132

138

Medium

1,665

1,679

– 14

320

305

High

1,556

1,572

– 16

435

430

Emphasis on monitoring and engagement

Low

1,591

1,604

– 13

346

334

Medium

1,268

1,277

– 9

352

352

High

1,941

1,942

– 2

189

187

(continued)

180

Exhibit E.10 (continued)

Sample Sizes:

FSS Control Difference

FSS Control

Outcome

Group

Group

(Impact) P– Value Group

Group

Average family share in month 60, if received HCV ($)

Emphasis on job search and post-employment services

Low

456

451

4

185

191

Medium

634

558

76

275

280

High

617

646

– 30

427

402

Emphasis on education and training

Low

547

493

54

267

265

Medium

573

605

– 32

393

371

High

667

618

50

227

237

Emphasis on financial services

Low

461

489

– 28

132

138

Medium

618

604

14

320

305

High

610

579

31

435

430

Emphasis on monitoring and engagement

Low

612

559

53

346

334

Medium

526

524

1

352

352

High

663

700

– 37

189

187

Average housing subsidy in month 60, if received HCV ($)

Emphasis on job search and post-employment services

Low

802

746

56

185

191

Medium

969

1,049

– 80

275

280

High

1,033

1,031

2

427

402

Emphasis on education and training

Low

776

820

– 43

267

265

Medium

981

989

– 8

393

371

High

1,139

1,142

– 3

227

237

Emphasis on financial services

Low

701

695

6

132

138

Medium

1,048

1,075

– 27

320

305

High

970

1,007

– 36

435

430

Emphasis on monitoring and engagement

Low

979

1,045

– 66

346

334

Medium

776

772

4

352

352

High

1,278

1,243

35

189

187

(continued)

181

Exhibit E.10 (continued)

Sample Sizes:

FSS Control Difference

FSS

Control

Outcome

Group Group

(Impact)

P-Value Group

Group

Average total family share in years 1 to 5 ($)

Emphasis on job search and post-employment services

Low

21,135 20,189

946

0.364

271

280

Medium

29,871 27,540

2,330 **

0.044

348

346

High

25,432 24,714

719

0.393

637

614

Emphasis on education and training

Low

23,388 22,069

1,320

0.167 381

386

Medium

24,759 24,298

461

0.586 562

553

High

30,021 28,244

1,777

0.185 313

301

Emphasis on financial services

Low

20,449 19,944

505

0.687 234

233

Medium

25,900 25,210

689

0.505 406

403

High

27,407 25,976

1,432 *

0.092 616

604

Emphasis on monitoring and engagement

Low

27,284 25,286

1,998 **

0.046 464

463

Medium

23,712 22,416

1,295

0.108 537

538

High

27,356 27,376

– 20

0.989 255

239

Average total housing subsidy in years 1 to 5 ($)

Emphasis on job search and post-employment services

Low

35,891 36,192

– 302

0.864 281

286

Medium

51,332 52,952

– 1,621

0.300 349

347

High

42,587 41,448

1,139

0.441 628

605

Emphasis on education and training

Low

37,797 38,955

– 1,158

0.369 384

387

Medium

41,312 40,585

727

0.617 562

551

High

53,780 55,326

– 1,546

0.484 312

300

Emphasis on financial services

Low

31,217 31,074

143

0.926 237

234

Medium

48,342 48,920

– 578

0.739 404

399

High

44,479 45,262

– 783

0.573 617

605

Emphasis on monitoring and engagement

Low

46,722 46,993

– 271

0.861 466

459

Medium

35,684 36,367

– 683

0.569 545

545

High

55,040 52,739

2,301

0.397 247

234

Sample size (total =2,548)

1,281 1,267

(continued)

182

Exhibit E.10 (continued)

Notes: The FSS impact sample includes Housing Choice Voucher households that were randomly assigned between Oct

ober 18, 2013, and December 22, 2014, and

with a hea d of house

hold a ge 18 to

61 at the time of random

as sig nment. Sample sizes for speci fic outcomes m ay vary because of missi ng values .

Esti mates

were regression-

adjusted using ordinary least squares, controlling for differences in sample member characteristics recorded at the

time of random assignment. No special weights were applied to responses to adjust for differences in sample size by

housing authority. Rounding may cause slight discrepancies in calculating sums and differences. A two-tailed t-test

was applied to differences between the FSS gr

oup and the control

gro

up for conti nu

ous variables and sele cted

outc

omes expressed as proportions. In addition, a chi-square test for categorical variables was run to determine w hether there is a difference in the distribution of related outcomes by res earch gro up. The p- value indicates the

likeli hood that the difference between the FSS group and control group arose by chance. Statistical significance l

evels are indicated as: * = 10 per cent; ** = 5 per cent; *** = 1 per cent. T he H-sta

tis

tic test w as used to test for

statis t ically significant differences in impact estimates across different subgroups. Statistical significance levels are i

ndicated as: † = 10 percent; †† =

5 percent; a

nd ††† = 1 percent. Resul ts dis played i

n ital ics are non-experime ntal.

No te sts of statistical significance

were performed on differences betwe en re search gr

oups i n mea

ns or proportions.

Sour ces: MDRC calculations using

baseline data, data collected from Individual Training and S er

vices Plan fo rms,

inf or mation provided by FSS ad

ministrators and

case managers, a

nd U.

S. Depart

me

nt of H ousing and U rban

D eve lopment Inventory Manageme nt System (IMS )/PIH Informati

on Center (PIC) da ta.

183

References

Abbi, Sarika. 2012. A Need for Product Innovation to Help LMI Consumers Manage Financial Emergencies. Allston, MA: Doorways to Dreams Fund.

Abraham, Katherine, John C. Haltiwanger, Kristin Sandusky, and James Spletzer. 2009. “Exploring Differences in Employment Between Household and Establishment Data.” NBER Working Paper No. 14805. Cambridge, MA: National Bureau of Economic Research.

Akin, Jim. 2020. “What is an Ideal Debt-to-Income Ratio?” experian.com/blogs/ask-experian/what-

is-an-ideal-debt-to-income-ratio/

Bloom, Howard S., Carolyn J. Hill, and James Riccio. 2001. “Modeling the Performance of Welfare-to-Work Programs: The Effects of Program Management and Services, Economic Environment, and Client Characteristics.” MDRC Working Papers on Research Methodology. New York: MDRC.

Bloom, Howard S., Stephen W. Raudenbush, Michael J. Weiss, and Kristin Porter. 2017. “Using Multisite Experiments to Study Cross-Site Variation in Treatment Effects: A Hybrid Approach with Fixed Intercepts and a Random Treatment Coefficient,” Journal of Research on Educational Effectiveness 10 (4): 817–842.

Castells, Nina, James Riccio. 2020. Executive Skills Coaching Plus Incentives in a Workforce Program Introducing the MyGoals Demonstration. New York: MDRC.

Center on Budget and Policy Priorities. 2017. “Policy Basics: The Housing Choice Voucher Program.” https://www.cbpp.org/research/housing/the-housing-choice-voucher-

program#:~:text=Policy%20Basics%3A%20The%20Housing%20Choice%20Voucher%20Program%2

0Created,is%20the%20nation%E2%80%99s%20largest%20source%20of%20rental%20assistance.

———. 2020. “Basic Facts About HUD’s Family Self-Sufficiency Program.” Washington, DC: Center on Budget and Policy Priorities.

Chetty, Raj, and Nathaniel Hendren. 2017a. “The Impacts of Neighborhoods on Intergenerational Mobility I: Childhood Exposure Effects.” NBER Working Paper No. 23001. Cambridge, MA: National Bureau of Economic Research. http://www.equality-of-

opportunity.org/assets/documents/movers_paper1.pdf.

———. 2017b. “The Impacts of Neighborhoods on Intergenerational Mobility II: County-Level Estimates.” NBER Working Paper No. 23002. Cambridge, MA: National Bureau of Economic Research. http://www.equality-of-opportunity.org/assets/documents/movers_paper2.pdf.

Clarity Experian. 2020. Alternative Financial Services Lending Trends: Insights Into the Industry and Its Consumers. Clearwater, FL: Clarity Services, Inc.

Code of Federal Regulations. 1996. “Part 984 – Section 8 and Public Housing: Family Self-Sufficiency Program. Source: 61 FR 9915, 984.303 Contract of Participation, 4(iii).

https://ecfr.federalregister.gov/current/title-24/subtitle-B/chapter-IX/part-984#p-984.303(b)(4)(iii).

184

Collins, J. Michael, and Leah Gjertson. 2013. Emergency Savings for Low-Income Consumers.

Madison, WI: Institute for Research on Poverty.

Consumer Financial Protection Bureau. 2012. Analysis of Differences Between Consumer-and Creditor-Purchased Credit Scores. Washington, DC: Consumer Financial Protection Bureau.

——. 2019. "What is a Debt-to-Income Ratio? Why is the 43% Debt-to-Income Ratio Important?"

Consumer Financial Protection Bureau. https://www.consumerfinance.gov/ask-cfpb/what-is-a-debt-

to-income-ratio-why-is-the-43-debt-to-income-ratio-important-en-1791/

DeNicola, Louis. 2019. “Everything You Need to Know About the VantageScore 4.0 Credit Scoring Model.” Credit Karma. https://www.creditkarma.com/advice/i/new-vantagescore-4-0-explained/.

Experian. 2020. “What Is a Good Credit Score?” Score Advice. https://www.experian.com/blogs/ask-

experian/credit-education/score-basics/what-is-a-good-credit-score/.

Folger, Jean. 2021. "What Constitutes a Good Debt-to-Income (DTI) Ratio?"

investopedia.com/ask/answers/081214/whats-considered-be-good-debttoincome-dti-ratio.asp

Geyer, Judy, Lesley Freiman, Jeffrey Lubell, and Micah Villarreal. 2017. Evaluation of the Compass Family Self-Sufficiency (FSS) Programs Administered in Partnership with Public Housing Agencies in Lynn and Cambridge, Massachusetts. Cambridge, MA: Abt Associates.

Guare, Dick, and Peg Dawson. 2016. A Manual for Executive Skills Coaching with Adults Affected by Conditions of Poverty and Stress. Fort Collins, CO: TuaPath.

Jacob, Brian, and Jens Ludwig. 2008. Improving Educational Outcomes for Poor Children. NBER

Working Paper No. 14550. Cambridge, MA: National Bureau of Economic Research.

Lopez-Fernandini, Alejandra. 2012. Unrestricted Savings: Their Role in Household Economic Security and the Case for Policy Action. Washington, DC: New America Foundation.

McKernan, Signe-Mary, Caroline Ratcliffe, and Katie Vinopal. 2009. Do Assets Help Families Cope with Adverse Events? Washington, DC: Urban Institute.

Mills, Greg, Daniel Gubits, Larry Orr, David Long, Judie Feins, Bulbul Kaul, and Michelle Wood.

2006. Effects of Housing Vouchers on Welfare Families. Washington, DC: U.S. Department of Housing and Urban Development, Office of Policy Development and Research.

Nuñez, Stephen, Nandita Verma, and Edith Yang. 2015. Building Self-Sufficiency for Housing Voucher Recipients: Interim Findings from the Work Rewards Demonstration in New York City. New York: MDRC.

Popkin, Susan J., Larry F. Buron, Diane K. Levy, and Mary K. Cunningham. 2000. “The Gautreaux Legacy: What Might Mixed-Income and Dispersal Strategies Mean for the Poorest Public Housing Tenants?” Housing Policy Debate 11 (4): 911–942.

Popkin, Susan J., Mary K. Cunningham, and Martha Burt. 2005. “Public Housing Transformation and the Hard to House,” Housing Policy Debate 16 (1): 1–24.

185

Popkin, Susan J., Brett Theodos, Liza Getsinger, and Joe Parilla. 2010. A New Model for Integrating Housing and Services. Washington, DC: Urban Institute.

Riccio, James A., Victoria Deitch, and Nandita Verma. 2017. Reducing Work Disincentives in the Housing Choice Voucher Program: Rent Reform Demonstration Baseline Report. New York: MDRC.

Rohe, William M. 1995. “Converting Public Housing to Cooperatives: The Experience of Three Developments,” Housing Policy Debate 6 (2): 439–479.

Rohe, William M., and Rachel Garshick Kleit. 1999. “Housing, Welfare Reform, and Self-Sufficiency: An Assessment of the Family Self-Sufficiency Program,” Housing Policy Debate 10 (2): 333–369.

Sard, Barbara, and Margy Waller. 2002. Housing Strategies to Strengthen Welfare Policy and Support Working Families. Washington, DC: Brookings Institution, Center on Urban and Metropolitan Policy.

Schwartz, Alex F. 2006. Housing Policy in the United States. New York: Routledge.

Shroder, Mark D. 2010. “Housing Subsidies and Work Incentives.” October 15.

http://dx.doi.org/10.2139/ssrn.1691112.

Smith, Robert E., Susan J. Popkin, Taz George, and Jennifer Comey. 2015. What Happens to Housing Assistance Leavers? Washington DC: U.S. Department of Housing and Urban Development, Office of Policy Development and Research.

U.S. Department of Housing and Urban Development (HUD). 2016. Report to Congress on Family Self-Sufficiency Best Practices. Washington, DC: U.S. Department of Housing and Urban Development. https://www.hud.gov/sites/documents/FY16FSS_BPREPORTCOCHRAN.PDF.

———. 2017a. Administering an Effective Family Self-Sufficiency Program: A Guidebook Based on Evidence and Promising Practices. Washington, DC: U.S. Department of Housing and Urban Development.

———. 2017b. Family Self-Sufficiency Program: Funder Report. Washington, DC: U.S. Department of Housing and Urban Development. https://www.hud.gov/sites/documents/FR-6100-N-

05_FY17_FSS_NOFA.PDF.

———. 2018. “Family Self-Sufficiency Performance Measurement System (“Composite Score”).”

HUD Notice 2018-24949. https://www.federalregister.gov/documents/2018/11/15/2018-

24949/family-self-sufficiency-performance-measurement-system-composite-score.

———. 2019. “Family Self-Sufficiency Program—Renewal.” Funding Opportunity Number FR-6300-N-04.

https://www.hud.gov/sites/dfiles/SPM/documents/FY19_FSS_Renewal_NOFA_final.pdf.

———. 2020. “Family Self-Sufficiency 2020 Summary of Resources.” Washington, DC: U.S.

186

Department of Housing and Urban Development.

https://www.hud.gov/sites/dfiles/CFO/documents/2020CJ-FSS.pdf.

Verma, Nandita, Stephen Freedman, Betsy L. Tessler, and Barbara Fink. 2021. Work, Engagement, and Well-Being at the Midpoint: Findings from the Family Self-Sufficiency Evaluation. Report prepared for the U.S. Department of Housing and Urban Development, Office of Policy Development and Research. New York: MDRC.

Verma, Nandita, Stephen Freedman, Betsy L. Tessler, Stephen Nuñez, and Barbara Fink. 2019.

Promoting Work and Self-Sufficiency for Housing Voucher Recipients: Early Findings From the Family Self-Sufficiency Program Evaluation. Report prepared for the U.S. Department of Housing and Urban Development, Office of Policy Development and Research. New York: MDRC.

Verma, Nandita, Betsy Tessler, Cynthia Miller, James A. Riccio, Zawadi Rucks, and Edith Yang.

2012. Working Toward Self-Sufficiency: Early Findings from a Program for Housing Voucher Recipients in New York City. New York: MDRC.

Verma, Nandita, Edith Yang, Stephen Nuñez, and David Long. 2017. Learning from the Work Rewards Demonstration. New York: MDRC

187

U.S. Department of Housing and Urban Development

Office of Policy Development and Research

Washington, DC 20410-6000

May 2023

Document Outline

Nearing the Finish Line: 5-Year Findings From the Family Self-Sufficiency Evaluation

DISCLAIMER

Foreword

Contents

List of Exhibits

Executive Summary

The Family Self-Sufficiency Program

Study Design and Sample Characteristics

Key Findings

Program Implementation

Enrollment, Graduation, and Escrow Disbursements

Employment and Earnings

Credit Use and Financial Well-Being

Housing Subsidy Receipt

Variation Across Sites in Program Impacts

Next Steps

Chapter 1. Introduction

Housing Assistance and Employment Context

The Family Self-Sufficiency Program

Core Features of the FSS Program

How Might the FSS Components Help Participants Advance?

The National Family Self-Sufficiency Impact Evaluation

The Study Sample

Structure of This Report

Chapter 2. Program Implementation Update and Reflections

Data Sources and Followup Period

Program Practice and Policy Updates

Staffing and Caseloads

Interim Escrow Disbursements

Requirements for Graduation

A New FSS Performance Measurement System

Case Management in the Final Years of the FSS Contract

Final Push in Years 4 and 5

Working with Participants Not Likely to Graduate

Balancing Accountability and Self-Reliance

Graduation

Service Referrals and the Role of the Program Coordinating Committee

Staff Reflections

Escrow Incentive

Interim Disbursements

Who Benefits from FSS?

Promising Practices

Making the FSS Program Mandatory or Keeping it Voluntary

Program Length

Conclusions

Chapter 3. FSS Enrollment Status, Escrow Accruals, and Graduations

How Family Self-Sufficiency Escrow Works

FSS Enrollment Status, Graduations, and Escrow Accrual and Disbursement

Variation in Graduation and Escrow Accrual for Selected Subgroups

Variation in Graduation Rates by Housing Agency

Who Appears to Be Benefiting the Most from Enrollment in Family Self-Sufficiency?

Conclusions

Chapter 4. Impacts on Employment and Earnings

Data Sources and Methods

Program Impacts on Employment and Earnings

Employment and Earnings Trends for Control Group Members

Impacts on Employment and Earnings

Conclusions

Chapter 5. Impacts on Credit Use, Housing Costs, Receipt of Housing Subsidies, and Financial Well-Being

Experian and Clarity Credit Data

Impacts on Credit Scores

Impacts on the Use of Traditional Financial Services and Sources of Credit

Impacts on the Use of Alternative Financial Services and Sources of Credit

Impacts on the Use of Credit (Traditional and AFS) and Incidence of Credit Problems

Impacts on Housing Status and Rent Subsidies

Impacts on Homeownership

Conclusions

Chapter 6. Variation in Program Impacts

Program Impacts by Subgroups: Who Benefits in Years 3 Through 5?

Subgroups Based on Participant Baseline Characteristics

Employment and Earnings

Credit Scores and Use of Credit

Variation in Housing Outcomes

Subgroups Based on Program Implementation Features

Program Clusters Based on Program Focus

Program Clusters Based on Site Monitoring and Engagement Practices

Conclusion

Chapter 7. Summary and Next Steps in the Evaluation

What the 5-Year Results Tell Us

The Evaluation Going Forward

Appendix A

Appendix B

Appendix C

Appendix D

Appendix E

References