DC Flexible Rent

Subsidy Program

Findings from the Program’s First Year

U.S. Department of Housing and Urban Development | Office of Policy Development and Research

DC Flexible Rent Subsidy Program

Findings from the Program’s First Year

Josh Leopold

Mychal Cohen

Kassie Scott

URBAN INSTITUTE

Maria Alva

Natnaell Mammo

Namita Mody

Ryan T. Moore

Sam Quinney

THE LAB @ DC

October 2020

ABOUT THE URBAN INSTITUTE

The nonprofit Urban Institute is a leading research organization dedicated to developing evidence-based insights that improve people’s lives and strengthen communities. For 50 years, Urban has been the trusted source for rigorous analysis of complex social and economic issues; strategic advice to policymakers, philanthropists, and practitioners; and new, promising ideas that expand opportunities for all. Our work inspires effective decisions that advance fairness and enhance the well-being of people and places.

ABOUT THE LAB @ DC

The Lab @ DC is an applied scientific team launched by District of Columbia Mayor Muriel Bowser in the Office of the City Administrator. The Lab @ DC was created because District residents deserve a government that asks questions, tests policies, and iteratively improves how it serves the community. The Lab @ DC partners with District agencies to do just that. The Lab @ DC works side-by-side with our agency partners to design resident-centered, evidence-based services and evaluate their effectiveness.

DISCLAIMER

The contents of this report are the views of the contractor and do not necessarily reflect the views or policies of the U.S. Department of Housing and Urban Development or the U.S. Government.

Contents

Acknowledgments

v

Foreword

vi

Executive Summary

viii

DC Flexible Rent Subsidy Program

1

Background

1

Evaluation Approach and Research Questions

3

Qualitative Approach

5

Quantitative Approach

6

Program Design and Implementation

9

Implementation Timeline

9

Eligibility Requirements

10

Program Enrollment and Participant Characteristics

13

Program Orientation

17

Flexible Subsidy Accounts

18

Participant Services

22

Recertification

23

Outcomes

25

Use of Flex Accounts

25

Homelessness and Housing Stability

30

Housing Mobility

31

Homelessness and Use of Continuum of Care Services

32

Economic Well-Being

36

Limitations

36

Conclusion

38

Appendix A. DC Flex Participant Survey Responses

40

Appendix B. Focus Group Interview Protocol

45

Appendix C. Inclusion Criteria for Households in the Year 1 Analysis

49

Appendix D. Blocking Variables, Balance Tables, and Regression Analyses for Al Enrolled

Participants

50

Appendix E. Baseline Outcomes

52

Appendix F. Inverse Probability Weights

53

Appendix G. Details of Pre-Registered Analyses Not Included in This Report, Non-Pre-

Registered Updates Included in This Report, and Potential Analyses to Be Pursued in Future

Reports

54

References

58

Acknowledgments

This report was funded by the U.S. Department of Housing and Urban Development. We are grateful to them and to all our funders, who make it possible for Urban to advance its mission.

The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders. Funders do not determine research findings or the insights and recommendations of Urban experts. Further information on the Urban Institute’s funding principles is available at urban.org/fundingprinciples.

The authors wish to acknowledge the support of Markia Smith and Martin Booker (formerly) of Capital Area Asset Builders; Peter Casey, Sarah Evans, Victoria Bishop, and David Yokum formerly of The Lab @ DC, Kristy Greenwalt, Dena Hasan, Noah Abraham, Danilo Pelletiere, Hersh Gupta, and Allison Tucker from the Government of the District of Columbia; Tom Fredericksen from The Community Partnership; Devlin Hanson and Mary Cunningham of the Urban Institute; and Anne Fletcher of the U.S. Department of Housing and Urban Development.

A C K N O W L E D G M E N T S

v

Foreword

In 2017, the Washington D.C. City Council used local funds to establish a pilot program, the DC

Flexible Rent Subsidy Program (DC Flex), to test the effectiveness of a shallow subsidy program.

Shallow subsidies are a novel approach to offering rental assistance that represents a mid-point in the duration and depth of assistance between a Housing Choice Voucher and other time-limited emergency assistance programs, such as rapid re-housing.

DC Flex offers participating families $7,200 per year in rental assistance that families can apply, in variable amounts of their choosing, toward their rent each month. Families can participate in the program for up to 4 years. Eligibility was limited to households with extremely low incomes that had children, a recent history of employment, and a recent application for homelessness or emergency rental assistance. These eligibility requirements reflected a goal to serve the families that were the most in need and yet also likely to succeed with a shallow subsidy. This report discusses the results of an evaluation to document program implementation and to capture the perspective of the approximately 100 families participating in the first year.

Because this evaluation focuses on the first year of implementation, it is too early to draw observations about program effectiveness. Families were generally satisfied with the program, with 88 percent opting to recertify for a second year. A review of the account data showed that 60

percent of program participants spent the full $7,200 over the course of the year, and those that did not rolled funds into the second year. It is important to note that the observations were made before the start of the COVID-19 pandemic.

The body of research on the effectiveness of shallow subsidies is limited; thus, this evaluation of the DC Flex Program helps to build the knowledge base. Although the sample is small and confined to a single metropolitan area, the documentation of one community’s approach to program design and implementation and the exploration of the opportunities and challenges with this program model the program designers, administrators, and participants identified provide a strong blueprint for other communities that may wish to establish a similar shallow subsidy program. The favorable feedback program participants provided and the fact that program participation remained strong moving into v i

F O R E W O R D

the second year of operation suggest that this model of assistance holds promise and warrants further research.

Todd M. Richardson

General Deputy Assistant Secretary for Policy Development and Research U.S. Department of Housing and Urban Development

F O R E W O R D

v i i

Executive Summary

In late 2017, the District of Columbia (DC) Department of Human Services began piloting the Flexible Rent Subsidy Program (DC Flex). DC Flex is a unique rent subsidy program that allows participants to choose how much financial assistance they need to pay rent each month. Unlike other programs, in which rental assistance goes to landlords, DC Flex assistance goes directly to participants. The program administrator deposited $7,200 in an escrow account for each family at the start of the program year. Participants then decide how much to withdraw from their account each month. The money must be used to pay rent, and participants cannot withdraw more than their total month’s rent each month. After 12 months, participants receive an additional $7,200 in their Flex accounts if they continue to meet the eligibility requirements of the program. Participants can stay in the program for up to 4 years (or longer if DC chooses to extend the program after the 4-year pilot ends).

Because DC Flex is a new and innovative housing subsidy, the District of Columbia saw an important need to rigorously assess the effects of the program on homelessness and housing stability in its pilot period. The Urban Institute and The Lab @ DC, an applied research team in the office of DC

Mayor Muriel Bowser, collaborated to evaluate the program’s first year. Urban’s qualitative evaluation found that the District was successful in creating the program, enrolling families, setting up and monitoring the Flex accounts, and making sure participants’ accounts were replenished in a timely fashion. The Lab @ DC’s quantitative evaluation—a randomized controlled trial—found no statistical difference in the use of emergency shelter or transitional housing, as both the treatment and the comparison groups entered these programs at very low rates. Families enrolled in DC Flex were far less likely to use other homeless assistance programs within the District of Columbia. This is primarily because a large subset of families chose to exit rapid re-housing, a rental assistance program intended to help families exit shelter with the help of temporary rental assistance, to enroll in DC Flex. Also, DC

Flex participants were initially not allowed to receive emergency rental assistance while in the program, although this rule was later changed.

DC Flex is an example of a “shallow subsidy” that offers a middle ground between a rental voucher program (paying a larger share of housing costs, often over many years) and emergency assistance (providing relatively small amounts of short-term rental assistance). Shallow subsidies are designed to stretch resources to serve more people, with the trade-off that recipients receive a lower subsidy than what is provided with a voucher. The DC Flex pilot offers a less expensive and more flexible subsidy alternative to housing vouchers for some part of the population that is at risk of homelessness or housing instability.

v i i i

E X E C U T I V E S U MM A R Y

Designing DC Flex was a collaborative effort between the DC Department of Human Services (DHS), the DC Interagency Council on Homelessness, the DC Department of Housing and Community Development, and The Lab @ DC. The DC Council allocated $5 million for the pilot. The legislation was approved by the DC Mayor in 2016 and provided 4 years of funding to assist up to 125 families.

DHS administers the program through a grant to Capital Area Asset Builders (CAAB), a local nonprofit specializing in financial education, matched savings, and consumer education programs in the Greater DC area. Evaluating the pilot program is a collaborative effort between the Urban Institute and The Lab @ DC. The Urban Institute tracked the experiences of program participants and staff involved in program design and administration during the first year of implementation, with support from a HUD

Research Partnership grant. The Lab @ DC conducted a quantitative analysis of the pilot using administrative data. Together, the qualitative and quantitative research aim to inform whether DC or other jurisdictions should expand this program beyond the pilot.

Eligibility for DC Flex was restricted to families that have children and had extremely low incomes.

Families must have at least one working adult, a recent application for homelessness or emergency rental assistance, and be living in a legal rental unit with their name on the lease. Heads of household applied for DC Flex from December 2017 through July 2018. Starting in January 2018, DHS and The Lab @ DC began selecting potential recipients using random lotteries. Applicants selected in the lottery went through an additional eligibility verification process starting in February. The first applicants who enrolled in the program began receiving funds in May 2018. By September 2018, 121

families had accepted the offer to enroll in DC Flex, and 102 families had set up accounts and begun using their subsidies. This report describes this initial cohort’s experiences and outcomes through September 30, 2019: its first year in the program.1 The Urban Institute’s qualitative evaluation included a survey that was conducted in February and March of 2019 and was completed by 61

percent (n = 74) of the 121 families originally enrolled in DC Flex, as well as two focus groups with DC

Flex participants in June 2019 and interviews with program staff throughout the year.

Surveyed participants almost unanimously (94 percent) reported that they were satisfied with the program. Participants reported mixed satisfaction levels with their homes and neighborhoods, 1 As of October 2020, this is the most recent time frame The Lab @ DC can report based on administrative data.

We expect that data through September 30, 2020, will become available by December 2020 and that the analysis pertaining to those records can be presented publicly by late 2021.

F O R E W O R D

i x

however.2 Some suggestions participants had to improve the program were to increase the amount of assistance offered, allow participants to find cheaper housing outside of DC, and provide employment services. While most participants liked the flexibility and independence that the program provided, some report that they would have benefited from having a case manager to help with employment, financial decision-making, and housing problems. Many participants said that the program had led to positive changes in their finances, including opening personal checking and savings accounts and establishing or improving their credit scores.

Early program data suggested that most participants were maxing out their DC Flex withdrawals each month and would deplete their annual assistance in the first 6 months of the program year. This helped motivate program changes to allow participants to also receive assistance through DC’s Emergency Rental Assistance Program (ERAP) while in DC Flex and to recertify for DC Flex even if they had rent arrears at the end of the program year. Although a few families took advantage of the rule changes related to ERAP during the program year, many families did have rent arrears. Ultimately, 88 percent of program participants in our study remained stably housed and were able to recertify for program year 2. In addition, one-third of participants did not use the full $7,200 in the first year and rolled over their savings into the next year.

The Lab @ DC led the quantitative evaluation.3 To evaluate the effects of DC Flex over the first program year, The Lab leveraged the lotteries used to select DC Flex applications as the basis for a randomized controlled trial. The evaluation compares the outcomes of applicants offered DC Flex to the outcomes of households assigned to “business as usual,” which includes access to all other homelessness prevention and housing support services available in DC. These services included both one-time and short-term services and more substantial supports like rent vouchers. In interpreting the results of the analysis, it is important to note that eligible applicants who were offered DC Flex in the random lottery had to choose DC Flex instead of these other services. They could still access those services after they enrolled, but for most services, that meant they could not continue in DC Flex.

Some but not all of the alternative options constituted a larger annual benefit than the $7,200

provided by DC Flex.

2 Note that the program does not determine where families reside. One of the eligibility criterion for the program is to have a lease prior to enrol ment.

3 The pre-analysis plan for the quantitative analysis is available on the Open Science Framework.

x

E X E C U T I V E S U MM A R Y

The primary outcome of interest is the rate at which participants experience homelessness, measured by the percentage of participants who stayed in emergency shelter or transitional housing at some point during the program year. Participants in both DC Flex and the control group were unlikely to experience homelessness, and The Lab @ DC did not find a statistically significant effect for DC Flex participants. During the first program year, 1.8 percent of DC Flex participants entered emergency shelter and transitional housing.4 Without the DC Flex program, The Lab @ DC estimates that 2.0 percent of participants would have entered these programs, meaning that there is a 0.2-percentage-point difference between DC Flex participants and nonparticipants. Larger differences may emerge over subsequent program years, especially as families in the control group stop receiving assistance from more short-term programs like rapid re-housing.

As a secondary outcome, The Lab measured the rate at which participants use homelessness services in addition to emergency shelter and transitional housing. Homelessness services can range from short assessments of need to permanent supportive housing placements. If a participant used a service at least once in the first program year, it was counted in this outcome. The Lab found that enrolling in DC Flex caused a statistically significant decrease in participants’ use of homelessness services5 during their first program year. The average participant experienced a 28.6-percentage-point decrease in the use of these services as a result of receiving DC Flex. This difference is largely driven by participants choosing DC Flex over other housing assistance programs they were receiving when they applied for DC Flex, primarily rapid re-housing. This result demonstrates that DC Flex was perceived as a viable alternative to existing services for many participants. This is noteworthy because the $7,200 in annual rental assistance offered in DC Flex was significantly less than the average annual assistance provided by programs like rapid rehousing accessed through the homeless Continuum of Care.

Finally, The Lab @ DC assessed whether DC Flex had any effect on (1) the rate of Temporary Assistance for Needy Families (TANF) receipt or (2) the amount of benefit received by participants.

The Lab found that DC Flex had no effect on the receipt rate or the monthly benefit amount in the first program year.

4 All entries were to emergency shelters; none were to transitional housing.

5 These services include emergency shelter and transitional housing, as wel as various other services like coordinated assessments, homeless prevention services, and the Family Rehousing and Stabilization Program (another rental subsidy provided by DC Government).

F O R E W O R D

x i

Overall, the quantitative results indicate that DC Flex is a viable housing support option for eligible applicants who choose to take up the program. Those applicants are at almost an identical—

but low— risk of homelessness after 1 year compared with those receiving business-as-usual options.

One year, however, is a short period of time to observe effects on homelessness. Over subsequent years, and with more data, it may become clear that residents who choose DC Flex are meaningfully better or worse off than the status quo. If the risk of homelessness remains similar, then the relative costs of DC Flex compared with the other homelessness prevention and rental support services will be vital to understanding its usefulness as a model.

Although the analysis period for this report ended before the onset of COVID-19, the pandemic severely limited the analysis of several pre-registered outcomes6 for housing stability and economic well-being that The Lab @ DC intended to include in this report. For example, the pandemic severely stressed the city’s unemployment insurance system, making it impossible to collect employment data for study participants.

Future reports will assess the impact of DC Flex during the pandemic. In an informal survey administered by CAAB in March 2020, all but one of the 34 respondents stated that they could not have paid rent during the pandemic without DC Flex. The pandemic will also affect outcomes for the business-as-usual group. For example, the District instituted an eviction moratorium during the emergency and relaxed rules for other housing assistance programs. Program take-up during the public health emergency has also been affected: fewer people appear to meet the eligibility criteria for DC Flex or are willing to give up deeper subsidies, adding unexpected variation to the lottery and enrollment process.

Despite these complications, additional research on DC Flex is planned and will be crucial. This report details only 1 year’s worth of data on a limited number of outcomes, which is not enough to determine whether the program was effective. In future years, The Lab @ DC intends to examine additional indicators of housing stability and economic security, including evictions, applications to other rental assistance programs, and employment outcomes. The costs of achieving the outcomes associated with DC Flex relative to the outcomes for other housing assistance programs used by the 6 Pre-registration is a process by which a research team publishes a study’s research questions, hypotheses, data to be collected, and analysis methods before any analysis for the study begins. Pre-registration promotes scientific research integrity by providing a public record of a researcher’s plans. Pre-registration helps keep us accountable and limits any intentional or unintentional tendencies toward data manipulation and other methodological decisions that could support pre-determined or desired conclusions.

x i i

E X E C U T I V E S U MM A R Y

control group will also be examined. That comparison will be important in assessing whether the program merits continuation and possibly expansion. Observing additional outcomes over multiple years will provide a fuller picture of the program’s effectiveness and will be crucial to the policymakers seeking to decide how to move forward after the pilot ends.

F O R E W O R D

x i i i

DC Flexible Rent Subsidy Program

This report summarizes the results of our evaluation of the DC Flexible Rent Subsidy Program (DC

Flex). It focuses primarily on the results of the qualitative analysis. The Lab’s quantitative analysis is briefly summarized here but will be discussed in more detail in an eventual submission to a peer-reviewed journal.

The report begins with the program’s background, in the context of other housing assistance programs. Next, the research questions and methods are detailed. This is followed by an overview of the program’s design and implementation, which details eligibility criteria, participant characteristics, the enrollment and account setup processes, participant services in addition to the $7,200, and recertification. This section is followed by a discussion of first-year outcomes, organized by topic: use of DC Flex accounts, homelessness and housing stability, used of homeless Continuum of Care services, housing mobility, and economic well-being. The body of the report ends with a discussion of analysis limitations and a conclusion.

The document concludes with seven appendices. The first two relate to the Urban Institute’s research: survey responses (A) and focus group interview protocols (B). The following four (C–F) add depth to The Lab @ DC’s quantitative analysis. The final appendix (G) details deviations from The Lab

@ DC’s pre-analysis plan.

Background

Rental housing assistance in the United States is defined by scarcity. There are “mainstream” rental assistance programs, the largest of which is the Housing Choice Voucher (HCV) program, for which most low-income renter households are eligible. In these programs, participants typically pay 30

percent of their income in rent, with the rest subsidized. These programs, which are not subject to time limits, have been shown to reduce housing costs, improve housing stability, and help children move out of poverty into the middle class (Chetty, Hendren, and Katz, 2016; Gubits et al., 2015; Wood, Turnham, and Mills, 2008). They are not, however, adequately funded to serve all (or even most) eligible households, and applicants can wait years before being offered assistance (Poethig, 2014).

On the other side of the spectrum are emergency assistance programs, like rapid re-housing, that provide short-term housing assistance for households in crisis. Rapid re-housing offers people

experiencing homelessness a rental subsidy that can last up to 24 months but is often much shorter in duration. While in a rapid re-housing program, recipients receive case management services to help budget and increase their incomes so that they can afford housing after the subsidy ends. Rapid rehousing has been shown to be effective at helping people exit emergency shelters or the street (Cunningham, Gillespie, and Anderson, 2015). Still, rapid re-housing does not appear to improve recipients’ long-term housing stability. HUD’s Family Options Study found that homeless families randomly assigned to receive rapid re-housing had similar levels of housing stability (as measured by time spent homeless or doubled-up or the number of residential moves) as families randomly assigned to receive usual care (Gubits et al., 2015; Gubits et al., 2016).

The DC Flex program is an example of a “shallow subsidy” that offers a middle ground between the voucher program and emergency assistance. Shallow subsidies are designed to stretch resources to serve more people, with the trade-off that recipients receive a lower subsidy than what is provided with a voucher and for a shorter period of time. The DC Flex pilot is testing whether a less expensive, but more flexible subsidy could provide a cost-effective alternative to HCVs for some part of the housing-cost-burdened population. It is the first shallow subsidy program to undergo a random assignment evaluation (Cunningham, Leopold, and Lee, 2014). The program was inspired by a similar rental subsidy in Chile designed to support young families (Ross and Pelletiere, 2014).

In DC Flex, participants each year are allotted $7,200, which is deposited in an escrow account.

Before the first of every month, the participant’s rent amount is made available in a checking account accessible to the participant. The only condition on the assistance is that the money must be used to pay rent. Participants are responsible for choosing how much assistance to use each month. After 12

months, participants receive an additional $7,200 in their escrow accounts if they continue to meet the eligibility requirements. Participants can stay in the program for a minimum of 12 months and for up to 4 years, or longer if the program is extended once the pilot ends.

The $7,200 annual subsidy provided by DC Flex was designed to be about one-half the average annual subsidy DC households receive in the voucher program. Unlike the voucher program, the subsidy amount does not vary based on household income or household size. The unique subsidy structure offers several potential advantages for participants. First, it avoids the “penalty” that households face in traditional subsidy models where households’ rent increases by 30 cents for every dollar of increased income. There is some evidence that this income-based rent subsidy decreases employment and earnings (Jacob and Ludwig, 2012). DC Flex does not reduce the subsidy amount when a household’s income increases. Second, by letting participants decide how much assistance to use each month, DC Flex may be better suited to fluctuations in participants’ income or expenses. The 2

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

incomes of poor households are often highly variable, with spikes and dips throughout the year. For these households, spending goes down when income decreases, suggesting a lack of available savings to cover basic needs (Hannagan and Morduch, 2015). A flexible subsidy that operates like a cash reserve may allow low-income households to increase their savings during income spikes and avoid material hardships during income dips. Finally, because the subsidy goes directly to the participant and does not increase when the rent goes up, it may not have the same inflationary effect on rental prices as traditional vouchers (Susin, 2002).

Although DC Flex is unique among current rental assistance programs, it shares several features with HUD’s Experimental Housing Allowance Program (EHAP) of the 1970s and shallow subsidies piloted in California and Chile. EHAP was a landmark study of the effects associated with housing allowances on households and the housing market overall (Public Housing: The Experimental Housing Allowance Program,” 1981). EHAP tested different types of rental subsidies, some similar to the current HCV Program and others more similar to the DC Flex program. The evaluators found that, when given the choice, most participants opted to use rental subsidies to reduce their housing costs, rather than improve their housing conditions (Struyk and Bendick, 1981). Households increased spending on housing only when it was required to meet program standards.

The most rigorously evaluated shallow subsidy program was Project Independence in Alameda County, California. The program provided a flat, shallow subsidy of $225 per month—about one-half the average subsidy for a single-person household in the HCV program at the time. A quasi-experimental evaluation of this program that used a comparison group of eligible nonrecipients constructed from administrative data found that over a 5-year period, the average time spent living in independent, rental housing was 4 years for the treatment group and 1 year for the comparison group (Dasinger and Speiglman, 2007). Data from the shallow rental subsidy program in Chile on which DC

Flex is modeled found that the subsidy reduced rent burden and overcrowding (Salvi del Pero, 2016).

Evaluation Approach and Research Questions

The Urban Institute and The Lab @ DC collaborated to evaluate the DC Flex pilot program. The Lab @

DC led the quantitative evaluation: a randomized controlled trial of the impact of the program on homelessness, housing stability, and economic well-being. The Urban Institute led the qualitative evaluation of DC Flex’s implementation, including the perspective of program staff and participants.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

3

The Lab’s quantitative evaluation of the first program year addresses the following research questions:

◼

How does participation in DC Flex affect the rate at which DC Flex applicants experience homelessness?

◼

How does participation in DC Flex affect the rate at which applicants experience housing instability?

◼

How does participation in DC Flex affect the rate at which applicants require additional monetary assistance through Temporary Assistance for Needy Families (TANF)?7

The quantitative results reported here are initial and have not been peer-reviewed. These results will be finalized and expanded upon for eventual submission to a peer-reviewed journal and are subject to change.

The qualitative components of the Urban Institute’s evaluation draw from document review, program observation, and interviews and focus groups with participants, staff, and other stakeholders to document the program’s design and the experiences of staff and participants. The evaluation addresses the following research questions:

◼

How were key decisions made about program eligibility and design?

◼

How does DC Flex compare with other local and national rental assistance programs?

◼

How did eligible applicants decide whether to enroll in the program?

◼

What were the initial challenges in creating and administering the flexible rent subsidy accounts, and how were they resolved?

◼

How did program participants make decisions about how much to withdraw from their accounts each month?

7 The original quantitative research design included this question: “How does participation in DC Flex affect the rate at which adults in families are employed and their income from wages?” Employment and earnings data have a 3- to 6-month lag in the DC unemployment insurance system. The data needed to answer the employment and income question were originally scheduled to be pulled in March 2020, but that work was delayed while the DC

Department of Employment Services managed the dramatic increase in unemployment insurance claims as a result of the COVID-19 pandemic.

4

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

◼

How did participants, staff, and stakeholders perceive the program’s effects on housing mobility, neighborhood location, housing affordability, and housing quality?

◼

What were participants’ overall impressions of the program? What did they like about the program, and what would they change?

Qualitative Approach

The qualitative evaluation is based on interviews with program staff and administrators; observation of program events and meetings; and interviews, a survey, and two focus groups with program participants.

Urban interviewed key program stakeholders in May and June of 2018 to understand the program’s design, target population and eligibility criteria, and intended outcomes. The team also observed program meetings where proposed changes to program rules were discussed.

Urban conducted interviews with participants focused on their early experiences in the program.

Questions focused on (1) how they heard about the program and their experience with the application process and eligibility requirements, (2) their current financial and housing statuses and goals for the future, and (3) their reasons for deciding to enroll in the program or not.

After the interviews, Urban administered a survey to program participants. In October 2018, the survey was tested for length and clarity with 11 participants in person (using a paper survey). The finalized survey was administered at about the midway point of the first program year in February and March 2019. At that time, most participants had been enrolled in DC Flex for 5 to 10 months. The survey was administered electronically to all 121 people who were offered enrollment at the time (including people who declined enrollment), and 74 responded, including the 11 people who participated in the pretest.8 The sample for the survey was the full population of presumptive enrollees, although ultimately some of these enrollees would exit the program before their Flex accounts were set up and they received funds. The survey responses were not connected to administrative data, so the responses are not weighted. If survey respondents are significantly different than nonrespondents, that would increase bias in the survey analysis.

8 The survey did not change significantly between the initial test and the final survey. The survey in appendix A notes where the versions differed.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

5

The survey included questions about participants’ employment, their families, their banking habits, their financial goals, and their feelings of financial stress, along with program-specific questions about accessing their DC Flex account, their use of the rent subsidy, and their plans for paying rent if the subsidy ran out.9

The Urban Institute also held two focus groups of DC Flex participants in June 2019, with 11 total participants. The focus groups gave participants an opportunity to speak about their experiences in the program following its first year of implementation before re-enrolling in the second year, if they were still eligible. Participants were asked about how they heard about the program, if they found the program orientation and financial coaching sessions to be helpful, how they used their initial $7,200

subsidy disbursement, if they experienced changes in their financial and housing situation, and, for those who expected to participate, how they planned to use their DC Flex accounts in year 2.10

Quantitative Approach

Within months of DC Flex’s establishment through legislation, staff members from the DC Interagency Council on Homelessness (ICH), Department of Human Services (DHS), and Department of Housing and Community Development (DHCD) began conversations with The Lab @ DC and the Urban Institute about how best to field a quantitative evaluation of DC Flex so that it could inform policy decisions in DC and nationally. The District decided that the Lab would work alongside District agencies to launch the program as a randomized controlled trial (RCT), and that evaluation was written into the program’s regulations. Because the program was intended to be a proactive support for residents at risk of homelessness and housing instability, the District used historical records to identify potential applicants for DC Flex and then encouraged them to apply through letters, text messages, emails, and referrals from community-based organizations starting in December 2017. This approach could serve as a model for other jurisdictions seeking to rigorously evaluate pilot programs.

DC Flex applications were accepted between December 2017 and July 2018, and in total, 719 DC

residents submitted eligible applications. To conduct the RCT, the 719 applicants were entered into five random lotteries between January 2018 and July 2018. A technique called blocked randomization was used to promote similarity in the characteristics of households offered DC Flex in the lottery and 9 See appendix A for the full participant survey results.

10 See appendix B for the focus group interview protocol.

6

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

those who were not. This technique promotes a fair comparison of DC Flex to “business as usual” and allows us to understand what outcomes are caused by DC Flex.

Those applicants not selected in each lottery were eligible to be chosen in future randomized lotteries. The five lotteries were of varying sizes to ensure that program staff could perform eligibility verifications efficiently. Overall, applicants had roughly a 1 in 3 chance of being selected in the lottery in the first program year, but households that applied earlier had a higher probability of being selected.

Those not selected by the fifth lottery became the control group against which to compare outcomes.

The pre-analysis plan11 discusses the experimental design, randomization methodology, and measurement strategy in depth.

Exhibit 1 below provides an overview of the lottery design, including where 51 participants were removed from the analysis because of data availability.12

11 The pre-analysis plan for the quantitative analysis is available on the Open Science Framework.

12 These participants could not be observed for a full year as either being in the treatment or control group. We conducted a sensitivity analysis that included the 26 participants in this group who were enrolled for 6 to 11

months, and there was no meaningful shift in the results. Results are presented in Appendix D.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

7

EXHIBIT 1

Consort Diagram for DC Flex

By the end of September 2018, 229 families had been assigned to the DC Flex treatment group through the lotteries. Once selected, participants were required to submit documentation to verify their final eligibility for the program. At that stage, some families were ruled ineligible, while others opted not to enroll in DC Flex. In all, 102 of the 229 families had enrolled in DC Flex by the end of September 2018. A total of 439 families had not been offered DC Flex and became the control group.

The 229 families offered DC Flex overlap with the sample for the Urban Institute’s survey, but they are different in that the survey population was everyone who had enrolled in DC Flex by February 2018. The RCT results focus on just the 229 participants offered DC Flex by the end of September 2018 and the 102 of that 229 who had successfully enrolled and received their first funds.

Because of this distinction, the survey results should not be interpreted as representative of the views of these 102 participants, and vice versa.

The Lab’s outcome evaluation draws from Homeless Management Information System (HMIS) data on the use of emergency shelter, transitional housing, and other Continuum of Care–funded services, as well as administrative data on the rate at which families receive cash benefits through DC’s TANF program and the amount of benefit received by participants. As detailed in the pre-analysis plan, the Lab calculated both the intent-to-treat (ITT) and the Complier Average Causal Effect (CACE). ITT tells us the average impact of offering DC Flex, regardless of whether households enrolled in the program, but not the impact of actually using it because of partial compliance (i.e., not all those 8

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

who are offered the program meet final eligibility and choose to enroll). The CACE, on the other hand, tells us the effect of receiving DC Flex funds on homelessness.

Program Design and Implementation

The DC Flex program was motivated by a desire to test a new, more flexible model of rental assistance and to provide more options for formerly homeless and at-risk families to improve their housing and financial stability. DHS led the development of program regulations with input from ICH, DHCD, and The Lab @ DC. The program was funded in the District’s fiscal 2018 budget and began enrolling participants in May 2018.

Implementation Timeline

The DC Flex program took months to put together and implement, and qualitative and quantitative analyses have occurred at different times. Initial applications to the program were received in December 2017. The Lab then conducted an initial screen of applications to determine eligibility. The Lab then randomized applications that made it through this initial screen into the treatment or control group. Applicants assigned to the treatment group were required to attend an in-person meeting where DHS and Lab staff reviewed their applications in more detail to certify their eligibility. DHS

contracted with Capital Area Asset Builders (CAAB), a local nonprofit specializing in financial education, matched savings, and consumer education programs in the Greater DC area, to administer the program. Once The Lab and DHS confirmed an applicant’s eligibility, they notified the CAAB

program administrator who helped eligible applicants complete the enrollment process, attend an orientation, and get their subsidy accounts setup. In May 2018, the first participants began enrolling in the program.

The quantitative analysis presented in this report includes participants who were successfully verified and enrolled in the program by October 1, 2018, and includes individuals who have participated in the program for at least one year. Administrative data for this report are available through September 30, 2019. The qualitative analysis included a survey fielded in February and March 2019 and two focus groups held in June 2019. The survey was sent to every active participant as of the fielding start date, and the focus groups was composed of active participants. Exhibit 2 details the project timeline.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

9

EXHIBIT 2

Project Timeline

Date

Event

October 1, 2017

The District of Columbia Fiscal Year 2018 budget includes $1,250,000 in annual funds to serve approximately 120 families for up to 4 years, through Fiscal Year 2021

December 2017

DC Flex application opens, and letter and text message outreach sent to 9,000

households who applied for the rapid re-housing program or the Homelessness Prevention Program or submitted multiple applications to the Emergency Rental Assistance Program within the previous 48 months

January 2018

Initial applications received for DC Flex; first of five rounds of program lotteries is conducted by the DC Department of Human Services (DHS) and The Lab @ DC

February 2018

DHS and The Lab @ DC begin verification process for households selected in the lottery

May 2018

Participants who successfully verified their eligibility and accepted a final offer begin to enroll in DC Flex

July 2018

Program application closes, and fifth round of program lottery is run (final lottery for first program year analysis cohort)

October 1, 2018

All initial program slots filled

February/March 2019

Qualitative surveys sent to participants by Urban Institute March 1, 2019

Program rules change, and participants are allowed to

◼

Use emergency rental assistance while in the program.

◼

Owe back-rent at recertification as long as their name is still on the lease of a legal rental within DC and they do not have an eviction process underway.

May 2019

The first DC Flex participants (37) reach 1 year of program participation and begin recertification

June 2019

Urban Institute holds two focus groups with program participants September 30, 2019

Last date of outcomes data included in first program year analysis; 102 participants have been in the program for at least 1 year

Eligibility Requirements

To be eligible for DC Flex, applicants must be 21 years or older and have physical custody of at least one minor child. Applicants must be considered at risk of homelessness, meaning they have incomes up to 30 percent of the area median income and have recently applied for or received emergency or temporary assistance from a government-funded housing program administered by the District.13

13 The income eligibility for the DC Flex program varied by household size. A household of two must have an annual income below $26,500. A household of six must have an annual income below $38,400.

1 0

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

Applicants must also have a lease on a legal rental unit and be employed or have a history of recent employment, defined as having worked in the 6 months before submitting their program application.

The program’s eligibility requirements are an attempt to thread the needle between serving families most likely to benefit from the program while targeting assistance to those with the greatest needs. Multiple stakeholders noted that the combination of a smaller subsidy, more flexibility, and no case management made the program a good fit for families with low service needs and some financial stability. Thus, families were required to have a recent work history and be living in their own homes with their names on the lease. DC Flex was also intended to help address family homelessness in the District and was authorized under the District’s Homeless Services Reform Act. The law has income requirements, and therefore the program required that families have incomes below 30 percent of the area median income and have a recent history of applying for, or receiving, housing assistance.

Unlike typical federal voucher programs, DC Flex does not have quality standards for the apartments in which participants live beyond the rule that they be legal rentals registered with the District. DC Flex participants could not use their subsidy to rent units outside the District. Given DC’s proximity to Virginia and Maryland, staff considered allowing participants to use the assistance in surrounding jurisdictions, but legal counsel ultimately determined that using funds outside the District would not be allowed under the law authorizing DC Flex. However, the option remains a policy of interest for staff.

To recruit eligible applicants for DC Flex, DHS used administrative records to identify households that had applied for the rapid re-housing program or the Homelessness Prevention Program or had submitted multiple applications to the Emergency Rental Assistance Program within the previous 48

months. The agency then sent by letter and text message information about the DC Flex program and instructions on how to apply to the roughly 9,000 households that met the criteria.14

In total, DHS received 3,692 applications to the DC Flex program. More than one-half (1,987) of these applications were incomplete (program eligibility criteria were presented early in the application, 14 The Lab @ DC ran three experiments to see what kind of outreach was most likely to prompt residents to apply for DC Flex. The three experiments compared different types of outreach (mailed letters and text messages) and message content (for example, highlighting a positive identity as a responsible renter versus highlighting the program’s financial benefits). The Lab is still finalizing the results, but when possible, governments should try multiple types of communication when reaching out to potential clients. Significantly more families who were sent both a letter and a text message submitted an application to DC Flex, compared with families who received only a letter. No definitive answer was reached on whether a specific message type or content yielded more eligible applications.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

1 1

so many would-be applicants likely determined they were not eligible). Among those who completed the application, 795 applicants were screened out because they did not meet the eligibility criteria based on self-reported information, and 189 were duplicate applications. The most common reason applicants were screened out as ineligible was that they had not applied for or received homelessness or emergency rental assistance within the previous 48 months. When the application closed in July 2018, there were 719 unique eligible applicants.

In January 2018, the first 50 applicants were randomly selected to receive a conditional offer to enroll in DC Flex. The first 50 applicants selected into the program were invited to attend an in-person eligibility intake session at the DHS office, where they submitted paperwork to demonstrate their eligibility for the program. Those that successfully documented their eligibility were given a final offer of enrollment in DC Flex. Many applicants failed to submit the necessary paperwork or were otherwise found to be ineligible at this step. The lottery process was repeated four times, and applicants who were not selected were included in subsequent lotteries. By October 1, 2018, all 120

program slots had been accepted by applicants, but only 102 participants had started to receive funds.

The 102 households that were enrolled in the program as of October 1, 2018, make up the DC Flex enrollment group in the quantitative evaluation. When participants withdrew or were terminated from the program after enrollment, additional households were selected through the same lottery process.

In interviews and focus groups, participants generally reported that the application and eligibility screening process went smoothly. At application, some participants were concerned about the income limits and whether they might exceed the income limits at recertification. Some households were eligible but decided not to enroll, generally because they felt they could receive more assistance through other programs that subsidized all or nearly all of the rent. These families calculated that they would get more assistance, at least in the near term, in their current programs. Similarly, a few households that were offered DC Flex were living in project-based Section 8 housing. These households typically decided to stick with those programs when they were told that they could not receive assistance from multiple programs at the same time.

Most interviewees and focus group participants indicated that program administrators were able to answer questions and explain the program well. Participants appreciated the efficiency and patience of the eligibility screening and felt that the required documentation was not onerous. A few participants noted that they had to miss work to verify their eligibility in person and that additional meeting times outside of work hours would have been helpful.

1 2

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

Program Enrollment and Participant Characteristics As noted earlier, The Lab @ DC used random lotteries to assign eligible families to either the treatment group, those offered DC Flex, or the control group. In RCTs, the goal is to achieve virtually identical treatment and control groups across any observed and unobserved characteristics before the program begins. This “balance” allows a researcher to attribute any differences in outcomes after the program has been implemented to the program itself. The DC Flex team was largely successful in achieving this balance. Exhibit 3 displays the mean self-reported information for participants offered spots in DC Flex and the control group with the standardized differences between the groups. The standardized difference is a statistic used to measure how different two groups are from each other; differences above 0.10 are notable, and those above 0.25 are concerning. Notable differences were observed for age, rent amount, days since the applicant had been employed (for those unemployed at the time of application), and the share who had previously applied for or received rapid re-housing in the 48 months before applying. These differences should be kept in mind when interpreting the quantitative results.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

1 3

EXHIBIT 3

The Treatment and Control Group Demonstrate Baseline Equivalence Offered

Standardized

Household information

DC Flex

Control

differences

Age

32.3 (7.9)

31.4 (7.8)

0.114

Number in household

3.3 (1.3)

3.3 (1.3)

0.009

Number of children

2.1 (1.1)

2.1 (1.2)

0.002

Currently working

79.0%

78.4%

0.017

Days since last reported date of employment*

(if not currently working)

87.3 (101.6)

97.9 (71.9)

0.121

Annual income

$17,195 (9,413)

$17,782 (9,723)

0.061

Rent amount

$863 (442)

$932 (450)

0.155

Split rent for housing unit

with someone not in their household

8.3%

8.0%

0.012

Sought services at Virginia Williams Family Resource

40.6%

40.8%

0.003

Center in 48 months before application

Applied for Emergency Rental Assistance Program

48.5%

44.0%

0.090

in past 48 months

Applied for Homelessness Prevention Program

13.5%

15.0%

0.043

in past 48 months

Applied for rapid re-housing in past 48 months

44.1%

49.9%

0.116

Applied for transitional housing in past 48 months

5.2%

4.6%

0.032

Currently receiving rapid re-housing

31.4%

36.7%

0.110

Currently receiving transitional housing

0.9%

1.6%

0.065

Currently receiving Housing Choice Voucher

6.1%

4.3%

0.080

Currently receiving project-based Section 8 housing

2.2%

2.1%

0.009

Source: DC Flex Application System self-reported application data.

Notes: The treatment group (“offered DC Flex”) was 229 families, and the control group was 439, for a total of 668 applicants who were eligible for the initial lottery and completed one year in the evaluation. Standard deviations are reported in parentheses. The standard deviation of a binary variable is sqrt[ n * P * ( 1 - P ) ] where P=mean.

* There were 51 (21 percent) unemployed individuals in the treatment group and 100 (22 percent) individuals unemployed in the control group at the time of the application.

Families assigned to the treatment group were required to attend an in-person appointment where they presented the necessary documentation to confirm their program eligibility. Some families chose not to attend this meeting; others attended and were deemed ineligible—often because their incomes were too high. Still other families made it through the in-person eligibility screening but decided not to enroll or dropped out before getting their accounts set up. Of the 229 households randomized into the DC Flex group, 102 ultimately enrolled and received rental assistance through the program by October 1, 2018. These 102 households are referred to as the “compliers.” Unfortunately, we do not have data on how many of the 127 “noncompliers” did not show up to their follow-up screening, how many were ineligible, and how many declined to enroll in the program.

Exhibit 4 shows the differences in demographics and past program participation between compliers and noncompliers. In interviews and focus groups, families who decided to enroll reported 1 4

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

their primary motivation was the offer of 4 years of rental assistance. Participants referred to this assistance as a “blessing” and a “life-saver.” Before enrolling in DC Flex, participants had to exit any other local or federal rental assistance programs. As shown in Exhibit 4, more than 28 percent of DC

Flex participants reported they were receiving rapid re-housing assistance at the time of enrollment.

This is noteworthy because the average monthly subsidy families receive in rapid rehousing ($1,713 in Fiscal Year 2021) was almost three times as high as the $600 average monthly subsidy offered in DC

Flex. One of the city’s goals in designing DC Flex was to offer an alternative to rapid rehousing that was more attractive to working families, and the data suggest they were at least partially successful.

Participants who chose DC Flex over rapid re-housing cited the predictability of knowing that they could receive assistance for 4 years as long as they remained eligible and the lack of required case management meetings as major factors in their decision.

Some eligible households decided not to enroll, generally because they felt they could receive more assistance through other programs. For example, some households were living in apartments where the market rent was $2,000 or more, but their rapid re-housing or transitional housing programs subsidized all or nearly all the rent. These families calculated that an annual subsidy of $7,200 from DC Flex would be depleted within a few months and that they could get more assistance, at least in the near term, in their current programs. Similarly, a few households that were offered DC

Flex were living in public housing or project-based Section 8 housing. These households typically decided to stick with their current programs when they were told they could not receive assistance from multiple programs at the same time.

Fifty-five percent of applicants offered DC Flex did not enroll. Households that were offered a subsidy were more likely to accept and enroll in DC Flex if they were working and had a higher average annual income and wage than noncompliers. Households offered a DC Flex slot were less likely to accept if they were receiving rapid re-housing or an HCV. This information should be informative to policymakers as it could be used to refine eligibility requirements if the program is expanded or piloted in other jurisdictions. The Lab @ DC plans to leverage some of these differences to attain a more precise causal estimate of the program in future program years, as described in the pre-analysis plan.15

Most of the 102 families that did enroll had one adult and two children, and 85 percent of adults in the program were working. The average annual household income reported was $18,505, or $1542 a 15 Page 26 of the pre-analysis plan.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

1 5

month, while the average monthly rent reported was $924; this suggests that most participants were extremely rent-burdened prior to participating in DC Flex. In interviews, several participants talked about having stable jobs as crossing guards, teacher’s assistants, or bus drivers that didn’t pay enough to consistently afford rent.

EXHIBIT 4

Comparison of DC Flex Participants (Compliers) with Those Offered DC Flex Who Did Not Enroll (Noncompliers)

P-value

difference

Standardized

Household information

Compliers

Noncompliers

in means*

differences**

Age

32.5 (8.4)

32.1 (7.5)

0.681

0.055

Number in household

3.2 (1.3)

3.4 (1.4)

0.362

0.122

Number of children

2.0 (1.1)

2.1 (1.2)

0.502

0.090

Currently working

85.3% (0.36)

73.8% (0.44)

0.035

0.286

Days since last reported date of

employment (if not currently working)

18.8 (73.0)

20.1 (47.3)

0.867

0.022

Annual income

$18,505 (8,918) $16,143 (9,797)

0.050

0.263

Rent amount

$924 (376)

$813 (484)

0.058

0.257

Split rent for a housing unit with

someone not in their household

7.8% (0.27)

8.7% (0.28)

0.811

0.032

Currently receiving rapid re-housing

28.4% (0.45)

34.1% (0.48)

0.360

0.123

Currently receiving transitional housing

1.0% (0.10)

0.8% (0.09)

0.881

0.020

Currently receiving Housing Choice

Voucher

2.9% (0.17)

8.7% (0.28)

0.071

0.248

Currently receiving project-based

Section 8 housing

0.0% (0.00)

4.0% (0.20)

0.042

0.286

Source: DC Flex Application System self-reported application data.

Notes: The “compliers” are the 102 families who were randomized into the DC Flex group, enrolled in the program, and received assistance. The “noncompliers” are the 127 families who were randomized into the DC Flex group and did not participate in the program. Standard deviations are reported in parentheses.

* <0.05 is statistically significant.

** >0.25 is concerning.

Exhibit 5 shows DC Flex enrollments by month during the first program year. Nearly two-thirds (64 percent) of enrollments occurred in the first 2 months of the program. Of the 102 families who enrolled in DC Flex and had accounts set up, 8 families voluntarily exited the program, and 2 had their participation terminated.

1 6

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

EXHIBIT 5

Monthly Enrollments into DC Flex, May 2018–October 2018

40

99%

100%

39

83%

78%

30

64%

26

20

38%

16

15

10

5

1

0

May 2018

June 2018

July 2018

August 2018 September 2018 October 2018

Source: Program data provided by Capital Area Asset Builders on January 29, 2019.

Program Orientation

Once enrolled, DC Flex participants were required to attend a 2-hour program orientation. The orientations were offered in a group setting in CAAB’s downtown office building. The first hour of the orientation provided participants an overview of the DC Flex program, the eligibility requirements for enrollment and recertification, and instructions on how to access their DC Flex accounts and remain in compliance with program rules. The second hour of the orientation was devoted to a more general introduction to financial literacy and budgeting titled “Can I Afford My Life?”

Given the unique and somewhat complicated structure of the DC Flex accounts (described in further detail below), the orientation was critically important for participants to understand the program rules before they began withdrawing funds. In focus groups, most participants reported that the orientation clearly communicated how the program worked and that there was nothing they wished had been covered in orientation that they only learned later. Some participants, however, thought their escrow accounts would replenish at the start of the calendar year or when they spent down their initial $7,200, rather than 12 months after their first withdrawal.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

1 7

CAAB staff believe the orientations improved over time. The orientations with the initial cohort of participants, those that enrolled in April and May of 2018, were the largest. There were often 20 or more participants in each session, which made it difficult for the trainer to ensure that all the participants fully understood the information. Staff felt that this might have contributed to the higher rate of program terminations among the initial enrollees, as some participants did not clearly understand program rules. In later cohorts, the pace of enrollment slowed, the orientation groups were smaller, and the trainers were able to warn participants of potential violations based on past experiences. This included more specific guidance on how to avoid overdrafts and ineligible uses of DC Flex funds. CAAB has also begun offering refresher orientations to participants and made their orientation materials available online.

Flexible Subsidy Accounts

Each DC Flex participant is set up with two accounts at a participating bank: an escrow account and a checking account. The escrow account holds the full amount of rental assistance the participant has available for the year, and the checking account holds the amount the participant has available to pay their rent each month. The escrow account starts off at $7,200. The account administrator, CAAB, then transfers an amount equal to each participant’s full monthly rent into the checking account. The participant can then decide how much they want to withdraw or direct debit from their checking account to pay their rent. If participants withdraw cash from their checking account, they must submit proof to CAAB that they paid their full rent to their landlord. Confirmation can be in the form of a picture (if using a money order) or a receipt from the landlord. Each month, CAAB transfers money from the escrow account to bring the checking account balance back to the amount of their monthly rent. This process continues until no money is left in the escrow account.

Exhibit 6 provides an example, taken from the DC Flex Program regulations, of how the flexible subsidy account might work for a participating family in its first year in the program. In this example, the family’s monthly rent is $1,600. In the first month, CAAB transfers $1,600 from the escrow account to the checking account. The family then withdraws $1,000 from its checking account to go toward rent and uses $600 from other funds for the remainder. In the second month, CAAB transfers $1,000 from the remaining balance of the escrow account to bring the checking account balance back to $1,600. During the year, the family’s use of the subsidy varies. In months 5 and 6, the family can pay the full rent on their own and does not make any withdrawals from the account, but in month 11

the family uses the subsidy to pay the full monthly rent. If, as in the example shown below, a participant still has a balance in their escrow account at the end of the program year, they can either 1 8

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

add it to their balance for the next program year or use up to $500 to pay for other housing-related expenses.

EXHIBIT 6

Hypothetical Annual Subsidy Account Withdrawals for Family with Monthly Rent of $1,600

Amount

Savings

Amount of

Amount of

remaining in

(escrow)

subsidy

Amount

subsidy

Amount

checking

balance to

transferred to

accessible

used by

paid by

account at

start month

checking

via checking

household

household

month’s end

($)

account ($)

account ($)

($)

($)

($)

Month 1

7,200

1,600

1,600

1,000

600

600

Month 2

5,600

1,000

1,600

1,000

600

600

Month 3

4,600

1,000

1,600

500

1,100

1,100

Month 4

3,600

500

1,600

300

1,300

1,300

Month 5

3,100

300

1,600

0

1,600

1,600

Month 6

2,800

0

1,600

0

1,600

1,600

Month 7

2,800

0

1,600

600

1,000

1,000

Month 8

2,800

600

1,600

400

1,200

1,200

Month 9

2,200

400

1,600

400

1200

1,200

Month10

1,800

400

1,600

800

800

800

Month 11

1,400

800

1,600

1,600

0

0

Month 12

600

600

600

200

1400

400

DHS provided funding to CAAB to administer the flexible subsidy accounts for DC Flex participants. Although the accounts were originally envisioned as operating like a line of credit, District officials had a difficult time finding local financial institutions interested in participating. Many banks felt the volume of business they would get from the program did not justify the amount of time it would take to administer it. Ultimately, the District settled on a more traditional bank account structure and recruited City First Bank to set up escrow and checking accounts for participants.

Setting up the accounts was delayed because the withdrawal process required joint management of the bank accounts between CAAB and DC Flex participants. Ultimately, the accounts were set up so that only participants could authorize withdrawals from their checking accounts but only CAAB could transfer money from participants’ escrow accounts to their checking accounts.

Roughly 40 percent of DC Flex participants did not qualify for a checking account from City First Bank because they failed its ChexSystems’ background check.16 Although what disqualified DC Flex 16 ChexSystems is a check verification service and consumer credit reporting agency used by 80 percent of commercial banks to screen applicants for checking and savings accounts

(https://en.wikipedia.org/wiki/ChexSystems).

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

1 9

participants is unclear, the most common reasons for failing ChexSystems’ background checks generally are bounced checks, insufficient fund notices, and instances of fraud. DHS identified another bank partner, United Bank, for participants who were denied accounts by City First. Because of some of the challenges in setting up the accounts during the first year, a small number of participants received more than their $7,200 annual allotment. For those families, CAAB took any overpayments out of the participants’ year 2 funds.

For the most part, participants were happy with the banks, although the lack of City First branch locations near participants’ homes was a problem for participants. Some focus group participants reported that they opened their own checking and savings accounts with their banks as a consequence of DC Flex and began relying on online bill pay and debit cards more than money orders.

Before they could begin withdrawing money from their accounts, DC Flex participants needed to show they were certified occupants of a legal rental property with the District. This meant providing to CAAB the business license for the owner or property management company and the certificate of occupancy to verify that the residence was a legal residence. The program designers anticipated that participants could gather this documentation on their own without the help of their landlord or property manager and advised participants not to inform their landlord or property manager about their program participation to avoid any increases in rent because of the availability of the subsidy. In the participant survey, however, 45 percent of respondents reported that their landlord knew they were in the program and an additional 16 percent were not sure whether their landlord knew (exhibit 7). In focus groups, several participants reported that their property manager learned they were in the program when they inquired about the business license. Program staff acknowledged that acquiring the business license and certificate of occupancy was more difficult than had been expected. The requirements also created challenges for some participants who were renting a room from a friend or family member and discovered that the unit was not registered as a legal rental.

2 0

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

EXHIBIT 7

Participant Survey: Does Your Landlord or Management Company Know That You Are in the DC Flex Program?

45%

39%

16%

Yes

No

Not sure

Source: DC Flex participant survey administered by the Urban Institute in February and March 2019.

DC Flex participants were generally able to withdraw funds and verify their rent payments without any problems. In the participant survey, 9 percent of participants reported any challenges with withdrawing funds from their accounts, and 3 percent reported challenges with sending confirmation of their rent payment to CAAB.

DC Flex participants were also able to log into a portal system to monitor their checking and escrow account levels.17 However, while 58 percent of participants reported monitoring how much money was available in their Flex checking accounts, only 42 percent reported monitoring how much was available in their escrow accounts. In focus groups, some participants reported that they were unaware that they had spent down their full $7,200 for the year until they were notified by their bank.

Several participants recommended an app for their phones that they could log into to see how much money they had left in their escrow account to help them manage the funds for the whole year.

17 In year 2 of the program, participants’ access to the escrow account was cut off after some participants used the information to withdraw money directly from their escrow accounts.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

2 1

Participant Services

DC Flex participants were required to attend a mandatory program orientation before their account could be set up and to attend one individual and one group financial coaching session within the first year of the program. Beyond these sessions, however, there is no requirement for participants to contact program staff beyond showing proof that they paid their rent to get money transferred from their escrow to their checking accounts each month. The DC Flex Program did not include case management, housing search assistance, employment services, or help accessing benefits. CAAB did employ a full-time program manager who helped some participants with these additional services when they requested assistance. The lack of required services was intentional to more clearly test the effectiveness of the subsidy. In a small pilot, it was important for DC government to understand the effect of the flexible funding on its own, rather than in combination with additional services.

DC Flex’s financial coaching sessions were tailored toward the specifics of the DC Flex program and included topics such as household budgeting, how to handle income fluctuations, setting long-term goals, and dealing with past-due rent. The first coaching session was a one-on-one session with the CAAB administrator, while the second was done in a group with other DC Flex participants. In the first session, participants reviewed their monthly income and expenses, completed a financial behavior survey, and developed a month-by-month plan for how much of their rent they would pay with their DC Flex accounts and how much from other sources of income. The group coaching sessions were less structured and often consisted of participants’ sharing information with one another.

Participants saw CAAB as a significantly positive part of the program. Many of the participants struggled with financial management and budgeting and thought that the coaching sessions were needed. Of the 59 participants who had had at least one session at the time of the survey, 95 percent found the coaching sessions helpful. In the focus groups, participants said the coaching sessions led to specific changes in their finances, including setting up automatic payments to avoid late fees and strategically deciding which bills to pay to improve their credit scores. Participants reported that the one-on-one sessions were valuable but preferred the group sessions because they could learn from other participants.

In focus groups, participants generally appreciated that the program let them decide how much assistance to use each month and did not require them to meet with a case manager regularly. Some participants cited this as a major benefit of the program compared with rapid re-housing. But some participants wished that the program had offered more robust services, including more frequent and 2 2

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

intensive financial coaching. One participant said: “I wish the program had someone sit down with me every month, here’s how to boost your credit, etc. Some people just need that guidance.”

CAAB staff felt that most participants were “thriving” in the program but that a smaller percentage (they estimated 20 percent) needed more support. All of the CAAB participants had the program administrator’s phone number, and some reached out to him frequently with questions or to seek advice. But the focus group showed that some participants who were struggling did not seek help.

Both administrators and focus group participants suggested that additional assistance in finding affordable units or accessing employment would be beneficial. Responses from the focus groups suggested that this was largely because of the difficulty of locating affordable units and well-paying jobs in Washington, DC. Specifically, both participants and staff expressed an interest in developing relationships with local employers to help participants find better-paying jobs.

Recertification

At the end of each program year, participants must recertify their eligibility. Once they recertify, another $7,200 is added to their escrow accounts. While applicants must have incomes at or below 30

percent of the area median to be eligible to enroll in DC Flex, at recertification their incomes must be at or below 40 percent of the area median. Program designers wanted to have a higher income limit to reduce any disincentives to earning; however, not exceeding the 40-percent limit was required to conform with the law that authorized DC Flex in the initial program rules. Participants also could not owe any back-rent to their landlord, and to prevent “double-dipping,” they could not receive assistance from any other local or federal rental assistance program after they began using their DC

Flex accounts.

The District changed the program rules regarding back-rent and use of other assistance programs during the first program year. The Emergency Rental Assistance Program (ERAP) is a local program available to District residents with incomes at or below 40 percent of the Area Median Income.

Eligible households can apply for assistance to cover up to five months of rental arrears.18 The original DC Flex rules prohibited families from receiving assistance through ERAP while also getting assistance through DC Flex. These rules were changed to all participants to use ERAP once per year but only after they had exhausted their annual DC Flex subsidy. Participants who owed back rent at 18 For more on ERAP see https://dhs.dc.gov/service/emergency-rental-assistance-program.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

2 3

recertification were permitted to use ERAP to pay it before recertifying. The changes were motivated by program data that showed that by January 2019, many of the first households to enroll in DC Flex had spent their entire $7,200 subsidy for the year in the first seven months of the program. In addition, some families were contacting CAAB because they could not pay their rent and were at risk of eviction. District officials feared that the initial recertification requirements could result in widespread program terminations, evictions, and returns to homelessness.

The District is still pursuing changes to the income requirements at re-certification to allow participants to further increase their incomes without losing eligibility for DC Flex. So far, however, there have been very few instances where participants have been denied recertification for exceeding the income threshold.

The desire to relax the recertification rules was weighed against several concerns. First, changing the program rules midway through the first year could confuse participants. Second, making changes to the program 1 full year into program implementation would reduce the generalizability of the quantitative evaluation results to future years because the experience would be different based on when participants joined and assessing the impact of the DC Flex program would be more difficult.

Programmatically, families who access ERAP funds would be receiving a higher annual total subsidy amount. This could be perceived as unfair by families who do not access ERAP.

The District ultimately decided to relax the requirements because these concerns were outweighed by the desire to reduce the risk of eviction and homelessness for participants. Although the District revised the program regulations in spring 2019, participants were not informed of the rule changes until their annual recertification meetings. DHS program data showed low take-up of ERAP

among DC Flex program participants, and in focus groups, some participants were still under the impression that they could not receive ERAP assistance while in DC Flex.

At the end of the first program year, DHS reports that at least 12 percent of DC Flex participants were not able to recertify for a second year of DC Flex. Among the reasons for not recertifying were nonresponse, failure to meet eligibility requirements, misuse of funds (two participants), and the purchase of a home (one participant).

2 4

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

Outcomes

The qualitative evaluation focused on participants’ use of the accounts, their perception of stability and mobility, and their self-defined economic well-being. Elements of The Lab @ DC’s quantitative analysis are included to help inform how participants used the accounts throughout the program.

Use of Flex Accounts

The survey of participants (in February and March of 2019) captured their early use of their DC Flex accounts and their planning around future use. The Lab @ DC conducted a quantitative analysis of the bank account transaction data provided by CAAB on the 102 DC Flex households enrolled between May 1, 2018, and October 1, 2018. The survey and the account analysis cover different time periods and slightly different samples of participants. Both samples include 13 participants who received DC

Flex funds but exited the program or were terminated before completing a year in the program. Their reasons for exit or termination varied: some exited the program voluntarily, some were terminated for misuse of funds, and some left DC.

The survey results offer some insight into how participants were thinking about account use roughly midway through the program year. Reviewing the survey results alongside this analysis shows interesting differences in how participants stated they would use the accounts and how the accounts were actually used. However, as stated earlier, survey respondents are not necessarily representative of all DC Flex participants.

Midway through the program, a plurality of survey respondents (32 out of 74, or 43 percent) reported that they used the DC Flex account to pay all or most of their monthly rent until it ran out (exhibit 8). Smaller shares of participants change their approach from month to month (28 percent), use a mixture of DC Flex and other income every month (15 percent), and use DC Flex only as a last resort (9 percent).

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

2 5

EXHIBIT 8

Participant Survey: How Have You Used Your DC Flex Account?

43%

28%

15%

9%

4%

To pay all/most rent Depends on my income Use a mix of Flex and Only use to pay rent as Missing/refused

until it runs out

and expenses that

other sources

a last resort

month

Source: DC Flex participant survey administered by the Urban Institute in February and March 2019.

Reviewing account use after 1 year in the program, however, shows that only 13 percent of the 102 families enrolled in DC Flex with at least 1 year of CAAB data exhausted their full $7,200 six months into the program.

Exhibit 9 illustrates the month after enrollment that a participant ran out of DC Flex funds. For example, 42 percent of participants had exhausted all of their DC Flex dollars by their ninth month in the program. Sixty of the 102 households who remained in the program for a full year exhausted their funds by the end of the first year.

2 6

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

EXHIBIT 9

When DC Flex Participants Exhausted Their First Year Program Funds Cumulative share of

participants who have

Month after enrollment

exhausted the $7,200

1

0

2

0

3

0

4

1

5

4

6

13

7

21

8

32

9

43

10

46

11

51

12

59

Source: Capital Area Asset Builders program data.

Notes: Based on the 102 participants who had accounts setup during the analysis period.

* People who have exhausted the $7,200 are those who have used $7,100 or more by the end of their first year.

There are a few possible explanations for why 33 percent of participants spent down their accounts by the eighth month. Participants may have entered the program with an immediate need for rental assistance, or they may have relied on DC Flex to pay rent while using other income, which would ordinarily have gone to rent, to build up savings.

There was much less range in how participants responded to the question of whether they would expend the full account by the end of the program year. Nearly 70 percent of respondents believed they were either very likely (41 percent) or somewhat likely (28 percent) to use up all their DC Flex subsidy before the end of the program year (exhibit 10).

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

2 7

EXHIBIT 10

Participant Survey: How Likely Is It That You Will Use All Your DC Flex Account Before Your Next Disbursement of $7,200?

41%

28%

12%

9%

9%

Very likely

Somewhat likely

Somewhat unlikely

Very unlikely

Not sure/missing

Source: DC Flex participant survey administered by the Urban Institute in February and March 2019.

However, the account data show that after 1 year in the program, DC Flex participants had an average of $543 (Standard Deviation: $1,683) of unused funds in their DC Flex accounts. For context, the average monthly rent based on flex account data was $1,147, indicating that, on average, participants had less than a month’s rent left at the end of the year.

Exhibit 11 tabulates the amount of DC Flex funds remaining for each of the 102 participants after 12 months. The left side of the table shows the percentage of participants with less than the specified amount remaining in the account. For example, by the end of their first program year, 58 percent of participants had less than $100 in their DC Flex accounts, indicating that they used the full $7,200

during the year. Eighty-six percent of households had less than $2,000 remaining in their accounts after 12 months.

2 8

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

EXHIBIT 11

Amount of Money Left in Participants’ Escrow Accounts at End of First Year Cumulative

share of

Range of dollar amount

Share of

Dollar amount left at month 12

participants

left at month 12

participants

Less than 1

47.06

0–0.99

47.06

Less than 100

58.82

1–99

11.76

Less than 500

62.75

100–499

3.93

Less than 1,000

69.61

500–999

6.86

Less than 2,000

90.2

1,000–1,999

20.59

Less than 3,000

92.16

2,000–2,999

1.96

Less than 4,000

95.1

3,000–3,999

2.94

Less than 5,000

97.06

4,000–4,999

1.96

Less than 6,922

100

5,000–5,761

2.94

Source: Capital Area Asset Builders program data.

Notes: Data is for 102 DC Flex participants with accounts setup during the analysis period.

The distribution of remaining balances after 12 months is displayed in the right side of Exhibit 11.

For example, 47 percent of participants had 99 cents or less in their account after 12 months, and an additional 12 percent had between $1 and $99. A meaningful number of participants (21 percent) had between $1,000 and $1,999 in their accounts. Because DC Flex funds will be carried over to the next year if the household remains eligible for DC Flex, this result suggests that some families may elect to keep roughly 1 month’s rent available as an additional buffer for the next program year. Future analysis will try to determine the motivations for participants that choose not to spend-down their full subsidy amount during the program year.

For several reasons, program staff expected that as the program matured fewer participants would exhaust their subsidy within the first 8 months of the program year. First, many participants entered the program with debt (including owing back-rent to their landlord) and used their DC Flex subsidy to pay their full rent so that they could use other income to catch up on other debts. Second, participants may be focusing on short-term stability, rather than longer-term planning, a possibility backed by research on the effects of scarcity (Mullainathan and Shafir, 2013). This pattern may change in the second year of the program as participants catch up on past debts and, hopefully, establish more trust in the program and the long-term availability of the funds. Third, program staff believed that DC Flex was influencing families’ financial decisions. Some families expressed interest in moving to apartments with lower rents to further stretch their DC Flex subsidy. Families were also becoming savvier about avoiding high-interest loans and other financial pitfalls. Finally, participants may feel less pressure to spend the money quickly now that they have established more trust and understanding of how the program operates.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

2 9

However, many participants may still follow the pattern of spending all their annual subsidy in the first half of the year and then trying to “hang on” until they reach recertification. Both participants and stakeholders indicated that the value of the subsidy—which was set at roughly half the average annual subsidy in the voucher program—was too low for many households with extremely low incomes to afford rent in Washington, DC. They recommended either increasing the subsidy amount or allowing participants to use their subsidy in Maryland suburbs with lower rents.

In general, participants said that they liked the flexibility of the DC Flex program. While some focus group participants struggled with planning or thought that the program required micromanagement of finances, most indicated that they liked the way the program was structured. For the most part, participants did not want to receive the full $7,200 as a lump sum or have the subsidy calculated based on their income. Many participants wished the funds could also be used for expenses besides rent.

Homelessness and Housing Stability

Interviewees and focus group respondents suggested that a larger subsidy amount was needed to meet rising housing costs in Washington, DC. The strain of meeting high housing costs, especially for larger families, was a common refrain. Many participants were previously receiving rapid re-housing assistance which covered all, or nearly all, of their monthly rent. DC Flex participants coming from that program may have experienced large increases in the amount of their income they had to contribute to rent. For example, a participant with a monthly rent of $2,500 that was previously being paid for with rapid re-housing assistance, would quickly exhaust it’s $7,200 annual subsidy from DC Flex. This was directly referenced by several focus group participants as a primary reason their families were unable to continue paying rent even with the DC Flex subsidy. Many participants and some program staff felt that the subsidy amount should be increased, or the program should be targeted to households with slightly higher incomes.

Despite these challenges, 76 percent of DC Flex survey respondents reported that they had enough money to pay their rent that month. Of those who believed they would spend all their subsidy before the end of the year, 80 percent reported that they had a plan for paying their rent after they had used up their subsidy. In focus groups, several participants indicated that access to the DC Flex subsidy is what allowed them to continue making ends meet after they were no longer eligible for other subsidy programs. For individuals who exited rapid re-housing to enroll in DC Flex, the program served as a crucial backstop for their housing stability. Even so, these households still indicated that at 3 0

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

times they needed to negotiate with their landlord and/or shift costs (e.g., delay payment of the electricity bill to pay rent) to make ends meet. This may be an indication that even households with relatively stable housing need additional subsidy to thrive in an expensive rental market, especially larger families given the scarcity of larger rental units. Families coming from programs with deeper subsidies may not be used to the shallow nature of the DC Flex subsidy or be used to planning out their subsidy use each month. The need to shift resources from one household expense to another will be an important metric for future evaluations.

Housing Mobility

During the first year of the study, the 74 DC Flex participants surveyed reported mixed rates of satisfaction with their homes and neighborhoods. A plurality of respondents (36 percent) felt “neutral”

about their current house or apartment, while 35 percent were very satisfied or satisfied and 24

percent were dissatisfied or very dissatisfied (exhibit 12). Similarly, 31 percent of respondents were neutral about the neighborhood where they lived, 38 percent were satisfied or very satisfied, and 25

percent were dissatisfied or very dissatisfied (exhibit 13).

EXHIBIT 12

Participant Survey: How Satisfied Are You with Your Current House or Apartment?

36%

19%

16%

12%

12%

5%

Very satisfied

Satisfied

Neutral

Dissatisfied

Very dissatisfied

Missing/refused

Source: DC Flex participant survey administered by the Urban Institute in February and March 2019.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

3 1

EXHIBIT 13

Participant Survey: How Satisfied Are You with Your Neighborhood?

31%

19%

19%

16%

9%

5%

Very satisfied

Satisfied

Neutral

Dissatisfied

Very dissatisfied

Missing/refused

Source: DC Flex participant survey administered by the Urban Institute in February and March 2019.

While relatively few families responding to the survey reported moving during the program year or believed that access to the DC Flex subsidy would increase their likelihood of moving, many focus group participants spoke of their desire to move. In the participant survey, 38 percent of respondents reported that being in DC Flex increased the likelihood that they would move, 39 percent reported it made no difference, and 18 percent said it made them less likely to move. Focus group participants and interviewees indicated their desire to eventually purchase homes in the District and saw DC Flex as a means for saving toward that goal. Other focus group attendees signaled their desire to move from one part of the city to another, and multiple participants wanted to move out of the city, largely because of the significant cost of living in Washington, DC. Stakeholders and focus group participants indicated that there are barriers to moving that the DC Flex program does not necessarily ameliorate, such as security deposits, moving expenses, and credit checks.

Homelessness and Use of Continuum of Care Services

The Lab @ DC’s randomized evaluation of DC Flex shows how participants’ housing stability and use of homeless services differs from the comparison group. Although many measures can illustrate a family’s housing stability, the primary outcome of interest for the quantitative analysis was entries into 3 2

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

the formal homeless services system, measured by the rate at which participants entered emergency shelter or transitional housing at any time during the first program year. As detailed in the pre-analysis plan, The Lab calculated both the Intent-To-Treat (ITT) and the Complier Average Causal Effect. ITT

tells us the average impact of offering DC Flex, regardless of whether households enrolled in the program, but not the impact of actually using it because of partial compliance (i.e., not all those who are offered the program meet final eligibility and choose to enroll). The CACE, on the other hand, tells us the effect of actually receiving DC Flex funds on homelessness. The results reported below are preliminary and will be the subject of a separate, more detailed report by The Lab @ DC.

Exhibit 14 shows the rate of homelessness for the 229 participants offered DC Flex (the treatment group) and the 439 participants in the control group. The ITT coefficient of -0.1 percentage point is the difference between the two groups. This small difference is not statistically significant.19

There is no meaningful difference in entries to emergency shelter or transitional housing between participants offered DC Flex and the control group.

EXHIBIT 14

Use of Emergency Shelter or Transitional Housing in Program Year 1 for Treatment and Control Groups

Share who entered emergency shelter or

transitional housing

Control group outcome

4.7%

Offered DC Flex outcome

4.6%

Intent-to-treat coefficient (SE)

-0.1 percentage point (1.7pp)

Source: Homeless Management Information Systems data.

Notes: The control group is 439 families, and the “offered DC Flex” group is 229 families. SE = standard error.

The goal of the DC Flex pilot is to determine whether the program should be extended and expanded to a larger scale. At scale, many of the issues of non-compliance seen in the pilot will be mitigated and those that remain will have relatively little cost. For this reason, the Lab’s main focus is on understanding what happened to the 102 participants who enrolled in DC Flex by the end of September 2018 – the CACE estimate.

19 We conduct a randomization inference test of significance for all outcomes in this report. The method is detailed in the pre-analysis plan and will be discussed in greater detail in further publications. We report statistical significance at the 0.05 level.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

3 3

Exhibit 15 shows the rate of homelessness for the 102 participants who receive DC Flex funds (compliers) and an estimate of what that rate would be for the control group (the counterfactual compliers, in statistical terms). In preliminary analysis of the first program year, The Lab found no meaningful or statistically significant differences in entries to emergency shelter or transitional housing between participants who received DC Flex funds and the counterfactual compliance group.

During the first program year, 1.8 percent of participants who received funds entered emergency shelter or transitional housing. Without the DC Flex program, we estimate that 2 percent of participants would have joined these programs. This small difference—a CACE coefficient of 0.2

percentage point—is not statistically significant.

EXHIBIT 15

Homelessness Rate in Year 1 for Compliers and Counterfactual Compliers Share who entered emergency shelter or

transitional housing

Complier outcome

1.8%

Counterfactual complier outcome

2.0%

CACE coefficient (SE)

-0.2 percentage point (3.7pp)

Source: Homeless Management Information Systems data

Notes: The compliers are the 102 families who were randomized into the DC Flex group, enrolled in the program, and received assistance. CACE = Complier Average Causal Effect. SE = standard error.

As a secondary outcome, the Lab examined the effect of DC Flex on participants’ interactions with the Continuum of Care (CoC). This measure includes the uses of emergency shelter and transitional housing in the homelessness outcomes above, but also captures less-intensive services like coordinated assessments, short-term homelessness prevention services, and rapid re-housing. For simplicity, this outcome is referred to as “support service use” and is measured as the percent of families applying for services during the program year. While this outcome includes many homelessness prevention services, this measure alone is not a comprehensive indicator of housing stability and notably does not include public housing, which is not included in the HMIS data, because it is administered the DC Housing Authority. Additional measures will be included in future results that separate out the programs used by the intensity of the services provided.

The same ITT and CACE analyses done for homelessness were conducted to measure the effect of DC Flex on support service use. Exhibit 16 shows support service use among the 229 participants offered DC Flex (the treatment group) and the 439 participants in the control group. The ITT

coefficient of -12.7 percentage points is the difference between the two groups. This difference is statistically significant.

3 4

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

EXHIBIT 16

Support Service Use by Treatment and Control Groups

Rate of Use of Services

Offered DC Flex outcome (N=229)

32.2%

Control group outcome

44.9%

Intent-to-treat coefficient (SE)

-12.7 percentage points (4.2pp)

Source: Homeless Management Information Systems data.

Notes: The “offered DC Flex” group is 229 families, and the control group is 439 families. SE = standard error.

Exhibit 17 shows the rate of service use for the 102 participants who receive DC Flex Funds (compliers) and an estimate of what the rate would be for the control group (the counterfactual compliers). During the first program year, 21.7 percent of the 102 participants who received DC Flex funds used at least one CoC service. Without the DC Flex program, Urban estimates that 50.3 percent of participants would have used at least one service. This difference—a CACE coefficient of -28.6

percentage points—is statistically significant.

EXHIBIT 17

Support Service Use by Compliers and Counterfactual Compliers Rate of Use of Services

Complier outcome

21.7%

Counterfactual complier outcome

50.3%

CACE coefficient (SE)

-28.6 percentage points (9.9pp)

Source: Homeless Management Information Systems data.

Notes: The compliers are the 102 families who were randomized into the DC Flex group, enrolled in the program, and received assistance. The counterfactual compliers are the 127 families that were conditionally offered a DC Flex slot but did not enroll.

CACE = Complier Average Causal Effect. SE = standard error.

While the CACE estimate is a statistically significant result, it should be interpreted in the context of other options available to the control group. Specifically, when participants are offered DC Flex, they must give up services they are currently receiving, while the control group is not faced with that option. For example, Urban finds that 30 percent of the control group received rapid re-housing at some point during the first program year, compared with only 9 percent of the 102 participants who received DC Flex. Urban interprets much of the overall difference in use of support services as being the result of participants’ choosing DC Flex over other subsidies, like rapid re-housing. For this reason, Urban recommends thinking of DC Flex as being an alternative to other existing support services for some families. Longer-term analysis will determine whether it actually prevents the use of CoC

services or achieves similar outcomes at a lower cost.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

3 5

Economic Well-Being

Our evaluation also provides some insight into the economic well-being of DC Flex participants.

According to the participant survey, 59 percent of respondents reported working full time, 33 percent were working part time or seasonally, and 5 percent were unemployed (3 percent of responses were other/missing). Feedback from DC Flex administrators indicates a potential change in participants’

financial health. Over the course of the first year, administrators saw more participants acquire personal bank accounts, fewer participants access high-interest loans, and generally improved financial decisionmaking.

Focus group participants modeled a potential avenue for improving economic well-being through peer learning. In both focus groups, participants shared information about how to improve credit scores, find and access affordable housing, and pay bills. Such peer information sharing may be a way to add support to the DC Flex program without requiring case management.

The Lab @ DC also conducted an exploratory analysis of whether DC Flex had any effect on the receipt of cash benefits through DC’s TANF program and the amount of benefits received by participants. There was no statistical or economically meaningful effect of DC Flex on the rate at which participants used TANF or the monthly benefit amount in their first program year. These results will be expanded on in future reports.

As data become available, the evaluation of future years of the DC Flex pilot will include an analysis of how participation in DC Flex affects (1) the rate at which adults in families are employed and (2) their income from wages. To answer these questions, the evaluation will compare three outcomes for the treatment and control group:

◼

Quarterly wages from DC-based nonfederal employers.

◼

The share of people who earned wages from DC-based, nonfederal employers at any point during the year of observation.

◼

The share of people who earned wages from DC-based, nonfederal employers in two consecutive quarters and four consecutive quarters, as a measure of stable earnings.

Limitations

Urban’s qualitative analysis is limited by nonrespondents to the participant survey. Although the survey had a respectable response rate of 61 percent, whether there was a response bias that may 3 6

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

have affected the results is unknown—for example, whether respondents felt more positively, or negatively, about the program than nonrespondents. Another limitation is a lack of data on the comparison group, which prevents comparisons about decision making, financial health, and geographic mobility between the two groups. Access to information on how the comparison group makes decisions on housing expenditures would provide valuable context for the psychological effect of the DC Flex program on participants. Additionally, participants who dropped out of the program were largely unresponsive to requests for interviews. Understanding the decisionmaking process for households that were selected but ultimately declined to participate in the DC Flex program would have informed how families were thinking about the requirements, subsidy amounts, and structure of the program. Households that did not recertify at the end of the first program year are also a gap in the data collection. Program administrators indicated that they were largely nonresponsive, so administrators were unable to speak with them to gain context for what caused them not to recertify.

Resources precluded qualitative data collection with landlords of tenants in the program. While the original conception of the program assumed landlords would be unaware of their tenants’

participation, interviews made it clear that many participants had already informed their landlords.

Identifying how landlords reacted to the subsidy, whether it made them more or less likely to take on a new tenant, and if they perceived changes in behavior related to the subsidy would have added useful context to the evaluation.

The most pressing limitations on The Lab @ DC’s quantitative results for DC Flex on homelessness, housing stability, and use of public benefits are compliance and attrition. As discussed in the Program Enrollment and Participant Characteristics section and detailed in the pre-analysis plan, roughly one-half of the applicants selected in the lottery did not enroll in the program, many because they were ultimately ineligible once they reached the final verification stage. This ineligibility poses analytical challenges. Urban does not know which households in the control group would have been ineligible. Some households selected in the lottery are simply “non-compliers”—in that they either did not respond or decided not to enroll—meaning they did not comply with the random lottery. The quantitative analyses in this report treat ineligibility and non-compliance the same. This might lead to less precise ITT and CACE estimates. The Lab @ DC will pursue strategies that differentiate between ineligibility and non-compliance and may lead to stronger estimates of the causal effect of DC Flex on each outcome.20 Despite these limitations, the results provide a reasonable picture of the impact of 20 See discussion of the Average Treatment on the Treated estimate in the Pre-Analysis Plan (page 26).

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

3 7

DC Flex if the pilot were expanded in future years, because a scaled up version of the program would only serve those that were eligible and interested in enrolling.

Regarding attrition, the primary risk of attrition is members of the treatment or control group moving out of the District of Columbia, because the outcome measures rely on administrative data maintained by the DC government. For example, if DC Flex were to increase housing stability, a family in the control group may decide that it is easier to afford housing in the Maryland or Virginia suburbs and move out of DC without the $7,200 in annual assistance. If the family established Maryland residency, they would not be captured in the DC homeless service administrative data or TANF

records. If DC Flex did not help participants achieve or maintain housing stability relative to other programs, households may be forced to leave homes in DC they can no longer afford. Attrition may introduce bias into the quantitative estimates if families with lower risk of homelessness and greater economic opportunity are more or less likely to move out of DC. If the control group families that are less likely to experience homelessness leave DC (as compared to similar treatment group families), DC

Flex would likely appear to have a greater impact than it would in the absence of attrition. Conversely, if the control group families that are more likely to experience homelessness leave DC, DC Flex would likely appear to have a lower impact than it would in the absence of attrition.

Conclusion

Urban’s qualitative evaluation demonstrates that the District was successful in getting the program set up, reaching full enrollment, and effectively implementing the DC Flex account system. This is no small feat given the program’s unique structure and the challenges it posed for regulators and program administrators. Program participants reported high levels of satisfaction with the program: 80 percent of respondents said they were very satisfied with the program, 14 percent were satisfied, and just 1

percent reported being dissatisfied. When asked what they liked best about the program, most participants cited either the ability to receive assistance for up to 4 years (27 percent) or the flexibility to decide how much assistance to use each month (27 percent). When asked what they would change about the program, the most common recommendations were to extend the program beyond 4 years (22 percent) and to increase the amount of assistance offered (22 percent). The program’s biggest challenge in its first year was deciding which families would most benefit from the program, given the shallowness of the subsidy and the lack of services.

Our evaluation could not determine whether the subsidy was more effective than traditional rental subsidy programs in helping participants better respond to financial shocks. However, the 3 8

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

flexibility was perceived as beneficial among participants and more empowering than the rapid rehousing model. Continuing the evaluation would be useful in understanding whether these patterns change in future years as participants hopefully become more financially stable.

Overall, the quantitative results indicate that DC Flex is a viable housing support option for eligible applicants who choose to take up the program. Those applicants are at almost an identical—

but low—risk of homelessness after 1 year, compared with households in the business-as-usual options. One year is a short time to observe effects on homelessness, and over subsequent years and with more data, it may become apparent that residents who choose DC Flex are meaningfully better or worse off than the status quo. If the risk of homelessness remains similar, then the relative costs of DC Flex, compared with the other homelessness prevention and rental support services, will be vital to understanding its usefulness as a model.

In future years, Urban intends to examine additional housing stability and economic security indicators, including evictions, applications to other rental assistance programs, and employment outcomes. The comparative costs of achieving the outcomes associated with DC Flex relative to the outcomes for other housing assistance programs used by the control group will also be examined, and the comparison will be an important factor in assessing the cost-effectiveness of the program.

D C F L E X I B L E R E N T S U B S I D Y P R O G R AM

3 9

Appendix A. DC Flex Participant

Survey Responses

The Qualtrics online survey was open from February 8 to March 5, 2019. Responses from 11

participants who completed a paper version in October 2018 are included.21 Urban sent the survey to 121 participants, defined as heads of household that had a DC Flex account set up, even if they had left the program. Urban received 74 completed surveys, for a response rate of 61 percent.

When did you attend your DC Flex orientation?

●

March-April 2018

14%

●

May-July 2018

50%

●

August-October 2018

36%

When did you first use DC Flex to help pay your rent?

●

April-June 2018

35%

●

July-September 2018

30%

●

October-December 2018

23%

●

January-February 2019

4%

●

Missing*

18%

* This question was not part of the original paper survey.

What is your current employment status?

●

Working full-time

59%

●

Part-time

30%

●

Unemployed

5%

●

Seasonal

3%

●

Other/Missing

3%

How satisfied are you with the program overall?

●

Very satisfied

80%

●

Satisfied

14%

●

Neutral

5%

●

Dissatisfied

1%

21 The 11 respondents filled out a pre-survey that was used to estimate survey completion time and to evaluate question clarity.

4 0

A P P E N D I X

[if neutral or dissatisfied] What parts of the program are you not satisfied with? [open-ended responses]

●

The yearly cap on drawing down funds (x3)

●

The amount of funds

●

The maximum income limits

Have you ever tried to check on how much money is available in your DC Flex checking account?

●

Yes

58%

●

No

40%

Have you ever had any difficulty getting that information?

●

Yes

5%

●

No

95%

Have you ever tried to check on how much money is left of the full $7,200 you have to use for the year for DC Flex?

●

Yes

42%

●

No

58%

Have you ever had any difficulty getting that information?

●

Yes

3%

●

No

97%

Have you had any difficulty accessing funds in your DC Flex account?

●

Yes

9%

●

No

91%

Have you had any issues sending your rent payment verification?

●

Yes

3%

●

No

97%

When you first entered the program, did you owe back rent to your landlord?

●

Yes

23%

●

No

77%

[If yes] Did you receive any assistance to help pay off back rent before entering the DC Flex program?

●

Yes, received assistance from another program

18%

●

No, but landlord forgave back rent

6%

●

No

76%

A P P E N D I X

4 1

Does your landlord or management company know that you are in the DC Flex program?

●

Yes

45%

●

No

39%

●

Not sure

16%

How many of the financial planning 1 on 1s with Capital Area Asset Builders have you had?

●

None

20%

●

One

55%

●

Two or more

25%

[If one or more] Have the financial coaching sessions been helpful?

●

Yes

95%

●

No

5%

Including your DC Flex Account, did you have enough money to pay your rent last month?

●

Yes

76%

●

No

18%

●

Missing

6%

Which of these statements best describes how you have used your DC Flex account?

●

I use the DC Flex account to pay all or most of my rent each month, until I have exhausted my account

43%

●

I don't follow a consistent pattern. It changes from month to month based on my income and expenses

28%

●

I use a mixture of the DC Flex account and other income sources each month.

15%

●

I mostly or fully pay my rent with other income sources and only use the DC Flex account as a last resort

9%

●

Missing

4%

How likely is it that you will use up all your DC Flex account before your next disbursement of $7,200?

●

Very likely

41%

●

Somewhat likely

28%

●

Somewhat unlikely

9%

●

Very unlikely

9%

●

Not sure

8%

●

Missing

4%

4 2

A P P E N D I X

[if very or somewhat likely] Do you have a plan for paying the rent after you have used up all your DC

Flex account?

●

Yes

80%

●

No

4%

●

Not sure

16%

How satisfied are you with your current house or apartment?

●

Very satisfied

19%

●

Satisfied

16%

●

Neutral

36%

●

Dissatisfied

12%

●

Very dissatisfied

12%

●

Missing

5%

How satisfied are you with your neighborhood?

●

Very satisfied

19%

●

Satisfied

19%

●

Neutral

31%

●

Dissatisfied

16%

●

Very dissatisfied

9%

●

Missing

5%

Does being in the DC Flex program make it more or less likely you will move?

●

More likely

38%

●

Less likely

18%

●

No difference

39%

●

Missing

5%

What do you like best about the DC Flex program?

●

That I can stay in the program for up to four years

27%

●

The flexibility to decide how much assistance to use each month 27%

●

The financial coaching

15%

●

That any money I don’t use can be rolled over into the next year 11%

●

That I can keep whatever money I have left over in my account at the end of the program

5%

●

The amount of rental assistance it offers

5%

●

Other

5%

●

Missing

5%

A P P E N D I X

4 3

If you could change one thing about the program what would it be?

●

Extend the program so that it lasts more than four years

22%

●

Increase the amount of assistance it offers

22%

●

Allow participants to use their accounts outside of DC

16%

●

Provide more help with finding an apartment

16%

●

Allow participants to use their accounts for non-housing expenses 3%

●

Provide more case management services

1%

●

Other*

12%

●

Missing

8%

* More than one person wrote in recommending increasing the maximum allowable household income. Several wrote in that there was nothing they would change.

4 4

A P P E N D I X

Appendix B. Focus Group Interview

Protocol

My name is [] and this is my colleague, []. We work for the Urban Institute, a nonprofit research organization in Washington, DC.

A focus group is a way to speak with a group of people at the same time about their perceptions, opinions, beliefs, and attitudes towards something. We are conducting this focus group to learn about your experiences with the DC Flex Pilot Program.

Before we begin, there are several important things for you to understand:

◼

Everyone who works on this study has signed a confidentiality pledge requiring us not to share your identity with anyone outside of the research staff or to identify you with anything you tell us. Although we may include what you tell us in research reports, we will combine what you say with what we hear from other participants. We will not share your individual names in any of the research reports we release. We will also not share your individual responses with CAAB or DHS or anyone else involved in the program.

◼

We want to emphasize, however, that we cannot offer you complete confidentiality because we cannot control what participants in today’s group say or do with the information you share outside of this meeting. In addition, we cannot offer total anonymity because other focus group participants may know you outside of this meeting. So, please remember that what you say has the possibility of being shared with someone outside of this room.

◼

Your participation in this discussion is entirely voluntary. If you don’t want to participate or don’t want to answer a particular question, you don’t have to. There will be no negative repercussions if you decide not to participate or decline to answer any questions you don’t want to answer. You will receive $30 as a thank you for taking the time to attend this focus group. If you decide to participate but later decide you do not want to continue, you may leave at any time and still receive the $30.

◼

We will be audio recording today’s discussion to make note-taking easier and ensure we don’t miss anything you say, but we will keep the recording and its transcription in a locked drawer and on a secure data server. Neither the notes nor the recording will be shared outside of the A P P E N D I X

4 5

Urban Institute. The notes and recordings that are taken during the discussion will be destroyed at the end of the project.

◼

When we begin the focus group shortly, please introduce yourselves using just your first names. Please do not mention the full names of others in the room (if you know them) so that the recordings do not contain identifying information. If you wish to share something privately, we will stick around for a few minutes after the focus group has concluded. The focus group will last no more than 2 hours.

Do you have any questions about the focus group or anything I have just said? [Pause for reply.]

[Hand out consent forms]

Some of the information I just provide is on this form. We will ask you to sign this form to confirm that you heard this information and agree to participate in this discussion. Let’s review what is on the form.

“I understand:

Everyone who works on this study has signed a Pledge of Confidentiality requiring them not to tel anyone outside the research staff anything I tel them during the focus group. However, I should remember that things I say in the discussion could be shared by other focus group participants.

The focus group will be audio recorded but the audio will only be used to back up the notes that are taken during the discussion and will subsequently be destroyed at the end of the project.

My participation is voluntary. I only have to answer questions I want to, and I can choose to stop participating at any time. I understand that my participation will not affect any service I may or will receive from partners or others.

I consent to participate in this focus group. I understand that my participation is voluntary and I understand that I can stop participating at any time or refuse to answer questions at any time.”

If you agree with what I just read and want to participate in this discussion, please check the first box at the bottom of your form. If you also agree to the focus group being recorded, please check the second box. If anyone has changed their mind and no longer wants to participate in today’s discussion, please let my co-worker know now.

[Pause, then facilitate signing forms]

Does anyone have any questions before we start?

OK, let’s begin.

4 6

A P P E N D I X

Here are a few guidelines to help us have a good discussion that includes everyone:

◼

This is an informal conversation about what you think—there are no right or wrong answers.

◼

Please respect what everyone in the room has to say.

◼

Please try not to use any names when telling a story or describing something you observed.

1. Introductions

2. Administration

a. Going around the room, can you tell us:

i. How you originally heard about the DC Flex Program?

b. What were your initial thoughts?

i. Probes:

1. What appealed to you about the program?

2. What concerns did you have?

c. What did you think of the application process and the orientation meeting? Looking back, are there things you learned about the program later on that you wish you had known at the beginning?

3. Structure

a. How do you feel about the flexibility of the program funds, meaning the ability to decide how much assistance to use each month, up to the full cost of your rent, until you hit the $7,200 cap?

i. Probes:

1. Is $7,200 for the year enough, too much or too little? Were you able to comfortably pay your rent each month without sacrificing other expenses or getting behind on rent?

2. Would you want to be able to use the funds for non-housing expenses?

3. Would you rather you received the full amount at once? Or received a set amount each month?

b. Did you think the financial planning sessions were useful?

i. Probes:

1. Did you make different financial decisions based on what you learned?

c. Are there other services you wish were a part of the program?

i. Probes:

1. Case management, housing search and move-in assistance, employment services

4. Participant experience

a. How did you plan to use your program funds? Were you able to stick with your plan?

i. Probes:

1. Were there any unexpected changes to your income or expenses that you had to deal with?

A P P E N D I X

4 7

2. If so, how useful was the program in helping you adjust to those changes?

b. How many of you told your landlord that you were in the program? How many did not?

i. Why did you decide to tell or not tell your landlord?

ii. If you told your landlord, do you think it affected how he or she treated you?

c. How many of you spent down the full $7,200 in less than 12 months?

i. Did you expect this to happen or was it a surprise?

ii. How did you manage paying rent after your assistance ended?

d. Has being in the DC Flex program affected where you live? Has it made you more or less likely to move, or the types of neighborhoods or homes you would consider moving to?

i. Probes:

1. Did anyone move to a new place since joining DC Flex? What were some of the challenges you faced in finding a new place?

2. What prompted your move (e.g., desire to save money or move into a nicer apartment or neighborhood)? Do you like your housing? Size, rooms, condition? If you moved, how does your new place compare to the old one?

e. Has being in the program had any effect on how you budget and pay your bills?

i. Probes

1. Opening a bank account, online bill pay, building or repairing credit 5. Future expectations

a. Do you plan to continue in the program for the full four years?

i. Probes:

1. [if not] What are some of the reasons you might leave the program?

b. After the 4 year run of the program, do you expect to be able to pay your rent without the subsidy? Why or why not?

c. What are some of the goals you hope to accomplish while in the program?

d. If there was one thing you could change or add to the program, what would it be?

e. To wrap up, do you have any final thoughts you want to be sure are conveyed about the DC Flex Pilot Program?

4 8

A P P E N D I X

Appendix C. Inclusion Criteria for

Households in the Year 1 Analysis

The first 102 spots in the DC Flex program were filled through five lotteries. After each participant selected in those five lotteries was offered DC Flex, a contingency list was created using the same lottery process. This allowed DHS to fill slots when participants exited the program but preserve the same lottery procedure. Participants were not notified of their status on the contingency list; they only became aware once DHS contacted them to offer enrollment. The table below outlines how participants in each lottery and the contingency lists are treated in the quantitative analysis.

EXHIBIT C.1

Sample Derivation for the Quantitative Analysis

All possible assignment

Are they included in this report

categories

Count

(the first program year analysis)?

Assignment

Not offered DC Flex at any time

420

Yes

Control

by 10/1/2019

Offered in lottery 1–5; received

102

Yes

Treatment

funds by 10/1/2018

complier

Offered in lottery 1–5; never

126

Yes

Treatment

received funds

noncomplier

Contingency list spot No. 1

1

Yes (We included the person in slot

Treatment

No. 1 on the contingency list because

noncomplier

the person in slot No. 2 received funds

by 10/1/2018.)

Offered in lottery 1–6; received

17

No; these are removed from main

None; treatment

funds after 10/1/2018 but before

analysis but used in sensitivity analysis. complier in

3/31/19

sensitivity analysis

only

Contingency list

9

No; these are removed from main

None; treatment

analysis but used in sensitivity analysis. noncomplier in sensitivity analysis

only

Contingency list

25

No; these are removed from the

Removed from

analytical sample entirely because they analytical sample were offered DC Flex with 1 to 6

but will be

months remaining in the program year.

included in future

The Lab @ DC deemed this too short a

years’ analyses.

period to observe a treatment effect.

Contingency list

19

Yes; these families were not offered

Control

funds by 10/1/2019.

A P P E N D I X

4 9

Appendix D. Blocking Variables,

Balance Tables, and Regression

Analyses for All Enrolled

Participants

To do random assignment, we used the blockTools (Moore and Schnakenberg, 2016) package in R on the following variables:

◼

Days since application at the time of the lottery.

◼

Binary indicator of ZIP code of applicant.

◼

Head of household age in years.

◼

Household size.

◼

Number of dependent minors.

◼

Binary indicator of current employment (if not employed at time of application, days since last employment at time of application).

◼

Annual income.

◼

Binary indicators for the type of prior temporary or emergency rental assistance program received.

◼

Binary indicators for current temporary or emergency rental assistance program participation.

◼

Current rental payment amount.

◼

Binary indicator of whether the applicant splits rent with another adult.

As noted in appendix C, there are 26 participants (17 compilers and 9 noncompliers) who first received DC Flex funds between October 2, 2018, and March 31, 2019. These participants were excluded from the main analyses because they did not experience a full year in DC Flex.

The table below shows the balance statistics for when those 26 participants are included in the study sample.

5 0

A P P E N D I X

EXHIBIT D.1

Balancing Table for All Enrolled Participants

P-value

Offered DC

difference in

Standardized

Household information

Flex

Control

means**

differences***

Age

32.0 (7.8)

31.4 (7.8)

0.327

0.078

Number in household

3.3 (1.3)

3.3 (1.3)

0.984

0.002

Number of children

2.1 (1.1)

2.1 (1.2)

0.866

0.014

Currently working

78.9% (0.4)

78.4% (0.4)

0.878

0.012

Days since last reported date of

employment (if not currently

working)*

69.4 (95.2)

97.9 (71.9)

0.031

0.338

Annual income

$16,984 (9,468) $17,782 (9,723)

0.299

0.083

Rent amount

$855 (443)

$932 (450)

0.030

0.173

Split rent for housing unit with

someone not in their household

8.1% (0.3)

8.0% (0.3)

0.942

0.006

Source: DC Flex Application System self-reported application data.

Notes: The “offered DC Flex” group is 255 families, and the control group is 439 families. Standard deviations are reported in parentheses.

* 68 individuals in the treatment group were not employed at the time of the application.

** <0.05 is statistically significant.

*** >0.25 is concerning.

To examine whether the exclusion of those 26 participants meaningfully affected the causal estimates in the report, The Lab @ DC included those 26 participants (n=255), ran the same ITT and CACE regressions, and compared those estimates against the original estimates. The inclusion resulted in no change in statistical significance, but a potentially meaningful change in the ITT and CACE point estimates for homelessness. Due to the shortened window of observation and low overall rates of homelessness, this change may result from random chance. If these differences persist, they will be captured in future years’ analyses. The same number of individuals in the control group are present in both sets of analyses, but small differences emerge due to how “pseudo” start dates are calculated for controls.

EXHIBIT D.2

Regression Analysis for All Enrolled Participants

Year 1 results

Homelessness (% pts)

Continuum of care (% pts)

TANF ($)

TANF (%)

ITT (SE)

1.8 (1.9)

-12.6 (4.0)

$167 (289)

1.2 (4.2)

CACE (SE)

3.9 (4.0)

-26.6 (8.5)

$352 (611)

2.6 (8.8)

Source: Homeless management information systems data.

Notes: ITT = intent-to-treat. CACE = Complier Average Causal Effect. TANF = Temporary Assistance for Needy Families. TANF

amounts represent annual value.

*** p-value <0.01

** p-value <0.05

* p-value < 0.1

A P P E N D I X

5 1

Appendix E. Baseline Outcomes

The following tables illustrate the balance of the treatment and control groups 1 year before randomization and illustrate the groups’ similarity in homeless management information systems and TANF outcomes during the 2017 calendar year, before anyone applied for or received DC Flex.

EXHIBIT E.1

Balance Table: Homeless Management Information System Records from 2017 (Pre-Treatment) P-value

Offered

difference

Standardized

DC Flex

Control

in means

differences

Use of Continuum of Care or support services

55.0%

59.5%

0.265

0.090

Entered emergency shelter or transitional housing

8.7%

10.3%

0.498

0.052

Source: Homeless Management Information Systems data.

Note: The “offered DC Flex” group was 229 families; the control group was 439 families.

EXHIBIT E.2

Balance Table: Temporary Assistance for Needy Families Receipt from 2017 (Pre-Treatment) P-value difference

Standardized

Offered DC Flex

Control

in means

differences

Dollar amount per month

$121

$142

0.161

0.116

% of recipients per month

48.7%

49.8%

0.786

0.022

Source: Department of Human Services administrative data.

Note: The “offered DC Flex” group was 229 families; the control group was 439 families.

5 2

A P P E N D I X

Appendix F. Inverse Probability

Weights

DC Flex applications were submitted between December 2017 and July 2018. Five lotteries were run between January 2018 and July 2018, followed by the rolling enrollment of participants into the program. This means that applicants were eligible for different lotteries, depending on when they applied. An applicant’s “randomization wave” refers to the date and number of the first lottery of which the applicant was a part. If someone was not picked for the program during their randomization wave, they were part of the lotteries that followed. This means that households that applied earlier had a higher probability of being selected, as shown in the table below.

EXHIBIT F.1

Probability of Being Randomized into the Treatment Group by Lottery Draw

# of people selected and

Lottery

included in year 1 study

Lottery dates

Treatment probability (%)

1

48

1/22/2018

38.51

2

54

2/14/2018

32.12

3

61

4/20/2018

28.98

4

36

6/18/2018

13.34

5 and contingency list

30

7/25/2018

6.40

When study participants have unequal probabilities of treatment, Urban risks bias in the treatment effect estimate caused by the different composition of treatment and control group participants. In the case of this study, (1) households that enter the lottery later might have unobserved characteristics that are correlated with worse outcomes, and (2) people who enter into the lottery later, by virtue of lottery design, have a lower probability of being treated.

The analyses used inverse probability weighting to adjust for the waves’ different probabilities of being treated Pr(T). This is important because if the results are estimated without proper reweighting for Pr(T), the estimates of the treatment effect are biased upwards because the control group (having a lower Pr(T) in later waves) is composed of people with unobserved characteristics that are correlated with worse outcomes. Weighting helps adjust for treatment effect bias caused by unobserved confounders correlated with the applicant’s randomization wave (Robins, 1998).

A P P E N D I X

5 3

Appendix G. Details of Pre-

Registered Analyses Not Included in

This Report, Non-Pre-Registered

Updates Included in This Report,

and Potential Analyses to Be

Pursued in Future Reports

Before the evaluation of the DC Flexible Rent Subsidy Program, The Lab @ DC published a pre-analysis plan that outlines the methods and analyses it intended to undertake for the evaluation.22 The goal of the pre-analysis plans is to promote scientific research integrity by reducing researcher discretion after experimental outcomes have been realized. While analyses were already underway, however, the research team identified some methodological changes that were necessary to improve the quality of the analyses and found these changes to be sufficiently different that an addendum was added to the pre-analysis plan. Below, the methods that were included in the pre-analysis plan and what methods will be explored in future analyses are summarized.

EXHIBIT G.1

Deviations or Additions to Pre-Registered Analyses

Method detailed

Method elaborated/updated

Analysis

in pre-analysis plan

in report

Reason

Unit of

The pre-analysis plan

All outcomes studied are

Recruitment forms only

observation

incorrectly said some

exclusively at the participant

included information on the

outcomes would be

level. No outcomes capture

applicant and not on other

measured at the participant

services household members

household members;

level and some at the family

other than the household head

therefore, administrative

level (appendix III).

received.

data could not be linked to

them.

22 The pre-analysis plan for the quantitative analysis is available on The Lab’s Open Science Framework page for DC Flex: https://osf.io/r47hb/.

5 4

A P P E N D I X

Inverse

Calculate a difference in

Adjust the control and

The pre-analysis plan did not

probability

means between those

treatment means using inverse

directly mention weighting

weighting

assigned treatment and

probability weights, based on

outcomes to reflect different

control for each outcome

the probability of a person

probabilities of treatment,

(page 20). Weights were not

being in the sample.

which are created by having

specified.

a rolling application process

and multiple lotteries. This

weighting is necessary to

account for observed and

unobserved differences

between groups that were

eligible for different

lotteries.

Use of

Defined in appendix III as

A binary variable equal to 1 if

A binary variable was used

services

“count of interactions with

anyone interacted with the

to align the measure with

D.C. Continuum of Care.”

Continuum of Care.

the homelessness outcome

measure, which is also

binary.

Housing

Appendix III listed all primary Before the COVID-19

It was not possible to obtain

stability and

and secondary outcomes.

pandemic, The Lab @ DC

administrative data

economic

intended to include more

(employment, income, court

well-being

diverse measures of housing

interactions, outcomes and

outcome

stability and economic wel -

proceedings) in the timelines

variables

being, including evictions and

required for the study. While

eviction proceedings, the rate

this report only includes

of application for the District’s

outcomes measured in the

Emergency Rental Assistance

HMIS and TANF databases.

Program, the rate of

The Lab @ DC intends to

application for emergency

include these additional

assistance with utilities

outcomes in future public

through the Low Income Home reports as data become

Energy Assistance Program,

available.

employment, and earnings.

Bayesian

Specified (page 22)

Not reported

Posterior probabilities were

posterior

not reported because of time

probabilities

constraints but are likely to

be included in future

analyses.

Treatment

Section G of the pre-analysis No ATT analysis was

Data on employment and

on the

plan outlines an approach to

conducted for this report.

income were viewed as vital

Treated

account for the fact that

to this analysis. Because

(ATT)

Urban does not know which

those data were not

estimate

members of the control

available (see above), this

group would not meet final

estimate was not calculated

eligibility requirements.

but will be in future analyses.

A P P E N D I X

5 5

EXHIBIT G.2

Potential Analytical Improvements for Future Reports

Method detailed

Method added/updated in

Analysis

in pre-analysis plan

future reports

Reason

Time units

Not specified

The time unit within

It will not be accurate to

participants will be months

analyze the entire post-

participation period

monolithically because of

phase-in in the control

group. Urban needs the

assignment indicator to

change for control group

participants who are

randomized into treatment

in the contingency list.

ANCOVA

Regression analyses in the

Chance imbalance is

The analyses reported

pre-analysis plan do not

common in RCTs and

control for imbalances in

include covariates.

something Urban has

the baseline outcome

encountered in the study.

between the treatment and

control group by controlling

in the regression for

baseline outcomes.

Treatment on

Specified, but not completed

May be updated to

Changes to the time units

the Treated

(see above).

accommodate other

or regression inputs may

(ATT) estimate

methods outlined here.

necessitate updating the

method.

Cost

Outlined as

Similar to pre-analysis plan;

The results in this report

comparison

“secondary/exploratory

insufficient data available

demonstrate the need for

analyses.”

for this report.

relative costs to be included

for the outcomes,

particularly the use of CoC

services.

Changing the

The definition of support

It might be important to

By construction, the CoC

definition of

services encompasses housing

update the definition to

variable in this analysis is

support

and nonhousing services

exclude housing services

favorable to the

services

(“wrap around services”).

that are alternatives to DC

intervention because

Flex, like rapid re-housing,

people who use DC Flex

and to include only services are no longer eligible to

that people are not forced

receive other housing

to give up by virtue of

services.

receiving DC Flex.

Date to define

The pre-analysis plan states:

In further work, Urban will

If there is a clear situation

post-

“Outcomes for treatment

investigate the time elapsed of “first come, first served,”

intervention

group members would have

across randomization

Urban might want to

period for the

been measured from their

waves between the date of

truncate the dates allowed

control group

program enrollment date (and

application, the date of

for noncompliers and

not their randomization date).

enrollment, and the date of

controls to fall in between

No corresponding date exists

receipt of funding.

the minimum and maximum

for the control group or for

dates for their

noncompliers. Our stated goal

corresponding

was to ensure that the

randomization wave group.

distribution of starting points

for outcome variables is

5 6

A P P E N D I X

proportionate for the

treatment and control group,

as best as possible. We

anticipated that we would start

dates to members of the

control group based on the

start dates of program

recipients in their same

blocking group.”

A P P E N D I X

5 7

References

Apgar Jr., William C. 1990. “Which Housing Policy Is Best?” Housing Policy Debate 1 (1): 1–32.

Chetty, Raj, Nathaniel Hendren, and Lawrence F. Katz. 2016. “The Effects of Exposure to Better Neighborhoods on Children: New Evidence from the Moving to Opportunity Experiment,” American Economic Review 106 (4): 855–902.

Cunningham, Mary, Josh Leopold, and Pamela Lee. 2014. "A Proposed Demonstration of a Flat Rental Subsidy for Very Low Income Households." Washington, DC: Urban Institute.

Cunningham, Mary, Sarah Gillespie, and Jacqueline Anderson. 2015. "Rapid Re-housing: what the research says."

Washington, DC: Urban Institute.

Dasinger, Lisa K., and Richard Speiglman. 2007. “Homelessness Prevention: The Effect of a Shallow Rent Subsidy Program on Housing Outcomes among People with HIV or AIDS,” AIDS and Behavior 11 (2): 128–39.

Gubits, Daniel, Marybeth Shinn, Stephen Bel , Michelle Wood, Samuel Dastrup, Claudia D. Solari, Scott R. Brown, et al. 2015. Family Options Study: Short-Term Impacts of Housing and Services Interventions for Homeless Families. Washington, DC: U.S. Department of Housing and Urban Development, Office of Policy Development and Research.

Gubits, Daniel, Marybeth Shinn, Michel e Wood, Stephen Bel , Samuel Dastrup, Claudia D. Solari, Scott R. Brown, et al. 2016. Family Options Study: 3-Year Impacts of Housing and Services Interventions for Homeless Families. Washington, DC: U.S. Department of Housing and Urban Development, Office of Policy Development and Research.

Hannagan, Anthony, and Jonathan Murdoch. 2015. “Income Gains and Month-to-Month Income Volatility: Household Evidence from the US Financial Diaries.” NYU Wagner Research Paper 2659883. New York: NYU

Wagner.

Jacob, Brian A., and Jens Ludwig. 2012. "The effects of housing assistance on labor supply: Evidence from a voucher lottery," American Economic Review 102.1: 272–304.

Moore, Ryan T., and Keith Schnakenberg. 2016. Package “blockTools. ” https://cran.r-

project.org/web/packages/blockTools/blockTools.pdf

Mullainathan, Sendhil, and Eldar Shafir. 2013. Scarcity: Why Having Too Little Means So Much. New York: Picador.

Poethig, Erika C. 2014. “One in Four: America’s Housing Assistance Lottery,” Urban Wire (blog). Washington, DC: Urban Institute. May 28.

“Public Housing: The Experimental Housing Allowance Program.” 1981. Urban Law Annual 21 (1).

https://openscholarship.wustl.edu/law_urbanlaw/vol21/iss1/16

Robins, James M. 1998. “Marginal Structural Models.” In 1997 Proceedings of the American Statistical Association, Section on Bayesian Statistical Science, 1–10. Alexandria, VA: American Statistical Association.

Ross, Lauren M., and Danilo Pelletiere. 2014. “Chile's New Rental Housing Subsidy and Its Relevance to US

Housing Choice Voucher Program Reform,” Cityscape 16 (2): 179–92.

Salvi del Pero, Angelica. 2016. “Housing Policy in Chile: A Case Study on Two Housing Programmes for Low-Income Households.” OECD Social, Employment and Migration Working Paper 173. Paris, FRA: Organisation for Economic Co-operation and Development.

Susin, Scott. 2002. “Rent Vouchers and the Price of Low-Income Housing,” Journal of Public Economics 83 (1): 109–52.

5 8

R E F E R E N C E S

Struyk, Raymond J., and Marc Bendick, eds. 1981. Housing Vouchers for the Poor: Lessons from a National Experiment. Washington, DC: Urban Institute Press.

Wood, Michel e, Jennifer Turnham, and Gregory Mills. 2008. “Housing Affordability and Family Well‐Being: Results from the Housing Voucher Evaluation,” Housing Policy Debate 19 (2): 367412.

R E F E R E N C E S

5 9

U.S. Department of Housing and Urban Development Office of Policy Development and Research

Washington, DC 20410-6000

August 2021

Document Outline

DC Flexible Rent Subsidy Program

Acknowledgments

Foreword

Executive Summary

DC Flexible Rent Subsidy Program

Background

Evaluation Approach and Research Questions

Qualitative Approach

Quantitative Approach

Program Design and Implementation

Implementation Timeline

Eligibility Requirements

Program Enrollment and Participant Characteristics

Program Orientation

Flexible Subsidy Accounts

Participant Services

Recertification

Outcomes

Use of Flex Accounts

Homelessness and Housing Stability

Housing Mobility

Homelessness and Use of Continuum of Care Services

Economic Well-Being

Limitations

Conclusion

Appendix A. DC Flex Participant Survey Responses

Appendix B. Focus Group Interview Protocol

Appendix C. Inclusion Criteria for Households in the Year 1 Analysis

Appendix D. Blocking Variables, Balance Tables, and Regression Analyses for All Enrolled Participants

Appendix E. Baseline Outcomes

Appendix F. Inverse Probability Weights

Appendix G. Details of Pre-Registered Analyses Not Included in This Report, Non-Pre-Registered Updates Included in This Report, and Potential Analyses to Be Pursued in Future Reports

References

Table of Contents

DC Flexible Rent Subsidy Program

Acknowledgments

Foreword

Executive Summary

DC Flexible Rent Subsidy Program

Background

Evaluation Approach and Research Questions

Qualitative Approach

Quantitative Approach

Program Design and Implementation

Implementation Timeline

Eligibility Requirements

Program Enrollment and Participant Characteristics

Program Orientation

Flexible Subsidy Accounts

Participant Services

Recertification

Outcomes

Use of Flex Accounts

Homelessness and Housing Stability

Housing Mobility

Homelessness and Use of Continuum of Care Services

Economic Well-Being

Limitations

Conclusion

Appendix A. DC Flex Participant Survey Responses

Appendix B. Focus Group Interview Protocol

Appendix C. Inclusion Criteria for Households in the Year 1 Analysis

Appendix D. Blocking Variables, Balance Tables, and Regression Analyses for All Enrolled Participants

Appendix E. Baseline Outcomes

Appendix F. Inverse Probability Weights

Appendix G. Details of Pre-Registered Analyses Not Included in This Report, Non-Pre-Registered Updates Included in This Report, and Potential Analyses to Be Pursued in Future Reports

References