HUD Jobs Plus

Outcomes

Evaluation

Long-term Effects from the

Original Jobs Plus Demonstration:

Employment and Earnings for Public

Housing Residents after 20 Years

U.S. Department of Housing and Urban Development | Office of Policy Development and Research

DISCLAIMER

The contents of this report are the views of the contractor and do not necessarily reflect the views or policies of the U.S. Department of Housing and Urban Development or the U.S.

Government.

HUD Jobs Plus Outcomes Evaluation

Long-term Effects from the Original Jobs Plus

Demonstration: Employment and Earnings for Public

Housing Residents after 20 Years

Cynthia Miller Nina Castells Nandita Verma Joshua Vermette Henry Kanengiser MDRC

September 24, 2021

Acknowledgments

We are grateful to the Office of Policy Development and Research (PD&R) in the U.S.

Department of Housing and Urban Development (HUD) for sponsoring a long-term impact analysis of the families who participated in the original Jobs Plus demonstration. These types of studies are rare, and this evaluation has allowed us to build long-term evidence on a program that showed initial promising results. Nathan Bossie served as our project officer and has provided valuable and thoughtful guidance on all aspects of the evaluation, including this report. We are also grateful to other staff at HUD, including Jayme Brown, Isaac Dozier, Leigh van Rij, Anne Fletcher, Carol Star, and Mark Shroder, who reviewed drafts of this report or participated in report briefings or webinars.

This report draws on the original Jobs Plus demonstration data and recent updates from HUD

and Office of Child Support Enforcement (OCSE). At MDRC, Johanna Walter, data manager for the original demonstration, dug into project archives and helped us assemble the file needed for the longer-term evaluation. At HUD, Lynn Rodgers served as the main contact for acquiring Inventory Management System/PIH Information Center (IMS/PIC) and National Directory of New Hires (NDNH) data.

Our Expert Panel advisors, Larry Katz of Harvard University and Kathy O’Regan of New York University, joined the final briefing on this report and provided important external reviews. We are grateful for their time and contributions.

At MDRC, our colleagues James Riccio and Howard Bloom served as senior advisors and reviewers and provided critical guidance on the methods used in the impact analysis and the interpretation of the analysis and findings. Riccio and Bloom played lead roles on the original Jobs Plus demonstration and brought a wealth or historical context to the current evaluation. We are grateful to Howard Bloom for helping us think through particular analytic issues and for developing a supporting appendix for this report. We also recognize Shawna Anderson’s diligent support on managing the project budget.

Foreword

The original Jobs-Plus demonstration program operated from 1998 to 2003 at public housing developments in areas of concentrated underemployment and joblessness. The overall goal of the demonstration was to take a comprehensive approach to increase economic empowerment and mobility for public housing residents. The residents at participating sites benefitted from employment-related services at onsite job centers, rent-based work incentives that allowed residents to keep more of their earnings, and activities to promote neighbor-to-neighbor exchanges that support work. The program targeted all working-age residents without disabilities at select public housing developments in six cities: Baltimore, Maryland; Chattanooga, Tennessee; Dayton, Ohio; Los Angeles, California; St. Paul, Minnesota; and Seattle, Washington.

A 2005 analysis by MDRC found that the program produced substantial earnings gains for residents in three of the six sites during the first 4 years of the program. These three sites had fully implemented and sustained the program. A followup study in 2010 found that the initial effects persisted, with residents in the three sites earning 16 percent more on average than those in comparison sites. HUD sponsored this long-term impact analysis to understand the long-term economic mobility and labor market effects of a fully implemented place-based employment initiative for adults and to explore intergenerational effects of economic well-being.

This report examines whether the earlier reported successes were sustained 15 years after the program ended and whether gains in residents’ earnings translated into improvements in their children’s employment and earnings. This report finds that participants at the three sites that fully implemented the original Jobs Plus model continued to have positive effects on adult residents’ earnings and employment between 2017 and 2019, two decades after the program launched. Moreover, exploratory analysis found positive effects for residents who were children at the time of the demonstration and are now of working age. By contrast, participants at sites that did not fully implement the Jobs Plus model experienced long-term negative effects on adults’ income and employment. These findings demonstrate the potential benefits of place-based interventions in areas of concentrated underemployment and joblessness. They also suggest, however, that for these interventions to have these long-term benefits they must be well implemented.

This report contributes to the growing literature on self-sufficiency program effectiveness and the value of investments in place-based interventions for intergenerational gains, particularly for residents who are most impacted by systemic disinvestment. HUD is committed to continuing to support the self-sufficiency of assisted families.

Solomon J. Greene

Principal Deputy Assistant Secretary for Policy Development and Research U.S. Department of Housing and Urban Development

ii

Table of Contents

Executive Summary .................................................................................................................................... vii

I.

Introduction ...................................................................................................................................... 1

II.

Long-Term Impacts Analysis: Research Questions, Study Sample, Data Sources,

and Analytic Approach ......................................................................................................... 8

III.

Long-Term Effects for Adults ....................................................................................................... 22

IV.

Long-Term Effects on Labor Market Outcomes of Children Living in Jobs Plus Households ..... 35

V.

Discussion of Findings ................................................................................................................... 44

VI.

Conclusion ..................................................................................................................................... 48

Appendix A: Analytic Model and Specifications ..................................................................................... A-1

Appendix B: Summary of Program Implementation for the Sites in the Jobs Plus Demonstration ......... B-1

Appendix C: Supplemental Impact Exhibits ............................................................................................. C-1

Appendix D: Estimating the Gain in Lifetime Earnings for Children of Adult Participants .................... D-1

Appendix E: Using Permutation Tests to Assess the Statistical Significance of Long-Term Jobs Plus

Impact Estimates .............................................................................................................. E-1

VII.

References .................................................................................................................................... 105

List of Exhibits

Exhibit ES- 1. Impacts on Average Annual Earnings and Average Quarterly Employment Rates

in Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the

Long-Term Impact Study and Children in Households from the 1998 Cohort,

Stronger Implementation Sites ............................................................................................ ix

Exhibit 1. Housing Developments in the Original Jobs Plus Demonstration ............................................... 4

Exhibit 2. Summary of Key Distinctions in the Approach Toward the Individual-Level Analysis in the

Original and Longer-Term Impact Analyses ........................................................................ 8

Exhibit 3. Key Research Questions for the Long-Term Impact Analysis ................................................... 10

Exhibit 4. Simplified Illustration of the Hypothesized Long-Term Effects of Jobs-Plus .......................... 11

Exhibit 5. Samples in the Original Jobs Plus Demonstration Evaluation and the HUD Long-Term

Impact Analysis .................................................................................................................. 14

iii

Exhibit 6. Selected Baseline Characteristics of Adults from the 1998 Cohort Included in the Long-Term

Impact Analysis .................................................................................................................. 16

Exhibit 7. Housing Subsidy Receipt Among Adults and Children in Year 21 of Followup for Jobs Plus

and Comparison Developments Combined by Site ............................................................ 17

Exhibit 8. Timeline of the Original Jobs Plus Demonstration and Followup Analysis .............................. 18

Exhibit 9. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the

1998 Cohort Who Are in the Long-Term Impact Study, All Sites ..................................... 23

Exhibit 10. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the

1998 Cohort Who Are in the Long-Term Impact Study, by Program Implementation

Strength .............................................................................................................................. 24

Exhibit 11. Mean Quarterly Earnings for the Work-Able Adults in the 1998 Cohort and the Long-Term

Impact Sample: Stronger Implementation Sites ................................................................. 26

Exhibit 12. Effect on Average Annual Earnings Distribution, Years 20 and 21 of Followup, Adults

from the 1998 Cohort Who Are in the Long-Term Impact Study, Stronger Implementation

Sites .................................................................................................................................... 28

Exhibit 13. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the

1998 Cohort Who Are in the Long-Term Impact Study, by Site, Stronger Implementation

Sites Summary Outcomes ................................................................................................... 29

Exhibit 14. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the

1998 Cohort Who Are in the Long-Term Impact Study, by Site, Weaker Implementation

Sites Summary Outcomes ................................................................................................... 31

Exhibit 15. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the

1998 Cohort Who Are in the Long-Term Impact Study, by Age at Baseline, Stronger

Implementation Sites Summary Outcomes ........................................................................ 33

Exhibit 16. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the

1998 Cohort Who Are in the Long-Term Impact Study, by Age at Baseline, Weaker

Implementation Sites Summary Outcomes ........................................................................ 34

Exhibit 17. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Children in

Households from the 1998 Cohort, Stronger Implementation Sites ................................... 36

Exhibit 18. Effects on Average Annual Earnings Distribution, Years 20 and 21 of Followup, Children

in Households from the 1998 Cohort, Stronger Implementation Sites .............................. 39

Exhibit 19. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Children in

Households from the 1998 Cohort, by Site, Stronger Implementer Sites Summary

Outcomes ............................................................................................................................ 41

Exhibit 20. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Children in

Households from the 1998 Cohort, by Age at Baseline, Stronger Implementation Sites

Summary Outcomes ........................................................................................................... 43

iv

Exhibit 21. Summary of Impacts on Earnings and Employment for Adults and Children in the

Jobs Plus Demonstration: Original Sample and Followup Study Samples ........................ 46

Exhibit B.1. Operational Highlights for Stronger Implementation Sites .................................................. B-2

Exhibit B.2. Operational Highlights for Weaker Implementation Sites ................................................... B-3

Exhibit B.3. The Original Jobs Plus Demonstration Implementation Timeline ....................................... B-4

Exhibit C.1. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of

Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study,

All Sites ........................................................................................................................... C-2

Exhibit C.2. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of

Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by

Program Implementation Strength.................................................................................... C-3

Exhibit C.3. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup,

Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Site, Stronger

Implementation Sites ........................................................................................................ C-4

Exhibit C.4. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of

Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study,

by Site, Stronger Implementation Sites ............................................................................ C-6

Exhibit C.5. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults

from the 1998 Cohort Who Are in the Long-Term Impact Study, by Site, Weaker

Implementation Sites ........................................................................................................ C-8

Exhibit C.6. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of

Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by

Site, Weaker Implementation Sites ................................................................................ C-10

Exhibit C.7. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults

from the 1998 Cohort Who Are in the Long-Term Impact Study, by Age at Baseline,

Stronger Implementation Sites ....................................................................................... C-12

Exhibit C.8. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of

Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study,

by Age at Baseline, Stronger Implementer Sites ............................................................ C-14

Exhibit C.9. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults

from the 1998 Cohort Who Are in the Long-Term Impact Study, by Age at Baseline,

Weaker Implementation Sites ......................................................................................... C-16

Exhibit C.10. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of

Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study,

by Age at Baseline, Weaker Implementer Sites ............................................................. C-18

Exhibit C.11. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup, Children

in Households from the 1998 Cohort, All Sites .............................................................. C-20

v

Exhibit C.12. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of

Followup, Children in Households from the 1998 Cohort, All Sites ............................. C-21

Exhibit C.13. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Children in

Households from the 1998 Cohort, by Implementation Strength .................................. C-22

Exhibit C.14. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of

Followup, Children in Households from the 1998 Cohort, by Program Implementation

Strength .......................................................................................................................... C-24

Exhibit C.15. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup, Children

in Households from the 1998 Cohort, by Site, Stronger Implementer Sites ................ C-25

Exhibit C.16. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of

Followup, Children in Households from the 1998 Cohort, by Site, Stronger Implementer

Sites ................................................................................................................................ C-27

Exhibit C.17. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup, Children

in Households from the 1998 Cohort, by Site, Weaker Implementer Sites ................... C-29

Exhibit C.18. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of

Followup, Children in Households from the 1998 Cohort, by Site, Weaker

Implementation Sites ...................................................................................................... C-31

Exhibit C.19. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup,

Children in Households from the 1998 Cohort, by Age at Baseline, Stronger

Implementation Sites ...................................................................................................... C-33

Exhibit C.20. Impacts Employment Stability and Earnings Distribution in Years 20 and 21 of Followup,

Children in Households from the 1998 Cohort, by Age at Baseline, Stronger

Implementation Sites ...................................................................................................... C-35

Exhibit C.21. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup,

Children in Households from the 1998 Cohort, by Age at Baseline, Stronger

Implementation Sites ...................................................................................................... C-37

Exhibit C.22. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup,

Children in Households from the 1998 Cohort, by Age at Baseline, Weaker

Implementation Sites ...................................................................................................... C-38

Exhibit C.23. Impacts Employment Stability and Earnings Distribution in Years 20 and 21 of

Followup, Children in Households from the 1998 Cohort, by Age at Baseline, Weaker

Implementer Sites ........................................................................................................... C-40

Exhibit E.1. Sampling Distribution and Permutation P-value for the Estimated Long-term Jobs Plus

Impact on Adult Earnings in Strong-Implementation Sites .............................................. E-4

Exhibit E.2. Estimates of Long-term Jobs Plus Impacts and their Alternative P-Values ......................... E-5

vi

Executive Summary

Public housing residents in the U.S. face many challenges in improving their economic situations: many developments are located in areas of concentrated poverty, and residents often struggle with poor work histories, limited education, lack of adequate childcare, health or medical problems, and concern about crime and safety. In the mid-1990s, the U.S. Department of Housing and Urban Development (HUD), the Rockefeller Foundation, and MDRC created the Jobs Plus demonstration to address these challenges. The Jobs Plus program takes a comprehensive approach to helping public housing residents improve their employment situations. It combines three key components—on-site employment-related services and activities, rent-based financial incentives, and community support for work—and leverages the place-based nature of the program to aim for “saturation,” targeting all residents in the development and intentionally providing opportunities for community building and social supports around residents’ economic self-sufficiency goals.

This demonstration, which operated from 1998 to 2003, was the subject of a rigorous evaluation which found that three of the six developments that were able to implement and sustain implementation of all three components of the model saw the program boost annual earnings. 1 Since the program was first implemented, and given its demonstrated positive and enduring effects on residents’ earnings, Jobs Plus has been expanded to at least 50 public housing agencies (PHAs) across the country through local replication efforts and more recent scale-up through HUD. 2 This report examines whether the significant gains observed in residents’ earnings due to the implementation of Jobs Plus—which did not show signs of fading during the full four years of implementation or the three years following the completion of the program—were sustained 15 years after the end of the intervention. 3 This report also explores whether the program results in long-term improvements in children’s employment and earnings, thus testing the possibility that a well-implemented place-based employment intervention can produce positive effects for adults exposed to the program and also spur intergenerational effects and improve the economic well-being of children growing up in these developments.

Data and Methods

This report examines the long-term effects of the Jobs Plus demonstration on labor market outcomes for two study samples: (1) the nonelderly, nondisabled adults living in the Jobs Plus and comparison study developments at the time that implementation began in 1998 and who had not reached retirement age (age 65) by the end of the long-term analysis study period, and (2) individuals who were children (ages 0 to 17) living in the Jobs Plus and comparison developments at the time of program launch. The long-term 1 Riccio (2010).

2 Since 2015, HUD has awarded approximately $108 million in four-year grants to 43 PHAs to implement Jobs Plus.

See Verma et al. (2019) for a description of some key departures in the design of the Jobs Plus model that is being scaled-up as part of the HUD replication effort.

3 Part of this effort includes an impact evaluation of the first 24 housing authorities implementing Jobs Plus as part of HUD’s national scale up of this program.

vii

effects for both samples are examined 20 to 21 years after program launch (14 to 15 years after program implementation ended), during the time period of July 2017 to June 2019. The analysis uses employment and earnings outcomes created with the National Directory of New Hires (NDNH) data for this time period. The NDNH data contain quarterly wage data for workers in employment covered by the unemployment insurance (UI) system.

Average annual earnings in years 20 and 21 and average quarterly employment in years 20 and 21 are the two confirmatory outcomes for this study. The confirmatory outcomes analysis is focused only on the strong implementation sites and for the adult sample. All remaining analyses of long-term effects, including those for children, are considered exploratory.

The study uses Ordinary Least Squares (OLS) to examine the long-term effects of Jobs Plus on labor market outcomes. This methodology leverages the very close match in baseline earnings and employment levels and trends in the 6 years leading up to Jobs Plus implementation to compare the labor market outcomes for the Jobs Plus program group with the outcomes of the comparison group about 15

years after the program ended in 2003, and it includes covariates to further increase the precision of the estimates and control for differences between the Jobs Plus and comparison groups at baseline. The methodology differs from that of the early impact analysis of the Jobs Plus demonstration, which used comparative interrupted time series, but that was not feasible for the long-term analysis because the 6

years of pre-Jobs Plus employment and earnings data could not be used to reasonably predict employment and earnings levels and trends 20 to 21 years later.

One limitation of this analytic method is that it does not account for the fact that the intervention was delivered at the development level (rather than to randomly selected individuals within developments) or for factors at the development level that will affect outcomes for all residents in that development. For this reason, this model most likely overestimates the precision and statistical significance of estimated impacts. Additional analyses are conducted to provide some assessment of how much the model overestimates statistical significance, and the findings are discussed in the report.

Key Findings

This long-term impact analysis aims to answer two main research questions. The primary research question asks: Were the Job-Plus-induced gains in residents’ earnings found for the strong implementation sites sustained 15 years after the end of the intervention? The secondary research question is: Did the Jobs Plus-induced gains in residents’ earnings from the original Jobs Plus demonstration translate into long-term improvements in their children’s employment and earnings?

The following are the key findings from the analysis:

• Jobs Plus continued to positively impact earnings of work-able residents in the stronger implementation sites 15 years after the program ended.

viii

The top panel of exhibit ES-1 presents the estimated effects of the Jobs Plus demonstration on the subset of nonelderly, nondisabled adults living in the Jobs Plus developments (and who had not yet reached retirement age by the end of the followup period) on employment rates and average earnings 20–21 years after the program was launched in 1998, for sites that fully implemented the Jobs Plus program during the demonstration. These sites include Dayton, Ohio, Los Angeles, California, and St. Paul, Minnesota.

The evaluation of the original demonstration found positive effects on average annual earnings for work-able Jobs Plus residents for the second 2 years of program implementation (after the 2-year rollout period), and these earnings gains were sustained for 3 years after the Jobs Plus program ended in those sites. Across these 5 years, the Jobs Plus program led to an average increase in annual earnings of about $1,300 per year in 2003 dollars, which is equivalent to about $1,800 in 2019 dollars.

Exhibit ES- 1. Impacts on Average Annual Earnings and Average Quarterly Employment Rates in Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study and Children in Households from the 1998 Cohort, Stronger Implementation Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Adults

Average Annual Earnings, Years 20 and 21 ($)

16,860

15,190

1,670 *

0.056

Average Quarterly Employment, Years 20 and 21 (%)

54.3

50.1

4.2 **

0.047

Sample size (total = 4,105)

914

1,073

Children

Total Annual Earnings, Average of Years 20 and 21 ($)

20,961

18,255

2,706 ***

0.000

Average Quarterly Employment, Years 20 and 21 (%)

72.8

66.6

6.2 ***

0.000

Sample size (total = 6,337)

1,622

1,700

Notes: The adult sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency. The child sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Stronger implementation sites include Dayton, Ohio, Los Angeles, California, and St. Paul, Minnesota.

P-values are based on standard errors that are adjusted to account for individuals from the same household using Huber-White standard errors.

The impact estimates for the two confirmatory outcomes (average annual earnings over Years 20 and 21 and average quarterly employment across Years 20 and 21) were each statistically significant, therefore, based on the Benjamini-Hochberg multiple hypothesis testing approach, no further adjustments to the p-values were needed.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires.

ix

The present study found that these earnings gains were sustained 15 years after the program ended, and there was little evidence that the magnitude of the effect faded over time. In this long-term followup period that covered 14 and 15 years after the program ended, the estimated effect on average annual earnings was $1,670 in 2018 dollars, or an 11-percent increase. There is evidence that these long-term earnings gains can be at least partly attributed to increased employment rates: the Jobs Plus group had an average quarterly employment rate of 54.3 percent, which is 4.2 percentage points higher than the comparison group during the same time period. This estimated effect was also statistically significant.

As noted, the analysis model used to estimate program impacts likely overestimates their statistical significance. Additional analyses suggest that the p-value of the effect on earnings is closer to 0.249 rather than 0.056, as shown in exhibit ES-1. The p-value for the effect on employment is likely closer to 0.166, rather than 0.047. Thus, the estimates should be considered with this limitation in mind and can be seen as somewhat suggestive of long-term impact. One factor that may suggest evidence of true impact, however, is that the effects observed here, in Years 20 and 21, are a continuation of a pattern found in the short-term.

• The children living in the Jobs Plus developments at the time that the program was implemented also experienced higher earnings and employment in adulthood, compared with their comparison group counterparts.

The bottom panel of exhibit ES-1 shows the estimated effects of the Jobs Plus demonstration on the average earnings and the employment rates of the children (any child under 18 years old) who were living in the three Jobs Plus developments with stronger implementation at the time the program started.

As discussed above, earlier analyses found positive effects on the average earnings of the work-able adults living in those households.

The results of the present long-term analysis show that living in one of these three Jobs Plus developments with stronger implementation led to increased employment and earnings 14 to 15 years after the program ended, when the children were between ages 18 and 38.

Exhibit ES-1 shows that children living in stronger implementation sites had average annual earnings that were $2,706 higher than children living in comparison developments that did not implement Jobs Plus. Children living in these three Jobs Plus developments earned, on average, $20,961 across Years 20 and 21, compared with earnings of $18,255 for children in the comparison group. Children living in Jobs Plus developments with stronger implementation also had higher annual employment rates than their counterparts living in comparison developments. As shown in exhibit ES-1, average quarterly employment rates were 72.8 percent in followup years 20 and 21 in the program group, compared with 66.6 percent in the comparison group, for an estimated effect of 6.2 percentage points. These estimated effects on average earnings and employment are statistically significant.

A similar analysis was conducted to assess the extent to which the analytic method overestimates the precision of the impact estimates for children. The findings suggested that the x

effects were somewhat less precisely estimated but still statistically significant at the 10 percent level.

Conclusion

The results described in this paper provide important evidence on the potential long-term effects of a comprehensive place-based employment program for public housing residents. Jobs Plus was designed to address the challenges of concentrated poverty and the barriers to economic mobility faced by residents of urban public housing developments. Although varied self-sufficiency programs for housing subsidy recipients have been implemented to address these types of challenges and some have been evaluated, the present study is one of the rare studies to follow program participants for many—in this case, 21—years to test whether program effects are sustained over the very long term, and whether they have intergenerational effects on their children.

Findings from the earlier analyses of the effects of Jobs Plus examined as part of the original demonstration and the followup analysis conducted 3 years after the implementation ended demonstrated that Jobs Plus, where it was implemented well, had sustained positive effects on the earnings of work-able adults. The findings from the present long-term study provide suggestive evidence that these effects were sustained over a very long time period: through 21 years post-program launch, or 15 years after implementation ended. Furthermore, it provides evidence that the program also led to long-term positive effects on the employment and earnings during adulthood for children living in the Jobs Plus developments at the time of implementation.

The main findings of the present study are based on three sites with strong implementation, which is a small number of sites for a test of a place-based program where the intervention occurs at the site level and not the individual level. The extended followup period exposes the long-term employment and earnings outcomes to further influences that may have affected one research group and not the other or affected the two groups disproportionately. Despite the very close match in baseline levels and trends in average earnings and employment rates and the random assignment of developments within sites, the study methodology (OLS) would not account for potential differences in the long-term trajectories of employment and earnings between the program and comparison group that might have existed even in the absence of Jobs Plus. Therefore, these findings should be interpreted with some caution.

While the findings for children are exploratory, the implications—if the findings are replicated—are important. While much attention has been paid to the benefits of children leaving public housing in high poverty areas for lower poverty areas in the Moving to Opportunity (MTO) study, the findings from the present study of the long-term effects of Jobs Plus suggest that children can also benefit meaningfully from public investments in improving the environments in which they already live. These findings also emphasize the importance of the robust implementation of the Jobs Plus program, since they provide evidence that when well-implemented, it can serve as a platform for producing lasting economic gains for adults and children.

xi

I. Introduction

What are the long-term economic mobility effects of a successful, place-based multicomponent employment initiative offered to public housing residents? Does it lead to long-term labor market effects for adults? Does it spur intergenerational effects and improve the economic well-being of children growing up in these developments? In the mid-1990s, the U.S. Department of Housing and Urban Development (HUD), the Rockefeller Foundation, and MDRC conceived the Jobs Plus demonstration to address the significant challenges to self-sufficiency that residents of public housing developments faced. 4 This demonstration, which operated from 1998 to 2003, was the subject of a rigorous evaluation that found that three of the six developments that fully implemented the model saw the program boost annual earnings. 5 Since the program was first implemented, and given its demonstrated positive and enduring effects on residents’ earnings, Jobs Plus has been expanded to 50 public housing agencies (PHAs) across the country through local replication efforts and more recent scale-up through HUD. 6 This report focuses on new evidence on the long-term impacts for families exposed to the original Jobs Plus demonstration.

To understand whether Job Plus is an effective economic mobility program in the long run, HUD has sponsored an evaluation of the Jobs Plus effort, including a special focus on the original Jobs Plus demonstration (hereafter referred to as the long-term impact analysis), to examine whether the significant gains observed in residents’ earnings due to the implementation of Jobs Plus—and which did not show signs of fading during in initial follow-up period—were sustained 15 years after the end of the intervention. 7 A related question of interest for this special focus on the original Jobs Plus demonstration is whether the program also results in long-term improvements in children’s employment and earnings, thus testing the possibility that a place-based employment intervention can produce positive effects for adults exposed to the program and also serve as a platform for enhancing the economic mobility of children in adulthood.

This report begins by providing some background on the original Jobs Plus demonstration: the model, the sites involved in the effort, features of the sites that successfully implemented this model and produced positive results, and the early evidence about its effectiveness. Following that, the report describes the data and methods for the long-term impact analysis for adults and children and highlights important distinctions between the original analysis and the current one. Findings for 4 The original model was named Jobs-Plus (with a hyphen), but the current program replicated by HUD is referred to as Jobs Plus (without the hyphen). This report uses the current treatment for all iterations of the program.

5As discussed later in this report, three of the six sites were identified as having implemented and sustained implementation of all three components of the Jobs Plus model for the duration of the demonstration: employment services, rent incentives, and community support for work. Also see Bloom, Riccio, and Verma (2005).

6 Since 2015, HUD has awarded approximately $108 million in four-year grants to 43 PHAs to implement Jobs Plus.

See Verma et al. (2019) for a description of some key departures in the design of the Jobs Plus model that is being scaled-up as part of the HUD replication effort.

7 Part of this effort includes an impact evaluation of the first 24 housing authorities implementing Jobs Plus as part of HUD’s national scale up of this program.

1

the adult and child samples are examined in separate sections, before wrapping up with discussion and conclusions.

The Jobs Plus Model8

This model’s theory of change rests on three components shown to be promising in prior research in welfare reform and other fields. It was hoped that combining them into a single model would be mutually reinforcing and make for a more robust intervention. 9 The three components include: on-site employment-related services, such as job search assistance, referrals to education and training programs, and support services coordinated by staff located within the development to make it easier to engage residents, and to make Jobs Plus staff more a part of the community they serve; rent-based financial incentives to help “make work pay” by reducing how much a family’s rent contributions increase when they enter work or increase their earnings, allowing them to see a bigger financial return from work; and, based on a growing recognition of the importance of social networks and social capital, the community support for work (CSW) component emphasizes resident-to-resident outreach, information-sharing, and mutual support, as well as connections to potentially instrumental external social networks.

Jobs Plus was also designed to operate at saturation levels—that is, to offer services, a rent incentive, and community support for work to everyone living in the development. Residents who formally enrolled in Jobs Plus and receive services or enroll in the rent incentive can influence other residents in the development without formal involvement in the program through strengthening and leveraging social networks among residents, sharing information about employment opportunities and availability of services in the community, and encouraging residents to attend Jobs Plus events that may benefit residents. In this way, it aspires to benefit residents beyond the personal benefit received through participating in employment and other support services and through the rent incentive.

Another unique feature of the original demonstration was its “mandatory collaborative,” an effort to promote governance, accountability, and support that was intended to craft, fund, and operate this comprehensive initiative. These collaboratives included local public housing agencies, resident representatives, and local public human services and workforce development agencies, which worked together and were accountable to one another. It was deemed that each partner had something special to offer, but each was also limited in what it could do alone. 10

8 This section draws heavily on Bloom, Riccio, and Verma (2005), which provides a fuller description of the demonstration and findings through 2003, when it ended; Riccio (2010) updates some of the main findings, covering another three years of follow-up.

9 See Bloom, Riccio, and Verma (2005); Blank and Wharton-Fields (2008); and Greenberg et al. (2015).

10 Kato and Riccio (2001).

2

The Jobs Plus Developments

Public housing developments in six cities were selected to implement Jobs Plus: Baltimore, Maryland; Chattanooga, Tennessee; Dayton, Ohio; Los Angeles, California; St. Paul, Minnesota; and Seattle, Washington. 11 Three developments (in Baltimore, Chattanooga, and Dayton) were populated almost exclusively by African American residents, who made up 94 percent or more of the Jobs Plus sample for those sites. 12 Residents of the other Jobs Plus developments included higher numbers of Hispanic and Asian/Pacific Islander households and substantial numbers of immigrants. The Jobs Plus developments also varied with respect to other demographic characteristics, such as the percentage of females (ranging from 65 percent to 91 percent of the sample) and households with two or more adults (ranging from 14 percent to 74 percent). The latter households were most highly represented in the sites with high proportions of immigrants (Los Angeles, St. Paul, and Seattle), which included many two-parent households.

Resident mobility was much higher than had been anticipated at the start of the demonstration. Nearly one-third (31 percent) of the residents in the 1998 cohort for all Jobs Plus developments combined moved out of their respective developments within 2 years of the program rollout (that is, before October 2000), and 42 percent had done so within 3 years (before October 2001). The move-out rates were highest in Baltimore, Chattanooga, and Dayton, cities with softer housing markets (especially Baltimore and Dayton) and where public housing appears to have served more as transitional housing for a large proportion of families. At the other three developments (Los Angeles, St. Paul, and Seattle), move-out rates were considerably lower. though hardly trivial, with 22 percent to 38 percent of residents leaving within 3 years. Residents in developments that experienced lower levels of mobility also had the opportunity to be exposed to a more mature and fully developed expression of the program.

Jobs Plus Implementation

Following a 2-year program rollout period, from 1998 to 2000, during which the participating agencies began planning and operating elements of the model, four of the six sites were determined to have developed programs of reasonable quality. Three sites, Dayton, Los Angeles, and St. Paul, were able to contend best with the operational challenges they confronted and fully implement the core components of the model. Baltimore and Chattanooga had a more difficult time getting and keeping all the Jobs Plus components in place. Finally, Seattle, which operated a strong program in the first few years of the demonstration, received a HOPE IV grant and had to contend with reconstruction and temporary dislocation of its residents at its Jobs Plus site. As a result, Jobs Plus ceased being a place-based intervention at this site, and Seattle was withdrawn from the national demonstration. However, because the site continued to operate a Jobs Plus program at the 11 Each development nominated had at least 250 units occupied by families with a working-age adult. No more than 30 percent of these families could have an employed member, and at least 40 percent had to be receiving welfare.

12 See Tables 2.1 and 2.2 in Bloom, Riccio, and Verma (2005).

3

development in modified form as the centerpiece of its HOPE VI community and supportive services plan (under the name “HOPE-Plus”), MDRC continued to evaluate the Seattle program. 13

Based on the implementation experiences of the sites, the original evaluation clustered the Jobs Plus developments into “stronger” and “weaker” implementation sites, summarized in exhibit 1

(see appendix exhibits B.1, B.2, and B.3 for additional highlights on each study site and its Jobs Plus implementation challenges and accomplishments). The analysis presented in this report continues to use the same classification for the study sites.

Exhibit 1. Housing Developments in the Original Jobs Plus Demonstration

Jobs Plus

Jobs Plus

Comparison

Implementation

City

Development

Development

Strength

Perkins Homes

Baltimore

Gilmor Homes

Weaker

Somerset Courts

College Hill Courts

Chattanooga

Harriet Tubman Homes

Weaker

Emma Wheeler Homes

Arlington Courts

Dayton

DeSoto Bass Courts

Stronger

Parkside Homes

Los Angeles

William Mead Homes

Dana Strand Village

Stronger

St. Paul

Mt. Airy Homes

Roosevelt Homes

Stronger

Seattle

Rainier Vista Garden

Community

Yesler Terrace

Weaker

Residents of the comparison sites were not necessarily devoid of the types of services and incentives available through Jobs Plus, but what was available to them was more limited. For instance, all the housing authorities offered residents of the comparison developments at least some self-sufficiency programs—on-site basic education, computer, or job readiness classes—other than Jobs Plus. Some housing authorities also offered a limited number of opportunities for apprenticeship training, such as in buildings maintenance and groundskeeping. The residents of the comparison developments also had access to rent-based work incentives that were available housing agency-wide. However, these incentives were less generous than those offered by Jobs Plus or were available only to residents who met certain conditions. But comparison developments were also much less 13 In Seattle, the housing authority received a federal HOPE VI grant in 1999, which was used to rebuild the Rainier Vista development where Jobs Plus was located. HOPE VI is a HUD program that is aimed at redeveloping the most

“severely distressed” housing projects across the country. The redevelopment process involves replacing public housing units with apartments or townhouses, some of which will become available at market rate to working families in an effort to reduce the concentration of poor households in the development communities.

4

aggressive in marketing and implementing their incentives to residents than were the developments implementing Job Plus. 14

Prior Evidence of Jobs Plus Effectiveness

MDRC’s evaluation of the original Jobs Plus demonstration examined the program’s impacts from two perspectives: an individual-level analysis and a development-level analysis, allowing the study to examine outcomes for individuals, whether or not they remain in public housing, and what happens to the earnings levels in the developments themselves as current residents leave and new ones move in. The individual-level analysis followed one cohort of residents (i.e., all the working-age, nondisabled residents living in the development in 1998), even after they moved out of the development. 15 This analysis showed that the program increased earnings for the six-site pooled sample by a small but statistically significant amount over the 4 years following the end of the rollout period, but the earnings effects were large for three of the six sites (Dayton, Los Angeles, and St.

Paul) that had fully implemented the essential elements of the model throughout the 4 years of post-rollout implementation. 16 These sites also had positive earnings effects across a range of subgroups.

Related to the program’s effects on employment, the rates were close, with no notable statistically significant differences for the Jobs Plus and the comparison developments. A subsequent analysis, focusing on additional years of followup, 4 years during program implementation and 3 years after the program ended, found that Jobs Plus households in these three sites continued to experience a gain in average annual earnings of $1,300, or a gain of 16 percent over the full 7-year followup period. 17

The “development-focused” analysis compared outcomes of the group of residents living in the Jobs Plus developments during each followup year with the group of residents living in the comparison group developments. This analysis differs from the individual-level analysis, mentioned above, which followed the 1998 cohort of residents over time; it focuses on residents in the development in each year of followup, regardless of when they moved into the development. The findings of this analysis showed that the stronger implementation sites produced positive effects on overall development-level earnings, but the magnitude of the effects varied according to each site’s resident turnover rates—which, in turn, were related to the tightness of the local private rental 14 Gardenhire (2004).

15 In this sense, this is not a voluntary sample enrolled in the study, as is the case in most employment-focused interventions: the “sample” was not “recruited” to be in the study. Furthermore, because many 1998 cohort members moved away during the first 2 years that Jobs Plus was being implemented, some were not exposed to the full program. To help account for this “dosage dilution,” the analysis was also repeated for a later cohort, including all working-age, nondisabled adults who were residents of a Jobs Plus development or comparison development in October 2000 (“the 2000 cohort”).

16 Bloom, Riccio, and Verma (2005).

17 The gain of $1,300 is in nominal dollars, or not adjusted for inflation.

5

housing market. In particular, the development-level impacts were lowest in Dayton, where tenant mobility (and access to affordable private rental housing) was highest. 18

Positive results from the original Jobs Plus demonstration prompted a community nonprofit that served the largest housing development in New York City (and the nation) to begin adopting elements of Jobs Plus, and subsequently, for the New York City government to implement an adapted version of Jobs Plus in multiple areas of the city. 19 While the New York City replication included more robust financial coaching, the Jobs Plus providers served much larger public housing developments, and the housing authority offered a less generous financial incentive to work. 20 The effort targeted housing developments operated by the New York City Housing Authority, starting with Jefferson Houses in 2009 and adding eight new sites between 2011 and 2014. An impact evaluation commissioned by New York City Mayor’s Office for Economic Opportunity found that Jobs Plus participants had significantly higher employment rates and average earnings in the year and a half following program entry, compared with other eligible residents in the same developments who had not (yet) enrolled. 21 Unlike the evaluation for the original demonstration, the New York City study did not have the benefit of a comparison group of individuals that were like Jobs Plus members but lacked access to the program (making it difficult to attribute effects of the program).

The analysis controlled for several demographic and employment characteristics, but it is uncertain how much of the estimated difference is attributable to selection bias (for example, at least hypothetically, residents who enrolled in Jobs Plus early or at all may have been more motivated to improve their employment situation than residents who enrolled much later or did not enroll at all) rather than program effects.

Neither of these studies examined the potential effects of these programs on children. For the original Jobs Plus evaluation, the younger children who were in households exposed to the program are now old enough to be engaged in the labor market. A baseline analysis of the children residing in the developments at the start of Jobs Plus examined whether the characteristics of their parents and the communities they lived in were associated with differences in their circumstances and wellbeing. 22 The findings showed that on some, but not all, measures of school and behavioral outcomes, 18 In Dayton, people with earnings moved out at a higher rate, and people with lower or no earnings replaced them, keeping the year-to-year averages at the development level lower. A development-level analysis was originally proposed for the HUD-funded long-term impact study but was dropped due to data limitations.

19 Building on the work by the East River Development Alliance (now Urban Upbound), a community organization in Queens, New York, to adapt Jobs Plus for Queensbridge Houses, the City of New York scaled up the replication of Jobs Plus.

20 See Leopold (2019). As described, the average Jobs Plus provider had more than 2,000 public housing units in its service area (for comparison, the original demonstration sites averaged about 560 units per housing development.

The New York housing agency also had less flexibility to alter the rent calculation of public housing residents to reduce financial disincentives to work.

21 Leopold et al. (2019)

22 See Morris and Jones (2002).

6

a substantial proportion of children living in public housing exhibited negative outcomes—and older children and boys were at greater risk than younger children and girls.

Beyond the above-described efforts to estimate program effects on employment and earnings, Jobs Plus has been the focus of in-depth implementation and process studies, documenting its scale-up, first as part of the Social Innovation Fund’s replication effort23 in the Bronx, New York, and San Antonio, Texas, and next as part of HUD’s national replication effort. HUD has also commissioned an impact study of its replication effort, focusing on the first 24 sites to implement the program. 24

The HUD-commissioned longer-term impact study for the original Jobs Plus participants offers an unusual opportunity to examine relatively long-term effects of an employment-focused program. Few studies, especially related to households receiving housing subsidies, have tracked employment outcomes for such extended periods of time; the Moving To Opportunity demonstration, a randomized controlled trial that offered families living in public housing vouchers to move to lower-poverty neighborhoods, is an exception, but helping families move to low-poverty neighborhoods is the main pathway out of poverty for that intervention. 25 Evaluations of welfare-to-work initiatives, mostly from the 1990s, along with more recent sector-focused employment programs, provide some context for assessing the lasting gains from interventions that offer employment and training services and financial assistance in the case of some. For the New Hope program, which offered services and financial incentives to low-income families in two inner-city neighborhoods in Milwaukee, Wisconsin, labor market effects were assessed over 8 years of followup. The results showed that New Hope’s effects on employment and income lasted for most families only during the 3 years in which the program operated; for a subgroup of individuals facing moderate barriers to work, New Hope increased employment, earnings, and income through Year 8. 26 A randomized controlled trial of Project Quest, a program in San Antonio, Texas, which offers to help low-income adults earn post-secondary credentials and access to jobs in specific sectors, found that program participants earned substantially more than control group members in the third through ninth years after random assignment—the program’s impact on annual earnings grew to $5,239 in year nine—a difference that was both statistically significant and the largest of any year. 27 Long-term impact analysis for three sites in the National Evaluation of Welfare-to-Work Strategies, a random assignment study testing the effects of alternative approaches to helping welfare recipients find jobs, showed that for approaches that were promising and effective in the short term (i.e., the first 5 years), 23 See Greenberg et al. (2015).

24 Program impacts are being examined by MDRC as part of a companion study.

25 The study finds that receiving housing vouchers to move from public housing in high-poverty neighborhoods to rental units in lower-poverty neighborhoods during childhood led to long-lasting influences on their well-being in adulthood (i.e., increased college attendance and earnings and reduced single parenthood rates). These effects were present for children who moved before turning 13 years old. See Chetty, Hendren, and Katz (2016).

26 See Miller et. al. (2008). From the start, and as hypothesized, New Hope was found to have larger effects on moderately disadvantaged individuals or those with only one barrier to employment. The large effects for this group stood out over time, persisting through Year 5 and growing larger by Year 8.

27 Roder and Elliot (2019).

7

their effects dissipated over time. 28 Whether the initial promising effects observed for the stronger implementation site in Jobs Plus are enduring—and whether they spur positive effects for children—

is an open question and the focus of the remainder of this report.

II. Long-Term Impacts Analysis: Research Questions, Study Sample, Data Sources, and Analytic Approach

The long-term impact analysis of the effects of the original Jobs Plus demonstration examines effects on labor market outcomes for both the work-able residents living in Jobs Plus developments at the time Jobs Plus was launched in 1998 (who have not yet reached retirement age by the end of the long-term follow-up period) and the children living in Jobs Plus developments at the time of the launch. The analyses use an Ordinary Least Squares (OLS) methodology, leveraging the very close match in baseline earnings and employment levels and trends in the 6 years leading up to Jobs Plus implementation to compare the labor market outcomes for the Jobs Plus program group with the outcomes of the comparison group about 15 years after the program ended in 2003. This section describes the study’s research questions, how the adult and child study samples were defined, the timeframe of the analysis, the data sources used for the analyses, and the analytic approach used to estimate long-term program impacts. Exhibit 2 offers a high-level summary of some of the key distinctions in the approaches used for the original individual-level analysis and the longer-term impact study, distinctions covered in the rest of this section.

Exhibit 2. Summary of Key Distinctions in the Approach Toward the Individual-Level Analysis in the Original and Longer-Term Impact Analyses

Feature

The Original

The HUD-Funded

Jobs Plus Evaluation

Long-Term Impact Analysis

Target population

Adult residents in the study

Adults and children in the study

developments

developments

Sample definition

All working-age, non-disabled

All working-age, non-disabled

residents, ages 21–62 in 1998

residents, ages 18–44 in 1998 (ages

38–64 during analysis period);

Children age 17 and under living in

the developments in 1998 (ages

20–38 during analysis period)

Analytic method

Comparative Interrupted Time

Ordinary Least Squares

Series

Data source for earning State Unemployment Insurance

National Directory of New Hires

and employment

data

Followup period

1998 to 2006

June 2017 to May 2019

28 Hamilton and Michalopoulos (2016).

8

Research Questions and Hypothesized Long-Term Effects

Two research questions are at the center of the long-term impact analysis for residents living in the original Jobs Plus sites in October 1998 (see exhibit 3). The first question focuses on the adult residents of the original Jobs Plus developments and seeks to understand whether the significant program-induced gains in residents’ earnings due to the implementation of Jobs Plus in the housing development—and which did not show signs of fading through 2006—were sustained 15 years after the intervention ended. The two confirmatory outcomes for the adult-focused analysis are 1) average

annual earnings over Years 20 and 21, and 2) average quarterly employment rates over Years 20

and 21. A 2-year average is a more accurate picture of earnings at the end of the followup period, given the variability in earnings from year to year. Confirmatory outcomes represent the key measures for assessing the effectiveness of an intervention and are typically based on a well-grounded theory of change. Given their primacy, impact findings on confirmatory outcomes are often subjected to further statistical adjustments to account for multiple hypothesis testing and reduce the risk of a false positive finding. The analysis for adults also includes exploratory outcomes, which are not the key outcomes of interest but can help to explain how a program had its effects, are more distal outcomes, or can uncover interesting patterns in the data. Included in this group are impacts for subgroups of interest, impacts by sites, impacts on earnings and employment in a given year, and impacts on measures of the persistence of employment. Impacts on exploratory outcomes are not typically subjected to statistical adjustments for multiple hypothesis testing.

The second question focuses on the potential intergenerational effects of this program and examines whether the gains in earnings of its participants also translated into long-term improvements in their children’s employment and earnings. All analysis of long-term effects for children, including effects for strong implementation sites, effects by individual sites, and effects for subgroups of interest, are considered exploratory, given that they are not the study’s key outcomes of interest.

The analysis for adults is conducted for the sample pooled across sites and by program implementation strength, for individual sites, and for age-specific subgroups within each sample.

However, the confirmatory outcomes analysis is focused only on the strong implementation sites.

The analysis of the long-term effects for children focuses on the group of sites with stronger implementation, where there was evidence of sustained effects for work-able adults, and effects are also examined for individual sites and for age-specific subgroups within that sample.

9

Exhibit 3. Key Research Questions for the Long-Term Impact Analysis Primary research question:

• Were the Job Plus-induced gains in residents’ earnings found for the strong implementation sites sustained 15 years after the end of the intervention?

Secondary research question:

• Did the Jobs Plus-induced gains in residents’ earnings found for the strong implementing sites translate into long-term improvements in their children’s employment and earnings?

Exhibit 4 offers a simplified illustration of the hypothesized pathways for the longer-term effects of Jobs Plus. While the program was operational, its multicomponent approach was fundamental to its vision for producing large impacts on employment and earnings: tackling residents’ obstacles to work through employment services, financial incentives, and social network strategies to enhance residents’ interest in and commitment to work, their capacity to look for and find work, their skills to qualify for better jobs, and their knowledge about job opportunities. 29 These changes, in turn, would increase their participation and success in the labor market. Sustained employment and earnings, if they occurred, could spur a range of family well-being outcomes.

Findings from the original evaluation showed that Jobs Plus’s positive effects on residents’ earnings were substantial and enduring, even after many residents moved out of public housing (and were no longer eligible for program services and incentives). Furthermore, 7-year trends in the program’s earnings impacts also indicate that Jobs Plus’s effects continued to be evident in the 3 years after the program ended (the duration of a follow-up analysis).

29 See Bloom et al. (2005) for a fuller depiction of the program’s underlying theory of change.

10

Exhibit 4. Simplified Illustration of the

Hypothesized Long-Term Effects of Jobs-Plus

Early and

Jobs

Impacts on

Impacts on

sustained

Plus

residents'

later work

impacts on

Offer

preparation

outcomes for

for work

parents’ work

children

outcomes

Mediators / Pathways*

Increases in parental income,

reduced economic stress,

improved parental emotional

well-being; residential mobility;

additional direct and indirect

effects of the program’s

community-wide activities and

supports.

*Long-term earnings for adults are the only mediators measured for the long-term impacts study.

Over time, as in the extended followup period captured by the long-term impact analysis, Jobs Plus may continue to affect participants earnings in a few ways. Individuals who were exposed to—and benefited from—the program might be more likely to continue working, move back into work more quickly if unemployed, and experience greater earnings growth by changing jobs or advancing on the job. Any training opportunities they took advantage of during the program period may also support their employment and earnings trajectories over the long term.

One factor to keep in mind when interpreting the impacts is that the environments in the developments (both Jobs Plus and comparison) may have changed over time as new policies or programs were developed. One example that was documented in the original evaluation report is that the Seattle Jobs Plus site underwent HOPE VI revitalization, meaning that large numbers of residents were relocated during the followup period. But other changes likely took place during the 15 years spanning the long-term followup, some of which are known and some of which are not known. For example, PHAs may have implemented new programs in comparison developments, designed to increase residents’ employment and earnings. When discussing the findings, the report documents any known changes in the developments that may have affected impacts over the long term. In a place-based study, however, where many people were still living in the developments for years after the program ended, it is important to consider that other factors or policies may have been occurring that worked to sustain or diminish Jobs Plus’s effects over time.

11

Related to the effects on children, child development theory points to two primary mechanisms through which increased parental income can improve children’s well-being. 30 The first is through reducing parental stress: with increased income, parents reduce their stress levels and improve their interactions and relationship with their children. 31 The second is through increased material investment: parents are able to provide more material goods that can improve children’s well-being, such as improved nutrition or higher quality childcare and education. 32 Several studies that analyzed the causal links between increased parental income and improved child well-being—

including rigorous quasi-experimental analyses leveraging random assignment designs and natural experiments—have consistently shown that increases in parental income lead to small but positive effects on children’s cognitive outcomes, behavioral outcomes, and academic achievement. 33

In the case of Jobs Plus, the effects on children would primarily occur through the effects of the program on the children’s parents or other adults in the household. In the stronger implementation sites, Jobs Plus had sustained impacts on the earnings of work-able adults living in the Jobs Plus developments. With earnings rising in both the program and comparison groups over the period of the demonstration, the rent incentive could have further contributed to increased disposable household income since households in Jobs Plus developments did not have to increase the amount of rent they paid when their income increased, as households in the comparison group did. There was no evidence that Jobs Plus increased employment rates across the three sites.

Therefore, it is reasonable to expect that a key pathway through which Jobs Plus might affect children’s later labor market outcomes in adulthood would be through increases in parents’ (and other adult household members’) earnings throughout and beyond the Jobs Plus program. It is possible, however, that while increased income from additional earnings and the rent incentive can improve children’s wellbeing, if those increased earnings are derived from working longer hours (which would not be captured by estimating effects on employment rates), reductions in time that parents spend with their children may have negative effects on their well-being. It is possible that increased household income was not the only pathway through which children could have benefited from Jobs Plus. As implied by the saturation nature of this initiative, children could have been affected in other ways, at least in theory, through various community-building experiences, mutual support among families, or through some employment services like childcare assistance. Finally, another factor that may affect children’s earnings as adults is residential mobility, either by the parents when the children were young or by the children as adults. Other research documents the positive effects of mobility to neighborhoods with more resources. 34

The findings described later in this paper will contribute to the body of evidence on the broader question of how program exposure affects child outcomes. As shown in exhibit 4, this study 30 For a fuller discussion of these pathways, see Gennetian et al. (2010).

31 McLloyd (1990).

32 Becker (1981); Coleman (1988).

33 Gennetian et al., (2010); Cooper and Stewart (2017).

34 See Chetty, Hendren, and Katz (2016).

12

does not measure the effects on any of the mediators that could lead to such long-term labor effects, such as other community-based inputs, parental stress and mental and emotional well-being, how parents invested their increased income, or any intermediate child outcomes such as academic or behavioral outcomes. Nevertheless, it provides important insight into one aspect of how a place-based employment program that has been shown to be effective in improving the earnings of adults in a housing subsidy program can also improve the long-term outcomes for children living in their households.

Subgroup Analyses

The study examines the long-term effects for two types of subgroups in both the adult and child analyses: (1) by site, and (2) by age at baseline. Effects are estimated separately for individual sites to understand if there was variation in long-term effects by site and to interpret any variation in effects in the context of the site-level patterns of effects found in the original demonstration. The 1998–2003 impact analysis had found large and positive effects on annual earnings of similar magnitude across all three stronger implementation sites. However, in that analysis, Dayton was the only site to show some evidence of effects on annual employment rates through the end of the program implementation period. Were the long-term effects also comparable across the three sites?

Did they translate into similar effects on children’s labor market outcomes?

Effects on employment and earnings were also examined separately for different baseline age groups and compared statistically across those groups. For the adult sample, the age categories were: (1) 18 to 24 years old; (2) 25 to 34 years old; and (3) 35 to 44 years old at the start of Jobs Plus implementation in 1998. The youngest age group was just entering the labor market at the time that Jobs Plus was first implemented, and some of those adults would have been participating in post-secondary education programs. The oldest age group includes adults nearing retirement age at the Years 20 and 21 follow-up period, and it is possible that the Jobs Plus program affects how long participants remain in the labor market. 35

For the child sample, effects were examined for the following age categories: (1) under age 5

(early childhood), (2) age 6 to 12 (middle childhood), and (3) age 13 to 17 (adolescence). Prior research shows that income increases can have larger benefits for younger children. 36 It is also important to keep in mind, however, that any differences in effects across different age groups may be driven not only by the age of the child at the time of the demonstration and the differential influence of increasing household income at different stages of child development, but also by the age of the child at the time of followup in Years 20 and 21 and when any influences of increased income during childhood might manifests themselves in adulthood. For example, any child who was under 5 years old at the time of the start of the demonstration will not yet have reached 25 years of 35 As shown in exhibit 5, this sample excludes individuals who were 45 to 62 in 1998 and were included in the impact analysis reported in Bloom et al. (2005) and Riccio (2010).

36 Morris et al. (2005); Vortruba-Drzal (2006).

13

age, a common threshold for studying effects on adult earnings, given post-secondary education, by the start of the long-term followup period.

Samples and Timeframe

The long-term employment and earnings outcomes are examined for two samples: an adult sample and a child sample. As shown in exhibit 5, the adult sample includes working-age, nondisabled members of the original cohort of residents living in program and comparison developments who were between the ages of 18 and 44 years in October 1998—in other words, they had not reached retirement age, or age 65, by the end of this study’s followup period in 2019. Exhibit 5 also shows how the sample for the long-term impact study overlaps with original study 1998 cohort.

Exhibit 5. Samples in the Original Jobs Plus Demonstration Evaluation and the HUD

Long-Term Impact Analysis

Age at Baseline (1998)

0–17

18–20

21–44

45–61

62+

Original Demonstration Sample

(Work-able, non-disabled)

Long-Term Impact Analysis Adult

Sample

(Work-able, non-disabled)

Long-Term Impact

Analysis Child

Sample

Although the adult sample in this report is not identical to the sample used in the original study, in aggregate there are very few differences in their baseline characteristics. Exhibit 6 shows baseline characteristics of the adult sample in this report. Across race, ethnicity, gender, and household composition, the adult sample in this report aligns closely with the 1998 cohort demographics. 37 The adult sample in this report does skew to a younger group than the 1998 cohort: the average age of adults in 1998 is around 30 years, and adults under the age of 34 make up almost 60 percent of the sample. This difference is to be expected given the focus of the analysis on assessing effects on residents 20 years after the implementation of Jobs Plus.

All children living in the Jobs Plus developments in 1998 are included in the child sample, regardless of whether the adults in their households were included in the long-term impact analysis 37 Baseline characteristics of the original study 1998 cohort can be found in Table 2.1 of Bloom et al. (2005).

14

sample. For example, if all the work-able adults in a household were excluded from the adult sample in the present long-term analysis because they were older than 44 years at baseline, any children in that household would still be included in the child sample of the present analysis. These children were between age 18 and 36 at the start of the long-term followup period. This report examines effects for children using the following age subgroups, based on age in 1998: early childhood (ages 0

to 5 years), middle to late childhood (ages 6 to 12 years), and adolescence (ages 13 to 17). In this report, there are 1,050 children who were under the age of 5 years in 1998, 1,566 in middle to late childhood (6 to 12 years), and 706 in adolescence (13 to 17 years).

Only a small proportion of individuals in both samples were still receiving some form of housing assistance at the end of the followup period covered in this analysis (Year 21). 38 Exhibit 7

shows housing subsidy receipt rates for the adult and child samples in each Jobs Plus site 21 years after the implementation of the program. Across the six sites, approximately 31 percent of adults in the long-term impact analysis sample were still receiving some type of housing assistance. 39 This rate ranged from 37.1 percent in Seattle to 24.1 percent in St. Paul. About 26 percent of the child sample were receiving some form of housing subsidy in Year 21, ranging from 32.4 percent in Chattanooga to 18.8 percent in St. Paul.

38 For those individuals no longer receiving some form of housing assistance in Year 21, it is unclear how long they continued to live in the development after Jobs Plus ended. Furthermore, as described, some of the developments in the study, both in Jobs Plus and comparison sites, experienced HOPE VI and RAD-related reconstruction and relocation, which could also have affected the subsidy receipt rates reported in exhibit 6. For these reasons, this analysis does not examine the effects of Jobs Plus on housing subsidy receipt.

39 This includes public housing, Section 8, project-based assistance, multifamily housing, and other types of housing subsidies.

15

Exhibit 6. Selected Baseline Characteristics of Adults from the 1998 Cohort Included in the Long-Term Impact Analysis All Developments

Outcome

Baltimore Chattanooga Dayton Los Angeles Seattle St. Paul

Combined

Race/ethnicity (%)

White (non-Hispanic)

0.3

3.8

6.2

0.8

7.0

6.4

4.0

Black (non-Hispanic)

99.0

94.1

91.5

8.8

34.8

24.5

64.6

Hispanic

0.1

1.6

0.2

79.2

1.3

3.2

12.8

Asian/Pacific Islander

0.0

0.0

0.1

8.0

28.8

55.4

12.1

Other

0.0

0.0

0.2

0.2

11.0

0.8

1.8

Missing

0.6

0.4

1.7

3.1

17.1

9.6

4.7

Household head (%)

80.4

86.8

86.0

48.3

62.0

62.2

73.1

Female (%)

82.7

87.6

80.9

61.8

72.1

69.8

76.9

Age (%)

18–24 years

23.2

34.9

35.1

30.1

25.1

27.5

29.6

25–34 years

23.2

34.9

35.1

30.1

25.1

27.5

29.6

35–44 years

36.2

24.5

24.0

39.3

36.1

28.5

31.1

Average age (years)

31.0

29.0

28.7

31.0

30.9

29.8

30.0

Lived in a household with (%):

Two or more adults

26.1

17.3

20.3

63.7

45.8

52.8

35.1

No children

31.9

19.4

24.7

25.7

23.7

9.2

23.3

Children ages

0–5 years

31.2

45.1

46.6

28.1

44.8

53.6

41.2

6–17 years

57.2

57.6

51.6

67.2

58.5

77.3

60.2

Sample size

792

728

874

615

598

498

4,105

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Source: MDRC calculations from housing authority records collected for the original Jobs Plus demonstration evaluation.

16

Exhibit 7. Housing Subsidy Receipt Among Adults and Children in Year 21 of Followup for Jobs Plus and Comparison Developments Combined by Site

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

The children sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

This table reflects receipt of any type of housing assistance captured in HUD IMS/PIC data, including public housing, housing choice vouchers, and multifamily housing.

Source: MDRC calculations using HUD IMS/PIC data.

The timeframe for the original and long-term outcomes analysis spans from 1992 to 2020. As shown in exhibit 8, the original Jobs Plus demonstration was implemented during the years 1998–

2003. Effects have been estimated in previous studies for the followup period 1998–2006. For both the adult and child samples, the analyses presented in this paper estimate the effects for the followup period Quarter 3 2017 to Quarter 2 2019, the quarters of data that approximate Years 20 and 21 after the program was launched. Specifically, Year 20 is estimated using data from Quarter 3 2017 to Quarter 2 2018 and Year 21 is estimated using data from Quarter 3 2018 to Quarter 2 2019.

17

Exhibit 8. Timeline of the Original Jobs Plus Demonstration and Followup Analysis

Year

2007-2016

Period covered by the HUD

long-term impacts analysis;

15–16 years post-program

Period covered by the original Jobs Plus

demonstration evaluation and followup analysis

Data Sources and Key Outcomes

The three data sources used in the analyses are described below.

• National Directory of New Hires (NDNH). NDNH data contain quarterly wage data for workers in employment covered by the unemployment insurance (UI) system. 40 These data are used to create two primary outcome measures (annual earnings and employment) and a set of secondary outcome measures (length of employment spells and earnings distributions) for sample members during the followup period, Quarter 3 of 2017 to Quarter 2 of 2019, roughly 20 and 21 years respectively following program launch.

• HUD Inventory Management System (IMS)/PIH Information Center (PIC) data . These data are used to describe residents’ rates of continued participation in subsidized housing programs in Years 20 and 21 of followup for both the adults of the original Jobs Plus evaluation and their children.

• Baseline data from the original demonstration . Housing authority records collected for the original Jobs Plus demonstration are used to identify the sample and to create measures that 40 These data miss some employment that is not covered by the UI system, including informal work and self-employment.

18

are used as covariates in the impact model. These include demographic characteristics (including age, race/ethnicity and gender), household composition, baseline earnings, and employment.

Analytic Approach

To determine whether Jobs Plus improved residents’ labor market outcomes, the original evaluation set up a careful research design that included a reliable comparison group. Within each city participating in the demonstration, several housing developments that were similar in size and in the demographic composition of their residents were identified. MDRC then randomly allocated one of these developments to a program group that got Jobs Plus, while the other one or two were allocated to the comparison group. This cluster random assignment strategy, combined with an interrupted time-series analysis using long-term trend data, allowed researchers to estimate the impact of living in a Jobs Plus housing development on employment and earnings, relative to what those outcomes would have been in the absence of Jobs Plus. The impact of Jobs Plus on work and welfare was estimated by comparing changes in outcomes after Jobs Plus was launched for the Jobs Plus group with changes in outcomes for the comparison group.

The present analysis of effects in Years 20 and 21 is conducted using a different method.

Impact models are estimated using individual-level data, in which earnings in Years 20 and 21 are regressed on treatment status, a series of variables capturing demographic characteristics, and a measure of the individual’s earnings in 1998, included as a series of variables capturing ranges of earnings. 41 Earnings in 1998 (the first year of Jobs Plus rollout, during which the program was not expected to generate impacts) is used to control for any pre-program differences between the Jobs Plus and comparison residents. The original evaluation found that the program and comparison developments within each site included residents who, on average, were very well matched, both in demographic characteristics and in long-term pre-program employment and earnings trends. Good matches were obtained for all sites combined, as well as within each site. The fact that the Jobs Plus and comparison groups were so well matched from the outset suggests that the regression method will provide a credible estimate of Jobs Plus’s long-term impact.

Nonetheless, a sensitivity analysis (not shown) was conducted using data from the original evaluation and a continuous measure of 1998 earnings. For the analysis sample, a regression model was used to estimate the effect of Jobs Plus on earnings in 1998. The resulting estimate, used as a measure of the pre-program difference between the two research groups, is then subtracted from the main estimate presented in the text, akin to a difference-in-difference analysis. The pre-program 41 Standard errors are adjusted to account for the clustering of outcomes for adults from the same household using Huber-White standard errors. Furthermore, it was not possible to include the continuous variable of baseline earnings as a covariate in the model due to restrictions on how the NDNH wage data could be linked with other data sources. In an effort to minimize the risk of identifying individuals, data on actual earnings could not be linked with NDNH data, but data aggregated into ranges of earnings could be linked.

19

difference in earnings was very small ($72 for sites with stronger implementation and $227 for sites with weaker implementation), so the results from the sensitivity analysis are very similar to those presented in the main report.

One limitation of this analytic method is that it does not account for the fact that the intervention was delivered at the development level (rather than to randomly selected individuals within developments) or for factors at the development level that will affect outcomes for all residents in that development. For this reason, this model most likely overestimates the precision and statistical significance of estimated impacts. Although there are standard statistical methods available to account for this issue, using them requires many sites, more than the six sites available for this analysis. To provide some assessment of how much the model overestimates statistical significance, we conduct a separate “permutation test” for all analyses and discuss the results in the text (see appendix E for details and full results). Thus, the estimates presented in the exhibits should be interpreted with this limitation in mind.

The same analytic method used for the adult analysis is used to estimate the effects on children’s employment and earnings in adulthood. 42 The child-level analysis focuses on the three Jobs Plus sites with stronger implementation, which had evidence of positive impacts on adult earnings at the time the program was implemented and for 3 years after the program ended.

However, effects on children were also estimated for the sites with weaker implementation (and that did not demonstrate effects on earnings for adults) to test whether the pattern of effects (or lack of effects) is consistent with the expectation that larger gains in parental earnings would lead to stronger effects on children’s later employment and earnings outcomes.

Finally, when many impacts are estimated, there is an increased risk that at least one estimate will be statistically significant simply by chance. For this reason, impacts on confirmatory outcomes are typically subjected to statistical adjustments to account for multiple hypothesis testing and reduce the risk of a false positive. Impacts on the two confirmatory outcomes presented in this report (average earnings over Years 20 and 21 and average quarterly employment over Years 20 and 21) are adjusted using the Benjamini-Hochberg method. 43

Covariates

The evaluation of the original demonstration used aggregate data for its comparative interrupted time series methodology and did not include any covariates. For the OLS model that will be used for this analysis, covariates are important for increasing the precision of the estimates and for controlling for differences between the Jobs Plus and comparison groups at baseline. For each followup year in the original CITS analysis, the comparison-group deviation from its baseline 42 Standard errors are adjusted to account for the clustering of outcomes for children from the same household using Huber-White standard errors.

43 Benjamini and Hochberg (1995).

20

outcome trend was used as the counterfactual for the corresponding deviation from the baseline trend of the Jobs Plus group. In the present OLS analysis however, the outcome of the comparison group for a given follow up year is the counterfactual for the outcome of the treatment group. As noted earlier, this counterfactual is credible because the baseline levels and trends in the earnings were very similar between each Jobs Plus group and its comparison group. Including covariates, however, will further improve the balance in baseline characteristics between the two groups. The covariates for the adult analysis include baseline employment, earnings, age, gender, race/ethnicity, number of children in the household, and site indicators. The models for the child analysis include the same set of covariates, except that they include household-level employment and earnings instead of individual-level employment and earnings. As noted, due to restrictions in analyzing NDNH data, both earnings and age are included as ranges in the impact models.

21

III. Long-Term Effects for Adults

Exhibit 9 presents effects on earnings and employment for residents from all sites combined.

The top panel of the exhibit shows Summary Outcomes, averaged for Years 20 and 21 combined, and the bottom panel presents the annual estimates (or Yearly Outcomes). Throughout the discussion, long-term effects are compared with the effects found from the original evaluation, with the dollar amounts adjusted for inflation and shown in 2018 dollars. Although the earlier effects were estimated using a different statistical method than for the original study sample, they provide a useful reference point for considering long-term effects.

First, starting with the top panel (Summary Outcomes), consider outcomes for the comparison group. Comparison group residents earned on average $15,531 per year over the 2-year period. This average includes zeros for those who did not work during a year. On average, about 50

percent of the comparison group worked in a given quarter over the 2-year period.

In terms of program impact, the estimated impact on earnings during the final year (2003) of the original followup period was $472, a statistically significant difference. In contrast, the impact on average earnings in Years 20 and 21 was -$236 and not statistically significant. One reason for the different impacts at the two points in time is that in the original evaluation, the positive effect for all sites was driven by positive effects in the sites that were deemed strong implementers of Jobs Plus.

Effects for the weaker implementing sites were close to zero. As the next table shows, in the long-term followup, the negative effect overall, although statistically insignificant, reflects both positive and negative effects across sites. Finally, effects on earnings and employment in individual years (shown in the Yearly Outcomes panel) show a similar story and are also statistically insignificant.

22

Exhibit 9. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, All Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Summary Outcomes

Average Annual Earnings, Years 20 and 21 ($)

15,295

15,531

– 236

0.694

Average Quarterly Employment, Years 20 and 21 (%)

50.2

50.8

– 0.5

0.713

Yearly Outcomes

Total Earnings ($)

Year 20

15,136

15,618

– 482

0.429

Year 21

15,858

15,833

25

0.969

Employed at least One Quarter (%)

Year 20

55.4

56.5

– 1.0

0.497

Year 21

55.1

55.7

-0.6

0.677

Average Quarterly Employment (%)

Year 20

50.5

51.5

– 1.1

0.462

Year 21

50.4

50.4

0.0

0.996

Sample size (total = 4,105)

1,780

2,325

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires.

Exhibit 10 presents effects by implementation strength. The left panel presents effects for the stronger implementation sites (Dayton, Los Angeles, and St. Paul), and the right panel present effects for the weaker implementation sites (Baltimore, Chattanooga, and Seattle). Jobs Plus continued to positively impact earnings in the strong implementing sites. The effect on average annual earnings is $1,670, an 11-percent increase that is statistically significant. Average quarterly employment rates were also higher for the Jobs Plus group, with an increase of 4.2 percentage points over the 2-year period. In the 3-year follow-up for the original study report, or 2000 through 2003, Jobs Plus increased earnings by about 14 percent. Thus, the effect is somewhat smaller in the longer term, but still notable.

23

Exhibit 10. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Program Implementation Strength Stronger Implementation Sites

Weaker Implementation Sites

Program Comparison

Program Comparison

Outcome

Group

Group Difference

P-Value

Group

Group Difference

P-Value

Summary Outcomes

Average Annual Earnings, Years 20 and

21 ($)

16,860

15,190

1,670 *

0.056

13,778

15,744

-1,967 **

0.019 †††

Average Quarterly Employment,

Years 20 and 21 (%)

54.3

50.1

4.2 **

0.047

46.4

51.1

-4.7 **

0.018 †††

Yearly Outcomes

Total Earnings ($)

Year 20

16,752

15,306

1,445

0.103

13,603

15,776

-2,174 **

0.010 †††

Year 21

17,357

15,461

1,896 **

0.038

14,406

16,077

-1,671 *

0.055 †††

Employed at least One Quarter (%)

Year 20

59.6

56.0

3.6

0.105

51.4

56.6

-5.2 **

0.015 †††

Year 21

59.4

55.3

4.1 *

0.065

50.9

55.9

-5.0 **

0.019 †††

Average Quarterly Employment (%)

Year 20

54.5

51.0

3.6 *

0.098

46.6

51.7

-5.1 **

0.013 †††

Year 21

54.4

49.6

4.8 **

0.026

46.5

50.8

-4.3 **

0.035 †††

Sample size (total = 4,105)

914

1,073

866

1,252

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5

percent; † = 10 percent.

Stronger implementation sites include Dayton, Los Angeles, and St. Paul. Weaker implementation sites include Baltimore, Chattanooga, and Seattle.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

The impact estimates for the two confirmatory outcomes (average annual earnings over Years 20 and 21 and average quarterly employment across Years 20

and 21) were each statistically significant; therefore, based on the Benjamini-Hochberg multiple hypothesis testing approach, no further adjustments to the p-values were needed.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires.

24

As noted earlier, the analysis method likely overestimates the precision of the impact estimates, as denoted by the p-value and asterisks. The p-value indicates the likelihood that the observed difference between the Jobs Plus and comparisons groups arose by chance. The p-value of .056 for the impact on earnings in the strong implementation sites suggests that the impact is statistically significant, having a very low (6 percent) probability of having arisen by chance. The permutation test (described in appendix E) was conducted to assess how much the precision may be overestimated. The findings from that analysis suggests that the p-value of the impact on earnings is 0.249, suggesting a 25 percent chance the effect arose by chance. The p-value for the effect on employment was 0.166 from the permutation test, versus 0.047 in the table.

Thus, the findings suggest that there is somewhat more uncertainty around the effects of Jobs Plus in the strong implementation sites. Nonetheless, the estimated effects reflect the same patterns that were observed previously for shorter-run outcomes, as illustrated in exhibit 11. Hence, the present findings provide suggestive, albeit weaker, evidence that longer-term Jobs Plus impacts for adults are qualitatively similar to the shorter-term impacts.

25

Exhibit 11. Mean Quarterly Earnings for the Work-Able Adults in the 1998 Cohort and the Long-Term Impact Sample: Stronger Implementation Sites

5,500

Jobs-Plus

Group

5,000

Comparison

Difference due to

Group

Jobs Plus (Years 20–

4,500

s)

21) = + $2,100/year

or + 13%

4,000

dolar

3,500

(in 2019 3,000

nings 2,500

2,000

Difference due to

ly Ear

Jobs Plus (2000–

ter

2006) =

1,500

uar

+ $1,800/year

Q

or + 16%

1,000

ean

Post-

M

Follow

500

program

Programs fully

up

Rollout

period

operational

Years

2004–2006

0

period

2000–2003

20–21

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

199

199

199

199

199

199

199

199

200

200

200

200

200

200

200

200

200

200

201

201

201

201

201

201

201

201

201

201

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Time Period

Note: All earnings amounts shown are in 2019 US dollars. In 2003 dollars, the earnings difference due to Jobs Plus was $1,300/year.

The original sample included work-able adults ages 20 to 61. The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Source: MDRC calculations using National Directory of New Hires (NDNH) data.

26

The estimated impact of $1,670 on earnings is an average effect across the full sample, but it does not mean that all sample members increased their earnings by $1,670. It could mean that the intervention had smaller, zero, or even negative effects on some parts of the sample and positive effects that are even larger than the average effect for other parts of the sample, based on individual levels of earnings. To assess these possibilities, a distributional analysis of the Jobs Plus and comparison group samples was conducted, where average earnings over Years 20 and 21 for both the Jobs Plus and comparison groups were rank-ordered, and instead of comparing the average outcomes, it compares the outcomes at multiple points in the distribution. Exhibit 12 shows the result: Jobs Plus had no measurable impact on the distribution over the lowest 50 percent of the sample, which is not surprising because much of the bottom 50 percent (about 40 percent) in both groups reported no earnings. 44 Earnings impacts for the top 40 percent of the sample, however, are statistically significant, with about a $1,500 positive impact seen at the 60th percentile and about a $3,500 difference at the 90th percentile.

Cases in which the long-term findings differ from the original study findings are for the weaker implementing sites, which show large negative effects on earnings—a statistically significant reduction of $1,967. (Findings from the permutation test yielded a p-value of 0.111, suggesting that the statistical significance is only slightly overestimated.) In the original study, the weaker sites did show a negative difference in earnings, but one that was relatively small and statistical insignificant.

This early slightly negative effect overall has grown into a relatively large and statistically significant negative effect in the long-term. Exhibit 13 illustrates which sites may be driving the results.

44 The dark line represents estimated effects, and the dashed lines represent 90 percent confidence intervals around those estimates. The confidence interval illustrates the uncertainty, or margin of error, around an estimate, and if that interval includes the value 0, then the estimate is not statistically different from zero. Since earnings are zero at the lower percentiles for both the Jobs Plus and comparison groups, the impact at those lower percentiles is also zero, and it is indistinguishable from the grey line indicating zero impact.

27

Exhibit 12. Effect on Average Annual Earnings Distribution, Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, Stronger Implementation Sites

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

The effects shown are calculated using quantile regression to estimate these effects at each decile of the annual earnings distribution. The measure used in this regression is the average annual earnings across Years 20 and 21.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires.

Exhibits 13 and 14 present effects for individual sites by implementation strength. For the stronger implementation sites, the positive overall effect is being driven by large effects in Los Angeles and St. Paul, although the impacts in St. Paul are not statistically significant at the 10

percent level. The effect in Dayton is not significant. Comparing these effects to the inflation-adjusted effects in Year 6 illustrates how the impacts changed over time: the impact changed from $1,754 to -$932 in Dayton, from $1,892 to $3,812 in Los Angeles, and from $2,919 to $2,902 in St.

Paul. Thus, the effects dissipated in Dayton, grew larger in Los Angeles, and stayed the same in St.

Paul.

The sustained positive effects in the stronger implementation sites as a whole are encouraging and consistent with the finding that earnings increase with work experience. It is not clear what may have driven the fading effects in Dayton. One potential factor, however, may be the 28

higher move-out rates in that site, which reduced program exposure. By Year 2, for example, 48

percent of Dayton residents had moved out of the Jobs Plus development, compared with 17 percent in Los Angeles and 27 percent in St. Paul. 45

Finally, another factor to consider, as noted earlier, is potential changes in either the Jobs Plus or comparison developments over time. The comparison development in Los Angeles went through a HOPE VI redevelopment during the later part of the original followup period. By the end of 2003, demolition was complete, and all residents had been relocated, with Section 8 vouchers.

Thus, it is possible that this disruption for the comparison residents led to increased impacts on earnings over time.

Similar to the positive effects in the stronger implementation sites, the negative effects in the weaker implementation sites are driven by individual sites. Comparing the long-term effects with those in Year 6 shows the following: the effect changed from -$420 to -$673 in Baltimore, from

-$1,311 to -$2,693 in Chattanooga, and from -$355 to -$2,684 in Seattle. Thus, the effect grew more negative in the latter two sites, particularly in Seattle, although the impact estimate is not statistically significant at the 10 percent level.

It is unlikely that Jobs Plus led to negative effects in the weaker implementation sites, and evidence suggests that it may be due to other changes to the Jobs Plus and comparison developments.

First, consider Seattle. The original study report provides some insight into why the early positive impacts in that site may have faded by Year 6. First, the Seattle site underwent a HOPE VI revitalization during the followup period, meaning that a large number of residents were relocated to other housing. Moving to housing potentially in other neighborhoods may have reduced access to jobs and supportive networks. Also, the comparison sites introduced new employment services and rent incentives as part of the housing authority’s Moving to Work demonstration. The latter factor may have also contributed to the long-term negative effects in this site.

Exhibit 13. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Site, Stronger Implementation Sites Summary Outcomes

Program Comparison

Outcome

Group

Group Difference

P-Value

Dayton

Average Annual Earnings, Years 20 and 21 ($)

13,567

14,499

– 932

0.399 ††

Average Quarterly Employment, Years 20 and 21 (%)

53.5

55.8

– 2.3

0.458 †

Sample size (total = 874)

308

566

Los Angeles

Average Annual Earnings, Years 20 and 21 ($)

18,241

14,429

3,812 **

0.038 ††

Average Quarterly Employment, Years 20 and 21 (%)

52.4

43.9

8.5 **

0.040 †

45 Bloom et al. (2005).

29

Program Comparison

Outcome

Group

Group Difference

P-Value

Sample size (total = 615)

326

289

St. Paul

Average Annual Earnings, Years 20 and 21 ($)

19,845

16,943

2,902

0.174 ††

Average Quarterly Employment, Years 20 and 21 (%)

53.9

47.6

6.3

0.173 †

Sample size (total = 498)

280

218

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires.

30

Exhibit 14. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Site, Weaker Implementation Sites Summary Outcomes

Program Comparison

Outcome

Group

Group Difference

P-Value

Baltimore

Average Annual Earnings, Years 20 and 21 ($)

13,820

14,493

– 673

0.636

Average Quarterly Employment, Years 20 and 21 (%)

43.2

45.4

-2.2

0.510

Sample size (total = 792)

276

516

Chattanooga

Average Annual Earnings, Years 20 and 21 ($)

11,483

14,176

– 2,693 **

0.018

Average Quarterly Employment, Years 20 and 21 (%)

51.2

57.1

-5.9 *

0.077

Sample size (total = 728)

269

459

Seattle

Average Annual Earnings, Years 20 and 21 ($)

16,881

19,565

– 2,684

0.164

Average Quarterly Employment, Years 20 and 21 (%)

45.3

51.0

-5.8

0.157

Sample size (total = 598)

321

277

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires.

31

Next, consider the Chattanooga site, which had negative effects by Year 6. Although it is easy to imagine that reductions in work and earnings in the short-term can lead to lasting effects, it is not clear why this site had a negative effect at the outset. As noted in the original study report, in Year 5 of the followup period, Chattanooga changed the Jobs Plus program to financial incentives only, given continued challenges in providing the other two components, and even this component was not strongly implemented. Thus, the program was never fully implemented, although the move to financial incentive only should not have led to negative effects. In terms of the larger negative effects in the long-term, one factor may be the relocation of the Jobs Plus residents. The Jobs Plus development was vacated in 2015 after the city bought the land on which it sits. 46 As with the HOPE

VI revitalization in Seattle, the resulting dislocation may have negatively affected former residents.

The extent to which former residents were affected, however, would depend on what proportion of the sample was still living in the development in 2015, 17 years after the start of Jobs Plus.

The original study report presented effects for several subgroups within the stronger implementation sites. Effects on earnings were larger, for example, for 1) residents not receiving TANF at program start, 2) residents who had lived in the development for 4 or more years, and 3) younger residents. Exhibits 15 and 16 present effects by age at program launch for stronger implementation sites and for weaker implementation sites. Effects do not significantly differ for the stronger implementation sites. Effects on earnings, for example, range from $1,038 to $2,154, variation that is not statistically significant. Negative effects on earnings in the weaker implementation sites are larger for younger residents, although the differences in impacts across the three age groups are not statistically significant.

Finally, the study examined effects on measures of employment stability (e.g., number of consecutive quarters employed) and measures of earnings levels (e.g., earnings above certain thresholds). Because these are secondary outcomes, the results are presented in appendix C. In general, the findings align with those presented for the primary outcomes of earnings and employment. The strong implementation sites increased sustained employment and increased the number of residents with moderately high earnings (or above $7,500 per year and above $10,000 per year). The weaker implementation sites reduced sustained employment and reduced the number of residents with earnings above these thresholds.

46 https://wdef.com/2019/01/16/former-site-harriet-tubman-homes-rezoned/

32

Exhibit 15. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Age at Baseline, Stronger Implementation Sites Summary Outcomes

Program Comparison

Outcome

Group

Group Difference

P-Value

Age 18-24

Total Annual Earnings, Average of Years 20 and 21 ($)

22,205

21,167

1,038

0.558

Average Quarterly Employment, Years 20 and 21 (%)

65.7

65.4

0.3

0.939

Sample size (total = 629)

285

344

Age 25-34

Total Annual Earnings, Average of Years 20 and 21 ($)

17,381

15,790

1,591

0.248

Average Quarterly Employment, Years 20 and 21 (%)

59.5

54.7

4.8

0.163

Sample size (total = 764)

362

402

Age 35-44

Total Annual Earnings, Average of Years 20 and 21 ($)

10,645

8,491

2,154

0.121

Average Quarterly Employment, Years 20 and 21 (%)

35.0

29.5

5.4

0.163

Sample size (total = 594)

267

327

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Stronger implementation sites include Dayton, Los Angeles, and St. Paul.

Ages referenced in this exhibit indicate the age of adults at baseline in 1998. In Year 20 of followup, the corresponding ages of these adults are 38–44, 45–54, and 55–64.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires.

33

Exhibit 16. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Age at Baseline, Weaker Implementation Sites Summary Outcomes

Program Comparison

Outcome

Group

Group Difference

P-Value

Age 18-24

Total Annual Earnings, Average of Years 20 and 21 ($)

18,540

21,907

– 3,367 *

0.063

Average Quarterly Employment, Years 20 and 21 (%)

60.3

70.6

– 10.2 ***

0.006

Sample size (total = 588)

230

358

Age 25-34

Total Annual Earnings, Average of Years 20 and 21 ($)

14,365

16,832

– 2,467 *

0.069

Average Quarterly Employment, Years 20 and 21 (%)

50.9

54.4

-3.5

0.292

Sample size (total = 849)

355

494

Age 35-44

Total Annual Earnings, Average of Years 20 and 21 ($)

9,316

9,154

162

0.897

Average Quarterly Employment, Years 20 and 21 (%)

29.4

30.6

– 1.2

0.735

Sample size (total = 681)

281

400

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Weaker implementation sites include Baltimore, Chattanooga, and Seattle.

Ages referenced in this exhibit indicate the age of adults at baseline in 1998. In Year 20 of followup, the corresponding ages of these adults are 38–44, 45–54, and 55–64.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires.

34

IV. Long-Term Effects on Labor Market Outcomes of Children Living in Jobs Plus Households

This section turns to an analysis of the program’s effects on employment and earnings for the children who were living in the Jobs Plus developments when the program launched in 1998. As noted earlier, the analysis of long-term effects for children are considered exploratory and includes all children living in the Job Plus developments, whether or not the adults in their households are included in the adult impact analysis. The patterns of labor market effects for children discussed in this section align with the patterns for adults: children living in households in the Jobs Plus developments that fully implemented the program had higher average earnings and employment rates about 20 years after program implementation began than their counterparts in comparison developments, whereas there was no evidence of any effects on employment rates or average earnings for children living in Jobs Plus developments where implementation was incomplete or weak. Effects on Children’s Later Labor Market Outcomes in Stronger Implementation Sites As with the exhibits in the previous section, the top panel of the exhibits in this section show Summary Outcomes, averaged for Years 20 and 21 combined, and the bottom panel presents the annual estimates (or Yearly Outcomes). The findings show that the Jobs Plus demonstration as a whole—pooling together all six sites that participated in the demonstration, regardless of implementation completeness—improved children’s labor market outcomes 20 years after the start of program implementation, when children who were 17 or under at the time of the 1998 program launch were ages 18 through 36 at the start of the long-term follow-up period and 20 to 38 years old at the end of the period. 47 As expected, these positive effects on children’s long-term labor market outcomes are driven almost entirely by the children who were living in the sites with stronger implementation (Dayton, Los Angeles, and St. Paul). The study hypothesized that any effects on children’s later labor market outcomes would have occurred in the stronger implementation sites, where there were positive and sustained effects on the work-able adult residents’ average earnings, rather than in the weaker implementation sites that did not demonstrate evidence of positive effects on work-able residents’ employment or earnings outcomes.

In these three sites combined, results from the evaluation of the original demonstration found positive effects on average earnings of work-able adults—an increase in about $1,300 (or 16 percent) per year in 2000–2006 dollars (the equivalent of about $1,800 in 2019 dollars), on average, across the 4 years of post-rollout program operations and the 3 years following, as mentioned earlier—but no overall effects on employment rates across these three sites. The results of the present long-term analysis show that these sustained effects on adult work-able residents’ earnings translated to substantial improvements to the long-term average earnings for the children living in those 47 Appendix exhibit C.12 shows the estimated long-term effects on average earnings and employment rates for children in all six sites that participated in the demonstration pooled together.

35

developments, whose effects on average annual earnings even exceeded those of the effects on adult earnings.

Exhibit 17 shows that children living in stronger implementation sites had average annual earnings that were about $2,700 higher 20 and 21 years later than children living in comparison developments that did not implement Jobs Plus. Children living in these three Jobs Plus developments earned about $21,000 annually, on average, across Years 20 and 21. In contrast, children in the comparison group had average annual earnings of about $18,300 during this followup period. These estimated effects on average earnings are highly statistically significant in both followup years. It is interesting to note that, for both the program and comparison groups, the earnings levels for the child sample during Years 20 and 21 are significantly higher than for the adult sample, which were approximately $17,000 annually for the program group and $15,000 annually for the comparison group during the followup period. This might be explained by the lower employment rates for the adult sample, likely because of their older ages.

Exhibit 17. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Children in Households from the 1998 Cohort, Stronger Implementation Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Summary Outcomes

Average Annual Earnings, Years 20 and 21 ($)

20,961

18,255

2,706 ***

0.000

Average Quarterly Employment, Years 20 and 21 (%)

72.8

66.6

6.2 ***

0.000

Yearly Outcomes

Total Earnings ($)

Year 20

20,322

17,977

2,345 ***

0.001

Year 21

22,143

19,095

3,048 ***

0.000

Employed at least One Quarter (%)

Year 20

81.9

76.6

5.3 ***

0.001

Year 21

81.5

75.7

5.7 ***

0.000

Average Quarterly Employment (%)

Year 20

72.7

67.4

5.3 ***

0.001

Year 21

73.2

66.1

7.1 ***

0.000

Sample size (total = 3,322)

1,622

1,700

36

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Stronger implementation sites include Dayton, Los Angeles, and St. Paul.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires.

The theory of change predicts that increasing parents’ earnings should increase their children’s earnings as adults. However, other research suggests that the size of Jobs Plus effects observed for the children are larger than what would be expected from parents’ earnings gain alone.

This research, on economic mobility, examines the correlation between parents’ and children’s earnings, and data from the U.S. suggests that increasing income for parents by 10 percent, for example, can lead to a positive effect on children’s adult earnings of between 3 and 5 percent. 48

Thus, the effect observed for Jobs Plus (an increase in earnings of 15 percent for the children) is large, given that the effects on earnings for adults were about 16 percent in the shorter-term and 11

percent in the long-term. This finding suggests that the effect on children may have operated through additional pathways. If parents moved to neighborhoods with more resources, for example, or if the children moved as adults, this would affect their adult outcomes. Data on mobility are not available to assess this hypothesis.

Finally, as was done for the adult impacts, the effects for children were estimated using the permutation test to assess how much the standard model overestimates the precision of the impact estimate. That analysis (shown in appendix D) suggested that the effects for children are statistically significant. The permutation test for effects on earnings and employment in the stronger implementation sites yielded p-values of 0.083. P-values of less than 0.10 are typically considered statistically significant.

In addition to the long-term effects of Jobs Plus on average later earnings for children living in the Jobs Plus developments, like the long-term impact analysis for the adult sample, the study also examined the long-term effects at various points throughout the distribution of earnings during this 2-year followup period. Exhibit 18 illustrates the effects on average annual earnings for Years 20 and 48 See Aaronson and Mazumder (2008). There is, however, some research showing that this association between parents’ and children’s earnings is stronger at the lower end of the income distribution (Palomino, Marrero and Rodriguez, 2018).

37

21 at each decile of the distribution. 49 It shows that the long-term impact on children’s earnings is consistently positive and statistically significant from the 10th percentile of the distribution to the 90th percentile, with the impact rising from about $1,000 at the 20th percentile to about $4,000 at the 90th percentile. Thus, Jobs Plus had positive impacts on all parts of the earnings distribution for children. Beginning around the 30th percentile, the effects ranged from about $2,800 to $4,300, without a clear pattern of effects being concentrated in any part of the distribution above the 30th percentile and without any clear upward or downward trend.

Children living in Jobs Plus developments with stronger implementation also had higher annual employment rates than their counterparts living in comparison developments. Exhibit 17

shows that average quarterly employment rates were 6.2 percentage points higher for program group children than comparison group children living in the Jobs Plus developments during the 2-year followup period. Average quarterly employment rates were 72.8 percent in the program group, compared with 66.6 percent in the comparison group. Like with annual earnings, the child sample had much higher employment rates in Years 20 and 21 compared with the adult sample. The adult sample had annual employment rates of 54.3 percent for the program group and 50.1 percent for the comparison group in these households living in the stronger implementation sites during the same time period.

The impact estimates on the yearly outcomes in exhibit 17 align with the impact estimates on these summary measures of average annual earnings and average quarterly employment across the 2-year followup period. The impact estimates on total annual earnings, annual employment, and average quarterly employment for Years 20 and Years 21 are all positive and highly statistically significant.

The estimated effects of Jobs Plus on employment stability and earnings levels (secondary outcomes for the study) align closely with these patterns of findings for average earnings and employment rates described above. These results are presented in appendix exhibit C.14. In stronger implementation sites, children in the program group are employed, on average, 5.8 quarters of the 8-quarter followup period 20 to 21 years after Jobs Plus began, compared with 5.3 quarters for the comparison group. The average length of the longest employment spell within the 8-quarter followup period is 5.5 quarters for the program group and 5.0 quarters for the comparison group. Seventy-three percent of the child sample in the program group worked for at least four consecutive quarters within this study period, compared with 66 percent of the comparison group children. The program group had a higher proportion of sample members who earned above each of the three earnings thresholds examined—$7,500, $10,000 and $15,000 annually—compared with the comparison group.

49 For example, a program group member with annual earnings during the long-term followup period that is at the median (the 50th percentile) of program group earnings had earnings during that period that were $3,451 higher than a comparison group member who had annual earnings at the median of the comparison group.

38

Exhibit 18. Effects on Average Annual Earnings Distribution, Years 20 and 21

of Followup, Children in Households from the 1998 Cohort, Stronger Implementation Sites

Mean Impact by Decile

90% Confidence Interval

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

The effects shown are calculated using quantile regression to estimate these effects at each decile of the annual earnings distribution. The measure used in this regression is the average annual earnings across Years 20 and 21.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires.

39

As expected, in weaker implementation sites where there was no evidence of effects on work-able adults during 4 years of the program or the 3 years following the end of implementation, there was also no evidence of long-term effects on labor market outcomes of children living in those developments. The results for the weaker implementation sites are presented in appendix exhibit C.13 and compared statistically with the effects on stronger implementation sites. The estimated effects on employment and earnings outcomes for children living in weaker implementation sites are close to zero and not statistically significant in the weaker development sites, both for the primary earnings and employment outcomes in appendix exhibit C.13 and for the secondary employment stability and earnings distributions outcomes in appendix exhibit C.14 The differences in effects between the two groups (children living in stronger versus weaker implementation sites) are highly statistically significant.

Effects on Children’s Later Labor Market Outcomes by Site

The present study also examined the effects of Jobs Plus on children’s later labor market outcomes separately for each of the three stronger implementation sites: Dayton, Los Angeles and St.

Paul. Impact estimates for individual sites should be interpreted with some more caution than the pooled results because the smaller sample sizes within each site lead to less certainty about the estimated effects, and results from single sites are less generalizable to other public housing developments than pooled results from three sites combined.

Exhibit 19 shows that the estimated effects of Jobs Plus on children’s later adult earnings are similar in magnitude across the three stronger implementation sites. The increases in average annual earnings range from about $1,900 to $2,300, depending on the site. Due to the smaller sample sizes by site, the differences in earnings between the program and comparison groups are only statistically significant in Dayton. A statistical test of the differences in the effects across the three sites also showed that these differences were not statistically significant for any of the earnings and employment outcomes in exhibit 19, though this could be due to the smaller samples by sites and the low statistical power to detect differences in effects across sites. In other words, it is uncertain whether the variation in estimated impacts across sites reflects a true difference.

While the estimated effects on earnings are comparable in size across the three stronger implementation sites, exhibit 19 shows a pattern of larger effects on employment rates in two of the three stronger implementation sites, Dayton and Los Angeles compared with St. Paul. The effects on average quarterly employment rates during the 2-year follow-up period are similar between Dayton and Los Angeles, with estimated effects of 7.5 and 7.0 percentage points, respectively. In St. Paul, however, the estimated effect on average quarterly employment rates during this 2-year period is much smaller (1.5 percentage points) and not statistically significant. The differences in effects on average quarterly employment, however, were not statistically significant for this 2-year summary measure, possibly due, in part, to the smaller sample sizes by site. Appendix exhibit C.15 presents the effects on the more detailed yearly measures by site and shows that the differences in effects on both 40

annual employment and average quarterly employment are statistically significant in followup Year 21, but not followup Year 20.

Exhibit 19. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Children in Households from the 1998 Cohort, by Site, Stronger Implementer Sites Summary Outcomes

Program Comparison

Outcome

Group

Group Difference

P-Value

Dayton

Average Annual Earnings, Years 20 and 21 ($)

14,841

12,613

2,228 ***

0.008

Average Quarterly Employment, Years 20 and 21 (%)

70.2

62.7

7.5 ***

0.001

Sample size (total = 1,379)

467

912

Los Angeles

Average Annual Earnings, Years 20 and 21 ($)

23,223

20,922

2,301

0.141

Average Quarterly Employment, Years 20 and 21 (%)

72.0

64.9

7.0 **

0.036

Sample size (total = 818)

417

401

St. Paul

Average Annual Earnings, Years 20 and 21 ($)

26,231

24,292

1,939

0.160

Average Quarterly Employment, Years 20 and 21 (%)

76.5

75.0

1.5

0.571

Sample size (total = 1,125)

738

387

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires.

Effects on Children’s Later Labor Market Outcomes by Child Age During Jobs Plus

It is possible that the effects of living in a Jobs Plus household as a child on later adult labor market outcomes may differ based on the child’s age at the time of Jobs Plus implementation. A subgroup analysis was conducted to compare the effects of living in a Jobs Plus household on later employment and earnings across the following three age groups: early childhood (0 to 5 years old), middle childhood (6 to 12 years old) and adolescence (13 to 17 years old).

41

Prior research shows that income increases can have larger benefits for younger children. It is also important to keep in mind, however, that any differences in effects across different age groups may be driven not only by the age of the child at the time of the demonstration and the differential influence of increasing household income at different stages of child development, but also by the age of the child at the time of followup in Years 20 and 21 and when any influences of increased income during childhood might manifests itself in adulthood. For example, any child who was under 5 years old at the time of the start of the demonstration will not yet have reached 25 years of age, a common threshold for studying effects on adult earnings, given post-secondary education, by the start of the long-term followup period. Therefore, if program effects emerge later in their adulthood, the estimated long-term effects for this age group may be smaller at the time of Year 20 and year 21

followup compared with potential effects 5 or 10 years later, making it inappropriate to directly compare estimated effects between the 0–5-year-old subgroup and the older, 6–12-year-old subgroup and draw conclusions about the relative influence of Jobs Plus based on the child’s age during the time of program implementation.

Exhibit 20 shows the results of this subgroup analysis. The estimated effects for both earnings and employment were positive for all three age groups; however, the pattern of effects shows that estimated effects on average annual earnings in both followup years were largest for children who were in middle childhood (age 6 to 12) at the time that Jobs Plus began in 1998, followed by those who were in early childhood (age 0 to 5)50. Average employment rates were largest for those who were in early childhood. While these patterns in effects by child age generally align with prior evidence of the effects of income on children, a statistical comparison of the effects shows that there are no statistically significant differences in the effects on any of the employment and earnings outcomes examined across the three age groups. It is possible that the sample sizes become too small when the sample is split into the three age groups (especially for the adolescent group) to detect differences in effects, but the lack of statistical significance in the test of effects across the three age groups indicates that, while there is evidence of effects for the full sample of children living in Jobs Plus developments, there is a high likelihood that the observed differences in effects across the three age groups are due to chance.

50 A sensitivity test was conducted that split the 6- to 12-year-old age group into two groups, 6- to 9-year-olds and 10- to 12-year-olds, and effects were estimated for those two groups separately. The estimated effects for each group were very similar to one another. The results are presented in appendix exhibit C.21.

42

Exhibit 20. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Children in Households from the 1998 Cohort, by Age at Baseline, Stronger Implementation Sites Summary Outcomes

Program Comparison

Outcome

Group

Group Difference

P-Value

Age 0-5

Average Annual Earnings, Years 20 and 21 ($)

15,893

14,288

1,605 *

0.066

Average Quarterly Employment, Years 20 and 21 (%)

75.2

68.7

6.5 ***

0.006

Sample size (total = 1,050)

482

568

Age 6-12

Average Annual Earnings, Years 20 and 21 ($)

22,353

19,202

3,151 ***

0.002

Average Quarterly Employment, Years 20 and 21 (%)

72.9

67.4

5.4 **

0.012

Sample size (total = 1,566)

770

796

Age 13-17

Average Annual Earnings, Years 20 and 21 ($)

25,044

22,899

2,146

0.248

Average Quarterly Employment, Years 20 and 21 (%)

68.1

62.3

5.8

0.118

Sample size (total = 706)

370

336

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Stronger implementation sites include Dayton, Los Angeles, and St. Paul.

Ages referenced in this table indicate the age of children at baseline in 1998. In Year 20 of follow-up, the corresponding ages of these children are 20-25, 26-32, and 33-37.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires.

43

V. Discussion of Findings

Exhibit 21 summarizes the effects on employment and earnings assessed so far: from the earlier analyses of the effects examined as part of the original demonstration, the followup analysis conducted shortly after the evaluation ended, as well as the long-term effects analyzed in the present study. The pattern of findings demonstrates that, where it was implemented well, Jobs Plus had sustained effects on the earnings of work-able adults living in the developments and that the program also results in long-term positive effects on the labor market outcomes of children living in the Jobs Plus developments at the time of implementation. The first row in exhibit 21 shows that, for the sites with stronger implementation together, there was evidence of positive effects on earnings across the 4 years of program implementation in the 2000–2003 analysis, the effects on earnings were sustained for 3 additional years after program operations ended (as shown in the 2000–2006) analysis, and that there was evidence of long-term effects on employment and earnings 15 years after program operations ended in the present 2017–19 long-term analysis. There was evidence of positive effects on employment and earnings for children in the same 2017–19 followup period.

For the three sites with weaker implementation (the second row in the exhibit), there was no evidence of effects during program implementation in the 2000–2003 analysis. Fifteen years after program operations ended, the comparison group had higher employment rates and average earnings than the program group (possibly due to disruptions to program group developments unrelated to Jobs Plus). There were no effects on children living in those developments with weaker implementation.

Study Limitations

While the findings from the long-term analysis are promising, it is important to recognize some of the caveats and interpret the findings with some caution. First is the study methodology. Two important strengths of the analytic approach are that (1) developments were randomly assigned within sites, and (2) there was a very close match in baseline levels and trends in average earnings between the program group and the comparison group. Despite these strengths, the study uses a methodology (OLS) that is less rigorous than the comparative interrupted time series methodology used in the original Jobs Plus demonstration evaluation (which was not feasible for the long-term analysis) and likely overestimates the statistical significance of the impact estimates. Separate analyses (using the permutation method) suggested that the effects on adults in the strong implementation sites were less precisely estimated that the standard analysis model would suggest.

The impact estimates, however, reflect the same patterns that were observed previously for shorter-run outcomes and thus provide suggestive evidence of longer-term effects for adults. The positive effects of children were also subject to the permutation test and remained statistically significant.

This analytic method also does not account for potential differences in the long-term trajectories of employment and earnings between the program and comparison group that might have existed even in the absence of Jobs Plus. In other words, there might exist unobserved differences between the 44

program and comparison group that might influence long-term labor market outcomes that OLS

cannot account for.

Second, the demonstration included six sites, and only three implemented the Jobs Plus model fully. It is a small number of sites for a test of a place-based program where the intervention occurs at the site level and not the individual level.

45

Exhibit 21. Summary of Impacts on Earnings and Employment for Adults and Children in the Jobs Plus Demonstration: Original Sample and Followup Study Samples

Work-Able Adults

All Children, 1998 Cohort

Jobs Plus Program

Implementation

1998 Cohort

1998 Cohort in Long-Term

Strength

Impacts Sample

2000–2003

2000–2006

2017–19

2017–19

Employment

Earnings

Employment

Earnings

Employment

Earnings

Employment

Earnings

Stronger

0

+

NA

+

+

+

+

+

Weaker

0

0

NA

NA

–

–

0

0

Notes: 0 indicates no statistically significant impacts. + indicates positive statistically significant impacts. – indicates negative statistically significant impacts. NA indicates that these impacts were not calculated.

Source: Bloom, Riccio, and Verma (2005), Riccio (2010), and MDRC calculations using quarterly wage data from the National Directory of New Hires.

46

Third, the long followup period introduces additional uncertainty given the potential influences unrelated to Jobs Plus that could have affected outcomes for either the program or comparison group. Two such instances were identified and discussed earlier in the report—a HOPE

VI redevelopment of the comparison development in the Los Angeles site early near the end of program implementation and the redevelopment of the program development in Chattanooga close to the long-term followup period—and these disruptions had implications for the interpretation of the differences in outcomes between the program and comparison groups in those sites. Since all the members of the program group and all the members of the comparison group within each site were living in different physical developments and in different neighborhoods at the start of Jobs Plus, the extended followup period exposes the long-term employment and earnings outcomes to further influences that may have affected one research group and not the other or affected the two groups disproportionately. Additional examples include self-sufficiency initiatives serving residents in one development in a site but not the other(s), or a transportation option being added to the location of one development and not the other(s).

47

VI. Conclusion

The results described in this paper provide important evidence on the potential long-term effects of a comprehensive place-based employment program for public housing residents. Jobs Plus was designed to address the challenges of concentrated poverty and the barriers to economic mobility faced by residents of urban public housing developments. Although varied self-sufficiency programs for housing subsidy recipients have been implemented to address these types of challenges, and some have been evaluated, the present study is one of the rare studies to follow program participants for many— in this case, 21—years to test whether program effects are sustained over the very long-term, and whether they have intergenerational effects on their children. 51

Furthermore, while much attention has been paid to the benefits of children leaving public housing in high-poverty areas for lower-poverty areas in the Moving to Opportunity (MTO) study, the findings from the present exploratory study of the long-term effects of Jobs Plus suggest that children can also benefit meaningfully from public investments in improving the environments in which they already live. Additionally, the findings from the present Jobs Plus long-term impact analysis might provide additional insights into the findings described in this report. Examining the effects for households with children separately from households without children might provide a more useful context for the estimated effects on children’s later labor market outcomes. Further exploratory analysis using survey data from the original demonstration—which included data on children’s behavioral and school outcomes in select sites—might provide some insight into the mechanisms of the effects on children’s later labor market outcomes.

What should the policymakers and practitioners make of the longer-term impacts for current replications and adaptations of the original Jobs Plus model? The general takeaways are encouraging, and they do seem to suggest that a robust implementation of Jobs Plus could serve as a platform for producing lasting economic gains for adults and children. As HUD continues to invest in the scale-up of the Jobs Plus program, it will be important to pay close attention to program implementation experiences and assess how best program exposure can be maximized for residents in developments where this program is being implemented. Another aspect of the model to assess is the strength of local partnerships in the current iteration of Jobs Plus, especially given the important role of mandatory collaboratives in the stronger implementation sites in the original demonstration.

Similarly, it will be important for HUD to pay attention to the evidence emerging from the replication study, focused on the early cohorts, to see whether those results also suggest any significant pattern of early impacts and what, if any, opportunities are presented for continuous program improvement.

Keeping in mind the caveats and cautions listed above, it is also worth considering the potential long-term benefits—or returns—of making investments in Jobs Plus type of place-based 51 Other evaluations of more traditional employment-focused programs, some which relied on sectoral strategies, followed participants for shorter periods of time (Project Quest, for instance, followed participants for 9 years).

While findings from these studies are positive and show encouraging long-term impacts on career advancement, for example, a question remains whether their impacts will persist as individuals change jobs or sectors.

48

self-sufficiency efforts. Until now, no information was available to assess whether, in the long term, Jobs Plus produces enduring outcomes. The results described in the report suggest a possible return on investment that more than covers the program’s net costs in a few years. 52 Drawing on the experience of the three stronger implementation sites (and without a formal cost analysis), the original evaluation developed a rough estimate of costs of operating the on-site features of the program (including the rent incentives): approximately $150 per targeted resident in any given month, or $1,800 per year. 53 Although not entirely comparable, the costs of the original Jobs Plus program are somewhat similar to the costs of the HUD Jobs Plus replication program, an analysis based on the first batch of HUD grantees that reached “steady state” operations. So far, HUD’s Jobs Plus grants have ranged between $1 to $3 million, depending on the development size and other factors, and are nonrenewable, one-time grants covering 4 years of program implementation and financial incentives—a scale of investment that may be justified for effective programs.

Finally, Jobs Plus programs around the country today serve roughly thousands of households in about 50 housing developments (HUD and non-HUD funded). The results presented in this report suggest that robust implementation of the program can prepare residents for successful transition to or advancement in the workforce. The report also just scratches the surface of unpacking the potential long-term and intergenerational effects of a successful place-based program.

52 Given the pattern of results for children, the study examined the feasibility of projecting lifetime earnings gains for children in the strong implementation sites (i.e., is Jobs Plus a good financial investment for families?).

Depending on the assumptions made for the pattern of impacts over time, the estimated gain in lifetime earnings for a given individual ranges from $19,000 to $79,000. See appendix D for a brief discussion.

53 Bloom, Riccio, and Verma (2005). Using this rough estimate, which the authors argue might represent a “high-end” estimate, PHAs with, for example, 250 eligible, working-age residents may need an annual budget in the vicinity of $450,000 per year to provide the on-site services and rent incentives.

49

APPENDIX A: Analytic Model and

Specifications

A-1

Building on the close match of the baseline trends of the Jobs Plus and comparison groups, this analysis will use an Ordinary Least Squares (OLS) model with prior earnings (four categorical annual earnings flags consisting of $5,000 intervals beginning with $1–$4,999 and ending with earnings of $15,000 or more) and other baseline variables as covariates in the model.

It will adjust the standard errors for clustering at the household level to account for the lack of independence among adults in the same household. The model is specified as follows: 𝑌𝑌𝑖𝑖𝑖𝑖 = 𝛼𝛼 + 𝛽𝛽T𝑖𝑖𝑖𝑖 + 𝛾𝛾𝛾𝛾𝑌𝑌𝑖𝑖𝑖𝑖+∑𝑘𝑘 𝛿𝛿𝑘𝑘 𝑋𝑋𝑖𝑖𝑘𝑘𝑖𝑖 + ∑𝑖𝑖 𝜑𝜑𝑖𝑖+𝑒𝑒𝑒𝑒𝑒𝑒 [1]

In this model:

Yij = annual earnings (or employment) for individual i in site j,

α = the mean outcome for the comparison development(s),

Ti = 1 if individual i lived in the Jobs Plus development and 0 if they lived in the comparison development,

𝛽𝛽 = the mean program effect,

𝛾𝛾𝑌𝑌𝑖𝑖𝑖𝑖 = the pre-test (P) value of Y over the baseline period for individual i in site j, broken into categories,

𝛾𝛾q = the coefficient for the pre-test variable,

∑𝑘𝑘𝑋𝑋𝑖𝑖𝑘𝑘𝑖𝑖 = a series of background characteristics k for participant i from site j, 𝛿𝛿𝑘𝑘= a set of coefficients for the background characteristics, and

∑𝑖𝑖𝜑𝜑𝑖𝑖= a series of indicators for each site to account for program fixed effects.

ei = A random error that varies independently across individuals with a mean of zero and a variance that can differ between treatment and comparison group members.

A-2

APPENDIX B: Summary of Program

Implementation for the Sites in the Jobs

Plus Demonstration

B-1

Exhibit B.1. Operational Highlights for Stronger Implementation Sites Dayton: DeSoto Bass Courts. A nonprofit agency, formerly the housing authority’s Resident Services division, administered Jobs Plus and provided stable and capable leadership and staff.

• Enduring high-level support in securing funding and program services of the housing authority and other collaborative partners, including Montgomery County’s multiservice “one-stop” job center.

• Program offered extensive outreach, short-term training, job readiness and search, intensive case management, colocated welfare caseworker, and on-line access to local one-stop’s employment database. First site to recruit residents for community support for work component (building captains).

• Rent incentives were implemented in May 2000.

Los Angeles: William Mead Homes. Jobs Plus was reconstituted in 2001 after a slow buildup due to PHA staffing gaps, leadership turnover, and equipment needs.

• Became a strong program that provided on-site job search, GED classes for Spanish speakers, and training, with intensive outreach and case management as well as on-site welfare-to-work caseworker and job developer from other public agencies.

• Rent incentives were implemented in June 2000, active promotion by PHA staff.

• Strong community support for work component. Beginning in November 2000, residents were hired and trained as community coaches to help Jobs Plus publicize activities and job opportunities and recruit participants; played leading role in bringing basic education classes on-site.

St. Paul: Mt. Airy Homes. Had consistent, professional staffing and colocation of staff from partner agencies.

• Offered on-site job counseling, job clubs, and case management, some customized short-term training classes, U.S. citizenship classes, and ESL and GED instruction. Hmong Women’s Support Group assisted with mental health and cultural issues. Head Start and after-school and summer programs were available for children and youth. Education and training were offered through referrals to local schools and agencies.

• Had to address special language- and immigrant-related barriers to outreach, service delivery, and employment.

• First site to implement rent incentives, beginning in November 1998, using PHA funds. Strong management office support in recruiting, orienting, and enrolling households for Jobs Plus as well as administering rent incentives.

Notes: ESL refers to English as a Second Language. GED refers to General Educational Development.

Source: Bloom, Riccio, and Verma (2005).

B-2

Exhibit B.2. Operational Highlights for Weaker Implementation Sites

Baltimore: Gilmor Homes

• Made a promising start with a full complement of staff and good relationships with an extensive network of local service agencies. Distinctive in its close coordination with a special on-site health office early on.

• Rent incentives were available in November 2000 but poorly administered by the housing authority.

• Jobs Plus had peaked by the end of 2000. Reductions in funding following the expiration of key grants starting in 2001 resulted in a steady loss of staff, including case managers, colocated welfare caseworkers, and job developers. Retrenchment of workforce services in local Sandtown-Winchester neighborhood eroded the capacity of referral network to offer employment services to Jobs Plus participants. On-site health office closed in 2002.

Chattanooga: Harriet Tubman Homes

• Until 2000, Jobs Plus foundered. The program was partially staffed by residents unprepared for their roles and given inadequate oversight by senior housing officials. Rent incentives were implemented in November 2000.

• Program was reconstituted between June 2000 and June 2002 with improved staffing, employment counseling, service referrals, and management. Overall, however, progress in strengthening program quality remained limited. Community support for the work component was never fully implemented.

• PHA gave low priority to Jobs Plus after a change in housing authority leadership. Its focus on privatizing its property management and resident services operations made it unlikely that the agency could oversee Jobs Plus adequately.

• In the summer of 2002, the PHA and national demonstration partners agreed to continue a scaled down, financial-incentives-only version of the program (although provision of some on-site services continued informally).

Seattle: Rainier Vista Garden Community. The program had strong and stable staff. HOPE VI funding led to the site’s formal withdrawal from the national demonstration at the end of 1999. Jobs Plus continued to operate with an expanded mission that included helping residents deal with issues related to relocation.

• Resident relocation under way from 2000 to mid-2002, part of the first stage of redevelopment.

• Rent incentives were implemented in September 1999. Enrollment in this component closed in April 2001 with phasing in of HOPE VI.

• With declining numbers of residents at Rainier Vista, the intensity of services on-site declined as staff were assigned additional responsibilities. Few services were provided to residents once they relocated out of public housing.

Source: Bloom, Riccio, and Verma (2005)

B-3

Exhibit B.3. The Original Jobs Plus Demonstration Implementation Timeline Source: Bloom, Riccio, and Verma (2005).

B-4

APPENDIX C: Supplemental Impact

Exhibits

C-1

Exhibit C.1. Impacts on Employment Stability and Earnings Distribution in Years 20

and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, All Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Number of quarters employed

4.0

4.1

0.0

0.713

Length of longest employment spell

3.9

3.9

– 0.1

0.567

Ever worked 4 consecutive quarters (%)

49.9

50.7

– 0.9

0.587

Highest earnings in Year 20 (%)

Earned above $7,500/year

47.2

48.5

–1.3

0.399

Earned above $10,000/year

44.5

45.7

–1.3

0.420

Earned above $15,000/year

38.1

41.1

-3.0 *

0.050

Highest earnings in Year 21 (%)

Earned above $7,500/year

48.6

47.7

0.9

0.565

Earned above $10,000/year

46.3

45.3

1.0

0.525

Earned above $15,000/year

40.1

41.2

– 1.1

0.472

Sample size (total = 4,105)

1,780

2,325

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires

C-2

Exhibit C.2. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Program Implementation Strength

Strong Implementers

Weak Implementers

Program Comparison

Program Comparison

Outcome

Group

Group Difference

P-Value

Group

Group Difference

P-Value

Number of quarters employed

4.3

4.0

0.3 **

0.047

3.7

4.1

– 0.4 **

0.018 †††

Length of longest employment spell

4.2

3.9

0.3 *

0.088

3.6

4.0

– 0.4 **

0.019 †††

Ever worked 4 consecutive quarters (%)

54.3

50.5

3.8 *

0.093

45.6

50.6

– 5.0 **

0.020 †††

Highest earnings in Year 20 (%)

Earned above $7,500/year

51.0

47.8

3.2

0.166

43.5

48.9

– 5.4 **

0.014 †††

Earned above $10,000/year

47.9

45.4

2.5

0.267

41.3

45.8

– 4.5 **

0.037 ††

Earned above $15,000/year

41.1

41.3

– 0.1

0.957

35.3

40.7

-5.4 **

0.012 †

Highest earnings in Year 21 (%)

Earned above $7,500/year

52.7

47.0

5.8 **

0.011

44.6

48.0

– 3.4

0.123 †††

Earned above $10,000/year

50.5

44.4

6.0 ***

0.008

42.5

45.8

– 3.3

0.130 †††

Earned above $15,000/year

42.8

40.4

2.4

0.279

37.5

41.7

– 4.2 *

0.052 ††

Sample size (total = 4,105)

914

1,073

866

1,252

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5

percent; † = 10 percent.

Stronger implementation sites include Dayton, Los Angeles, and St. Paul. Weaker implementation sites include Baltimore, Chattanooga, and Seattle.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-3

Exhibit C.3. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Site, Stronger Implementation Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Dayton

Total Earnings ($)

Year 20

13,157

14,491

– 1,334

0.235 ††

Year 21

14,065

14,845

-780

0.504 ††

Employed at least One Quarter (%)

Year 20

59.5

61.8

– 2.3

0.482

Year 21

60.5

61.3

-0.8

0.817

Average Quarterly Employment (%)

Year 20

53.0

56.9

– 3.9

0.224 ††

Year 21

54.2

55.1

– 0.9

0.781

Sample size (total = 874)

308

566

Los Angeles

Total Earnings ($)

Year 20

18,549

15,155

3,394 *

0.072 ††

Year 21

18,834

14,339

4,495 **

0.018 ††

Employed at least One Quarter (%)

Year 20

58.3

50.8

7.5 *

0.087

Year 21

57.4

49.3

8.2 *

0.066

Average Quarterly Employment (%)

Year 20

53.5

45.0

8.4 **

0.044 ††

Year 21

52.1

43.4

8.7 **

0.039

Sample size (total = 615)

326

289

St. Paul

Total Earnings ($)

Year 20

19,575

16,612

2,963

0.165 ††

Year 21

20,236

17,473

2,763

0.219 ††

Employed at least One Quarter (%)

Year 20

57.9

52.3

5.6

0.251

Year 21

56.4

52.3

4.2

0.388

Average Quarterly Employment (%)

Year 20

54.0

47.6

6.4

0.177 ††

Year 21

53.9

47.7

6.1

0.194

Sample size (total = 498)

280

218

Note: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

C-4

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Stronger implementation sites include Dayton, Los Angeles, and St. Paul.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-5

Exhibit C.4. Impacts on Employment Stability and Earnings Distribution in Years 20

and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Site, Stronger Implementation Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Dayton

Number of quarters employed

4.3

4.5

– 0.2

0.458 †

Length of longest employment spell

4.1

4.3

– 0.2

0.316 †

Ever worked 4 consecutive quarters (%)

53.0

57.8

– 4.9

0.147 ††

Highest earnings in Year 20 (%)

Earned above $7,500/year

48.2

52.7

– 4.5

0.190 †

Earned above $10,000/year

44.9

48.8

– 3.9

0.260

Earned above $15,000/year

36.1

42.1

– 6.0 *

0.079

Highest earnings in Year 21 (%)

Earned above $7,500/year

51.4

52.1

– 0.6

0.853

Earned above $10,000/year

48.7

47.9

0.8

0.820

Earned above $15,000/year

40.1

43.5

– 3.4

0.327

Sample size (total = 874)

308

566

Los Angeles

Number of quarters employed

4.2

3.5

0.7 **

0.040 †

Length of longest employment spell

4.0

3.4

0.6 *

0.062 †

Ever worked 4 consecutive quarters (%)

51.9

42.1

9.8 **

0.028 ††

Highest earnings in Year 20 (%)

Earned above $7,500/year

50.0

43.1

6.9

0.120 †

Earned above $10,000/year

47.2

42.2

5.0

0.254

Earned above $15,000/year

41.6

39.4

2.2

0.612

Highest earnings in Year 21 (%)

Earned above $7,500/year

50.4

41.6

8.8 **

0.047

Earned above $10,000/year

48.5

41.1

7.4 *

0.096

Earned above $15,000/year

41.1

38.0

3.2

0.465

Sample size (total = 615)

326

289

St. Paul

Number of quarters employed

4.3

3.8

0.5

0.173 †

Length of longest employment spell

4.2

3.7

0.5

0.196 †

Ever worked 4 consecutive quarters (%)

54.0

48.1

5.9

0.226 ††

(continued)

C-6

Exhibit C.4 (continued)

Program Comparison

Outcome

Group

Group Difference

P-Value

Highest earnings in Year 20 (%)

Earned above $7,500/year

51.7

45.8

5.9

0.237 †

Earned above $10,000/year

49.0

44.5

4.5

0.367

Earned above $15,000/year

45.5

42.5

3.0

0.542

Highest earnings in Year 21 (%)

Earned above $7,500/year

53.4

45.0

8.5 *

0.088

Earned above $10,000/year

51.9

43.2

8.7 *

0.079

Earned above $15,000/year

45.5

38.2

7.3

0.138

Sample size (total = 498)

280

218

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Standard errors are adjusted to account for the clustering of outcomes for adults from the same household.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-7

Exhibit C.5. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Site, Weaker Implementation Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Baltimore

Total Earnings ($)

Year 20

14,228

14,223

5

0.997

Year 21

14,409

14,849

– 440

0.763

Employed at least One Quarter (%)

Year 20

46.6

50.2

– 3.6

0.312

Year 21

46.7

50.9

– 4.2

0.231

Average Quarterly Employment (%)

Year 20

43.6

45.1

– 1.5

0.656

Year 21

43.7

45.9

– 2.2

0.511

Sample size (total = 792)

276

516

Chattanooga

Total Earnings ($)

Year 20

10,762

13,842

– 3,080 ***

0.006

Year 21

12,186

14,669

– 2,483 **

0.042

Employed at least One Quarter (%)

Year 20

57.8

63.5

– 5.7

0.101

Year 21

56.5

61.9

– 5.3

0.129

Average Quarterly Employment (%)

Year 20

51.3

57.8

– 6.5 *

0.056

Year 21

51.1

56.5

– 5.4

0.114

Sample size (total = 728)

269

459

Seattle

Total Earnings ($)

Year 20

16,774

20,594

– 3,820 *

0.051

Year 21

17,344

19,688

– 2,344

0.243

Employed at least One Quarter (%)

Year 20

50.8

55.9

– 5.2

0.233

Year 21

49.8

54.9

– 5.1

0.240

Average Quarterly Employment (%)

Year 20

45.8

53.2

– 7.4 *

0.079

Year 21

45.1

49.9

– 4.9

0.239

Sample size (total = 598)

321

277

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

C-8

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-9

Exhibit C.6. Impacts on Employment Stability and Earnings Distribution in Years 20

and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Site, Weaker Implementation Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Baltimore

Number of quarters employed

3.5

3.6

– 0.2

0.510

Length of longest employment spell

3.4

3.5

– 0.1

0.623

Ever worked 4 consecutive quarters (%)

43.4

44.1

– 0.7

0.843

Highest earnings in Year 20 (%)

Earned above $7,500/year

42.1

43.7

– 1.6

0.649

Earned above $10,000/year

40.7

41.1

– 0.4

0.916

Earned above $15,000/year

36.3

35.1

1.2

0.724 ††

Highest earnings in Year 21 (%)

Earned above $7,500/year

41.4

43.9

– 2.5

0.481

Earned above $10,000/year

41.1

42.9

– 1.8

0.610

Earned above $15,000/year

37.7

38.4

– 0.7

0.849

Sample size (total = 792)

276

516

Chattanooga

Number of quarters employed

4.1

4.6

– 0.5 *

0.077

Length of longest employment spell

3.9

4.4

– 0.5 **

0.048

Ever worked 4 consecutive quarters (%)

49.7

56.0

– 6.3 *

0.085

Highest earnings in Year 20 (%)

Earned above $7,500/year

45.1

52.3

– 7.2 *

0.056

Earned above $10,000/year

41.0

47.7

– 6.7 *

0.075

Earned above $15,000/year

30.9

41.6

– 10.7 ***

0.003 ††

Highest earnings in Year 21 (%)

Earned above $7,500/year

46.9

51.2

– 4.3

0.251

Earned above $10,000/year

42.9

47.8

–5.0

0.186

Earned above $15,000/year

36.2

43.7

– 7.4 **

0.042

Sample size (total = 728)

269

459

Seattle

Number of quarters employed

3.6

4.1

– 0.5

0.157

Length of longest employment spell

3.5

4.0

– 0.5

0.164

Ever worked 4 consecutive quarters (%)

44.7

52.8

– 8.1 *

0.065

(continued)

C-10

Exhibit C.6 (continued)

Program Comparison

Outcome

Group

Group Difference

P-Value

Highest earnings in Year 20 (%)

Earned above $7,500/year

44.3

51.8

– 7.5 *

0.088

Earned above $10,000/year

43.0

50.2

– 7.2

0.102

Earned above $15,000/year

40.6

47.4

– 6.9

0.115 ††

Highest earnings in Year 21 (%)

Earned above $7,500/year

46.3

49.1

– 2.8

0.513

Earned above $10,000/year

44.3

46.4

– 2.1

0.622

Earned above $15,000/year

39.3

43.5

– 4.2

0.331

Sample size (total = 598)

321

277

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Sources: MDRC calculations using quarterly wage data from the National Directory of New Hires C-11

Exhibit C.7. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Age at Baseline, Stronger Implementation Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Age 18–24

Total Earnings ($)

Year 20

22,416

21,659

757

0.683

Year 21

23,033

21,163

1,870

0.307

Employed at least One Quarter (%)

Year 20

72.5

71.9

0.6

0.886

Year 21

73.9

71.1

2.9

0.458

Average Quarterly Employment (%)

Year 20

65.5

66.7

– 1.2

0.761

Year 21

66.7

64.4

2.3

0.543

Sample size (total = 629)

285

344

Age 25–34

Total Earnings ($)

Year 20

17,061

15,477

1,584

0.242

Year 21

17,809

16,331

1,479

0.312

Employed at least One Quarter (%)

Year 20

64.8

61.5

3.3

0.353

Year 21

63.7

61.3

2.4

0.507

Average Quarterly Employment (%)

Year 20

60.0

55.1

4.9

0.157

Year 21

59.1

54.6

4.5

0.197

Sample size (total = 764)

362

402

Age 35–44

Total Earnings ($)

Year 20

10,388

8,700

1,688

0.226

Year 21

10,920

8,731

2,189

0.133

Employed at least One Quarter (%)

Year 20

38.9

33.5

5.4

0.208

Year 21

38.3

32.6

5.7

0.170

Average Quarterly Employment (%)

Year 20

35.0

30.5

4.5

0.260

Year 21

35.0

29.1

5.9

0.137

Sample size (total = 594)

267

327

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

C-12

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Stronger implementation sites include Dayton, Los Angeles, and St. Paul.

The ages referenced in this exhibit indicate the age of adults at baseline in 1998. In Year 20 of followup, the corresponding ages of these adults are 38–44, 45–54, and 55–64.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-13

Exhibit C.8. Impacts on Employment Stability and Earnings Distribution in Years 20

and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Age at Baseline, Stronger Implementer Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Age 18–24

Number of quarters employed

5.3

5.2

0.0

0.939

Length of longest employment spell

5.0

5.1

0.0

0.884

Ever worked 4 consecutive quarters (%)

65.5

66.8

– 1.2

0.764

Highest earnings in Year 20 (%)

Earned above $7,500/year

61.0

62.6

– 1.6

0.699

Earned above $10,000/year

58.5

59.9

– 1.4

0.742

Earned above $15,000/year

51.4

55.7

– 4.2

0.319

Highest earnings in Year 21 (%)

Earned above $7,500/year

65.9

62.1

3.8

0.359

Earned above $10,000/year

61.9

59.3

2.6

0.530

Earned above $15,000/year

53.7

53.7

0.1

0.987

Sample size (total = 629)

285

344

Age 25–34

Number of quarters employed

4.8

4.4

0.4

0.163

Length of longest employment spell

4.6

4.2

0.3

0.223

Ever worked 4 consecutive quarters (%)

59.4

54.8

4.6

0.207

Highest earnings in Year 20 (%)

Earned above $7,500/year

55.4

51.8

3.6

0.338

Earned above $10,000/year

51.6

48.2

3.4

0.364

Earned above $15,000/year

45.0

42.8

2.2

0.556

Highest earnings in Year 21 (%)

Earned above $7,500/year

57.1

50.8

6.2 *

0.094

Earned above $10,000/year

54.5

47.4

7.1 *

0.060

Earned above $15,000/year

46.9

43.4

3.6

0.338

Sample size (total = 764)

362

402

Age 35–44

Number of quarters employed

2.8

2.4

0.4

0.163

Length of longest employment spell

2.7

2.3

0.4

0.189

Ever worked 4 consecutive quarters (%)

35.3

29.5

5.8

0.157

(continued)

C-14

Exhibit C.8 (continued)

Program Comparison

Outcome

Group

Group Difference

P-Value

Highest earnings in Year 20 (%)

Earned above $7,500/year

34.1

28.2

5.9

0.147

Earned above $10,000/year

31.6

27.4

4.2

0.295

Earned above $15,000/year

24.9

24.9

0.0

0.994

Highest earnings in Year 21 (%)

Earned above $7,500/year

33.0

27.4

5.6

0.161

Earned above $10,000/year

32.8

26.4

6.4

0.105

Earned above $15,000/year

25.8

23.8

1.9

0.606

Sample size (total = 594)

267

327

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Stronger implementation sites include Dayton, Los Angeles, and St. Paul.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-15

Exhibit C.9. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Age at Baseline, Weaker Implementation Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Age 18–24

Total Earnings ($)

Year 20

18,353

21,541

– 3,188 *

0.081

Year 21

19,475

22,702

– 3,227 *

0.089

Employed at least One Quarter (%)

Year 20

67.7

77.7

– 10.0 ***

0.008

Year 21

66.9

77.5

-10.6 ***

0.006

Average Quarterly Employment (%)

Year 20

61.1

70.3

– 9.2 **

0.015

Year 21

60.1

71.0

– 10.9 ***

0.005

Sample size (total = 588)

230

358

Age 25–34

Total Earnings ($)

Year 20

14,229

16,959

– 2,730 **

0.048

Year 21

15,062

17,139

– 2,076

0.145

Employed at least One Quarter (%)

Year 20

55.2

59.7

– 4.5

0.202

Year 21

56.6

59.8

– 3.2

0.368

Average Quarterly Employment (%)

Year 20

50.3

55.5

– 5.2

0.132

Year 21

51.9

53.8

– 2.0

0.564

Sample size (total = 849)

355

494

Age 35–44

Total Earnings ($)

Year 20

9,119

9,345

– 226

0.857

Year 21

9,578

9,076

502

0.698

Employed at least One Quarter (%)

Year 20

33.6

34.8

– 1.2

0.749

Year 21

30.5

33.0

– 2.5

0.498

Average Quarterly Employment (%)

Year 20

30.4

31.4

– 1.0

0.769

Year 21

28.7

30.0

– 1.3

0.716

Sample size (total = 681)

281

400

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

C-16

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Weaker implementation sites include Baltimore, Chattanooga, and Seattle.

The ages referenced in this exhibit indicate the age of adults at baseline in 1998. In Year 20 of followup, the corresponding ages of these adults are 38–44, 45-54, and 55–64.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-17

Exhibit C.10. Impacts on Employment Stability and Earnings Distribution in Years 20

and 21 of Followup, Adults from the 1998 Cohort Who Are in the Long-Term Impact Study, by Age at Baseline, Weaker Implementer Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Age 18–24

Number of quarters employed

4.8

5.6

– 0.8 ***

0.006

Length of longest employment spell

4.6

5.5

– 0.8 ***

0.006

Ever worked 4 consecutive quarters (%)

58.2

70.0

– 11.9 ***

0.005

Highest earnings in Year 20 (%)

Earned above $7,500/year

54.4

65.2

– 10.9 **

0.012

Earned above $10,000/year

53.4

59.7

-6.4

0.150

Earned above $15,000/year

45.4

52.5

– 7.1

0.109

Highest earnings in Year 21 (%)

Earned above $7,500/year

57.6

65.0

– 7.4 *

0.083

Earned above $10,000/year

53.1

61.7

– 8.6 **

0.049

Earned above $15,000/year

48.5

56.5

– 8.1 *

0.069

Sample size (total = 588)

230

358

Age 25–34

Number of quarters employed

4.1

4.4

– 0.3

0.292

Length of longest employment spell

4.0

4.3

– 0.3

0.268

Ever worked 4 consecutive quarters (%)

51.1

54.0

– 2.8

0.433

Highest earnings in Year 20 (%)

Earned above $7,500/year

47.3

53.0

– 5.7

0.116

Earned above $10,000/year

44.7

51.1

– 6.5 *

0.072

Earned above $15,000/year

38.2

45.7

– 7.4 **

0.037

Highest earnings in Year 21 (%)

Earned above $7,500/year

49.8

51.5

– 1.7

0.628

Earned above $10,000/year

48.7

49.0

– 0.4

0.924

Earned above $15,000/year

41.5

44.8

– 3.3

0.366

Sample size (total = 849)

355

494

Age 35–44

Number of quarters employed

2.4

2.4

– 0.1

0.735

Length of longest employment spell

2.3

2.4

– 0.1

0.824

Ever worked 4 consecutive quarters (%)

28.4

30.2

– 1.8

0.622

(continued)

C-18

Exhibit C.10 (continued)

Program Comparison

Outcome

Group

Group Difference

P-Value

Highest earnings in Year 20 (%)

Earned above $7,500/year

30.0

30.0

0.0

0.997

Earned above $10,000/year

27.3

27.4

– 0.1

0.978

Earned above $15,000/year

24.0

24.4

-0.4

0.896

Highest earnings in Year 21 (%)

Earned above $7,500/year

27.8

29.1

– 1.4

0.703

Earned above $10,000/year

26.4

28.0

– 1.6

0.650

Earned above $15,000/year

24.1

24.9

– 0.8

0.817

Sample size (total = 681)

281

400

Notes: The sample for this analysis includes all adults living in a Jobs Plus or comparison development who were between the ages of 18 and 44 years in October 1998 and were not listed as disabled by their public housing agency.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Weaker implementation sites include Baltimore, Chattanooga, and Seattle.

P-values are based on standard errors that are adjusted to account for adults from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires

C-19

Exhibit C.11. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup, Children in Households from the 1998 Cohort, All Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Total Earnings ($)

Year 20

18,394

16,962

1,432 ***

0.003

Year 21

20,032

18,496

1,537 ***

0.002

Employed at least One Quarter (%)

Year 20

78.9

76.4

2.6 **

0.019

Year 21

78.7

75.5

3.2 ***

0.005

Average Quarterly Employment (%)

Year 20

69.6

66.1

3.5 ***

0.002

Year 21

69.8

66.0

3.9 ***

0.001

Sample size (total = 6,337)

2,902

3,435

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-20

Exhibit C.12. Impacts on Employment Stability and Earnings Distribution in Years 20

and 21 of Followup, Children in Households from the 1998 Cohort, All Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Number of quarters employed

5.6

5.3

0.3 ***

0.001

Length of longest employment spell

5.3

5.0

0.3 ***

0.000

Ever worked 4 consecutive quarters (%)

69.5

64.9

4.6 ***

0.000

Highest earnings in Year 20 (%)

Earned above $7,500/year

60.4

57.1

3.3 **

0.011

Earned above $10,000/year

55.8

52.3

3.5 ***

0.007

Earned above $15,000/year

47.2

43.3

3.9 ***

0.003

Highest earnings in Year 21 (%)

Earned above $7,500/year

62.3

59.3

3.1 **

0.019

Earned above $10,000/year

58.0

55.4

2.6 **

0.048

Earned above $15,000/year

50.1

47.7

2.4 *

0.071

Sample size (total = 6,337)

2,902

3,435

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-21

Exhibit C.13. Impacts on Earnings and Employment in Years 20 and 21 of Followup, Children in Households from the 1998 Cohort, by Implementation Strength

Stronger Implementation Sites

Weaker Implementation Sites

Program Comparison

Program Comparison

Outcome

Group

Group Difference

P-Value

Group

Group Difference

P-Value

Summary Outcomes

Average Annual Earnings, Years 20 and

21 ($)

20,961

18,255

2,706 ***

0.000

16,697

16,370

327

0.615 †††

Average Quarterly Employment,

Years 20 and 21 (%)

72.8

66.6

6.2 ***

0.000

66.1

64.7

1.5

0.338 ††

Yearly Outcomes

Total Earnings ($)

Year 20

20,322

17,977

2,345 ***

0.001

16,328

15,695

634

0.339 †

Year 21

22,143

19,095

3,048 ***

0.000

17,753

17,616

137

0.847 †††

Employed at least One Quarter (%)

Year 20

81.9

76.6

5.3 ***

0.001

75.8

75.6

0.2

0.900 ††

Year 21

81.5

75.7

5.7 ***

0.000

75.9

74.9

1.1

0.521 ††

Average Quarterly Employment (%)

Year 20

72.7

67.4

5.3 ***

0.001

66.5

64.3

2.2

0.179

Year 21

73.2

66.1

7.1 ***

0.000

66.2

65.3

1.0

0.563 †††

Sample size (total = 6,337)

1,622

1,700

1,280

1,735

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5

percent; † = 10 percent.

C-22

Stronger implementation sites include Dayton, Los Angeles, and St. Paul. Weaker implementation sites include Baltimore, Chattanooga, and Seattle.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

The impact estimates for the two confirmatory outcomes (average annual earnings over Years 20 and 21 and average quarterly employment across Years 20 and 21) were each statistically significant; therefore, based on the Benjamini-Hochberg multiple hypothesis testing approach, no further adjustments to the p-values were needed.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-23

Exhibit C.14. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of Followup, Children in Households from the 1998 Cohort, by Program Implementation Strength

Strong Implementers

Weak Implementers

Program Comparison

Program Comparison

Outcome

Group

Group Difference

P-Value

Group

Group Difference

P-Value

Number of quarters employed

5.8

5.3

0.5 ***

0.000

5.3

5.2

0.1

0.338 ††

Length of longest employment spell

5.5

5.0

0.5 ***

0.000

5.0

4.8

0.1

0.241 ††

Ever worked 4 consecutive quarters (%)

72.8

65.5

7.3 ***

0.000

66.0

63.7

2.3

0.205 †

Highest earnings in Year 20 (%)

Earned above $7,500/year

64.4

59.4

5.0 ***

0.006

56.1

54.2

1.9

0.322

Earned above $10,000/year

60.0

54.7

5.2 ***

0.004

51.4

49.3

2.1

0.275

Earned above $15,000/year

51.3

46.1

5.2 ***

0.004

43.0

39.9

3.1

0.101

Highest earnings in Year 21 (%)

Earned above $7,500/year

66.2

60.6

5.6 ***

0.002

58.3

57.4

1.0

0.616 †

Earned above $10,000/year

62.4

57.5

5.0 ***

0.006

53.2

52.7

0.5

0.794 †

Earned above $15,000/year

54.6

50.1

4.5 **

0.015

45.2

44.7

0.5

0.805

Sample size (total = 6,337)

1,622

1,700

1,280

1,735

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5

percent; † = 10 percent.

Stronger implementation sites include Dayton, Los Angeles, and St. Paul. Weaker implementation sites include Baltimore, Chattanooga, and Seattle.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-24

Exhibit C.15. Yearly Impacts on Earnings and Employment in Years 20 and 21

of Followup, Children in Households from the 1998 Cohort, by Site, Stronger Implementer Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Dayton

Total Earnings ($)

Year 20

14,096

12,284

1,812 **

0.033

Year 21

15,596

13,029

2,567 ***

0.004

Employed at least One Quarter (%)

Year 20

79.9

75.2

4.6 *

0.054

Year 21

82.2

74.4

7.9 ***

0.001 ††

Average Quarterly Employment (%)

Year 20

69.2

63.4

5.8 **

0.015

Year 21

71.2

62.0

9.3 ***

0.000 †

Sample size (total = 1,379)

467

912

Los Angeles

Total Earnings ($)

Year 20

22,743

20,719

2,024

0.214

Year 21

24,682

22,324

2,358

0.166

Employed at least One Quarter (%)

Year 20

78.6

72.4

6.2 *

0.076

Year 21

79.9

71.8

8.1 **

0.021 ††

Average Quarterly Employment (%)

Year 20

71.7

65.7

6.0 *

0.086

Year 21

72.8

64.9

7.9 **

0.023 †

Sample size (total = 818)

417

401

St. Paul

Total Earnings ($)

Year 20

25,565

23,823

1,742

0.220

Year 21

27,677

25,235

2,442

0.101

Employed at least One Quarter (%)

Year 20

86.0

82.4

3.6

0.169

Year 21

81.9

82.6

– 0.7

0.794 ††

Average Quarterly Employment (%)

Year 20

77.5

75.1

2.3

0.401

Year 21

75.8

75.0

0.8

0.782 †

Sample size (total = 1,125)

738

387

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

C-25

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-26

Exhibit C.16. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of Followup, Children in Households from the 1998 Cohort, by Site, Stronger Implementer Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Dayton

Number of quarters employed

5.6

5.0

0.6 ***

0.001

Length of longest employment spell

5.2

4.6

0.6 ***

0.002

Ever worked 4 consecutive quarters (%)

69.7

61.1

8.6 ***

0.002

Highest earnings in Year 20 (%)

Earned above $7,500/year

54.0

50.1

3.9

0.183

Earned above $10,000/year

48.3

43.0

5.4 *

0.062

Earned above $15,000/year

38.0

33.9

4.1

0.141

Highest earnings in Year 21 (%)

Earned above $7,500/year

57.7

50.5

7.2 **

0.013

Earned above $10,000/year

51.7

46.0

5.7 *

0.051

Earned above $15,000/year

41.4

37.1

4.3

0.131

Sample size (total = 1,379)

467

912

Los Angeles

Number of quarters employed

5.8

5.2

0.6 **

0.036

Length of longest employment spell

5.5

5.0

0.5 **

0.043

Ever worked 4 consecutive quarters (%)

73.6

63.2

10.4 ***

0.006

Highest earnings in Year 20 (%)

Earned above $7,500/year

69.4

62.2

7.3 *

0.060

Earned above $10,000/year

66.4

60.2

6.2

0.111

Earned above $15,000/year

58.6

50.4

8.1 **

0.043

Highest earnings in Year 21 (%)

Earned above $7,500/year

70.7

64.6

6.1

0.111

Earned above $10,000/year

68.9

62.5

6.4

0.103

Earned above $15,000/year

63.0

57.3

5.7

0.159

Sample size (total = 818)

417

401

St. Paul

Number of quarters employed

6.1

6.0

0.1

0.571

Length of longest employment spell

5.9

5.8

0.1

0.575

Ever worked 4 consecutive quarters (%)

76.3

75.0

1.3

0.679

(continued)

C-27

Exhibit C.16 (continued)

Program Comparison

Outcome

Group

Group Difference

P-Value

Highest earnings in Year 20 (%)

Earned above $7,500/year

73.1

70.0

3.2

0.329

Earned above $10,000/year

69.2

67.1

2.1

0.531

Earned above $15,000/year

61.4

60.3

1.0

0.767

Highest earnings in Year 21 (%)

Earned above $7,500/year

73.5

72.6

0.8

0.796

Earned above $10,000/year

70.9

70.1

0.8

0.798

Earned above $15,000/year

64.5

62.8

1.7

0.617

Sample size (total = 1,125)

738

387

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-28

Exhibit C.17. Yearly Impacts on Earnings and Employment in Years 20 and 21

of Followup, Children in Households from the 1998 Cohort, by Site, Weaker Implementer Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Baltimore

Total Earnings ($)

Year 20

15,592

14,377

1,214

0.316

Year 21

16,989

16,746

243

0.852

Employed at least One Quarter (%)

Year 20

72.7

71.2

1.5

0.632

Year 21

72.0

72.3

– 0.2

0.940

Average Quarterly Employment (%)

Year 20

62.8

60.1

2.7

0.381

Year 21

62.2

64.0

– 1.8

0.566

Sample size (total = 967)

306

661

Chattanooga

Total Earnings ($)

Year 20

13,467

13,764

– 297

0.730

Year 21

14,956

15,495

– 539

0.570

Employed at least One Quarter (%)

Year 20

79.7

80.4

– 0.7

0.757

Year 21

80.2

78.2

2.0

0.402

Average Quarterly Employment (%)

Year 20

69.8

68.9

0.9

0.720

Year 21

69.8

68.6

1.2

0.625

Sample size (total = 1,174)

471

703

Seattle

Total Earnings ($)

Year 20

21,088

19,771

1,317

0.385

Year 21

22,498

21,525

972

0.553

Employed at least One Quarter (%)

Year 20

74.2

74.2

0.0

0.997

Year 21

73.5

73.7

– 0.2

0.959

Average Quarterly Employment (%)

Year 20

65.4

62.9

2.5

0.438

Year 21

64.4

62.2

2.2

0.501

Sample size (total = 874)

503

371

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

C-29

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-30

Exhibit C.18. Impacts on Employment Stability and Earnings Distribution in Years 20 and 21 of Followup, Children in Households from the 1998 Cohort, by Site, Weaker Implementation Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Baltimore

Number of quarters employed

5.0

5.0

0.0

0.873

Length of longest employment spell

4.7

4.7

0.1

0.804

Ever worked 4 consecutive quarters (%)

61.6

61.2

0.4

0.918

Highest earnings in Year 20 (%)

Earned above $7,500/year

54.7

50.8

3.9

0.266 †

Earned above $10,000/year

47.7

48.1

– 0.3

0.924

Earned above $15,000/year

41.7

39.2

2.5

0.462

Highest earnings in Year 21 (%)

Earned above $7,500/year

55.7

57.1

– 1.4

0.698

Earned above $10,000/year

53.0

52.2

0.7

0.837

Earned above $15,000/year

46.0

44.1

1.9

0.600

Sample size (total = 967)

306

661

Chattanooga

Number of quarters employed

5.6

5.5

0.1

0.689

Length of longest employment spell

5.2

5.1

0.1

0.660

Ever worked 4 consecutive quarters (%)

68.9

67.9

1.0

0.722

Highest earnings in Year 20 (%)

Earned above $7,500/year

52.0

55.6

– 3.6

0.228 †

Earned above $10,000/year

47.7

47.3

0.5

0.876

Earned above $15,000/year

37.3

35.8

1.4

0.619

Highest earnings in Year 21 (%)

Earned above $7,500/year

56.5

56.7

– 0.2

0.942

Earned above $10,000/year

48.0

51.1

– 3.1

0.303

Earned above $15,000/year

37.8

41.9

– 4.0

0.173

Sample size (total = 1,174)

471

703

Seattle

Number of quarters employed

5.1

5.0

0.2

0.476

Length of longest employment spell

4.9

4.7

0.2

0.347

Ever worked 4 consecutive quarters (%)

65.6

60.4

5.3

0.147

(continued)

C-31

Exhibit C.18 (continued)

Program Comparison

Outcome

Group

Group Difference

P-Value

Highest earnings in Year 20 (%)

Earned above $7,500/year

61.8

56.2

5.6

0.119 †

Earned above $10,000/year

58.7

53.7

5.0

0.167

Earned above $15,000/year

51.0

46.9

4.1

0.259

Highest earnings in Year 21 (%)

Earned above $7,500/year

62.4

58.4

4.0

0.278

Earned above $10,000/year

59.5

55.1

4.3

0.244

Earned above $15,000/year

53.5

49.8

3.7

0.326

Sample size (total = 874)

503

371

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-32

Exhibit C.19. Yearly Impacts on Earnings and Employment in Years 20 and 21

of Followup, Children in Households from the 1998 Cohort, by Age at Baseline, Stronger Implementation Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Age 0–5

Total Earnings ($)

Year 20

14,471

13,461

1,010

0.250

Year 21

17,305

15,206

2,099 **

0.030

Employed at least One Quarter (%)

Year 20

86.6

80.8

5.8 **

0.016

Year 21

86.5

78.9

7.6 ***

0.003

Average Quarterly Employment (%)

Year 20

74.1

69.7

4.4 *

0.081

Year 21

76.2

67.8

8.4 ***

0.001

Sample size (total = 1,050)

482

568

Age 6–12

Total Earnings ($)

Year 20

21,732

19,020

2,712 ***

0.009

Year 21

23,482

19,918

3,564 ***

0.001

Employed at least One Quarter (%)

Year 20

81.7

77.4

4.2 *

0.060

Year 21

80.7

76.5

4.1 *

0.071

Average Quarterly Employment (%)

Year 20

72.9

68.3

4.5 **

0.046

Year 21

73.1

66.9

6.3 ***

0.006

Sample size (total = 1,566)

770

796

Age 13–17

Total Earnings ($)

Year 20

25,307

23,280

2,028

0.290

Year 21

26,088

23,764

2,325

0.239

Employed at least One Quarter (%)

Year 20

74.3

69.2

5.1

0.193

Year 21

74.6

69.9

4.7

0.233

Average Quarterly Employment (%)

Year 20

69.1

62.7

6.4 *

0.094

Year 21

68.0

62.7

5.2

0.175

Sample size (total = 706)

370

336

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

C-33

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Stronger implementation sites include Dayton, Los Angeles, and St. Paul.

The ages referenced in this exhibit indicate the age of children at baseline in 1998. In Year 20 of followup, the corresponding ages of these children are 20–25, 26–32, and 33–37.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-34

Exhibit C.20. Impacts Employment Stability and Earnings Distribution in Years 20 and 21 of Followup, Children in Households from the 1998 Cohort, by Age at Baseline, Stronger Implementation Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Age 0–5

Number of quarters employed

6.0

5.5

0.5 ***

0.006

Length of longest employment spell

5.6

5.2

0.5 **

0.020

Ever worked 4 consecutive quarters (%)

75.3

67.0

8.2 ***

0.005

Highest earnings in Year 20 (%)

Earned above $7,500/year

58.2

56.2

2.0

0.539

Earned above $10,000/year

52.1

48.8

3.3

0.306

Earned above $15,000/year

41.5

36.6

4.9

0.122

Highest earnings in Year 21 (%)

Earned above $7,500/year

64.6

58.8

5.8 *

0.065

Earned above $10,000/year

58.6

54.2

4.4

0.166

Earned above $15,000/year

46.6

43.4

3.2

0.318

Sample size (total = 1,050)

482

568

Age 6–12

Number of quarters employed

5.8

5.4

0.4 **

0.012

Length of longest employment spell

5.6

5.1

0.5 ***

0.006

Ever worked 4 consecutive quarters (%)

72.8

66.7

6.1 **

0.017

Highest earnings in Year 20 (%)

Earned above $7,500/year

67.6

62.2

5.3 **

0.042

Earned above $10,000/year

63.2

58.0

5.2 *

0.050

Earned above $15,000/year

54.3

50.4

3.8

0.159

Highest earnings in Year 21 (%)

Earned above $7,500/year

67.0

61.7

5.3 **

0.045

Earned above $10,000/year

63.7

59.6

4.2

0.115

Earned above $15,000/year

57.5

53.2

4.3

0.111

Sample size (total = 1,566)

770

796

Age 13–17

Number of quarters employed

5.4

5.0

0.5

0.118

Length of longest employment spell

5.3

4.8

0.5

0.106

Ever worked 4 consecutive quarters (%)

68.2

61.6

6.6

0.114

(continued)

C-35

Exhibit C.20 (continued)

Program Comparison

Outcome

Group

Group Difference

P-Value

Highest earnings in Year 20 (%)

Earned above $7,500/year

65.6

59.0

6.6

0.111

Earned above $10,000/year

63.3

57.6

5.7

0.172

Earned above $15,000/year

58.5

52.0

6.5

0.127

Highest earnings in Year 21 (%)

Earned above $7,500/year

66.5

61.2

5.3

0.205

Earned above $10,000/year

64.8

58.3

6.5

0.124

Earned above $15,000/year

59.8

54.1

5.6

0.191

Sample size (total = 706)

370

336

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Stronger implementation sites include Dayton, Los Angeles, and St. Paul.

The ages referenced in this exhibit indicate the age of children at baseline in 1998. In Year 20 of followup, the corresponding ages of these children are 20–25, 26–32, and 33–37.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-36

Exhibit C.21. Yearly Impacts on Earnings and Employment in Years 20 and 21 of Followup, Children in Households from the 1998 Cohort, by Age at Baseline, Stronger Implementation Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Age 6–8

Total Earnings ($)

Year 20

19,677

17,177

2,500 *

0.057

Year 21

21,587

18,156

3,431 **

0.013

Employed at least One Quarter (%)

Year 20

82.4

79.0

3.4

0.271

Year 21

82.0

77.4

4.6

0.148

Average Quarterly Employment (%)

Year 20

73.8

69.6

4.2

0.182

Year 21

74.7

67.8

6.9 **

0.029

Sample size (total = 775)

363

412

Age 9–12

Total Earnings ($)

Year 20

23,607

21,021

2,586

0.104

Year 21

25,191

21,873

3,317 **

0.043

Employed at least One Quarter (%)

Year 20

81.2

75.5

5.7 *

0.080

Year 21

79.6

75.4

4.2

0.207

Average Quarterly Employment (%)

Year 20

72.2

66.8

5.4

0.100

Year 21

71.8

65.6

6.2 *

0.064

Sample size (total = 791)

407

384

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Stronger implementation sites include Dayton, Los Angeles, and St. Paul.

The ages referenced in this exhibit indicate the age of children at baseline in 1998. In Year 20 of followup, the corresponding ages of these children are 26–32, and 33–37.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

A test for significant differences across subgroups was done. Like the main results that combine these two age categories, there was no statistically significant difference in the effects across the four age groups.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-37

Exhibit C.22. Yearly Impacts on Earnings and Employment in Years 20 and 21

of Followup, Children in Households from the 1998 Cohort, by Age at Baseline, Weaker Implementation Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Age 0–5

Total Earnings ($)

Year 20

12,409

11,529

880

0.305

Year 21

15,127

13,635

1,492

0.123

Employed at least One Quarter (%)

Year 20

77.1

79.7

– 2.6

0.322

Year 21

77.6

79.9

– 2.3

0.379

Average Quarterly Employment (%)

Year 20

66.3

64.7

1.5

0.559

Year 21

67.7

67.3

0.4

0.889

Sample size (total = 1,051)

443

608

Age 6–12

Total Earnings ($)

Year 20

17,857

17,157

699

0.493

Year 21

18,872

19,236

– 364

0.744

Employed at least One Quarter (%)

Year 20

75.8

74.8

1.0

0.680

Year 21

76.0

72.8

3.1

0.202

Average Quarterly Employment (%)

Year 20

67.3

64.7

2.6

0.261

Year 21

65.8

64.9

0.9

0.725

Sample size (total = 1,404)

601

803

Age 13–17

Total Earnings ($)

Year 20

19,849

20,151

-302

0.874

Year 21

20,023

21,231

– 1,207

0.535

Employed at least One Quarter (%)

Year 20

73.3

69.8

3.6

0.383

Year 21

72.6

70.0

2.6

0.532

Average Quarterly Employment (%)

Year 20

64.7

62.5

2.3

0.573

Year 21

64.8

62.3

2.5

0.542

Sample size (total = 560)

236

324

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

C-38

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Weaker implementation sites include Baltimore, Chattanooga, and Seattle.

The ages referenced in this exhibit indicate the age of children at baseline in 1998. In Year 20 of followup, the corresponding ages of these children are 20–25, 26–32, and 33–37.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-39

Exhibit C.23. Impacts Employment Stability and Earnings Distribution in Years 20 and 21 of Followup, Children in Households from the 1998 Cohort, by Age at Baseline, Weaker Implementer Sites

Program Comparison

Outcome

Group

Group Difference

P-Value

Age 0–5

Number of quarters employed

5.4

5.3

0.1

0.685

Length of longest employment spell

5.0

4.9

0.2

0.397

Ever worked 4 consecutive quarters (%)

67.1

62.8

4.3

0.163

Highest earnings in Year 20 (%)

Earned above $7,500/year

51.4

48.7

2.7

0.410

Earned above $10,000/year

44.6

41.8

2.9

0.380

Earned above $15,000/year

33.8

30.4

3.4

0.261

Highest earnings in Year 21 (%)

Earned above $7,500/year

57.0

54.9

2.1

0.519

Earned above $10,000/year

51.2

47.6

3.6

0.273

Earned above $15,000/year

41.9

37.8

4.1

0.200

Sample size (total = 1,051)

443

608

Age 6–12

Number of quarters employed

5.3

5.2

0.1

0.473

Length of longest employment spell

5.0

4.9

0.1

0.539

Ever worked 4 consecutive quarters (%)

65.5

65.2

0.3

0.915

Highest earnings in Year 20 (%)

Earned above $7,500/year

57.7

57.5

0.1

0.958

Earned above $10,000/year

53.7

53.0

0.7

0.805

Earned above $15,000/year

47.2

43.6

3.5

0.200

Highest earnings in Year 21 (%)

Earned above $7,500/year

57.8

58.5

– 0.7

0.797

Earned above $10,000/year

53.6

55.1

– 1.4

0.613

Earned above $15,000/year

46.9

47.9

– 1.0

0.730

Sample size (total = 1,404)

601

803

Age 13–17

Number of quarters employed

5.2

5.0

0.2

0.578

Length of longest employment spell

4.9

4.7

0.2

0.527

Ever worked 4 consecutive quarters (%)

66.0

61.4

4.6

0.296

(continued)

C-40

Exhibit C.23 (continued)

Program Comparison

Outcome

Group

Group Difference

P-Value

Highest earnings in Year 20 (%)

Earned above $7,500/year

60.7

56.5

4.1

0.359

Earned above $10,000/year

58.1

54.9

3.2

0.479

Earned above $15,000/year

49.8

49.0

0.8

0.863

Highest earnings in Year 21 (%)

Earned above $7,500/year

62.4

59.3

3.1

0.492

Earned above $10,000/year

55.9

56.3

– 0.4

0.938

Earned above $15,000/year

47.1

50.3

– 3.2

0.491

Sample size (total = 560)

236

324

Notes: The sample for this analysis includes all children living in a Jobs Plus or comparison development who were between the ages of 0 and 17 in October 1998.

Year 20 of followup aligns with calendar months July 2017 to June 2018. Year 21 of followup aligns with calendar months July 2018 to June 2019.

Rounding may cause slight discrepancies in sums and differences.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Weaker implementation sites include Baltimore, Chattanooga, and Seattle.

The ages referenced in this exhibit indicate the age of children at baseline in 1998. In Year 20 of followup, the corresponding ages of these children are 20–25, 26–32, and 33–37.

P-values are based on standard errors that are adjusted to account for children from the same household using Huber-White standard errors.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires C-41

APPENDIX D: Estimating the Gain in

Lifetime Earnings for Children of Adult

Participants

D-1

The positive effects on earnings for adults who were children during Jobs Plus are encouraging and raise questions related to benefit-cost analyses. Do the gains in lifetime earnings for these children outweigh the costs of the program? An informal analysis suggests that the gain in lifetime earnings per individual is somewhere between $19,000 and $79,000. While the exact number is uncertain, the range suggests that the gain is positive.

Although a benefit-cost analysis is beyond the scope of this report and was also not conducted as part of the original Jobs Plus study, this appendix presents a brief analysis of the potential gains in lifetime earnings for children in the strong implementation sites, based on the impacts observed during Years 20 and 21. For all children, Jobs Plus led to an increase in earnings of 14.8 percent over the control group average of about $19,000. Over these 2 years, the individuals ranged in age from 20 to 38.

Several assumptions are necessary to produce an estimate of the net present value of the gains in earnings over individuals’ working lives. One assumption, for example, is about the trajectory of earnings levels over time for the control group. It is assumed, based on literature in economics, that earnings increase with experience (at about 8 percent per year), but at a decreasing rate, following the typical inverted U-shaped pattern over the working life. 54 In this case, earnings start at about $11,000

at the age of 20, peak at just over $27,000 at age 36, and fall to $25,500 by age 65. Note that this assumption results in earnings levels that match those presented in exhibit 20, showing control group earnings by child age group. Another assumption required to present the stream of future gains in today’s dollars (or the net present value) is the interest rate, which is assumed to be 3 percent.

The most important assumption, however, is whether the impacts on earnings remain constant over time or whether they decay. In this case, an assumption must also be made about whether the observed impact on earnings of 14.8 percent appeared at the start of their working lives.

At one extreme, assuming a constant percentage impact on earnings from ages 20 to 65 produces a net present value gain in lifetime earnings of just under $79,000. This is likely an upper-bound estimate, given that sustained earnings gains from an intervention tend to be the exception rather than the norm. At the other extreme, assuming that the impact on earnings is 14.8 percent through age 30, but zero after that, produces a net present value gain of about $19,000. The actual value is probably somewhere between these two estimates.

The original study did not produce a formal estimate of programs costs but suggested that the costs per family served might run between $1,000 to $1,800 per year, or $5,000 to $9,000 over the 5-year program period. A formal benefit-cost analysis would need to compare the costs of the program with its benefits on participating adults and their children.

54 Polachek, S.W. 2007. “Earnings Over the Lifecycle: The Mincer Earnings Function and Its Applications,” IZA Discussion papers No. 3181, Institute for the Study of Labor, Bonn.

D-2

APPENDIX E: Using Permutation Tests to

Assess the Statistical Significance of Long-

Term Jobs Plus Impact Estimates

E-1

The present appendix explains:

1. The problem with using an ordinary least squares (OLS) standard error to assess the statistical significance of a long-term Jobs Plus impact estimate.

2. The strengths and limitations of using a permutation test for this purpose.

3. The mechanics of such a permutation test.

4. How to interpret the results of this test.

The Problem with Ordinary Least Squares Standard Errors for the Present Analysis As described in the main report, the present analysis estimates Jobs Plus impacts on long-term mean earnings levels and employment rates for the three strong-implementation Jobs Plus sites combined and the three weak-implementation Jobs Plus sites combined using an Ordinary Least Squares (OLS) regression of the following form:

𝑌𝑌

3

𝑀𝑀

𝑖𝑖 = ∑𝑘𝑘=1 𝛼𝛼𝑘𝑘𝑆𝑆𝑘𝑘𝑖𝑖 + 𝛽𝛽𝛾𝛾𝑖𝑖 + ∑𝑚𝑚=1 𝛾𝛾𝑚𝑚𝑋𝑋𝑚𝑚𝑖𝑖 + 𝑒𝑒𝑖𝑖

(1)

where:

𝑌𝑌𝑖𝑖 = a measure of long-term earnings or employment for sample member i, 𝑆𝑆𝑘𝑘𝑖𝑖 = one if sample member i is from site (city) k and zero otherwise, 𝛾𝛾𝑖𝑖 = one if sample member i is from a Jobs Plus housing development and zero otherwise,

𝑋𝑋𝑚𝑚𝑖𝑖 = baseline covariate m for sample member i and

𝑒𝑒𝑖𝑖 = a random error for sample member i that is independently and identically distributed with a normal distribution, a mean of zero, and a variance of 𝜎𝜎2.

The estimated value of 𝛽𝛽 in Equation 1 is the estimated impact of Jobs Plus on long-term earnings or employment, and the OLS standard error of this impact estimate is an appropriate basis for assessing the impact estimate’s statistical significance if sample members had been assigned individually to Jobs Plus or its comparison group. However, sample members for the Jobs Plus evaluation were randomly assigned to Jobs Plus or its comparison group in “clusters”

defined by the public housing development in which they lived. Consequently, an OLS standard error will tend to underestimate the true standard error of the impact estimate and thereby overstate its statistical significance (understate its p-value). 55

The Possibilities and Mechanics of a Permutation Test for the Present Analysis Several statistical approaches, such as multi-level regression (Raudenbush and Bryk, 2002) or cluster-robust standard errors (Angrist and Pischke, 2009, p. 312–15), are often used to estimate standard errors of impact estimates which adjust for the clustering of sample members. However, 55 See Jacob et al. (2010) for a discussion of this issue.

E-2

to provide meaningful results, these methods require many clusters. 56 Unfortunately, this is not the case for Jobs Plus, with a total of seven clusters (developments) for its three strong-implementation sites and eight clusters for its three weak-implementation sites.

To address this problem, we used an alternative approach from the statistics literature called a permutation test (Wilcox, 2003). This approach can, in principle, accommodate cluster assignments with few clusters. To implement the approach, we proceeded as follows, separately for the three strong-implementation sites and the three weak-implementation sites.

Step #1: Use Equation 1 to estimate the true Jobs Plus impact (𝛽𝛽̂𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇) from individual data for sample members using the true definition of all Jobs Plus and comparison developments (true values for 𝛾𝛾𝑖𝑖).

Step #2: Use Equation 1 to estimate a Jobs Plus pseudo-impact (𝛽𝛽̂𝑃𝑃𝑃𝑃𝑇𝑇𝑇𝑇𝑃𝑃𝑃𝑃) from individual data for sample members for a series of “re-definitions” of 𝛾𝛾𝑖𝑖 that represent all possible permutations of Jobs Plus and comparison developments. In other words, one permutation would reassign treatment status to residents in the comparison development in one city (i.e., pretend they are in a Jobs Plus development) and comparison status to residents in the Jobs Plus development in that city. All other assignments in other cities would be unchanged.

The total number of such permutations for strong-implementation sites (including that for the true definition of all Jobs Plus developments) is 12 (3 permutations for Dayton times 2 permutations for Los Angeles times 2 permutations for St. Paul), and its counterpart for weak-implementation sites is 18 (3 permutations for Baltimore times 3 permutations for Chattanooga times 2 permutations for Seattle).

Step #3: Rank-order the resulting true and pseudo impact estimates from most to least positive. This result is a sampling distribution of cluster-randomized impact estimates under the strong null hypothesis of uniformly zero Jobs Plus impacts (Wilcox, 2003).

Step #4: Locate 𝛽𝛽̂𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 in its sampling distribution and determine the corresponding p-value based on this location. For example, with 12 permutations, if 𝛽𝛽̂𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 had the largest positive value in its sampling distribution, an estimate this positive or larger could occur by chance under a null hypothesis of uniformly zero impacts with a probability/p-value of 56 A small number of clusters provides few degrees of freedom for estimating a cluster-level variance component, which in turn produces very low power (substantial uncertainty) for program impact estimates. Because the original Jobs Plus impact analysis was based on comparative interrupted time-series analysis, which leveraged time-series data for 6 baseline years at each housing development in the study sample. For each study site, annual baseline earnings levels and employment rates for the Jobs Plus development were consistently very similar to those for its comparison development/s (i.e. there was very little year-to-year fluctuation in this relationship); for this reason, the statistical precision of that analysis did not face a serious degree of freedom problem.

E-3

1

= 0.083. Correspondingly, if 𝛽𝛽̂

12

𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 were the second most positive value in its

sampling distribution, its p-value would be 2 or 0.167; and so on. 57

12

For a negative value of 𝛽𝛽̂𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇, the direction of p-values should be reversed. Thus, with 12 permutations, if 𝛽𝛽̂𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 is the most negative impact estimate in the sampling distribution, it would have a p-value of 0.083; if it were the second most negative impact estimate, it would have a p-value of 0.166.

Correspondingly, with 18 permutations for weak-implementation sites, the p-value when 𝛽𝛽̂𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 is the most or least positive estimate in the sampling distribution is 1 = 0.056.

18

Note that the difference between seven clusters for strong-implementation sites and eight clusters for weak-implementation sites creates a corresponding difference between 12 permutations and 18 permutations. This, in turn, produces a higher-resolution (more continuous) sampling distribution for weak-implementation sites.

To summarize the preceding steps, exhibit E.1 illustrates the determination of a permutation-based p-value for our OLS estimate of the Jobs Plus impact on the long-term earnings of adults in strong-implementation sites. Note that permutation #1, which represents all true Jobs Plus and comparison clusters and is highlighted in yellow, is 3rd from the top of the sampling distribution.

This implies a p-value of 0.249.

Exhibit E.1. Sampling Distribution and Permutation P-value for the Estimated Long-term Jobs Plus Impact on Adult Earnings in Strong-Implementation Sites

Permutation Number

Impact Estimate

Permutation P-value

11

2,710

0.083

7

2,380

0.166

1

1,670

0.249

2

1,251

0.332

12

819

0.415

6

347

0.498

9

– 370

0.581

4

– 877

0.664

8

– 1,259

0.747

3

– 1,706

0.830

57 This p-value is for a one-sided hypothesis test whose direction is specified in advance. Such a test is most appropriate when existing theory or empirical evidence strongly suggest the direction of an impact. This seems reasonable for the present analysis because it seeks to determine whether the long-term pattern of earnings and employment for Jobs Plus developments versus their comparison-group counterparts is qualitatively the same as their previously reported shorter-term pattern.

E-4

Exhibit E.1. (continued)

10

– 2,205

0.913

5

– 2,498

1.000

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires

Interpreting Permutation Test Results for the Present Analysis When assessing the quality of long-term Jobs Plus impact estimates, it is important to distinguish between their statistical significance and their internal validity (lack of selection bias). To assess their statistical significance requires determining the extent to which an estimated mean outcome difference for residents of Jobs Plus and comparison developments represents a true difference or estimation error. This is the goal of determining p-values, which is the subject of the present appendix.

To assess the internal validity of long-term Jobs Plus impact estimates requires determining whether an observed true long-term outcome difference was caused mainly by Jobs Plus or by other factors, a topic which is discussed elsewhere in the report.

To focus on statistical significance, exhibit E.2 presents estimates of mean Jobs Plus impacts on the long-term earnings and employment of adults and children in strong- and weak-implementation sites. For each impact estimate, the exhibit also reports its OLS p-value, its permutation p-value, and the location of the impact estimate in its sampling distribution.

Exhibit E.2. Estimates of Long-term Jobs Plus Impacts and their Alternative P-Values Sample and Outcome Measure

Actual Impact

OLS

Permutation

Permutation

Estimate

P-value

P-Valuea

Positionb

Strong Jobs Plus Implementation Sites

Adults

Earnings ($)

1,670

0.056

0.249

3rd from top

Employment Rate (%)

4.2

0.047

0.166

2nd from top

Children

Earnings ($)

2,706

0.000

0.083

top

Employment Rate (%)

6.2

0.000

0.083

top

Weak Jobs Plus Implementation Sites

Adults

Earnings ($)

– 1,967

0.019

0.111

2nd from bottom

Employment Rate (%)

– 4.7

0.018

0.111

2nd from bottom

E-5

Exhibit E.2. (continued)

Children

Earnings ($)

327

0.615

0.278

5th from top

Employment (%)

1.5

0.338

0.222

4th from top

a This p-value reflects a one-sided hypothesis test of whether the direction of a long-term program- and comparison-group outcome difference is the same as that for its shorter-term counterpart.

b The “permutation position” of a real Jobs Plus impact estimate is its location in its sampling distribution.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires Consider first the results for strong-implementation sites. Here, estimated Jobs Plus impacts on annual earnings and employment rates for adults are $1,670 and 4.2 percentage points, respectively, with corresponding OLS p-values of 0.056 and 0.047. These p-values, however, overstate statistical significance. On the other hand, permutation p-values, which are consistent with the clustered random assignment of sample members, are equal to 0.249 and 0.166, respectively. This suggests that we cannot determine with a high degree of confidence whether the estimated positive long-term outcome differences represent mainly true positive differences or positive impact error.

Nonetheless, the two impact estimates for adults are near the top of their sampling distribution and reflect the same patterns that were observed previously for shorter-run outcomes. Hence, one might conclude that the present findings provide suggestive but weak evidence that longer-term Jobs Plus impacts for adults are qualitatively similar to their shorter-run counterparts.

In contrast, findings in exhibit E.2 for children strongly suggest that those who had lived in a Jobs Plus development during the program period experienced long-term earnings levels and employment rates that were appreciably higher than those of their comparison-group counterparts. Not only are the magnitudes of these estimated differences large, but their OLS and permutation p-values suggest that they probably represent, at least in part, true differences.

Now consider the results for weak-implementation sites. Note first that these results for children provide no evidence of a positive or negative program- versus comparison-group outcome difference. The magnitudes of these estimated differences are quite small, and they are not at all statistically significant (based on OLS or permutation p-values).

However, results in the exhibit for adults provide evidence of a negative program- versus comparison-group outcome difference, with OLS and permutation p-values that are statistically significant at well below or near the 0.10 level. These findings suggest that this negative difference probably represents more of a real difference than estimation error. The remaining question, in this case, is whether this negative difference was caused by Jobs Plus or by other factors. As the report discusses, it is likely due to factors other than Jobs Plus that affected residents in these developments.

E-6

VII. References

Aaronson, Daniel, and Bhashkar Mazumder. 2008. “Intergenerational Economic Mobility in the United States, 1940 to 2000,” Journal of Human Resources 43, no. 1 (2008): 139–172.

Angrist, J.D., and J.S. Pischke. 2009. Mostly Harmless Econometrics, Princeton University Press, Princeton, NJ: 312–315.

Becker, Gary S. 1981. A Treatise on the Family. Cambridge: Harvard University Press.

Benjamini, Y., and Y. Hochberg. 1995. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing,” Journal of the Royal Statistical Society, Series B

(Methodological), 57, 289–300.

Blank, Susan, and Donna Wharton-Fields. 2008. Helping Public Housing Residents Find and Keep Jobs: A Guide for Practitioners Based on the Jobs-Plus Demonstration. New York: MDRC.

Bloom, Howard S., James Riccio, and Nandita Verma. 2005. Promoting Work in Public Housing: The Effectiveness of Jobs-Plus Final Report. New York: MDRC.

Chetty, Raj, Nathaniel Hendren, and Lawrence F. Katz. 2016. “The Effects of Exposure to Better Neighborhoods on Children: New Evidence from the Moving to Opportunity Experiment,”

American Economic Review, 106 (4): 855–902.

Coleman, James S. 1988. “Social Capital in the Creation of Human Capital,” American Journal of Sociology, 94 (1988): 95–120.

Cooper, Kerris, and Kitty Stewart. 2017. “Does Money Affect Children’s Outcomes? An Update,”

LSE Research Online Documents on Economics 103494, London School of Economics and Political Science, LSE Library.

Duncan, Robert J., Megan M McClelland, and Alan C. Acock. 2017. “Relations Between Executive Function, Behavioral Regulation, and Achievement: Moderation by Family Income,” Journal of Applied Developmental Psychology 49 (2017): 21–30.

Gardenhire, Alissa. 2004. Implementing Financial Work Incentives in Public Housing Lessons from the Jobs-Plus Demonstration. New York: MDRC.

Gennetian, Lisa, Nina Castells, and Pamela A. Morris. 2010. “Meeting the Basic Needs of Children: Does Income Matter?” Children and Youth Services Review, 32(9), 1138–1148.

Greenberg, David M., Aurelia Aceves, Victoria Quiroz-Becerra, David H. Greenberg, and Ari Oppenheim. 2015. The Second Generation of Jobs-Plus Programs: Implementation Lessons from San Antonio and the Bronx. New York: MDRC.

Hamilton, Gayle, and Charles Michalopoulos. 2016. Job Search or Basic Education Participation First: Which Improves Welfare Recipients’ Earnings More in the Long Term? New York: MDRC.

Jacob, R, P. Zhu, and H.S. Bloom. 2010. “New Empirical Evidence for the Design of Group Randomized Trials in Education,” Journal of Research on Educational Effectiveness, 3(2): 157–198.

105

Kato, Linda Yuriko, and James A. Riccio, with Jennifer Dodge. 2001. Building New Partnerships for Employment: Collaboration Among Agencies and Public Housing Residents in the Jobs-Plus Demonstration. New York: MDRC.

Leopold, Josh, Theresa Anderson, Marla McDaniel, Christopher R. Hayes, Sade Adeeyo, and Rob Pitingolo. 2019. Helping Public Housing Residents Find Jobs and Build Careers Evaluation Findings from New York City’s Jobs-Plus Expansion. Washington, DC: Urban Institute.

McLloyd, Vonnie C. 1990. “The Impact of Economic Hardship on Black Families and Development,” Child Development 65(2): 562–589.

Miller, Cynthia, Aletha C. Huston, Greg J. Duncan, Vonnie C. McLoyd, and Thomas S. Weisner.

2008. New Hope for the Working Poor Effects After Eight Years for Families and Children.

New York: MDRC.

Morris, Pamela, Lisa Gennetian, and Greg J. Duncan. 2005. “Effects of Welfare and Employment Policies on Young Children: New Findings on Policy Experiments Conducted in the Early 1990s,” Social Policy Report, 19 (2). Society for Research in Child Development.

Morris, Pamela, and Stephanie Jones. 2002. Children in Public Housing Developments: An Examination of the Children at the Beginning of the Jobs-Plus Demonstration. New York: MDRC.

Polachek, S.W. 2007. “Earnings Over the Lifecycle: The Mincer Earnings Function and Its Applications” IZA Discussion Papers No. 3181. Institute for the Study of Labor, Bonn.

Palomino, Juan C., Gustavo A. Marrero, and Juan G. Rodríguez. 2018 “One Size Doesn’t Fit All: A Quantile Analysis of Intergenerational Income Mobility in the U.S. (1980–2010),” The Journal of Economic Inequality 16, no. 3: 347–367.

Raudenbush, S.W., and A.S. Bryk. 2002. Hierarchical Linear Models: Applications and Data Analysis Methods, 2nd edition, Sage Publications.

Riccio, James A. 2010. “Sustained Earnings Gains for Residents in a Public Housing Jobs Program: Seven-Year Findings from the Jobs-Plus Demonstration.” Policy Brief. New York: MDRC.

Roder, Anne, and Mark Elliot. 2019. Nine-Year Gains: Project QUEST’s Continuing Impact. New York: Economic Mobility Corporation.

Verma, Nandita, Betsy L. Tessler, David H. Greenberg, Edith Yang, Sophia Sutcliffe, Michael D.

Webb, William M. Rohe, Atticus Jaramillo, Amy T. Khare, Emily K. Miller, and Mark L.

Joseph. 2019. HUD’s Jobs Plus Pilot Program for Public Housing Residents: Ongoing Implementation Experiences. New York: MDRC.

Votruba-Drzal, Elizabeth. 2006. “Economic Disparities in Middle Childhood Development: Does Income Matter?" Developmental Psychology 42 (6): 1154.

Wilcox, R.R. 2003. “Comparing Two Independent Groups,” Applying Contemporary Statistical Techniques, Science Direct, Academic Press.

106

U.S. Department of Housing and Urban Development

Office of Policy Development and Research

Washington, DC 20410-6000

May 2023

Document Outline

HUD Jobs Plus Outcomes Evaluation

DISCLAIMER

Acknowledgments

Foreword

Table of Contents

List of Exhibits

Executive Summary

Data and Methods

Key Findings

Jobs Plus continued to positively impact earnings of work-able residents in the stronger implementation sites 15 years after the program ended.

The children living in the Jobs Plus developments at the time that the program was implemented also experienced higher earnings and employment in adulthood, compared with their comparison group counterparts.

Conclusion

I. Introduction

The Jobs Plus Model88 This section draws heavily on Bloom, Riccio, and Verma (2005), which provides a fuller description of the demonstration and findings through 2003, when it ended; Riccio (2010) updates some of the main findings, covering another three years of follow-up.

The Jobs Plus Developments

Jobs Plus Implementation

Prior Evidence of Jobs Plus Effectiveness

II. Long-Term Impacts Analysis: Research Questions, Study Sample, Data Sources, and Analytic Approach

Research Questions and Hypothesized Long-Term Effects

Subgroup Analyses

Samples and Timeframe

Data Sources and Key Outcomes

Analytic Approach

Covariates

III. Long-Term Effects for Adults

IV. Long-Term Effects on Labor Market Outcomes of Children Living in Jobs Plus Households

Effects on Children’s Later Labor Market Outcomes in Stronger Implementation Sites

Effects on Children’s Later Labor Market Outcomes by Site

Effects on Children’s Later Labor Market Outcomes by Child Age During Jobs Plus

V. Discussion of Findings

Study Limitations

VI. Conclusion

APPENDIX A: Analytic Model and Specifications

APPENDIX B: Summary of Program Implementation for the Sites in the Jobs Plus Demonstration

APPENDIX C: Supplemental Impact Exhibits

APPENDIX D: Estimating the Gain in Lifetime Earnings for Children of Adult Participants

APPENDIX E: Using Permutation Tests to Assess the Statistical Significance of Long-Term Jobs Plus Impact Estimates

The Problem with Ordinary Least Squares Standard Errors for the Present Analysis

The Possibilities and Mechanics of a Permutation Test for the Present Analysis

Interpreting Permutation Test Results for the Present Analysis

VII. References