Evaluation of the Resident

Opportunity and Self-Sufficiency –

Service Coordinator Program

U.S. Department of Housing and Urban Development | Office of Policy Development and Research

Evaluation of the Resident

Opportunity and Self-Sufficiency

Service Coordinator Program

Corianne Payton Scally

Eric Burnstein

Nicole DuBois

Marcus Gaddy

Chris Hayes

Clare Salerno

Patrick Spauster

Yipeng Su

Elsa Falkenburger

Sue Popkin

Urban Institute

Kathy Carton

Nina Liou

Rhae Parks

EJP Consulting Group, LLC

November 2019

Foreword

The Resident Opportunity and Self Sufficiency (ROSS) Program was created in 1998 to consolidate several small programs intended to assist residents of public housing and tribal designated housing entities. In 2008, HUD merged these programs further into one ROSS Service Coordinator (ROSS-SC) program. The direct service provision component was replaced with a service coordination model focused on connecting public housing residents to local services to help residents attain economic and housing self-sufficiency or to age-in-place and maintain independent living. This report responds to a Congressional request for an evaluation of the ROSS-SC program after a 2013 report by the United States Government Accountability Office (GAO) identified significant gaps in data needed to document the effectiveness of the program.

This study is the first comprehensive evaluation of the ROSS-SC program. The report presents findings from an analysis of HUD and grantee administrative data from a system that has since been upgraded, and a national survey of Service Coordinators employed with grants over three-year period (Fiscal Years 2014-2016). It also includes qualitative findings on program effectiveness from interviews of service coordinators, PHA staff, and focus groups with residents at 10 sites.

Significant takeaways from this study suggest (1) service coordinators are highly experienced and qualified professionals, (2) the ROSS-SC program provides public housing residents with vital services and so, increasing access to service coordinators for all public housing residents would seem to be beneficial, (3) barriers to accessing services exist for residents and for PHAs that are geographically-isolated and/or do not have other service coordinator funding or service partners, and (4) while HUD

responded to the GAO’s recommendations for improved data collection and improved the grantee reporting system (the FY 2014 Logic Model) assessed by this study, grantee reporting could be improved to better document program outcomes. Based on these findings, future research could consider whether there are any disparities in access based on race, gender, disability status, broader community investment in services, or other socioeconomic factors.

Around the time the study’s data collection was underway, the Department began to pilot a new online data system and implement other program improvements. The new online data system and those improvements were not able to be captured by this study. The most significant of these program improvements include:

• Issuing a Data Guide, which is updated annually and provides definitions of each metric, how they tie to areas of need grantees are focused on, and Key Performance Indicators.

The Data Guide is available on the HUD Exchange ROSS home page.

• Providing annual reporting training for grantees.

• Reporting is now aligned with the applications. Applicants select areas of need upon which they will focus, based on findings from a community needs assessment, and then use the

new reporting mechanism, Standards for Success, to report using metrics related to the areas of need. In addition, grantees may submit a narrative to accompany their data submissions.

• As of the end of October 2020, a data visualization tool that aggregates grantees’ data is available on HUD Exchange. This tool allows grantees to view their data in chart form and monitor their progress. HUD has begun providing in-depth training for how to use this tool which will include one-on-one assistance to around twenty percent of the grantees.

• Establishing a list-serve that allows grantees to communicate directly with HUD program staff.

• An online guidebook providing step-by-step guidance for implementing a ROSS program available on the HUD Exchange ROSS home page.

• A webinar series covering key training topics for grantees such as “motivational interviewing;” and

• An annual onboarding webinar for both new and renewal grantees.

These findings will help inform program improvements, while adding to the evidence of success of this critical program for delivering a wide variety of services to assisted households. The report can aid policymakers in guiding funding decisions and programmatic improvements, together with Public Housing Agencies that administer the program on behalf of their communities. The report also documents the role of perhaps the most important part of the program - the service coordinators themselves, who connect vital services to our assisted families.

Todd M. Richardson

General Deputy Assistant Secretary for Policy Development and Research

U.S. Department of Housing and Urban Development

Contents

Acknowledgments

ix

Executive Summary

1

Key Findings

2

Implications and Recommendations for HUD

3

Chapter 1. Introduction

7

Overview of the Resident Opportunity and Self-Sufficiency Program

7

Reason for Assessment

13

Research Goals

14

Report Overview

17

Chapter 2. Methodology

19

Introduction

19

Site Visits

22

Survey of ROSS-SC Service Coordinators

25

Administrative Data Analysis

29

Chapter 3. Residents and Services

35

Introduction

35

Key Findings

35

Targeting and Engaging Residents

37

Aligning Services to Meet Resident Needs

41

Relationships with Partners

48

How ROSS-SC Enhances Public Housing Operations

56

Conclusion

58

Chapter 4: Performance Monitoring and Reporting

59

Introduction

59

Key Findings

60

Logic Model Structure and Processes

61

Assessing Logic Model Changes

62

Conclusion

70

Chapter 5. Service Coordinators: Work Responsibilities, Qualifications, and Supports

72

Introduction

72

Key Findings

72

Core Work Responsibilities

74

Service Coordinator Qualifications

83

Work Supports and Training

89

i

Conclusion

93

Chapter 6. Resident Experiences and Local Contexts

95

Introduction

95

Key Findings

96

Resident Experiences

97

Local Contexts

106

Conclusion

111

Chapter 7: Study Implications and Recommendations

112

Introduction

112

Establishing a Theory of Change

113

Defining and Measuring Grantee Performance

119

Improving Peer Learning and Communications

128

Filling Service Gaps and Facilitating Partnerships

130

Meeting Resident Needs

132

Sustaining ROSS-SC

135

Appendix A. Sample Individual Training and Services Plan (ITSP)

138

Appendix B. Interview Protocols

139

Informed Consent

139

Service Coordinator

141

Grantee Supervisor/Manager

144

Partner

147

Appendix C. Focus Group Protocols

148

Informed Consent

148

Resident Focus Group Guide

149

Appendix D. Service Coordinator Survey

151

Appendix E. Service Coordinator Survey Weights

183

Appendix F. Question Variations Between Fielded Versus OMB-Approved Service

Coordinator Survey

184

Appendix G. ROSS-SC Service Coordinators—Needs and Service Partners

(HUD Form 52769)

187

Appendix H. FY 2014-2016 Logic Model

189

Appendix I. FY 2014 Application-Logic Model Service Classification

212

Appendix J. Guidance for Completing the 2014 ROSS-SC Logic Model

213

ii

Appendix K. 2011-2014 Crosswalk

229

Appendix L. FY 2014 Outputs Versus Outcomes by Metric

235

Appendix M. Application-Logic Model Crosswalk

238

References

239

Additional Reading

242

iii

List of Exhibits

Exhibit 1: Timeline of Resident Opportunities and Self-Sufficiency Service Coordinator Program

and Related Programs................................................................................................................................................ 8

Exhibit 2: Resident Opportunities and Self-Sufficiency Service Coordinator Program Description ......... 9

Exhibit 3: Timeline of ROSS-SC Grantees and Appropriations ....................................................................... 12

Exhibit 4: Research Questions for Resident Opportunity and Self-Sufficiency Program Evaluation ..... 16

Exhibit 5: Primary Data Source for Key Research Questions.......................................................................... 19

Exhibit 6: Timeline of Data Col ection Activities ............................................................................................... 21

Exhibit 7: Distribution of FY 2014 Grantees and Samples by Selection Criteria ........................................ 23

Exhibit 8: Site Interviews ........................................................................................................................................ 24

Exhibit 9: Distribution of Service Coordinators Who Responded to the Service Coordinator

Survey by Grantee Type ......................................................................................................................................... 27

Exhibit 10: FY 2014 ROSS-SC Application Form: Needs ROSS-SC Grantees Plan to Address ............... 29

Exhibit 11: Summary of Grantee Samples Across Three Quantitative Analyses ........................................ 30

Exhibit 12: Grantee Classifications ....................................................................................................................... 36

Exhibit 13: Types of Residents Grantees Target ................................................................................................ 37

Exhibit 14: Types of Residents Targeted on FY 2014 Applications and Logic Model Projections .......... 38

Exhibit 15: Grantee Perceptions of Outreach to Residents and Residents’ Participation ......................... 40

Exhibit 16: Partner Referrals by Type of Service ............................................................................................... 42

Exhibit 17: Service Coordinators’ Additional Activities .................................................................................... 45

Exhibit 18: Partners Meeting Resident Demands .............................................................................................. 47

Exhibit 19: Partner Contacts Versus Service Agreements ............................................................................... 52

Exhibit 20: Logic Model Definitions ..................................................................................................................... 61

Exhibit 21: Change in Data Reporting Between the FY 2011 and FY 2014 Logic Models....................... 62

Exhibit 22: Grantee Means of Tracking Client Interactions ............................................................................ 64

Exhibit 23: Data Quality Definitions ..................................................................................................................... 65

Exhibit 24: Service Coordinator Recommended Functions ............................................................................. 74

Exhibit 25: Identifying Resident Needs................................................................................................................ 76

Exhibit 26: Use of Formal Assessments ............................................................................................................... 77

Exhibit 27: Total Residents Served Per Service Coordinator .......................................................................... 78

Exhibit 28: Service Coordinator Caseload by Month ........................................................................................ 79

Exhibit 29: Service Coordinator Caseload versus Meetings to Work Toward Personal Goals ................ 80

Exhibit 30: Frequency and Length of Meetings with Residents ..................................................................... 81

iv

Exhibit 31: Use of Individual Training and Service Plans ................................................................................. 82

Exhibit 32: Service Coordinator Work Experience ............................................................................................ 83

Exhibit 33: Service Coordinator Months in Current Position by Grant Year ............................................... 84

Exhibit 34: Service Coordinator Educational Attainment ................................................................................ 86

Exhibit 35: Service Coordinator Compensation by Tenure ............................................................................. 89

Exhibit 36: Service Coordinator Training Provided by Grantees .................................................................... 90

Exhibit 37: Satisfaction of Support from Grantees............................................................................................ 92

Exhibit 38: Referrals Resulting in Outcomes, FY 2014 Grantees ................................................................... 99

Exhibit 39: What is a Theory of Change? ......................................................................................................... 113

Exhibit 40: Implicit Theory of Change for Achieving Resident Outcomes through ROSS-SC

Program, Case Management Approach ............................................................................................................ 115

Exhibit 41: Implicit Theory of Change for Achieving Resident Outcomes through ROSS-SC

Program, Service Connection Model ................................................................................................................. 119

v

Preface

This Preface outlines HUD-implemented program improvements that began taking place while this study was underway and which, while mostly outside the scope of the study, are supported by the findings from the site visits, service coordinator survey, and importantly, the study’s recommendations.

Urban Institute’s evaluation of the ROSS-SC program presents an analysis of the program using grantee-submitted program data covering 2014-2016 (which was col ected from April 2017 through October 2018). Program data was submitted by grantees using the Logic Model, an excel spreadsheet which aggregated data and was submitted annually. One of the core components of the study was to examine whether changes the Office of Public Housing (PIH) made to the Logic Model in response to an audit by the Government Accountability Office translated into improved data. This study determined that in fact, data had improved, but further improvements could be made.

While the study was underway, PIH was implementing a new data system. In 2016, HUD’s Office of Grants Management & Oversight (GMO) which oversees policies and procedures for all HUD grant programs, began piloting a new reporting system, Standards for Success. ROSS was one of the few programs selected for this pilot. Grantees began transitioning data col ection from the Logic Model to the Standards for Success Platform in 2016, and in 2017 the full transition was complete for all 2016

and subsequent grantees. Therefore, the evaluation findings refer to the decommissioned Logic Model reporting structure and do not pertain to the new Standards for Success system.

Nevertheless, the findings and recommendations resulting from the study were not only useful but served to buttress the program improvements already underway.

After the data col ection period began for this study (April 2017 through October 2018), HUD

released a series of technical assistance products to guide grantees in the use of the new data reporting system and to assist with overall program implementation. These products were posted on a new HUD Exchange ROSS page. The products include a Data Guide, a Step-by-Step program guidebook, webinars, and additional program information. As a result, most of the recommended program enhancements have been implemented and are noted in italics.

• Create a standard for reporting that reflects the priorities of the ROSS-SC program office.

HUD’s new reporting system responds to this recommendation.

vi

o The new reporting system has an online data guide which explains which metrics are required and which ones are not, what each metric is, and which metrics are tied to various areas of need.

o HUD’s new reporting system al ows grantees to attach a narrative statement to accompany their data reports.

• Provide training to service coordinators and grant managers in meeting the data standards in order to generate data that are comparable across sites. HUD provides annual training to grantees on the new data system.

• Peer learning on best practices could improve the capacity for all grantees, given that some sites do have high-quality and efficient data col ection practices.

• Distribute a list of available case management systems to aid in standardizing tracking systems by resident. HUD has responded to this by al owing grantees to use a portion of their grant funds to purchase case management systems. Grantees use the case management systems to submit their annual reports to HUD’s Standards for Success reporting system.

• Provide additional funds for lower-resourced grantees to purchase and maintain hardware and software. HUD al ows grantees to use grant funds to purchase necessary hardware and software to support the work of the ROSS service coordinator.

• Offer technical assistance to grantees. HUD has developed a series of webinars, written guidance, and a data visualization tool that wil allow grantees to see their data in graph form.

In addition,

• Reporting is now aligned with the applications. Applicants select areas of need upon which they wil focus, based on findings from a community needs assessment, and then use the new reporting mechanism, Standards for Success, to report using metrics related to the areas of need. In addition, grantees may submit a narrative to accompany their data submissions.

• As of the end of October 2020, a data visualization tool that aggregating grantees’ data is available on HUD Exchange. This tool allows grantees to view their data in chart form and monitor their progress. HUD has begun to provide in-depth training for using this tool, which also includes one-on-one assistance to 20 percent of the grantees.

• Since FY 2018, the ROSS-SC NOFA has asked applicants to assess residents’ areas of need in the targeted communities and focus their programs and reporting on those areas. This is vii

which is different from the ROSS-SC NOFAs that were in effect during the study data collection period.

• Since this report was drafted, data projections are no longer required as part of the application process. Instead, applicants are now required to assess at least 20 percent of the residents in the developments they wil target with their grant (if they are awarded) and use the findings to develop their application (proposing to address at least three areas of need that emerge from the needs assessment).

Other program improvements were beginning to be implemented which also coincided with the timing of the study and which are supported by the study’s findings and recommendations.

• The study team recommended improved communications between service coordinators and HUD.

o Since the study data col ection period ended, PIH Housing has been seeing Service Coordinators on the ROSS-SC list-serve and has provided peer learning webinars

featuring ROSS grantees on the HUD exchange website:

https://www.hudexchange.info/programs/ross/.

o Establishing a LISTSERV and email address that allows grantees to communicate directly with HUD program staff, and grantees are encouraged to connect with their local HUD field office staff. Service coordinators may opt-in to the HUD LISTSERV on a voluntary basis. HUD uses this LISTSERV to communicate training opportunities, important program updates, and service coordinators and other grantee staff may

send questions directly to the designated email address to ask questions and receive answers from HUD staff.

o An online guidebook providing step-by-step guidance for implementing a ROSS

program.

o A webinar series covering key training topics for grantees such as “motivational interviewing.”

o An annual onboarding webinar for both new and renewal grantees.

HUD will continue to make program improvements based on comprehensive data analysis and input from HUD field staff and the grantee community.

viii

Acknowledgments

This report was funded by the U.S. Department of Housing and Urban Development (HUD) through a contract awarded to the U.S. Department of Health and Human Services under contract number HHSP2332015000641. We are grateful to them and to all our funders, who make it possible for Urban to advance its mission.

We wish to thank Marina Myhre, HUD Office of Policy Development and Research (PD&R), for her technical guidance on this project and Blair Russel for data assistance. David Hardiman, Teresa Souza, Mark Shroder, and Carol Star at HUD PD&R also provided helpful comments. Finally, HUD

Resident Opportunity and Self-Sufficiency Service Coordinator (ROSS-SC) program officers Dina Lehmann-Kim and Tremayne Youmans provided valuable insights, ensuring our analysis answered the questions most relevant to the program. Thank you as wel to Charles Eldridge from the Office of Field Operations and Chad Trepinski from the Office of Public and Indian Housing in the Portland, OR Field Office for their helpful comments, and J’Vaughn Hawkins from HUD Office of Public and Indian Housing for providing assistance with obtaining grant application data forms.

We would also like to thank Sue Popkin for her thoughts and leadership and many contributions to this work and our technical reviewer, Mary Cunningham. We also thank Brice McKeever, Sierra Latham, Megan Thompson, Ilana Brody, and Ruth Gourevitch, all of whom contributed to this research during their time at the Urban Institute.

The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders. Funders do not determine research findings or the insights and recommendations of Urban experts. Further information on the Urban Institute’s funding principles is available at urban.org/fundingprinciples. Any errors or omissions remaining in this report are the responsibility of the authors.

ix

Executive Summary

Since 2008, the U.S. Department of Housing and Urban Development (HUD) has provided funding for the Resident Opportunity and Self-Sufficiency Service Coordinator (ROSS-SC) program, which al ows eligible grantees to hire service coordinators to connect public housing residents to needed services.

The program’s goals are to help improve economic and housing self-sufficiency and, in the case of elderly residents or people with disabilities, to improve conditions for independent living. This evaluation is aimed at understanding how housing agencies use service coordinators, whether those agencies make effective and efficient use of ROSS-SC resources, and whether those efforts lead to positive experiences and outcomes for residents. In addition, this report outlines implications and recommendations for improving program implementation, reporting, and future evaluation.

ROSS-SC is intended to assist three groups of public housing residents: seniors, persons with disabilities, and very low-income families. Four types of grantees are eligible for ROSS-SC: public housing agencies (PHAs), Tribal y Designated Housing Entities (TDHEs), resident associations affiliated with a PHA, and nonprofit organizations that must be supported by a PHA, tribe, or resident association. Grants provide 3 years of funding, and all grantees must contribute a 25-percent match in cash or in-kind donations. Grantees may apply for funds for up to three service coordinators, depending on the number of housing units in the grantee’s portfolio. Twenty-five percent of ROSS-SC

funds are set aside for qualified resident association applicants. Applicants seeking grant renewals receive priority over those seeking new grants. From $30 million to $40 million is appropriated annually to ROSS-SC, and about 10 percent of PHAs (approximately 350) nationally have active grants at any one time.

Data used for this evaluation came from multiple sources: a national survey of service coordinators conducted by the Urban Institute, HUD administrative data, and a series of 10 site visits conducted by the Urban Institute and EJP Consulting Group. The national survey of service coordinators which gathered data from 215 grantees yielded information about service coordinator activities and qualifications, resident needs, partnerships, and available work supports and trainings. In addition, using administrative data submitted by grantees using a Logic Model reporting tool, the research team conducted three separate analyses: (1) a comparison of planned services and activities to be coordinated, as reported by grantees in their grant applications, with what they actually provided according to their post-award reporting; (2) an analysis of data reported by repeat grantees receiving grants in both FY 2011 and FY 2014; and (3) a performance period analysis of FY 2014

grantees using reported data over their 3-year grant period. Site visits included interviews with public 1

housing agency staff, grant managers, service coordinators, and community partners, as wel as a resident focus group.

Key Findings

Residents access critical services with the help of their service coordinators but also face barriers to getting the help they need.



Many public housing residents—particularly those experiencing unemployment—rely on service coordinators to help with emergency and longer-term needs. Outcomes were not clearly captured by administrative data, but residents reported positive program experiences.



Residents are most commonly connected to health and adult education services, as wel as access to food and transportation. Other services include youth programming and emergency-related services.



Common barriers to accessing services include lack of local transportation and childcare, limited English proficiency, and a dearth of service providers in lower-resourced, more geographically isolated communities with smaller populations. 1 Those obstacles may require resources beyond what is possible for ROSS-SC to provide.

Service coordinators are equipped for and take multiple approaches to their work and are valued for what they do.



Service coordinators are educated, experienced, and adept at reaching out to residents and seeking out community partners to help meet identified resident needs.



Service coordination approaches vary from case management to a focus on making broader service connections through outreach and events. Those approaches can be blended together and affect how service coordinators report on their interactions with residents.

1 Lower-resourced refers to PHAs with no other federal service coordinator funding and/or operating in communities with few service partners due to population size and geographic isolation.

2



Residents value the role that service coordinators play in connecting them to resources in the community.

Partners are the key to accessing services and meeting program goals.



Service coordinators build relationships with community partners.



Community partners are key to the implementation of ROSS-SC; however, local gaps may exist between resident needs and available services. This situation is particularly true for grantees in smaller, more remote places with more-limited resources.



Property managers are critical partners on the ground. They are the number one source of resident referrals to service coordinators, with whom they often work to troubleshoot issues around housing stability, such as unit maintenance and eviction prevention.

Grantee reporting could improve and expand to tell a more comprehensive story.



Grantee performance reporting became more complete with the adoption of the FY 2014

Logic Model. The introduction of mandatory reporting on a standard set of metrics helped.



Issues with the FY 2014 Logic Model reporting structure and data quality made program inputs, outputs, and outcomes for residents unclear. 2



Grantees want to tel the ful story of their work. Existing metrics do not reflect al of what grantees do to meet resident needs through ROSS-SC, and they would like the opportunity to share more.

Implications and Recommendations for HUD

Establishing a Theory of Change

Grantees currently have two pathways to leverage ROSS-SC to achieve resident outcomes. The case management approach assumes regular and iterative interaction between service coordinators and 2 HUD replaced the Logic Model reporting structure with the online Standards for Success reporting system, which started in 2016 as a pilot and moved to ful implementation in 2017 for al 2016 and subsequent grantees.

3

residents to assess needs, refer to services, access services, achieve outcomes, and then circle back on additional needs. The service connection approach focuses on broader outreach, leading to attendance at ROSS-SC-sponsored events that facilitate direct connections between residents and community service partners. This model of interaction is just as important as the first, but it makes it harder to assess whether residents access services and, if so, what any related outcomes may be. Both models can operate simultaneously within a single grantee or through a single service coordinator, which can present measurement chal enges when asked to report on services and activities that may be appropriate for one program pathway but not the other. Acknowledging those different paths could improve program implementation and measurement by HUD and grantees.

Defining and Measuring Grantee Performance

Targeted improvements by HUD in measuring performance could make grantee services easier to track, improve reliability (defined in exhibit 23) of entered data, and al ow grantees to tell their ful ROSS-SC story. Aligning services and activities proposed by grantees in their applications with those reported on after the grant award wil ensure continuity between what grantees proposed they would do and what they actually do during the grant period. Improving reporting instructions in several ways—including providing more detailed guidance on how to report client interactions and outcomes under the case management versus service connection approach—would increase the reliability of reported data. Finally, giving grantees more options of how to report—expanding the use of optional metrics, al owing new fields for grantees to report on challenges encountered, and allowing narrative entries—could enhance the picture of ROSS-SC on the ground and provide more thorough data for future evaluations.

Improving Peer Learning and Communications

Service coordinators, particularly those from lower resourced grantees, are relatively isolated from one another and could benefit from improved peer-learning opportunities to share best practices for engaging residents, leveraging partnerships, and overcoming barriers to accessing necessary services.

ROSS-SC can be used to support those opportunities but may not be enough. Improved direct, two-way communications between service coordinators and HUD could ensure that service coordinators 4

receive appropriate program news and resources and can regularly share feedback with HUD on program implementation and challenges. 3

Filling Service Gaps and Facilitating Partnerships

Service gaps in some places suggest the need for more direct service resources to improve access and address resident needs. Although the ROSS-SC program’s precursors used to provide funding for direct services, this is no longer an al owable use of funds within the current program iteration. This could be revisited, particularly for lower resourced communities that simply have few service options.

HUD could also assist in facilitating local and national partnerships with commonly identified chains, such as grocery stores and optometrists, to build a pool of organizations that could support ROSS-SC

grantees at the local level.

Meeting Resident Needs

To better assess and address resident needs, HUD could facilitate grantee access to resident data from public housing developments being served. This could al ow grantees to better understand the residents they serve as well as identify some possible service connections that might help them, on the basis of key characteristics such as adult age, presence of children, earnings, and sources of income. Best practices could also be culled from other promising models of resident services and suggested for ROSS-SC implementation, such as coaching and two-generation models. Finally, removing resident barriers should be made more of a priority by allowing more grant resources to be used for resources such as transportation and childcare which can pose significant barriers to self-sufficiency, and to identify further local partnerships to overcome those and other obstacles.

Sustaining ROSS-SC

This evaluation highlights how ROSS-SC service coordinators provide a valuable connection between public housing residents and partners in the surrounding community to help address resident needs and improve self-sufficiency and independent living. To sustain ROSS-SC into the future, HUD could help mitigate several issues. Grantees face funding concerns when the ROSS-SC funding is zeroed out in Presidential, Congressional, or Department budgets. They are also concerned about losing funding if the public housing developments served by ROSS-SC are converted through the Rental Assistance 3 Please refer to the HUD Preface for an update on ROSS-SC grantee and HUD communication.

5

Demonstration program and become ineligible for future ROSS-SC grants as a result. In addition to seeking legislative assistance in addressing those realities, HUD may benefit from strengthening relationships among service coordinators through peer-learning opportunities and between service coordinators and HUD via more direct, two-way lines of communication. 4 Final y, the ROSS-SC

program could be better tailored to address the specific needs of lower resourced grantees by providing additional targeted funding and al owing direct service provision to address unmet resident needs in the absence of community service partners.

4 Please refer to the HUD Preface for an update on ROSS-SC grantee and HUD communication.

6

Chapter 1. Introduction

The Resident Opportunities and Self-Sufficiency Service Coordinator (ROSS-SC) program funds service coordinators to connect residents of public housing with local services to enable them to achieve economic and housing self-sufficiency and live independently (HUD n.d.a). ROSS-SC has never been formal y evaluated; this report seeks to fill that gap by reviewing and assessing key components of the program. This study presents an opportunity to understand how housing agencies use service coordinators, whether those agencies make effective and efficient use of ROSS-SC

resources, and whether those efforts lead to positive outcomes for residents. In addition, this report analyzes changes that were made in performance reporting to help inform future data reporting efforts for the program. 5 This introductory chapter provides an overview of the ROSS-SC program, including its history, structure, and funding. For context, it then describes existing federal and non-federal programs that either provide or coordinate services for residents of federally funded or assisted housing and reviews evidence about their effectiveness. Final y, it outlines the goals of this study, the research questions that this study answers, and the content of the report.

Overview of the Resident Opportunity and Self-

Sufficiency Program

Program Inception

The U.S. Department of Housing and Urban Development (HUD) operated a series of social service programs that preceded ROSS-SC (exhibit 1).

5 HUD replaced the Logic Model reporting structure with the online Standards for Success reporting system, which started in 2016 as a pilot and moved to ful implementation in 2017 for all 2016 and subsequent grantees.

The evaluation findings are based on the decommissioned Logic Model reporting structure.

7

EXHIBIT 1: TIMELINE OF RESIDENT OPPORTUNITIES AND SELF-SUFFICIENCY SERVICE COORDINATOR

PROGRAM AND RELATED PROGRAMS

Sources: American Association of Service Coordinators n.d.; Expectmore.gov n.d.; HUD n.d.b, n.d.c, 1999; HUD PD&R 2000

Note: This timeline shows the major programs that preceded ROSS-SC.

Established in 1966, the first of those programs, the Economic Development and Supportive Services Program (EDSS), funded housing authorities and tribal y designated housing entities (TDHEs) to provide direct services that helped residents obtain the education, training, and social services needed to obtain and maintain employment (HUD 1999, HUD n.d.c, 1998; HUD PD&R 2000). HUD began funding service coordinators in the 1990s through the Cranston-Gonzalez National Affordable Housing Act of 1990 (Levine and Johns 2008). The Housing and Community Development Act of 1992 expanded the authority beyond Section 202 properties and began awarding grants to fund service coordinators in federally subsidized and public housing (American Association of Service Coordinators n.d., McNickle 2007, Chavis 2018, Levine and Johns 2008), The Tenant Opportunities Program (TOP) was original y established in 1994 and funded resident councils to provide direct services to families, elderly, and disabled populations (HUD 1998). Final y, in 1998, the Quality Housing and Work Responsibility Act of 1998 (QHWRA) established the ROSS program (Expectmore.gov n.d.).

During its first decade, from 1999–2007, ROSS included provisions for service coordinators (or program coordinators, as they were then cal ed) to provide direct services to residents. The program 8

supported a wide range of activities, including resident management and business development, capacity building, resident service delivery models, homeownership supportive services, neighborhood network centers, and public housing service coordinators, as wel as specific grants for targeted populations, such as seniors and people with disabilities (Federal Register 67, no. 58; HUD PD&R

2003). In 2008, the program was consolidated and renamed ROSS-SC, and the direct services component was eliminated in favor of a service coordinator model focused on linking residents to existing services in the community (Expectmore.gov n.d.; HUD n.d.a). The goal was to provide public housing residents with access to services in a more cost-effective manner (HUD n.d.a). Exhibit 2

provides the current program description.

EXHIBIT 2: RESIDENT OPPORTUNITIES AND SELF-SUFFICIENCY SERVICE COORDINATOR PROGRAM

DESCRIPTION

“The purpose of the ROSS-SC Service Coordinator program is to provide funding to hire and maintain service coordinators who will assess the needs of residents of conventional public housing or Indian housing and coordinate available resources in the community to meet those needs. This program works to promote the development of local strategies to coordinate the use of assistance under the public housing program with public and private resources for supportive services and resident empowerment activities. These services should enable participating families to increase earned income, reduce or eliminate the need for welfare assistance, make progress toward achieving economic independence and housing self-sufficiency, or, in the case of elderly or disabled residents, help improve living conditions and enable residents to age in place.”

Source: HUD 2014: 2

Eligible Grantees and Target Population

Four types of grantees are eligible for ROSS-SC: public housing agencies (PHAs), TDHEs, resident associations affiliated with a PHA, and nonprofit organizations that must be supported by a PHA, tribe or resident association. Grants provide 3 years of funding, and all grantees must contribute a 25-percent match in cash or in-kind donations. These contributions usual y come from local organizations or service providers, but federal, state, or foundation money may also be permitted. Grantees enter into agreements with local organization(s) to provide services to residents as an in-kind contribution.

Grantees can apply for funds for one, two, or three service coordinators, depending on the number of housing units the grant wil serve (HUD 2017). One service coordinator wil serve 50–1,000 units, two service coordinators wil serve 1,001–2,500 units, and three wil serve 2,501 or more units (HUD

2017). Until 2017, eligible grantees who submitted complete applications were chosen using a lottery, with 25 percent of funds set aside for qualified resident associations, and priority was given to renewal applicants across all applicant types (HUD 2008-2017 notices of funding availability 9

[NOFAs]). In 2018, a lottery was only used to break ties on score ratings; scores were based on past performance (capacity to meet ROSS-SC program requirements and timely use of HUD funds) and either soundness of approach for renewal applicants or capacity and soundness of approach for new applicants (HUD 2018). By statute, 25 percent of ROSS-SC funds are still set aside for resident association applicants, and applicants seeking to renew their grants continue to receive priority over applicants seeking new ones (HUD 2018).

ROSS-SC is intended to assist three groups of public housing residents: seniors, persons with disabilities, and very low-income families. About 10 percent, or about 350, of PHAs nationally have active grants at any one time. Although a variety of HUD programs support low-income residents, ROSS-SC is currently the only standalone program receiving appropriated funds for service coordination for seniors in public housing. The Multifamily service coordinator program serves seniors living in HUD-assisted privately owned housing, and the Elderly/Disabled service coordinator Program is funded from the PHA Operating Fund, rather than receiving its own appropriation (HUD n.d.a, HUD

n.d.e). The Elderly/Disabled service coordinator program was formerly part of ROSS until its 2004

transition to the Operating Fund, although it has not funded any new PHAs since 1995 (McNickle 2007). The program operates like ROSS-SC by funding a service coordinator whose role is to link elderly residents and residents with disabilities to services (McNickle 2007). Although these other service coordinator programs exist for elderly residents, ROSS-SC is becoming increasingly critical as a means to support aging in place as people over the age of 62 make up higher proportions of PHA populations each year (Chavis 2018).

Role of the ROSS-SC Service Coordinator

The service coordinator’s main charge is to assess residents’ needs and offer relevant information and referrals to an array of local services aimed at supporting their economic, physical, and social wel -

being. Service coordinators are encouraged to perform a variety of recommended tasks to support residents, including the following:



Coordinating a local Program Coordinating Committee of local service providers.



Program marketing.



General case management of resident needs and referrals to local service providers.



Coordinating and overseeing service provision.

1 0



Coordinating educational or other informational events.



Establishing formal and informal resident networks that encourage self-sufficiency.



Monitoring and tracking resident participation in local services.



Evaluating program success (HUD, 2017).

Services Coordinators connect residents to a range of services related to self-sufficiency, including education, health care, job training, youth programs, financial literacy, and resources to assist with aging in place. The coordinated services depend on individual needs and available community resources. Service coordinators are encouraged to set goals and implement individual training and service plans (ITSPs) with residents to help them on their path to either economic self-sufficiency or aging in place. Unlike other HUD programs, such as Family Self-Sufficiency (FSS), however, ROSS-SC

does not require an individual contract with participants, provide financial incentives for participation, or have mandatory requirements for completing or graduating from the program. Many ROSS-SC

participants do complete an ITSP (see appendix A for an example), which provides a template for them to set goals and chart their progress, but it is not universal y used nor is it required to receive service coordination or referrals.

ROSS-SC Funding

Congress has appropriated an average $35.6 million, adjusted for inflation, 6 for ROSS-SC annually since 2008 (exhibit 3). The median number of grantees receiving a new 3-year contract each year is 113. Although ROSS-SC has ultimately received annual funding, initial Presidential or Congressional budget proposals regularly omit it, often in favor of other programs like Jobs Plus. On several occasions, Congress appropriated as much as $52 mil ion for ROSS-SC, but ultimately funneled about $10 mil ion of those funds for Jobs Plus and other public housing education programs (HUD 2014, HUD 2015). HUD has also stated that PHAs should use Operating Funds to support self-sufficiency activities instead of additional appropriated funds (NYPHA 2012, HUD 2012). PHA Operating Funds, however, are under strain because appropriations for the program have declined over the long term and are typically inadequate to fund large scale services beyond basic operations and maintenance.

ROSS-SC therefore continues to serve as a critical funding stream for service coordinators (Chavis 2018).

6 This and all further dol ar amounts are inflation adjusted to 2019 dol ars.

1 1

EXHIBIT 3: TIMELINE OF ROSS-SC GRANTEES AND APPROPRIATIONS

Dol ars (2019)

Funding Amount

Number of Grantees

Number of Grantees

$45,000,000

140

$40,000,000

120

$35,000,000

100

$30,000,000

80

$25,000,000

$20,000,000

60

$15,000,000

40

$10,000,000

20

$5,000,000

$-

0

2008

2009

2010

2011

2012

2013

2014

2015

2016

2017

2018

Fiscal Year

Sources: White 2010, White 2009, HUD 2011, HUD NOFAs 2007-2018

Notes: In 2014 and 2015, ROSS-SC received $52 mil ion through congressional appropriations. In 2014, however $9.7 million was then set aside for Jobs Plus. In 2015, $9.7 mil ion of ROSS-SC funds was also set aside for Jobs Plus, and an additional $2.2

mil ion went to an unnamed public housing education program. Dollars are adjusted for inflation to 2019.

Even after a grantee receives ROSS-SC funding to support or create resident-related services, program continuity is not ensured due to the fol owing reasons:



Grantees are not guaranteed that their grant wil be renewed at the end of the 3-year funding period. Though HUD has prioritized applications of repeat grantees to mitigate some of this uncertainty, funding is awarded on a competitive basis. HUD evaluates applicants based on past performance. Nonetheless, in the years 2015–2017, between 55 and 70 percent of total ROSS-SC funding went to repeat applicants (HUD 2018).



HUD reserves the right to lower funding amounts for renewal applicants based on a grantee’s financial need, capacity, funding balance, and prior performance (HUD 2018).



As of this writing, ROSS-SC grantees where units are converting from public housing to other funding streams through the Rental Assistance Demonstration (RAD) housing are uncertain about their ability to continue the program.

1 2



The program has been regularly zeroed out from Presidential and Congressional budget proposals, creating doubt around its sustainability as a source of future grants.

Reason for Assessment

Congress mandated this assessment of the ROSS-SC program in response to findings from a 2013

Government Accountability Office (GAO) report identifying gaps in reporting across HUD’s self-sufficiency programs, including ROSS-SC. GAO made multiple recommendations, including that HUD

better tailor reporting processes to ROSS-SC’s programming and that HUD more accurately capture and analyze data and therefore better assess program outcomes (GAO, 2013). GAO noted that grantee reporting was often inconsistent, and HUD program performance tracking and measurement needed improvement.

ROSS-SC grantees funded between 2008 and 2015 used an Excel-based reporting tool called a Logic Model to track the number of residents projected to receive, and the number who were ultimately connected to, services through ROSS-SC. GAO found participation data from FY 2008–

2011 were either missing or unreliable as a result of insufficient instructions for entering data in the Logic Model (GAO, 2013). For example, the 2011 Logic Model contained 121 different fields for entering activities and 99 options for outcomes achieved. No metrics were mandatory for reporting, so grantees left many fields blank. Additional y, no instructions were provided regarding how grantees should define and count the units for each metric, adding to reporting confusion. This made it impossible to produce a clear unduplicated count of program activities and services coordinated.

These shortcomings and problems with missing data meant HUD was not able to use the Logic Model data to accurately report on the number of residents ROSS-SC served between 2008 and 2011

(GAO 2013). Instead, HUD estimated the number of residents served based on the number of service coordinators funded by the program, the population of the site, and expected number of units to be served (GAO 2013).

In response to the findings of the GAO report, HUD began to implement improvements to its data collection methodology. These changes included updating the Logic Model starting in FY 2014. The FY 2014 Logic Model, which included fewer metrics, more detailed instructions, and a set of mandatory metrics, had 29 fields for services and activities and 34 outcome fields; grantees were required to provide data for 14 of each.

1 3

Since FY 2017, HUD has phased out the Logic Model in favor of the new Standards for Success reporting system, a method of reporting on discretionary-funded programs. The pilot began in FY

2017 with a subsample of HUD programs and participants, including some ROSS-SC grantees. It expanded in FY 2018 to include ROSS-SC grantees from FY 2016 and FY 2017, as well as the Multifamily Housing Service Coordinator Program grantees, and has now replaced the Logic Model for all subsequent ROSS-SC grantees (HUD n.d.g). The changes in reporting brought by Standards for Success were implemented during the course of this study and therefore are outside the scope of this evaluation. This assessment, however, aims to provide recommendations for future data col ection and reporting improvements with these current standards in mind.

Research Goals

This evaluation is one of the few studies that review HUD Service Coordinator programs, and the first study to do so for HUD’s ROSS-SC program. As such, it has the potential to influence the discussion of service coordination in the housing context, both within government programs and elsewhere. It is also intended to respond to the 2013 GAO report’s call for clearer and more consistent data reporting and outcome evaluation. Specifically, this report is both a process evaluation and a modified outcome evaluation7 and has three objectives:

1. To assess the efficacy of the changes in program reporting since changes were made to the program’s Logic Model in FY 2014.

2. To examine the breadth and depth of ROSS-SC program implementation by current service coordinators across all grantee types.

3. To analyze reporting requirements and performance metrics to suggest how they could be revised to improve future data col ection and program evaluations.

Our full list of research questions is provided in exhibit 4. We addressed these questions through a mix of original data col ection obtained through several methods, described in the list below and in greater detail in chapter 2.

7 For the purposes of this report, a modified outcome evaluation seeks to identify areas of potential impact for resident participants in ROSS-SC services and for grantee organizations. A ful outcomes evaluation was not possible due to data limitations. Instead, this report discusses what evidence exists that indicates potential successes and failures and provides recommendations for how to col ect data moving forward to strengthen future evaluation efforts.

1 4



Site visits to 10 distinct grantees, during which the research team conducted interviews with PHA staff, grant administrators, service coordinators, local service partners, and focus groups with ROSS-SC participants.



A national survey of all service coordinators employed through grants from FY 2014–FY

2016. 8



Analysis of administrative data:

» Repeat grantee analysis: reported data from a subset of grantees that received grants in both FY 2011 and FY 2014 and therefore had used both versions of the Logic Model for reporting.

» Performance period analysis: 3 years of reported data from FY 2014 grantees.

» Application vs. Logic Model analysis: data from applications for FY 2014 grantees compared with data they reported post-award.

8 Several questions on the Office of Management and Budget (OMB)-approved survey were omitted when fielded, limiting the research team’s abilities to answer some research questions using survey data. Answers were triangulated from other survey questions or data sources. See chapter 2 for a ful er discussion.

1 5

EXHIBIT 4: RESEARCH QUESTIONS FOR RESIDENT OPPORTUNITY AND SELF-SUFFICIENCY PROGRAM

EVALUATION

Program Structure and Processes

How did implementation of the FY 2014 Logic Model affect grantee reporting and outcomes? How did data quality and tracking change? How did grantee processes for populating the Logic Model change? How did grantee processes for retaining data and documentation change?

What logic and processes are used by other HUD Service Coordinator programs, how do they compare with ROSS-SC, and what lessons can be learned to improve ROSS-SC?

How wel do the ROSS-SC grantee annual reporting requirements define and measure grantee activities and performance outcomes? Have the mandatory metrics changed the structure of the ROSS-SC program and the services that are being coordinated? Did the mandatory metrics help? How adequate are the resources to support grantees’ effectiveness? Do the grantees have the capacity to col ect data for reporting requirements?

What evaluation tools are grantees using to measure performance? What outcome measures are used?

How do ROSS-SC program and staff fit into the organizational structure of grantee?

Which populations are targeted? How do participants learn about the program? Who decides to participate?

What are their characteristics?

What services are proposed versus the services actually being offered to program participants, and how are these managed and coordinated?

How are participants tracked over time and with what type of data and systems? Who sets targets for and implements the Logic Model, and who reports corresponding outcome data? What types of records are kept on participants, and how long are these records kept?

Service Coordinator Characteristics & Functions

What are their qualifications (education, experience), and how are they compensated?

What type of training and technical assistance do they receive from grantee, Office of Field Operations, and Field Office staff?

What are their caseloads, active assessment and service plans, frequency of meetings with program participants, contacts with agencies, referrals to services completed and accepted, and participant turnover rates?

What recommended functions out of the 10 specified by HUD in the program Notice of Funding Availability (NOFA) do they perform versus what additional functions do they add?

How do they assess program and participant needs? How do they develop, implement, and track progress on Individual Training Service Plans that set goals and benchmarks for individual participants?

What are the types, strength, and quality of partnerships between service coordinators and local service providers? How well do the local service providers meet community needs?

Outcome Evaluation

What influences success?



Capacity of grantee.



Capabilities of service coordinator.



Characteristics of program participants.



Availability and coordination of services.



Local economic context.



Presence of other programs.



Additional factors.

What are outcomes for participants?



Increase in participant incomes, by population served.



Success in participants meeting goals in their Individual Training Service Plans, by grantee type, by participant type.



Frequencies and types of referrals and service take-up rates.



Additional outcomes.

In what ways are programs leveraged to improve effectiveness and success rates of participants?



Providing referrals to services—differences by grantee type and population served.



Sharing services with other government-funded programs and avoiding duplication.



Additional leveraging.

1 6



Resources and technical assistance, or TA, needed by grantees.

Report Overview

Throughout the report’s seven chapters, we describe how ROSS-SC is implemented in communities around the country from the perspective of service coordinators and the 10 sites we visited. We assess how implementation of the FY 2014 Logic Model affected both grantees’ activities and reporting.



Chapter 1 includes the Introduction, program history, and overview of the context and need for the report.



Chapter 2 summarizes our data col ection and analysis methods, including the challenges we encountered.



Chapter 3 discusses how ROSS-SC programs are structured locally, including what types of residents ROSS-SC service coordinators support, how service coordinators find and engage residents, and what types of services grantees provide or link to residents.



Chapter 4 focuses on the effectiveness of performance tracking and reporting using the FY

2011 and FY 2014 Logic Models and grantee application data.



Chapter 5 details the functions of the service coordinators, including their day-to-day tasks and responsibilities, their caseloads and contacts with residents, and the work they do to build partnerships in their communities.



Chapter 6 discusses resident experiences and contextual factors that influence the ability of individual ROSS-SC grantees to achieve desired outcomes for residents.



Final y, chapter 7 discusses the implications of our study for ROSS-SC residents, grantees, and the program office, and offers suggestions of ways that the program could be changed to strengthen desired outcomes, data tracking, and future evaluation efforts.

1 7

Chapter 2. Methodology

Introduction

In order to best understand the breadth of program implementation of ROSS-SC across the country as wel as the details of program operations on the ground, the evaluation team used a mixed-methods approach. The study combined a qualitative analysis of data col ected during a series of 10 site visits, a national survey of service coordinators, and quantitative analysis of administrative data submitted by grantees via a program Logic Model. This chapter provides an overview of these three methods and how they were used to answer the evaluation questions. Most research questions were answered by data spanning multiple data col ection and analysis methods. Exhibit 5 shows the research questions along with the primary data sources for addressing them. The remainder of this chapter discusses each method in depth and in the order of implementation.

EXHIBIT 5: PRIMARY DATA SOURCE FOR KEY RESEARCH QUESTIONS

Survey of

Administrative

Service

Key Research Questions

Data Analysis

Site Visits

Coordinators

Program Structure and Processes

How did implementation of the FY 2014 Logic Model affect

x

x

grantee reporting and outcomes?

How did data quality and tracking change?

x

x

How did grantee processes for populating the Logic Model

x

x

Change?

How did grantee processes for retaining data and

x

documentation change?

What logic and processes are used by other HUD service

coordinator programs, how do they compare with ROSS-SC,

x

and what lessons can be learned to improve ROSS-SC?

How wel do the ROSS-SC grantee annual reporting

requirements define and measure grantee services and

x

x

activities and performance outcomes?

Have the mandatory metrics changed the structure of the

ROSS-SC program and the services that are being

x

x

coordinated?

Did the mandatory metrics help?

x

How adequate are the resources to support grantees’

x

effectiveness?

1 9

Survey of

Administrative

Service

Key Research Questions

Data Analysis

Site Visits

Coordinators

Do the grantees have the capacity to col ect data for

x

reporting requirements?

What evaluation tools are grantees using to measure

x

performance?

What outcome measures are used?

x

x

How does the ROSS-SC program and staff fit into the

x

organizational structure of grantee?

Which populations are targeted?

x

x

How do participants learn about the program?

x

Who decides to participate?

x

What are the characteristics of participants?

x

What services are proposed versus actually being offered to

program participants, and how are these managed and

x

x

coordinated?

How are participants tracked over time, with what type of

x

data and systems on site?

Who sets targets for and implements the Logic Model, and

x

who reports corresponding outcome data?

What types of records are kept on participants, and how long

x

are these records kept?

Service Coordinator Characteristics & Functions

What are service coordinators’ qualifications (education,

x

experience)?

How are service coordinators compensated?

x

What type of training and technical assistance do service

coordinators receive from grantee, Office of Field Operations,

x

x

and Field Office staff?

What is a typical service coordinator caseload?

x

x

x

How many active assessment and service plans do service

x

x

x

coordinators maintain?

How frequently do service coordinators meet with program

x

x

participants?

How frequently do service coordinators have contacts with

x

x

x

partner agencies?

How frequently are referrals to services completed and

x

x

x

accepted?

How high are participant turnover rates?

x

What recommended functions out of the 10 specified by

HUD in the program NOFA do service coordinators perform,

x

and what additional functions do they add?

2 0

Survey of

Administrative

Service

Key Research Questions

Data Analysis

Site Visits

Coordinators

How do service coordinators assess program and participant

needs?

x

x

How do service coordinators develop, implement, and track

progress on individual Training Service Plans which set goals

and benchmarks for individual participants?

x

x

What are the types, strength, and quality of partnerships

between service coordinators and local service providers?

x

x

Outcome Evaluation9

What influences success?

What are outcomes for participants?

x

x

x

In what ways are partner programs leveraged to improve

effectiveness and success rates of participants?

x

x

NOFA = notice of funding availability.

Source: ROSS-SC Evaluation RDDCAP, FY 2014-FY 2016 Logic Model data, site visit data, and survey of service coordinators.

At the outset of this project, HUD tasked the evaluation team with conducting an analysis of outcome data from individual ROSS-SC grantees that they may collect in addition to required HUD reporting. This did not seem to be a common practice across the sites visited, and we were unable to design a timely method with HUD for identifying grantees that might have such data. The evaluation team and HUD decided not to pursue this analysis; thus the research questions related to outcomes were reduced compared with the other categories of questions.

Data col ection for each of our three methods occurred between June 2017 and December 2018. As can be seen in exhibit 6, much of this work occurred simultaneously.

EXHIBIT 6: TIMELINE OF DATA COLLECTION ACTIVITIES

2017

2018

J J A S O N D J F M A M J J A S O N D

Administrative Data Analysis

X X X

X X

Grantee Survey Fielded

X X X

Survey of Service Coordinators

Fielded

X X X X

Site Visits

X X X

X X X

9 At the outset of this project, HUD tasked the evaluation team with conducting an analysis of outcome data from individual ROSS-SC grantees that they may col ect in addition to required HUD reporting. This did not seem to be a common practice across the sites visited and we were unable to design a timely method with HUD for identifying grantees that might have such data. The evaluation team and HUD decided not to pursue this analysis, thus the research questions related to outcomes were reduced compared with the other categories of questions.

2 1

Site Visits

Overview

The evaluation team used the site visits to gain perspective on how the ROSS-SC program operates on the ground across a range of environments and grantee types. In addition to getting a first-hand look at the program in operation with 12 grantees across the country (two reconnaissance site visits, three pilot site visits prior to Office of Management and Budget (OMB) approval, and seven additional visits after approval), the site visits provided the evaluation team with experience that al owed them to better understand survey responses and the results of their data analysis. Interviews with multiple program stakeholders while on site helped the team to understand the activities, partnerships, and outcomes of the ROSS-SC program and to answer research questions regarding program structure and process, service coordinator characteristics, functions, and program outcomes. Through gaining a deeper understanding of general chal enges on the ground from site visits, the team learned how individual grantees approach their own data and reporting practices.

Reconnaissance Site Visits and Scoping

The evaluation team conducted two reconnaissance site visits to gain a better understanding of how ROSS-SC is implemented by grantees and to better inform the development of Urban Institute and EJP Consulting Group’s data col ection that would be used to conduct this study. These preliminary site visits provided the team with the grounding in the ROSS-SC program and relevant contextual factors to enhance the interview protocols, determine the site sampling criteria, develop an effective approach for resident focus groups, and refine site visit logistics to reduce burdens on each of the sites that would be visited and make efficient use of project resources. In addition to these two preliminary site visits, representatives from the evaluation team met with the HUD District of Columbia Field Office to better understand the HUD structure and systems supporting the ROSS-SC program and grantees.

Site Selection and Site Visit Preparation

The evaluation team selected a collection of sites that would, as much as possible, reflect the diversity of the ROSS-SC program grantees. As the initial charge was to review the data col ection and reporting practices for the FY 2014 grantees, the team selected sites from the universe of 77 FY 2014

2 2

grantees that had also received FY 2011 grants—PHAs, nonprofits, resident associations, and TDHEs—

that met basic criteria for study inclusion. The team selected sites to maximize distribution by geographic region, grantee type, PHA size, and whether the community is within or outside of a metropolitan statistical area (MSA). The team also included one TDHE grantee. 10 The members also limited sites with high numbers of RAD units, as it was unclear at the time (and still is at the time of this report) whether these sites would be able to continue to receive ROSS-SC support once the units converted from Section 9 funding to Section 8. Exhibit 7 shows the distribution of FY 2014 grantees within the selection categories and the distribution of the final sites visited.

EXHIBIT 7: DISTRIBUTION OF FY 2014 GRANTEES AND SAMPLES BY SELECTION CRITERIA

FY 2014 Grantees

Final Site Visit Sample

Selection Criteria

Category

Number (%)

Number (%)

Grantee Type

PHA

56 (73)

5 (50)

Nonprofit Organization

10 (13)

2 (20)

Resident Association

7 (9)

2 (20)

TDHE

4 (5)

1 (10)

PHA Size

Large (1,250 Units or More)

15 (20)

5 (50)

Medium (250 to 1,249 Units)

40 (52)

2 (20)

Small (249 Units or Fewer)

22 (29)

3 (30)

Geographic Region

Northeast

22 (29)

3 (30)

Southeast

16 (21)

2 (20)

Midwest

21 (28)

2 (20)

Northwest

4 (5)

2 (20)

Southwest

13 (17)

1 (10)

Community Type

In MSA

70 (91)

7 (70)

Not in MSA

7 (9)

3 (30)

MSA = metropolitan statistical area. PHA = public housing agency. TDHE = Tribal y Designated Housing Entity.

Sources: HUD list of 2014 ROSS-SC grantees and 2017 and 2018 site visits.

Notes: One PHA in an outlying territory was not included in this table. Percentages may not add to 100 percent due to rounding.

The evaluation team took several steps in preparation for site visits, including the fol owing:



Designed interview protocols tailored to interview service coordinators, grant managers, and partners (see appendix B).



Designed focus group protocols to provide key contextual evidence about resident needs and to inform answers of research questions related to program processes, structure, and outcomes (appendix C).

10 To preserve the confidentiality of site visit respondents within this grantee when quoting them in this report, we designate them instead as a “lower-resourced PHA” in a category with several other PHAs to avoid singling them out. This does not affect the interpretation of the quotes.

2 3



Conducted background research to assess the particular context of each grantee program, developing a site profile prior to arriving on site.



Worked with service coordinators at the sites to recruit residents for focus groups by publicizing the event through flyers and word of mouth.

Site Visit Procedures

On the site visits, team members interviewed ROSS-SC service coordinator(s), ROSS-SC grant managers, ROSS-SC partners, and where applicable, other PHA or grantee organization staff and resident council presidents. In addition, the team facilitated one focus group with residents at each site. Due to mixed results in recruitment and availability of ROSS-SC participants, focus group participation ranged from 2 to 11 residents. A complete tal y of individuals interviewed by site is available in exhibit 8.

EXHIBIT 8: SITE INTERVIEWS

PHA Manager/

ROSS-SC

Staff/Resident

Number of

Service

Grant

Association

Focus Group

Coordinators

Administrators

President

Partners

Participants

Site 1

2

2

1

3

9

Site 2

2

1

0

4

8

Site 3

1

1

0

4

6

Site 4

1

1

1

2

8

Site 5

2

0

1

4

8

Site 6

1

0

1

3

2

Site 7

1

1

2

2

5

Site 8

2

1

0

3

11

Site 9

2

1

0

3

4

Site 10

2

1

1

3

9

Total

16

9

7

31

70

Source: 2017 and 2018 site visits

ANALYSIS

On completion of each site visit, the evaluation team reviewed and cleaned their notes from the interviews and focus groups and saved them to a secure server at the Urban Institute. After the majority of site visits were complete, the team developed an analytical rubric, or codebook, to categorize the information col ected and tie it to the initial research questions. The team built the codebook on the structure of the interview and site visit protocols and adjusted categories based on observations from the site visits. The evaluation team then created a crosswalk between the codebook categories and the project’s key research questions as identified by HUD. Although this codebook 2 4

formed the basis for the analysis of al site visit data, the team used it as a living tool, al owing for alterations to categories as members reviewed the material and the creations of new categories as members identified new themes.

The evaluation team then analyzed al cleaned interview and focus group notes, categorizing text based on the codebook in NVivo qualitative data analysis software. NVivo also allowed the team to categorize each set of notes by interview subject (service coordinator, grant manager, resident focus group, partner, or other grantee staff), and grantee type (PHA, nonprofit organization, TDHE, or resident association). This structure allowed the team to run reports producing al data by codebook category or separated by interview subject or grantee type.

To analyze the qualitative data, the evaluation team studied the text coded to each thematic category (or node) and developed key takeaways. While reviewing each node, team members matched data to the appropriate research question and identified il ustrative quotations from respondents for potential use in the final report. As a final step, the team gathered all material matched to each research question and organized relevant findings.

Survey of ROSS-SC Service Coordinators

Overview

The evaluation team designed the survey of ROSS-SC service coordinators to answer research questions regarding program structure and process, service coordinator characteristics, functions, and outcomes. The information received from the survey provided the evaluation team with generalizable knowledge on service coordinators to complement the data gathered in the site visits.

Survey Design

SURVEY TOOL DEVELOPMENT



In designing the service coordinator survey, the evaluation team developed the survey questions to maximize the useful information col ected and to match the key research questions as defined by HUD. See appendix D for the complete survey. The following are samples of some of the topic areas covered in the survey:



Service coordinator qualifications and experience.

2 5



Whether the ROSS-SC grant funds all of their activities or whether they have responsibilities outside the ROSS-SC grant.



Average caseloads.



The frequency of service coordinator contacts with residents served.



The types of functions and activities performed by the service coordinator.



Specific populations targeted for service coordination.



Service needs among residents.



Resources available to the service coordinator.



The nature of partnerships and other relationships with service providers.



Basic demographic characteristics of the service coordinators: age, gender, race/ethnicity, and education level.

After the initial draft of service coordinator survey questions was complete, the team performed pretesting and interviews with four service coordinators at three sites to ensure survey questions were easy to understand and to address any feedback. Based on the testing, the survey team made changes in the wording of a small number of questions. Pretesting also revealed that the survey took longer to complete than originally planned so the evaluation team cut several questions in areas where data could be gathered from other sources.

Preparation for Fielding

As HUD has a relationship with the grantee organization (i.e. a PHA) and the grantee organization then hires the service coordinators, HUD does not obtain or maintain a list of active service coordinators funded by ROSS-SC grantees, or their contact information, the evaluation team determined that they needed to col ect service coordinator names and contact information directly from current grantees.

To do so, the team created a basic survey to be sent to HUD’s ROSS-SC grantee points of contacts.

Due to delays in government review and approval of the survey materials by HUD and the Office of Management and Budget, this preliminary survey of grantees was fielded later than expected, and some ROSS-SC grants that were active for the original fielding timeline had ended. In response, the evaluation team changed the grantee survey population to those grantees funded in FY 2014–FY

2 6

2016 instead of the original y planned FY 2013–FY 2015 grantees, as the FY 2013 grants were no longer active. Because of this change, the team ultimately fielded the grantee survey to 331 grantees, similar in number to the originally estimated 330 grantees from the FY 2013–FY 2015 cohorts, but with some differences in population.

In total, 236 of the grantees (71 percent) responded to the contact survey. The survey ultimately yielded contact information for 351 service coordinators. Once the grantee survey was closed, the evaluation team used the results to field the service coordinator survey.

SURVEY FIELDING

Using the contact information col ected from the grantee contact survey, the evaluation team launched the service coordinator survey in August 2018 to service coordinators funded by FY 2014–

FY 2016 ROSS-SC grants. Throughout the fielding, the team regularly analyzed the responses to ensure that response rates were appropriately distributed across grantee types, geographic location, and populations served. Due to lower-than-expected response rates across subgroups, the team took steps to ensure that all service coordinators were receiving the initial request email. This effort yielded an increased response rate.

The evaluation team closed the service coordinator survey in the middle of November 2018, for a fielding period of 3.5 months. The final response rate was 61 percent (215 service coordinators).

Although this was lower than targeted, the response rate across subgroups remained consistent with national proportions, suggesting that the data col ected were representative of the full population of service coordinators and grantee type. As seen in exhibit 9, most responses came from service coordinators at PHAs (76.3 percent), fol owed by resident associations (13 percent), nonprofit organizations (7.5 percent), and TDHEs (3.2 percent).

EXHIBIT 9: DISTRIBUTION OF SERVICE COORDINATORS WHO RESPONDED TO THE SERVICE

COORDINATOR SURVEY BY GRANTEE TYPE

Grantee Type

Number of

Percent of

Responses

Responses

Public Housing Agency

164

76.3

Resident Association

28

13.0

Nonprofit Organization

16

7.5

Tribal y Designated Housing Entity

7

3.2

Total

215

100

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

2 7

Survey Analysis

To make reliable and robust conclusions about the active service coordinator population based on the survey data, the evaluation team summarized the survey responses by grantee year and population(s) served. The team then developed a grantee weight based on the total universe of grantees and the response rate by categories. Specifical y, they identified the distribution of grantees across the categories of grant year and service type from the HUD contact list as identified by the analysis of the FY 2014–FY 2016 application forms, then divided that by the distribution of grantees where at least one service coordinator completed the survey. For a full discussion of survey weights and methods of analysis, see appendix E.

Chal enges in Survey Fielding and Analysis

There were two major challenges in survey fielding and analysis. First, the service coordinator survey as it was fielded differed slightly from the survey as approved by OMB. The approved survey as programmed into the software was accidental y lost during staff turnover, resulting in an earlier version mistakenly being fielded. The impact on overal response burden was negligible. In addition to missing a few response categories, the primary omissions were (1) questions on services to disabled residents and to non-English speakers, and (2) a question on the type of residents living at the property. We were able to triangulate answers in most cases from other data col ected, although we missed the opportunity for more specific answers. A ful list of question variations is provided in appendix F.

Second, the smaller-than-expected sample size of the service coordinator survey limited the team’s ability to do subgroup comparisons or any correlation analysis by grantee type, including TDHEs or nonprofit grantees. After adjusting for grantee organization type, the resulting population for certain grantee organization type subgroups was extremely small and could not be reported without infringing on the confidentiality of the individual service coordinator respondents. This problem limited the team’s ability to provide in-depth subgroup analysis in certain areas. Looking at a breakdown of service coordinators responding by grantee organization type shows that responses from TDHE and nonprofit organization grantees were so few that the team was unable to draw conclusions about the differences across these types.

2 8

Administrative Data Analysis

Overview

The evaluation team used three sources of HUD administrative data in their analysis: the FY 2014

ROSS-SC applicant data from HUD Form 52769, where applicants identify the service needs of their residents and the service partners they intend to work with (see appendix G), FY 2011 Logic Model data submitted by FY 2014 grantees, and 3 years of Logic Model data from FY 2014 grantees using the revised Logic Model, submitted over their FY 2014–FY 2016 period of performance (appendix H).

The FY 2014 ROSS-SC application form asks that grantees identify the needs they propose to address across 32 categories, such as childcare and transportation; it also includes a self-defined

“other” category (exhibit 10). Applicants only indicate whether they anticipate a service need and whether they have a partnering agency to provide the service. Applicants provide no indication of the level or intensity of the service on the form. See appendix I for more information on the service classifications on the Application and the Logic Model.

EXHIBIT 10: FY 2014 ROSS-SC APPLICATION FORM: NEEDS ROSS-SC GRANTEES PLAN TO ADDRESS

 Life Skil s Training

 Nutrition classes

 Financial Literacy/Credit

 Youth Programming: Tutoring/Mentoring/

 Counseling/Credit Repair

After School/Summer

 Literacy Training

 Childcare





ESL

Transportation





GED/High School Equiv.

Tax Preparation Assistance

 Mentoring

 Community Safety

 Job Soft Skil s Training

 Resident Empowerment/Capacity Building

 Job Hard Skills

 Resident Business Development

 Training/Certification

 Assistance with Activities of Daily Living

 Job Search and Placement

 Meals to Meet Nutritional Needs for Elderly

 Job Retention/Promotion

 Disability Services Counseling

 ISAs/IDAs*

 Personal Emergency Response Resources

 Homeownership

 Wel ness Programs





Counseling

Other (please describe)



Computer Classes

Source: HUD Form 52769: ROSS Service Coordinators—Needs and Service Partners (6/2015).

Note: Categories are listed here in the same order they appear on the form. *Individual Savings Accounts/Individual Development Accounts

2 9

The FY 2011 Logic Model data reflect grantees’ self-reported projected and actual service amounts across 121 service and activity metrics and 99 outcome metrics. None of the metrics on the Excel-based form are mandatory. Grantees self-select which metrics to report for each grant year during the period of performance. Any metrics that grantees do not select have no values reported at all.

The Logic Model data for the FY 2014–FY 2016 period of performance come from a revised Logic Model that HUD implemented for the FY 2014 ROSS-SC grantees. This Excel-based Logic Model has a comparatively reduced set of metrics: 29 service and activity metrics and 34 outcome metrics.

Fourteen of both the outcome metrics and the service and activity metrics are mandatory, requiring every grantee to report a service amount or outcome, or zero if it is not applicable. The remainder of the metrics are optional. Any optional metrics that grantees do not select have no values reported. See appendix J for instructions on completing the FY 2014 Logic Model.

The evaluation team conducted three distinct quantitative analyses.

1. Repeat grantee analysis: A comparison of grantees’ projected and actual services and activities as reported in the first year of FY 2011 grantee Logic Model data versus the comparable projected and actual services and activities they reported 3 years later using the updated FY 2014 Logic Model.

2. Performance period analysis: An analysis of FY 2014 grantees’ reported ROSS-SC services and activities and outcomes over their 3-year FY 2014–FY 2016 grant performance period, based on Logic Model data across all 3 years.

3. Application vs. Logic Model analysis: A comparison of grantees’ proposed ROSS-SC services and activities on their FY 2014 grant application versus the projected services and activities they input into their FY 2014 Logic Model.

The sample size of grantees differs for each analysis according to the constraints necessary to perform the analyses and the data that were available to the evaluation team. The samples are summarized below (exhibit 11).

EXHIBIT 11: SUMMARY OF GRANTEE SAMPLES ACROSS THREE QUANTITATIVE ANALYSES

Analysis

Application vs. Logic

Repeat Grantee

Performance Period

Model

(Number of Grantees)

(Number of Grantees)

(Number of Grantees)

Initial Sample

88

106

89

Reason for Removal

3 0

Grants Rescinded

7

4

0

No Available Data

14

13

8

Sample for Analysis

67

89

81

Sources: FY 2014-FY 2016 Logic Model data and HUD communications.

Repeat Grantee Analysis

SAMPLE

With the repeat grantee analysis, the team sought to understand whether and how HUD’s changes to the Logic Model’s structure in FY 2014 affected data quality. As a first step to isolate the effects of the revised Logic Model implementation, the evaluation team limited the initial sample exclusively to ROSS-SC grantees that received grants in both FY 2011 and FY 2014. The evaluation team further refined the sample, only keeping grantees who reported on metrics in the FY 2011 Logic Model that are analogous to the mandatory metrics in the FY 2014 Logic Model. After removing grantees with missing data, the team ended with a final sample of 67 grantees for this analysis.

The final sample was almost entirely made up of Public Housing Agencies (PHAs, 58 out of 67

grantees, 87 percent). Of the remaining nine non-PHA grantees, two grantees are TDHEs, four are resident associations, and three are nonprofit organizations. The very small sample size of non-PHA grantees precluded analysis by grantee type, as any results would not yield generalizable findings.

METHODS

Due to evidence of unreliable data values, the evaluation team recoded all service and activity data values to binary codes, labeling any positive value as “yes” (1) and any zero value or missing value as

“no/missing” (0). This al owed us to focus on analyzing whether any data had been input, rather than using more unreliable specific values. Using a crosswalk to compare analogous variables across both Logic Models, the evaluation team summed the number of service and activity metrics coded as “yes”

for each grantee, in each year. The difference between the FY 2014 and FY 2011 Logic Models’ values informed findings on how much more data were available in the FY 2014 Logic Model (see exhibit 21).

3 1

Performance Period Analysis

SAMPLE

With the performance period Logic Model analysis, the evaluation team sought to understand grantees’ services and activities over time. In selecting the sample for this analysis, the team only kept grantees that reported data across FY 2014, FY 2015, and FY 2016 in order to have the best opportunity to see patterns in the data, leaving 89 grantees.

The sample is skewed toward PHAs, with 69 of the 89 grantees being PHAs (78 percent). Of the 20 remaining non-PHAs, 3 grantees are TDHEs, 7 are resident associations, and 10 are nonprofit organizations. Because the number of some grantee-types (i.e. TDHEs) was too small, we could not conduct a segmentation analysis for each type of non-PHA grantee separately. However, some analyses contain a comparison of PHA (69 grantees) and non-PHA (20 grantees).

METHODS

The evaluation team coded all missing data as “0” to perform the analyses of FY 2014 grantee-reported services and activities and outcomes using the Logic Model data provided. Depending upon the research question, the evaluation team did one of the following:



Recoded the service/activity data to binary codes, labeling any positive value as “yes” (1) and any zero value or missing value as “no/missing” (0), to determine whether a grantee engaged in a service or activity or achieved an outcome.



Used the actual value reported to quantify the grantees’ projected or actual services, activities, and outcomes.

For analyses that used the actual number provided, the evaluation team transformed the data in different ways depending on the research question. Methods included using the data as-is to assess the amount of discrete service or activity for a particular metric; combining data across metrics to arrive at an aggregate number for a set of related services and activities; and comparing projected and actual services and activities, projected and actual outcomes, projected services and activities and projected outcomes, or actual services and activities and actual outcomes.

3 2

Application vs. Logic Model Analysis

SAMPLE

With the application vs. Logic Model analysis, the team sought to uncover how grantees’ proposed services and activities differed from what they delivered to clients. For consistency, the team used the 81 grantees in the performance period analysis sample of FY 2014 grantees for which we had complete application forms.

SCOPE OF ANALYSIS

For the descriptive analysis, we only used data from the FY 2014 applications and the Logic Model for grant year FY 2014. Because the applications show grantees’ intended services prior to receiving ROSS-SC funds, they offer a view of grantees’ proposed services at a point when they were not bound by the Logic Model’s new reporting requirements. By analyzing the degree to which the applications and the Logic Model data overlap or are different, we were able to infer grantee-level changes in anticipated service.

METHODS

We used the service needs marked on the grantees’ FY 2014 applications to split grantees into three categories: family-serving grantees, elderly/disabled-serving grantees, and grantees that served both.

To assign grantees to the categories, the evaluation team first used the needs marked on the FY 2014

application that had analogues on the FY 2014 Logic Model. This strategy was used to ensure that there would be one-to-one alignment between the two forms. The team assigned the labels as fol ow:



Family-serving grantees are those that marked a need for childcare.



Elderly/disabled-serving grantees are those that marked a need for assistance with activities for daily living, meals to meet nutritional needs for the elderly, or disability counseling services.



Both indicates grantees who marked needs for one or more families and for elderly/disabled residents.

The team adjusted the classification for 10 grantees that could not be classified in this way to reach an appropriate approximation using HUD documentation and expanded Logic Model metrics. The evaluation team classified grantees by their FY 2014 Logic Model data, using the same grantee-type 3 3

categories, for services and activities. We display the comparison between these groups in chapter 3, exhibit 14.

Data Chal enges

Several data chal enges informed the evaluation team’s sampling strategies and the analyses, as outlined below:



Grantees are able change their Logic Model data after they have submitted it to HUD. To deal with this issue, the Urban Institute, in col aboration with HUD, established a cut-off date of September 5, 2017, for the Logic Model data.



Preliminary analysis of the Logic Model data showed unexplained, large discrepancies between projected and actual services and activities. To circumvent the problems with unreliable values, the evaluation team shifted to a binary approach that recoded grantees’

reported values as present (1) or missing (0). This strategy de-emphasizes the exact values of the services, activities, and outcomes, making their unreliability far less of an analytical concern.



Multiple metrics across both Logic Models have no analogue, or to the extent that they do, some definitions differ. Therefore, the evaluation team created a crosswalk to compare the FY

2011 and FY 2014 Logic Models (see appendix K for crosswalk). Only mandatory metrics from FY 2014 that have a corresponding metric in the FY 2011 Logic Model are covered in this analysis.



Although the research team had initial y planned to use expenditure data from HUD’s Line of Credit Control System (LOCCS) data showing grantee expenses over time, they found that these data were not usable for this evaluation. Because the grant drawdowns did not contain a description of the actual services being provided, the evaluation team determined that LOCCS data would not yield meaningful or generalizable results. The evaluation team also did not use the LOCCS data for the performance period analysis because the data could not be aligned with grantees’ performance periods.

3 4

Chapter 3. Residents and Services

Introduction

A primary goal of the ROSS-SC program is to connect public housing residents to services in their communities provided by outside organizations. In this chapter, we examine how service coordinators target grantees for services, to what services they typical y refer residents, how they find and maintain partnerships with organizations that provide those services in their communities, and how service coordinators col aborate with other PHA staff to improve service delivery.

This chapter responds to the fol owing key research questions:



Which populations are targeted?



How do participants learn about the program?



Who decides to participate?



What are participants’ characteristics?



How does the ROSS-SC program and staff fit into grantees’ organizational structure?



What are the types, strength, and quality of partnerships between service coordinators and local service providers?



How well do the local service providers meet community needs?



What services are proposed versus actually being offered to program participants, and how are these managed and coordinated?

Key Findings



Grantees conducted outreach to a wide range of residents, including unemployed residents, residents with physical disabilities, working residents, elderly residents, families with children, and residents with mental health needs.

3 5



Outreach took a variety of forms, including door-to-door engagement, flyers, resource fairs, and organized activities and celebrations. Unemployed residents were most likely to participate, while non-English-speaking residents were least likely.



Grantees most commonly referred residents to health and adult education services. During site visits, service coordinators also highlighted that they provide linkages to food, transportation, and youth programming.



Service coordinators maintained partnerships through local Program Coordinating

Committees, preexisting community networks, and on a one-on-one basis. They worked with a wide variety of organizations, both local and national, to develop formal partnerships through Memoranda of Understanding as wel as informal agreements.



Service coordinators regularly identified new partners to meet emergent needs. Generating or accessing community resource lists helped.



The ROSS-SC grant was often the sole source of funds for service coordination within lower-resourced grantees, al owing residents access to activities and services they would not have otherwise had (see exhibit 12).



Service coordinators and PHA property management may benefit by working together, with property managers alerting service coordinators of residents in crisis, and service coordinators helping residents avoid lease violations, missed rent payments, and evictions.

There were also some notable chal enges encountered:



Grantees in rural and otherwise isolated contexts may have more difficulty finding partners to ful y cover resident needs.



Some gaps remained between the partners that service coordinators identified and those partners’ ability to meet resident demand for services, with the largest gap between the demand for mental and behavioral health services and the capacity of community service partners to meet it.



Data sharing was uncommon within partner agreements, creating challenges for service coordinators seeking to track resident use of partner services and outcomes.

EXHIBIT 12: GRANTEE CLASSIFICATIONS

Site visits revealed two levels of resources available to grantees. As a result, this report wil refer to grantees as one of the following:

3 6



Lower-resourced grantees: refers to PHAs with no other federal service coordinator funding and operating in communities with few service partners due to population size and geographic isolation.



Better-resourced grantees: associated with PHAs with other federal service coordinator funding such as FSS and within a larger, service-rich community. These grantees often had funding to provide direct services themselves through other government or philanthropic funding.

Targeting and Engaging Residents

Target Populations

GRANTEES TARGET A WIDE CROSS-SECTION OF RESIDENTS

The survey results show that most grantees reach out to residents of different ages, needs, and life circumstances, reflecting the ROSS-SC program’s broad scope on the ground (exhibit 13). At least three out of four grantees we studied conduct outreach to unemployed residents (85 percent), residents with physical disabilities (80 percent), working residents (79 percent), elderly residents (79

percent), families with children (78 percent), and residents with mental health needs (76 percent).

More than one-half of respondents conduct outreach to non-English-speaking residents.

EXHIBIT 13: TYPES OF RESIDENTS GRANTEES TARGET

Grantees (%) (n=178)

Type of Resident

Unemployed Residents

85

80

Residents with Physical Disabilities

Working Residents

79

Elderly Residents

79

Families with Children

78

76

Residents with Mental Health Needs

Non-English-Speaking Residents

54

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: This table reflects 215 service coordinators’ responses, which have been weighted and aggregated to the grantee level.

The total does not sum to 100 percent because the categories are not mutually exclusive.

3 7

GRANTEES TEND TO SERVE BOTH FAMILIES AND ELDERLY/DISABLED RESIDENTS

Grantees’ FY 2014 Logic Models show that they have targeted a broader population than what they originally projected in their grant application. Grantees’ target populations generally include families as wel as seniors and residents with disabilities in their Logic Model service projections, even if their ROSS-SC grant applications had specified serving just one or the other. For example, childcare service projections most likely indicate a family-serving grantee. Although only 38 percent of FY 2014

applicants proposed services for both families and seniors/residents with disabilities in their grant application, 60 percent of grantees projected coordinating services for both groups in their FY 2014

Logic Model reports (exhibit 14).

EXHIBIT 14: TYPES OF RESIDENTS TARGETED ON FY 2014 APPLICATIONS AND LOGIC MODEL

PROJECTIONS

Application

Logic Model

100%

80%

60%

60%

39%

38%

40%

25%

23%

20%

15%

0%

Families only

Seniors/residents with disabilities

Both families and

only

seniors/residents with disabilities

Source: FY 2014 Logic Model data and FY 2014 application data.

Notes: N=89. This figure reflects grantees for which we had application and Logic Model data for FY 2014.

Grantee Outreach Efforts and Resident Participation

GRANTEES DO INTENSIVE OUTREACH

Grantees regularly engage in intensive outreach to link residents to services. Across the sites we visited, service coordinators employed many in-person outreach strategies, including door-to-door resident engagement, resource fairs, and organized activities around holidays. Although service coordinators used other methods of outreach, such as social media and flyers, they emphasized how important more personal outreach is for building relationships and trust. In-person interactions 3 8

between service coordinators and residents, as wel as word of mouth among residents, can be more effective at fostering participation in ROSS-SC activities, as service coordinators may not be able to reach residents easily via phone or email.

One grantee the research team visited found that connecting with seniors and residents with disabilities when they first became tenants was an effective way to let them know how ROSS-SC

service coordinators could serve as a resource for them.

“One of the things we try to manage…is building rapport, building trust with the residents so that they actual y…know that there’s a group out there that they can literal y walk to and use as a resource. And we’re finding that it’s more beneficial to try to get them on the front end as they’re coming in to get the keys, as they’re coming in to have their first orientation. That’s beneficial because then we can get some information from them, we can actual y pull them in and talk to them about the various programs.”

—Executive director of a better-resourced PHA grantee

RESIDENT PARTICIPATION DEPENDS ON EMPLOYMENT STATUS AND ENGLISH PROFICIENCY

Despite in-person outreach efforts, participation in ROSS-SC activities varies by residents’

employment status and language background. At the grantee level, service coordinators thought unemployed residents were the most likely to participate in services (81 percent) and reported the highest level of outreach to these residents. Although more than one-half (54 percent) thought that working residents would likely engage in services, that number appears low when compared to the emphasis service coordinators placed on conducting outreach to working residents. In fact, working residents show the largest difference between outreach versus the perceived likelihood of participation (exhibit 15). An unobserved reason for this may be that working residents have fewer barriers to self-sufficiency and less need and time for ROSS-SC service coordinator assistance than other residents. Employment-related time commitments may also contribute to this inverse relationship for working residents, who otherwise would participate in ROSS-SC activities to advance their career skills and opportunities. Working residents do not always have schedules conducive to meeting with a service coordinator during the workday. Some service coordinators emphasized the importance of having classes at different times of the day to accommodate schedules and ensure participation.

3 9

“Maybe [the resident] wants to get their GED, but GED classes are only at night. Wel , they work 2nd shift, so that’s not an option for them.”

—ROSS-SC grant supervisor at lower-resourced resident council grantee

EXHIBIT 15: GRANTEE PERCEPTIONS OF OUTREACH TO RESIDENTS AND RESIDENTS’ PARTICIPATION

Targeted for

Outreach

Likely to Participate

Grantees (%)

Grantees (%)

(n=178)

(n=178)

Type of Resident

Unemployed Residents

85

81

Residents with Physical Disabilities

80

65

Working Residents

79

54

Elderly Residents

79

79

Families with Children

78

74

Residents with Mental Health

Needs

76

61

Non-English-Speaking Residents

54

38

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Note: This table reflects 215 service coordinators’ responses, which have been weighted and aggregated to the grantee level.

The total does not sum to 100 percent, because the categories are not mutual y exclusive.

Only 38 percent of service coordinators reported on the study survey that it was likely that non-English-speaking residents would participate in ROSS-SC programming, which is substantially below any other category. Residents’ varying English abilities were sometimes a chal enge for service coordinators and their local service partners—local partnering institutions and agencies that provide direct services to residents. Multiple sites we visited had bilingual ROSS-SC staff. Local policy in some jurisdictions requires that services be provided in Spanish as wel as English. Site visit observations suggest that Spanish-speaker participation in ROSS-SC activities at these sites may be higher, but this cannot be generalized across all ROSS-SC grantees or all language groups. In fact, it was clear across sites visited that grantee staff did not always have the capacity necessary to meaningfully engage residents who speak less commonly spoken languages like Russian, Burmese, Vietnamese, or Oromo.

4 0

“Creating a flyer and putting it in their door doesn’t work; they don’t read them. Some can’t read in English, so we’l be doing it in English [and other languages]. We’re finding that contacting residents isn’t as easy as making a flyer on my laptop and printing 2,000

out. One, it’s expensive; and two, it doesn’t work.”

—Service coordinator at a lower-resourced resident council grantee

Aligning Services to Meet Resident Needs

Services Referred and Available to Residents

ALMOST ALL SERVICE COORDINATORS REFER RESIDENTS TO HEALTH AND EDUCATION

SERVICES

Service coordinators are expected to assess the resident needs of the public housing communities they serve, understand the needs of individual residents who come to them for assistance, and help address those needs by providing appropriate referrals to community service partners. We asked service coordinators about resident needs and the types of services to which they commonly referred their residents in both the survey and site visits (discussed below). A majority of service coordinators responding to the survey reported referring residents to all service areas we asked about (exhibit 16).

Almost all service coordinators reported referring residents to healthcare services (94 percent), fol owed closely by adult education services (90 percent) and mental or behavioral health services (87

percent). Less common but still widespread partnerships include child or family services at just over three-fourths (76 percent) of service coordinators and community services (69 percent).

4 1

EXHIBIT 16: PARTNER REFERRALS BY TYPE OF SERVICE

Type of services partner organizations provide

Healthcare services

Adult education services

Mental health or behavioral health services

Employment related

Financial education services

Child or family services

Community services

0%

20%

40%

60%

80%

100%

Yes

No

Other

Service Coordinators (percent)

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: This table reflects 215 service coordinators’ responses, which have been weighted to reflect the pool of service coordinators. Percentages may not add up to 100 due to rounding. The ‘other’ category includes ‘Don’t Know,’ ‘Not Available,’

and those that didn’t answer the question.

FOOD, TRANSPORTATION, AND YOUTH SERVICES ARE ALSO COMMON

The top areas of service provision mentioned by service coordinators during our site visits were similar to those identified in the survey, with some variation. Service coordinators at all 10 grantees we visited

noted food and meal services as an important area of resident need and partner services. Key partners noted in this area include Meals on Wheels, local churches, and food banks. Service coordinators reported connecting residents with prepared meals and food boxes. Mobile grocers—services that bring fresh foods to sel from a van or other vehicle on a regularly scheduled basis—served at least three of the sites we visited. At one site, we heard about staff giving residents vouchers to a local farmer’s market. Food services can have income or age restrictions, and not al residents at the sites we visited had access.

Assistance with transportation was one of the most common needs residents and service coordinators noted in our site visits. Service coordinators at al the sites we visited mentioned linking residents to and/or providing at least one form of transportation service. They noted that residents 4 2

frequently need assistance in the form of transit subsidies or rides to reach remote training facilities, employers, and other services. The most common types of transportation assistance service coordinators reported arranging were bus tickets and van or car rides, typically through a local church or government partner. One service coordinator observed that providing bus tickets through ROSS-SC

helps residents to get to work, thus overcoming what is for many a significant barrier to obtaining and maintaining employment. At this same site, an education partner noted they provide rides to residents so that they can get to their nursing classes and pass the program. At least one site’s service coordinators also shared that they would at times provide rides to residents themselves or would integrate transportation assistance with programming aimed at car ownership.

As reflected in the survey results above, health services were also commonly noted, with service coordinators at 9 of 10 sites mentioning this area. In addition to standard primary health and dental care, service coordinators reported connecting residents with mental health services, substance use and addiction services, exercise programs, and insurance supports, among others. General y, the grantees we visited had some sort of partnership with a regional healthcare provider. Most (seven sites) reported having service coordinators assist residents with insurance, including obtaining insurance coverage, navigating reimbursements, and applying for Medicare and Medicaid. Generally, sites did not have healthcare services onsite, although one-half of the sites (five sites) reported hosting health fairs to connect residents with local providers and screen for health needs. Although less common, three grantees we visited reported having both health professionals and in-home aides available to assist with independent living, provided through partnerships and external funds. Service coordinators mentioned sometimes addressing other needs related to public health (such as smoking cessation classes) and mental health.

“We had an Alcoholics Anonymous group here once and people would open up about domestic violence, and we were able to help people to get the help they needed with addiction and get them out of that situation.”

—Service coordinator (former) at lower-resourced PHA grantee

Service coordinators at 6 of the 10 grantees we visited identified financial literacy as a key service area related to resident needs. The financial literacy services they described ranged from financial education to matched-savings accounts. They noted that these activities are conducted frequently in 4 3

partnership with local financial or educational institutions and are often comprehensive, covering subjects such as credit repair, loan repayment, budgeting, and banking. Some financial programs we heard about focus on homeownership, such as classes on homebuying or home maintenance; others target access to public benefits, particularly for seniors on a fixed income. At one grantee we visited, the service coordinators told us about a 3-month class they were developing, which emphasizes the use of online financial education tools and printed paper budgets for residents. They also publish a regular online newsletter that includes money-saving tips.

“[The PHA] had a program where, for every dollar you put in, they would put in three

[dol ars]. It was a wonderful program, and there were a lot of people in it.”

—Resident focus group participant at better-resourced PHA grantee

Education, mentioned by service coordinators at 6 of the 10 sites, included adult education and English for Speakers of Other Languages (ESOL). Service coordinators reported connecting residents with educational services at local universities, libraries, technical col eges, and community col eges.

Most frequently among the 10 sites we visited, service coordinators reported connecting residents with high school equivalency and ESOL classes. Adult education appeared to often be combined with soft skil development or vocational training in health-based fields, among others. One site, for example, had a close partnership with a Nurse’s Assistant (NA) training school that leveraged local philanthropic funds to allow qualified residents to enrol in the program tuition-free. Another site had a similar connection with a phlebotomy training program. Service coordinators we spoke with noted that residents were more likely to participate when provided with convenient options, such as night classes and programs offered onsite.

Youth programming, mentioned by service coordinators at 6 of the 10 sites, primarily came in the form of afterschool and summer programming. Of the six sites where service coordinators mentioned youth programming, four maintained after-school programs. Two sites operated in-house programs, whereas the other two sites reported bringing in a partner agency to run the program. These afterschool programs offered multiple supports for school-age children, including homework help, field trips, and community engagement activities like volunteering and a community garden. At least one site placed emphasis on youth programming by hosting after-school programs aimed at providing youth alternatives to perceived high-risk behaviors, such as drug and alcohol abuse and problem 4 4

gambling. Another site offered programming in partnership with their local library for preschool-aged children and their parents that helps develop literacy and early childhood learning. Two sites we visited encouraged youth and parents to spend time together by providing a community room with food, games, and other activities. These grantees intended to bring families in to engage with each other and to use these times as opportunities to connect with the service coordinators for individual supports.

Both survey results and site visits illustrated how service coordinators across grantees connect residents with important services addressing broad areas of need, including healthcare, food, education, transportation, youth services, employment, and financial services. The types of services available varied across sites on the basis of available partners, resident needs, and the entrepreneurial approaches service coordinators take to make connections.

SERVICE COORDINATORS SPEND A LARGE PORTION OF THEIR TIME ADDRESSING

EMERGENCY SERVICE NEEDS

Most service coordinators report going beyond the functions described above to address client needs (exhibit 17). These activities are often in response to immediate resident needs. A majority of service coordinators are involved in food insecurity activities (81 percent), transportation and eviction prevention (both 75 percent), property management and maintenance issues (69 percent), health emergencies (61 percent), domestic violence issues (55 percent), and childcare needs (52 percent). At more than half the sites the research team visited, service coordinators described spending at least half their time directly working with residents and addressing immediate needs.

EXHIBIT 17: SERVICE COORDINATORS’ ADDITIONAL ACTIVITIES

Service Coordinators (%)

(N=215)

Ad

ditional Activities

Food Security

81

Transportation

75

Eviction Prevention

75

Property Management/Maintenance

69

Health Emergencies

61

Domestic Violence

55

Childcare

52

Working with Child Protective Services

34

Other Immediate/Emergency Problems

32

Drug-Related Emergencies

30

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: This table reflects service coordinators’ responses weighted to reflect the pool of service coordinators.

4 5

Types of Partners

SERVICE COORDINATORS ACROSS SITES WORKED WITH COMMON NATIONAL AND LOCAL

PARTNERS

Service coordinators work with a wide variety of organizations to link residents to needed services.

Their partners range from large institutions such as national organizations with local branches, universities, churches, and regional health institutions to smal er organizations such as nonprofit organizations or local government services. The grantees we visited cited the fol owing specific national nonprofit and government entities as partners providing myriad services to residents:



Adult Protective Services (providing home energy assistance and weatherization)



American Lung Association



Goodwil (resumé assistance and job training)



United Way (financial literacy classes)



Volunteers of America (youth prevention services and community building after-school programs)



YWCA/YMCA (smoking cessation services; Livestrong support group; chronic disease prevention programs; domestic violence support)



Meals on Wheels (food boxes)

In addition, grantees we visited discussed the fol owing organization types as common service partners:



Banks and financial institutions.



Social assistance centers, such as nonprofit disability assistance programs.



County departments of social services, youth and community centers.



Healthcare providers, such as health centers, mental health services, senior care services, and health education providers.



Food and nutrition providers, including food banks, nonprofit summer meal programs for youth, and grocery stores offering community outreach.

4 6



Public institutions including mayor’s offices, public libraries, sheriff and police departments, and unemployment offices.



Educational institutions, including universities and community col eges, high schools, vocational/work training programs (frequently, Certified Nursing Assistant programs), and col ege/career navigation organizations.



Women’s outreach programs and domestic violence advocacy organizations.



Childcare centers.

Understanding Service Gaps

SERVICE SUPPLY DOES NOT NECESSARILY MEET ALL RESIDENT DEMAND

Although most service coordinators reported referring residents to a diverse set of services, not al of the service coordinators reported that their service partners were actually able to meet their residents’

specific service needs. For example, although 94 percent of service coordinators reported that they referred residents to healthcare services, only 82 percent reported that partner organizations providing healthcare services were able to meet their residents’ needs (exhibit 18). The disparity for mental and behavioral health services is larger stil , with 87 percent of service coordinators having reported that they referred to services in this area but only 70 percent reporting that they thought their partners could meet resident demands for these services.

EXHIBIT 18: PARTNERS MEETING RESIDENT DEMANDS

Types of Services Partner Organization

Service Coordinators (%)

Provide

N=215

Don’t

Not

Didn’t

Yes

No

Know

Available

Answer

Healthcare Services

94

0

2

0

4

Adult Education Services

90

2

2

1

4

Mental Health or Behavioral Health Services

87

4

2

3

4

Financial Education Services

86

4

2

4

4

Employment-Related Services

86

4

2

3

5

Child or Family Services

76

9

4

6

6

Community Services

69

12

7

8

5

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: This table reflects service coordinators’ responses, weighted to reflect the pool of service coordinators.

The reasons for this discrepancy likely vary by grantee and may include limited numbers of partners in a particular service area, limited resources or capacity on the part of available partners, or a 4 7

particularly high demand for services. In al cases, though, this gap between referrals and demand indicates that service coordinators are referring residents to needed services and that partner services are in high demand.

COMMUNITIES WITH FEWER RESOURCES STRUGGLE WITH GAPS IN SERVICES

Despite the discrepancies indicated in the survey data, service coordinators we spoke with during site visits reported that they are general y able to meet resident needs. Observations during site visits, however, suggest that some grantees have access to more and higher-quality services than are available in other communities. Grantees in small and rural contexts, for example, have limited options for career development and training compared with those in larger cities. Those in small and rural contexts also reported depending on a single office such as county social services for a range of services such as emergency rent assistance, healthcare referrals, and transportation assistance; in larger urban contexts, service coordinators noted having access to other community organizations such as immigrant or refugee support centers, nonprofit healthcare or counseling support, and community development organizations. A service coordinator at one better-resourced site even reported organizations approaching their housing development to request to use their space as a venue to provide services to residents and community members. This difference in available resources may explain some of the discrepancy between referrals and resident demands met. Other factors, such as the specific characteristics of local partners may come into play as wel , however, and even in the highest-resourced communities, some resident demands remain unmet.

Relationships with Partners

Building and Maintaining Partner Relationships

SERVICE COORDINATORS SUBSTANTIALLY ADVANCE PARTNERSHIPS

Outreach to community partners and the development and maintenance of partnerships are core elements of a service coordinator’s job, enabling them to connect residents to needed services. FY

2014 grantees, on average, reached out to about 19 partners per year over the course of the 3-year grant, making an average of 58 contacts with potential partners. Staff at 5 of the 10 grantees we visited mentioned that their ability to make new partnerships was due to the ROSS-SC grant, with some noting that this work to build relationships was one of the biggest accomplishments of their ROSS-SC program.

4 8

“We’ve learned along al these years we’ve had ROSS-SC […] that real y, the most important piece is building relationships with our community partners and doing the coordination of services. Obviously, we can’t do everything ourselves, and also having those partners at the table with us is extremely helpful and important.”

—ROSS-SC grant administrator at better-resourced PHA grantee

REGULAR SERVICE COORDINATOR MEETINGS WITH PARTNERS ARE KEY TO STRONG

GRANTEE/PARTNER RELATIONSHIPS

The ROSS-SC Notice of Funding Availability (NOFA) recommends that service coordinators organize local Program Coordinating Committees (PCCs) to convene their program partners and other community stakeholders, and to serve as a regular point of contact between service coordinators and their partners. Weighted to the grantee level, survey results indicate that 70 percent of grantees have a PCC, with respondents from PHAs reporting a slightly higher rate (72 percent) than the average across all grantees, which includes resident associations and nonprofit organization grantees. The type of residents served (elderly/people with disabilities, families, or both) is not associated with a grantee’s likelihood of having a PCC. Of those grantees who responded that they had a PCC, more than one-half (52 percent) indicated it meets quarterly, one-fourth (25 percent) indicated it meets monthly, and 8

percent indicated it meets as necessary. Respondents general y view the PCCs positively, with 43

percent agreeing it is “very effective in meeting their goals” and 47 percent of respondents saying it was “somewhat effective” in helping the program meet its goals.

Not all service coordinators establish a formal PCC to coordinate with partners. Some service coordinators appear to primarily interact with partners on a one-on-one basis, while others tap into preexisting regularly scheduled community meetings of partners. Although service coordinators at only 3 of the 10 sites we visited used the term PCC, all reported regularly coordinating with partners in some fashion, including holding or attending regular meetings. Given this finding, our survey results could be an undercount, as some service coordinators participate in or even organize ad hoc meetings or similar events and do not count them as formal PCCs. Some service coordinators we spoke with viewed initiating partner meetings as the core component of their jobs. One grantee we visited, for 4 9

example, employed separate case managers at the housing agency funded through the local PHA’s operating funds, enabling the ROSS-SC service coordinators to focus predominantly on creating and coordinating partnerships. At that site, service coordinator-initiated partnership meetings brought together about 30 stakeholders, including resident representatives, staff, and community partners.

These meetings were strategy sessions, where participants reflected on services provided and plan for the future. At another site, interviewees reported that a similar meeting brought together a wide array of partners across sectors, including social service providers, behavioral health providers, prosecutors, law enforcement, community health, housing, and women’s outreach. In sites we visited with both ROSS-SC and FSS programs, interviewees noted that these meetings often incorporated FSS staff as well.

“[Meetings with our partners are] the opportunity to share any updates on the program; or if we need any type of feedback from community partners; if we’re changing any policies, that’s an opportunity to bring partners who are serving our residents, so we can inform them and ask for their feedback. […] It’s an opportunity for community partners we bring to the table to get to know al of us and each other.”

—ROSS-SC grant administrator at better-resourced PHA grantee

SERVICE COORDINATORS BENEFIT FROM BROADER COMMUNITY NETWORKS

At several grantees we visited, we found that service coordinators participate in community convenings of local partners that predate the start of the ROSS-SC grant or the service coordinator’s tenure. In these cases, the service coordinators are able to plug their work into these preexisting structures and benefit from this established venue. Although these meetings are typical y more expansive than a ROSS-SC-focused PCC, both service coordinators and partners report that the convenings offered valuable opportunities for service coordinators to build relationships and learn from other providers in the community. For service coordinators who have prior background and experience working in the community, these convenings can function as a gateway for creating more formal and even new partnerships. This was the case for two service coordinators we interviewed, one of whom held a leadership role in this community coordination entity.

5 0

Maintaining Individual Connections and Seeking New Partners

ONE-ON-ONE RELATIONSHIPS WITH PARTNERS ARE VALUABLE

Service coordinators in smal er, more rural communities we visited mentioned spending time with individual partners, often citing long-term relationship building with partners as a key aspect in their work. Some staff members we interviewed from small housing agencies, for example, reported having worked with the local social services department for decades. This longstanding relationship led to stronger communication and coordination; partners at one such site says they spoke with the service coordinator weekly. Service coordinators reported regular one-on-one communication with partners in at least two of the seven urban locations we visited as wel . Partners we spoke with also noted the value of these relationships.

Anything we’re going to do here I run by [the service coordinator], just as a courtesy. We need to keep a clear line of communication, coordinating schedules… when they have events, I always ask how I can help. That could be weekly, biweekly, often.

—Staff member at partner organization of a better-resourced PHA grantee

NEW PARTNERS ARE IDENTIFIED TO MEET EMERGENT NEEDS

Service coordinators often respond to ad hoc requests for assistance from residents. Many of these cannot be met through established partnerships and require rapid identification of new providers. At two grantee sites we visited, the service coordinators reported creating lists of community resources.

At one site, this took the form of a Google Doc that each service coordinator updated any time they looked up a new service and referred someone to it. At another, service coordinators noted using a spreadsheet for similar purposes. Both of the grantees who do this employ multiple service coordinators, so this strategy was especially useful to facilitate internal resource sharing. At a third site, the county compiled their own resource list, which the service coordinator used. Across sites, such resource lists are valuable resources for service coordinators, especial y those early in their careers or new to the community.

5 1

Formalizing Partnerships and Sharing Data

GRANTEES REPORT SUCCESS IN CONTACTING PARTNERS AND REACHING SERVICE

AGREEMENTS

Grantees frequently formalize their partnerships through Memoranda of Understanding or similar agreements, as shown in data reported by FY 2014 grantees. During the 3-year grant cycle, grantees made 39 agreements on average with agencies out of the average 58 providers they contacted, or about 12–14 per year (exhibit 19). Over one-half of grantees (54 percent) were successful in turning outreach contacts into agreements 75 percent or more of the time. For those who reported on the optional metric measuring outreach to healthcare providers, the rate of “established partnerships” was even higher: an average grantee made 15 agreements with agencies to provide health-related services out of the 18 total providers they contacted, or about 4–7 per year.

EXHIBIT 19: PARTNER CONTACTS VERSUS SERVICE AGREEMENTS

Partners Contacted

Agreements Made with Partners

Number of Providers

Number of Providers

Total

5,106

3,451

Average per Grantee

58

39

Median per Grantee

46

31

Source: FY 2014–2016 Logic Model data.

Note: The total number of grantees represented in this table is 88.

INFORMAL AGREEMENTS ARE ALSO POPULAR, MAKING PARTNERS DIFFICULT TO COUNT

Site visit interviews revealed variability in the level of formality of partner agreements. Al sites we visited had at least one type of formal MOU in place. But service coordinators and grant managers at 9

of the 10 grantees spoke of a mix of informal and formal partnerships. Furthermore, staff understanding of what constitutes a partnership to be reported on the Logic Model appeared to vary across grantees, with some believing that any stated agreement—verbal or written—would count, where others only counted agreements formalized through an MOU or similar document. The FY

2014 Logic Model defines partnerships as agreements that “entail accepting direct referrals from ROSS-SC.” As such, data recorded in the Logic Model may be an undercount of the partnerships that service coordinators undertake if some sites only count those formalized with an MOU. In addition, a handful of grantees reported more agreements than outreach attempts, indicating that a formal agreement may have already been in place.

5 2

AGREEMENTS CAN COVER USE OF SPACE, FEES, DATA SHARING, AND RESOURCES

Some service coordinators and grant managers we spoke with described several different kinds of formal partner agreements. Agreements they mentioned covered the use of space, data sharing, types of programming and services, and the financing of services and activities. Some of these partnerships were formalized through the PHA, and not through the ROSS-SC program specifically, although they may have been initiated by ROSS-SC service coordinators. The fol owing examples il ustrate the variety of agreements:



One housing agency agreed to pay the fees for their residents to take classes from a partner.



One partner agreed to share participation data monthly, along with future plans for providing services to those participants.



One partner agreed to prioritize clients of the housing agency and set aside funding and other resources for them.

DATA SHARING WITH PARTNERS IS UNCOMMON, MAKING IT DIFFICULT TO TRACK RESIDENT

USE AND OUTCOMES

Although we did hear examples of data sharing between partners and the grantees during site visits, that relationship was rare. Some service coordinators we spoke with were frustrated by their inability to obtain this information as it impeded their ability to track residents’ progress. Although they may be able to schedule check-ins with residents they are working with closely, those that are receiving one-off referrals or receiving services in groups remain difficult to track.

Health information is a particular challenge due to concerns for resident privacy. This appeared to be particularly true in the case of healthcare providers where Health Insurance Portability and Accountability (HIPAA) concerns complicate sharing health data. Although releases are possible—we did hear about one case where this occurred—the level of effort for service coordinators to establish and maintain HIPAA forms with all providers for each resident receiving services would be very high.

Education can also be a chal enge for data sharing, although less so than health. According to the Family Educational Rights and Privacy Act (FERPA), some data considered to be directory information, which may include student names and dates of attendance, do not require special permission to obtain (DOE, n.d.). On site visits, we heard about education service data being col ected from residents themselves, as service coordinators required verification of participation in a class or activity such as 5 3

an attendance sheet to receive reimbursement for transportation expenses. We also heard examples of partners sharing attendance information directly for these purposes.

“[A service coordinator] wil check-in about what my numbers are. It’s been pretty unofficial. He wants to know about students coming for attendance records.”

—Staff member of partner organization at a better-resourced PHA grantee

Service coordinator followups on referrals typical y occurred on an informal or ad hoc basis, frequently through self-reporting by residents. Some partners we interviewed mentioned that they kept their own internal records, sometimes through an extensive database. Most partners we interviewed, however, shared that they did not track the people who were specifical y referred to them through the service coordinator, or identify those that live in public housing, which complicates fol owing up on these clients. As discussed below in chapter 4, service coordinators have a variety of systems for tracking residents, but most survey respondents (approximately 75 percent) indicated they did not use case management software. 11

Partnership Benefits to Service Coordinators and Residents

GRANTEES VIEW PARTNERS AS BENEFICIAL TO RESIDENTS AND SUPPORTIVE OF THEIR

GOALS

In our site visits, grantees also reported positive perceptions of partnerships and the outreach they conducted to residents, much of which, the grantees shared, would not have been possible without ROSS-SC funds. PHA staff we interviewed, including service coordinators, grant managers, and others, noted that in addition to providing residents with valuable services, partners provide residents with important social connections. At one grantee, for example, volunteer partners ran clubs for residents in areas such as cooking and gardening. Working closely with the service coordinators, they used these clubs as opportunities to improve communication and conflict resolution skills among resident participants, using trauma-informed approaches to their work. Service coordinators also reported that connecting with partners helped to build and extend their network, improving their own abilities to 11 Please refer to the Preface for an update on case management software usage.

5 4

connect residents with needed services and the partners’ abilities to provide services to residents in need. They also noted that strong partners provide service coordinators with a sense of a shared mission within their community.

“Dive in. Once you have a core of partners, they can help network to others and share new funding opportunities. If you keep at it, the partner network starts to build. Keep talking the talk and exchanging cards and working together to create programs. However, you must be sure the services are needs based.”

—Service coordinator at better-resourced nonprofit grantee

STRONG RELATIONSHIPS AND TRUST BETWEEN SERVICE COORDINATORS AND

PARTNERS ARE KEY

Relationship building and trust are key to successful partnerships. In addition to service coordinators and residents benefiting from partnerships, our site visit data suggest that partners benefit from their association with ROSS-SC as well. Several partners we interviewed spoke positively about the service coordinators, their local PHA, or the ROSS-SC program itself. Specifical y, they cited that working with service coordinators helps to improve their own programming: “It’s like a family,” noted one partner,

“Even though we’re not employees, just partners, we’re included. There’s a lot of communication, emails, and meetings with coordinators.” Several partners also praised the work ethic of the service coordinators and the demonstrated passion for their jobs. Sometimes partners reported difficulties in getting in touch with service coordinators. This could create challenges in the delivery of regular services. Partners acknowledged that this was likely due to the high demands on service coordinators’

time, however, rather than poor communication skil s or a lack of fol ow-through. One partner summed up this observation by noting that while service coordinators certainly aim for open communication and responsiveness, this is “easier said than done when you’re paid little [and] your job is big.”

5 5

How ROSS-SC Enhances Public Housing Operations

Integrating ROSS-SC with Other Grantee Programs

ROSS-SC BOLSTERS LESS-RESOURCED GRANTEES, DEEPENS WORK OF BETTER-RESOURCED

ONES

On our site visits, some grantees expressed that they rely on ROSS-SC as their sole source of funds to support service coordination. As one service coordinator at a lower-resourced grantee expressed, the work of the PHA was significantly smal er in scope before ROSS-SC funding. With ROSS-SC funding, the PHA could afford to hire another staff person for the service coordination and expand existing staff roles to complement and support the new service coordinator’s work. This led to a substantial y increased presence in the community.

“When we did get ROSS-SC…we started having classes, doing activities, a lot of community involvement.”

—Service coordinator at a lower-resourced PHA grantee

Other sites we visited already had some resident outreach or engagement capacity established, which ROSS-SC enhanced. For example, some housing authorities already employed case managers to work with residents. These better-resourced sites tended to add ROSS-SC-funded positions like

“resident relations” roles, as one site cal ed them, to their existing staffing, adding new capacity. At one of these sites, the acquired new staff could dig deeper into residents’ underlying needs, enhancing the grantee’s self-sufficiency efforts.

“[Before ROSS-SC] it was compliance, you do inspections, you do research. If someone cal ed for help…you couldn’t refer them out, you didn’t give them that helpfulness, because…it was al compliance and monitoring.

—Service coordinator and long-time lower-resourced PHA grantee employee

5 6

Prior to ROSS-SC funding, even the better-resourced sites may not have had the capacity to be actively involved with partners in their area or to develop new partnerships. The additional staff funded by the ROSS-SC grant enhanced their ability to reach out to partners and formalize these partnerships. These relationships are crucial to leveraging resources available in the community to better serve residents, broaden the ability of the PHA to respond to a variety of resident needs, and can result in programming brought on-site to where the residents live.

“Service coordinators will do an intake process, develop a relationship, and often uncover the true issues that are underlying the chal enges [residents] are having with lease violations. Residents like that the program is focused on self-sufficiency and feel comfortable engaging with the program because of it. However, the topics often go much deeper than self-sufficiency.”

—ROSS-SC grant manager at better-resourced PHA grantee

Integrating ROSS-SC with Property Management

SERVICE COORDINATORS AND PROPERTY MANAGEMENT MAY BENEFIT FROM

WORKING TOGETHER

The service coordinator often works col aboratively with property management. Almost all sites (at least 8 of the 10) visited for this study discussed the mutual benefits of working together: property managers, by virtue of being on-site, may identify residents in crisis before they come to the attention of the service coordinator and refer them to the ROSS-SC service coordinator. The service coordinator can then intervene, helping to connect residents to needed services. This comprehensive approach can help stabilize residents and demonstrate to residents that the PHA staff is invested in their wel -being at the same time. Property managers have an incentive to participate in this relationship; as one PHA staff person expressed it, “The managers care about their residents too, and they don’t want turnover because that’s potential vacancy issues [leading to operational costs to the PHA].”

Sometimes service coordinators might intervene before an issue escalates to property management, or they can step in to assist with dispute resolution. One service coordinator spoke about how they provided wellness checks in partnership with property managers: “We may have our differences, but they know my job…. I can ask them to hold off on evicting a resident and give me a chance to work with them. Sometimes it works and sometimes not. I’l also try to talk with residents to let them know when they have a good manager and get them to work with them, which has helped.”

5 7

In addition, a grant manager at another PHA spoke of how their property managers play a role encouraging resident engagement with ROSS-SC.

Conclusion

ROSS-SC is a crucial asset in many public housing communities where it operates. ROSS-SC service coordinators make valuable connections to external partners and expand PHAs’ roles in supporting residents’ progress toward economic and housing self-sufficiency and the ability to live independently while aging in place. The program’s flexibility means that service coordinators are able to target specific groups of residents for outreach, although they frequently serve a wide range of residents.

Service coordinators work to identify the best partners in the community to meet resident needs and are frequently successful in establishing partnerships. Although partners are not always able to meet al resident demand for services, they provide valuable services that benefit the lives of residents.

Because ROSS-SC depends on existing community resources, service coordinators in small and rural areas may depend on a smal er number of partners for services, and gaps in services may be a bigger chal enge than in better-resourced areas. Regardless of resources or size, ROSS-SC’s value is clear to service coordinators, to their partner organizations, and to other PHA operations.

5 8

Chapter 4: Performance Monitoring

and Reporting

Introduction

This chapter reviews the key features of the FY 2014 Logic Model to explore how it affected the amount and quality of data grantees reported compared to the prior version of the Logic Model and whether the FY 2014 version was an improvement. It draws on qualitative and quantitative data to answer whether and how the revised Logic Model improved the data that grantees report. 12

This chapter answers the fol owing research questions:



Have the mandatory metrics changed the structure of the ROSS-SC program and the services that are being coordinated?

» How did grantee processes for populating the Logic Model change?

» How did grantee processes for retaining data and documentation change?



How are participants tracked over time, and with what type of data and systems?

» Who sets targets for and implements the Logic Model, and who reports corresponding outcome data?

» What types of records are kept on participants, and how long are these records kept?

» Do grantees have the capacity to col ect data for reporting requirements?

» How adequate are the resources to support grantees’ effectiveness?



Did the mandatory metrics help?

» How wel do the ROSS-SC grantee annual reporting requirements define and measure grantee activities and performance outcomes?

» What outcome measures are used?

12 HUD replaced the Logic Model reporting structure with the online Standards for Success reporting system, which started in 2016 as a pilot and moved to ful implementation in 2017 for al 2016 and subsequent grantees.

5 9

Key Findings

Mandatory Metrics



The addition of mandatory metrics with the FY 2014 Logic Model increased the percentage of grantees projecting service provision and reporting actual services provided compared to data from the FY 2011 Logic Model.



The implementation of mandatory metrics with the FY 2014 logic model did not appear to affect grantees’ choice of programming or services. Some grantees may have interpreted the mandatory metrics to mean that they were obligated to provide those services, however.

FY 2014 Logic Model Limitations



The list of activities included in the ROSS-SC application did not entirely match the activities grantees could report on in the FY 2014 Logic Model. This impedes analysis of whether grantees performed the service and activities they applied to provide.



The FY 2014 Logic Model introduced some ambiguity to the data by asking grantees to input a zero to indicate mandatory metrics for both services and outcomes that were not applicable to the grantee as wel as those services and outcomes the grantee did not provide but that would be applicable.



The FY 2014 Logic Model did not provide clear distinctions between inputs, outputs, and outcomes, or clear causal pathways connecting these. It captured some inputs and outputs aligned with service coordinators’ activities but was less suited to reflect resident outcomes, which may be beyond service coordinators’ activities, observation, or control.

Grantee Reporting



Grantees’ ability to reliably and accurately col ect data for reporting on the Logic Model varied widely. More relied on paper records than on case management software.



Grantee reporting practices compromised data quality: how projections are made, how outreach activities are counted, and who inputs the data all affect the validity and reliability of the data.

6 0



Grantees we visited general y did not feel that the FY 2014 Logic Model sufficiently captured their work.

Logic Model Structure and Processes

The ROSS-SC Logic Model served to track grantees’ services, activities, and outcomes, both in terms of projected services and outcomes and the actual services delivered and corresponding outcomes observed (exhibit 20). At the beginning of each year of the grant period, ROSS-SC grantees were required to project the type and number of outputs and outcomes for each relevant service and activity. For actual annual reporting, grantees recorded the actual, or observed, data for each of their selected services and activities, as well as the associated outcomes. Although these data were due within 30 days of the award date each year, grantees sometimes revise these data in future periods’

submissions.

EXHIBIT 20: LOGIC MODEL DEFINITIONS



The Logic Model’s services and activities metrics included both inputs (that is, the people and resources that go into a service or activity) and outputs (that is, direct evidence of the successful performance or completion of a service or activity). For example, some of the Logic Model’s services/activities metrics capture inputs with enrol ment numbers (for example, the number of persons enrol ed in financial literacy classes), whereas others capture outputs with referral numbers (for example, the number of elderly persons and persons with disabilities whom the service coordinator referred to services).



The Logic Model’s outcome metrics include both outputs and outcomes, or the result of completion of an activity or service. For example, the number of people who completed a financial literacy class is an output of enrol ment in the class. Improved quality of life is an outcome of the services to which the service coordinator referred elderly persons or persons with disabilities.



There is a causal pathway between inputs, outputs, and outcomes. The structure of the Logic Model is best suited to capture inputs and outputs that align with service coordinators’

activities. It is less suited to reflect outcomes, which may be beyond service coordinators’

activities, observation, or control.

6 1

Assessing Logic Model Changes

THE MANDATORY METRICS IMPROVED THE LOGIC MODEL

The analysis of data submitted by repeat grantees showed that the introduction of the mandatory metrics in the FY 2014 Logic Model facilitated more reporting and more data than under the FY 2011

Logic Model. 13 All 67 repeat grantees projected that they would provide service on at least one of the mandatory metrics in FY 2014, with 78 percent of them projecting service provision on eight or more mandatory metrics. In contrast, none of the 67 grantees projected service on more than seven comparable metrics on the FY 2011 Logic Model; 10 percent of grantees did not project service on any of the FY 2011 metrics.

The introduction of the mandatory metrics had an even larger effect on the 67 repeat grantees’

reporting of actual services provided. On the FY 2011 Logic Model, 34 grantees (51 percent) reported no data on actual services provided on comparable metrics. On the FY 2014 Logic Model, that dropped to only three grantees (4 percent), with no actual service data provided. Across the board, grantees reported significantly more data on actual services provided. Overal , 58 grantees (87

percent) reported more data on comparable metrics in the FY 2014 Logic Model than the FY 2011

Logic Model (exhibit 21).

EXHIBIT 21: CHANGE IN DATA REPORTING BETWEEN THE FY 2011 AND FY 2014 LOGIC MODELS

Grantees (%)

N

More Data

87

58

4

3

Same Amount of Data

Less Data

9

6

Source: Urban Institute analysis of FY 2011 and FY 2014 Logic Model data.

Notes: This analysis only includes service and activity metrics that appear in both Logic Models. This analysis also only counts the 67 grantees that received grants in both grant cycles.

The analysis of repeat grantees’ reporting also showed that the introduction of mandatory outcomes meant that grantees reported outcome data more uniformly. More grantees reported data under the FY 2014 Logic Model’s mandatory metrics, but all outcome metrics—whether mandatory or not—had some data reported from multiple grantees. The only outcomes that have more than 90

percent of grantees reporting data—completion of financial literacy or management classes and the 13 Only counts metrics and grantees that appeared on both the FY 2011 and FY 2014 Logic Models.

6 2

completion of an agreement with an agency for services—are both mandatory outcomes. Al three outcomes with less than 20 percent of grantees reporting data are optional outcomes. 14

THE FY 2014 LOGIC MODEL INTRODUCES SOME AMBIGUITY TO THE DATA

The FY 2014 Logic Model’s revised data process, which has grantees input zeroes for mandatory services and outcomes that do not apply to their programs, introduced some ambiguity to the data.

The ambiguity comes from the fact that zero now can mean two distinct things: (a) not applicable, which indicates that a service or outcome is outside of a grantee’s programmatic scope; or (b) zero, which only indicates an absence of service or outcomes, even when it is within the scope of a grantee’s program.

The Logic Model data show that grantees use zero in ambiguous ways. More than one-fifth (21

percent or 17 grantees) of the 81 FY 2014 grantees projected zero for at least one mandatory service or activity at some point in the grant cycle, only to later report some level of service was actual y provided to clients on the same metric. This makes it unclear whether the grantee thought that activity or service was not applicable (zero = not applicable) but changed its plans based on an actual change in residents’ needs, or if the grantee simply expected only minimal participation and undercounted (zero

= applicable, but not projected).

MANDATORY METRICS DID NOT WIDELY AFFECT SERVICES

The introduction of mandatory metrics generally did not influence grantees’ choice of programming or services. During the site visits, most grantees either did not directly discuss that the mandatory metrics had any effect, or, as one grantee explicitly stated, that they have “not impacted the program’s design and mission,” The few grantees we visited that changed their metrics in response to the mandatory metrics felt that they were complying with a HUD obligation. These grantees said they added programs, regardless of resident need or interest, because they believed that mandatory metrics on the Logic Model reflected that the service was now mandatory to provide.

“[The service coordinators] sometimes add more programs in to try to fit the Logic Model.

But do the residents want that? Probably not.”

14 It is unclear how the implementation of the FY 2014 Logic Model affected the services and outcomes beyond the presence and absence of data. Because of widespread missing data in FY 2011, the lack of clear definitions in FY 2011, and grantees’ documented conceptual misunderstandings and reporting deficiencies, a reliable estimate of how outcomes changed (that is how many outcomes were different) as a result of the Logic Model is unknown.

6 3

—ROSS-SC grant administrator at a better-resourced nonprofit grantee Grantee Data Capacity

GRANTEE CAPACITY FOR DATA COLLECTION VARIES WIDELY

Grantees we surveyed and visited had widely different capacity to col ect and report their data; this capacity encompasses infrastructure, processes, and staff. Some grantees have very advanced capacity whereas others have much less. For example, the survey results show that the most common method (39 percent) grantees use to track resident interactions is via paper records, whereas another 30

percent have case management software and 23 percent use spreadsheets (exhibit 22). From what we observed during our 10 site visits, grantees who rely on paper records demonstrated lower data capacity than those that use spreadsheets or case management software. At sites that rely on paper records, program staff admitted that standardized data processes were largely absent. Staff used paper to record data as tallies, kept records in binders, or captured information on Post-it® Notes; some staff even relied on memory to record their interactions with residents. These grantees did not use tracking tools outside of the Logic Model.

EXHIBIT 22: GRANTEE MEANS OF TRACKING CLIENT INTERACTIONS

50%

39%

40%

30%

30%

23%

20%

10%

5%

2%

0%

Paper

Case

Spreadsheets Multiple

Did not

records management

means

answer

software

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: This figure reflects 215 service coordinators’ responses, which have been weighted and aggregated to the grantee level.

The total number of grantees represented in this table is 178. The total does not sum to 100 percent because of rounding.

At the sites we visited with more rudimentary data practices, staff rarely entered data into a computer as part of their data management process. Although some staff we interviewed expressed a desire to switch to case management software, which grantees can purchase with ROSS-SC funds, or 6 4

spreadsheets, they faced significant time constraints in doing so. Instead, staff opted to make service to their clients the priority.

“She gives me al of these handwritten numbers, and she explains each of the activities, and then I play dot-to-dot with the Logic Model.”

—Executive Director of lower-resourced PHA grantee

Grantees we visited whose sites use case management software had much more capacity for systematic and standardized data col ection. During the site visits, staff reported some form of quality control in their data reporting process, such as designating a staff member to be responsible for reviewing data before it goes to HUD or having a routine, monthly, multi-staff review of the data. One site even had a training specifical y on the Logic Model for the service coordinators. These grantees also tend to have routine data entry procedures, and staff frequently input data on a daily or weekly basis. 15

Data Quality

EXHIBIT 23: DATA QUALITY DEFINITIONS



Validity concerns the metrics themselves, specifical y the extent to which a given metric measures the object it says it measures.



Reliability relates to the recorded estimates or observations, specifical y the degree to which they accurately reflect reality.

SOME GRANTEES LACKED UNDERSTANDING OF HOW THEIR REPORTED SERVICES AND

ACTIVITIES SHOULD LINK WITH OUTCOMES REPORTED

According to completed Logic Models for FY 2014 grantees, there were multiple inconsistencies between associated services/activities and outcomes, where grantees reported more outcomes (for 15 It is unclear whether the implementation of the FY 2014 Logic Model changed grantee processes in any way concerning staff roles or how staff retained data and documentation. During the site visits, we learned that there had been staff turnover at some sites, leaving us without reliable information about what was done before the implementation of the FY 2014 Logic Model. Even in sites that had not experienced turnover, staff appeared to not distinguish between different versions of the Logic Model. It is unclear if they did not understand the differences or if so much time had elapsed that they could not differentiate the two when we the team spoke with them is unclear.

6 5

example, families receiving childcare) than services and activities (for example, families the grantees referred to childcare services). In other words, this would mean that there were more families receiving childcare services than families that had been referred to childcare services. FY 2014

grantees reported more outcomes than services and activities for all but 1 of the 14 mandatory metrics, 3 of which had more than 15 percent of grantees reporting more outcomes than services and activities. The optional metrics were even more inconsistent. Al but 2 out of the 15 optional metrics had more outcomes than services and activities; one-third of the optional metrics had more than 15

percent of grantees reporting more outcomes than services and activities (appendix L).

The sites’ lack of understanding also appears to have affected how they reported some services and activities, specifical y the three “referrals for senior/disability services” mandatory metrics. These three metrics al have the exact same text on the FY 2014 Logic Model and are only distinguishable by their connection to three separate outcomes: (1) improved living conditions and quality of life; (2) independent living, aging in place, and avoiding long-term care placement; and (3) seniors and disabled residents obtaining needed services. Although the glossary is explicit that they are different, grantees often reported the same numbers for all of three outcomes as if they were the same. Grantees would have had to understand the intended connection between the services/activities and the outcomes as described in the glossary’s guidance to properly input the data.

RESPONSIBILITY FOR INPUTTING DATA VARIES

The staff responsible for inputting Logic Model data varied across sites we visited. Although a single person was sometimes in charge of inputting data, we saw a great deal of variance as far as whose responsibility that was and what specific responsibilities they held. Service coordinators were sometimes only involved in submission of their own individual caseload data, as opposed to overseeing the data for multiple service coordinators. In other instances, service coordinators helped make sense of the data, but they were not themselves responsible for setting the targets or inputting the data into the Logic Model. For example, the service coordinator’s supervisor—who might be the department head, or even the executive director—or some other grant manager may be responsible for data management and entry.

GRANTEES’ PROJECTIONS MAY NOT BE RELIABLE

Some grantees we visited shared that they were not confident in their estimates of services and activities entered in the Logic Model. They noted that resident needs vary and can change over time, partnerships fluctuate, and the work of a service coordinator is not always the same, making it difficult for grantees to project future activities. For example, a coordinator might develop a partnership that 6 6

brings a new class to the residents, but residents may not participate. Conversely, a need might arise among several residents at once (for example, bedbugs) that was previously unknown to the service coordinator and was never estimated. This led at least one grantee to describe their estimates as more of a “guess.”

We also heard from more than one grantee about their concern that failure to meet their estimates would reflect poorly on them. This provided an incentive to underestimate anticipated service and activity values to ensure that grantees met or exceeded their targets. One service coordinator for a PHA shared, “I’d rather have lower numbers and then exceed them.” Another grant administrator pointed out that the risk of failing to meet estimates is even higher when having to predict resident needs farther into the future.

“You’re looking at [what to project on the Logic Model] easily two years in advance…. And honestly, do you set yourself up to fail by projecting higher and performing lower?”

—Grant administrator at better-resourced nonprofit organization grantee

Further complicating these challenges is the fact that multiple staff did not always see a clear line between what they do on a day-to-day basis and the categories on the Logic Model: “Resident services is not a clean dataset. It’s not something that leads from this output to this outcome. That’s not a clean line. It’s more about us knowing the effect of our work, and I can tel you that’s not something we’l ever be able to report on, and that’s unfortunate.” As discussed in greater detail later, grantees experienced understandable difficulties quantifying or categorizing certain activities or milestones with residents, viewing their work with residents as more nuanced, less linear or countable.

HUD DEFINITIONS ARE NOT USED CONSISTENTLY ACROSS GRANTEES

Despite the instructions and extended glossary guidance in the FY 2014 Logic Model, our observations during the site visits suggest that grantees counted the same metrics differently, particularly for outreach. Outreach is at the core of multiple mandatory and optional service and activity metrics, including outreach to partners, outreach to individuals who are not elderly or disabled, and outreach to seniors and persons with disabilities. Additional y, outreach is an implied activity in other service and activity metrics, such as information provided on banking, credit, and individual development accounts (IDAs); the glossary instructs sites to “count each person who receives 6 7

information related to opening or maintaining a bank or IDA account or credit.” In effect, discrepancies in counting outreach activities can easily cascade throughout the Logic Model.

Through site visits, we also learned that some grantees only counted what they considered

“meaningful contacts” so as to not inflate their numbers, even when the contacts seemed to meet the definition in the Logic Model, while others counted more liberal y. As such, some sites reported no outreach and other sites appear to have drastically underreported their outreach activity on the FY

2014 Logic Model, based on their stricter, site-level definition. For example, despite reporting on making hundreds of contacts through events like resource fairs in a site visit interview, one service coordinator reported no outreach on the Logic Model.

6 8

“If you just come to the fair and I tel you about programs, unless I come in and sit down with you, I don’t consider that a contact for my numbers. ”

—ROSS-SC service coordinator at a lower-resourced PHA grantee

THE GRANT APPLICATION AND LOGIC MODEL ARE NOT ALIGNED

The Application vs. Logic Model analysis showed that the FY 2014 Logic Model is not completely aligned with the ROSS-SC grant application (HUD Form 52769), meaning that the full scope of grantees’ activities and outcomes cannot be captured by the Logic Model (appendix M). The FY 2014

Logic Model was streamlined to focus on core activities of interest to HUD, so metrics for some activities no longer appear in the form (as they did in the prior version). Although grantees could engage in a variety of activities, many were not tracked. For example, 73 out of the 81 FY 2014

grantees (90 percent) applied for funding to connect residents to English as a Second Language (ESL), youth programming, homeownership counseling, or computer classes, none of which has a corresponding reporting metric in the FY 2014 Logic Model (appendix M).

EXTENT OF GRANTEES’ WORK EXCEEDED WHAT WAS TRACKED IN THE LOGIC MODEL

During site visit interviews, grantees shared that they felt the Logic Model was insufficient to capture what their work entails and the context in which their work is situated. On site visits, grantees noted that the Logic Model presents a one-dimensional representation of their work, leaving no space for context, narrative, or explanation. Staff felt that the narrative and context is critical y important because it il uminates what the data show. As one staff member explained: “At first, we’re just removing barriers and dealing with crises. And then we have to stabilize them before we get to a place where we can think about jobs.” Another staff member noted that people do not always follow a straight path between the service coordinator’s work and an outcome. Sometimes residents experience setbacks and require multiple touches, particularly when dealing with more serious needs, such as mental health challenges. Areas that the Logic Model emphasizes, such as education and employment outcomes, are viewed as secondary to residents’ emergency needs to keep them financial y afloat and in housing.

Grantees also felt that there was a need for their day-to-day work to inform the data that HUD

col ects on the Logic Model. In our interviews at almost every site, staff expressed frustration with at least some component of the Logic Model. Some felt particularly constrained by the Logic Model’s pre-selected metrics. Additional y, multiple service coordinators reported dealing with ad hoc requests from residents every single day, ranging from assisting with tax preparation or paying utility bills to 6 9

liaising with teachers on behalf of their kids who are having problems in schools. Although these might be documented in case notes, there is no ability to input this information into the Logic Model. At least one grantee director thought this thwarted HUD’s ability to clearly document all impacts that the work of ROSS-SC service coordinators has on public housing residents, including unanticipated ones.

“HUD has said: ‘These are our priorities, but don’t tel us about other good things that have happened outside our priorities.’ Most funders want to know about outcomes that they didn’t anticipate, but not HUD. I think that, probably, the program is not sufficiently appreciated because of that—because they’re not letting programs tel everything that is happening.”

—CEO of a better-resourced nonprofit grantee

Even when data may appear in the Logic Model, the mismatch with services provided on the ground calls into question whether the data capture the actual scope and scale of what sites do. For example, the application’s “job search and placement” category, according to site visit interviews, means things as disparate as holding job fairs that attract hundreds of people, to assistance for paying for classes, to coordinating across services like adult education and workforce development. Based on site visit interviews, it is unclear where grantees are reporting these activities in their Logic Model data. Based on our site visit interviews, it seems unlikely that grantees are reporting al of these under the Logic Model’s metrics for job skil s assessment or referrals to employers. In some sites we visited, grantees stated they have not reported some of these activities at al .

“I consider the e-Logic Model to be a problem, in the sense that it doesn’t give us anywhere to add our actual outcomes. It’s too rigid. We have many outcomes that we have mentioned today that we can’t put in there at al .”

—CEO of a better-resourced nonprofit grantee

Conclusion

It is clear that HUD has responded to GAO’s recommendation for improved data col ection. As a data col ection instrument, the FY 2014 Logic Model has improved potential compared to the FY 2011

7 0

Logic Model. The FY 2014 Logic Model’s instructions and glossary provide key guidance to grantees, its streamlined form has made more complete data col ection possible, and its mandatory metrics drastically reduced the amount of missing data.

There is stil room for improvement in several areas: reporting on program inputs, outputs, and outcomes; data quality; grantee capacity; and the ability to capture the full impacts of the wide array of work service coordinators engage in while serving public housing residents. First, there is a causal pathway between inputs, outputs, and outcomes not clearly captured by the Logic Model. The Logic Model is best suited to capture inputs and outputs that align with service coordinators’ activities and less suited to reflect outcomes, which may be beyond service coordinators’ activities, observation, or control. Second, while our 10 site visits cannot speak conclusively to the entire cohort of ROSS-SC

grantees, they suggest multiple issues with Logic Model implementation. Data reliability and data validity (as defined in exhibit 23) are recurring issues that weaken the data at multiple points, sometimes from a lack of understanding and other times out of concern for how data may reflect on grantees. Third, the capacity of grantees to col ect and report data is a core area for improvement.

Final y, the alignment between the application, the Logic Model (or subsequent reporting system), and grantees’ day-to-day milestones and achievements should be better coordinated and expanded to cover more of what service coordinators do to address resident needs. 16

16 For updates on program progress since completion of this evaluation, see HUD’s Preface to this report.

7 1

Chapter 5. Service Coordinators:

Work Responsibilities,

Qualifications, and Supports

Introduction

This chapter presents data and information we col ected via our site visits and service coordinator survey about the activities service coordinators perform, the number and types of residents they serve, and how they interact with their clients. The chapter also summarizes service coordinators’

experience and qualifications and the nature and quality of training and support they receive. This chapter responds to the fol owing research questions:



Which of the 10 functions recommended by HUD do service coordinators perform?



What additional functions do they add?



What are the service coordinators’ qualifications (education, experience), and how are they compensated?



How do they assess participant needs?



How are Individual Training Service Plans (ITSPs) used?



What is their total caseload? How many of those residents actively receive support?



What kinds of training and technical assistance do service coordinators receive from grantees, and do the service coordinators consider it adequate?

Key Findings



Service coordinators are highly experienced and qualified professionals, with most having 10

or more years of work experience and at least a 4-year col ege degree.

7 2



Service coordinators most commonly coordinate and oversee the delivery of services, market the program to residents, and monitor and track the ongoing provision of services. Less common functions include encouraging the formation of civic engagement and/or self-help groups and coordinating a local Program Coordinating Committee.



Service coordinator learned of resident needs through property managers and both formal and informal assessments.



Most service coordinators maintained a caseload of 50 or more residents (the minimum required by the grant), but over one-half see more than 150 residents. Those who saw fewer said this al ows them to better support the residents they do see make progress toward their goals.



About one-half of service coordinators met with individual residents every other week or more frequently. One-half met individual residents once a month or less.



Although more than half of service coordinators reported using Individual Training Plans (ITSPs), a smaller proportion reported using them with all participants. Most service coordinators reported viewing them as effective tools, especial y with unemployed residents.



The median compensation for full-time (35 or more hours a week) service coordinators was between $45,000 and $60,000, which is on par with the annual median wage for ful -time social workers in the United States.



Nearly all service coordinators receive training support from their employer, including guidance on service coordinator responsibilities.



Service coordinators we spoke with reported receiving limited technical assistance from HUD.

We note that new resources have been produced and distributed since the completion of our data col ection for this study, however.

7 3

Core Work Responsibilities

Service Coordinator Functions

SERVICE COORDINATORS PERFORM EXPECTED OUTREACH, COORDINATION, AND

MONITORING FUNCTIONS; FEWER COORDINATE PARTNER AND RESIDENT GROUPS

Survey respondents report performing most of the 10 recommended functions presented in exhibit 24

below, defined in the FY2014–FY2016 NOFAs, but also perform other activities to support residents more directly. 17 Of the recommended functions, more than four out of every five survey respondents coordinate services, perform quality assurance, monitor and track service provision, and market the program to residents (exhibit 24). Fewer (57 percent) say they are involved in helping to form resident groups to promote self-sufficiency, or in the formation of civic engagement or self-help groups (38

percent).

EXHIBIT 24: SERVICE COORDINATOR RECOMMENDED FUNCTIONS

Service Coordinators (%)

(N=215)

Expected Functions

Coordinate and oversee the delivery

90

of services

Market the program to residents

86

Monitor and track the ongoing

85

provision of services

Coordinate and sponsor educational

83

events

Track and report to HUD on the

81

progress of residents enrol ed in

the program

Evaluate the overal success of the

77

program

Assist grantee to create a resident

57

group to promote self-sufficiency

efforts

Encourage the formation of civic

38

engagement and/or self-help

groups

Coordinate a local Program

37

Coordinating Committee (PCC)

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

17 Services cannot be paid for directly from grant funds.

7 4

Notes: The service coordinators’ responses have been weighted to reflect the pool of service coordinators. Respondents were not asked about case management functions due to an error in survey fielding.

Local Program Coordinating Committees (PCC) are bodies that service coordinators coordinate to connect with “local service providers to ensure that program participants are linked to supportive services needed to achieve self-sufficiency” (HUD 2014). It is one of the 10 recommended service coordinator functions, but not a requirement. Although most service coordinators (70 percent) reported that the grantee had a local program coordinating committee, only 37 percent of survey respondents reported that they were directly involved in coordinating the PCC’s activities (exhibit 24).

Information from the site visits suggests service coordinators are heavily engaged in coordination activities even if they are not working through a formal PCC. Although the survey found only slightly more than a third of service coordinators coordinated a PCC, in at least 5 out of 10 of the sites we visited, service coordinators were actively involved with working with partners on residents’

programming needs.

Assessing Residents’ Needs

MOST RESIDENT REFERRALS COME FROM PROPERTY MANAGERS AND SERVICE

COORDINATOR INTAKE; FEWER COME THROUGH PARTNERS

Service coordinators have some flexibility to shape their program activities and partnerships to meet the needs of the public housing residents they serve. To inform these activities, they use a variety of methods for identifying resident needs. Service coordinators who responded to our survey reported that they learn about resident needs most frequently from property managers (85 percent), their own informal assessments (83 percent), and formal intake assessments (78 percent) (exhibit 25). Less common ways of identifying resident needs include referrals from a service provider operating at a property (44 percent), a service provider not operating at a property (36 percent), and institutional knowledge from the grantee (31 percent). Twenty-one percent of respondents indicated that they use other methods (written in under “other” in survey): 10 percent reported that residents reach out to service coordinators through walk-ins and self-referrals, and 6 percent reported getting information from other residents, neighbors, and family members. Other methods mentioned on the survey by one or two service coordinators each include learning about residents’ needs through a survey, personal observation, or during weekly site visits.

7 5

EXHIBIT 25: IDENTIFYING RESIDENT NEEDS

Service

Coordinators (%)

(N=215)

Source of Information

Property Managers

85

Informal Assessments

83

Formal Intake Assessments

78

Partner at the Property

44

Partner not at the Property

36

Institutional Knowledge from Grantee Organization

31

Walk-ins or Self-referrals (Write-in)

10

Other Residents, Neighbors, or Family (Write-in)

6

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: The service coordinators’ responses have been weighted to reflect the pool of service coordinators.

Most service coordinators we interviewed on site visits emphasized the utility of making direct contact with residents through activities such as walking around buildings to get first-hand information, build trust with the residents, and encourage participation in ROSS-SC programs and activities. At one better-resourced grantee, a service coordinator tracked move-ins, identifying residents who could benefit from ROSS-SC in the process and conducting an assessment to determine needs, whether for long-term or short-term assistance.

“[Our service coordinator] doesn’t hide in his office. [Out in the community he] does some face-to-face check on people. Meetings can happen with [him] to give advice. What can I do as a parent? What can I tel my son? He’s very responsive and very respectful.”

—Focus Group Participant at a lower-resourced PHA grantee

FORMAL ASSESSMENTS HAPPEN MOSTLY AT INTAKE, AND FOR FEWER THAN HALF OF ALL

RESIDENT PARTICIPANTS

The majority (78 percent) of service coordinators surveyed identified formal assessments—typical y one-on-one meetings where the service coordinator seeks to identify specific resident needs—as an important tool for identifying residents who need assistance. Service coordinators do not conduct formal assessments consistently, however, and not for all active participants. In terms of frequency of assessments, more than one-third (37 percent) indicated that they conduct a formal assessment only at intake, when the resident first moves into the development or first seeks out the service 7 6

coordinator’s assistance (exhibit 26). Another fourth of coordinators perform assessments annual y (23

percent), whereas others do so semi-annual y (15 percent) or monthly (8 percent). Resident association grantees were more likely to do annual (35 percent) and semi-annual (24 percent) assessments than PHA grantees (21 and 13 percent, respectively). PHA grantees were more likely to use a formal assessment for intake purposes only.

Less than one-half of service coordinators (45 percent) said that all participants had a formal assessment (exhibit 26). An additional 33 percent said that “most participants” had one, and 18

percent said “some participants had one”—only 1 percent said that none of the participants were formal y assessed. Service coordinators at resident association grantees were less likely to say that all participants were formally assessed (34 percent) than service coordinators at PHAs (45 percent).

EXHIBIT 26: USE OF FORMAL ASSESSMENTS Service

Coordinators (%)

Grantees (%)

(N=215)

(N=178)

Total

PHAs

Resident

Associations

Frequency Conducted

(N=137)

(N=21)

Only at Intake

37

39

22

Annual y

23

21

35

Semi-annual y

15

13

24

Monthly

8

7

5

Other

13

16

12

Share of Participants

Al Participants

45

45

34

Most Participants

33

31

45

Some Participants

18

20

19

None of the Participants

1

2

0

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: The service coordinators’ responses have been weighted to represent the pool of service coordinators and weighted to represent the pool of grantees. Percentages may not add up to 100 due to rounding.

Managing Caseloads

SERVICE COORDINATORS REPORT SERVING THE REQUIRED NUMBER OF RESIDENTS

Service coordinators are responsible for serving al eligible residents at their assigned public housing developments. They general y interact more intensively on a monthly or weekly basis with a smaller share of residents, however. The ROSS-SC NOFA requires that each ROSS-SC-funded service 7 7

coordinator actively serves at least 50 occupied housing units. This 50-unit measure is often used in the fields of service coordination and direct service provision to understand the level of effort required by a coordinator or provider to perform their job duties. We reviewed survey results using two definitions of caseloads to provide a detailed understanding of service coordinator work: the total number of residents they reported serving, and the number they reported serving in an average month.

More than 40 percent of service coordinators surveyed reported serving more than 150 residents in total, another 7 percent served between 101 and 150, and about 45 percent reported serving 100

residents or fewer (exhibit 27). Service coordinators hired by resident association grantees were more likely to serve more than 150 people (58 percent), compared with 37 percent for those working for PHAs.

EXHIBIT 27: TOTAL RESIDENTS SERVED PER SERVICE COORDINATOR

By Service

Coordinator

By Grantee

(N=215)

(N=178)

Number of Residents

Served by Each Service

Resident

Coordinator

Associations

PHAs

(N=21)

(N=137)

Fewer than 25

3

5

2

26 to 50

15

17

16

51 to 100

26

14

31

101 to 150

7

2

12

More than 150

45

58

37

Don’t Know

2

2

1

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: The service coordinators’ responses have been weighted to represent the pool of service coordinators and weighted to represent the pool of grantees. Percentages may not add up to 100 due to rounding.

In contrast to the uneven caseload distribution (exhibit 27), the number of individual residents who participate in ROSS-SC services or activities each month (exhibit 28) is more evenly distributed.

Although only 18 percent of service coordinators reported seeing more than 50 residents per month on a regular basis, another 46 percent were seeing more than 20 residents per month. Thirty-three percent were seeing 20 residents or fewer per month. Service coordinators we spoke to during site visits corroborated that they had regular, sustained contact with residents through a more formal case management approach, meeting with between 10 and 25 residents per week.

7 8

EXHIBIT 28: SERVICE COORDINATOR CASELOAD BY MONTH

Service Coordinators

(N=215)

Number of Residents

Participate Every Month

Fewer than 10

6

10 to 20

27

21 to 30

15

31 to 40

16

41 to 50

15

51 or More

18

Don’t Know

1

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: The 215 service coordinators’ responses have been weighted to reflect the pool of service coordinators. Percentages may not add up to 100 due to rounding.

RESIDENTS ARE MORE LIKELY TO WORK TOWARD PERSONAL GOALS WITH SERVICE

COORDINATORS WHO HAVE LOWER MONTHLY CASELOADS

About 60 percent of service coordinators answered the survey by saying that they see 20 or fewer residents per month specifical y to work with them toward their personal goals. As the monthly caseload increased, the percentage of residents using these meetings to work toward goals declined, dropping to 19 percent for service coordinators with a monthly caseload of 21 to 30, 10 percent for those with a caseload of 31 to 40 residents, and 2 percent for those seeing 41 to 50 residents regularly per month (exhibit 29). This pattern was observable across grantees regardless of the type of population served (family, elderly/disabled, or both). Because service coordinators do not use a standard set of qualifications for who to count in their caseload, these numbers should be understood as a broad trend, rather than a specific observation.

7 9

EXHIBIT 29: SERVICE COORDINATOR CASELOAD VERSUS MEETINGS TO WORK TOWARD PERSONAL

GOALS

Service Coordinators (%)

(N=215)

Number of Residents

Meet to Work Toward Personal Goals

Fewer than 10

27

10 to 20

33

21 to 30

19

31 to 40

10

41 to 50

2

51 or More

5

Don’t Know

1

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: The service coordinators’ responses have been weighted to reflect the pool of service coordinators. Percentages may not add up to 100 due to rounding.

Meeting with Residents and Tracking Progress

MOST SERVICE COORDINATORS MEET WITH INDIVIDUAL RESIDENTS AT LEAST ONCE A

MONTH FOR 45 MINUTES OR LESS

About one-half of service coordinators who responded to the survey reported that they met with individual residents frequently, at least every other week. The other half of service coordinators reported meeting individual residents monthly or less often (exhibit 30). One-on-one meetings typical y last 30-45 minutes (40 percent), or sometimes less than 30 minutes (33 percent). Information from the site visits indicate that service coordinators also tend to have many residents coming in as needed for requests, to fill out forms, or ask questions. Although these ad hoc meetings remain relatively brief, they can take up significant portions of service coordinators’ time.

8 0

EXHIBIT 30: FREQUENCY AND LENGTH OF MEETINGS WITH RESIDENTS

Service Coordinators (%)

(N=215)

F requency of Meetings

Two to Three Times a Week

10

At Least Once Every Two Weeks

22

At Least Once a Week

18

At Least Once a Month

20

Intermittently or as Needed

27

Length of Meetings

Less than 15 Minutes

2

15 to 30 Minutes

33

30 to 45 Minutes

40

45 to 60 Minutes

20

More than an Hour

4

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: The service coordinators’ responses have been weighted to reflect the pool of service coordinators. Percentages may not add up to 100 due to rounding.

NOT ALL COORDINATORS USE INDIVIDUAL TRAINING AND SERVICES PLANS, BUT THOSE

THAT DO USE THEM TO GUIDE AND TRACK RESIDENT ENGAGEMENT AND PROGRESS

The Individual Training and Services Plan (ITSP) is a tool provided by HUD that offers service coordinators a structure by which to identify resident needs, set goals, and recommend or refer them to specific partner services or programs (appendix A). ITSPs are not used by every service coordinator, nor for every participant. The majority (68 percent) of service coordinators reported using ITSPs, while 21 percent indicated they did not use them (exhibit 31).

As with formal assessments, the share of participants with an ITSP varied widely: 38 percent of service coordinators using ITSPs said that all participants had an ITSP; 36 percent said most participants; and 25 percent said some participants (exhibit 31). Service coordinators reported that they typical y use ITSPs as a means of tracking resident progress (87 percent), for resident guidance (85 percent) and as a plan for resident engagement (81 percent) (not shown).

8 1

EXHIBIT 31: USE OF INDIVIDUAL TRAINING AND SERVICE PLANS

Service Coordinators (%)

(N=215)

Service Coordinator Uses ITSPs

68

Share of Residents with ITSPs

(N=148)

Al Participants

38

Most Participants

36

Some Participants

25

None of the Participants

1

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: The service coordinators’ responses have been weighted to reflect the pool of service coordinators. Percentages may not add up to 100 due to rounding.

Logic Model data indicated much more widespread use of ITSPs than respondents reported on our survey. While both data sources reflect that they are not universally used, ITSPs are often used. Using the required fields in the Logic Model on the numbers of enrol ees with new or stil open ITSPs, the 2014 grantees reported 6,659 active ITSPs across 84 grantees in the first year of the grant, for an average of 79 service plans per grantee. This remained stable during the next 2 years of reporting.

Because the grantees reported serving 12,003 individuals in that first year, we conclude that 55

percent of participants had a new or continuing ITSP.

INDIVIDUAL TRAINING AND SERVICES PLANS ARE MOST USEFUL FOR RETAINING RESIDENTS

WHO ARE ABLE TO WORK

Reviews on the effectiveness of ITSPs in retaining participants in ROSS-SC were general y positive.

Among those service coordinators who reported using ITSPs, 81 percent said they were either “very”

or “somewhat” effective in retaining participants (28 percent and 53 percent, respectively). Only 11

percent said they were “somewhat ineffective,” and 2 percent said, “very ineffective.” Another 5

percent of respondents were unsure if the ITSPs were effective or not. Effectiveness may vary based on resident characteristics.

Service coordinators surveyed who reported using ITSPs believe that ITSPs are most effective with people who are able to work: unemployed residents (41 percent), families with children (26

percent), and working residents (15 percent). Very low numbers of service coordinators reported that ITSPs are helpful in retaining the participation of elderly residents (10 percent), residents with mental health needs (4 percent), and residents with physical disabilities (2 percent). Also, few service coordinators included non-English-speaking residents among the groups for whom ITSPs were most effective (2 percent); non-English speakers could be part of any of the other groups listed, however.

8 2

Service Coordinator Qualifications

Education and Work Experience of Service Coordinators

SERVICE COORDINATORS ARE EXPERIENCED, AND MOST HAVE BEEN IN THEIR POSITION FOR

MORE THAN 2 YEARS

In the ROSS-SC Notice of Funding Availability (NOFA), service coordinators are required to have “two or three years’ experience in social service delivery for low-income youth, adults, senior citizens and/or people with disabilities,” (HUD 2015). The survey results suggest the service coordinators meet these criteria, and most service coordinators have considerably more work experience. When asked how many years of total work experience they have, three out of every four service coordinators who responded had more than 10 years of experience (exhibit 32). Only 3 percent of service coordinators reported work experience that was less than 3 years. We cannot conclude, however, whether all work experience was social service delivery experience or as a service coordinator because the survey did not ask for specific roles in prior experience.

EXHIBIT 32: SERVICE COORDINATOR WORK EXPERIENCE

Service Coordinators (%)

(N=215)

Work Experience

1-2 Years

3

3-5 Years

9

6-10 Years

13

More than 10 Years

74

Didn’t Answer

2

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: The service coordinators’ responses have been weighted to reflect the pool of service coordinators. Percentages may not add up to 100 due to rounding.

Service coordinators also tend to stay in their positions; more than 66 percent of survey respondents have been in their position for more than 2 years, and more than 80 percent of coordinators have worked in their current position for more than a year (exhibit 33). Specifically, more than 70 percent of service coordinators from grant years 2014 and 2015 have been at their current position for more than 2 years, and a smaller share of coordinators employed via a ROSS-SC grant awarded in 2016 have held the job for more than 2 years, partly due to the shorter life of the grant to 8 3

date. This longevity is beneficial, as service coordinators build knowledge of their resident communities and the program over time.

EXHIBIT 33: SERVICE COORDINATOR MONTHS IN CURRENT POSITION BY GRANT YEAR

Grant Year (%)

(N=215)

Months in Current Position

2014

2015

2016

Fewer than 6 Months

7

7

5

7-12 Months

11

10

11

12-24 Months

11

9

24

More than 24 Months

72

70

56

Don’t Know

0

0

1

Didn’t Answer

0

4

2

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: The 215 service coordinators’ responses have been weighted to reflect the pool of service coordinators. Percentages may not add up to 100 due to rounding.

The relatively high retention rate of service coordinators also confirms the site visit findings.

Although not specifical y asked, respondents from at least 7 out of the 10 grantees we visited, including grantee staff and service coordinators, mentioned that they rarely have any turnover, except for internal promotions. Their longevity has benefits for both the program and the residents: Service coordinators demonstrate their commitment to the residents and are in a better position to ensure continuity of the program and maintain partnerships. One executive director who had experienced turnover in the program until hiring a particular service coordinator a few years ago noted that “folks

[that is, residents] have been locked in and focused” ever since.

SERVICE COORDINATORS ARE HIGHLY EDUCATED AND HOLD PROFESSIONAL

CERTIFICATIONS IN VARIOUS FIELDS

Although a 4-year degree is not a requirement under the ROSS NOFA for service coordinators, the col ege attainment rate among service coordinators is 75 percent, which is consistent with industry norms, where a bachelor’s degree in social work (BSW) is the most common requirement for entry-level administrative positions in social work (BLS n.d.). According to the survey results, of the service coordinators who do not have a 4-year col ege degree (24 percent), 71 percent had some col ege or associate’s degree, whereas others have either completed a GED or have earned vocational/technical certificates (exhibit 34). Col ectively, grantees serving elderly/disabled residents had the highest share 8 4

of service coordinators with at least a col ege degree (80 percent), compared with grantees that serve family populations (69 percent) or both groups (72 percent).

8 5

EXHIBIT 34: SERVICE COORDINATOR EDUCATIONAL ATTAINMENT

Service Coordinators (%)

(N=215)

Educational Attainment

Serving Family

Serving

Elderly/Disabled

Serving Both

(N=69)

(N=40)

(N=102)

Less than High School

1

0

0

High School Diploma, GED, or

Equivalent

4

3

3

Some Technical, Vocational, or

Business Courses

3

0

3

Vocational/Tech/Business

Certificate or Diploma

0

0

3

Some College

9

10

9

Associate’s Degree or Technical

Certificate

11

6

8

Four-Year College Degree

45

40

28

Some Graduate School

4

10

9

Graduate or Professional

Degree

20

30

35

Didn’t Answer

4

0

2

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: The service coordinators’ responses have been weighted to reflect the pool of service coordinators. Percentages may not add up to 100 due to rounding.

In addition to a traditional col ege education, service coordinators have acquired professional certificates that help them perform their work. Sixty percent of service coordinators indicate they have at least one professional certification. The percentage of service coordinators serving elderly/disabled residents who have a professional certificate (75 percent) is greater than the percentage for those who serve families (58 percent). Professional certificates that service coordinators self-reported in an open-ended question in the survey cover an array of topics including social work, self-sufficiency counseling, homeownership education, eviction prevention, mental health first aid, suicide prevention, and mediation services. Each of these enhances the abilities of service coordinators to identify residents’ needs and potential services and strategies that can help meet them. Some service coordinators also have certificates in administrative and management fields, such as financial planning, public housing management, paralegal, and grant writing.

8 6

EFFECTIVE SERVICE COORDINATION REQUIRES MORE THAN WORK EXPERIENCE AND

EDUCATION

In more than half of the 10 sites we visited, respondents—including partners, grantee staff, and service coordinators themselves—spoke about skil s beyond prior work experience and education that contribute to service coordinators’ strengths. These included having existing ties to the community, existing partnerships in place in the community, and, when applicable, language skil s. It also included having an understanding and vision for the community, empathy, and motivation to build relationships, both with residents and partners.

“(We are) Looking for someone with a personality that brings people in. Someone that is going to motivate and mobilize residents. Someone that is going to go door-to-door and spend time on couches.”

—ROSS-SC grant administrator at a better-resourced PHA grantee

Residents we met on site visits spoke positively about their service coordinators, and a handful of service coordinators made a strong impression on the residents they serve because of their extended level of commitment. Residents described these service coordinators as a core resource in their lives, even mentioning that service coordinators are accessible via home and cel phones outside of work hours. Residents trust these service coordinators with many life issues that may pose financial, logistic, or emotional barriers to meeting their self-sufficiency or aging-in-place goals. One resident said, “[The service coordinator] will call you after hours. If you bring it to her, she’l make it happen for you.” In interviews, these service coordinators frequently referred to their clients as like family, noting how they view their positions as a core aspect of their lives. These service coordinators typical y have been in their roles from the beginning of the current ROSS-SC grant program in 2008. Although not all service coordinators are able or expected to maintain this level of commitment, soft skil s, such as availability when on and off site, flexibility, and responsiveness, can contribute to higher levels of trust from residents.

8 7

“[With the service coordinator] there’s never ‘Sorry about your luck’ — there’s never that.

There’s ‘I don’t know, let me find out for you.’ There’s never, ever, ever been a ‘I can’t help you.’”

—Resident focus group participant at better-resourced PHA grantee

Service Coordinator Compensation

COMPENSATION OF SERVICE COORDINATORS IS ON PAR WITH THE MEDIAN ANNUAL

SALARY FOR SOCIAL WORKERS

The 2018 NOFA stated that service coordinators’ salary must be based on comparable salaries for similar professions; the amount that can be funded was capped by the NOFA at $70,000 (per year) for all three grant fiscal years covered by the survey. The median compensation for full-time (35 or more hours a week) service coordinators is between $45,000 and $60,000, which is on par with the annual median wage for full-time social workers in the United States, which was $49,483 annually or $23.69

hourly as of May 2018, according to the Bureau of Labor Statistics (exhibit 35, BLS 2019). More recent hires tend to have higher compensation than service coordinators who have been on the job for more than 2 years (exhibit 35). The exception is those service coordinators earning $60,000–$70,000

annually, more of whom have been on the job longer. Most service coordinators (77 percent) work ful time, and another 18 percent worked more than 20 hours a week.

8 8

EXHIBIT 35: SERVICE COORDINATOR COMPENSATION BY TENURE

Service Coordinators (%)

(N=210)

Compensation

Less than 2 Years

More than 2 Years

(N=70)

(N=140)

Less than $15,000

0

2

$15,000-$30,000

9

3

$30,000-$45,000

35

33

$45,000-$60,000

38

37

$60,000-$70,000

6

16

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: This table reflects 210 service coordinators’ responses (5 respondents did not answer this question), which have been weighted to reflect the pool of service coordinators. Percentages may not add up to 100 due to rounding.

Work Supports and Training

SERVICE COORDINATORS RECEIVE A VARIETY OF TRAINING AND SUPPORT FROM GRANTEES

Nearly all service coordinators receive training support from their employer: more than 90 percent listed at least one type of support they received from their grantee in their response to the survey.

The types of supports vary, however: service coordinators are most likely to receive guidance on their responsibilities as service coordinator (66 percent) and least likely to receive guidance on local service providers (46 percent) from grantee organizations (exhibit 36). In addition to the regular training opportunities listed in the survey, service coordinators wrote in that they receive other support such as financial management trainings, internal learning community, and staff management meetings.

8 9

EXHIBIT 36: SERVICE COORDINATOR TRAINING PROVIDED BY GRANTEES

Service

Coordinators (%)

(N=215)

Types of Support

Yes

No

Didn’t Answer

Guidance on Service

Coordinator Responsibilities

66

25

9

Regular Training Opportunities

63

28

9

Oversight

55

36

9

Guidance on Residents’ Needs

53

38

9

Guidance on Local Partners

46

45

9

Other

4

96

N/A

n/a = data not available.

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: The service coordinators’ responses have been weighted and aggregated to represent grantee level (178) results.

Percentages may not add up to 100 due to rounding.

Information from the site visits provided a more nuanced picture of what grantee training and support looks like on the ground. Most grantees we visited provide a mixture of in-house and outside support to service coordinators. During 6 of our 10 site visits, respondents—including partners, grantee staff, and service coordinators—referred to opportunities to attend conferences and conventions such as the National Coordinators Convention, American Association of Service Coordinators, 18 and NeighborWorks, 19 as wel as certification trainings, praising these opportunities for providing useful information and resources relevant to their jobs. These trainings cover a range of topics in service delivery, including suicide prevention, conflict resolution, child abuse, assertive engagement, trauma-informed care, coaching, and intergenerational community building. Service coordinators also described seeking out further training opportunities, with some noting that they could use more support in finding relevant conferences to attend. A few service coordinators spoke of attending conferences organized by HUD specifical y for ROSS-SC in the past and reinforced that this opportunity was a helpful learning opportunity and connected them to other grantees.

Although more limited, some service coordinators spoke of informal collaboration among themselves to support and learn from each other. At least two grantees also noted benefiting from col aboration on best practices with other grantees in the area. One service coordinator from a PHA grantee explained how service coordinators supported one another, “I talked to other ROSS-SC

18 American Association of Service Coordinators: https://www.servicecoordinator.org/.

19 NeighborWorks America: https://www.neighborworks.org/Training-Services/Training-Professional-

Development.

9 0

coordinators, actually FSS20 coordinators, but they were encouraging and came to talk to me about how I was running the ROSS-SC program, and they wanted to pick my brain about that as wel .”

SERVICE COORDINATORS ARE GENERALLY SATISFIED WITH SUPPORTS PROVIDED BY

GRANTEE ORGANIZATIONS TO HELP THEM ACCOMPLISH THEIR WORK

Service coordinators general y report high levels of satisfaction with supports provided by grantees.

Eighty-five percent of service coordinators surveyed responded that they are very or somewhat satisfied with resources available to performing their job duties (exhibit 37). Around three-fourths of service coordinators were very or somewhat satisfied with the training opportunities provided by grantees (79 percent), the frequency of feedback from grantees (74 percent), the quality of information on residents (74 percent), the quality of grantee feedback (72 percent), and the number (72 percent) and the quality (71 percent) of relationships grantees have developed with service providers.

20 Family Self-Sufficiency (FSS) is a separate program operated by HUD, which, like ROSS-SC, provides grantees with funding to connect participants to services available in their communities. FSS participants must be residents of public housing, live in a project-based Section 8 development, or hold a Housing Choice Voucher. Where households in any of these housing arrangements general y pay 30 percent of their income in rent, FSS

participants’ rent is frozen at the amount they pay at program initiation, even if their income increases. The difference between this amount and their new rent is stil col ected and placed in the escrow account, which they can access at program completion.

9 1

EXHIBIT 37: SATISFACTION OF SUPPORT FROM GRANTEES

Service Coordinators (%)

(N=215)

Very or

Very or

Type of Support Offered to Service Coordinators

Somewhat

Somewhat

Don’t

Didn’t

Satisfied

Unsatisfied

Know

Answer

Resources made available to you by grantee to perform

your job duties

85

8

5

2

Training opportunities provided by grantee

79

14

5

3

How often grantee provides you with feedback

74

13

10

3

Quality of information on resident needs grantee provides

74

13

10

3

Quality of feedback grantee provides

72

15

11

3

Number of relationships grantee has developed with

service providers

72

12

13

3

Quality of relationships grantee has developed with service

providers

71

13

13

4

Source: Urban Institute ROSS-SC Service Coordinator survey (2018).

Notes: The service coordinators’ responses have been weighted and aggregated to represent grantee level (178) results.

Percentages may not add up to 100 due to rounding. Due to a version control issue during the survey, question on satisfaction of “Guidance on your responsibilities as service coordinators” was not fielded and col ected.

TECHNICAL ASSISTANCE FROM HUD WAS LIMITED

Technical assistance from HUD appeared limited, compared with the various support service coordinators receive from their grantee organization and other relevant associations. 21 We did not receive consistent feedback during the site visits from service coordinators about technical assistance they may have received from HUD, although one grantee organization who had received the grant for multiple years noted that there used to be a more robust training component to ROSS-SC, but this had not been the case for some time.

In the absence of technical assistance specifical y focused on the ROSS-SC program, some grantees spoke during the site visits of leveraging other HUD programs. One grantee supervisor from a PHA attributed the successful ROSS-SC program partly to a strong agency col aboration between their FSS team and their ROSS-SC team, “When we talk about best practices for ROSS-SC, HUD has not really provided us with that much, but we are able to really lean on the FSS team. Just having that internally has been really supportive.”

21 As of data col ection in 2017 and 2018 conducted for this evaluation, a new HUD Exchange guidebook (2019) with webinars and additional program information and resources was not yet available:

https://www.hudexchange.info/programs/ross/guide/tracking-activities-measuring-performance-and-outcomes-

and-preparing-for-hud-monitoring-visits-and-audits/why-benchmarks-and-outcomes-matter/tracking-outcomes-

in-ross/.

9 2

“When ROSS-SC first got started, once HUD funded groups and contracts were signed, there was a training set aside. They pul ed everyone to DC and would tel all of the tenant presidents/leaders and employees they hired how to manage the program, showed them what papers they had to fil out.

That has not happened in a long time. The last webinar must have been about 2-3 years ago [that I last remember hearing about].”

—Service coordinator at a lower-resourced nonprofit grantee

THE AVAILABILITY AND ACCESSIBILITY OF TRAININGS AFFECTS THE LEVEL OF SERVICES AND

SKILLS THAT SERVICE COORDINATORS CAN OFFER RESIDENTS

Although grantees in a variety of contexts noted the need for more training support, the grantees we visited in smal er communities reported more difficulty accessing high-quality trainings. Service coordinators working in communities far from large urban centers must travel farther to access in-person trainings and use a significant proportion of their training budgets to get to and from training sites, according to service coordinators and grant managers. Sites we visited in large, urban areas reported lower levels of difficulty finding nearby training opportunities. 22

Conclusion

Service coordinators are the core of the ROSS-SC program, performing recommended HUD functions including assessing resident needs, managing caseloads, and tracking resident progress over time. The survey and site visit data show that service coordinators use a combination of formal and informal assessment tools to do resident intake and track progress. One-half to two-thirds of service coordinators reported having used an ITSP with at least some of their participants, whereas others reported never using it. In addition, while some service coordinators have noted the benefits of having a forum to coordinate with partners and other service providers, many leverage preexisting networks 22 As of data col ection in 2017 and 2018, a new HUD Exchange guidebook (2019) with webinars and additional program information was not available: https://www.hudexchange.info/programs/ross/guide/tracking-activities-

measuring-performance-and-outcomes-and-preparing-for-hud-monitoring-visits-and-audits/why-benchmarks-

and-outcomes-matter/tracking-outcomes-in-ross/.

9 3

in their communities and their own individual relationships with service providers rather than convening a formal Program Coordinating Committee.

Service coordinators are wel -educated and experienced, in line with qualifications detailed in the NOFA. Most have at least a col ege degree, three-fourths have at least 10 years of experience, and a majority also have some form of professional certification. Those who work with the elderly or disabled are more likely to have a degree or a professional certificate. Turnover in the position is low, so most service coordinators surveyed have 2 or more years of experience in the role. Compensation is consistent with the national median for social workers, and rarely approaches the cap set in the NOFA.

Most service coordinators receive guidance and training from their grantee organizations to help them perform their roles, and most reported being satisfied with training opportunities and other support they receive. Service coordinators indicated, however, that not al grantee organizations provide regular training, and that there is demand for more information about available training opportunities.

9 4

Chapter 6. Resident Experiences

and Local Contexts

Introduction

According to HUD’s definition, ROSS-SC program participants achieve self-sufficiency by “increasing earned income, reducing or eliminating the need for welfare assistance, making progress toward achieving economic independence and housing self-sufficiency, and helping to improve living conditions and enabling residents to age-in-place” (for elderly/disabled residents) (HUD n.d.a). This chapter focuses on how residents make strides toward the types of self-sufficiency ROSS-SC is intended to support and how ROSS-SC service coordinators and partners help them on this path.

The findings in this chapter are primarily gleaned from the 10 resident focus groups that evaluation team members facilitated during site visits, as wel as site visit interviews, survey, and administrative data analyses. Because the evaluation team did not have access to consistent baseline and outcome data for individual resident participants and across sites, the findings stop short of drawing causal conclusions about specific resident outcomes. Instead, this chapter documents areas where the evidence suggests that the ROSS-SC program has helped residents and identifies local contextual factors that impede or enable residents’ progress toward self-sufficiency or successfully aging in place.

The fol owing are the core research questions we address in this chapter:



What are outcomes for participants? (Increase in participant incomes, by population served; frequencies and types of referrals, and service take-up rates; additional outcomes)



What influences success? (Capacity of grantee, capabilities of service coordinator, characteristics of program participants, availability and coordination of services, local economic context, presence of other programs, additional factors)

In this chapter, we summarize areas in which residents have achieved gains in economic and housing self-sufficiency partial y attributed to participation in the ROSS-SC program, as wel as common chal enges they have faced. We also address possible benefits to residents that extend beyond the stated goals of the ROSS-SC program. Although these specific findings are not 9 5

generalizable, they indicate that ROSS-SC frequently acts as a vital resource for public housing residents. This chapter wil discuss resident experience with the ROSS-SC program, which includes the following:



Connections to resources, including referrals to services and resident uptake.



Finding employment and increasing income.



Moving toward economic independence.



Moving toward housing self-sufficiency.



Achieving and maintaining independent living while aging in place.



Other improvements to resident experiences.

It wil also highlight how local contexts affect residents’ abilities to effectively use the services to which service coordinators connect them, including:



Resources available from the local PHA.



Geographic and operational characteristics of public housing developments.



Resident access to transportation.



Resident access to childcare.



Resident access to health care.



Safety and security in the housing developments and surrounding communities.



Local cost of living.

Key Findings

Resident Experiences



Service coordinators are advancing both the ROSS-SC program’s goals and the goals of residents. Their efforts lead to residents becoming more aware of community resources, which in turn helps them access employment, increase income, live independently, and age in 9 6

place. Service coordinators and their community partners may also help residents increase their self-confidence, especial y through community building activities.



Site visit data suggest that residents access concrete resources like job training, health care, and government benefits through ROSS-SC service coordinators and partners. Logic Model data from FY 2014 grantees support these findings with additional information about earnings and health referrals.



The Logic Model did not al ow for reporting on and tracking individual participant outcomes and did not track other factors that may affect participants and their outcomes. Data available from the site visits and survey of service coordinators provide some evidence of the program’s overall benefit to residents but were not designed to measure specific individual outcomes.

Local Contexts



Strained PHA budgets and management can limit grantees’ capacities to support residents beyond the resources provided by service coordinators.



A sense of community among residents, or levels of isolation and security within and between public housing developments, influence residents’ abilities to move toward self-sufficiency.



Lack of access to transportation and childcare further limit resident progress toward self-sufficiency, as do high and rising costs of living in local communities.

Resident Experiences

Connection to Resources and Referrals to Services

RESIDENTS ARE MORE AWARE OF RESOURCES IN THE COMMUNITY

ROSS-SC grant administrators and partners alike emphasized that service coordinators increase resident familiarity with local resources and opportunities available to meet their needs. Resident focus group and interview participants from at least four grantees that we visited explicitly mentioned service coordinators’ impact on resident awareness of resources. These four grantees were lower-resourced sites, which may indicate that ROSS-SC service coordinators serve as a particularly crucial connector for residents in under-resourced and often geographically isolated areas.

9 7

“The service coordinator gave residents the resources of other community resources here in our county that they had never been aware of.”

—Staff member at partner organization of a lower-resourced PHA grantee

RESIDENTS ARE BETTER EQUIPPED TO ACCESS LOCAL RESOURCES

Service coordinators not only lead residents to become more aware of resources, they likely better equip residents to take advantage of them. A service coordinator observed the following outcome from their homeownership education and financial literacy program: “[Residents] are educated about the homebuying process and walking into a bank for anything. You know what to expect. There’s not that intimidation factor. Now they’re more aware of what’s going to happen, questions to ask. I feel like ROSS-SC has made a huge impact.” Overal , the ability of service coordinators to provide residents with a greater understanding of resources available in their communities and to put them in a better position to take advantage of them is key to their abilities to help residents achieve their self-sufficiency or independent-living goals.

RESIDENTS FOLLOW UP ON REFERRALS TO LOCAL RESOURCES

In their Logic Model data, FY 2014 ROSS-SC grantees reported positive outcomes23 through referrals to childcare, health care, employment, transportation, and other services for older adults and people with disabilities. Referrals for services over al 3 of their grant years resulted in positive outcomes for more than one-half of the recipients for childcare, services for older adults and people with disabilities, and health care (exhibit 38). Referrals associated with employment (for employment itself or for transportation services related to employment) resulted in fewer positive outcomes (less than half of recipients). While these data show generally positive outcomes from referrals made, they remain inconclusive because some metrics are optional and are therefore not reported across all grantees.

The data that do exist may be inaccurate, as discussed previously in chapter 4, including the number of outcomes exceeding the number of actual referrals and the inability to directly connect ROSS-SC

services and activities with reported resident outcomes independently of other influential factors.

23 As noted in chapter 4, we are using the “outcomes” variables reported in the Logic Model, noting that these are actual y a mix of outputs and outcomes.

9 8

EXHIBIT 38: REFERRALS RESULTING IN OUTCOMES, FY 2014 GRANTEES

Average Number of

Outcomes

Percent

Referrals per Site

Achieved

Outcomes

Outcome

Referral

Achieved

Elderly/Disabled

Services (N=70)

128

123

96

Obtained needed

services

Lived

Elderly/Disabled

independently/aged

Services (N=66)

120

107

89

in place and avoided

long-term care

placement

Elderly/Disabled

Services (N=71)

159

137

86

Improved quality of life

Childcare (N=72)

45

33

73

Obtained childcare

services

Health Care (N=57)

116

65

56

Obtained health care

services

Employment (N=54)

93

44

47

Obtained part- or ful -

time employment

Source: FY 2014-FY 2016 Logic Model data

Notes: We are using the “outcomes” variables reported in the Logic Model, noting that these are actual y a mix of outputs and outcomes. Average number of referrals per site only reflects sites that reported these numbers on the Logic Model. Mandatory metrics include childcare and elderly/disabled services referrals and outcomes. Optional metrics include health care and employment referrals and outcomes.

Despite issues with Logic Model reporting on individual resident outcomes, site visit interviews gave insights into positive trends in real outcomes that residents seem to be achieving.

“More touches, more stories of success, having folks that didn’t have a high school diploma, now have their GEDs. A person who didn’t have daycare for a child, now does, and now he or she is able to work. Those kinds of little wins, like that, they add up: instead of being once or twice a quarter, if they’re twenty times a quarter, that’s saying a lot I think.”

–PHA Director at lower-resourced PHA grantee

9 9

Employment and Income

MANY RESIDENTS CAN ACCESS JOB TRAINING, ALTHOUGH EMPLOYMENT OUTCOMES ARE

LESS EVIDENT

Residents can access a variety of supports for skills development and experience that prepare them for employment through the diverse set of job training programs made available through ROSS-SC

service coordinators, although the range and quality of these programs vary by site. Service coordinators at more than one-half of the 10 sites we visited offer connections to employment by organizing job fairs. Likewise, in more than one-half of the sites we visited, including elderly/disabled sites, we heard about service coordinators who partnered with employment assistance programs to provide residents with reliable access to job training. Residents who participated in focus groups reported finding these resources helpful, though they often wished there were even more training opportunities available to them. In terms of employment outcomes, one site noted that 17 residents obtained jobs through a job fair they hosted. A service coordinator at a different site said they connect resident teens to summer jobs by helping employment partners to hire 20 of them every summer so the teens can build job experience for their resumés.

Residents noted that even if they had not yet obtained a job through ROSS-SC partners or otherwise, these programs likely put them in a better position to obtain employment in the future.

One resident observed that the interviewing and resumé courses they took “also locked me in to getting resources they have through [a partner].”

“They have a computer system, and you can get into the websites and things so you can look up jobs. It’s a little more in depth…. I’m thankful for housing. I do want to move on, and I would like to have a job and I’l pay whatever I can to have my own house.”

—Resident focus group participant at better-resourced PHA grantee

ROSS-SC service coordinators and partners appear to help move residents toward employment and overall stability even if the data on how many residents obtain employment through ROSS-SC are limited.

1 0 0

RESIDENTS MAY SEE EMPLOYMENT AND EARNINGS GAINS, ALTHOUGH DATA ARE

INCOMPLETE

Logic Model and site visit data suggested that residents also experience positive earnings outcomes.

Performance period analysis on Logic Model data from FY 2014 grantees on earnings shows positive trends; a majority of grantees (64 percent, N=18) who reported on this metric met or exceeded their projections for the number of residents that would increase their income. Only 32 percent of grantees reported this metric, however. Grantees serving families achieved their income projections more frequently than those that serve only elderly/disabled residents. This discrepancy is not surprising as older adults and people with disabilities are less likely to be employed and have earned income that would grow over time, and therefore their incomes would have lower projected increases. Although we do not know the amount by which earnings increased and we cannot causal y connect any increased earnings exclusively to ROSS-SC participation, the positive employment and earnings trends shown in the Logic Model taken together with site visit data show that ROSS-SC service coordinators and partners are connecting residents to employment readiness programs, and in some cases to actual jobs.

Economic Independence

SERVICE COORDINATORS AND PARTNERS HELP ELIGIBLE RESIDENTS ACCESS INCOME

SUPPORTS AND BENEFITS

Service coordinators and partners in at least four of the sites we visited reported helping residents access a broad range of federal, state, and local government benefits. These included financial disability support, Social Security, veteran’s assistance, utility assistance, child support, TANF, SNAP, adult protective services, and tax preparation assistance. Service coordinators also helped residents understand and access health insurance through the Affordable Care Act, Medicaid, and Medicare.

One grantee we visited partners with the county Department of Social Services to link residents to those supports. Across all 10 sites, service coordinators most frequently work one-on-one with residents to assess their needs and help them fill out necessary forms to access these benefits.

1 0 1

“One resident with super high anxiety, she had trouble getting her Social Security and proving she had a disability… [the other service coordinator] and I were both persistent with her about following through with her appointments; she wasn’t going. At some point she started going. She started going more and more…and let us know she’s doing fine.”

—Service coordinator at a better-resourced PHA grantee

Although site visit data suggest residents who participate in ROSS-SC access benefits that were previously unknown or inaccessible to them, current data do not allow us to identify specifically how much assistance they receive, either in cash assistance or in level of services.

Housing Self-Sufficiency

HOMEOWNERSHIP AND FINANCIAL LITERACY PROGRAMS MAY IMPROVE HOUSING

STABILITY AND OPPORTUNITY

As noted in the partnerships chapter, at least four sites provided homeownership education, weatherization, or other home assistance programs, usually through financial literacy programming. A resident at one of these sites described how a service coordinator connected her to a service that helped her improve her credit score by 200 points, putting her in a better position to purchase a home.

Although we do not know whether participation in these programs actually led to homeownership, residents appear to have accessed training to assist them with potential y becoming homeowners in the future. In the case of Tribal y Designated Housing Entities (TDHEs), participants may have improved their living conditions through weatherization of their homes; this is due to the expanded authorities TDHEs have to serve their communities more comprehensively, including work with homeowners. 24

24 TDHEs are different from public housing agencies (PHAs) because they may be a department within the tribe, tribal housing authority with separate board of commissioners, or a nonprofit organization. Additional y, the Indian Housing Block Grant provides a funding stream and pathway for tribal nations to plan and develop affordable housing on Indian reservations and Native communities. TDHEs are also eligible borrowers for Section 184 Indian Home Loan Guarantee Program, which is a tribal homeownership program. See

https://www.hud.gov/program_offices/public_indian_housing/ih/codetalk/tribalhousing for more information.

1 0 2

Independent Living while Aging in Place

ACCESS TO HEALTH SERVICES SUPPORTS INDEPENDENT LIVING

Residents who participated in the two focus groups held at elderly/disabled grantee sites cited many ways the efforts of service coordinators supported them to age in place by receiving assistance with choosing doctors and getting to appointments, as wel as bringing in home health aides and participating in health screenings, breast cancer screenings, and smoking cessation programs. Such services allow older adults and people with disabilities to take proactive measures to improve or stabilize their health and live independently. At least two sites’ service coordinators engaged area or county aging services as partners in this area.

“With our seniors, we like to help them and listen as much as we can…making sure their bills are being paid on time, [and that they are] going to doctors’ appointments.”

—Service coordinator at lower-resourced PHA grantee

One older adult resident shared that receiving transportation to health appointments through a ROSS-SC partner makes a huge difference in the ability to live independently and age in place, saying,

“[A partner I was connected to through the service coordinator] takes care of all of my medical needs.

They take me to the doctor. I don’t need to have my son come and get me.” Transportation assistance further al ows residents to access health and other services and mitigates residents’ need to rely on family members or others for assistance.

FOOD ASSISTANCE HELPS BUT MAY NOT ALWAYS BE ENOUGH TO MEET RESIDENTS’ NEEDS

While food assistance is a common service that many populations access through the ROSS-SC

program, it is especially important for elderly/disabled grantees, as their residents are more likely to have challenges accessing and preparing food due to mobility limitations. In addition, partners serving older adults frequently use meals and meal preparation as a way to bring clients together, fighting isolation in addition to providing food. Residents from one focus group at a better-resourced elderly/disabled grantee noted that food assistance was available and used through ROSS-SC partners, but suggested that many residents needed more assistance than they were receiving. A resident in that group stated, “In this building there are people who don’t get regular meals. I started a program 1 0 3

where once a month I cook a big dinner where anyone can come and eat for free. Paid out of my pocket.” Residents did acknowledge a variety of food assistance programs offered through ROSS-SC

partners, however, such as a mobile market and food boxes that were available on site. Residents in a focus group at a lower-resourced PHA noted that though they may be struggling financial y, they stil may not be eligible for food assistance programs because of income or age restrictions that are outside the control of service coordinators.

SOCIAL ACTIVITIES HELP BUILD COMMUNITY AND REDUCE ISOLATION

Older adults and residents with disabilities also noted many social activities coordinated through

ROSS-SC enable them to build community with their fel ow residents and reduce isolation.

Programming included such activities as dancing, art, exercise classes, a boating trip, excursions out to dinner and to theater performances, and weekly community meals. Residents, partners, and service coordinators in at least six sites we visited indicated that these kinds of socialization programs were available in their communities. Though these programs were helpful, residents suggested even more programming could be beneficial, such as computer literacy programs (or computer access in general) or a weekly tea on a rotating topic. Such socialization programs enable residents to forge relationships with each other and to participate in both their communities: the one within their housing development and the larger community surrounding it.

Other Improvements in Resident Experiences

PARTICIPANTS EXPERIENCE INCREASED ACCESS TO HEALTH SERVICES AND SUPPORTS TO

ADOPT OR MAINTAIN HEALTHY BEHAVIORS

Residents we spoke with highlighted access to health services as a key benefit of ROSS-SC

programming. Notably, service coordinators help residents access preventative care through referrals and transportation to appointments, insurance enrol ment, and bringing in partners to provide health education, screenings, or help with activities of daily living, especial y in the case of older adult residents. Although the data do not allow us to conclude that ROSS-SC has led to concrete improvements in health outcomes, residents discussed adopting improved health behaviors that are associated with improved health. For example, residents shared stories of quitting smoking, losing weight, and improving their diet through exercise and nutrition programs provided through or by service coordinators. One grant manager told a poignant story of a resident who received pro bono eye surgery thanks to the efforts of a service coordinator. With her improved vision, the resident no longer relies on her mobility-impaired husband to help her walk to and from her job. Stories like this 1 0 4

demonstrate the vital role ROSS-SC service coordinators and partners can play in improving resident overall health and quality of life.

Much of the health programming that residents access through ROSS-SC service coordinators and service partners targets prevention and education, making it difficult to measure direct impact on improved health outcomes. Site visit data suggest, however, that residents do find value in these services. Residents in one focus group noted that the healthcare providers who come on site to give presentations would call them back to make sure they are able to obtain needed health services.

Service coordinators at multiple sites also pointed out that helping residents to enrol in health insurance represents a cornerstone of their health programming. They noted that accessing health insurance provides economic stability in addition to improved health by reducing costs and opens the door for residents to be more proactive about their health. The PHA director at an elderly/disabled grantee similarly indicated that programs coordinated between ROSS-SC grantees and service partners both improve residents’ health and al ow them to remain in their units, rather than moving to a nursing home or other care facility.

“We real y looked at these individuals with mental health issues, and they have specific needs. That’s one of the reasons too that we saw the need for service coordinators, to make sure [residents] stay lease compliant, and make sure that there isn’t this revolving door of people with mental health conditions.”

—PHA director at better-resourced PHA grantee

ROSS ACTIVITIES BUILD COMMUNITY AND RESIDENT SELF-ESTEEM

Residents and partners noted that participation in ROSS-SC programs and services have led them and other ROSS-SC participants to have a more positive view of their abilities and participate in activities they would not otherwise join. Both partners leading group programs and residents who were participating in them reported that self-esteem grew as residents forged friendships and were able to share about their goals and experiences. A staff member at a partner organization providing mental health services to low-income families through ROSS-SC, mostly in the form of group counseling, noted how residents build friendships, share experiences with each other, and improve their self-esteem. This partner observed, “There are some [residents] that were very shy and wouldn’t open up.

[Now they are] not afraid to share their problems. If something’s stressing them out they share it. The 1 0 5

others offer advice and encouragement….I have one lady that just started coming, she had been here for 20 years and did not know anyone else in the group. Now she’s connected. They go shopping or get breakfast or go to the park.” Even other programs with an explicit focus outside of mental health or socialization, such as career development courses, often led to positive self-confidence or socialization outcomes for participants. One ROSS-SC grant administrator observed how engagement in ROSS-SC activities could spil over into greater involvement and larger life pursuits: “Residents who are part of ROSS-SC are more engaged overall—involved in their kid’s school, getting a job, working toward their education.”

“The exciting thing when you go to one of [the resident] graduations [from a career development program] is the bond they’ve developed among themselves. They’ve developed a support team just from doing a class together. My opinion is that the single biggest barrier for our residents is self-confidence. If you don’t have self-esteem you can’t get to that next step. The teamwork they build in those classes real y helps that piece of it.

It’s become the prerequisite foundation course for almost everything else we do with the residents. It’s to teach them the basics, but the self-esteem is the primary benefit.”

—ROSS-SC grant administrator at a lower-resourced PHA grantee

Local Contexts

By design, ROSS-SC service coordinators depend on external resources to help residents meet their self-sufficiency and aging-in-place goals. In chapter 3, we discussed the important role that partners play in determining the specific nature of the ROSS-SC program at each grantee and the diversity of approaches that service coordinators employ in engaging with them. ROSS-SC service coordinators operate in diverse contexts and with varying resources, both within their grantee organizations and in the housing developments themselves. Furthermore, characteristics of the broader community, from its geographic location to the local economy and service infrastructure, play a role in how effective service coordinators can be in helping residents reach their goals. These contextual factors can either create an enabling environment that provides supports and encouragement to residents who seek higher levels of self-sufficiency or better ability to age in place, or they can create barriers to achieving these goals.

1 0 6

This section describes contextual issues that we identified during our research as influential on the ability of ROSS-SC to help residents achieve the program’s goals. Taken together, these issues indicate areas that the ROSS-SC program and other service coordination efforts should consider for setting program goals and achieving outcomes and how best to leverage opportunities and anticipate and navigate through chal enges.

PHA Resources and Property Management

LIMITED RESOURCES FROM PHA OPERATION FUNDS CAN UNDERMINE SERVICE

COORDINATOR EFFECTIVENESS

Housing authority staff and grant managers both mentioned limited funding for PHA programs and general operations as inhibiting their ability to meet the needs of residents. Lack of funding for PHAs also affects the availability and coordination of services because sites frequently leverage multiple funding streams and in-kind resources for purposes other than the ROSS-SC program. For example, PHA and nonprofit staff members at some sites discussed only having enough funding to refer residents to services off site, rather than to bring partners on site. As noted in chapter 4, onsite services often see higher participation rates, potential y due to fewer logistical barriers to participation.

MAINTENANCE CHALLENGES CAN PREVENT RESIDENTS FROM FOCUSING ON HIGHER GOALS

When asked what residents in their public housing community struggle with most in their living situation, the dominant answer in focus groups at all 10 sites was maintenance issues. Issues ranged from slow response times to larger problems, such as the reported lack of door locks across a development. Although addressing these kinds of issues fal s outside the scope of the service coordinator’s role, the lack of predictability in property management, combined with the compliance-oriented nature of public housing management, possibly create higher levels of stress and distrust among residents for al staff they associate with the housing authority, including service coordinators.

Characteristics of Residents, the Developments, and the Surrounding Communities SENSE OF COMMUNITY AMONG RESIDENTS CREATES AN ENABLING ENVIRONMENT

Residents in 3 of the 10 sites we visited cited a strong sense of community as a significant asset in their developments. These residents noted specifical y that this support enables a sense of security 1 0 7

when seeking out opportunities because they can count on neighbors to watch children and keep an eye on the property while they are away. As one resident put it, “We got each other’s backs.” Many of the focus group participants at one grantee we visited had been relocated from a development that had been demolished. Most participants are currently living in private housing using vouchers to subsidize their rent while they wait for the PHA to rebuild. Although they acknowledged that their former development was in a state of disrepair, they lamented the loss of the community they had, including a sense of security and freedom due to the supports of their neighbors. Residents said of their former home, “They push the bad part, and they never say the good. People that lived there loved it. [The development] wasn’t as bad as people thought.…We were doing big things.”

THE LOCATION OF PUBLIC HOUSING CREATES CHALLENGES FOR RESIDENTS IN ACCESSING

SERVICES, EMPLOYMENT, AND COMMUNITY AMENITIES

Many of the public housing developments we visited are located far from the services and amenities that residents need to work toward self-sufficiency. This isolation is particularly severe for sites in small communities; at one site, the public housing development is almost 3 miles from the human services office, a partner for many key services. Most centers of employment and health care are even farther. In other cases, service coordinators that split time between multiple developments reported spending significant time traveling between the sites. At one site, a service coordinator responsible for residents at two developments told us that he was unable to provide equal services to the two sites, since only one development has space to host partner activities, and the other is situated on the opposite end of town. When asked, service coordinators at 3 of the 10 sites we visited reported distances between developments as a barrier to providing quality services or resident progress toward self-sufficiency. Lack of adequate or inexpensive transportation options often compounded geographic isolation between developments, surrounding communities, and services.

LANGUAGE BARRIERS CREATE CHALLENGES FOR RESIDENTS IN CONNECTING WITH ROSS-SC

AND SERVICES

At the grantees we visited, we heard about language barriers affecting the level of engagement with ROSS-SC activities. Staff at partners shared that they were aware that non-English-speaking residents wanted services such as education, but language was a substantial barrier to enrol ing in services, let alone pursuing activities that might help them become self-sufficient. One partner that provided GED

classes recounted how difficult it was to get non-English-speaking students to participate in programming due to language barriers.

1 0 8

“We offered GED courses before, and the [non-English-speaking] students just didn’t show up. But I wil say, if you were to pull dozens of ESL files right now, what does it say number 1 that they want? GED—and they don’t even know what it stands for.”

—Staff member at partner organization of a lower resourced resident council grantee Grantees did their best to manage when there was not sufficient language capacity on staff to work with residents. As one service coordinator stated: “Sometimes I have to use Google Translate, and it’s not perfect, but it works; you do what you have to do to help residents, period. It’s whatever it takes.” Some grantees are better equipped to reach out and develop relationships with residents who do not primarily speak English. One grantee we visited with a high proportion of Spanish-speaking residents, for example, employed bilingual service coordinators. Another grantee we visited, however, had residents from a variety of linguistic backgrounds, making the hiring of staff based on linguistic skills impractical. The challenge is also greater for grantees with populations who speak less commonly known languages. Service coordinators noted that having to duplicate efforts or services in multiple languages and cultural contexts was another barrier in an environment where time and resources are frequently scarce and staff with the requisite backgrounds are few.

RESIDENTS’ PERCEPTION OF VIOLENCE AND SAFETY WHERE THEY LIVE INDICATES AN

UNADDRESSED RESIDENT NEED

Perceived violence or lack of security also surfaced as a clear priority for residents participating in focus groups. Residents reported violence as an issue in al 10 sites; 6 sites noted that it was a significant problem. Common complaints included police responding slowly or not at all to 911 calls from the developments; a lack of security measures such as door locks, area lighting and cameras; and non-residents using public housing developments as a venue for illegal activity. Residents reported feeling unsafe and having poor relations with local police, citing incidents of harassment. In interviews, service coordinators corroborated these concerns, noting that violence is a significant chal enge for providing services to residents. Residents and service coordinators both noted that feeling unsafe keeps residents in their houses at times during which they could otherwise access educational or other services.

1 0 9

THE RELATIVE COST OF LIVING MAKES ACHIEVING HOUSING AND ECONOMIC SELF-SUFFICIENCY MORE DIFFICULT

Although living in areas of economic growth can provide opportunities for employment, increases in the cost of living can create significant barriers to public housing residents who are seeking to work toward self-sufficiency. High and rising rents can make it difficult for residents to save enough to cover nonsubsidized housing costs, especial y when the costs of other necessities—such as transportation, food, utilities, and health care—increase as wel . Although this issue was apparent when we visited grantees in large urban centers, we also heard about this in smal cities and rural communities where residents and service coordinators reported rising costs offsetting gains made through services.

Although resident focus group participants didn’t speak extensively about supports for these issues, PHA staff observed that the presence of other programs such as other housing supports can offset costs and help residents save money, allowing them to build a personal safety net. One grant manager suggested that the inclusion of an escrow account for ROSS-SC similar to the FSS program would allow residents to better prepare for rising costs as they move toward greater self-sufficiency.

Access to Transportation and Childcare

LACK OF ACCESS TO RELIABLE TRANSPORTATION MAKES ACCESSING SERVICES, LIFE

NECESSITIES, AND OPPORTUNITIES FOR EMPLOYMENT DIFFICULT

At 8 of the 10 sites we visited, residents, service coordinators, and grant managers reported that transportation represented the biggest barrier to resident success. Lack of transportation is a particular challenge for those who frequently travel outside of regular commuting hours, such as second- and third-shift workers. Limited transportation also creates challenges for older adults because they may have trouble accessing essential services and medical care. Residents specifical y mentioned lack of access to employment centers such as warehouses and industrial manufacturing zones as a barrier to employment. While noting that transportation stil represents the largest barrier to resident employment and growth, staff and residents at one rural site expressed that the buses they do have are a lifeline for connecting people to employment and basic amenities like shopping. Of the two sites where respondents did not mention transportation as a barrier, one is in a major urban area with a robust bus network, which they cited as necessary for the success of the program. The other site leverages external funds that provide extensive transportation support for activities, including job training and other types of education.

1 1 0

LACK OF ACCESS TO AFFORDABLE CHILDCARE INHIBITS ABILITY TO PARTICIPATE IN ROSS-SC

SERVICES

Residents at 4 of the 10 sites we visited mentioned lack of childcare or children’s programming as a deterrent to resident participation in the ROSS-SC program. Service coordinators at two sites did mention partnering with local Head Start programs to connect residents to childcare. Residents at one grantee noted that they used to be able to bring their children when they attended ROSS-SC partner programming, but it is no longer allowed. Partner programming at this site is mostly in the evening, after work hours, but without included childcare.

“If we had a designated childcare agency, that would get people in [work] training

[programs].”

—Service coordinator at better-resourced PHA grantee

Conclusion

Although the data available preclude drawing firm conclusions about outcomes associated with participation in ROSS-SC, our data suggest that the program has connected participants with services that they would not have accessed otherwise and supported their efforts to move toward self-sufficiency or to age in place. Interviews with service coordinators suggest that they frequently connect residents with resources and services of which they would not otherwise have been aware.

Through service coordinator efforts, residents gain access to valuable training opportunities, including job training, and in some cases may obtain employment and/or increase their earnings. Residents reported improvements in their quality of life, building confidence and skills to seek out education, employment, and other resources. Barriers to success in the program continue to exist but may be opportunities for future program adaptations or resources. Issues such as transportation, childcare, and maintenance could be addressed with appropriate community partners, or through increased federal dol ars and an expansion of the service coordinator’s role to provide direct services that may be missing in the community.

1 1 1

Chapter 7: Study Implications and

Recommendations

Introduction

ROSS-SC plays a critical role in expanding access to services that help public housing residents move toward self-sufficiency and successful aging in place. This important role is true for both high- and low-resourced PHAs and grantees. This first evaluation of ROSS-SC focuses primarily on process and performance and secondarily on understanding program outcomes for residents. Through our data col ection and analysis of Logic Model data, a national survey of service coordinators, 10 site visits and resident focus groups, we have learned a significant amount about (1) how reporting and monitoring were conducted through Logic Model grantee data, (2) what service coordinators do, what their work looks like national y, and the value they bring to their communities, (3) how local service partners interact with ROSS service coordinators and their residents, and (4) grantee and resident perspectives on the opportunities and chal enges of the ROSS-SC program. Outcomes attributable to ROSS-SC are more difficult to assess. Activities and services available vary across sites, and sometimes vary within a single grantee that is serving multiple public housing developments. Service coordinators are encouraged to tailor their activities to the specific needs of their residents and may be either supported or constrained in how they do this by resources available in the larger community. This contributes further to differences in actual program implementation across sites.

This chapter considers findings across al evaluation activities and presents a series of recommendations for improving how the program works and what it can achieve. This will help improve future evaluations of ROSS-SC outcomes. Recommendations focus on the following:



Establishing a theory of change.



Improving grantee performance measurement.



Improving understanding and supports for what service coordinators do.



Maximizing service partnerships.



Meeting resident needs.

1 1 2



Sustaining the ROSS-SC program. 25

Establishing a Theory of Change

Two Pathways to ROSS-SC Implementation

STRONGER LINKS BETWEEN PROGRAM GOALS, OUTPUTS, AND DESIRED OUTCOMES COULD

IMPROVE PROGRAM DESIGN AND IMPLEMENTATION

Charting an explicit path between ROSS-SC resources and desired outcomes for grantees and the residents they serve could provide useful guidance to grantees about how they could improve their program and provide HUD data which that wil al ow program managers to evaluate the effectiveness of program design. Ultimately, in order to better understand grantee performance, outcomes reporting should be grounded in an explicit theory of change (exhibit 39) that shows expected causal links between what service coordinators do and the outcomes residents achieve.

EXHIBIT 39: WHAT IS A THEORY OF CHANGE?

A theory of change outlines a sequence of steps expected to lead to a desired set of goals. These steps should reflect the key program design elements linked together in a causal pathway. A clear set of assumptions help explain why one step should lead to another as part of a necessary path to achieving desired outcomes. Whereas logic models general y reflect a linear path connecting a specific set of inputs, outputs, and outcomes, with a focus on implementation and monitoring, theories of change focus on big picture causal connections in a more flexible format, including allowing for multiple non-linear paths and feedback loops.

Based on evidence col ected across application and Logic Model data, service coordinator survey, site visits, and the review of other program materials such as NOFAs over time, we observed two different theories of change that service coordinators appear to fol ow for improving resident outcomes: a case management approach versus a service connection approach. Although both start from an initial assumption that the primary barrier residents face in achieving their desired goals is a lack of connection to appropriate community service partners, they diverge in the level of direct involvement service coordinators take in participants’ uptake and use of partner services.

25 Please see the HUD Preface for program updates.

1 1 3

• Case management approach: This approach reflects formal ROSS-SC program descriptions, service coordinator functions, and reporting requirements through the FY 2014 Logic Model. It assumes that service coordinator outreach results in a formal intake process with interested residents, fol owed by a detailed assessment of their needs and referrals to services to meet those needs. Residents then access the services to which they have been referred and achieve associated outcomes. They remain in regular contact with service coordinators who help them track their service access and outcomes. Through an iterative process, residents and service coordinators continue to identify unmet resident needs and link to needed services to achieve a set of diverse resident outcomes over time.

• Service connection approach: This alternate approach appeared to be in use at multiple sites visited for this study. According to this theory of change, service coordinators focus on creating opportunities for residents to connect directly with community service partners through activities and events, rather than focusing on structured intake, needs assessment, and one-on-one case management. Service coordinators may or may not be aware of the ultimate service accessed, or resulting resident outcomes, if they do not have formal relationships with the community service providers or residents.

These two pathways can be fol owed simultaneously by grantees, and even by participants.

However, the resource and partner environments within which grantees work may affect their approach. Service coordinators may also have different skil s, communication styles, and different approaches to service provision. Resident needs and priorities may vary in nature and shift over time.

Some needs may be more conducive to case management, whereas others can be met through a service connection approach. Having multiple potential pathways to achieving the same goal provides options and flexibility, even while they may complicate tracking and measurement, as discussed later in this chapter.

A Case Management Model

A CASE MANAGEMENT MODEL PROMOTES STRONG LINKS BETWEEN SERVICE

COORDINATORS, RESIDENTS AND PARTNERS, AND OUTCOMES TRACKING, BUT MAY FACE

IMPLEMENTATION BARRIERS

Exhibit 40 below presents the assumed steps leading toward outcomes using a case management approach. We discuss each step along with a summary of what our evidence reveals on whether or how that step is occurring in current grantee program implementation, and barriers that arise.

1 1 4

EXHIBIT 40: IMPLICIT THEORY OF CHANGE FOR ACHIEVING RESIDENT OUTCOMES THROUGH ROSS-SC

PROGRAM, CASE MANAGEMENT APPROACH

1 1 5



Outreach: Service coordinators are creative in reaching out to residents, spending a significant proportion of their time—as much as one day a week or more—on outreach and trust-building activities. This level of effort speaks to the difficulty of engaging a population with diverse needs and resources, language barriers, low levels of trust, and high levels of trauma.



Intake: Many service coordinators use a formal intake process, but less than 40 percent surveyed do this with all their residents. There are many reasons—both positive and negative—that a resident may never sit down for a one-on-one intake process with a service coordinator. This includes having their needs met through some other ROSS-SC-sponsored activity or encountering barriers that keep them from participating.



Needs Assessment: Assessing resident needs can be a key part of the intake process, as well as an ongoing activity over a longer-term engagement between a service coordinator and resident to reassess changing needs over time. Sometimes residents may approach service coordinators with emergencies that trump more long-term planning.



Referrals to Services: Service coordinators are resourceful in building service partnerships to meet resident needs, and partners often see service coordinators as valuable connectors to potential clients whom they may be able to serve. Sometimes, however, needed services may not be available in the community, may have limited capacity compared to demand, or may be difficult to access due to location, schedule, or costs.



Services Accessed: The goal of providing a referral to a resident is to have them access the referred service so their needs can be addressed. On site visits, residents told us how valuable the services they have accessed are in meeting their needs. We also learned about some barriers to access. These included difficulties in accessing services located off site due to inadequate transportation services and a lack of childcare services. Residents may also choose not to access services; they may have more pressing concerns around housing stability, personal emergencies, and personal and family safety. We also heard evidence of community service partners closing with no other local alternative to which to refer residents. Final y, site visits clarified that there is little evidence of active data-sharing regarding resident participation and outcomes between ROSS-SC service coordinators and service partners, placing the impetus on service coordinators to fol ow up with residents regularly and rely on resident recall to document services accessed.



Resident Outcomes: Using a case management approach promotes regular communication between service coordinators and residents regarding services accessed and any outcomes 1 1 6

achieved. Although this regular communication may encourage residents to continue to participate in service activities, it is hard to link outcomes directly to resident interaction with a service coordinator. Residents may achieve positive or negative outcomes due to a number of factors that go beyond their interaction with a ROSS-SC service coordinator or that may be beyond the service coordinator’s control. For example, ROSS-SC participants may report increased income or education despite not having used ROSS-SC-referred services in these areas. Personal or family barriers may also intervene, such as losing a job due to employer restructuring, having a health emergency, or experiencing an eviction. There may also be a mismatch between services accessed or available and specific resident needs, such as a resident connecting to a job training program but not having access to consistent childcare and transportation services necessary to attend.

A Service Connection Model

A SERVICE CONNECTION MODEL CAN BE EASIER TO IMPLEMENT TO REACH MORE RESIDENTS

BUT MORE DIFFICULT FOR TRACKING SERVICE ACCESS AND OUTCOMES

A common strength cited by ROSS-SC grantees, service coordinators, and residents is the program flexibility that al ows service coordinators to interact with residents in a variety of ways. The service connection approach, outlined in exhibit 41, also starts with outreach to residents but diverges from there.



ROSS-SC-Sponsored Activity: Many service coordinators host valuable events geared toward bringing together a variety of partners through a service fair and publicizing the event to residents. Residents may not go through a formal intake process with the service coordinator prior to attending the activity. In fact, these events may reach residents who are less likely to schedule and participate in a one-on-one meeting with a service coordinator.



Connection to Partners: These events al ow residents to connect directly with a partner.

Community service partners at multiple sites expressed their success at connecting with residents at these types of events.



Services Accessed: Because the ROSS-SC-sponsored activity directly connects residents and community partners, neither the resident nor the community partner may fol ow up directly with the service coordinator to report a successful connection. Some residents may never directly interact with the service coordinator. Fol owup in these instances may be based on 1 1 7

formal or informal data-sharing agreements with the community partners, if present. The barriers to accessing services under this approach are similar to those present under the case management approach.



Resident Outcomes: If neither community partners nor residents report back on outcomes as a result of a connection made during a ROSS-SC-sponsored activity, service coordinators have no method of tracking resident outcomes from this service connection approach. If tracking does occur, outcomes may not always be directly associated to resident ROSS-SC

participation, and personal barriers and service-needs mismatches might prevent outcomes achievement.

1 1 8

EXHIBIT 41: IMPLICIT THEORY OF CHANGE FOR ACHIEVING RESIDENT OUTCOMES THROUGH ROSS-SC

PROGRAM, SERVICE CONNECTION MODEL

Defining and Measuring Grantee Performance

Although we are aware that the ROSS-SC program has adopted the Standards for Success for future data col ection and reporting, these recommendations focus on the FY 2014 Logic Model, since this was the ROSS-SC data reporting system being used at the time this study was commissioned and was the specific focus of the evaluation. Despite the current use of Standards for Success, we believe the 1 1 9

recommendations below wil offer significant insights that can be applied to future data collection efforts, both for ROSS-SC and other service coordination efforts. In addition, these lessons will be of value to researchers designing future evaluations of ROSS-SC and similar programs.

Aligning Applications and Reporting

SERVICES LISTED IN ROSS-SC APPLICATIONS SHOULD MATCH THOSE THAT GRANTEES

REPORT ON AFTER AWARD

As discussed in chapter 3, the application forms for FY 2014 grantees requested that applicants list specifically needed services. Many of the areas of need listed frequently by applicants, however, were not included on the FY 2014 Logic Model, meaning that grantees who used ROSS-SC resources to connect residents to these services could not report on their successes or failures in these areas.

Better alignment between resident needs listed on the application and the service and activities reported via the Logic Model (or other future data system) would ensure that submitted applications reflect program goals and desired outcomes; it would also improve future evaluations’ abilities to compare what applicants intended to do with what they actually report doing after grants are awarded to them. Enabling the grant application, HUD Form 52769 (appendix G) to be completed and submitted online, as wel as the Logic Model annual data, could assist future evaluations and monitoring efforts to easily and efficiently match grantees’ anticipated versus actual activities. 26

Improving Reporting Instructions

HELP GRANTEES MAKE BETTER PROJECTIONS

The 2014 Logic Model required grantees to report projected numbers (at the time of application) as wel as actual service and activity numbers achieved during grant implementation for mandatory and optional metrics. Although HUD did not specifical y identify projections as performance monitoring tools, we heard concerns from service coordinators that not meeting their projections would reflect poorly on their organizations. This could result in grantees inputting more conservative estimates of projected services and activities to ensure that they are easy to achieve or exceed. This in turn may mask the true need for a particular service in their resident population.

26 Please refer to the HUD Preface for ROSS-SC NOFA updates.

1 2 0

Service coordinators reported that their methods for determining projected needs were frequently informal, either simply carrying the prior year’s numbers forward or making a guess based on resident interactions and their own experience. New service coordinators and grant managers who do not have experience in the program or with the resident population’s needs may have difficulty making accurate projections.

HUD could improve this process by providing clearer guidelines on best practices for making projections, providing grantees with a clearer planning process. Improving instructions for developing projections would also promote more standardized and comparable measures across grantees, improving tools for future monitoring and evaluation. 27

HELP GRANTEES UNDERSTAND HOW PROJECTIONS ARE USED

Transparency around how HUD reads and assesses projections, as wel as describing how these measures wil or wil not be used in assessing performance, would provide grantees with a better understanding of the weight that is put on these projections, as wel as a clearer benchmark to assess their own work. In addition, clear standards for passing or failing performance standards related to projections would help grantees to orient their services and efforts to ensure that they are meeting those standards. 28

IMPROVE GUIDANCE ON COUNTING AND RECORDING INTERACTIONS WITH RESIDENTS

In our study, we noted some confusion from service coordinators and grant managers around exactly what units are to be counted for many Logic Model measures, specifically how formal, extensive, or official interactions had to be in order to be counted. In addition, grantees could use greater clarity on how to record and whether to distinguish between one-on-one interactions via a case management approach versus a service connection approach involving larger interactions that reach many people, such as group workshops, community meetings, or community-wide events such as job or health fairs.

27 As of the 2017 ROSS NOFA, projections are no longer required as part of the application process:

https://www.hud.gov/sites/dfiles/SPM/documents/ROSS_FR-6100-N-05.pdf

28 As of the 2017 ROSS NOFA, projections are no longer required as part of the application process:

https://www.hud.gov/sites/dfiles/SPM/documents/ROSS_FR-6100-N-05.pdf.

1 2 1

Some reporting metrics may need to be modified to differentiate reporting between these two pathways to achieving resident outcomes. 29

MANDATORY VERSUS OPTIONAL METRICS REQUIRE STRONGER JUSTIFICATION

AND EXPLANATION

The FY 2014 Logic Model created a more complete data col ection system than had previously existed for ROSS-SC, introducing mandatory metrics as wel as opportunities for grantees to voluntarily report additional data points in the form of optional metrics. Some service coordinators and grant managers we spoke to expressed confusion over why some metrics in the FY 2014 Logic Model were mandatory and others were optional. Some believed that services included as mandatory metrics were services that al grantees were required to provide, regardless of whether or not they were listed on their ROSS-SC application, whereas others did not. More concrete information about how HUD uses mandatory versus optional metrics, along with specific instructions about how to report on mandatory metrics and how to select which optional metrics to include, including examples, could improve reporting efficiency and accuracy. 30

GRANTEES NEED INCREASED CAPACITY FOR STANDARDIZED REPORTING

In addition, some service coordinators reported recording interactions based on memory, whereas others keep notes and logs throughout the day. Grantees reported using data recording practices that range from sophisticated case management software to unwieldy spreadsheets saved on shared drives or paper records. Some service coordinators reported inputting their data on a daily basis, whereas others centralized their notes once per week. These variations in reporting practices make comparisons across grantees difficult.

There are several program enhancements that could support more effective and equitable tracking systems, allowing all grantees to maintain similar qualities of data:



Create a standard for reporting that reflects the priorities of the ROSS-SC program office.

29 As of data col ection in 2017 and 2018 conducted for this evaluation, the newly released (2019) HUD Exchange guidebook with webinars and additional program information was not available:

https://www.hudexchange.info/programs/ross/guide/introduction/ .

30 As of data col ection in 2017 and 2018 conducted for this evaluation, a new HUD Exchange guidebook (2019) with webinars and additional program information and resources was not yet available:

https://www.hudexchange.info/programs/ross/guide/tracking-activities-measuring-performance-and-outcomes-

and-preparing-for-hud-monitoring-visits-and-audits/why-benchmarks-and-outcomes-matter/tracking-outcomes-

in-ross/.

1 2 2



Provide training to service coordinators and grant managers in meeting these standards in order to generate data that are comparable across sites.



Peer learning on best practices could improve the capacity for all grantees, given that some sites do have high-quality and efficient data col ection practices.



Distribute a list of available case management systems to aid in standardizing tracking systems by resident.



Provide additional funds for lower-resourced grantees to purchase and maintain hardware and software.



Offer technical assistance.

Besides al owing for more reliable cross-site analysis, standards for reporting could allow program officers to identify grantees with special needs, such as high numbers of referrals to addiction treatment. It can also highlight grantees that have been particularly successful in achieving desired outcomes, to further investigate the potential for best practices or program innovations.

We note that HUD released a Data Guide31 which addresses some of the chal enges related to col ecting accurate data, including implementation of the newer Standards of Success reporting system. These include more detail on which interactions to count and how to count them, some information on how metrics relate to outcomes, and examples to accompany these descriptions. The guidebook also includes recommendations for case management systems.

Expanding Data and Reporting To Tell the Full Story

Allowing more flexibility for grantees to report on activities differently, based on whether they are engaging residents using a case management approach, a service connection approach, or both, could better capture what service coordinators do on the ground; such an approach could produce a strong, ongoing narrative about the impacts ROSS-SC is having on assisted residents. Changes that could help with creating a model that more accurately reflects grantee success and challenges, including al owing service projections to be updated to reflect changing resident needs or environment, providing 31 U.S. Department of Urban Development. “Running ROSS Step-by-Step.”

https://www.hudexchange.info/programs/ross/guide/introduction/.

1 2 3

optional metrics that better align with grantee activities, adding some new standardized metrics for both case management and service connection approaches, and including qualitative input.

UPDATING PROJECTIONS TO REFLECT SHIFTING NEEDS COULD HELP

Clear guidance on how grantees may update their projections during the grant period could ensure that real changes in priorities based on changing needs, opportunities, and resources throughout the grant period are accurately captured. 32 Rather than making this invisible in reporting, HUD could incorporate opportunities for grantees to shift their forecasts to take advantage of new opportunities and address challenges that emerge during the grant period. By retaining both the new projections in addition to the original projections, HUD could create a richer dataset that would show how grantees adjust to changing conditions and more accurately project their residents’ needs during the course of a reporting period. 33

REFORMING OPTIONAL METRICS TO BETTER ALIGN WITH LOCAL NEEDS COULD IMPROVE

GRANTEES’ ABILITY TO REPRESENT THEIR WORK

In addition to better aligning the resident needs listed in the grant application with reporting metrics during the grant period and clarifying the role of mandatory versus optional reporting metrics, metrics could be better aligned to capture grantee work with residents on the ground. On site visits, grantees frequently reported that the optional metrics included stil did not fully capture the extent of their work. Al owing grantees to more fully customize their reporting through improved optional metrics could better capture grantee program activities and outcomes and provide more data on what ROSS-SC is helping residents achieve. Some examples (discussed in more detail below) include interactions with property management, associated resident outcomes, and a variety of emergency services that service coordinators responded via survey that they are also providing, such as transportation and food services.

SHARING BEST PRACTICES ON PARTNER DATA-SHARING AGREEMENTS COULD IMPROVE

OUTCOMES TRACKING UNDER THE CASE MANAGEMENT APPROACH

Promoting agreements with service partners to report on ROSS-SC participant outputs (for example, attendance at training) and outcomes (for example, got a job as a result of the training received) could help grantees report more precisely on program outputs and outcomes, and tie them more closely to partner interventions. This data sharing would require a higher bar for partners but could be mutually 32 As of the 2017 ROSS NOFA, projections are no longer required as part of the application process:

https://www.hud.gov/sites/dfiles/SPM/documents/ROSS_FR-6100-N-05.pdf.

33 Please refer to the HUD Preface for data use updates.

1 2 4

beneficial and it would also benefit residents through more integrated tracking of the services they access. Best practices could be shared with grantees through written materials, webinars, and other peer-learning opportunities, and could include examples of successful data-sharing agreements and types of data commonly shared through such agreements, as wel as sample data-sharing MOUs.

HOUSING STABILITY MEASURES COULD BE ADDED AS AN IMPORTANT PROGRAM ACTIVITY

AND OUTCOME

To promote housing stability and self-sufficiency, many service coordinators work directly with public housing property managers to identify and address issues such as unit maintenance needs, late or nonpayment of rent, or other compliance issues. As noted in chapter 6, residents we spoke to reported having high levels of trust in service coordinators and noted that they reach out to them for help when they have emergency needs. Where residents frequently view property managers as front-line enforcers of compliance, they often perceive service coordinators as their advocates. This provides the opportunity for service coordinators to work with property managers to find solutions to compliance issues and potential y avoiding evictions, fines, or other repercussions that could be harmful to residents’ housing stability and costly for the PHAs. On site visits, we heard examples of service coordinators helping with crisis management and connections to emergency rent payment assistance, as wel as facilitating resident relations with property management around maintenance issues.

Although the FY 2014 Logic Model focused on services geared toward improving resident self-sufficiency, it did not capture work that service coordinators do on housing stability, a prerequisite for self-sufficiency, and an important aspect of service coordinators’ work. Potential metrics to track could include whether and how frequently service coordinators meet with property managers, whether they work on maintenance issues, and whether they work together to help residents avoid fees and eviction. 34 In addition, higher-level indictors, such as rent payment history across residents receiving services, could identify both baseline resident needs for such services and the effectiveness of coordinated interventions over time.

HUD would need to consider the level of burden increased reporting would place on service coordinators, property managers, and grantees. If more metrics are adopted, HUD would also need to determine whether reporting on housing-related ROSS-SC services and activities should be mandatory or optional, based on program goals, although these interactions appeared widespread across our site visits. HUD should also assess whether such metrics are equally applicable to both case management and service connection approaches to achieving resident outcomes. Site visit data suggest that service 34Please refer to the HUD Preface for updated information.

1 2 5

coordinators are persistent advocates for residents with PHA property managers and may be able to report on these outcomes more readily than those that depend on external partners.

OUTCOME REPORTING FOR ELDERLY/DISABLED SUCCESS IN INDEPENDENT LIVING COULD BE

IMPROVED FOR A CASE MANAGEMENT APPROACH

For grantees serving elderly/disabled residents using a case management approach, metrics could be improved to report on specific case management outcomes. For the FY 2014 Logic Model, there was no indication of what the “outcome” listed as “age in place successfully” actually measured. Sample measures could be taken from other HUD-funded programs, such as the Vermont Support and Services at Home (SASH) program, which successfully tracked and documented outcomes related to the presence of an onsite Wel ness Nurse and Care Coordinator (Kandilov et al., 2017). Measurable outcomes included reductions in the number of falls residents experienced, emergency room visits and hospitalizations, and the length of hospital stays and delayed transitions to long-term institutional care (Kandilov et al., 2017). These are being further piloted through HUD’s Supportive Services Demonstration and Evaluation, at the conclusion of which, further learnings may be available ( Federal Register 83, no. 32). Resident intake and tracking models from these programs could be explicitly adopted for ROSS-SC, or may require modification to better align with service coordinator roles and variation in available community partners (for example, some more remote, rural communities may not have local hospitals or emergency departments with which to partner on data).

MEASURING BARRIERS FACED IN DELIVERING ROSS-SC SERVICES AND ACTIVITIES COULD

CLARIFY GRANTEE PERFORMANCE ISSUES

The FY 2014 Logic Model also does not capture barriers to resident success external to the ROSS-SC

program. Al owing grantees to report about whether they have faced particular chal enges during the reporting period can put the services, activities, and outcomes they report on into clearer context, particularly if these metrics are indeed being used by HUD to measure grantee performance. These could include a series of metrics counting challenges, such as number of months with a vacant service coordinator position(s), numbers of partner deficiencies (including lack or loss of a local partner needed to meet resident needs), number and type of onsite services available, and presence of public transportation options. These could help to distinguish between low performing grantees and those that are in particularly challenging contexts. Taken in aggregate, these measures could also allow HUD

to target extra resources and supports to grantees with higher needs and/or a where strategic investment of limited resources could have the greatest impact.

1 2 6

GRANTEES WOULD LIKE TO INCLUDE QUALITATIVE STORIES IN ADDITION TO NUMERIC DATA The most common piece of feedback we heard from service coordinators and grant managers when asked about data reporting is that they would like to have an opportunity to provide qualitative, narrative data. Given the diversity of contexts in which ROSS-SC programs operate, and the flexibility that grantees have to tailor the program to meet local needs and leverage local resources, even an expanded quantitative data reporting tool could not capture all the ways residents benefit from the program. Although reviewing these data would require more effort from HUD program staff, it would provide a deeper connection between HUD administrators and grantees, while offering grantees the opportunity to fully report on their activities more comprehensively. This narrative input could be reviewed at critical program junctures, such as while making general program decisions, as wel as when working one-on-one with a grantee to troubleshoot performance issues. It could also help future evaluations by explaining trends and shifts in grantee data, informing selection criteria for future site visits, and identifying fruitful research questions and data col ection tools to highlight future ROSS-SC

achievements. 35

STANDARDS FOR SUCCESS MAY IMPROVE DATA RELIABILITY AND AGGREGATION BUT MAY

REQUIRE FUTURE IMPROVEMENTS TO REFLECT THE TWO THEORIES OF CHANGE

Standards for Success, implemented by HUD for the ROSS-SC program beginning in FY 2017, is a reporting system tied to case management at the individual level. It al ows grantees to choose from four different case management reporting tools and from over 90 data elements. These data elements capture activities and services, and of the 98, 18 capture outcomes. The data elements cover employment, financial information, education, health, substance abuse, and housing categories. These reflect the many needs residents have, the areas of need grantees choose to focus on, and the types of services provided by ROSS-SC partners. Indicators are rol ed up to capture such things as average quarterly earnings or participation rate in GED courses for al residents participating in the ROSS-SC

program. They are meant to address overall program impact by showing trends such as whether economic independence and self-sufficiency improve, whether the ability to live independently improves, whether participation in ROSS-SC improves other quality-of-life outcomes, and whether ROSS-SC housing services help participants meet their housing needs (HUD n.d.d, HUD n.d.h). A detailed review of Standards for Success is outside the scope of this evaluation. However, several of the model’s features, such as the incorporation of case management software and a specific tie between activities and outcomes reflect improvements suggested in this chapter.

35 Please refer to the HUD Preface for updated reporting system information.

1 2 7

ADOPTED STANDARDS SHOULD CONSIDER CONTRAINTS FACING

LOWER-RESOURCED GRANTEES

Resource differences across grantees affect the approach they take to implementing ROSS, whether they share responsibilities with other service coordinators or PHA staff, and how they track data. On site visits, we learned how better-resourced PHAs may have other case management programs, funded by a grantee’s budget, other local funding streams, or through other federal programs. Service coordinators at lower-resourced PHAs often have to take on both direct case management and service provision tasks as wel as coordinating partners and services. This blended role can create increased demands on service coordinators’ time, potential y requiring them to balance the needs of their individual residents against the high levels of communication and coordination that partner cultivation and maintenance requires.

HUD program managers and future evaluators could benefit from increased awareness of these differing contexts in which service coordinators work when designing performance reporting systems, and when interpreting program performance data. Service coordinators at higher-resourced grantees, for example, may be able to report serving a higher number of residents than those at low-resourced grantees because the residents require a lighter touch, and are directed elsewhere for higher intensity case management. Service coordinators at lower-resourced grantees, on the other hand, may report lower engagement with partners and partner recruitment, due to the fact that fewer partner organizations exist in the community, which results in spending more of their time working directly with residents.

Improving Peer Learning and Communications

Train New Service Coordinators During Onboarding

Service coordinators reported in interviews and the service coordinator survey that they would like expanded opportunities for trainings, opportunities to learn from their peers working at other PHAs, and general training and technical assistance. Onboarding both when a new service coordinator is hired and when a new grant cycle begins provides a natural opportunity for expanded and updated training, including best practices on building partnerships, providing service coordination, tracking resident participation and outcomes, and reporting properly on services and activities. The launch of 1 2 8

the new ROSS-SC resource website in October 2018 provides significant new resources in this area (HUD n.d.f), including an onboarding webinar each year for both renewal and new grantees. 36

FOSTERING PEER LEARNING OPPORTUNITIES ACROSS GRANTEES COULD

IMPROVE PARTNERSHIPS

Formal communication across sites has the potential to improve grantee performance across the board. In addition to the benefits of knowledge sharing, grantees could have the opportunity to improve partnerships. By sharing information about partners, grantees who are geographically proximate could identify strong regional partners, and could, for example, leverage combined funds to provide a service to residents that neither could access individually. Moreover, grantees who have partnered with national or regional chains, such as grocery stores, health, vision or dental services, or educational services, would be able to share these resources with other grantees who have not yet tapped those resources.

Improving Communication Between HUD and Service Coordinators

STRENGTHENING CONNECTION BETWEEN HUD AND SERVICE COORDINATORS COULD

IMPROVE COMMUNICATION AND DATA COLLECTION

Although HUD runs a voluntary listserv that service coordinators may choose to join to connect to training opportunities and program updates and ask questions, service coordinators interviewed were unaware of this resource and felt disconnected from the national program. HUD’s formal relationship is with the grantees. Because the grantees hire the service coordinators, it can be hard to know whether service coordinators themselves are receiving information from HUD pertaining to training and other communications. In sending out communications to grantees but not directly to service coordinators, HUD has no assurance about whether service coordinators actually receive important information. Grant managers may not pass communication on to the service coordinators, either due to focusing on other aspects of their job or because they do not understand the importance of the message. Likewise, service coordinators may have no means to respond with requests for clarification, other questions, or feedback.

While grantees may email the HUD program staff with grant implementation questions, a uniform, direct, two-way line of communication between HUD and service coordinators would enable more 36 The ROSS-SC resource website is available here: https://www.hudexchange.info/programs/ross/.

1 2 9

efficient collection of evaluation data in the future, including accurate counts of active service coordinators in real time and updated contact information. This would allow grantees to be evaluated based on the number of service coordinators available to the grantee and actually performing the work. Performance measures for grantees that are not staffed at full capacity due to turnover or other issues would then be adjusted, allowing for a more realistic measure. Contact information would also allow future evaluation surveys, similar to the service coordinator survey conducted through this evaluation, to be administered directly to service coordinators more quickly and cost-effectively. Such enhanced communication and tracking, however, would require that HUD mobilize additional resources and staff to devote to this task.

REGULAR FEEDBACK FROM SERVICE COORDINATORS COULD HELP HUD IMPROVE ROSS-SC

AND IDENTIFY TRAINING AND TECHNICAL ASSISTANCE GAPS

A feedback mechanism for service coordinators and/or grantees to share program insights could help HUD make programmatic course corrections, if necessary. It could also point to training and technical assistance gaps that HUD could help fil . For example, lower-resourced grantees may need help connecting to existing training opportunities and may need additional resources to access what is available (for example, transportation costs may be higher for more remote grantees). HUD could also use feedback to match lower-resourced grantees with better-resourced grantees for virtual or in-person peer learning engagements. Updates to HUD trainings and materials could also take feedback into account to ensure service coordinators have access to common information requested.

Although it may require a greater investment from the HUD program office, a feedback mechanism holds the potential for improved performance and efficiencies as grantees could receive needed resources more quickly. Feedback col ection could take the form of more frequent surveying, incorporating a feedback module into annual reporting, or more frequent program evaluations.

Filling Service Gaps and Facilitating Partnerships

Partnerships are the core of the ROSS-SC program. Partners are the primary sources of services and resources for residents at most ROSS-SC sites, and their availability, quality of service, and accessibility can radically affect a grantee’s ability to achieve desired outcomes for residents.

Unfortunately, however, some needed partners may not historical y exist in a given community, or may close, leaving significant service gaps and jeopardizing the ability of ROSS-SC service coordinators to help meet resident needs. Other partners, including businesses, may not be listed on a ROSS-SC

1 3 0

grant application or be considered to be a service provider in a traditional sense, but could nevertheless provide valuable resources for service coordinators to leverage.

Addressing Missing Services

MORE RESOURCES FOR ROSS-SC GRANTEES TO PROVIDE DIRECT SERVICES COULD HELP

ADDRESS LACK OF COMMUNITY PARTNERS AND SERVICES

Some communities have significant service gaps historically, with fewer resources available to meet resident needs. Prior to its transformation into ROSS-SC in 2008, the original ROSS program and its precursors provided service funds. Given how critical the availability of community services is to the success of ROSS-SC and advancing residents toward meeting programs goals, HUD could reexamine the need for service funding, particularly in lower-resourced communities with few service partners. In addition to filling service gaps in the community, increased funding for direct service provision and more enhanced case management could enable more lower-resourced grantees to take a case management approach to ROSS-SC: as the service provider, grantees could directly track resident access to services, as wel as outcomes. Implementation would require a number of changes in the program, however, including adding authority to provide services and appropriating additional resources to pay grantees to provide such services. It would also necessitate establishing different methods for tracking program activities, costs, and outcomes for services provided directly versus services coordinated through referrals to other community partners.

Service coordinators at several grantees we visited reported having to end programs or seek out new services due to a partner closing or changing their programming. As ROSS-SC depends on leveraging partner services, the loss of a key partner has the potential to result in a significant decline in resources available to residents. It also places an additional burden on the service coordinator who must seek out a new partner for a previously available service. In addition to affecting the quality of service available to residents, partner stability has the potential to affect grantee program performance and the performance of the ROSS-SC program overall. As partner closures are more likely in areas with less funding available for social services, whether public or through private foundations or other funding streams, it is likely that some communities served by lower-resourced grantees are more affected by this than others. By tracking this more carefully, HUD could identify which areas may need greater supports to provide a base level of services.

1 3 1

Exploring Regional or National Partnerships

NATIONAL OR REGIONAL CONSORTIA COULD IMPROVE EFFICIENT ACCESS TO PRIVATE-SECTOR PARTNERS ACROSS GRANTEES

Given that a primary goal of the ROSS-SC program is to leverage existing services and resources, establishing some mechanism for col aboration on a national or regional level could improve service coordinators’ abilities to match residents to services on the ground. Although service coordinators at a single grantee frequently share resources and partner leads with other service coordinators at the same grantee, each grantee typical y is responsible for identifying local partners and establishing relationships. When asked to list current partners, however, several service coordinators we interviewed mentioned local branches of national or regional business chains, such as grocery stores and optometrists. These established local relationships create an opportunity for the ROSS-SC

program to establish national partnerships with these national chains, which could provide service coordinators across the country with a base set of resources, allowing them to focus on establishing or expanding on other local partnerships that they may not otherwise be able to focus on. This kind of collaboration would be of particular value in areas with sparse social services or a weak network of nonprofits. Although national or regional partnerships are possible in a wide range of service areas, supermarkets and pharmacies have particular promise, as they have large and efficient distribution networks, and frequently have established community engagement and assistance programs to encourage access to food, medicine, and basic medical services.

Meeting Resident Needs

Assessing Resident Needs

PARTNERING WITH HUD TO ACCESS AVAILABLE RESIDENT DATA COULD HELP IDENTIFY

RESIDENT NEEDS

HUD could provide grantees with resident data that public housing agencies already col ect to assess resident needs and develop their ROSS-SC activities and services. Whereas the survey conducted for this evaluation asked service coordinators to make an informed guess about resident characteristics in their developments, HUD has these data already submitted through HUD Form 50058 annually on households living in public housing. These data could be shared with service coordinators to identify key populations they could serve through ROSS-SC. Data of interest could include numbers of adults and their ages, disability status, presence and age of children, and wages and sources of income. If service coordinators then col ected similar resident characteristic data on ROSS-SC participants, they 1 3 2

could compare the characteristics of the residents they are serving against the characteristics of all potential participants to assess whether they are missing important segments of the population in their current services and activities. This could help service coordinators plan strategical y on how to engage missing populations through future programming.

Exploring Other Promising Models

USING LESSONS LEARNED FROM PROMISING MODELS OF RESIDENT SERVICES COULD

ENHANCE ADDRESSING RESIDENT NEEDS

Although we have posited two different theories of change guiding ROSS-SC implementation on the ground, there are other program models targeting improved outcomes for public housing residents from which future ROSS-SC programming could draw lessons, including coaching and two-generation models. HUD could consider making information on these other approaches available to grantees and service coordinators for consideration. At a program level, HUD could decide if any elements of these models could be adopted for broader implementation.

Whereas ROSS-SC takes a needs-based approach and focuses on a pre-defined set of desired outcomes, coaching programs allow participants to set individual goals and tailor services to build on their particular strengths. Coaching also identifies pathways that empower individuals to reach their own goals, building self-confidence and autonomy while enabling access to necessary resources and information. Early evidence indicates that these programs may lead to greater levels of commitment by participants to sticking with plans of action as wel as lasting economic and employment outcomes (Burnstein, Gal agher, and Oliver 2019; Theodos, Stacy, and Daniels 2018). 37

Other programs seek to address families as a unit, identifying interventions that help parents and their children simultaneously. Urban Institute’s work on HOST (Housing Opportunities and Services Together) offers lessons about how to use public and assisted housing as a platform for delivering services using this two-generation, whole family approach (Popkin, Falkenburger, and Haight 2018).

Lessons from HOST include using a tiered model that targets the most intensive services to those residents with the greatest needs; taking the time to build trust and engage residents; forming partnerships with strong local providers; and using a trauma-informed, community-engaged approach (Scott et. al. 2016a). The HOST approach has evolved over time and involves conducting initial assessments and then targeting the residents with the greatest need with the most intensive services, while providing those with less need with referrals and emergency supports. Coaches and case 37 Please see HUD Preface for updated information.

1 3 3

managers worked directly with family members using strength-oriented approaches to identify specific strategies for moving to meet individualized goals (Popkin et al. 2012). The HOST site in Bangor, ME, is currently refining the tiered two-generation service model, and the approach is showing promise in both helping adults move to self-sufficiency and children improve school attendance.

Some ROSS-SC service coordinators are using elements of this approach. Having other service coordinators incorporate these best practices could help service coordinators ensure their services match the needs and preferences of their residents—and help ensure that they are using their service dol ars effectively.

Addressing Barriers to Participation

TO ACHIEVE HIGHER GOALS, ROSS-SC IMPLEMENTATION NEEDS TO ADDRESS RESIDENTS’

BARRIERS TO PARTICIPATION

Our conversations with service coordinators and residents uncovered barriers to participation for some residents that they may need help addressing through stronger partnerships and additional resources. Being able to adequately address transportation, childcare, and language needs of residents would enable a wider array of individuals to access ROSS-SC activities and local services who may otherwise have limited or no access, as wel as potential y building increased engagement from residents and trust in service coordinators. To fil these gaps may require forging new partnerships with public, private, or nonprofit partners, reallocating ROSS-SC grantee resources to provide some of these al owable services, or tweak program operations to accommodate these resident needs, as much as possible. HUD may provide suggestions and encourage grantees to implement best practices within existing program regulations, including the following:



Providing supportive transportation to secure employment, such as to a job interview.



Allowing the presence of children at ROSS-SC activities in some circumstances to help parents participate in programs or arranging for childcare services on site.



Encouraging the expansion of the hours that service coordinators are available for residents who are not home during the day.



Hiring multilingual service coordinators or pay for translator services for key interactions and events to allow ROSS-SC to reach residents who do not speak English wel .

1 3 4

Sustaining ROSS-SC

There are several targeted improvements and reforms that can strengthen the future of the ROSS-SC

program and enhance its ability to meet resident needs, support service coordinators in their work, and boost the capacity of lower-resourced grantees. Some proposed changes outlined below may require congressional support and specific legislative action, whereas others may fal within the bounds of existing regulations.

Serving Residents

ENSURE THAT CURRENT RESIDENTS CONTINUE TO RECEIVE SERVICES

Several grantees we interviewed discussed concerns around funding chal enges facing ROSS-SC. This includes mentions of continued proposed federal cuts to the program, as wel as concerns about losing ROSS-SC funding as some properties were scheduled to convert under the Rental Assistance Demonstration (RAD) program. At the time sites were selected for our visits, grantees were al owed to complete their current ROSS-SC contract if al units converted through RAD, but they were not eligible for renewal to serve those converted units in the future. There were therefore concerns about residents losing access to ROSS-SC service coordinators.

With the expansion of the RAD and Moving to Work Demonstration programs, it is vital to ensure that residents who currently benefit from ROSS-SC can continue to receive the same or similar services. This may require changing program guidelines to allow developments formerly operated as public housing to remain eligible for ROSS-SC grants, expanding the program to serve voucher holders in addition to public housing residents, or working with affected PHAs and ROSS-SC grantees to identify replacement funding to continue to provide service coordinators after unit conversion.

Supporting Service Coordinators

EMPOWER SERVICE COORDINATORS THROUGH A STRENGTHENED RELATIONSHIP WITH HUD

AND AMONG EACH OTHER

The service coordinators and grant managers we spoke to during the April 2017 through October 2018 data col ection period regularly noted the need for greater clarity on their roles, including what tools were available to them to engage residents and recruit and maintain partnerships, as wel as how to col ect and report data. More extensive and regular communications with HUD would provide the opportunity to share resources and provide clarity on an as-needed basis. We note that the recently released program manual is a step in the right direction here, although it was not available to ROSS-SC

1 3 5

sites during our period of data col ection. 38 In addition, a national newsletter and regular webinars highlighting promising practices and challenges could improve a feeling of national engagement and mutual learning. Final y, ensuring that ROSS-SC grantees know they can use their administrative and training budgets to access peer learning organizations such as the American Association of Service Coordinators39—a membership organization for service coordinators and affordable housing professionals that offers trainings and conferences—could provide a strong opportunity for grantees to learn from the field and to have a point of contact with each other.

Improving Program Resources

PROVIDE LOWER-RESOURCED GRANTEES WITH IMPROVED RESOURCES

As noted throughout this chapter, not all grantees have the same levels of overall resources, both organizationally as wel as in their surrounding community. On our site visits, we noted that those service coordinators who were able to do the most with ROSS-SC resources functioned within an ecosystem of local partners that were already wel -funded and had strong inter-organizational networks that included nonprofit, governmental, and private actors. In addition to having diverse and high-quality options for local partners, service coordinators working in these contexts had strong educational backgrounds and work experience that enabled them to take full advantage of these systems. They also often had easier and closer access to high-quality training resources. Sometimes the PHAs themselves had other resources available to operate onsite case managers separate from those funded through ROSS-SC or other federal grants. These high-resourced grantees we visited were frequently, although not always, in large urban areas.

In contrast, the lower-resourced grantees we visited both depended more on the ROSS-SC

program for connecting their residents to services and were able to do less due to an absence of other supports within the PHA and sparse local service landscapes. These grantees may be working in communities that do not have a strong philanthropic presence, and despite highly dedicated and competent people working in social services, do not have the economic, social, or physical infrastructure than can provide core resources. For example, at some grantees we visited, residents do not have the transportation services they need to access services. At others, service coordinators do not have access to the training and professional resources needed to leverage their positions. These 38 Please see HUD Preface for updated ROSS-SC grantee and HUD communication information.

39American Association of Service Coordinators: https://www.servicecoordinator.org/.

1 3 6

grantees are frequently, although not always, in rural areas, and would benefit from additional resources to increase their capacity to meet resident needs.

Conclusion

This evaluation highlights how ROSS-SC service coordinators provide a valuable connection between public housing residents and surrounding community partners to help address resident needs and improve self-sufficiency and independent living. Grantees, service coordinators, community partners, and residents we spoke with agreed that the resources ROSS-SC provides are crucial in making these connections and advocating for residents and meeting their needs. From helping residents confront emergencies, to connecting them to key service providers, to providing individual, ongoing case management to support achieving long-term goals, service coordinators are a critical resource to public housing residents. They seek out needed partners, build and maintain a variety of relationships within the PHA and across the community, and track the progress that results from resident engagement with these resources.

To sustain ROSS-SC into the future, HUD should consider how to address grantees’ funding concerns, help strengthen relationships among service coordinators through peer-to-peer training, continue to strengthen the connection between service coordinators and HUD, and consider tailoring the ROSS-SC program to address the specific needs of lower-resourced grantees

1 3 7

Appendix A. Sample Individual

Training and Services Plan (ITSP)40

40 U.S. Department of Housing and Urban Development. “Running ROSS Step by Step: Sample Individual Training and Services Plan.” https://files.hudexchange.info/resources/documents/ROSS-Step-by-Step-Individual-Training-

and-Services-Plan.pdf.

1 3 8

Appendix B. Interview Protocols

Informed Consent

[DISCUSS THE PURPOSE OF OUR PROJECT/CONTRACT BASED ON THE DESCRIPTION IN THE

CONSENT FORM. THEN READ THE FOLLOWING OUT LOUD TO THE INTERVIEW PARTICIPANTS

BEFORE STARTING.]

Before we begin, I want you to know that the information you share in this interview wil be kept private to the extent permitted by law. We will not include your name in our notes or reporting, and our notes and recordings wil not be shared outside of our Urban Institute research team. When we report our findings, we wil combine information from everyone we talk to and present it in a way that individual answers cannot be easily identified. Every effort wil be made by the research team to preserve your privacy by not using your name or any other identifying information that can be linked to a specific comment in our report.

We want to be sure that you freely consent to participate in this interview and that you are aware that you are not obligated to answer any questions you do not wish to. Do you consent to participate in the interview?

(IF THE CONTRACT MONITOR IS IN THE ROOM, HAVE HIM OR HER READ THIS)

My name is _________, and I am with the Office of Policy Development and Research at the U.S. Department of Housing and Urban Development national office, the agency that funds [GRANTEE NAME] to provide the Resident Opportunity and Self-Sufficiency Service Coordinator program and that is funding this program evaluation. If you agree, I would like your permission to observe how the Urban Institute conducts this interview to ensure the quality of their research process. I would like you to know that I am only here to monitor the research, and perform no role in the collection or analysis of the data you provide to them. I will not use any of your personal information or discuss any of the experiences you describe during this interview for any other purpose. Your responses wil remain private to the extent permitted by law and wil not be reported to anyone. Do you consent to my staying in the room to observe this interview?

(IF THERE IS MORE THAN ONE PARTICIPANT IN THE INTERVIEW):

However, because there is more than one participant in this interview we cannot ensure that what is shared during this conversation not be shared with outside stakeholders. We encourage participants not to share what other respondents say in respect of their privacy.

1 3 9

[IF THE INTERVIEWEE RESPONDS YES, NOTE THE TIME. IF NO, THANK THEM FOR THEIR TIME AND

ALLOW THEM TO LEAVE.]

[THE INTERVIEWER SHOULD CONTINUE BY READING THE FOLLOWING]

With your permission, we would also like to record our interview for note-taking purposes. Only our research team wil have access to the recording and we wil erase all recordings after our research is finished. Do you consent to have the interview recorded?

[IF YES, BEGIN RECORDING]

Before we begin, do you have any questions for us about the interview?

1 4 0

Service Coordinator

First, we would like to ask you about yourself and your professional experience. Then we would like to ask you about the ROSS program and your role as a ROSS service coordinator.

1. How did you come to this job?

2. How long have you been here?

3. How do you like it?

4. What did you do before this job?

5. What training/certification/education/experience did you have prior to starting this job?

6. What training have you received while you were working in this job?

Now, we would like to ask you about the ROSS program and your role as a ROSS service coordinator.

7. How would you describe the ROSS program in your own words?

8. What does your typical day look like from beginning to end?

9. How much time did you spend in the last month:

a. With residents

b. With work col eagues [IF NEEDED, CLARIFY BETWEEN SAME PROGRAM

STAFF/DIFFERENT PROGRAM/SUPERVISOR]

c. With service partners

d. On administration [IF NEEDED, CLARIFY PAPERWORK, REPORTING & DATA

ENTRY, ETC.]

i. How much of this is for [INSERT GRANTEE NAME] vs. HUD?

ii. Do you ever find you have to enter the same data in multiple places?

iii. Can you explain where and in what context?

10. What do you like most about your work?

11. What do you like least about your work?

[FACILITATOR READS]: We would like to hear more about the activities you perform during an average week.

12. What is a considered a full caseload for you in terms of the number of residents you work with?

a. How many residents are you expected to see on a weekly basis?

b. How many of them would you say you work with on a regular basis (at least monthly) or are considered “active”?

13. What do you do to reach out to residents and get to know them?

a. How do you make yourself available to help them?

i.

One-on-one meetings [IF NEEDED, PROBE IF THESE ARE SCHEDULED VS.

WALK-INS]

ii.

Events

iii.

Other

14. What outreach methods do you find to be most effective?

a. Are there any particular events or communications strategies that have been more effective than others to increase resident participation?

15. What kinds of conversations do you have with families?

1 4 1

a. For elderly/people with disabilities? [AS APPROPRIATE]

16. What are some of your residents’ greatest needs?

17. Are any residents reluctant to or unwil ing to participate?

a. Why do you think that is?

18. Do you have residents complete an Individual Training Service Plan (ITSP)—or a plan that defines their final and interim goals, the steps they need to reach those goals, and barriers they may face?

a. At what point do you encourage a resident to complete such a plan?

19. How many people have an active plan right now? Are the ROSS participants similar to or different from participants in other programs or services that you offer?

a. If yes, how? [PROMPT: BY TYPE OF NEEDS, INTERESTS, DEMOGRAPHICS, OTHER

CHARACTERISTICS]

20. Do you refer residents to services/agencies/ resources outside of your agency?

a. If yes, can you provide a few examples?

b. How do you know about these various resources?

c. Are these resources available on or off site?

i. If off site, how do residents access them?

d. Do you meet with or speak to representatives within these agencies/organizations regularly?

e. Do you strategize with them about how to meet your residents’ needs?

i. Give them feedback on your residents’ experiences with their services?

f. Are there referral services that are necessary, but not available or don’t have the capacity to meet all of your residents’ needs?

g. Which resources for residents do you wish you could improve or expand?

21. Which referral services do you feel best enhance residents’ self-sufficiency?

22. How do you track progress and fol ow up with residents?

23. How often do you follow up with each resident?

a. Do you wait for them to reach out to you?

b. Do you schedule regular appointments/check-ins to fol ow up/update Individual Training Service Plans [INTERVIEWER SHOULD REPLACE WITH THE NAME THEY

USE AT THE SITE] with participants?

24. What is the most chal enging aspect of your job?

25. What has been your greatest accomplishment?

26. What would help you most in your work supporting ROSS participants?

a. Training

i.

[IF YES] can you specify what type of training?

ii.

Did you find it helpful?

b. Compensation

c. Activities

d. Services/Partners

e. Coordination with other programs/grantee resources

f. Technical Assistance or help from HUD: Did the help come from

i.

HUD HQ?

ii.

The local Field Office?

iii.

What type of help did they offer?

1 4 2

[IF NEEDED, PROMPT WITH EXAMPLES: DATA COLLECTION; OUTREACH

TO PARTNERS; ENGAGING RESIDENTS]

27. Do you have any other recommendations you have that would make the ROSS program stronger?

[INTERVIEWER SHOULD TAKE THIS TIME TO NOTE IN WRITING ANY RELEVANT INFORMAL

OBSERVATIONS]

1 4 3

Grantee Supervisor/Manager

1. How long have you had a ROSS-SC program running at this location?

2. Do you have multiple ROSS awards serving different public housing sites?

a. [IF SO, SPECIFY THAT YOU WOULD LIKE THEM TO ANSWER FOR ALL ACTIVE

SITES]

[INTERVIEWER READS]

We’ d like to ask you some questions now about the ROSS-SC Program Structure and Processes 3. How many residents would you estimate are served at the ROSS-SC sites currently?

a. In the past year?

4. According to our data, the budget for the ROSS-SC program at this site is $NUMBER. Is this about right?

5. The data also show that you have had about $NUMBER of this amount dispersed. Does this seem about right?

a. Do you expect to use the ful amount prior to the expiration in DATE?

b. Why or why not?

6. How does the ROSS-SC program and staff fit into your overall organizational structure?

a. How is the SC’s work managed? To whom does s/he report?

b. How is the work divided between multiple SCs?

c. Does the ROSS-SC staff work coordinate with other programs you may be running?

i.

What other self-sufficiency or service-related programs do you run?

ii.

Do they take steps to avoid duplication of program activities or participants

served? If so, what kind of steps do they take?

iii.

Do you use the ROSS program to leverage funding or resources from within

your agency? From external funders or agencies?

d. Do you expect to continue applying to be a ROSS site in the future?

i.

If not, why?

I.

[PROBE: RAD, OTHER LOSS OF PUBLIC HOUSING, STRATEGIC

PLAN CHANGES, OTHER]

7. Do you run any other self-sufficiency focused programs?

a. [IF NEEDED, PROMPT]:

i.

Family Self-Sufficiency (FSS) program?

ii.

Are there any interactions or overlaps between the ROSS and FSS or other

self-sufficiency programs?

[INTERVIEWER READS]

Now we’ d like to turn the conversation to talk a bit about the characteristics and activities of your service coordinator/s

8. What are the qualifications (education, experience) for the service coordinator position and the service coordinator manager position?

1 4 4

a. How are they compensated?

9. What type of training and technical assistance do you provide to your service coordinators?

a. Do your service coordinators receive any training from other sources, such as the Office of Field Operations, or the HUD Field Office staff?

10. How frequently do service coordinators turn over?

11. What activities do the service coordinators conduct as part of the services they provide to residents?

12. How do coordinators conduct outreach to residents?

13. Are the ROSS participants similar to or different from participants in other programs or services that you offer?

a. If yes, how? [PROMPT: BY TYPE OF NEEDS, INTERESTS, DEMOGRAPHICS, OTHER

CHARACTERISTICS]

14. What types of partners do you interact with?

a. What types of services do these partners provide?

b. Who maintains the relationships with these various partners or referral agencies?

c. Is someone charged with maintaining regular communication with these

partners/agencies?

d. Does someone strategize with them about how to meet your residents’ needs?

i.

Give them feedback on your residents’ experiences with their services?

e. Who runs the partner organizations?

i.

State/local governments?

ii.

Nonprofits?

iii.

Charities?

f. Which resources do you wish you could improve or expand for residents?

[THESE QUESTIONS MAY EITHER SAVED OR REPEATED FOR THE INTERVIEW WITH THE DATA

MANAGER]

We would like to ask you some questions about the data that you submitted both in your proposed Logic Model as wel as the reporting data submitted to HUD. These questions are only so that HUD can better understand how grantees are interpreting the data fields and to improve their process moving forward. We are not evaluating your performance specifically.

15. How do you, as a ROSS-SC grantee, compile and track their data throughout the year?

a. Do they use a case management system?

16. What documentation do you keep on the program and its participants and for how long?

17. Who set the goals you use in the ROSS-SC Logic Model during the application phase?

a. What role does the SC play in determining the goals on the Logic Model?

b. Who completes the Logic Model during grant implementation?

18. Did the FY 2014 mandatory metrics affect the services you provide, the partners you work with, or your program’s overal design or mission? How?

19. How do you count the number of residents “served” by the service coordinator reflected in the Logic Model?

1 4 5

a. [PROMPT] Do you count individual sessions with residents?

b. The number of individuals served?

c. Number of referrals or programs each individual is involved in? [DISPLAY A COPY OF

THE LOGIC MODEL FOR CLARIFICATION]

20. The Logic Model provides a space for projected and for recorded numbers. How do you determine the values for the projected numbers?

a. When fil ing out the form, are the projected numbers always referring to next year’s goals, or are they goals you set for yourselves for the current reporting period?

21. In looking at the data we received from HUD, we see that there are larger discrepancies between your projected and recorded numbers, such as in the Seniors/Disabled-Service Coordination-Referrals for senior/disability services sections [DISPLAY A COPY OF THE

DATA REPORTED FOR CLARIFICATION]. Do these numbers look right to you?

a. What are some of the reasons for this discrepancy?

b. Why are the projections the same for each senior category, but the recorded numbers vary?

22. How do you understand the difference between services/activities and outcomes?

a. Both categories seem to have a projected and recorded option. Can you explain how you view the difference between these two areas?

b. Why do the numbers vary between these two areas of reporting, for both the

projected and recorded values?

[INTERVIEWER READS]

Finally, we have a few questions we would like to ask related to our Outcome Evaluation 23. How successful do you think your program is?

a. What indicates success to you?

24. What do you think contributes most to program success?

a. Capacity of [grantee]

b. Capabilities of service coordinators

c. Characteristics of program participants

d. Availability of services/partners

e. Additional factors

25. What are the biggest challenges to the program’s success?

a. Costs

b. Staffing

c. Resident participation rates

d. Caseloads

e. Service/Partner availability

f. Reporting metrics–appropriate way to measure outcomes?

g. Etc.

[INTERVIEWER SHOULD TAKE THIS TIME TO NOTE IN WRITING ANY RELEVANT INFORMAL

OBSERVATIONS]

1 4 6

Partner

1. What types of services does your organization provide?

2. What are your organization’s strengths?

a. What do you think contributes most to program success for your clients/participants?

b. What indicates success to you?

3. How long have you been working with [GRANTEE]?

a. Do you know that [GRANTEE] is receiving a ROSS Service Coordinator grant?

b. What do you know about the ROSS-SC program?

4. How long have you been working specifical y with the ROSS Service Coordinator program run by [GRANTEE]?

a. What governs your partnership with them?

i.

A Formal partnership, such as a subcontract, or a Memorandum of

Understanding?

ii.

An Informal relationship or referrals?

iii.

A Local Partnership Council?

5. How do you serve clients referred to you from [GRANTEE]?

a. What kinds of supports do you provide those clients?

b. Do you have regular communication with the service coordinators (include names of SC’s, if known)? General communication or about specific clients?

c. How do you track and report on participant progress?

i.

To whom do you report?

d. How do you keep participants engaged?

i.

What are the major obstacles?

6. What are your biggest challenges to meeting their need for services?

a. Funding

b. Need

c. Coordination

7. What could improve your ability to serve clients?

a. Is there anything that might strengthen the partnership between [GRANTEE] and you/your organization?

[INTERVIEWER SHOULD TAKE THIS TIME TO NOTE IN WRITING ANY RELEVANT INFORMAL

OBSERVATIONS]

1 4 7

Appendix C. Focus Group Protocols

Informed Consent

[READ THE FOLLOWING OUT LOUD TO THE FOCUS GROUP PARTICIPANTS AT THE BEGINNING

OF THE SESSION]

Good morning/afternoon.

Our goal for this conversation is to learn more about your experiences with the ROSS-SC program.

[ESTABLISH HOW RESIDENTS AT THIS SITE REFER TO THE ROSS PROGRAM AND USE THIS

TERMINOLOGY THROUGHOUT THE FOCUS GROUP].

Before we begin, I want you to know that the information you share in this interview wil be kept private to the extent permitted by law. We wil not include your name in our notes or reporting, and any notes and recordings that are shared with HUD wil include no identifying information. When we report our findings, we wil combine information from everyone we talk to and present it in a way that individual answers cannot be easily identified. Every effort wil be made by the research team to preserve your privacy by not using your name or any other identifying information that can be linked to a specific comment in our report. While we ask all focus group participants to keep what is shared today to themselves, we cannot offer confidentiality, as there is always a chance that other focus group participants may share what is said in our discussion today. We ask you to be particularly mindful when it comes to sharing any information back with your ROSS service coordinator. Please share only your own thoughts and not the comments or opinions of others in the room.

We want to be sure that you freely consent to participate in this interview and that you are aware that you are not obligated to answer any questions you do not wish to. Do you consent to participate in the interview? (If yes, note time. If no, thank them for their time and allow them to leave.) (If the contract monitor is in the room, have them read this) My name is _________, and I am with the Office of Policy Development and Research at the U.S. Department of Housing and Urban Development national office, the agency that funds [GRANTEE NAME] to provide the Resident Opportunity and Self-Sufficiency Service Coordinator program and that is funding this program evaluation. If you agree, I would like your permission to observe how the Urban Institute conducts this interview to ensure the quality of their research process. I would like you to know that I am only here to monitor the research and perform no role in the col ection or analysis of the data you provide to them. I wil not use any of your personal information or discuss any of the experiences you describe during this interview for any other purpose. Your responses wil 1 4 8

remain private to the extent permitted by law and wil not be reported to anyone. Do you consent to my staying in the room to observe this focus group?

[THE FOCUS GROUP FACILIATOR SHOULD CONTINUE BY READING THE FOLLOWING]

With your permission, we would also like to record our interview for note-taking purposes. Only our research team wil have access to the recording and we wil erase all recordings after our research is finished. Do you consent to have the interview recorded?

[IF YES, BEGIN RECORDING] .

Before we begin, do you have any questions for us about the interview?

Resident Focus Group Guide

[THESE ARE BACKGROUND QUESTIONS. TIME SPENT HERE SHOULD BE LIMITED

TO 10 MINUTES]

1. How long have you all been living in public housing here in [CITY]? Which development do you live in (LIST DEVELOPMENTS)?

2. What are some good things about living here?

3. What are some things that people do not like?

4. What are some common challenges that residents face?

5. Where do people turn for support when they need help with something?

[FACILITATOR READS] Now I’ d like to turn the conversation to talk more about the ROSS-SC program run by [GRANTEE].

6. Tel us a little bit about involvement with the ROSS-SC program.

a. How do residents find out about ROSS-SC?

b. Why do people decide to participate?

c. What ROSS-SC activities are available to residents? Where are these activities located?

d. How can people get in touch with the ROSS-SC? Are the service coordinators on site on a regular basis? How often? When?

[FACILITATOR READS]: We have some questions about the kinds of support services residents receive, first at your development, and then services provided at other places in the city.

7. What services are available to residents at your development?

[IF NOT MENTIONED BY RESPONDENTS, COVER EACH OF THE FOLLOWING

CATEGORIES]:

1 4 9

For Families – Employment; Childcare; Education; Job training; Certification program; Financial education or counseling; Other

For Elderly/Disabled – Independent living; Coordination of services; Alternatives to long-term care; Aging in place supports; Meals; Socialization; Other]

a. Who provides these services?

b. Where are these services provided? [Office, community center, in-home, other?]

c. Which ones have been helpful to residents, and how? Which ones haven’t been

helpful, and why not?

8. Do the service coordinators refer residents to other agencies or organizations for help or support?

d. IF so: What are some of these referral agencies or organizations?

e. Where are they located, and how do residents get there?

f. What supports do they provide?

[IF NOT MENTIONED BY RESPONDENTS, COVER EACH OF THE FOLLOWING

CATEGORIES]:

For Families – Employment; Childcare; Education; Job training; Certification program; Financial education or counseling; Other

For Elderly – Independent living; Coordination of services; Alternatives to long-term care; Aging in place supports; Meals; Socialization; Other

g. Which ones have been helpful to residents, and how?

h. Which ones haven’t been helpful, and why not?

9. Do people have plans or goals to move out of public housing? If so, do the service coordinators help people achieve these goals?

10. Are there any other services that people receive through the service coordinators or

[GRANTEE]? Do you know what program those services are a part of?

[FACILITATOR READS]: Now we have a few last questions about how you think the ROSS-SC program could be improved.

11. Are there certain resident needs that are hard for the ROSS-SC program to meet? Certain resident needs for which there are no opportunities for support or help?

12. How could your service coordinator better help you and other residents meet your needs?

13. Do you have any other feedback on the ROSS-SC program? Are there any other questions that we should be asking of residents to evaluate the ROSS-SC program?

[FACILITATOR SHOULD TAKE THIS TIME TO NOTE IN WRITING ANY RELEVANT INFORMAL

OBSERVATIONS]

1 5 0

Appendix D. Service Coordinator

Survey

Note: questions marked with asterisks (***) were weighted to the grantee level for the analysis (see appendix E for the weighting amounts).

Public Burden Statement

The public reporting burden for this information col ection is estimated to be half an hour. You are not required to respond to this collection of information unless a valid OMB control number is displayed.

OMB Number: 2528-0316

Expiration Date: 4/30/2021

Introduction

The purpose of this research is to understand the ROSS Service Coordinator (ROSS-SC) characteristics, interaction with public housing authorities (PHAs), and activities. [Grantee] has received a ROSS-SC grant from the U.S. Department of Housing and Urban Development (HUD) in order to fund your efforts to help their public housing residents attain economic and housing self-sufficiency, or to age in place and maintain independent living. This survey asks you for details about the following:

• The role and activities of your current position as a service coordinator

Characteristics and needs of the residents you serve

• How you interact with these residents

• The local service providers you work with in your role

• The types of support you receive from grantee

Your responses wil provide us with information on the type of work done by service coordinators and wil help us accurately represent the scope of service coordinators’ work in our report to HUD.

The survey consists of 73 questions and should take about 25 minutes to complete. Your participation in this survey is entirely voluntary. Your refusal to participate wil not affect your program’s funding or your employment as a service coordinator. You may also discontinue the survey at any time with no penalty.

1 5 1

The risks to participating are minimal; we wil report the results in aggregate, and the name of your employer wil not be attached to your responses. We wil make sure that no one sees your survey responses without approval. However, because we are using the Internet, there is a chance that someone could access your online responses without permission. In some cases, this information could be used to identify you.

For questions or concerns, please contact Patrick Spauster at 202-261-5874 or pspauster@urban.org.

Section 1

This section asks about your current role as a service coordinator for grantee.

Q1 How many months have you worked in your current position as a service coordinator?

o Fewer than 6 months

o 7-12 months

o 12-24 months

o More than 24 months

o Don’t know

Q2 On average, how many hours per week do you work for the ROSS-SC program?***

o Fewer than 20 hours per week

o Between 20-35 hours per week

o More than 35 hours per week

o Don’t know

1 5 2

Q3 Do you work on any other [grantee] programs?

o Yes

o No

o Don’t know

1 5 3

Section 2

This section asks about your activities as a service coordinator.

Q4 I’d like you to think now about all of the residents who are eligible to work with you in your job as a service coordinator. About how many residents do ROSS service coordinators currently serve?***

o Fewer than 25

o 26-50

o 51-100

o 101-150

o More than 150

o Don’t know

Q5 How many individual residents come to you every month to receive any type of assistance, on average?***

o Less than 10

o 10-20

o 21-30

o 31-40

o 41-50

o More than 50

o Don’t know

1 5 4

Q6 How many residents come to you at least once per month to work toward specific personal goals?

o Less than 10

o 10-20

o 21-30

o 31-40

o 41-50

o More than 50

o Don’t know

Q7 How often do you meet with the residents that you see on a regular basis?

o Two to three times a week

o At least once a week

o At least once every two weeks

o At least once a month

o Intermittently or as needed

o Don’t know

1 5 5

Q8 How long do meetings with residents typically last?

o Less than 15 minutes

o 15-30 minutes

o 30-45 minutes

o 45-60 minutes

o More than 60 minutes

Q9 Where do you meet with residents? Select al that apply.

▢ In the resident’s home

▢ In an office located in the public housing development

▢ In an office or other space located in the same neighborhood

▢ In an office or other space located in a different neighborhood

Q10 Does your organization have a Local Program Coordinating Committee (PCC) or something similar? The purpose of a PCC is to secure public and private resources to support ROSS-SC by establishing a network of advisors and service providers. The PCC may include representatives from 1 5 6

the PHA, public housing residents, local government, local service providers, and/or local employers.***

o Yes

o No

o Don’t know

Display This Question:

If Does your organization have a Local Program Coordinating Committee (PCC) or something similar?

Th. . = Yes

Q11 How frequently does the PCC meet?

o More than once a month

o Monthly

o Every other month

o Quarterly

o Annually

o Intermittently or as needed

o Other

o Don’t know

Display This Question:

If Does your organization have a Local Program Coordinating Committee (PCC) or something similar?

Th. . = Yes

1 5 7

Q12 How effective is the PCC in helping grantee achieve its goals?

o Very effective

o Somewhat effective

o Not at al effective

o Don’t know

Q13 Which of the fol owing functions are you performing as a service coordinator? Select all that apply.***

▢ Organizing a Local Program Coordinating Committee

▢ Marketing the program to residents

▢ Coordinating delivery of services

▢ Coordinating and sponsoring educational events

▢ Tracking and reporting to HUD the progress of residents enroled in the program

▢ Documenting overall program performance

1 5 8

Q14 Have you assisted families in resolving any of the fol owing issues that require immediate attention? That is, issues that had to be addressed that day? Select all that apply.

▢ Eviction prevention

▢ Domestic violence

▢ Food insecurity

▢ Property management/ maintenance

▢ Providing transportation to appointments

▢ Childcare

▢ Health emergencies

▢ Drug-related emergencies

▢ Working with child protective services

▢ Other immediate/emergency problems (specify):

________________________________________________

1 5 9

Q15 For each of the fol owing activities, please indicate whether you perform them in your duties as a service coordinator, or if grantee provides the services through another program.

Perform myself

Grantee

Service not

provided

Don’t know

provided

Coordinating

services on behalf

o

of individual

o

o

o

residents

Ensuring quality

of services

o

o

o

o

delivered

Tracking service

o

provision

o

o

o

Developing a

resident group to

o

promote self-

o

o

o

sufficiency efforts

Developing a

resident civic

o

engagement

o

o

o

group

Developing a

resident self-help

o

o

o

o

group

1 6 0

Section 3

This section asks about the type of people you serve and their needs.

Q16 What types of people do you target for ROSS-SC? Select all that apply.***

▢ Elderly residents

▢ Families with children

▢ Residents with physical disabilities

▢ Residents with mental health needs

▢ Unemployed residents

▢ Working residents

▢ Non-English-speaking residents

▢ Other (Please specify) ________________________________________________

1 6 1

Q17 For each of the fol owing potential service areas, please indicate whether it is one of the needs of your target population. Note that we are not asking whether grantee or its partners provide the service.***

1 6 2

Yes

No

Don’t know

Employment services,

such as job training

and placement

o

assistance, provision of

o

o

professional clothing,

or career planning

Education services,

such as GED training

programs, technical

o

education/job skills

o

o

training, or soft skills

training

Financial education

services, such as

o

financial coaching or

o

o

money management

Child or family

services, such as

childcare, early

o

o

o

childhood education,

or parenting guidance

Healthcare services,

such as healthcare

coordination, nutrition

o

o

o

education, or wellness

programs

Mental health or

o

behavioral services

o

o

Community services,

such as good neighbor

programs and

o

o

o

community safety

coordination

1 6 3

Services for seniors or

people with

disabilities, such as

transportation, meal

o

o

o

provision, homemaker

assistance, or personal

care

Other services (Please

o

specify)

o

o

Section 4

This section asks about how you interact with the residents you serve.

Q18 What types of residents are most likely to use services? Select all that apply.***

▢ Elderly residents

▢ Families with children

▢ Residents with physical disabilities

▢ Residents with mental health needs

▢ Unemployed residents

▢ Working residents

▢ Non-English-speaking residents

1 6 4

Q19 What types of residents are least likely to use services? Select al that apply.

▢ Elderly residents

▢ Families with children

▢ Residents with physical disabilities

▢ Residents with mental health needs

▢ Unemployed residents

▢ Working residents

▢ Non-English-speaking residents

1 6 5

Q20 How do you know if a resident needs help? Select al that apply.

▢ Information from property managers

▢ Information from service provider at a property

▢ Information from a service provider not operating at the property

▢ Institutional knowledge from {grantee}

▢ Informal assessment

▢ Formal intake assessment

▢ Other (please specify) ________________________________________________

Q21 How often do you conduct a formal assessment for participants?***

o Only at intake

o Monthly

o Semi-annually

o Annually

o Other (please specify) ________________________________________________

1 6 6

Q22 What share of participants has a formal intake assessment?***

o Al participants

o Most participants

o Some participants

o None of the participants

o Don’t know

1 6 7

Q23 Do you use Individual Training Service Plans, which are intended to help residents identify actions needed to become self-sufficient?***

o Yes

o No

o Don’t know

Display This Question:

If Do you use Individual Training Service Plans, which are intended to help residents identify actio... =

Yes

Q24 How do you use Individual Service Plans? Select all that apply.***

▢ For resident guidance

▢ As a plan for resident engagement

▢ As a means of tracking resident progress

▢ Other (please specify) ________________________________________________

Display This Question:

If Do you use Individual Training Service Plans, which are intended to help residents identify actio... =

Yes

1 6 8

Q25 What share of residents has an Individual Training Service Plan?***

o Al participants

o Most participants

o Some participants

o None of the participants

o Don’t know

Display This Question:

If Do you use Individual Training Service Plans, which are intended to help residents identify actio... =

Yes

Q26 How effective are Individual Training Service Plans in retaining participants?

o Very effective

o Somewhat effective

o Somewhat ineffective

o Very ineffective

o Don’t know

Display This Question:

If Do you use Individual Training Service Plans, which are intended to help residents identify actio... =

Yes

1 6 9

Q27 For which types of residents are Individual Training Service Plans most effective?***

o Elderly residents

o Families with children

o Residents with physical disabilities

o Residents with mental health needs

o Unemployed residents

o Working residents

o Non-English speaking residents

1 7 0

Section 5

This section asks about how you interact with grantee, how grantee supports your work as service coordinator, and what services grantee provides to the residents you serve.

Q28 What types of support do you receive from grantee? Select all that apply.

▢ Regular training opportunities

▢ Oversight

▢ Guidance on resident needs

▢ Guidance on local service providers

▢ Guidance on your responsibilities as service coordinator

▢ Other (please specify) ________________________________________________

1 7 1

Q29 Please indicate how satisfied you are with each of the fol owing ways in which grantee supports you in your role as service coordinator.

Very satisfied

Somewhat

Somewhat

Very

satisfied

Don’t know

unsatisfied

unsatisfied

Resources

made

available to

you by

o

o

o

o

o

grantee to

perform your

job duties

Training

opportunities

o

provided by

o

o

o

o

grantee

How often

grantee

o

provides you

o

o

o

o

with feedback

Quality of

feedback

o

grantee

o

o

o

o

provides

Quality of

information

on resident

o

o

o

o

o

needs grantee

provides

Number of

relationships

grantee has

o

developed

o

o

o

o

with service

providers

Quality of

relationships

grantee has

o

developed

o

o

o

o

with service

providers

1 7 2

Q30 Do you coordinate your activities with staff from any other HUD-funded programs run by grantee?

o Yes

o No

Section 6

This section asks about your partnerships with local service providers.

Q31 For the fol owing types of services, do you refer residents to service providers in that category? If so, are the service providers able to meet the demand for that service?***

Refer to partner organizations to

Partner organization able to meet

provide services?

resident demand for service?

Yes

Don’t

Not

Don’t

Not

No

know

Yes

No

available

know

available

1 7 3

Employment-

related

services: for

example, job

training and

placement

o

assistance,

o o o o o o o

provision of

professional

clothing, or

career

planning

Adult

education

services: for

example, GED

training

programs,

o o o o o o o o

technical

education/job

skills training,

or soft skills

training

Financial

education

services: for

example,

o

financial

o o o o o o o

coaching or

money

management

Child or

family

services: for

example,

childcare,

o

early

o o o o o o o

childhood

education, or

parenting

guidance

1 7 4

Healthcare

services: for

example,

healthcare

coordination,

o

nutrition

o o o o o o o

education,

clinics, or

wellness

programs

Mental health

or behavioral

o

health

o o o o o o o

services

Community

services: for

example,

good

neighbor

o o o o o o o o

programs, or

community

safety

coordination

Other

services:

o o o o o o o o

please specify

1 7 5

Section 7

This section asks about how you track your interactions with engaged residents.

Q32 How do you track client interactions?***

o Paper records

o Spreadsheets

o Off-the-shelf case management software

o Custom-design case management software

o Other (please specify) ________________________________________________

o Don’t know

1 7 6

Q33 Please indicate whether you track the fol owing resident outcomes.***

Yes

No

Don’t know

Adult educational

outcomes, such as

getting a GED, col ege

acceptance,

o

o

o

completing college

courses, getting a

college degree

Child educational

outcomes, such as

high school

o

graduation, improved

o

o

grades, college

enrol ment

Employment

outcomes, such as

finding a job, finding a

full-time job, keeping a

o

o

o

job for a certain length

of time, earning a

promotion

Housing outcomes,

such as avoiding

eviction, decrease in

o

lease violations,

o

o

moving to non-

subsidized housing

Health outcomes, such

as having a medical

home, obtaining

o

health benefits,

o

o

decreased negative

health reports

Outcomes for elderly

or disabled residents,

such as aging-in-place

services, placement in

o

o

o

independent-living

facilities, enrollment in

meals program

1 7 7

Section 8

This final section asks you to describe yourself.

Q34 How do you describe your gender?

o Female

o Male

o Other

o Don’t know

Q35 Do you describe your ethnicity as Hispanic or Latino/a?

o Yes

o No

o Don’t know

1 7 8

Q36 How do you describe your race?

o White

o Black or African American

o American Indian or Alaska Native

o Asian

o Hawaiian or Pacific Islander

o Other (please specify) ________________________________________________

o Don’t know

Q37 In what year were you born? (Please provide in format XXXX)

________________________________________________________________

1 7 9

Q38 What is the highest level of education that you have ever completed?

o Less than high school

o High school diploma, GED, or equivalent

o Some technical, vocational, or business courses

o Vocational/technical/business certificate or diploma

o Some colege

o Associate degree or technical certificate

o Four-year colege degree

o Some graduate school

o Graduate or professional degree

1 8 0

Q39 How many years of total work experience do you have?

o Less than 1 year

o 1-2 years

o 3-5 years

o 6-10 years

o More than 10 years

o Don’t know

Q40 Do you have any professional certifications that help you in your role as service coordinator?

o Yes

o No

o Don’t know

Q41 Please list your certifications that help you in your role as service coordinator.

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

1 8 1

Q42 What is your total compensation from grantee?

o Less than $15,000

o $15,000-$30,000

o $30,000-$45,000

o $45,000-$60,000

o $60,000-$70,000+

o Don’t know

1 8 2

Appendix E. Service Coordinator

Survey Weights

The team calculated service coordinator weights to account for different amounts of service coordinators at different grantees. A grantee weight was developed based on the total universe of grantees and the response rate by categories. Specifical y, we identified the distribution of grantees across the categories of grant year and service type from the HUD contact list, then divided that by the distribution of grantees where at least one service coordinator completed the survey.

Grant

Population Served

All Grantees

Grantees with at Least One

Weight

Year

(%)

Service Coordinator

Completion (%)

2014

Both

14

17

0.781

Families

12

7

1.613

Elderly

6

7

0.851

2015

Both

14

13

1.053

Families

11

12

0.929

Elderly

9

9

1.008

No data

1

1

0.538

2016

Both

19

19

1.043

Families

10

10

1.012

Elderly

4

4

1.076

No data

0

1

0.538

Total

100

100

Source: Survey of Service Coordinators and analysis.

1 8 3

Appendix F. Question Variations

Between Fielded Versus OMB-

Approved Service Coordinator

Survey

Question numbers refer to the OMB-approved version of the survey.



Between Q2 and Q3: Does the SC work on any other programs for the grantee? Yes/No.



Between Q4 and Q5: How many residents come to you every month to work toward specific personal goals?



Q5, on how often they meet with residents:

» Instead of asking about residents they meet with “at least once a month,” asked about residents they meet with “on a regular basis”



Q9, missing text that helps clarify the question

» OMB version:

 How frequently does the PCC schedule meetings? Please consider both

meetings attended by al members of the PCC, as well as those attended by a

smaller number of members who are available.

» Fielded version:

 How frequently does the PCC meet?



Q11, NOFA-specified functions performed by the SC

» OMB version

 11 items (“tracking” and “reporting” are split into two activities)

» Fielded version

 6 items

 Missing “overseeing routine delivery of services”

 Missing “ensuring quality of services delivered”

1 8 4

 Missing “encouraging residents to build informal self-sufficiency support

networks”

 Missing “supporting community-based groups to support self-sufficiency

efforts”

 Combined “tracking service provision” and “reporting to HUD the progress of

residents enrol ed in the program” into “tracking and reporting to HUD the

progress of residents enrol ed in the program”



Between Q12 and Q13: This question combined with Q11 address the NOFA-specified

functions.

» Q19. For each of the fol owing activities, please indicate whether you perform them in your duties as a service coordinator, or if grantee provides the services through another program.

 Coordinating services on behalf of individual residents

 Ensuring quality of services delivered

 Tracking service provision

 Developing a resident group to promote self-sufficiency efforts

 Developing a resident civic engagement group

 Developing a resident self-help group



Q13, types of residents living at the property served, not included



Q15, whether disabled residents are served by the SC, not included



Q16, proportion of disabled clients that are not elderly, not included



Q17, proportion of residents eligible for services that don’t speak English, not included



Q18, provisions made for non-English-speaking residents, not included



Q23, services for youth or older children, not included



Q29, types of residents most likely to use services, missing “single-parent families” as a response option



Between Q29 and Q30, types of residents least likely to use services



Q30, how the SC knows if a resident needs help, missing “direct community outreach” and

“neighbor referral” response options



Q34, how ITSPs are used, response options changed

» OMB version

1 8 5

 To identify resident needs

 To direct residents to available services

 To track resident engagement with service providers

 To help residents set personal goals

 To track resident progress toward goals

 Other (please specify)

» Fielded version

 For resident guidance

 As a plan for resident engagement

 As a means of tracking resident progress

 Other (please specify)



Q36, how effective are ITSPs in assisting residents with their goals, instead asked how effective ITSPs are in retaining participants



Q37, which type of residents do the ITSPs help, missing “single-parent families” response option



Additional question in fielded version

» What types of support do you receive from grantee? Select all that apply.

 Regular training opportunities

 Oversight

 Guidance on resident needs

 Guidance on local service providers

 Guidance on your responsibilities as service coordinator

 Other (please specify)



Q47, on satisfaction with support from grantee for guidance on SC responsibilities, not included



Between Q53 and Q54, whether the SC coordinates activities with staff from other HUD-funded programs run by the grantee

1 8 6

Appendix G. ROSS-SC Service

Coordinators—Needs and Service

Partners (HUD Form 52769)

NEED?

SERVICE PROVIDER/

NEEDS

(check all that

apply–see NOFA

PARTNER(s)

Value of Match*

for requirements)

(list all)

Life Skil s Training

Financial Literacy/Credit

Counseling/Credit Repair

Literacy Training

ESL

GED/High School Equiv.

Mentoring

Job Soft Skil s Training

Job Hard Skills

Training/Certification

Job Search and Placement

Job Retention/Promotion

ISAs/IDAs

Homeownership

Counseling

Computer Classes

Drug/Alcohol Treatment

Health/Dental Care

Home Maintenance Classes

Parenting Classes

Nutrition Classes

Youth Programming —

Tutoring/Mentoring/

Afterschool/Summer

Childcare

Transportation

Tax Preparation Assistance

Community Safety

Resident

Empowerment/Capacity

Building

Resident Business

Development

Assistance with Activities

of Daily Living

1 8 7

Meals To Meet Nutritional

Need for Elderly

Disability Services

Counseling

Personal Emergency

Response Resources

Wel ness Programs

Other (please describe)

Other

Other

Other

Other

TOTAL

$

1 8 8

Appendix H. FY 2014-2016

Logic Model

This page intentional y left blank.

1 8 9

1 9 0

1 9 1

1 9 2

1 9 3

1 9 4

1 9 5

1 9 6

1 9 7

1 9 8

1 9 9

2 0 0

2 0 1

2 0 2

2 0 3

2 0 4

2 0 5

2 0 6

2 0 7

2 0 8

2 0 9

2 1 0

2 1 1

Appendix I. FY 2014 Application-

Logic Model Service Classification

Family-Serving Grantee Indicator Variables:



Application (HUD Form 52769)

» Parenting classes.

» Youth programming–tutoring/mentoring/afterschool/summer.

» Childcare.



FY 2014 Logic Model

» Childcare–Families referred to childcare services.

» Training–Parenting/household skills/life skills–Enrolled.

Elderly/Disabled-Serving Grantee Indicator Variables:



Application (HUD Form 52769)

» Assistance with activities of daily living.

» Meals to meet nutritional needs for elderly.

» Disability counseling services.



FY 2014 Logic Model

» Seniors/disabled–Service coordination–Referrals for senior/disability services.

» Outreach–Outreach to seniors/persons with disabilities.

» Seniors/disabled–Food and nutrition–Congregate meals coordinated.

2 1 2

Appendix J. Guidance for

Completing the 2014 ROSS-SC

Logic Model

Glossary of Terms—Services/Activities

Services/Activities

Units

How to Count

Examples

Count the number of service coordinators

Administration-Hire

you hired during the reporting period.

ROSS-SC Service

Persons

Coordinator

Mandatory metric.

Count each household that is referred to a

new childcare arrangement during the

reporting period.

Childcare-Families

Do not count if a family is still engaged in a

referred to childcare Households

childcare arrangement from a previous

services

reporting period.

Mandatory metric. If not applicable to your

program enter “0.”

Count each participant who enrol s or

Examples of Adult

continues during the reporting period.

Basic Education

include basic

A person who enrol s or continues in formal

reading, writing,

Education-Adult

(e.g. uses textbooks and handouts) classes

literacy, math skills,

Basic

Persons

or one-on-one adult basic education

pre-admission

Education/Literacy

tutoring during the reporting period may

col ege prep

classes-Enrol ed

be counted.

courses and other

adult continuing

Mandatory metric. If not applicable to your

education classes

program enter “0.”

(non-credit).

Count each participant who enrol s or

continues during the reporting period.

A person who participates (enrol s or

Education-High

continues) in formal (e.g. uses textbooks

School/GED

Persons

and handouts) high school classes or one-

program-Enrol ed

on-one GED tutoring during the reporting

period may be counted.

Mandatory metric. If not applicable to your

program enter “0.”

2 1 3

Services/Activities

Units

How to Count

Examples

Count each ROSS-SC participant who enrol s or

Education-Post-

continues in post-high school courses during

secondary classes-

Persons

the reporting period.

Enrol ed

Optional metric.

Count each skil assessment begun for

participants during the reporting period.

Employment-Job

Persons

skil s assessment

Mandatory metric. If not applicable to your

program enter “0.”

Count each participant that enrol s or continues

Financial Literacy-

in a financial literacy or financial

Financial

management course.

Persons

literacy/managemen

t classes-Enrol ed

Mandatory metric. If not applicable to your

program enter “0.”

Count each person who receives (starts or

continues) individualized financial

Financial Literacy-

literacy/budget/credit counseling during the

Individual

Persons

reporting period.

Counseling-Enrol ed

Mandatory metric. If not applicable to your

program enter “0.”

Count each person who receives information

Financial Literacy-

related to opening or maintaining a bank or

Banking, credit and

Persons

IDA account or credit.

IDA information

provided

Optional metric.

Count each person that the service coordinator

provides information to about the ROSS-SC

Outreach-Outreach

program whether through telephone or in-

to seniors/persons

Persons

person contact.

with disabilities.

Optional metric.

Count each individual that the service

Outreach-Outreach

coordinator provides information to about

to individuals (non-

the ROSS-SC program whether through

Persons

elderly/non-

telephone or in-person contact.

disabled) residents

Optional metric.

Count the number of organizations/service

providers the service coordinator

successfully made contact with during the

reporting period to discuss the ROSS-SC

Outreach-Service

Providers

program and potential partnership

providers contacted

opportunities.

Mandatory metric. If not applicable to your

program enter “0.”

2 1 4

Services/Activities

Units

How to Count

Examples

Policy Priority 1-

Count the number of program participants

Healthcare

whose health insurance status has been

Assessment-Health

Persons

evaluated during the reporting period.

insurance

assessment

Optional metric.

conducted

Example 1: If

Participant A was

referred for

mental health

services in year 1

Count the number of participants that have

and again in year

been referred to healthcare services

2, Participant A

Policy Priority 1-

(physical, mental, dental) during the

should be

Healthcare

reporting period.

counted one time

Assistance-Referral

Persons

for each year.

for healthcare

You may count a participant in each reporting

services

period for which a referral was made.

Example 2: If

Participant A is

Optional metric.

referred multiple

times for mental

health services in

year 1, count the

referral only

once.

Policy Priority 1-

Count the number of residents the service

Healthcare

coordinator has identified as high-risk for

Assistance-Target

Persons

health-related problems.

High-Need Clients-

Most vulnerable

Optional metric.

residents identified

Policy Priority 1-

Healthcare

Count the number of high-risk (health-related)

Assistance-Target

residents whose health needs have been

High-Need Clients-

Persons

assessed.

Baseline needs

identified for most

Optional metric.

vulnerable residents

Examples of

healthcare

Policy Priority 1-

providers:

Partnership with

Count the number of health-related providers

Federal y

Healthcare

the service coordinator has successful y

Qualified Health

Organization-

Providers

contacted during the reported period.

Clinics, other

Outreach to

local health

Healthcare

Optional metric.

clinics, hospitals,

Providers

nurses’

associations, etc.

2 1 5

Services/Activities

Units

How to Count

Examples

Policy Priority 2-

Count the number of participants who have

Promote Economic

been referred to employers during the

Development-

Persons

reporting period.

Participants referred

to employers

Optional metric.

Count the number of participants who have

enrol ed in job training classes during the

Policy Priority 2-

reporting period.

Promote Economic

Development-Job

Persons

Do NOT count the person if they are

training classes-

continuing a program in which they enrol ed

Enrol ed

during a prior reporting period.

Optional metric.

Count the number of participants who received

Seniors/Disabled-

congregate meals offered through partners

Food and Nutrition-

Persons

during the reporting period.

Congregate Meals

coordinated

Optional metric.

Count the number of participants who have

been referred to needed services during the

reporting period.

Seniors/Disabled-

Do NOT count the number of referrals

Service

individual participants have been given.

Coordination-

Persons

Referrals for

NOTE: This service/activity is listed three times

senior/disability

in the LM as it relates to three different

services

outcomes.

Mandatory metric. If not applicable to your

program enter “0.”

Count al participants that enrol ed and began

completing a new ITSP during the reporting

Service

period.

Coordination-

Service

You may count participants as “new” that have

Coordination-New

been enrol ed previously but dropped out.

Persons

Participants

Enrol ed-Individual

Do NOT count the participant if they continue

Training Service

to have an open ITSP (these participants are

Plans (ITSPs) begun

counted in the row below).

Mandatory metric.

2 1 6

Services/Activities

Units

How to Count

Examples

Service

Coordination-

Participants with

ITSPs continuing to

receive services-

Count al individuals that were enrol ed with an

Year 1

ITSP prior to this reporting period that

continue to have an open ITSP at the

Service

beginning of the reporting period This

Coordination-

category may include participants who no

Participants with

Persons

longer have an open ITSP at the end of the

ITSPs continuing to

reporting period (i.e. due to accomplishment

receive service-Year

of all goals in ITSP or services no longer

2

needed).

Service

Mandatory metric.

Coordination-

Participants with

ITSPs continuing to

receive services-

Year 3

Count al INDIVIDUAL residents that are

seeking services during the reporting period

that do NOT have an ITSP, (e.g. other

individuals or family members of participants

with ITSP, even if you did only a one-time

referral or service connection). Do not count

the number of services, just the individuals.

Count children/youth if services were

sought for training/educational/

Service

teen employment/financial literacy, after

Coordination-Non-

school/enrichment services, health/mental

ITSP individuals

Persons

health services, etc. Do NOT count children

seeking services

for childcare (if a household was linked to

(unduplicated count)

childcare – this is captured under the

Childcare-Families referred to childcare

services).

Note: “unduplicated count” refers to counting

each individual only once in this category.

Do not count individuals more than once in

this category.

Mandatory metric. If not applicable to your

program enter “0.”

2 1 7

Services/Activities

Units

How to Count

Examples

Examples of

Parenting/

Household

Skills/Life Skills

Count each participant that enrol s in a

include: Good

parenting/household skills/life skills

neighbor training,

course/training during the reporting period.

parenting classes,

Training-

household

Parenting/Househol

Persons

Do NOT count the person if they are

management,

d Skills/Life skills

continuing a program in which they enrol ed

nutrition classes,

classes-Enrol ed

during a prior reporting period.

civic engagement,

navigating

Optional metric.

community

resources,

citizenship

classes, driver’s

education, etc.

Examples of

transportation

services include

bus

passes/tokens,

Count each participant who receives (starts or

rides in a

Transportation-

continues) transportation assistance (to

grantee-owned

Referrals to

enable service provision) during the

van, arranging car

transportation

Persons

reporting period.

pools, connecting

services to enable

to city/county

service provision

Optional metric.

special

transportation

opportunities,

assistance with

personal auto

repair, etc.

Examples of

transportation

services include

bus

passes/tokens,

rides in a

Count each participant who receives (starts or

Transportation-

grantee-owned

continues) transportation assistance (to

Referrals to

van, arranging

enable employment) during the reporting

transportation

Persons

carpools,

period.

services to enable

connecting to

employment

city/county

Optional metric.

special

transportation

opportunities,

assistance with

personal auto

repair, etc.

2 1 8

Glossary of Terms—Outcomes

Outcomes

Units

How to Count

Examples

Count the number of ROSS-SC service

Administration-

coordinators hired during the reporting

ROSS-SC service

Persons

period.

coordinator hired

Mandatory metric.

Count each household that successful y

obtained new childcare services as a result

of the ROSS-SC’s referral.

Childcare-Families

obtain childcare

Households

Do not count if a family was referred during a

services

prior reporting period.

Mandatory metric. If this metric does not

apply to your program enter “0.”

Examples of Adult

Basic Education

Count each participant who successful y

include basic

completes a class in this category/receives

Education-Adult

reading, writing,

certificate of completion, even if the

Basic

literacy, math skills,

participant was referred during a prior

Education/Literac

Persons

pre-admission

reporting period.

y classes-

col ege prep

Completed

courses and other

Mandatory metric. If applicable to your

adult continuing

program enter “0”.

education classes

(non-credit).

Count each participant who receives a High

School diploma/GED certificate or other

high school equivalency certification during

Education-High

this reporting period. Count even if the

school

Persons

participant was referred during a prior

diploma/GED

reporting period.

obtained

Mandatory metric. If not applicable to your

program enter “0.”

Count each ROSS-SC participant who

Education-Post

completes a post-high school course(s)

secondary classes-

Persons

during the reporting period.

Competed

Optional metric.

2 1 9

Outcomes

Units

How to Count

Examples

Count each participant for

whom job skil s assessments

were completed during the

Employment-Job skills

reporting period.

Persons

determined

Mandatory metric. If not

applicable to your program

enter “0.”

Count each participant who

successfully completes a

financial literacy or financial

management course during

Financial Literacy-

the reporting period. Count

Financial

even if the participant was

Persons

literacy/management

referred during a previous

class-Completed

reporting period.

Mandatory metric. If not

applicable to your program

enter “0.”

Count each participant who

completes individualized

financial

literacy/budget/credit

Financial Literacy-

counseling during the

Individual Counseling-

Persons

reporting period.

Completed

Mandatory metric. If not

applicable to your program

enter “0.”

Count the number of

organizations/service

providers the service

coordinator successful y

entered into a partnership

Outreach-Providers

with to provide services to

make agreements with

ROSS-SC program

Providers

agency to provide

participants. Count

services

agreements made during the

reporting period.

Mandatory metric. If not

applicable to your program

enter “0.”

2 2 0

Outcomes

Units

How to Count

Examples

Count al participants that

completed an ITSP during

reporting period.

Service Coordination-

New Participants

You may count participants

Enrol ed-Individual

Persons

having completed an ITSP

Training Service Plans

even if they began their

(ITSPs) completed

ITSP in a prior reporting

period.

Mandatory metric.

Count al individuals that were

enrol ed with an ITSP who

have completed their goals

and no longer require

services. The difference

between the number of

participants counted under

the service “Service

Service Coordination-

Coordination-Participants

Services no longer

Persons

with ITSPs continuing to

needed-ITSP goals

receive services” and this

achieved-Year 1

metric will tell HUD how

many participants continue

to need services during Year

1.

Mandatory metric. If not

applicable to your program

enter “0.”

Count al individuals that were

enrol ed with an ITSP who

have completed their goals

and no longer require

services. The difference

between the number of

participants counted under

the service “Service

Service Coordination-

Coordination Participants

Services no longer

Persons

with ITSPs continuing to

needed- ITSP goals

receive services” and this

achieved-Year 2

metric will tell HUD how

many participants continue

to need services during Year

2.

Mandatory metric. If not

applicable to your program

enter “0.”

2 2 1

Outcomes

Units

How to Count

Examples

Count al individuals that were

enrol ed with an ITSP who

have completed their goals

and no longer require

services. The difference

between the number of

participants counted under

the service “Service

Service Coordination-

Coordination Participants

Services no longer

Persons

with ITSPs continuing to

needed-ITSP goals

receive services” and this

achieved-Year 3

metric will tell HUD how

many participants continue

to need services during Year

3.

Mandatory metric. If not

applicable to your program

enter “0.”

Count al INDIVIDUALS who

were connected to services

during the reporting period

that do NOT have an ITSP,

(e.g. other individuals or

family members of

participants with ITSPs who

themselves do not have an

ITSP, even if you did only a

one-time referral or service

connection).

Count children/youth if they

were connected to

Service Coordination-

training/educational/teen

Non-ITSP individuals

Persons

employment/financial

served (unduplicated

literacy, after

count)

school/enrichment services,

health/mental health

services etc. Do NOT count

children for childcare (if a

household was linked to

childcare - this is captured

under the Childcare-Families

referred to childcare

services).

Note: “unduplicated count”

refers to counting each

individual only once in this

category. Do not count

2 2 2

individuals more than once

in this category.

Mandatory metric. If not

applicable to your program

enter “0.”

Count each number of

participants who have

successfully received

services. Count each

participant only once, even

if the participant was

referred to multiple services

during the reporting period.

Do NOT count the number of

Seniors/Disabled-Service

referrals individual

Coordination-

Persons

participants have been

Seniors/disabled obtain

given.

needed services

Note: This outcome is linked to

the activity/service:

“Seniors/Disabled-Service

Coordination-Referrals for

senior/disability services.”

Mandatory metric. If not

applicable to your program

enter “0.”

Count the number of

participants whose living

conditions/quality of life has

improved as a result of

services received during the

reporting period.

Count each participant only

Seniors/Disabled-

once.

Improved living

Persons

conditions/quality of life

Note: This outcome is linked to

the activity/service:

“Seniors/Disabled-Service

Coordination-Referrals for

senior/disability services.”

Mandatory metric. If not

applicable to your program

enter “0.”

2 2 3

Outcomes

Units

How to Count

Examples

Count the number of

participants whose living

conditions/quality of life has

improved as a result of

services received during the

reporting period.

Count each participant only

Seniors/Disabled-Live

once.

independently/age in

Persons

place and avoid long-

Note: This outcome is linked to

term care placement

the activity/service:

“Seniors/Disabled-Service

Coordination-Referrals for

senior/disability services.”

Mandatory metric. If not

applicable to your program

enter “0.”

Count each participant who

opened a checking/savings

account after receiving

information from the ROSS-

SC during the reporting

period.

Financial Literacy-

Checking/Savings

Persons

NOTE: This outcome is related

account established

to service/activity:

“Financial Literacy-Banking,

credit and IDA information

provided.”

Optional metric.

Count each participant whose

credit score improved after

receiving information from

the ROSS-SC during the

reporting period.

Financial Literacy-Credit

Persons

NOTE: This outcome is related

score improved

to service/activity:

“Financial Literacy-Banking,

credit and IDA information

provided.”

Optional metric.

2 2 4

Outcomes

Units

How to Count

Examples

Count each participant who

established an IDA account

after receiving information

from the ROSS-SC during

the reporting period.

Financial Literacy-IDA

Persons

NOTE: This outcome is related

account established

to service/activity:

“Financial Literacy-Banking,

credit and IDA information

provided.”

Optional metric.

Count each person that the

Outreach-Eligible

service coordinator provides

seniors/persons with

information to who enrol s

Persons

Add to Outcomes

disabilities enrol ed in

in the ROSS-SC-program.

ROSS-SC program

Optional metric.

Count each person that the

Outreach-Eligible

service coordinator provides

individuals (non-

information to who enrol s

Individuals

elderly/non-disabled

in the ROSS-SC program.

enrol ed)

Optional metric.

Count the number of program

participants who have

Policy Priority 1-Health

obtained health insurance

Care Assessments-

after insurance assessment

Persons

Residents obtain

conducted during the

insurance

reporting period.

Optional metric.

Count the number of

participants that have

received healthcare services

(physical, mental, dental)

Policy Priority 1-Health

during the reporting period.

Care Assistance-

Persons

Healthcare services

Count each participant once

obtained

even if multiple referrals

were provided during a

reporting period.

Optional metric.

2 2 5

Outcomes

Units

How to Count

Examples

Count the number of

vulnerable residents that

were addressing their

baseline needs during the

Policy Priority 1-Target

reporting period.

High-Need Clients-Most

vulnerable residents

Persons

Count an individual once

linked to services

during a reporting period,

addressing their baseline

even if that participant was

health needs

referred to multiple times

during the reporting period.

Optional metric.

Count the number of

Policy Priority 1-

healthcare organizations

Partnerships with Health

that have entered into an

Such a partnership would

Care Organizations-

agreement/partnership

entail accepting direct

Providers

Partnership established

arrangement with the

referrals from the ROSS-

with healthcare

ROSS-SC program.

SC.

providers

Optional metric.

Count the number of

Policy Priority 1-

participants for whom a

Partnership with Health

medical home has been

Care Organizations-

Persons

established.

Medical home*

established for residents

Optional metric.

Count each participant who

moves from a status of

unemployed to part-time

employed (including

apprenticeship) during the

reporting period.

Count each participant only

Policy Priority 2-

once even if they change

Promote Economic

status several times.

Development-

Persons

Participants obtain part-

Count part-time and seasonal

time employment

employment as

Do not count in this category if

they maintain employment

achieved in a previous

reporting period.

Optional metric.

2 2 6

Outcomes

Units

How to Count

Examples

Count each participant who

moves from a status of

unemployed, employed

part-time to ful -time

employment during the

reporting period.

Policy Priority 2-

Promote Economic

Count each participant only

Development-

Persons

once even if they change

Participants obtain ful -

status several times.

time employment

Do not count in this

category if they maintain

employment achieved in

previous reporting period.

Optional metric.

Count the number of

Policy Priority 2-

participants whose wages

Promote Economic

increased during the

Persons

Development -Earned

reporting period.

income increases

Optional metric.

Count the number of

participants who completed

Policy Priority 2-

job training courses and

Promote Economic

whose skills improved as a

Development-

Persons

result during the reporting

Participants’ job skills

period.

increase

Optional metric.

2 2 7

Outcomes

Units

How to Count

Examples

Count the number of

participants who were

referred to and participated

in a congregate meal

Seniors/Disabled-Food

program during the

and Nutrition-

Persons

reporting period.

Participants have

adequate nutrition

Count each individual

participant once.

Optional metric.

Count each participant that

Examples of Parenting/

completed a

Household Skills/Life Skills

parenting/household

include: Good neighbor

skills/life skills

Training-

training, parenting classes,

course/training during the

Parenting/Household

household management,

Persons

reporting period. You may

Skills/Life Skills classes-

nutrition classes, civic

count the completion even

Completed

engagement, navigating

if the enrol ment began in a

community resources,

prior enrol ment period.

citizenship classes, driver’s

education, etc.

Optional metric.

Examples of transportation

services include: bus

Count each participant who

passes/tokens, rides in a

obtained a service as a

grantee-owned van,

Transportation-Service(s)

result of transportation

arranging carpools,

obtained as a result of

Persons

assistance obtained during

connecting to city/county

transportation

the reporting period.

special transportation

opportunities, assistance

Optional metric.

with personal auto repair,

etc.

Examples of transportation

services include: bus

Count each participant who

passes/tokens, rides in a

obtains employment as a

grantee-owned van,

Transportation-

result of transportation

arranging carpools,

Employment obtained as

Persons

assistance obtained during

connecting to city/county

a result of transportation

the reporting period.

special transportation

opportunities, assistance

Optional metric.

with personal auto repair,

etc.

*A Medical Home is a patient-centered practice providing comprehensive primary care that facilitates partnerships between individual patients, their personal providers, and, when appropriate, the patient’s family. The medical home team of care providers is responsible for a patient’s physical and mental health, including preventive, acute, and chronic care. The provision of medical homes can allow better access to health care, increased satisfaction with care, and improved or maintenance health.

2 2 8

Appendix K. 2011-2014 Crosswalk

Key

No

A good match: No major issues with comparison over time.

color

No exact match: Denotes metrics that sum to an analogous metric (e.g., S13+S14

=P2)

An inexact match: Denotes metrics that should only be compared with caution

(e.g., S78 is “referrals for disability services”, while P10 includes both senior and disability services; numerical changes may be artifacts of this change in scope, or

“scope mismatch”).

Poor match: Denotes metrics for which no comparable metrics exists. If there is

an approximation, matching is problematic (e.g., S12’s target population is

unspecified, while PA8 specifies high-needs residents; if S12 includes everyone, the data are incomparable).

Services/Activities

Mandatory Services/Activities

Var.

FY 2011

Var. FY 2014

Notes

ID

ID

S1

Administration-Hire ROSS-SC

P1

Administration-Hire ROSS-SC

Service Coordinator |Persons

service coordinator

S13

Childcare-Working households

P2

Childcare-Families referred to

S13+S14=P2

linked to childcare services

childcare services

|Households

S14

Childcare-Non-working households

under contract linked to childcare

services |Households

S16

Education-Adult Basic Education-

P3

Education-Adult Basic

S16+S26=P3

Enrol ed |Persons

Education/Literacy classes-Enrol ed

S26

Education-Literacy class-Enrol ed

|Persons

S24

Education-GED program-Enrol ed

P4

Education-High school/GED

S24+S25=P4

|Persons

program-Enrol ed

S25

Education-High school-Enrol ed

|Persons

S37

Employment-Skills assessment

P5

Employment-Job skil s assessment

|Persons

2 2 9

S40

Financial Literacy-Financial

P6

Financial Literacy-Financial

literacy/Financial management

literacy/management classes-

education-Enrol ed |Persons

Enrol ed

S39

Financial Literacy-Credit repair

P7

Financial Literacy-Individual

S39+S42=P7

counseling-Enrol ed |Persons

Counseling-Enrol ed

S42

Financial Literacy-Financial

management counseling-Enrol ed

|Persons

S70

Outreach-Service Coordination-

P8

Outreach-Service providers

Service providers contacted

contacted

|Providers

S78

Service Coordination-Referrals for

P9

Seniors/Disabled-Service

Scope

disability services connected

Coordination-Referrals for

mismatch

|Persons

senior/disability services

Nul

No match

P10

Seniors/Disabled-Service

Repeats P9

Coordination-Referrals for

senior/disability services

Nul

No match

P11

Seniors/Disabled-Service

Repeats P9

Coordination-Referrals for

senior/disability services

S10

Case Management-Service

P12

Service Coordination-New

Scope

Coordination-Individual Training

Participants Enrol ed-Individual

mismatch

Service Plans (ITSPs) developed

Training Service Plans (ITSPs) begun

|Persons

Nul

No match

P13

Service Coordination-Non-ITSP

individuals seeking services

(unduplicated count)

Nul

No match

P14

Service Coordination-Participants

with ITSPs continuing to receive

services-Year 1

Services/Activities, Continued

Optional Services/Activities

Var.

FY 2011

Var.

FY 2014

Notes

ID

ID

S29

Education-Post secondary

PA1

Education-Post secondary

Classes-Enrol ed |Persons

classes-Enrol ed

S44

Financial Literacy-IDA accounts

PA2

Financial Literacy-Banking, credit Scope mismatch

established-Persons |Persons

and IDA information provided

Nul

No match

PA3

Outreach-Outreach to

individuals (NON-ELDERLY-

NON-DISABLED)

2 3 0

S63

Outreach-Outreach to elderly

PA4

Outreach-Outreach to

Scope mismatch

persons with disabilities

seniors/persons with disabilities

|Households

S112 new- Outreach to the elderly

|Persons

Nul

No match

PA5

Policy Priority 1-Healthcare

Assessment-Health insurance

assessment conducted

S55

Health-Referral for dental

PA6

Policy Priority 1-Healthcare

S55+S56+S57=PA6

services |Persons

Assistance-Referral for

healthcare services

S56

Health-Referral for healthcare

services |Persons

S57

Health-Referral for mental

health services |Persons

Nul

No match

PA7

Policy Priority 1-Partnership

with Healthcare Organization-

Outreach to Healthcare

Providers

S12

Case Management-Service

PA8

Policy Priority 1-Target High-

Unclear scope

Coordination-Needs

Need Clients-Baseline needs

assessments conducted |Persons

identified for most vulnerable

residents

Nul

No match

PA9

Policy Priority 1-Target High-

Need Clients-Most vulnerable

residents identified

S84

Training-Employment-Job

PA10 Policy Priority 2-Promote

training classes-Enrol ed

Economic Development-Job

|Persons

training classes-Enrol ed

S67

Outreach-Service Coordination-

PA11 Policy Priority 2-Promote

Unclear scope

Employers contacted |Employers

Economic Development-

Participants referred to

employers

S47

Food and Nutrition-Health-

PA12 Seniors/Disabled-Food and

Congregate meals coordinated

Nutrition-Congregate Meals

|Persons

coordinated

S94

Training-Parenting classes-

PA13 Training-Parenting/Household

S94+S88+S92=PA13

Enrol ed |Persons

Skills/Life skills classes-Enrol ed

S88

Training-Household skil s

training-Enrol ed |Persons

S92

Training-Life skills training-

Enrol ed |Persons

2 3 1

Nul

No match

PA14 Transportation-Referrals to

transportation services to enable

employment

Nul

No match

PA15 Transportation-Referrals to

transportation services to enable

service provision

Outcomes

Mandatory Outcomes

Var. ID

FY 2011

Var. ID FY 2014

Notes

Nul

No match

O1

Administration-ROSS-SC

service coordinator hired

Nul

No match

O2

Childcare-Families obtain

childcare services

Nul

No match

O3

Education-Adult Basic

Education/Literacy class-

Completed

O12

Education-GED obtained

O4

Education-High school

|Persons

diploma/GED obtained

O13

Education-High school

diploma obtained |Persons

O15

Employment-Certification

O5

Employment-Job skills

Scope

from private industry

determined

mismatch

|Persons

Nul

No match

O6

Financial Literacy-Financial

literacy/management class-

Completed

Nul

No match

O7

Financial Literacy-Individual

counseling-Completed

O56

Number of new Supportive

O8

Outreach-Providers make

Poor match

Service Partners providing

agreements with agency to

services as a result of the

provide services

gap analysis |Partners

O68

Self-Sufficiency-Improved

O9

Seniors/Disabled-Improved

Unclear

living conditions/quality of

living conditions/quality of

scope

life |Persons

life

O70

Self-Sufficiency-Live

O10

Seniors/Disabled-Live

Unclear

independently/age in place

independently/age in place

scope

and avoid long-term care

and avoid long-term care

placement |Persons

placement

O96

new- Residents connected to O11

Seniors/Disabled-Service

Poor match

services/programs

Coordination-

|Persons

Seniors/disabled obtain

needed services

Nul

No match

O12

Service Coordination-New

Participants Enrol ed-

Individual Training Service

Plans (ITSPs) completed

Nul

No match

O13

Service Coordination-Non-ITSP

individuals served

(unduplicated count)

2 3 2

Nul

No match

O14

Service Coordination-Services

no longer needed-ITSP goals

achieved-Year 1

Outcomes, Continued

Optional Outcomes

Var. ID

FY 2011

Var. ID FY 2014

Notes

O7

Education-Associates degree OA1

Education-Post secondary

Poor match

obtained |Persons

classes-Completed

O8

Education-Bachelor’s degree

Poor match

obtained |Persons

O10

Education-Certification from

Poor match

post-secondary school

|Persons

O26

Financial Literacy-

OA2

Financial Literacy-

Checking/Savings account

Checking/Savings account

established |Persons

established

O29

Financial Literacy-Credit

OA3

Financial Literacy-Credit score

score improved |Persons

improved

O41

Financial Literacy-IDA

OA4

Financial Literacy-IDA account

Scope

account deposits-Persons

established

mismatch

|Persons

Nul

No match

OA5

Outreach-Eligible

seniors/persons with

disabilities enrol ed in ROSS-

SC program

Nul

No match

OA6

Outreach-Non-elderly/non-

disabled residents enrol ed

Nul

No match

OA7

Policy Priority 1-Healthcare

Assessments-Residents

obtain insurance

Nul

No match

OA8

Policy Priority 1-Healthcare

Assistance-Healthcare

services obtained

Nul

No match

OA9

Policy Priority 1-Partnership

with Healthcare

Organizations-Medical

home* established for

residents

Nul

No match

OA10

Policy Priority 1-Partnerships

with Healthcare

Organizations-Partnership

established with healthcare

providers

O96

new- Residents connected to OA11

Policy Priority 1-Target High-

Scope

services/programs

Need Clients-Most

mismatch

|Persons

vulnerable residents linked

to services addressing their

baseline health needs

O31

Financial Literacy-Earned

OA12

Policy Priority 2-Promote

Unit

income increased-

Economic Development-

mismatch

Households |Households

Earned income increases |

Persons

2 3 3

Nul

No match

OA13

Policy Priority 2-Promote

Economic Development-Job

training classes-Completed

O24

Employment-Section 3-

OA14

Policy Priority 2-Promote

Employment obtained

Economic Development-

part-time |Persons

Participants obtain part-time

employment

O25

Employment-Full time

OA15

Policy Priority 2-Promote

Unit

equivalent (FTE) |FTE

Economic Development-

mismatch

Participants obtain ful -time

employment| Persons

Nul

No match

OA16

Policy Priority 2-Promote

Economic Development-

Participants’ job skills

increase

Nul

No match

OA17

Seniors/Disabled-Food and

Nutrition-Participants have

adequate nutrition

O80

new- Training-Parenting

OA18

Training-Parenting/Household

Scope

Classes-Completed

Skills/Life skills classes-

mismatch

|Persons

Completed

Nul

No match

OA19

Transportation-Employment

obtained as a result of

transportation

Nul

No match

OA20

Transportation-Service(s)

obtained as a result of

transportation

2 3 4

Appendix L. FY 2014 Outputs

Versus Outcomes by Metric

Service/Activities (Outputs)

Outcomes

# of

# of

As a % of

Grantees

Grantees

Grantees who

where

who

provided

Outcomes

provided

service

exceeded

services

Mandatory

Outputs

P1

Administration-Hire ROSS-SC service

O1

Administration-ROSS-SC service

coordinator

coordinator hired

2

75

2.7%

P2

Childcare-Families referred to

O2

Childcare-Families obtain childcare

childcare services

services

2

64

3.1%

P3

Education-Adult Basic

O3

Education-Adult Basic

Education/Literacy classes-Enrolled

Education/Literacy class-Completed

2

59

3.4%

P4

Education-High school/GED program-

O4

Education-High school diploma/GED

Enrolled

obtained

1

60

1.7%

P5

Employment-Job skil s assessment

O5

Employment-Job skills determined

2

68

2.9%

P6

Financial Literacy-Financial

O6

Financial Literacy-Financial

literacy/management classes-

literacy/management class-Completed

Enrolled

3

69

4.3%

P7

Financial Literacy-Individual

O7

Financial Literacy-Individual

Counseling-Enrolled

Counseling-Completed

1

68

1.5%

P8

Outreach-Service providers contacted O8

Outreach-Providers make agreements

with agency to provide services

2

87

2.3%

P9

Seniors/Disabled-Service

O9

Seniors/Disabled-Improved living

Coordination-Referrals for

conditions/quality of life

senior/disability services

10

66

15.2%

P10

Seniors/Disabled-Service

O10

Seniors/Disabled-Live

Coordination-Referrals for

independently/age in place and avoid

senior/disability services

long-term care placement

17

62

27.4%

P11

Seniors/Disabled-Service

O11

Seniors/Disabled-Service

Coordination-Referrals for

Coordination-Seniors/disabled obtain

senior/disability services

needed services

10

66

15.2%

P12

Service Coordination-New

O12

Service Coordination-New Participants

Participants Enrol ed-Individual

Enrolled-Individual Training Service

Training Service Plans (ITSPs) begun

Plans (ITSPs) completed

3

80

3.8%

P13

Service Coordination-Non-ITSP

O13

Service Coordination-Non-ITSP

individuals seeking services

individuals served (unduplicated

(unduplicated count)

count)

7

76

9.2%

P14

Service Coordination-Participants

O14

Service Coordination-Services no

with ITSPs continuing to receive

longer needed-ITSP goals achieved-

services-Year 1

Year 1

0

75

0.0%

2 3 5

Optional

PA1

Education-Postsecondary classes-

OA1

Education-Postsecondary classes-

Enrolled

Completed

2

33

6.1%

PA2

Financial Literacy-Banking, credit and

OA2

Financial Literacy-Checking/Savings

IDA information provided

account established

3

31

9.7%

OA3

Financial Literacy-Credit score

improved

0

28

0.0%

OA4

Financial Literacy-IDA account

established

0

28

0.0%

PA3

Outreach-Outreach to individuals

OA6

Outreach-Non-elderly/non-disabled

(NON-ELDERLY-NON-DISABLED)

residents enrolled

0

43

0.0%

PA4

Outreach-Outreach to

OA5

Outreach-Eligible seniors/persons with

seniors/persons with disabilities

disabilities enrol ed in ROSS-SC

program

3

37

8.1%

PA5

Policy Priority 1-Healthcare

OA7

Policy Priority 1-Healthcare

Assessment-Health insurance

Assessments-Residents obtain

assessment conducted

insurance

1

39

2.6%

PA6

Policy Priority 1-Healthcare

OA8

Policy Priority 1-Healthcare Assistance-

Assistance-Referral for healthcare

Healthcare services obtained

services

5

57

8.8%

PA7

Policy Priority 1-Partnership with

OA9

Policy Priority 1-Partnership with

Healthcare Organization-Outreach to

Healthcare Organizations-Medical

Healthcare Providers

home* established for residents

6

51

11.8%

OA10 Policy Priority 1-Partnerships with

Healthcare Organizations-Partnership

established with healthcare providers

6

54

11.1%

PA8

Policy Priority 1-Target High-Need

OA11 Policy Priority 1-Target High-Need

Clients-Baseline needs identified for

Clients-Most vulnerable residents

most vulnerable residents

linked to services addressing their

baseline health needs

11

34

32.4%

PA9

Policy Priority 1-Target High-Need

OA11 Policy Priority 1-Target High-Need

Clients-Most vulnerable residents

Clients-Most vulnerable residents

identified

linked to services addressing their

baseline health needs

17

35

48.6%

PA10 Policy Priority 2-Promote Economic

OA13 Policy Priority 2-Promote Economic

Development-Job training classes-

Development-Job training classes-

Enrolled

Completed

2

54

3.7%

OA16 Policy Priority 2-Promote Economic

Development-Participants’ job skil s

increase

9

55

16.4%

PA11 Policy Priority 2-Promote Economic

OA12 Policy Priority 2-Promote Economic

Development-Participants referred to

Development-Earned income

employers

increases

10

51

19.6%

OA14 Policy Priority 2-Promote Economic

Development-Participants obtain part-

time employment

3

49

6.1%

OA15 Policy Priority 2-Promote Economic

Development-Participants obtain full-

time employment

5

48

10.4%

PA12 Seniors/Disabled-Food and Nutrition-

OA17 Seniors/Disabled-Food and Nutrition-

Congregate Meals coordinated

Participants have adequate nutrition

3

18

16.7%

2 3 6

PA13 Training-Parenting/Household

OA18 Training-Parenting/Household

Skil s/Life skil s classes-Enrolled

Skil s/Life skil s classes-Completed

3

41

7.3%

PA14 Transportation-Referrals to

OA19 Transportation-Employment obtained

transportation services to enable

as a result of transportation

employment

0

23

0.0%

PA15 Transportation-Referrals to

OA20 Transportation-Service(s) obtained as

transportation services to enable

a result of transportation

service provision

4

38

10.5%

2 3 7

Appendix M. Application-Logic

Model Crosswalk

Application Needs

ROSS Services/Activites on the FY 2014 Logic Model

Life Skills Training

Training–Parenting/Household skills/Life skills classes

Financial Literacy/Credit Counseling/Credit Repair

Financial Literacy–Financial literacy/management classes

Financial Literacy–Individual counseling

Literacy Training

Education–Adult Basic Education/Literacy classes

ESL

No match

GED/High School Equivalency

Education–High school/GED program

Mentoring

No match

Job Soft Skills Training

Policy Priority 2–Promote Economic Development–Job training classes

Job Hard Skills Training/Certification

Policy Priority 2–Promote Economic Development–Job training classes

Job Search and Placement

Employment–Job skills assessment

Policy Priority 2–Promote Economic Development–Participants referred to employers Job Retention/Promotion

No match

Individual Savings Accounts (ISAs)/Individual Development Accounts (IDAs)

Financial Literacy–Banking, credit and IDA information provided

Homeownership Counseling

No match

Computer Classes

No match

Drug/Alcohol Treatment

Policy Priority 1–Health Care Assistance–Referral for health care services

Mental Health Treatment

Policy Priority 1–Health Care Assistance–Referral for health care services

Health/Dental Care

Policy Priority 1–Health Care Assistance–Referral for health care services

Policy Priority 1–Health Care Assessment–Health insurance assessment conducted

Policy Priority 1–Partnership with Health Care Organization–Outreach to Health Care Providers Home Maintenance Classes

Training–Parenting/Household skills/Life skills classes

Parenting Classes

Training–Parenting/Household skills/Life skills classes

Nutrition Classes

No match

Youth Programming–tutoring/mentoring/after school/summer

No match

Child Care

Child Care–Families referred to child care services

Tax Preparation Assistance

No match

Community Safety

No match

Resident Empowerment/Capacity Building

No match

Resident Business Development

No match

Assistance with Activities of Daily Living

Seniors/Disabled–Referrals for senior/disability services

Meals to Meet Nutritional Need for Elderly

Seniors/Disabled–Food and Nutrition–Congregate Meals coordinated

Disability Services Counseling

Seniors/Disabled–Referrals for senior/disability services

Personal Emergency Response Resources

No match

Wellness Programs

No match

No match

Administration–Hire ROSS Service Coordinator

No match

Outreach–Service providers contacted

No match

Service coordination–Individual Training Service Plans (ITSPs) begun

No match

Service coordination–Non-ITSP individuals seeking services

No match

Service coordination–Participants with ITSPs continuing to receive services

No match

Education–Post Secondary Classes

No match

Outreach–Outreach to individuals (non-elderly, non-disabled)

No match

Outreach–Outreach to seniors/persons with disabilities

No match

Policy Priority 1–Target High-Need Clients–Baseline needs identified for most vulnerable No match

Policy Priority 1–Target High-Need Clients–Most vulnerable residents identified

No match

Transportation–Referrals to transportation services to enable employment

No match

Transportation–Referrals to transportation services to enable service provision

2 3 8

References

American Association of Service Coordinators (AASC). n.d. “History of Service Coordination.”

https://www.servicecoordinator.org/page/SCHistory/History-of-Service-Coordination.htm.

Burnstein, Eric, Megan Gal agher, and Wilton Oliver. 2019. Economic Mobility Services for Affordable Housing Residents: Exploring Resident Services as a Vehicle for Economic Success. Washington, DC: Urban Institute.

https://www.urban.org/research/publication/economic-mobility-services-affordable-housing-residents.

Chavis, Judith. 2018. HUD-Funded Service Coordination Programs: ROSS, Family Self-Sufficiency, and Service Coordinators in Multifamily Housing for Elderly and Disabled. Washington, DC: National Low-Income Housing Coalition. .

ExpectMore.gov. n.d. “Detailed Information on the Resident Opportunities and Self Sufficiency Assessment.”

https://obamawhitehouse.archives.gov/sites/default/files/omb/assets/omb/expectmore/detail/10009091.20

08.html.

Falkenburger, Elsa, Sarah Haight, and Susan J. Popkin. 2018. Place Matters: A Two-Generation Approach to Housing.

Washington, DC: Urban Institute.

Federal Register Notice. “Funding Availability for Public Housing Resident Opportunities and Self Sufficiency (ROSS) Program,” Federal Register 67, no. 58 (March 26, 2002); 14207.

Kandilov, Amy, Vincent Keyes, Noel e Siegfried, Kevin Smith, Patrick Edwards, Jenna Brophy, Aubrey Col ins, Ann Larsen, and Martijn Van Hasselt at RTI International and Alisha Sanders and Robyn Stone at LeadingAge Center for Applied Research. 2017. “Support and Services at Home (SASH) Evaluation: Evaluation of the First Four Years.” Washington, DC: U.S. Department of Health and Human Services.

https://aspe.hhs.gov/system/files/pdf/257936/SASH4.pdf.

Levine, Cheryl, and Ashaki Robinson Johns. 2008. Multifamily Property Managers’ Satisfaction with Service Coordination. Washington, DC: U.S. Department of Housing and Urban Development Office of Policy Development and Research. https://www.huduser.gov/Publications/PDF/Multifamily_prop.pdf.

McNickle, Larry. 2007. “Service Coordinators: An Essential Link for Seniors to Age in Place,” Journal of Housing and Community Development 64 (4): 40–47.

https://ogg.osu.edu/media/documents/firehouse/SCEssentialLink_McNickle_25954214.pdf.

New York City Housing Authority (NYCHA). “Executive Summary: NYCHA’s Final Agency Plan for FY 2015.”

https://www1.nyc.gov/assets/nycha/downloads/pdf/fy2015-nycha-agency-plan-exec-summary-en.pdf.

Popkin, Susan J., Elsa Falkenburger, and Sarah Haight. 2018. Place Matters: A Two-Generation Approach to Housing.

Washington, DC: Ascend at the Aspen Institute. https://ascend.aspeninstitute.org/resources/place-matters-a-

two-generation-approach-to-housing/.

Popkin, Susan J., Mol y M. Scott, Joe Parilla, Elsa Falkenburger, Marla McDaniel, and Shinwon Kyung. 2012.

Planning the Housing Opportunity and Services Together Demonstration: Challenges and Lessons Learned.

Washington, DC: Urban Institute.

Theodos, Brett, Christina Plerhoples Stacy, and Rebecca Daniels. 2018. “Client Led Coaching: A Random Assignment Evaluation of the Impacts of Financial Coaching Programs.” Washington, DC: Urban Institute.

U.S. Department of Education (DOE). “Frequently Asked Questions About FERPA.”

https://www2.ed.gov/policy/gen/guid/fpco/pdf/ferpafaq.pdf.

U.S. Department of Housing and Urban Development (HUD). n.d.a. “About the Resident Opportunities and Self Sufficiency (ROSS) Grant Program.”

https://www.hud.gov/program_offices/public_indian_housing/programs/ph/ross/about.

———. n.d.b. “Budget Line Items for the Consolidated Economic Development and Supportive Services and Tenant Opportunities Program,” https://www.hud.gov/sites/documents/DOC_11245.PDF.

———. n.d.c. “Economic Development and Supportive Services Program (ED/SS).”

https://www.hud.gov/programdescription/ed-ss.

———. n.d.d. “HUD Standards for Success Data Integrity Frequently Asked Questions (FAQs).”

https://www.hudexchange.info/resources/documents/Standards-for-Success-Data-Integrity-Frequently-

Asked-Questions.pdf.

———. n.d.e. “Multifamily Housing – Service Coordinator Program.”

https://www.hud.gov/program_offices/housing/mfh/scp/scphome.

———. n.d.f. “Running ROSS Step-by-Step.” https://www.hudexchange.info/programs/ross/guide/introduction.

———. n.d.g. “Standards for Success.” https://www.hudexchange.info/programs/standards-for-success.

———. n.d.h. “Standards for Success Data Integrity Reference Manual.”

https://www.hudexchange.info/resources/documents/Standards-for-Success-Data-Integrity-Reference-

Manual.pdf.

———. 1998. “Processing of Grant Applications for Fiscal Year (FY)1998 SuperNOFA Public Housing Tenant Opportunities Program (TOP). https://www.hud.gov/sites/documents/98-42pihn.doc.

———. 2000. “Success in the New Welfare Environment: An Assessment of Approaches in HUD’s Employment and Training Initiatives.” https://mefassociates.com/wordpress/wp-content/uploads/2015/07/NewWelfare-

HUD%E2%80%99s-Employment-and-Training-Initiatives.pdf.

———. 2002. “Resident Opportunities and Self-Sufficiency (ROSS) Program,”

https://archives.hud.gov/funding/2002/nofa02/rosssec.pdf.

———. 2011. “2011 Public Housing FSS, ROSS-SC, Multifamily Housing SC.”

https://archives.hud.gov/news/2011/FSSRSCPHACHART.pdf.

———. 2012. “Public and Indian Housing Public Housing Capital Fund 2012 Summary Statement and Initiatives.”

https://www.hud.gov/sites/documents/ph_capital_fund_2012.pdf.

———. 2014. “FY 2014 Notice of Funding Availability: ROSS-SC.”

https://www.hud.gov/sites/documents/2014ROSS-SCNOFA.pdf.

———. 2015. “FY 2015 Notice of Funding Availability: ROSS-SC.”

https://www.hud.gov/sites/documents/2015ROSS-SC-NOFA.pdf.

———. 2017. “FY 2017 Notice of Funding Availability: ROSS-SC.”

https://www.hud.gov/sites/dfiles/SPM/documents/ROSS_FR-6100-N-05.pdf.

———. 2018. “FY 2018 Notice of Funding Availability: ROSS-SC.”

https://www.hud.gov/sites/dfiles/SPM/documents/FY%2018%20ROSS%20NOFA%20-%20FR-6200-N-

05.pdf.

U.S. Department of Housing and Urban Development Office of Policy Development and Research (HUD PD&R).

2000. “Success in the New Welfare Environment: An Assessment of Approaches in HUD’s Employment and Training Initiatives. ” https://mefassociates.com/wordpress/wp-content/uploads/2015/07/NewWelfare-

HUD%E2%80%99s-Employment-and-Training-Initiatives.pdf.

———. 2003. “Resident Opportunities and Self-Sufficiency (ROSS) Program.”

https://archives.hud.gov/funding/2003/grpross.cfm.

U.S. Department of Labor Bureau of Labor Statistics (BLS). n.d. “Occupational Outlook Handbook, Social Workers” https://www.bls.gov/ooh/community-and-social-service/social-workers.htm.

U.S. Government Accountability Office (GAO). 2013. HUD Data on Self-Sufficiency Programs Should Be Improved.

Washington, DC: GAO. https://www.gao.gov/assets/660/655798.pdf.

White, Donna. 2009. “HUD awards nearly $28 mil ion to public housing agencies across the country to promote self-sufficiency.” U.S. Department of Housing and Urban Development.

https://archives.hud.gov/news/2009/pr09-037.cfm.

2 4 0

———. 2010. “HUD awards nearly $28 mil ion to public housing agencies across the country to promote self-sufficiency.” U.S. Department of Housing and Urban Development.

https://archives.hud.gov/news/2009/pr10-097.cfm.

Additional Reading

Arena, Olivia and Mary Bogle. 2019. “Bangor, Maine, Residents Show the Importance of Supporting Each Other’s Economic Goals,” Urban Wire (blog), Urban Institute. https://www.urban.org/urban-wire/bangor-maine-

residents-show-importance-supporting-each-others-economic-goals.

Bloom, Howard, James Riccio, and Nandita Verma. 2005. Promoting Work in Public Housing: The Effectiveness of Jobs-Plus. Final Report. New York: MDRC. https://www.mdrc.org/sites/default/files/ful _485.pdf.

Geyer, Judy, Lesley Freiman, Jeffrey Lubell, and Micah Vil arreal. 2017. Evaluation of the Compass Family Self-Sufficiency (FSS) Programs Administered in Partnership with Public Housing Agencies in Lynn and Cambridge, Massachusetts. Bethesda, MD: Abt Associates.

Gillespie, Sarah, and Susan J. Popkin. 2015. “Building Public Housing Authority Capacity for Better Resident Services: Lessons from HOST.” Washington, DC: Urban Institute.

https://www.urban.org/sites/default/files/publication/65441/2000333-Building-Public-Housing-Authority-

Capacity-for-Better-Resident-Services.pdf.

“HUD Supportive Services Demonstration/Integrated Wel ness in Supportive Housing: Privacy Act of 1974; System of Records,” Federal Register 83 (32) February 15, 2018: 6875.

Popkin, Susan J. 2018. “The Limits of Housing Revitalization as a Platform for Improving Residents’ Economic Well-being: Lessons from the U.S.,” International Journal of Urban Sciences 22 (4): 461–472.

https://www.tandfonline.com/doi/ful /10.1080/12265934.2018.1455531.

Proscio, Tony. 2006. More Than Roof and Walls: Why Resident Services Are an Indispensable Part of Affordable Housing. Columbia, MD: Enterprise Community Partners.

Riccio, James A. 2010. “Sustained Earnings Gains for Residents in a Public Housing Jobs Program: Seven-Year Findings from the Jobs-Plus Demonstration.” New York: MDRC.

https://www.mdrc.org/sites/default/files/policybrief_33.pdf.

Scal y, Corianne Payton, Samantha Batko, Susan J. Popkin, and Nicole DuBois. 2018. The Case for More, Not Less: Shortfalls in Federal Housing Assistance and Gaps in Evidence for Proposed Policy Changes. Washington, DC: Urban Institute.

Scott, Molly, Susan J. Popkin, and Priya Saxena. 2016a. Making a Two-Generation Model Work in the Real World: Lessons from the HOST Demonstration. Washington, DC: Urban Institute.

www.urban.org/research/publication/making-two-generation-model-work-real-world.

Scott, Molly M., Susan J. Popkin, and Jasmine Simington. 2016b. A Theoretical Framework for Two-Generation Models: Lessons from the HOST Demonstration. Washington, DC: Urban Institute.

www.urban.org/research/publication/theoretical-framework-two-generation-models.

Theodos, Brett, Susan J. Popkin, Joe Parilla, and Liza Getsinger. 2012. “The Challenge of Targeting Services: A Typology of Public-Housing Residents,” Social Service Review 86 (3): 517–44.

2 4 2

U.S. Department of Housing and Urban Development

Office of Policy Development and Research

Washington, DC 20410-6000

May 2021

Document Outline

- Evaluation of the Resident Opportunity and Self-Sufficiency – Service Coordinator Program

- Foreword

- Contents

- List of Exhibits

- Preface

- Acknowledgments

- Executive Summary

- Key Findings

- Implications and Recommendations for HUD

- Chapter 1. Introduction

- Overview of the Resident Opportunity and Self-Sufficiency Program

- Reason for Assessment

- Research Goals

- Report Overview

- Chapter 2. Methodology

- Introduction

- Site Visits

- Survey of ROSS-SC Service Coordinators

- Administrative Data Analysis

- Chapter 3. Residents and Services

- Introduction

- Key Findings

- Targeting and Engaging Residents

- Aligning Services to Meet Resident Needs

- Relationships with Partners

- How ROSS-SC Enhances Public Housing Operations

- Conclusion

- Chapter 4: Performance Monitoring and Reporting

- Introduction

- Key Findings

- Logic Model Structure and Processes

- Assessing Logic Model Changes

- Conclusion

- Chapter 5. Service Coordinators: Work Responsibilities, Qualifications, and Supports

- Introduction

- Key Findings

- Core Work Responsibilities

- Service Coordinator Qualifications

- Work Supports and Training

- Conclusion

- Chapter 6. Resident Experiences and Local Contexts

- Introduction

- Key Findings

- Resident Experiences

- Conclusion

- Chapter 7: Study Implications and Recommendations

- Introduction

- Establishing a Theory of Change

- Defining and Measuring Grantee Performance

- Improving Peer Learning and Communications

- Filling Service Gaps and Facilitating Partnerships

- Meeting Resident Needs

- Sustaining ROSS-SC

- Appendix A. Sample Individual Training and Services Plan (ITSP) 40. 40. U.S. Department of Housing and Urban Development. “Running ROSS Step by Step: Sample Individual Training and Services Plan.” https://files.hudexchange.info/resources/documents/ROSS-Step-by-Step-Individual-Training-and-Services-Plan.pdf.

- Appendix B. Interview Protocols

- Informed Consent

- Service Coordinator

- Grantee Supervisor/Manager

- Partner

- Appendix C. Focus Group Protocols

- Informed Consent

- Resident Focus Group Guide

- Appendix D. Service Coordinator Survey

- Appendix E. Service Coordinator Survey Weights

- Appendix F. Question Variations Between Fielded Versus OMB-Approved Service Coordinator Survey

- Appendix G. ROSS-SC Service Coordinators—Needs and Service Partners (HUD Form 52769)

- Appendix H. FY 2014-2016 Logic Model

- Appendix I. FY 2014 Application-Logic Model Service Classification

- Appendix J. Guidance for Completing the 2014 ROSS-SC Logic Model

- Appendix K. 2011-2014 Crosswalk

- Appendix L. FY 2014 Outputs Versus Outcomes by Metric

- Appendix M. Application-Logic Model Crosswalk

- References

- Additional Reading

Table of Contents

Evaluation of the Resident Opportunity and Self-Sufficiency – Service Coordinator Program

Foreword

Contents

List of Exhibits

Preface

Acknowledgments

Executive Summary

Key Findings

Implications and Recommendations for HUD

Chapter 1. Introduction

Overview of the Resident Opportunity and Self-Sufficiency Program

Reason for Assessment

Research Goals

Report Overview

Chapter 2. Methodology

Introduction

Site Visits

Survey of ROSS-SC Service Coordinators

Administrative Data Analysis

Chapter 3. Residents and Services

Introduction

Key Findings

Targeting and Engaging Residents

Aligning Services to Meet Resident Needs

Relationships with Partners

How ROSS-SC Enhances Public Housing Operations

Conclusion

Chapter 4: Performance Monitoring and Reporting

Introduction

Key Findings

Logic Model Structure and Processes

Assessing Logic Model Changes

Conclusion

Chapter 5. Service Coordinators: Work Responsibilities, Qualifications, and Supports

Introduction

Key Findings

Core Work Responsibilities

Service Coordinator Qualifications

Work Supports and Training

Conclusion

Chapter 6. Resident Experiences and Local Contexts

Introduction

Key Findings

Resident Experiences

Conclusion

Chapter 7: Study Implications and Recommendations

Introduction

Establishing a Theory of Change

Defining and Measuring Grantee Performance

Improving Peer Learning and Communications

Filling Service Gaps and Facilitating Partnerships

Meeting Resident Needs

Sustaining ROSS-SC

Appendix A. Sample Individual Training and Services Plan (ITSP) 40. 40. U.S. Department of Housing and Urban Development. “Running ROSS Step by Step: Sample Individual Training and Services Plan.” https://files.hudexchange.info/resources/documents/ROSS-Step-by-Step-Individual-Training-and-Services-Plan.pdf.

Appendix B. Interview Protocols

Informed Consent

Service Coordinator

Grantee Supervisor/Manager

Partner

Appendix C. Focus Group Protocols

Informed Consent

Resident Focus Group Guide

Appendix D. Service Coordinator Survey

Appendix E. Service Coordinator Survey Weights

Appendix F. Question Variations Between Fielded Versus OMB-Approved Service Coordinator Survey

Appendix G. ROSS-SC Service Coordinators—Needs and Service Partners (HUD Form 52769)

Appendix H. FY 2014-2016 Logic Model

Appendix I. FY 2014 Application-Logic Model Service Classification

Appendix J. Guidance for Completing the 2014 ROSS-SC Logic Model

Appendix K. 2011-2014 Crosswalk

Appendix L. FY 2014 Outputs Versus Outcomes by Metric

Appendix M. Application-Logic Model Crosswalk

References

Additional Reading