Evaluation of the HUD-DOJ

Pay for Success Permanent

Supportive Housing

Demonstration

Year 3 Report

U.S. Department of Housing and Urban Development | Office of Policy Development and Research

Evaluation of the HUD-DOJ Pay

for Success Permanent

Supportive Housing

Demonstration

Year 3 Report

Kelly Walsh

Nicole DuBois

Samantha Batko

Clare Salerno

Eleanor Noble

Akiva Liberman

Mary Cunningham

Urban Institute

May 2020

Foreword

In 2020, the U.S. Department of Housing and Urban Development (HUD) estimated about 580,000

people experience homelessness on any given night, including just over 110,000 individuals who are chronically homeless. Research shows a strong link between homelessness and incarceration.

Individuals sleeping on the street are frequently cited for public-nuisance offenses such as public intoxication, panhandling, and trespassing. As they are arrested and cycle in and out of jail, homelessness becomes a law enforcement issue.

In 2016, HUD and the Department of Justice (DOJ) formed an interagency collaboration that combines DOJ’s mission to promote safer communities by focusing on the reentry population with HUD’s mission to end chronic homelessness. HUD-DOJ launched the Pay for Success (PFS) Permanent Supportive Housing (PSH) demonstration through a Notice of Funding Availability, which awarded just over $8.6 million to seven grantees and funded three phases of PFS projects: feasibility analysis, transaction structuring, and implementation—which includes evaluations and success payments. HUD

is conducting a formative evaluation of this demonstration to assess whether providing PSH within a PFS framework is a successful and cost-effective way of using PSH to provide housing stability and reduce social service use and recidivism for a population continually cycling between homeless services and the criminal justice system.

This Year 3 report summarizes the grantees’ progress in the second year of the demonstration from January through December 2019, focusing on the transaction structuring phase and the requirements, challenges, and accomplishments as most sites progressed toward implementation.

While it is too early to tell if PFS is successful, the fact that the sites are still progressing toward implementation is a testament to the early success of the PFS framework. The PFS model has catalyzed partnerships and collaborations across sectors while shifting governments’ programmatic focus from outputs to outcomes. Future reports will explore the impact of the outcome-based success metrics on project evaluations and evaluations of the overall success of the PFS framework to provide PSH for this target population. This current study builds knowledge around using this innovative financing model to fund PSH projects for high users of social services who also cycle in and out of the criminal justice system.

Todd Richardson

General Deputy Assistant Secretary for Policy Development and Research

U.S. Department of Housing and Urban Development

F O R E W O R D

i

Contents

Acknowledgments

v

Executive Summary

vi

Background

vi

Evaluation

vii

Overview of 2019 Site Progress

viii

Transaction Structuring Cross-Site Themes

x

Implementation Cross-Site Themes

xii

Benefits of Pay for Success

xiii

Time Costs of Pay for Success

xiv

I. Introduction

1

The HUD-DOJ Pay for Success Permanent Supportive

Housing Demonstration

3

Defining the Problem

3

Permanent Supportive Housing

4

The Pay for Success Model

5

The Demonstration Theory of Change

7

The Demonstration Evaluation in Year 3

9

Process Study Methodology

10

Annual Partnership Survey Methodology

11

Time Use Survey Methodology

15

II. Demonstration Site Overview and Progress in 2019

18

Transaction Structuring Sites without Pilots

21

Montgomery County and Prince George’s County, Maryland

21

Rhode Island

23

Transaction Structuring Sites with Launched Pilots

25

Anchorage, Alaska

26

Austin/Travis County, Texas

29

Implementation Sites

31

Lane County, Oregon

32

Los Angeles County, California

34

III. Transaction Structuring Cross-Site Themes

38

Sites Employed Different Project Management Structures to Suit Current Needs

38

Sites Engaged Additional Technical Assistance Providers for Project Support

39

Feasibility and Transaction Structuring Phases Blended Together

40

C O N T E N T S

i i

Community Landscape and Stakeholders Shaped Project Design

43

Transaction Structuring Challenges

45

Finding and Solidifying Financial Partners

45

Securing Housing Resources

50

Designing the Evaluation

51

Pilots Represented an Opportunity to Refine Program Model and Demonstrate Effectiveness

52

Pilots Helped Refine Program Design and Operations

53

Pilots Helped Demonstrate Ability to Implement

54

IV. Implementation Cross-Site Themes

55

Sites Added Coordination Mechanisms for Stakeholders involved in Permanent

Supportive Housing Implementation

55

Sites Reported Successful Lease-Ups and High Rates of Retention

55

Implementation Challenges

56

Building Capacity to Serve “High-Need” Participants

56

Building Capacity in New Service Providers

57

Finding Housing and Developing Relationships with Landlords

57

Blending Service Provider and Criminal Justice Roles in Permanent Supportive

Housing Implementation

58

Classifying Participant Exits

59

V. Benefits of Pay for Success

61

Pay for Success Strengthened Partnerships and Collaboration

61

Breaking Down Silos between Sectors

61

Raising Awareness of the Target Population and Possible Cross-Sector Solutions

62

Strengthening Local Data Capacity and Cross-Sector Sharing

64

Increasing Local Capacity through External Technical Assistance Providers

66

Implementation and Pilots Increased Capacity to Serve Target Population

66

VI. Time Costs of Pay for Success

68

Time Spent on the Project Varied by Type of Stakeholder

68

Total Time Spent Varied by Phase and by Site

71

VII. Conclusion

74

Appendix A. Example Site Visit Stakeholder List

77

Appendix B. Detailed Partnership Survey Methodology

78

Appendix C. Evaluation Methodology Limitations

82

Appendix D. Partnership Survey Instrument

84

C O N T E N T S

i i i

References

103

i v

C O N T E N T S

Acknowledgments

This report was funded by the U.S. Department of Housing and Urban Development and the U.S.

Department of Justice. We are grateful to them and all our funders who make it possible for the Urban Institute to advance its mission.

The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders. Funders do not determine research findings or the insights and recommendations of Urban experts. Further information on the Urban Institute’s funding principles is available at urban.org/fundingprinciples.

Thanks to Matthew Eldridge and Lauren Farrell, who contributed to this work, and to Marina Myhre, Susan Anderson, and Andre Bethea, who provided comments on this report. We also wish to thank the grantees for making time to participate in the evaluation.

A C K N O W L E D G M E N T S

v

Executive Summary

In 2016, the U.S. Department of Housing and Urban Development (HUD) and the U.S. Department of Justice (DOJ) launched the Pay for Success (PFS) Permanent Supportive Housing (PSH) demonstration (the demonstration), funding seven sites across the country. The goal of the demonstration is to strengthen communities’ abilities to reduce recidivism and homelessness among the reentry population by increasing PSH through PFS, a financing mechanism that aims to unlock private and philanthropic investment in social programs, build the evidence base on effective programs, and ensure public funds pay for outcomes. The Urban Institute (Urban) evaluates this work to determine whether PFS can be used to scale PSH to end chronic homelessness. By the end of 2019, two sites had launched a PSH

program through PFS, four were working on structuring the PFS transaction (two of which were operating pilots), and one had withdrawn from the demonstration, deciding to fund PSH without PFS.

This report describes the challenges sites encountered and milestones they reached in the third year of the demonstration.

Background

PSH is an evidence-based program model that improves housing stability through intensive supportive services and housing assistance. In the demonstration, PSH is intended to reduce recidivism and homelessness among people reentering the community after being in the criminal justice system. These people may cycle in and out of other crisis services, including hospital emergency departments and psychiatric and detoxification centers, as well as jails. Homelessness and cycling are detrimental to people’s health and well-being and come at a high cost to the public. Despite strong evidence supporting its effectiveness, PSH is not available at the scale many communities need and is not routinely part of jail diversion or prison reentry efforts.

The demonstration uses the PFS model to expand PSH programming. PFS planning requires creating new or leveraging existing partnerships among government agencies, social service providers, and private and philanthropic investors. Under PFS, investors pay the up-front costs of PSH and are repaid by an end payor—usually a state or local government entity—if the program succeeds.

Demonstration sites define success as positive housing and justice outcomes like reductions in rearrests or reconvictions or program retention and housing stability. HUD-DOJ defined three phases of PFS projects: feasibility analysis, transaction structuring, and implementation. The implementation phase includes evaluation and success payments.

v i

E X E C U T I V E S U M M A R Y

In October 2015, HUD issued a Notice of Funding Availability (NOFA) for the demonstration. HUD

and DOJ evaluated applications from across the country and ultimately awarded $8.7 million to seven demonstration sites: Anchorage and Matanuska-Susitna Borough, Alaska; Austin/Travis County, Texas; Lane County, Oregon; Los Angeles County, California; Montgomery County and Prince George’s County,1 Maryland; Pima County, Arizona; and Rhode Island.

This report summarizes grantee progress in Year 3 of the demonstration (January through December 2019). It focuses on milestones and challenges, the time necessary to participate in PFS by core partners, and the partners’ perceptions of the benefits and trade-offs of PFS planning and implementation. Throughout 2019, six sites were engaged in the demonstration.

Evaluation

Urban is conducting a national evaluation of the HUD-DOJ PFS PSH demonstration. The goals of the Urban evaluation are to examine how PFS is implemented in each site, capture lessons learned across sites, and examine the benefits of using PFS to fund PSH for a reentry population. The national evaluation is not intended to replace the local evaluations that determine PFS success payments. Our qualitative, formative evaluation will describe the PFS projects that develop in each site. Our research design seeks to document the reality on the ground as each community began its PFS project and, in hindsight, how that may have affected the project as it moved through each stage. Three objectives drive the Urban evaluation strategy :

◼

Document the progress and processes of partners.

◼

Document time use and costs incurred by partners.

◼

Document the benefits of the PFS model to the target population.

This report details the evaluation data collection, analysis, and findings for Year 3 of the demonstration. This documentation includes information on the site progress, milestones met, challenges faced, site partner perceptions of the benefits of the project, and estimates of the time spent by the project partners on PFS planning and implementation activities in 2019. Previous evaluation publications are in exhibit 1.

1 The grant originally included both Prince George’s County and neighboring Montgomery County, but Montgomery County left the demonstration in September 2019. The project is now a single county effort in Prince George’s County.

E X E C U T I V E S U M M A R Y

v i i

EXHIBIT 1

Demonstration Evaluation Publications

Evaluation of the HUD-DOJ Pay for Success Permanent Supportive Housing Demonstration:

Baseline Report. Urban describes the accomplishments and challenges of the seven projects during the first period of the demonstration (through December 2017) in a baseline report focusing on the feasibility analysis phase of PFS.

Data Use and Challenges in Using Pay for Success to Implement Permanent Supportive Housing:

Lessons from the HUD-DOJ Demonstration. Urban describes use of data at demonstration sites in all PFS phases and some common challenges accessing high-quality, usable data across homelessness, healthcare, and criminal justice sectors.

Evaluation of the HUD-DOJ Pay for Success Permanent Supportive Housing Demonstration: Year 2

Report. Urban describes the progress and challenges of the remaining six projects in 2018. The report focuses on the process of transaction structuring and implementation.

Demonstration infographic. An at-a-glance dashboard of the demonstration project design and milestones.

Overview of 2019 Site Progress

In 2019, the six demonstration sites made progress and encountered challenges. Each site has been actively involved in the demonstration since October 2016, although they entered in different phases of work. By the end of 2019, Lane County and Los Angeles County had implemented programs, and the remaining sites had continued transaction structuring work.

Sites in Transaction Structuring Without Pilots

Montgomery County and Prince George’s County, Maryland, entered the demonstration as one site with a plan to launch a joint project. By mid- to late-2019, it became clear that the needs of the two counties’ diverged; because Montgomery County reports they effectively ended chronic homelessness, they no longer saw a need for more PSH, and they left the demonstration before the end of 2019. Prince George’s County continued work on evaluation design planning, investor outreach strategy, and service provider procurement. Challenges included educating newly elected officials and the level of effort necessary to finalize the evaluation design. Rhode Island made progress on data-sharing agreements, linking, and analysis that revealed important information on its target population’s Medicaid use. In v i i i

E X E C U T I V E S U M M A R Y

Rhode Island, the principle challenge was securing the state as the end payor for the project. Although the governor backed the project, the state legislature declined to include funding for the project in the annual state budget.

Sites in Transaction Structuring with Launched Pilots

Anchorage and Austin/Travis County continued transaction structuring work while operating pilot programs of the proposed PFS-funded PSH interventions. Site stakeholders felt that piloting a program before launching with the PFS contract (i.e., investor dollars, end payor commitment, and evaluation) provided an opportunity to test assumptions about enrollment pathways and rates and to find and fix unexpected challenges before investor dollars are at risk. Anchorage began enrolling participants in a pilot in July 2019, placing 17 people in housing before the end of the year. While the Municipality of Anchorage, the primary potential end payor, continued to support the project, efforts to secure funding for that payment or get buy-in at the state level were not yet successful. Austin/Travis County completed enrollment of 25 people in the project pilot in the spring of 2019 and continued to serve the people enrolled in the pilot. The site faced challenges following leadership changes at the intermediary organization. Despite this, the site identified its evaluator and secured and maintained commitments from multiple end payors.

Sites Implementing Permanent Supportive Housing Programming with Pay For Success

Los Angeles County and Lane County launched their PFS-funded PSH programs in 2017 and 2018, respectively. Los Angeles County is using PFS to expand an existing jail diversion program. At the end of September 2019, it ended enrollment as planned after 2 years and enrolling over 300 participants. The site continued monitoring participant housing status and issued $2,264,900 in success payments to investors in 2019 based on observed participant outcomes. Lane County was in the early stages of implementation at the beginning of 2019. By the end of the year, the project housed almost one-half of its target number and prepared its first outcome payment report. Additional capital is needed to continue to house the new clients necessary to meet the overall goal of the site. Efforts to secure those resources were not successful by the end of 2019, but partners remain committed to finding a path forward.

Exhibit 2 illustrates the enrollment and retention progress made in 2019 for sites in the implementation phase and sites with pilots.

E X E C U T I V E S U M M A R Y

i x

EXHIBIT 2

Demonstration Site Enrollment Tracker

Total eligible

Total

Total ever

individuals

individuals

housing

Site

Phase

referred

ever housed

exits

Anchorage, AK

Transaction structuring

33

17

0

Austin/Travis County, TX

Transaction structuring

29

29

4

Lane County, OR

Implementation

123

62

9

Los Angeles County, CA

Implementation

n/a

349

95

Notes: Lane County does not consider an exit from housing to be an exit from the program as participants can continue to participate in services and be rehoused if needed or desired. If the person is in stable housing at the 6- or 12-month follow-up points, they are considered to be a successful housing outcome. This is true for six of the nine “total housing exits ever.” The Los Angeles project did not track referrals. Enrollment ended on September 30, 2019. Housing exits are current through December 2019 and include both positive and negative exits.

Source: Monthly calls and communications with sites

Transaction Structuring Cross-Site Themes

Five sites—Montgomery County and Prince George’s County, Maryland; Rhode Island; Anchorage; Austin/Travis County; and Lane County—continued some transaction structuring activities in 2019 and shared some experiences and challenges. Some of these challenges were unique to PFS, while others arise from any effort to expand PSH in a community.

Sites employed different project management structures to suit current needs. Depending on the needs of the project, stakeholder involvement ebbed and flowed. Some sites set up working groups to concentrate on different aspects of project planning—for example, in Rhode Island, the project team assigned stakeholders to housing, healthcare, and legal working groups.

Sites engaged additional technical assistance providers for project support. Sites either brought on for the first time or continued working with outside experts to support different project design areas.

Social Finance is involved with three sites to serve as a financial adviser or intermediary and to support the development of the PFS contract. The Corporation for Supportive Housing is also involved in three sites, as intermediary or technical assistance provider for program design and implementation, supporting the design of the PSH project. Additionally, demonstration sites have access to technical assistance from the Council on State Governments provided by the Bureau of Justice Assistance.

x

E X E C U T I V E S U M M A R Y

The feasibility and transaction structuring phases blended together. While all sites had officially transitioned out of the feasibility analysis phase of PFS by the start of 2019, many spent time continuing or redoing activities typically associated with that phase. This included ongoing community outreach, education of key elected officials and agency heads, the creation or renewal of data-sharing agreements, and data analysis of the target population. Sites found these activities useful to establish more accurate estimates about the size of and costs incurred by the target population. However, additional planning work can lead to more accurate program and contract design; it may also have contributed to perceptions that the PFS process was lengthy and costly among some stakeholders.

Community landscape and stakeholders shaped project design. As sites developed project designs and launched pilots, it became clear that project design aspects were determined by the dominant sector involved and by local priorities. Although all sites will be serving a high-needs population involved with both the homelessness and criminal justice systems, which system was predominantly involved shaped the definition of the target population; the identification and referral pathways; service provider selection and service model; the type of housing used; and the outcome metrics identified for repayment.

Sites dealt with the following common challenges during transaction structuring:

◼

Finding and solidifying financial partners. Several sites struggled to secure end payors, and stakeholders in most sites expressed surprise and frustration at the challenges they faced.

Government partners and service providers were unlikely to view the project as a risk, but some budget staff worried about the risks of the PFS financial model. Securing health payors proved particularly difficult even though health costs were commonly the highest identified cost of the target population to the community.2 Additionally, state budgets and their approval timelines proved to be not easily compatible with PFS funding. Projects that require approval in a state budget are bound to legislative calendars, and when not approved, it can set a project back a full year. Sites that successfully moved PFS contract negotiations forward found that local champions, including local government or elected officials, were key. Ultimately, three sites decided to proceed with philanthropic investments to launch a pilot or full project implementation without signing a PFS contract.

◼

Securing housing resources. Most sites decided to leverage existing housing assistance in the community for the project but found cobbling these resources together challenging. Sites 2 See discussion of financial feasibility in:

https://www.huduser.gov/portal/sites/default/files/pdf/PayforSuccess.pdf.

E X E C U T I V E S U M M A R Y

x i

attempted to access tenant-based vouchers through state or local housing authorities and turnover in existing units but could not keep up with demand through these mechanisms. Sites also faced resistance from landlords and community-based providers to serve the target population.

◼

Designing the evaluation. As sites moved closer to implementation, evaluators and project partners worked through design considerations. They grappled with data constraints, including tracking participants after they exited the program and whether the target population was large enough to construct a robust comparison group.

Implementation of Cross-Site Themes

Four sites—Anchorage, Austin/Travis County, Lane County, and Los Angeles County—implemented PSH programming as pilots or launched PFS projects. These sites also organized their project work into teams: service providers and core team members who worked directly with the target population, and financial and other partners who oversaw the PFS contract. This streamlined communication and helped projects be responsive to the service needs of their clients. So far, respondents have indicated that participants successfully enrolled and leased up, and for those who accessed housing, many have remained housed. Los Angeles County completed project enrollment in 2019. Many stakeholders interviewed believed the project was improving the lives of participants.

These sites also worked through the following challenges in PSH implementation, some of which stem from or are heightened by PFS and others that could arise to expand PSH:

◼

Building capacity to serve “high-need” participants. Service providers in three sites—

Austin/Travis County, Anchorage, and Los Angeles County—added or increased clinical capacity and lowered caseloads to respond to service provider feedback on the support level for participants. The fourth site, Lane County, hired a housing location specialist to facilitate lease-ups.

◼

Building capacity in new service providers. In all four sites, at least one of the service providers was new to Housing First and PSH. Some sites engaged outside trainers to help the providers transition. All sites continued to monitor practices that might conflict with a Housing First model, such as using the criminal justice partner as a punitive tool to enforce compliance.

◼

Finding housing and developing relationships with landlords. Securing enough housing for the project remained a challenge in all sites except for Los Angeles County, which funds tenant-x i i

E X E C U T I V E S U M M A R Y

based vouchers. Even when housing resources were available in Lane County, matching participants to units could sometimes be challenging because of unit location. Participants with prior sex offenses can only be placed in the program within The Oaks or other housing types that the implementation partners can find that are not Section 8 or public housing.

Respondents in Anchorage and Austin/Travis County indicated that landlords were not always receptive to project clients.3

◼

Blending service provider and criminal justice roles in PSH implementation. The two sites in the implementation phase—Lane County and Los Angeles County—enroll participants who are on probation and parole. Supervision requirements in these two sites have caused tension with fidelity to the Housing First model.

◼

Classifying participant exits. Los Angeles County and Lane County tracked participant outcomes for success payments. They found that this was not a straightforward exercise, particularly in the cases of temporary housing instability (such as if a client moves back to shelter or transitional housing in between permanent housing placements). The two sites dealt with this differently, with Los Angeles County counting a move to interim housing (a term used to describe shelter or transitional housing placement) in between housing placements as a negative exit, and Lane County defining housing stability at the time of measurement, meaning a client could have moved or exited a housing placement in the interim.

Benefits of Pay for Success

As we have reported since the beginning of the demonstration, stakeholders consistently noted increased community collaboration in support of the target population as one of the primary benefits of the project. In some places—particularly Los Angeles County—this level of collaboration was not new.

But even stakeholders in places with existing PSH capacity reported benefits resulting from using PFS, including:

◼

Breaking down silos between sectors.

◼

Raising awareness of the existence of the target population, their needs, and possible cross-sector solutions.

3 The Oaks was a new 54-unit affordable housing complex called The Oaks at 14th (The Oaks) that Sponsors Inc used during the pilot program. “The Oaks,” Sponsors, Inc. , https://sponsorsinc.org/tag/the-oaks/.

E X E C U T I V E S U M M A R Y

x i i i

◼

Strengthening local data capacity and cross-sector sharing.

◼

Increasing local capacity through external technical assistance providers.

Stakeholders in sites in implementation or running a pilot reported an increased capacity to serve the target population, such as increasing capacity or skills in providers, new staff, and lower data service challenges.

Time Costs of Pay for Success

In the demonstration, one of the consistently reported challenges of PFS is the time commitment for project stakeholders. In 2019, we began to collect data on time spent by different stakeholders in each site. Between quarter 4 (Q4) 2018 and Q3 2019, the sites report a combined 26,572 hours spent on PFS

planning and PSH implementation. Almost one-half of those hours were reported by the sites in transaction structuring that launched pilots during this time. We found that intermediary organizations spend the most time on the projects, particularly before launching a pilot or implementing a project. We will continue to track and report on this information for the remainder of the demonstration.

x i v

E X E C U T I V E S U M M A R Y

I. Introduction

In 2016, the U.S. Department of Housing and Urban Development (HUD) and the U.S. Department of Justice (DOJ) entered an interagency agreement and launched the Pay For Success (PFS) Permanent Supportive Housing (PSH) demonstration. The purpose of the demonstration is to help communities reduce recidivism and homelessness among the reentry population by expanding PSH through PFS

financing. DOJ made the Second Chance Act (SCA) funds, managed through the Bureau of Justice Assistance (BJA), available for the work.4 HUD issued the Notice of Funding Availability (NOFA) in 2015

and awarded $8.7 million to seven grantees. Grantees were funded for as many as three phases of PFS: feasibility analysis (up to $250,000), transaction structuring (up to $600,000), and project implementation, including evaluation (up to $250,000) and success payments (up to $1.3 million). The original seven grantees were Montgomery and Prince George’s Counties in Maryland, as a single two-county site; the state of Rhode Island; the Municipality of Anchorage and Matanuska-Susitna Borough in Alaska; Austin/Travis County, Texas; Los Angeles County, California; Lane County, Oregon; and Pima County, Arizona. Pima County exited the demonstration in late 2017, and Matanuska-Susitna Borough and Montgomery County had left the projects in Alaska and Maryland in 2018 and 2019, respectively.

Technical assistance was available to demonstration sites through a BJA-contracted provider, The Council on State Governments Justice Center.

Although each demonstration site has its own structure, goals, and strategies, all sites must conform to HUD and DOJ requirements and be designed to ensure a minimum level of consistency while allowing sites the flexibility to meet the needs of their specific communities. This includes creating at least 100 PSH units for high-needs, justice-involved people experiencing homelessness and adopting a Housing First philosophy to program enrollment. Specific requirements are listed in exhibit 3.

4 Congress passed the SCA (Public Law 110-199) with bipartisan support in 2008. The goals of SCA are to increase reentry programming and improve outcomes for people released from state prisons and local jails.

1

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

EXHIBIT 3

HUD-DOJ Requirements for the Target Population and the Permanent Supportive Housing Intervention

Target population

PSH model components

◼ People with criminal justice involvement, defined as

◼ Outreach

multiple jail or prison stays within a 3-year period,

◼

including at least one in the past year

High-quality, permanent, and affordable housing

◼ Accessible transportation and employment

◼ People with homelessness history, defined as

meeting the definition of chronic homelessness

opportunities

according to 24 CFR 578.3 or being homeless for 12 ◼ Housing First approach months over the past 3 years or for one or more

nights during each year in the past 3 years

◼ Housing stability services

◼

◼ People with high needs, defined as having a history

Informed property or landlord management

of high-cost use of services or significant physical or

◼ Care management and service coordination

behavioral health challenges that require high-cost

support

◼ Coordination with the criminal justice system (for

example, courts or community corrections)

PSH = permanent supportive housing.

Source: Liberman et al. (2020)

HUD contracted Urban to conduct a national evaluation of the HUD-DOJ PFS PSH demonstration.

Urban’s evaluation goals are to examine how PFS is implemented in each site, capture lessons learned across sites, and examine the benefits of using PFS to fund PSH for a reentry population. The national evaluation is not intended to replace the local evaluations. Our research seeks to document the reality on the ground as each community began its PFS project and, in hindsight, how that may have affected the project as it moved through each stage. Three objectives drive Urban’s evaluation strategy:

◼

Document the progress and processes of partners in each site.

◼

Document costs and time use incurred by partners.

◼

Document the benefits of the model to the target population.

This report documents evaluation data collection, analysis, and findings for Year 3 of the demonstration. This includes information on site progress, challenges faced, site partner perceptions of the benefits of the project, and estimates of the time spent by the project partners on PFS planning and implementation activities in 2019.

2

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

The HUD-DOJ Pay for Success Permanent Supportive

Housing Demonstration

The HUD-DOJ PFS PSH demonstration is designed to test the feasibility of using the PFS model to expand PSH to improve housing stability and other outcomes for justice-involved people experiencing homelessness. It addresses specific challenges at the intersection of housing and criminal justice with a theory of changes that blends the systems-change potential of the PFS financial model with the demonstrated benefits of an evidence-based PSH programming model.

Defining the Problem

Homelessness is a crisis in the United States, and there is a considerable need for additional resources to address this crisis. In 2019, nearly 568,000 people across the country experienced homelessness on a given night, a 3-percent increase from the previous year (Henry et al., 2020). This increase marks the third consecutive year in overall homelessness increases following steady decreases from 2010 to 2016. The number of people experiencing chronic, or long-term, homelessness increased 8 percent from the previous year to just over 96,000 people. This is the third year that chronic homelessness has increased after 8 consecutive years of decreases from 2008 to 2016.

People experiencing chronic homelessness are more likely to stay in unsheltered conditions, on the streets, or in places considered unfit for human habitation than those who are not chronically homeless.

People living in unsheltered situations have higher rates of previous involvement with the juvenile and criminal justice systems than those in shelters (Flaming, Burns, and Carlen, 2018; Lee et al., 2016; Linton and Shafer, 2014). Furthermore, people living in unsheltered conditions are exposed to the possibility of interactions with the criminal justice system because they may be jailed for public nuisance offenses like panhandling, trespassing, and public intoxication. In turn, incarceration can exacerbate housing instability (Metraux, Roman, and Cho, 2007). This link between criminal justice involvement and homelessness is important because the number of people with criminal justice involvement in the United States is large and can significantly increase homelessness. In 2016, more than 6.6 million people were held in prisons and jails or under supervision through probation and parole in the United States. Of those people, 740,700 were held by local jails (Kaeble and Cowhig, 2018).

Criminal justice involvement also compounds the difficulty of addressing underlying health and mental health conditions. When people become incarcerated, they may experience interruptions in medications or treatments or cease receiving care at all. When they reenter the community, transitions E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

3

back to care can be difficult (Mallik-Kane and Visher, 2008). Changes in access to care associated with reentry may force people to use costly crisis systems, including sobering centers and hospital emergency rooms. Relying on these systems is an ineffective strategy for maintaining long-term health.

Although reentry populations experiencing homelessness and health challenges generate significant costs to local institutions, including costs of jail stays and use of shelters and emergency health care, no one system is adequately positioned or prepared to address the needs of this population comprehensively.

Permanent Supportive Housing

PSH is an intervention intended to provide housing without time limits and supportive services to people who are experiencing homelessness and cannot otherwise exit homelessness and stabilize in housing. PSH can be provided in scattered-site or project-based configurations. In scattered-site supportive housing, the housing assistance is typically provided through a tenant-based voucher used to access various housing options on the private rental market. In project-based supportive housing, all the units are in one building or on one site. Project-based supportive housing is usually developed using various funding streams and relies on project-based vouchers to provide ongoing operating capital.

Supportive services in supportive housing vary greatly and are intended to be individualized to the participant. Programs often use motivational interviewing5 and trauma-informed care practices.6 Some programs have more intensive structured service models, including Assertive Community Treatment and similar evidence-based interventions.7

PSH is intended to follow a Housing First model, meaning that people are placed in housing without preconditions like sobriety and that services to address health and other needs are included but are not required. As its name implies, PSH provides “permanent” or long-term housing assistance (rather than temporary or transitional support) to make housing affordable. Residents pay no more than 30 percent of their income toward rent.

PSH can improve housing outcomes and reduce incarceration among a justice-involved population.

The Denver Supportive Housing Social Impact Bond Initiative (Denver SIB) aims to support people 5 For more information on motivational interviewing, see: samhsa.gov/homelessness-programs-resources/hpr-

resources/empowering-change.

6 For more information on trauma-informed care practices and homelessness, see:

https://www.usich.gov/news/trauma-informed-care-building-on-our-commitment-to-strengths-based-

approaches-to-ending-homelessness/.

7 For more information on Assertive Community Treatment, see: https://store.samhsa.gov/product/Assertive-

Community-Treatment-ACT-Evidence-Based-Practices-EBP-KIT/SMA08-4344.

4

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

struggling with homelessness, substance use, and mental health challenges, a vulnerable group known to cycle in and out of jail. From January 2016 through December 2018, 330 people were housed through this project. Two years after entering housing, 79 percent of participants were still housed (Cunningham et al., 2019). In a quasi-experimental study of supportive housing for frequent users, Aidala and colleagues (2014) found a 40 percent reduction in jail days over 24 months for people in supportive housing compared with a quasi-experimental comparison group.

PSH capacity has increased dramatically in recent years. In 2019, there were nearly 370,000 PSH

beds nationwide, 181,505 of which were dedicated to people experiencing chronic homelessness—3.6

times as many beds dedicated to chronic homelessness than 2009 (Henry et al., 2020). Despite this growth, there are still not enough PSH units to solve chronic homelessness, and the rate of growth has slowed in part because of the cost (NASEM, 2018). This demonstration will test the ability of PFS to finance expanding PSH for justice-involved populations.

The Pay for Success Model

PFS is an innovative financing model that is designed to link payment for preventive social programs to outcomes achieved. In a typical procurement model, governments provide up-front funding for service provider operations. It pays for activities and hopes that those activities produce positive outcomes.

Withholding payment until outcomes can be verified is often unrealistic because most service providers cannot wait for months or years to receive payment.

In the most common PFS model, known as a social impact bond or PFS financing, an outside investor provides upfront capital for program costs. If the program achieves predetermined outcomes for the people served, then the end payor (typically a government entity) repays the investor, usually with interest. These payments are typically referred to as “success payments.” This model, illustrated in exhibit 4, ensures that end payors only pay for achieved outcomes (i.e., success) and, at the same time, helps build evidence on program effectiveness through independent evaluations. Since the emergence of PFS in 2012, communities worldwide have been using a continuum of PFS model variations in their efforts to improve program performance and people’s lives.8

PFS is best used for programs with the following:

8 For more context, see “The global impact bond market in 2019: A year in review,” The Brookings Institution,

https://nff.org/report/pay-success-first-25.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

5

◼

A strong evidence base. Previous evidence of effectiveness is necessary to help define outcomes and repayment thresholds. Without that precedent, project planners are more likely to set unreasonable thresholds by over- or under-estimating program effect.

◼

Difficulty finding traditional funding. When one agency would be responsible for paying for a program, but most of the benefits would accrue to another, it is less likely to decide to spend scarce resources. This “wrong pockets” problem inhibits the funding of programs like PSH that can improve outcomes and possibly reduce costs in the housing, corrections, and healthcare sectors (NASEM, 2018). The PFS model is supposed to overcome this and some other political barriers to direct government funding (Roman, 2015).

◼

Short-term impacts: In PFS, payments are tied to outcomes. To be feasible, those outcomes must be measurable in 1 to 5 years of program operation. Longer-term outcomes are not feasible for the model because most investors will not wait that long for repayment.

EXHIBIT 4

Pay for Success Financing Process and Partners

Source: Urban Pay for Success Initiative Slide Library

6

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

All PFS efforts include some core activities necessary to get a project off the ground. These include managing multi-stakeholder collaboration, building local capacity, assessing and possibly improving data systems, defining the problem and the target population, and conducting a baseline cost analysis.

The model is not without risks to the participating partners. Investors face financial and reputational risks (Walsh et al., 2017). Service providers face reputational risk if the evaluation shows that the program was not effective (Giantris and Pinakiewicz, 2013). Such a finding could lead to financial risk if the project is terminated, and expansion to other markets is prevented.9 Local project champions face reputational risk if local leaders think the effort required for PFS was not worth the benefit. All parties experience termination risk if one or more parties to the contract decide to stop participating.

HUD and DOJ defined three discrete phases for the demonstration:

◼

Feasibility analysis: Institutional and programmatic feasibility is examined through informal and formal collaborations and data analysis. The phase concludes with a report that documents feasibility and the next steps for transaction structuring.

◼

Transaction structuring: PFS planners secure commitments from all financial partners and identify service providers. This phase may include a pilot study to test assumptions and revise the model. Parties cement the PFS project in a legal contract, often accompanied by several other legal agreements that outline the implementation and governance of the initiative and the parameters by which government will make success payments to investors.

◼

PFS contract implementation: Once a PFS contract closes, the work of implementation begins.

Success payments are made if the project meets agreed-upon outcomes as determined by an independent outcome evaluation.

The Demonstration Theory of Change

A theory of change provides an overview of how program activities are expected to lead to specific outcomes. It includes key assumptions about how change can occur and provides a platform to test and refine those assumptions

9 Service providers may also take on financial risk if they agree to directly fund pilots or other services prior to identifying an end payor or investor.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

7

Exhibit 5 represents the PFS theory of change developed for the demonstration. This model combines activities and their expected outcomes for PFS planning (such as data sharing, evidence base use, barrier reduction, or funding increase) and PSH implementation with the outcomes of the service provision itself (for example, increased housing stability and decreased criminal justice system involvement and return to custody). The actual impact of these benefits depends on two things: the ability of partners to act effectively and the amount these actions differ from the status quo.

Demonstration sites using PFS to bring a new PSH program to a community have the potential for greater systems change than a community using PFS to expand an existing program already supported by cross-sector collaborations and data-sharing practices.

EXHIBIT 5

A Blended Theory of Change for Permanent Supportive Housing Supported by Pay For Success Cost avoidance,

Individual-level

offsets, or savings

PFS

PFS outcomes

Intervention

outcomes

accrued

Bring together

Data sharing

Housing First

Increase housing stability

Avoid costly

cross-sector

shelter, jail/prison

◼

◼

◼

partners,

To define the

Remove any

Reduce homelessness

stays and use of

◼

including

target

preconditions and

Provide a safe, healthy, crisis health

private

population

barriers for housing

and stable housing unit services

◼

investors, to

eligibility

Decrease shelter use

investigate the

Use evidence base

Decrease criminal justice

feasibility of

involvement and reentry

PFS, structure

◼ To define

Housing subsidy

to jail and prison

the

outcome-based

◼ Provide rent

transaction,

payments

◼ Decrease arrests

assistance in a

and develop a

◼ Decrease jail and

housing unit that is

PFS contract

prison days

Identify and

safe, sustainable,

◼ Decrease jail and

remove system-

functional, and

prison reentry

level barriers

conducive to tenant

stability

Increase appropriate

◼ To increase

healthcare services and

access to

improve health

effective

Case management

services

services

◼ Decrease detox visits

◼ To improve

◼ Decrease avoidable

◼

outcomes

Develop a case plan

emergency room and

◼ Facilitate access to

hospital visits

benefits

◼ Connect to mental and

Increase funding

◼ Provide referrals

physical health care

and PSH

◼ Coordinate care

and substance abuse

and system

◼ For the target

treatment

involvement

population

◼ Decrease severity of

illness

Coordinate with

◼ Improve mental health

community

◼ Improve physical

corrections

health

Source: Liberman et al. (2019)

8

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

Launching or expanding PSH for a justice-involved population with any funding model requires coordination across the justice, housing, and healthcare sectors. The people typically served by PSH

may frequently interact with all those sectors, and PSH provides an opportunity to improve outcomes in all. It also involves coordination across city, county, and state governments. PFS requires the same level of coordination, as well as outreach to the impact investment community. These similarities—along with a strong evidence base, observable short-term (3 to 5 years) outcomes, and a need to overcome bureaucratic challenges—make PSH a likely candidate for expansion through the PFS funding model.

The ability of the PFS model to expand PSH is still in question. By the end of 2019, at least 24

localities have explored whether PFS could be used to support supportive housing in their community.10

By the end of 2019, five had launched. They are the two from this demonstration—Los Angeles County, California, and Lane County, Oregon—and PFS PSH projects in Massachusetts; the County of Santa Clara, California; and the City and County of Denver, Colorado. Of these, only Denver and Los Angeles use both housing and justice outcomes as success payment metrics (housing retention and impact on jail stays).11 The HUD-DOJ demonstration is the largest test of using PFS to expand PSH that includes both housing and justice-system outcomes.

The Demonstration Evaluation in Year 3

The demonstration evaluation methodology that Urban used was designed to collect and analyze data on places engaged in PFS planning and determine if this process led to expanded PSH capacity for communities. After the project, we will observe whether the assumptions used to create the theory of change for the demonstration were correct and what unanticipated outcomes occurred for both the PFS planners and the people served by the PSH intervention. As such, the Urban Institute evaluation strategy is driven by three objectives.

Evaluation objective one: Document progress and processes of partners. By studying the process, we learn how the PFS model is implemented in diverse settings with different structures, populations, and community contexts. Research questions include the following:

10 This includes the demonstration sites discussed in this report and others supported by other efforts.

11 For more on the Denver social impact bond initiative, see “Denver Supportive Housing Social Impact Bond Initiative,” Urban Institute, https://www.urban.org/policy-centers/metropolitan-housing-and-communities-policy-

center/projects/denver-supportive-housing-social-impact-bond-initiative. For more on the other launched PFS

PSH projects, see “Pay for Success: The First 25,” Nonprofit Finance Fund, https://nff.org/report/pay-success-first-

25.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

9

◼

What tasks are accomplished in each PFS phase, and how are they accomplished?

◼

What are the primary challenges faced in each PFS phase, and how are they mitigated?

◼

What milestones or performance metrics in each PFS phase indicate progress and, in hindsight, are viewed as critical to the success of the PFS project?

Evaluation objective two: Document time use and costs incurred by partners. By studying time use and costs, we are learning about the staff time required of each key partner and other budget expenditures.

Research questions include the following:

◼

How much time do partners spend on developing the PFS PSH project in each phase? How does this vary by site?

◼

What other funding is leveraged to implement the PFS PSH project?

◼

What are the aggregate labor costs by PFS partner type? By PFS phase? By site?

Evaluation objective three: Document benefits of model to target population. By studying benefits, we are learning about both the indirect benefits produced by the PFS model as a forcing mechanism for systems change and the direct benefits of the program through positive cross-sector outcomes for enrolled people. Research questions include the following:

◼

Throughout PFS phases, how do PFS partner perceptions and interactions change, and how does business as usual with partners change in ways that change community-level systems and benefit the target population?

◼

In the implementation, how do PFS projects produce program-level benefits for the target population, particularly in terms of new supportive housing units and program performance?

◼

In the implementation, what are the individual-level outcomes for people participating in supportive housing funded by a PFS project, particularly in the areas of housing, criminal justice, and health?

Process Study Methodology

In 2019, Urban continued regular communication with sites and other data collection methods that have been going on since the start of the demonstration. This included the following:

◼

Calls with sites. Each month, the evaluation team leads a call with representatives from the sites to understand progress and any barriers to progress. Site representatives typically include 1 0

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

the intermediary and other core project members, including government partners and technical assistance providers.

◼

Project enrollment tracker. One component of monthly site communication is learning about ongoing PSH enrollment for sites that are operating pilots or are in contract implementation.

The evaluation team asks for various metrics about project progress, including the number of people referred to and placed in housing in the previous month and the number of exits.

◼

Document review. The evaluation team reviews any relevant documents the sites provide. At the beginning of the demonstration, this included feasibility reports and other planning information. In 2019, this included success payment reports and data dashboards on participant outcomes developed after two sites progressed into PSH contract implementation.

◼

Key informant interviews. Each year, the evaluation team conducts a 2-day visit to each site to interview project stakeholders, including staff at the intermediary organization, the grantee organization, local government partners, confirmed and potential end payors and investors, pilot funders, confirmed and potential service and housing providers, and evaluators (see appendix A for example stakeholder list). Where possible and applicable, the visits included observing relevant meetings or events. For example, the 2019 visit to the Los Angeles site included observing proceedings of the Los Angeles County to the Office of Diversion and Reentry Housing Court, Department 44, which is the referral pipeline for participants in that project. Respondents are asked about the key tasks accomplished and challenges faced within the prior year and the benefits and costs of the PFS project. The questions are tailored to the role of the respondent in the project and the progress of the site.

Transcripts from the monthly calls with sites and the key informant interviews were analyzed using NVivo software. The evaluation team revised the coding scheme for the 2019 data analysis to account for updated interview protocols and site progress. All coding output was reviewed for common themes, challenges, perceived benefits, site progress, and other elements relevant to the evaluation research objectives.

Annual Partnership Survey Methodology

Urban fielded the Annual Partnership Survey for the second time this year. The survey was designed to measure changes in the strength of the partnerships and the benefits at the population- and system-levels. The survey asks questions about respondent backgrounds and roles in the project, collaboration with partners, data sharing, and barriers to service provision. The survey was administered using E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

1 1

Qualtrics, a web-based software, and was designed to take 15 to 20 minutes to complete. It was fielded from December 2019 through February 2020.

Eligible respondents included any person who participated in key planning or implementation activities in 2019, even if that participation was infrequent. Exhibit 6 shows the characteristics of those who completed the survey and the overall response rates by site and organizational role on the project.

The 2019 sample included 136 people from 67 active partner organizations (between 9 and 16

organizations in each site). For this wave, 104 people completed the survey, a response rate of 76

percent.

Some sites (like Anchorage) were represented more heavily than others (like Rhode Island) because the overall number of partners varies across the sites and can depend on the PFS phase. We categorized the sample by whether the role of respondent and non-respondent organizations were more focused on the PFS planning or PSH implementation aspects of the demonstration, as well as evaluation partners.

Response rates for PFS and PSH partners closely matched the overall survey response rate of 76

percent, and all evaluators completed the survey. Fifty-six percent of respondents work at organizations that are directly involved in serving the target population. Respondents tended to be actively involved in the project: 73 percent of respondents had been involved in the project for a year or longer, and 57 percent had attended more than 10 meetings related to the project in the previous year.

1 2

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

EXHIBIT 6

Annual Partnership Survey: Respondent Overview

Percent of

overall

Entire

survey

sample

N

respondents

(%)

Site

Montgomery and Prince George’s Counties, MD

15

14

83

Rhode Island

9

9

64

Anchorage, AK

29

28

81

Austin/Travis County, TX

22

21

76

Lane County, OR

12

12

71

Los Angeles County, CA

17

16

77

Total

104

100

76

Organization role in PFS Demonstration

PFS planning partner

63

61

75

Intermediary

19

18

86

Technical assistance

12

12

92

Financial partner

15

14

75

Other local partner

17

16

59

PSH implementation partner

32

31

74

Direct service for the target population

22

21

73

Housing partner for the target population

3

3

60

Criminal justice: referral source for the target population

4

4

100

Other criminal justice partner

3

3

75

Evaluator

9

9

100

Total

104

100

76

Tenure on PFS Demonstration

Less than 6 months

13

13

6–12 months

15

14

12–18 months

9

9

18–24 months

12

12

24–30 months

20

19

More than 30 months

35

34

Total

104

100

PFS = pay for success. PSH = permanent supportive housing. Notes: A total sample of 136 people received the survey. The response rate is not available for tenure on PFS demonstration because that information comes from the survey and is not available for non-respondents.

Source: 2019 annual partnership survey

Because the survey has more than 70 questions, we created indices that group the questions by topic to simplify reporting and interpretation. For example, we grouped questions about different types of data sharing or collaboration on one scale. This method allows us to draw broader conclusions about the level of partnerships, collaboration, data sharing, and challenges across demonstration sites.

After creating the indices, we compiled responses of respondents involved in the PFS planning or PSH implementation. We also compiled responses by the PFS phase to determine how partnerships, collaboration, data sharing, and challenges may vary between transaction structuring and contract implementation. We ran regressions to test the significance of the differences between different roles E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

1 3

and phases, and we report only on the indices determined to be significant. No adjustments were made to account for variation in total respondents across sites. For a full description of this methodology, see appendix B.

EXHIBIT 7

Partnership Survey Indices

Number of

Questions

Index name

Index definition

in Index

Partnerships (roles, planning, and infrastructure)

Project roles and

The PFS demonstration is meeting its aims through data use and is led by 4

progress

organizations with distinct roles.

Organizational role and Partner organizations and staff members understand the problem the 4

awareness

project addresses, their role on the PFS demonstration, and how that role is related to supporting the target population.

Organization’s level of

Respondent’s organization plays an active role in the PFS supportive

1

involvement (self)

housing project.

Vision alignment of

Project partners work well together and have a common vision for the

8

partners

project and understanding of PFS, PSH, and how they intersect.

Community and

The community supports the PFS demonstration by prioritizing and

8

provider support and

leveraging resources and expertise, and the community and service

capacity

providers reduce barriers to serving the target population by making

housing accessible and educating policymakers.

Ability to bill Medicaid

Supportive housing providers can bill Medicaid for covered services.

1

for services

Data use and sharing

Performance data use

Partners use data to identify the target population and the level of

3

services needed.

Data sharing for PFS

Partners use data to identify the target population and the level of

2

planning

services needed.

Service provider data-

How often service providers are sharing and receiving data about clients 2

sharing frequency (self)

in the target population.

Criminal justice data-

How often service providers are sharing criminal justice client data (such 4

sharing frequency (self)

as criminal history, jail/prison release, or probation/parole information).

Client data-sharing

How often service providers are making referrals to other organizations, 5

frequency (self)

as well as information about other services, assessments, or outcomes

obtained.

1 4

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

Number of

Questions

Index name

Index definition

in Index

Collaboration

PSH provider

PSH providers are collaborating with other organizations on the PFS

2

collaboration

demonstration to provide housing stability and health services to the

target population.

Cross-system

Law enforcement, homeless assistance, courts, jails, and

7

collaboration

health/behavioral health services are collaborating.

Organizational

How often a respondent organization is sharing or leveraging resources, 6

collaboration

trainings, or service information.

frequency

Organizational

Change in overall collaboration—including data sharing, formal

3

collaboration increase

agreements, and meetings—over the past year (1 = decrease, 2 = no

change, 3 = increase)

Increase in partners on

Change in number of PFS partner organizations represented on the local

4

local criminal justice

criminal justice coordinating committee over the past year (1 = decrease, coordinating

2 = no change, 3 = increase)

committee

Challenges

Collaboration

The project has faced challenges to collaboration that include distrust, 4

competition for resources, conflicting visions, lack of working

relationships, and skepticism about Housing First.

Data service

The project has faced challenges in providing services, including

8

coordination of entry for homeless services, data information sharing

across organizations, and limited time and resources.

Accessing clients in jail

The project has faced challenges accessing clients in jail.

1

PSH service

The project has faced challenges because of a lack of access to housing 6

and services, such as housing eligibility challenges or general lack of services.

PFS = pay for success; PSH = permanent supportive housing. Notes: Scales are from 1 to 4 except where noted. See appendix B for more detail.

Source: 2019 annual partnership survey

Time Use Survey Methodology

In 2019, Urban launched a time use survey to understand how much time different stakeholders spent in each site on the PFS project. Data collection involved outreach to up to six working on the demonstration project in each site, and organizations were selected based on their roles in PFS planning and PSH implementation. Participants included the PFS intermediary, the service provider, the government partner or end payor, funders, and technical assistance providers. We collected data on individual staff time, but given the dynamic nature of who works on the project in a given quarter, we E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

1 5

consider each organization a single respondent. Therefore, the response rate for this data collection is at the organization, not the individual, level.

We use three methods of time use data collection. The primary method is direct email outreach to an organization point of contact every quarter requesting time-use data for everyone in their organization who has spent time on the project. The email describes the purpose of the data collection and how the information will be used and says that reporting is voluntary. We attached an Excel table with fields for name, position, and hours spent to facilitate consistency across respondents (for an example, see exhibit 8). Our secondary data collection method is to capture time-use data from the invoices submitted to HUD through the DRGR system. For any respondent who did not participate in these two methods, we estimated time use through site visit interview responses. We shared these estimates with the respondents and asked them to verify the accuracy of the data or provide a correction. If the person responded and verified, we recorded that response in the dataset. For the email survey, we sent three reminder emails and followed up with phone calls to minimize nonresponse.

For points of contact who did not respond, we worked with the site to identify other people to contact or strategies to reach the primary point of contact.

EXHIBIT 8

Time Use Data Collection Template

Sources of

Staff members

Hours spent on project

information

Where did you

Staff

Staff member

Q4

Q1

Q2

Q3

Q4

find this

name

role/level

2018

2019

2019

2019

2019

information?

Person 1

Executive director

12

6

6

8

6

Outlook calendar

Person 2

Policy associate

36

42

36

40

48

Timecard

Exhibit 9 shows the number of respondents for each quarter. The data collection period began in Q4 2018. Between 31 and 36 organizations were surveyed, depending on the quarter. During the project, some organizations responded that they were no longer involved in the project, usually because their work was completed or their funding ran out. These organizations were dropped from future data collection. If an organization indicated a pause in the work, it was still included in data collection, but its time for that quarter was reported as zero. These distinctions are important for understanding how the response rate changes from quarter to quarter. As organizations report that they have left the project, they are no longer included in the response rate. This means that over time, the number of organizations reporting data changes. The data in this report cover a four-quarter period, from the 1 6

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

fourth quarter of 2018 through the third quarter of 2019. During that time, there was a 74 percent cumulative response rate from partners at each site. The survey response rate lagged over time.

EXHIBIT 9

Time Use Study Response Rate

Response rate

Fiscal year quarter

Organizations responded

Total organizations surveyed

(%)

2018 Q4

29

36

81

2019 Q1

29

36

81

2019 Q2

24

35

69

2019 Q3

20

31

65

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

1 7

II. Demonstration Site Overview and

Progress in 2019

The demonstration sites have substantive variations in almost all aspects of their PFS planning and implementation. These variations include the number and type of partners, program model, eligibility criteria, and referral pathways. Exhibits 10 and 11 provide an overview of the target population and project partners for all demonstration sites. The accompanying text summarizes the project details and the broader landscape of state and local efforts to end homelessness and implement criminal justice reform. Four sites began and ended 2019 in the transaction structuring phase (Prince George’s County, Rhode Island, Anchorage, and Austin/Travis County); two of these sites operated pilots during the year.

Two sites began and ended the year in implementation (Lane County and Los Angeles County).

EXHIBIT 10

Target Population Criteria by Site

Site

Criminal justice

Homelessness

“High needs”

Prince

A score of moderate or high

A vulnerability for living in unsafe

Unmet behavioral or somatic

George’s

risk to re-offend, as measured

or unstable housing; measured by

health needs, including

County, MD

by the Level of Service

VI-SPDATb and a county-

substance use, mental health,

Inventory-Revised (LSI-R)

developed housing vulnerability

and disability needs measured

assessment

by multiple tools

Rhode Island

Frequent users of the Department of Corrections, homeless shelters, and Medicaid services; specific criteria under development

Anchorage,

Has been released from a

“Chronically homeless” per 24

Has a history of high-cost

AK

correctional facility two+

CFR 578.3 OR homeless at least

utilization of crisis services

times in the prior 36 months,

once in each of prior 36 months

(e.g., emergency rooms) OR has

including once in the prior 12

OR homeless at least 12 months

significant medical health or

months

cumulatively in prior 36 months

behavioral health challenges

Austin/Travis Minimum of 1 jail day or jail

Meets HUD’s definition of

Minimum of 1 inpatient day or

County, TX

booking in last year and a

chronic homelessness

four emergency department

minimum of two jail bookings

visits in the past 18 months

in the last 3 years

Lane County,

Released from state prison

Homeless since most recent

Determined by CAT-R. There is

OR

into Lane County with a

release from prison; a history of

a priority for women, families,

medium to very high risk of

housing instability as defined by

or clients with criminal

recidivating as defined by a

the CAT-Rc

histories excluded from other

risk assessment tool

federal housing options.

Los Angeles

Discharge from county facility

Homeless at least one night

Upper third quadrant on VI-

County, CA

in 30–120 days, eligible for

during the year in each of the last

SPDAT, disabling condition

diversion services through the 3 years

alternative court

Notes: aLSR-I is a standardized criminogenic risk assessment. b VI-SPDAT is a combination of the Vulnerability Index and the Service Prioritization Decision Assistance Tool and assigns a score based on a person’s risk of dying on the street. cThe Coordinated Assessment Tool for Reentry (CAT-R) was customized for the reentry population based on the framework of the National Alliance to End Homelessness’s Coordinated Assessment Tool, suited for the chronically homeless population.

Source: Monthly calls and interviews with sites

1 8

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

EXHIBIT 11

Summary of Demonstration Project Partners in 2019

Sites Planning Projects

Sites Implementing Project Pilots

Sites Implementing Projects

Montgomery County

and Prince George’s

Austin/Travis County,

County, MD

Rhode Island

Anchorage, AK

TX

Lane County, OR

Los Angeles County, CA

Intermediary

American Institutes for

Rhode Island Coalition for

United Way of Anchorage Ending Community

Third Sector Capital

(1) Corporation for

(grantee)

Research

the Homeless

Homelessness Coalition

Partners

Supportive Housing (CSH)

(ECHO)

(2) National Council on

Crime and Delinquency

Technical

AIR and Third Sector

Social Finance

(1) CSH

(1) CSH

Third Sector Capital

assistance

(as Intermediary Advisor)

(2) Social Finance

(2) Social Finance

Partners

Evaluator

ICF International

Not yet identified

NPC Research

Abt Associates

Mark Eddy at New

RAND Corporation

York University

Service

Not yet identified

Not yet identified

(1) RurAL CAP

(1) Caritas of Austin

(1) Sponsors

(1) Amity Foundation

provider(s)

(2) South Central

(2) Integral Care1

(2) Homes for Good2

(2) Project 180

Foundation

(3) The People Concern

(4) Volunteers of America

Housing

Not applicable

Not yet identified

RurAL CAP

ECHO

(1) Sponsors

Brilliant Corners

provider(s)

(2) Homes for Good

Pilot funders

Identified but not

n/a

(1) Rasmuson Foundation

St. David’s Foundation

Meyer Memorial

n/a

finalized

(2) Alaska Mental Health

Trust

Trust Authority

End payors

Prince George’s County

Not yet identified

Not yet confirmed

(1) City of Austin3

HUD

Los Angeles County

(2) Episcopal Health

Foundation3

(3) Central Health3

(4) Travis County3

(5) Ascension-Seton3

Other

(1) Dept of Social Services (1) Governor’s office

(1) Municipality of

n/a

(1) Parole and

(1) Office of Diversion and

government

(2) Dept of Corrections

(2) Office of Housing and

Anchorage

Probation

Reentry

planning

(3) Dept of Health

Community Development

(2) Oregon Criminal

(2) Housing for Health

partners4

(4) Office of Management (3) Executive Office of

Justice Commission

Division at the Department

and Budget

Health and Human

(3) Trillium

of Health Services

Services

Community Health

(4) BHDDH5

Plan

Investors

Identified but not

Not yet identified

Not yet identified

Not yet confirmed;

Oregon Community

(1) Hilton Foundation

finalized

Grant provided by St.

Foundation

(2) UnitedHealthcare

David’s Foundation3

Notes: 1Caritas provided services for the pilot and contract implementation, and Integral Care only during contract implementation. 2Homes for Good is the Lane County Housing Authority and provides services to people in public housing and those with vouchers, but not all services they provide are available to all participants in the way that Sponsors’ services are. 3Not yet confirmed. 4At the level of site jurisdiction unless otherwise noted (i.e. All partners listed for Prince George’s County are County Departments, and all listed for Rhode Island are State Departments). 5Department of Behavioral Healthcare, Development Disabilities, and Hospitals.

Source: Monthly calls with sites and stakeholder interviews

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

1 9

Transaction Structuring Sites without Pilots

Montgomery County and Prince George’s County, Maryland, and Rhode Island began and ended the year in transaction structuring. Neither site operated a pilot in 2019, although Prince George’s County plans to do so in the future (it is not currently part of the Rhode Island plan).

Montgomery County and Prince George’s County, Maryland: Background

Montgomery County and Prince George’s County joined together on a cross-county effort to use PFS to expand PSH and partnered with the American Institutes for Research (AIR). Each planned on using PFS

to provide scattered-site PSH to improve outcomes for people being released from jail with a moderate or high risk of reoffending, vulnerability for living in unsafe or unstable housing, and an unmet behavioral health need. The PFS planning partnership includes the intermediary and grantee, AIR; Prince George’s County government planning partners, including the Department of Social Services, Department of Corrections, and Department of Health; Montgomery County government planning partners, including the Department of Health and Human Services, Department of Housing and Community Affairs, and the Department of Finance; potential end payor the county Office of Management and Budget; evaluator ICF International; and technical assistance partner Third Sector Capital Partners. This work aligned with other efforts to improve outcomes for justice-involved people in one county. Both Montgomery County and Prince George’s County participate in the Data-Driven Justice Initiative to better identify people cycling in and out of the justice system and connect them to services. Prince George’s County also received support from the Department of Justice’s (DOJ’s) Smart Reentry: Focus on Evidence-Based Strategies for Successful Reentry from Incarceration to Community program.

Feasibility work, headed by AIR, Montgomery County Department of Health and Human Services; and Prince George’s County Department of Social Services, began in 2017 continued through early 2018. During this time, they established partnerships with other stakeholders in health, finance, contracts, and corrections. The dual-county focus of the project created challenges due to different requirements from each county’s legislative and contracting offices and distinct service and housing landscapes. The outcome of the 2018 elections also created challenges for maintaining momentum and leadership support in Montgomery County when key agency leaders transitioned out of the Administration. After the elections, agencies in both counties had to wait for more than six months to E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

2 1

appoint and hire new staff members, who needed to be briefed on the Pay For Success (PFS) project and their roles as key decisionmakers. This turnover caused delays in transaction structuring work goals.

Montgomery County and Prince George’s County, Maryland: 2019 Progress

In 2019, Montgomery County and Prince George’s County continued their transaction structuring work with AIR; it was the first year that the site was fully in the transaction structuring phase. Once counties filled their vacancies after the 2018 elections, AIR met with key leaders to brief them on the project and make sure they were informed and onboard. AIR established and facilitated working groups in each county to cover major topics, including investors, financing, evaluation, and intervention. Those groups completed their work in August 2019, and AIR used results from this work to lead investor outreach and draft the request for applications for the service provider in November 2019.

Third Sector has continued to serve as an intermediary adviser to AIR. In 2019, this work included sharing contract templates and documents with AIR and the counties, providing support in monthly meetings and calls, advising on evaluation design options and related payment structures, and planning site visits for government partners to visit other PFS Permanent Supportive Housing (PSH) sites around the country. In February, the Director of Prince George’s County Department of Social Services attended the Sorensen Summit in Salt Lake City. In September, Third Sector arranged for a larger cohort of representatives from Prince George’s County, AIR, and ICF International to meet with PFS planners in Santa Clara County, California.

The site continued to make progress on selecting investors. Members of the steering committee from each county reviewed and ranked potential investors and presented their recommendations to the executive committee for feedback. AIR started preliminary conversations with investors in mid-2019

but felt it would be best to wait for a more direct engagement until the service provider was selected and the program model finalized.

County steering committee members also participated in a working group that in early 2019

drafted a logic model of the proposed intervention that informed both service provider selection and the evaluation plan that ICF International drafted over the year. The request for applications was also informed by ones used in Salt Lake City. It was released in November and closed in December 2019. AIR

expected the provider(s) to be selected in early 2020. Initial drafts of the ICF International evaluation plan received extensive notes from AIR staff who have expertise in program evaluation in the criminal justice system. Throughout 2019, AIR and ICF International worked together to get closer to a plan that 2 2

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

both organizations were confident would work to measure improvement in housing outcomes for a justice-involved population. This iterative work was ongoing at the end of 2019.

This site underwent a major shift after the 2018 election when Montgomery County decided to leave the demonstration. The county reports they have ended veteran homelessness in 2015 and made significant progress toward meeting its goal of ending chronic homelessness at the end of 2018. As a result, new Montgomery County leaders concluded that its population of individuals experiencing chronic homelessness was small. The target population for this demonstration would be best served by interventions such as Rapid Rehousing that are less costly than PSH. AIR discussed with HUD whether Montgomery County could continue in the demonstration using interventions other than PSH, but because PSH is a core component and common thread across all sites, an alternative approach was not approved. Montgomery County expressed that it could finance other programs with existing resources and withdrew from the project in September 2019. As a result, the site work continues with one county, Prince George’s County, in partnership with AIR and its grant.

Rhode Island: Background

Rhode Island plans to use PFS to create PSH for frequent utilizers of the Department of Corrections, homeless shelters, and Medicaid services, but the specific target population is not yet defined. The core PFS planning team included the intermediary, Rhode Island Coalition for the Homeless (RICH); government partner and potential end payor, the Rhode Island governor’s office; and the technical assistance provider, Social Finance. This PFS planning effort aligns with homelessness and criminal justice priorities in the state. While the demonstration is not specifically named in the Rhode Island strategic framework to end homelessness, a guiding principle of that framework is that safe and affordable housing is a human right. On the criminal justice side, the governor established a Justice Reinvestment Working Group in 2015 that led to the implementation of six bills in 2017 to reform the statewide parole and probation system and provide judges with more diversion options for people dealing with mental health or substance use disorders, the target population of the project. Rhode Island also participates in the Data-Driven Justice Initiative to use data to help develop strategies that divert people with low-level offenses and mental illnesses from the criminal justice system and that employ alternatives to pre-trial incarceration.

In 2017, the site completed its feasibility analysis, and the project oversight committee recommended that the project advance to transaction structuring. The governor’s office accepted this recommendation, and transaction structuring formally kicked off in spring 2018 by creating a steering E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

2 3

committee and working groups composed of key stakeholders. These efforts were led by RICH and Social Finance, with the governor’s office providing strategic direction. Structuring conversations were paused after June 2018 when the Rhode Island legislature passed the state budget without a provision for a PFS outcome payment fund, which had been requested in the governor’s proposed budget.

Rhode Island: 2019 Progress

Rhode Island began 2019 in the transaction structuring phase. The focus of Year 3 was to secure the state as the project end payor, advance conversations on the funding needs for the project, and better understand the target population through data sharing and analysis.

The proposed budget from the governor for fiscal year (FY) 2020, released in January 2019, included $500,000 for this project. Project partners, specifically RICH and Social Finance, spent time educating legislators on the project, the PFS model, and PSH. Despite the optimism of project partners, PFS funding was not included in the legislature version of the budget. An effort by legislative supporters of the project to include funding through a budget amendment was unsuccessful, and the FY 2020

budget passed in June 2019 without funding for PFS. Stakeholders reflected that this was due in part to legislators not understanding of the PFS model as well as political realities that made funding new projects challenging. This setback caused project partners to pause project planning conversations again and reconsider their strategy for securing project end payors (e.g., exploring healthcare partners).

In contrast to the legislative setback, the healthcare working group made significant progress on data integration. In the face of this legislative setback, the site attempted to secure other, partial, end payor funding by submitting an application to the Social Impact Partnerships to Pay for Results Act (SIPPRA), a new federal funding source committed in 2018 to provide additional funding for locally developed PFS projects.12 This effort had the added benefit of spurring significant progress on data integration with data user agreements among Medicaid, RICH, and the Department of Corrections. The analysis identified the 400 costliest users of shelter, criminal justice, and healthcare services and estimated potential costs of serving them now and potential savings of this intervention to these systems. This level of integration and analysis was not possible in the feasibility phase and revealed a higher level of use than previously estimated. Although the SIPPRA application at the site was 12 The SIPPRA legislation includes plans to leverage a $100 million fund held by the U.S. Treasury Department to provide support through (1) project funding for success payments and evaluation costs and (2) feasibility studies.

These resources are intended to help the PFS model overcome the wrong pockets problem for places where the planned intervention could generate savings at the federal level by enabling the federal government to become a PFS end payor.

2 4

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

unsuccessful, project stakeholders reflected positively on this data-sharing effort as a potential model for future collaboration.

The end payor uncertainty remains the largest project challenge for Rhode Island, and project partners have limited interest in advancing other elements of project structuring (e.g., developing an evaluation plan, finalizing the service delivery model, or building the financial model) until an end payor (the state is still the likeliest possibility) is secured. Failing to secure the state as an end payor in the summer of 2019 encouraged RICH and Social Finance to adjust their staffing on the project and consider alternative options for the project. Separately, the primary champion of the project in the office of the governor left his position in 2019. Project partners ended the year with a decision to try again in 2020 to secure state funding for PFS by pursuing a more aggressive advocacy and education campaign with legislators in both houses and a grassroots campaign to galvanize support.

Transaction Structuring Sites with Launched Pilots

Anchorage, Alaska, and Austin/Travis County, Texas, continued their transaction structuring work while launching PSH pilots in their communities. Exhibit 14 summarizes project characteristics for the sites that are implementing a project, either as a pilot or in full implementation.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

2 5

EXHIBIT 14

Summary of Demonstration Project Characteristics for Sites with Pilots or Launched Programs Sites Implementing Project Pilots

Sites Implementing Projects

Austin/Travis

Lane

Los Angeles

Anchorage, AK

County, TX

County, OR

County, CA

Number to

1901 (including 45

225 (including 25

100 (plus 92 served by

300

be served

served by pilot)

served by pilot)

pilot)

Referral

Targeted street

Targeted street

Service connection

Jail in-reach

pathway

outreach

outreach

made post-release

while in transitional

housing

Service

◼ Housing First

◼ Housing First

◼ Housing First

◼ Housing First

model

◼ Intensive case

◼ Modified

◼ Intensive case

◼ Case management

management

Assertive

management

◼ Service linkages

services

Community

services

◼ Probation

◼ Landlord liaison

Treatment

◼ Cognitive behavioral

supervision

and housing-

◼ Service linkages

therapy

◼ Landlord liaison and

related case

◼ Probation

housing-related

management

supervision

case management

◼ Service linkages

Housing

Scattered site and

Scattered site

Single site; some

Scattered site

model

single site

scattered site

Project

$7,600,000

$16,300,000

$2,200,000

$12,982,446

budget

Payable

TBD

(1) Housing

(1) Housing stability

(1) Housing stability

outcomes

stability2

(2) Recidivism

(2) Rearrests

(2) Jail bookings2

(3) Jail days2

(4) Emergency

Department visits2

(5) Inpatient days2

Notes: 1Anchorage is planning to serve 190 people over the life of the project with 150 units of PSH, expecting turnover of participants. 2These will be confirmed in final contracting.

Source: Monthly calls and interviews with sites

Anchorage, Alaska: Background

Anchorage plans to use PFS to provide single-site and scattered-site PSH to improve outcomes for people who are experiencing chronic homelessness and have a history of contacts with municipal emergency services, including police. The PFS planning partnership includes the intermediary, United Way of Anchorage; potential end-payor the Municipality of Anchorage; local philanthropic funders Rasmuson Foundation and Alaska Mental Health Trust Authority; service providers RurAL CAP and South Central Foundation; potential housing providers Neighborworks; technical assistance providers 2 6

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

Corporation for Supportive Housing (CSH) and Social Finance; and evaluator NPC Research. Specific outcomes that will be linked to success payments are yet to be determined. This project is listed as a primary component of the city’s strategic plan to end homelessness, known as Anchored Home, and complements another local effort to build capacity and improve data quality, Built for Zero.

This work began in 2016 as a collaboration between the city of Anchorage and the Matanuska-Susitna Borough and was led by a team from the United Way of Anchorage, the office of the mayor of Anchorage, and consulting firm Agnew::Beck. PFS feasibility work continued until May 2018. During this time, partners received data from the Alaska Departments of Corrections and Health Services but failed to connect with local Homeless Management Information System (HMIS) data because of concerns about privacy. The United Way and the Anchorage office of the mayor remained core partners during transaction structuring as new partners were brought in to lead specific working groups.

Matanuska-Susitna Borough dropped out of the project in 2018, for a number of reasons, including that the feasibility analysis did not include data from the Borough and that Matanuska-Susitna Borough stakeholders were not as engaged in the financial and program design planning processes throughout the feasibility phase. Partners pursued planning for a project in Anchorage.

CSH was brought in as a technical advisor to focus on the design and implementation of the supportive housing program. A technical assistance grant provided resources for Social Finance to help partners with issues surrounding end payors, investors, success payments, and the overall financial model. While their outreach to philanthropic funders like the Rasmuson Foundation and the Alaska Mental Health Trust Authority resulted in funding to launch an implementation pilot, the site ended 2018 without buy-in from necessary investors and end payors.

Anchorage, Alaska: Progress in 2019

In 2019, Anchorage continued work to secure financial support for the project. In May 2019, the Municipality of Anchorage, with key support from the United Way of Anchorage and Social Finance, submitted a SIPPRA application to help fund the HUD-DOJ PFS project and expand PSH with intensive case management in Anchorage by 150 units. At the time, this was a critical component of their financing strategy; the process of applying for the grant also encouraged partners to solidify data sharing agreements that were key to the development of final eligibility criteria and prioritize a list of people in the community who met that criteria. The project was selected as a finalist, but by the end of 2019 they did not know if they won SIPPRA funds. The site also launched and began enrolling E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

2 7

participants in a pilot in July 2019. The site aims to enroll 41 people in the Pilot; exhibit 12 shows enrollment progress through the end of 2019.

EXHIBIT 12

2019 Project Enrollment in Anchorage’s Pilot

Total eligible

individuals

Total individuals

Total housing

Phase

referred

housed ever

exits ever

Transaction structuring

33

17

0

Source: Monthly calls with site

Throughout 2019, as in the previous year, United Way worked closely with the technical assistance providers at the site, CSH and Social Finance, to develop a financial strategy and refine and launch a pilot of the program model. The municipality remained the project’s strongest political support after state stakeholders became less engaged following the November 2018 elections and change in governor. One reason for the continued municipal support is the growth of the unsheltered homeless population in downtown Anchorage, which has received media attention and frustrated some residents.

Through the pilot, the site hopes to refine the participant identification and enrollment pathway. At the start of the pilot, the Mobile Intervention Team, housed within the municipality, used its knowledge of the unsheltered population to refer people it thought would fit the eventual criteria while a formal identification process was developed. During 2019, the project worked to secure additional data-sharing agreements and develop a mechanism to aggregate and analyze data from multiple sources.

From seven sources of data over the past 3 years, the Alaska Justice Information Center built a list totaling over 300 people: those who are chronically homeless; have had at least one contact with the Mobile intervention team; have 10 or more police department calls for service; have two or more arrests from the police department; have two or more medical transports from emergency medical services; have two or more intakes at Anchorage Safety Center; and have two or more contacts with Anchorage safety patrol. Social Finance worked on a de-identified version of this list to help figure out how it should be prioritized. This identification process worked outside the coordinated entry system of the site and proved a point of contention among local partners.

Other goals for the pilot were to refine the service model and to vet providers. Budget negotiations and staff turnover at the providers slowed service model refinement in the lead-up to and during the early months of the pilot. The housing provider selected during the request for qualifications process in 2018 proved reluctant to house the target population without additional compensation. These negotiations stretched unresolved to the end of the calendar year. These struggles with service and 2 8

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

housing providers occurred with the backdrop that Anchorage struggles, and has struggled for years, with supportive housing and services capacity.

In addition to the difficulty with defining the model and securing housing and service providers, the site continued to face challenges to secure end payors. The municipality remained verbally committed as the primary end payor, but the planned financial mechanism for producing the revenue for the project, an alcohol tax, failed as a ballot measure in April 2019. For the remainder of the year, other financial pathways for the municipality failed to gain traction. Additionally, significant state budget cuts not only resulted in the state not being a likely end payor but added pressure on existing services and infrastructure by decreasing funding for homeless assistance providers and connected sectors.

Despite these challenges, site stakeholders were optimistic that the project would continue and cited commitment by the municipality and longtime dedicated philanthropic groups.

Austin/Travis County, Texas: Background

Austin/Travis County plans to use PFS to provide scattered-site PSH to improve outcomes for people who are experiencing chronic homelessness and are frequent users of jails and hospitals The PFS

planning partnership includes the intermediary, Ending Community Homelessness Coalition (ECHO); a technical assistance provider that provides additional intermediary functions, Social Finance; potential end payors the City of Austin, Travis County, Central Health, and Episcopal Health Foundation; service providers Caritas of Austin and Integral Care; additional technical assistance provider CSH; and the evaluator, Abt Associates. The project will track outcomes across three areas: health, housing, and criminal justice. A pilot for 24 units of PSH launched in 2018 and was funded by St. David’s Foundation.

Lessons from the pilot informed program design for the full project, particularly outreach and partner communication and roles.

This project is part of a broader action plan to end homelessness in Austin/Travis County. Last updated in July 2018, this plan identifies the demonstration as a pathway out of homelessness that specifically addresses disparities in access to housing among specific subgroups and as a tool for increasing system effectiveness. This project is also aligned with other local efforts to reduce jail use, including new diversion efforts for people with serious mental illnesses, extended funding for mobile outreach crisis teams, expansion of the use of citations as alternatives to arrests, and early conversations between judges and advocates on ways to reform the Travis County bail system to reduce or eliminate cash bail requirements (Weber, 2020).

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

2 9

The site completed its feasibility phase in early 2016. This work included improving data systems to identify the 500 costliest users of criminal justice, homelessness, and health services. Throughout 2017, the site struggled to secure commitments from end payors because of stakeholder concerns about the necessity of the model and a desire to only pay on outcomes aligned with their priorities. In 2018, the site made progress in its negotiations with five potential end payors but continued to face difficulty managing the various concerns and motivations of these partners. Progress was also made with the launch of a program pilot in February 2018 and the selection of Abt Associates as project evaluator.

Austin/Travis County, Texas: Progress in 2019

Austin/Travis County began and ended 2019 in the transaction structuring phase. The site continued negotiating commitments from four end payors: the City of Austin Public Health department (paying on housing stability outcomes); Travis County (paying on criminal justice outcomes); Episcopal Health Foundation (paying on health outcomes); and Central Health (paying on health outcomes).13 In 2019, the site completed enrollment in the pilot (launched in 2018) and continues to serve up to 24 people at any given time (exhibit 13). The site plans to roll this group into the larger project.14

EXHIBIT 13

2019 Project Enrollment in Austin/Travis County’s Pilot

Total eligible

individuals

Total individuals

Total housing

Phase

referred

housed ever

exits ever

Transaction structuring

29

29

6

Note: Enrollment in the pilot ended in November 2019.

Source: Monthly calls with site

Lessons from the pilot have been used to refine the program model. For example, the pilot revealed the need to develop clearer communication protocols among project partners, particularly the service providers, and clarify the coordination role of ECHO. Pilot challenges also included managing the service needs of participants and the need to rehouse some participants multiple times before they were stabilized in a unit. Although it was a challenge for housing navigators at ECHO to continually rehouse clients, they noted that the focus of PFS on outcomes helped motivate the pilot to lease clients faster than usual. The need to keep people housed also meant they rehoused people that might otherwise have fallen off their radar. The sudden death of three pilot participants in quick succession 13 The fifth potential end payor, Community Care Collaborative, merged with Central Health in 2019.

14 The outcomes of this pilot population will not contribute to determining project success payments, however.

3 0

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

placed significant stress on service provider staff, which contributed to staff turnover. This, in turn, encouraged partners to focus on hiring the right staff and ensuring they had appropriate skills, training, and supports. As a result of the pilot and the lessons it provided, partners indicated they felt more confident going into the full project launch.

ECHO and Social Finance continued negotiating the contract with end payors and refining the financial model for the project in pursuit of a signed contract. Despite these efforts, some end payors expressed frustration with the PFS process and requested additional details on various contract points.

Consequently, the project did not have a signed contract at the end of 2019. Having different end payors with different concerns made it difficult to finalize a contract that would satisfy all parties, and modifications requested by one party had implications for the others. These dependencies extended to the financing agreements with the identified impact investors as well, with changes in either document necessitating review by both groups of stakeholders. The delays also created challenges for service providers that had been preparing for the full implementation throughout 2019 by hiring and training new staff to serve PFS clients. Delays in project launch kept new employees from filling their caseloads and strained the budgets of service providers, which were relying on PFS funding to pay the new staff.

Several stakeholders made efforts to maintain project momentum. The mayor of Austin publicly reaffirmed commitment to the project by the city, and the pilot funder, St. David’s Foundation, used its influence in the community to encourage other partners to remain focused.

A leadership change in mid-2019 at ECHO was also a challenge for the site. The departing executive director had been a driving force on the project and had important relationships with local partners.

With her departure, stakeholders perceived that the project lost some of its political momentum and focus. The project also faced challenges in identifying housing financing mechanisms and at the end of 2019 had identified only enough vouchers to cover the first year of the program. Regardless, the site was cautiously optimistic that the project would successfully move forward in early 2020, though concerns raised by Travis County at the end of 2019 (such as data sharing and early termination clauses) were expected to cause additional delays.

Implementation Sites

Lane County, Oregon, and Los Angeles, California, began and ended 2019 in the implementation phase.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

3 1

Lane County, Oregon: Background

Lane County is using a non-traditional PFS financial model to provide PSH (project-based and scattered site) to improve outcomes among county residents who have experienced homelessness or housing instability since their release from state prison. The PFS planning partnership includes the intermediary, Third Sector Capital Partners; service and housing provider partner, Sponsors Homes for Good (the county housing authority), and Lane County Parole and Probation; investor Oregon Community Foundation; and evaluator New York University. Partners sought the creation of a state program to be funded based on performance but were not successful in the most recent state legislature sessions. In lieu of a state program, the program has used traditional grants from philanthropic organizations and a loan from a community investor, the Oregon Community Foundation. This loan was made directly to the service provider who is repaid for performance by HUD. The target population has a medium to very high risk of recidivating, and priority is being given to those excluded from federal housing assistance options, such as sex offenders. The project measures and links success payments to housing stability at 6 and 12 months and avoidance of a new incarceration for a felony conviction at 12 months. A program pilot informed the transaction structuring phase contract development prior to full implementation, particularly the establishment of outcome metrics. This work is aligned with an existing focus to expand PSH in the county, increase support for returning residents, and reduce barriers to housing. In 2018, the county conducted an assessment that generated a recommendation that the county maintain its emphasis on increasing PSH capacity and ensure that all PSH units be used to maximum capacity (Technical Assistance Collaborative, 2018).

Lane County has taken a blended approach to the three-phase model. The site spent the first few months of the demonstration in the feasibility phase, during which it moved forward on initial financial structuring activities, including end payor and investor engagement. This component has faced hurdles because of the requirements—and timing—of the state budgeting process. In September 2017, during the feasibility phase, Lane County launched a pilot program to help orient providers to the Housing First model. The pilot demonstrated project effectiveness, helped refine the approach to the Housing First program model for the site, and tested enrollment pathways and rates. Armed with these lessons and a loan from the Oregon Community Foundation, Lane County moved on to implementation in October 2018 but continued to seek state funding necessary to fund the project fully to serve the projected number of participants in the program.

3 2

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

Lane County, Oregon: Progress in 2019

Lane County began and ended 2019 in the implementation phase. By December 2019, the project had housed 58 people (exhibit 15) and was preparing its first report for outcome payment. The project planned to house 125 people in total but has not yet met that goal due to housing supply constraints.

EXHIBIT 15

2019 Project Enrollment in Lane County

Total eligible

Total individuals

Total housing exits

Phase

individuals referred

housed ever

ever

Implementation

123

62

9

Notes: Lane County does not consider an exit from housing to be an exit from the program because participants are able to continue to participate in services and be rehoused if needed or desired. If the person is in stable housing at the 6- and/or 12-month follow up points, they are considered to be a successful housing outcome. This is true for six of the nine “total housing exits ever.”

Source: Monthly calls with site

During the year, the site balanced two priorities: supporting participants in obtaining and maintaining housing and securing additional resources for the project. The evaluation of the project continued without challenges.

Participants are referred by case managers in the Sponsors Transitional Housing Program and Lane County Parole and Probation. The site is utilizing a variety of housing options in the community, including the Oaks at 14th, a project-based supportive housing program for people reentering the community from incarceration; public housing; and Section 8 vouchers. For prior sex offenders, these participants can be housed at The Oaks, a property developed through the Low Income Housing Tax Credit program and operated by Sponsors. At this property the rents are low enough that many tenants do not receive subsidies; they may be employed, receiving Social Security Disability Insurance, or have income or rental assistance through other sources, including other, non-federal project funds through Sponsors. During project planning, stakeholders anticipated relying heavily on turnover in The Oaks for the majority of placements. During implementation, project partners found that turnover was slower than anticipated and ended up relying more heavily on Section 8 vouchers as the year progressed. This brought its own challenges, specifically struggles with securing units in the private rental market for people with criminal histories. To address this, Sponsors hired a full-time housing liaison. Homes for E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

3 3

Good, the housing provider partner, and Parole and Probation, the criminal justice partner, continued to provide services to participants.

When launching the project in August 2018, the site had secured financing only for a subset of the target number of participants. This funding was secured through a loan from the Oregon Community Foundation to the service provider, Sponsors. If the project was successful in meeting performance benchmarks, HUD, the end payor for this first part of the project, would make success payments. For this first investment, success payments would be made for housing stability, regardless of whether a participant remained in the program, and the lack of a new felony conviction. The plan for 2019 was to secure a sustainable, performance-based funding source through a state program to be administered by the state Criminal Justice Commission. Unfortunately, because of unrelated legislative matters, the Oregon 2019 legislative session closed without that bill being brought to the floor. Luckily, because of a slower-than-expected enrollment pace, Sponsors could still use the Oregon Community Foundation loan that launched the project in 2018. In August, Sponsors received a grant from Trillium Community Health Plan to address Medicaid outcomes with the target population, extending the time the project had to secure additional funding. Project partners had a number of plans for potential funding at the end of 2019, including hoping to secure the performance-based funding from the state at the Oregon legislature “short session” scheduled for February 2020.

Toward the end of 2019, it was expected that Third Sector, the intermediary for the project from its inception, would step back from its coordinating role. It would remain the financial intermediary for the purposes of processing success payments. Local partners were slated to take on coordination and project oversight in early 2020.

Los Angeles County, California: Background

Los Angeles County used PFS to provide scattered-site PSH to improve outcomes for people who have been discharged from county jails, are eligible for diversion services through the alternative court, have a history of homelessness, score in the upper third quadrant on VI-SPDAT, and have a disabling condition. This is an expansion of an existing diversion program, Just in Reach, which has a history of successful outcomes for participants. The need for this type of diversion is due in part to state legislative decisions that changed the roles of jails in the state and reallocated resources to criminal justice reform 3 4

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

(including Proposition 47, SB 678, and AB 109).15 The PFS planning partnership included co-intermediaries CSH and the National Council on Crime and Delinquency; local government planning partners the Department of Health Housing for Health and the Office of Diversion and Reentry; end payor the Los Angeles County CEO; investors UnitedHealthcare and the Hilton Foundation; housing and service provider partners Brilliant Corners, Project 180, Amity Foundation, the People Concern, and Volunteers of America; and evaluator RAND Corporation. The project links success payments to housing stability at 6 and 12 months and the number of rearrests at 24 months.

Los Angeles County began the demonstration in the transaction structuring phase and was 3

months into implementation by the end of 2017. During this early period, the site faced challenges obtaining access to criminal justice data from the Sheriff’s Department, but this did not affect the overall timeline of the project. By the end of 2018, the county had placed 182 people in housing. Success payments based on housing retention outcomes for these participants through the second quarter of 2018 totaled more than $1.3 million.

Los Angeles County, California: Progress in 2019

Los Angeles completed project enrollment in September 2019. In total, the PFS project housed 349

people at some point; by December 2019, 254 people remained in the project (exhibit 16).

EXHIBIT 16

2019 Project Enrollment in Los Angeles County

Total eligible

Total individuals

Total housing exits

Phase

individuals referred

housed ever

ever

Implementation

n/a

349

95

Notes: Los Angeles does not track referrals. Enrollment ended September 30, 2019. Housing exits are current through December 2019 and include both positive and negative exits.

Source: Monthly calls with site

During 2019, including after the close of enrollment, project stakeholders focused on supporting clients in housing. The PFS project is a part of, and is therefore nearly indistinguishable from, the broader Los Angeles County Office of Diversion and Reentry (ODR) housing program. As of July 2019, of the roughly 1,500 clients ODR had housed since its creation in 2015, about 300 of them were PFS

15 Proposition 47 alleviated jail overcrowding by reducing penalties associated with certain drug and property offenses, requiring that prosecutors charge them as misdemeanors in most cases. SB 678 provided incentives to county probation departments to reduce the number of felons under supervision that return to state prisons. AB

109, Public Safety Realignment, resulted in shifting the responsibility of corrections for people with low-level felonies from the state to local county jails.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

3 5

clients. Changes that ODR makes therefore affect both the PFS clients and the broader population it serves in PSH. Service providers do not know the difference between PFS and non-PFS clients because they are contracted by ODR to provide services and report data in the same way for both groups.

In 2018, ODR made two key changes to the program model, adding additional clinical support for clients and reducing the case ratio from 1:20 to 1:15. In 2019, service providers reported they were able to staff up and fully implement the new case ratios. The housing provider, Brilliant Corners, reported that locating and securing housing for clients was no longer a challenge. Their housing navigators have worked to establish relationships with landlords throughout the area and expand their landlord network over the years. Brilliant Corners is also able to pay landlords to hold units as an incentive (for example, pay up to 3 months of rent on a vacant unit while they work to find a client for the unit). Staff also reported that the landlords appreciated being paid directly by Brilliant Corners (instead of the tenant), and some recognized the value of the services provided through the program to stabilize the clients in housing. In doing all of this, Brilliant Corners has continued to find new units. ODR and Housing for Health continued to provide technical assistance and oversight to providers, including triaging specific cases, easing processes between County agencies and providers (such as coordinating a hand-off to the boarding care team if a client is leaving housing for a higher level of care), and facilitating the sharing of best practices (such as, available trainings on interpersonal violence or other relevant topics). Program staff found the depth of knowledge within the Housing for Health office and across providers to benefit client outcomes greatly. For example, providers would share template forms and processes with each other.

By March, the project had exhausted the HUD-DOJ funding for success payments and started drawing down from the county. In total in 2019, end payors made $2,264,900 in success payments for 6- and 12-month housing outcomes of participants enrolled in the project between October 1, 2017, and September 30, 2018. Exhibit 17 provides details on the payments. The quarters listed are at the time of payment, so the cohort shown for 6-month housing stability in Q3 2018 and 12-month housing stability in Q1 2019 were enrolled in the first quarter of project implementation (October through December 2017). The evaluator, RAND, has been engaged in the project from the beginning and independently verifies client outcomes for the success payments. RAND is also in ongoing conversations with the county and CSH to finalize details of the evaluation plan.

3 6

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

EXHIBIT 17

Los Angeles County Success Payment Outcomes Reported in 2019

Total for

Four

Q3 2018

Q4 2018

Q1 2019

Q2 2019

Quarters

Eligible participants for 6-month

39

36

42

39

156

housing retention

Participant met outcome

36

34

37

33

140

Negative exit prior to month 6

3

1

3

5

12

Positive exit prior to month 6

1

1

1

3

Deceased

1

1

Payment for 6-month housing

$218,736

$206,584

$224,812

$200,508

$850,640

retention

Eligible participants for 12-month

39

36

75

housing retention

Participant met outcome

27

27

54

Negative exit prior to month 12

8

8

16

Positive exit prior to month 12

4

1

5

Deceased

0

0

0

Payment for 12-month housing

$707,130

$707,130

$1,414,260

retention

Total payment

$218,736

$206,584

$931,942

$907,638

$2,264,900

Q = quarter.

Source: Urban Institute PFS PSH Process Evaluation

At this phase, stakeholders did not report spending much time on the PFS project activities, aligning with the decrease in time use from Q4 of 2018 to Q3 2019 as seen in the time use survey (exhibit 28).

CSH is responsible for the bulk of the ongoing PFS project work, which is generally limited to monitoring enrollment, tracking housing and criminal justice outcomes, and working with RAND to verify quarterly success payments. Monthly operations meetings became shorter as implementation progressed, and processes for collecting and reporting data typically ran smoothly. Other stakeholders, including Brilliant Corners and ODR, spend a small amount of time reporting data for the project and attending steering committee meetings. Housing for Health staff and service providers do not know how much time they spend on the project because they interact with the clients in the same way, regardless of their PFS status. One stakeholder described the project as “a fairly well-oiled machine.” Everyone interviewed spoke highly of the collaboration among the providers and across the county government, healthcare, and criminal justice systems. Although stakeholders generally reported favorably on the progress of the project and accomplishments in engaging the target population in housing, multiple people interviewed reflected negatively on the process of launching the PFS component. They viewed the level of effort it took to get to a final contract as disproportionate with the results; at least two stakeholders were not interested in being involved in future PFS projects because of their experience.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

3 7

III. Transaction Structuring Cross-

Site Themes

In 2019, five sites—Montgomery and Prince George’s County, Rhode Island, Anchorage, Austin/Travis County, and Lane County—continued their transaction structuring work. Lane County is unique among sites in this group because it moved into the implementation phase without completing all the transaction structuring work, namely solidifying financial resources needed to serve the full number of projected participants for the project.16 The goal of the transaction structuring phase is for stakeholders to agree on a legal Pay For Success (PFS) contract that establishes success payments based on agreed-upon outcomes. This is also the phase in which sites may opt to launch a pilot. Two sites—Anchorage and Austin/Travis County—operated pilots in 2019. The following section reflects the experiences of these five sites in 2019.

Sites Employed Different Project Management

Structures to Suit Current Needs

The involvement of various stakeholders ebbed and flowed with project activities, and methods of coordinating shifted as sites moved from planning to implementation. In early phases, sites often convened large groups of government partners, service providers, and other community stakeholders to provide education around PFS and permanent supportive housing (PSH) and to brainstorm various aspects of the project. During feasibility, for example, sites engaged a broad coalition of stakeholders on advisory boards and on specific work groups. In some sites, these advisory boards and work groups met regularly and moved key elements of the project forward, including data sharing and matching, identifying the target population, and planning financial strategies. As sites started narrowing in on planning, some concentrated different project aspects into work streams. For example, in Rhode Island, the project team assigned project stakeholders into housing, healthcare, and legal working groups.

Some stakeholders involved earlier became less involved. For example, a consulting firm in Anchorage, Agnew::Beck, wrote the feasibility analysis, and were less involved as the project moved into 16 Lane County partners secured a loan from the Oregon Community Foundation to start their program with the goal of securing a state program that would fund the project using a pay for performance funding mechanism.

3 8

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

implementing its pilot. Stakeholder engagement also ebbed in sites that experienced project stalls,17

including Anchorage, Montgomery County and Prince George’s County, and Rhode Island.

Sites Engaged Additional Technical Assistance Providers

for Project Support

Sites had access to technical assistance from the Council on State Governments provided by the Bureau of Justice Assistance as part of this demonstration. Some sites decided to engage additional technical assistance support. Social Finance was involved in three sites—Anchorage, Austin/Travis County, and Rhode Island—as a financial adviser and/or intermediary. In these sites, Social Finance played an important role in communicating with stakeholders and managing working groups, as well as advising on financial components of projects and data and evaluation. In Austin/Travis County and Rhode Island,18

Social Finance will also serve as the financial intermediary, meaning that it will host the PFS transaction by holding and transferring funds between investors and end payors. Third Sector was formally involved in two sites: Montgomery County and Prince George’s County as an intermediary advisor and Lane County as the intermediary. A typical role for Third Sector on projects like these is financial intermediary, as in Lane County, but it advised AIR, the intermediary in Montgomery County and Prince George’s County, on financing and contracting.

Sites also relied on technical assistance advisers for program design and implementation assistance.

The Corporation for Supportive Housing (CSH) was involved in three sites as either the intermediary (Los Angeles County) or a technical assistance provider for program design and implementation (Anchorage and Austin/Travis County). In 2018, in Lane County, the service provider, Sponsors, accessed a 1-day training from a Housing First technical assistance provider, in addition to various public resources, to help transition its partners to a Housing First model.

17 As detailed in Chapter 2. Demonstration Site Overview and Progress in 2019.

18 Pending confirmation.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

3 9

Feasibility and Transaction Structuring Phases Blended

Together

By the end of 2019, all sites had transitioned out of the feasibility phase and spent at least several months in in the transaction structuring phase.19 As noted above, these phases are defined by the HUD-DOJ grant agreements, and progressing from one phase to the next has financial implications for the project because it triggers the potential for additional grant payments.

Where the two-phase PFS planning framework originated is unknown, but it became a field norm in 2014 when the Corporation for National and Community Service used these phases as two distinct categories for grant applications to the Social Innovation Fund PFS program. Between 2014 and 2018, this program provided resources to test PFS feasibility in more than 50 locations in the United States.

Several demonstration sites reported that some tasks conducted in the feasibility phase needed to be continued, repeated, or refined during transaction structuring. This includes outreach to and education of newly elected officials and new agency heads, creation or renewal of data-sharing agreements, and reanalysis of key information like local need, population size, and the impact of different eligibility criteria on that population. Exhibit 18 illustrates how key tasks performed in the feasibility phase were continued—or repeated—in the transaction structuring phase by demonstration sites.

19 Los Angeles County began the demonstration at the transaction structuring phase because pre-grant work addressed feasibility.

4 0

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

EXHIBIT 18

Key Tasks for Feasibility Analysis and Transaction Structuring as Experienced by Sites Feasibility

Transaction

Pilot

PFS contract

analysis

structuring

period

implementation

Establish partnerships

Educate stakeholders about PSH and PFS

X

X

Identify the target population

Confirm the need

X

X

Confirm the population size

X

X

Define eligibility criteria

X

X

X

X

Establish (financial) feasibility of PFS

Assess costs and possible savings

X

X

Find non-PFS funding

X

X

Secure end payors

X

X

X

Secure investors

X

X

Define success payments

X

Establish the feasibility of implementing PSH

Confirm capacity for services

X

X

Assess housing capacity using Housing First

X

X

X

Finalize evaluation plan

Establish success metrics

X

X

Create an evaluation design

X

X

X

Involve evaluators

X

X

X

PFS = Pay For Success. PSH = permanent supportive housing.

Note: Non-PFS funding includes funding for Medicaid, housing vouchers, and philanthropy.

Source: Liberman et al., 2020

Specific examples of this include the following:

◼

In Anchorage and Austin/Travis County, stakeholders expressed that the data analysis findings from the feasibility phase were not accurate when compared with new analyses conducted during the transaction structuring phase. This includes the cost of service provision and the level of need in the target population.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

4 1

◼

In Rhode Island, new data-sharing agreements enabled the site to use individual-level data to test cost assumptions made during the feasibility phase with aggregate data. This revealed that the feasibility analysis underestimated utilization.

◼

In sites with staff turnover or newly elected political leadership—like Anchorage and Montgomery County and Prince George’s County—staff had to educate the new stakeholders about PSH and PFS to establish support and political will to continue with the project.

This fluidity between these two phases can be a benefit when it allows for the refinement of early assumptions and the creation of more accurate estimates of key PFS and service delivery metrics like the costs incurred by the target population, the costs of service provision, enrollment pathways and rates, and the level of need in a community. In fact, this is central to the role of a program pilot that may be launched during transaction structuring. This type of continued refinement can lead to better PFS

contracts, outcome payment thresholds, and service provision practice.

However, this lack of distinction between the two phases, especially in places where tasks were repeated (even for good reason), risks contributing to negative impressions of the planning process.

Partners, especially those new to PFS, that thought a task or analysis had been completed with the close of the feasibility phase may be displeased when the need to repeat or revise that work emerges in the transaction structuring phase. This can contribute to perceptions that the process to launch a PFS

project is time- and resource-intensive.

In practice, it seems that PFS projects have a long period of project design in which assumptions evolve into specific conclusions, underpinned by data, rather than two distinct planning phases.

There are early indications that blending is also happening between the transaction structuring phase and implementation. Lane County moved into implementation prior to securing the full financial investment needed to serve all intended participants and has spent time post-implementation trying to secure a sustainable, performance-based funding source through the state. For service providers in sites with launched pilots, the distinction between transaction structuring and implementation is almost one without a difference. Providers are housing and serving pilot program participants as they would post-implementation.

Despite these similarities in activities, the partnership survey revealed some significant differences in perceptions by other partners of the project in different phases in 2019. Sites in transaction structuring noted greater challenges, such as accessing clients in jail, using data for services, and collaboration more broadly. It is likely that sites in PFS contract implementation have had more time to 4 2

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

work out these challenges, particularly those related to services. In a similar way, the data show higher levels of criminal justice data and performance data use for PFS planning and implementation among sites in PFS contract implementation. Sites in implementation also reported better collaboration across systems (for example, courts, jail, homelessness, and health) and among PSH providers, as well as higher vision alignment among all partners. These findings demonstrate that the challenges and collaboration stakeholders perceive can change in positive ways as the PFS demonstration progresses toward contract implementation. It is important to note that these phase differences are also confounded with site identity, and therefore some of these observations may be attributed to differences across sites, as opposed to PFS phase. As more sites in the demonstration move to implementation, we will see whether the blending of structuring and implementation emerges as a cross-site trend.

Community Landscape and Stakeholders Shaped Project

Design

The housing and criminal justice landscapes in each community, and the main backers of the PFS

project, shaped the development of the PFS projects. Some projects emerged from the homelessness sector (Anchorage, Austin/Travis County, and Rhode Island), while others emerged from the criminal justice sector (Lane County and Los Angeles County). Although all sites included stakeholders from both sectors and sought to serve a population that touched both systems, the different backgrounds informed key decisions about program design. The Montgomery County Continuum of Care (CoC), however, did not view increased PSH capacity as in alignment with the needs of its homeless population, which contributed to its exit from the project.

This section overviews the variation in site approaches to program design.

◼

Defining the target population: Depending on the motivation for the project and the end payors on board, a community might prioritize homeless history and vulnerability (through the VI-SPDAT or other mechanisms) or criminal justice history and likelihood of recidivism (through recent jail stays or validated assessments such as the Coordinated Assessment Tool for Reentry). Demonstrating that these clients also have high health needs helped bring the Alaska Mental Health Trust on as a pilot funder in Anchorage and Central Health on as an implementation end payor in Austin/Travis County.

◼

Establishing identification and referral pathways: The eligibility criteria inform when and how clients will be identified for services, through more traditional homeless street outreach or E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

4 3

through jail in-reach. In Anchorage, first responders, including a mobile intervention team that is part of the police department, identify eligible clients. Austin/Travis County also used street outreach during its pilot and plans to do so for the contract implementation; it is also the only site so far enrolling in this project through the local coordinated entry system.20 In Los Angeles County, public defenders refer clients, who are screened; if they are found eligible, a judge orders them to the program. Service providers then connect with clients before they are released from jail, and clients must report back to the court and probation throughout the program.

◼

Recruiting and working with service providers and creating the service model: In Anchorage, Lane County, and Los Angeles County, the PFS project contracted with providers that do not typically provide PSH or serve a homeless population. These providers varied in their approach to clients and in some cases needed Housing First training and/or support. Some service providers in Los Angeles County, for example, had primarily been mental healthcare providers or reentry providers; in collaboration with traditional homeless assistance providers, stakeholders viewed this variety of perspectives as beneficial in promoting the best outcomes for all PFS clients.

◼

Finding housing for the project: Some projects are considering using CoC vouchers or CoC

apartment complexes to house clients in this project. The Rhode Island project expects to receive 30 to 35 CoC units. In Anchorage, because of the tight housing market, clients are sometimes placed within a CoC-funded provider building. In both Anchorage and Austin/Travis County, the expansion of PSH is integrated with CoC plans to end homelessness and scale up PSH capacity.

◼

Determining outcome metrics for repayment: Both end payors and potential investors can influence the selection of outcome metrics linked to repayment. They will be most interested in metrics that are aligned with their own mission. For example, healthcare system end payors may be more interested in paying on health outcomes than paying on improvements in housing stability or criminal justice involvement. Although the contract is not final in Austin/Travis County, healthcare system end payors will likely pay on health outcomes for clients while other end payors pay on the other metrics.

20 Coordinated entry systems are required in local continuums of care for the purposes of coordinating and prioritizing the distribution of resources, to provide a standard process for deciding who receives what, and to provide clear and consistent points of access for people seeking assistance.

4 4

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

All these decisions can cause tension between stakeholders as projects get off the ground. It can be difficult for all players to get on board as some factors are prioritized over others. The bullets above illustrate how partners, especially end payors, can drive key project decisions.

Transaction Structuring Challenges

Every site in this demonstration has experienced challenges during the transaction structuring phase.

Some challenges are due to work required by the PFS model, while others result from the work necessary to increase PSH capacity and would likely be faced regardless of the funding source.

Challenges related to finding and securing buy-in from organizations to invest in the program and others to agree to repay those investors are unique to PFS. In the demonstration, most sites leverage existing resources like vouchers to fund the housing component of the work. Challenges related to securing housing resources are not unique to PFS-funded efforts to expand PSH, but, notably, they are challenges the PFS model is supposed to help overcome. Sites faced challenges related to evaluation design that could have occurred for any outcome evaluation effort; however, it is not unreasonable to assume that because the evaluation findings trigger financial payments, the presence of the PFS model contributed to these challenges.

Finding and Solidifying Financial Partners

Led by project intermediaries and their advisers, several sites (Anchorage, Austin/Travis County, and Rhode Island) adopted the same similar approach to sequencing: secure end payors and finalize project details before securing investors. In places where some organizations had different funding role options (for example, investor, end payor, or supplier of catalytic grants for pilots or full implementation), conversations could advance at the same time. However, at least one end payor at a site (Austin/Travis County) indicated that it would have preferred that investor terms be secured prior to signing on. Sites were challenged with how to best sequence these processes to optimize the leverage they had to get sufficient capital raised and investors signed on.

Stakeholders in the four sites (Anchorage, Austin/Travis County, Lane County, and Rhode Island) expressed surprise and frustration at the effort required to of secure financial partners. This came through particularly in comments submitted through the partnership survey. The time commitment needed to structure a PFS deal was the most common criticism of PFS. Some stakeholders did not believe the additional time it took added value to project implementation. One financial partner found E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

4 5

the contract agreements too complex, and a service provider felt the funding delays kept it from being able to serve people who needed support urgently.

The holdup on funding and ability to begin full project implementation has been extremely frustrating. People who should’ve been housed months ago through this program continue to languish and die on the streets.

—Direct service provider for target population

LOCAL CHAMPIONS ARE KEY TO SECURING BUY-IN FROM FINANCIAL PARTNERS

Local government partners have driven projects forward in Austin/Travis County and in Anchorage. The mayor and the city council in Austin have publicly expressed support for the project, which has been cited as beneficial by other stakeholders, and helped one organization overcome hesitations to participate. The Municipality of Anchorage has been a backbone of the project, and local stakeholders were excited by the prospect of the city funding PSH for the first time. Both sites relied on the enthusiasm and experience of local partners and funders to launch pilot projects. Both sites are also hoping to find local investors to sign on to their PFS contracts in implementation.

Rhode Island and Montgomery County and Prince George’s County also continued with general local partner collaboration to explore how to secure commitments from end payors and investors and find service providers. Although the office of the Governor of Rhode Island had been an important political backer since the project’s beginning, the project continued struggling to gain traction with state legislators. This complicated efforts to move state funding forward. By 2019, leaders in Montgomery County and Prince George’s County did not agree on key project parameters due to differences in characteristics of people experiencing homelessness and existing service-provider and housing capacity. At the same time, department-level stakeholders in Montgomery County were becoming less convinced that this was the right path for their county, and leadership changes due to the 2018

elections created a need to re-establish buy-in with the Montgomery County endpayors. This led to a reduced willingness to proceed with the project. After Montgomery County left the demonstration in September 2019, stakeholders in Prince George’s County had more autonomy to design a project that fit into their homeless assistance landscape.

4 6

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

BRINGING HEALTH PARTNERS TO THE TABLE WAS CHALLENGING

The sites’ feasibility analyses and broader literature demonstrated that the people likely to be served by the PFS project have high healthcare costs. Because they are unlikely to have adequate access to medical care while experiencing homelessness and/or residing in jail and can face harsh conditions in these places, they are likely to fall back on expensive emergency care, including ambulance rides and avoidable hospitalizations. In fact, in all four of the feasibility reports that included health data in the cost analyses, healthcare costs were the highest costs incurred by this population (Anchorage, Austin/Travis County, Rhode Island, and Montgomery County and Prince George’s County, Maryland).

Yet sites have largely been unsuccessful in getting health payors to the table. This could change, of course, as sites move to solidify their contracts in 2020. So far, only Austin/Travis County has firm commitments from health entities, including Central Health and the City of Austin public health department. The decision by the site to have each end payor pay on a different outcome metric helped secure a cross-sector group of end payors, but this came with its own challenges. The deal structure is complex, and coordinating across these different partners has contributed to delays in contracting (not complete at the end of 2019). Some Austin stakeholders questioned whether it would have been better to pay for PSH outright but noted the benefits of having many systems at the table to meet the challenge of homelessness. These benefits included addressing the “wrong pockets” problem and acknowledging that the city/county had yet to fund PSH at the scale proposed in the demonstration through more conventional means.

PFS planners face several challenges when trying to involve healthcare stakeholders in PFS

transactions, especially when those systems have few incentives to reduce healthcare use and spending (Skopec, 2018). In this demonstration, potential health end payors saw risk in miscalculating how program participants would use the healthcare system and were concerned that costs might not go down significantly, despite participants being in housing and having access to appropriate and preventative care. This was the case in Austin/Travis County but was also raised in Anchorage. In Anchorage, the project was asked to provide the potential health end payor with participant-level data so it could see whether the targeted participants would save the health end payor resources. In Texas, at least, local entities are paying for these services because the state has not expanded Medicaid.

Additionally, around one-half of the target population for the Anchorage project is Alaska Native, and their healthcare costs are covered by the federal government through the Indian Health Service. In states that have expanded Medicaid, 21 the health costs incurred by the target population may be 21 Of the six states with demonstration sites, five have expanded Medicaid: California, Oregon, Maryland, and Rhode Island effective January 1, 2014, and Alaska effective September 1, 2015. For more information on Medicaid E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

4 7

covered by the federal government instead, disincentivizing local actors from signing on as end payors.

But, healthcare system payors are still being considered in Anchorage and in Prince George’s County.

SECURING THE STATE GOVERNMENT AS AN END PAYOR PROVED DIFFICULT

Three sites (Anchorage, Lane County, and Rhode Island) have attempted to pursue a state funding source as an end payor. All three have found, however, that legislatures and their budgeting processes are challenging to navigate. Budgeting cycles are long—a year in Rhode Island and two years in Oregon—and can be highly unpredictable because of unrelated political dynamics. Opposition to a bill by legislators, including funding for a PFS project, might be due to party loyalty or dislike of the sponsor or champion of the bill. In Oregon, the state legislature was mired in conflict in 2019 as some legislators walked out and refused to vote on any bills in May and again in June.

Elections can upend collaboration as well. When officials or administrations turn over, PFS

stakeholders have to invest time in fostering new relationships and educating legislators and staff again and face uncertainty about how this work fits into the priorities of a new administration. In Anchorage, for example, the state had been considered a potential end payor for the project from the beginning, particularly because the state has a unified corrections system. A governor had set aside $1.5 million in his outgoing budget proposal for the intermediary in the PFS project, but the election of a new governor prevented the funding from being provided. Project members contacted the new administration and set up meetings, but by the summer 2019 budgeting process, it was clear state funding would not be appropriated for this project. Instead, the governor severely cut social services spending, including in the areas of homelessness and health care.

State budgets, and their timelines, are also not easily compatible with PFS funding. Oregon state legislators have previously rejected PFS as a funding mechanism. As a result, the project is not pursuing a typical PFS structure with the state and is instead looking to establish a performance-based state program. A stakeholder in Rhode Island wondered whether savings would accrue in time to meet the requirement for a yearly balanced budget. Although 2019 was the third year in a row that PFS funding was proposed in the budget from the governor, legislators in Rhode Island signaled confusion around the concept of PFS and committing funding for a new project.

expansion, see: https://www.kff.org/medicaid/issue-brief/status-of-state-medicaid-expansion-decisions-

interactive-map/

4 8

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

OPINIONS ON THE NEED FOR PAY FOR SUCCESS TO SCALE PERMANENT SUPPORTIVE HOUSING

VARIED ACROSS STAKEHOLDERS

Government partners, including project planning partners and entities that may become end payors, and service providers expressed very little concern around the financial risks associated with providing PSH. All sites had at least some PSH capacity in their community, even if it might have varied from these proposed projects in terms of target population or scale. Representatives of local government departments and service providers in the community saw this project as typical of the programs they fund or operate, including other homeless assistance or reentry programs. PSH in particular has a documented track record from a long history of studies. Some expressed that they saw risk in not moving the project forward and not creating housing for this population.

One argument for PFS is that it shifts the financial risk of the program from governments to investors: if the program does not produce intended outcomes, the investors are not paid back. This is no longer an attractive argument if the government does not see much risk in operating the program—

as was the case for some partners in several sites. Pima County, for example, dropped out of the demonstration to fund PSH outright. Furthermore, some partners even expressed concern about the risks created by the PFS financial mechanism. In Austin/Travis County, Montgomery County, and Rhode Island, budget staff worried about the risks of PFS including appropriating new money as a debt and the use of the messaging around “cost savings” that they did not expect to be realized.

PROJECTS MOVED AHEAD WITH PHILANTHROPIC INVESTMENTS

Although no formal PFS contract had been signed in Lane County, Austin/Travis County, or Anchorage by the end of 2019, all three sites had progressed into implementing pilots or full projects. Each site had an acknowledgement of the need for PSH and deep local commitment. Project stakeholders believed that they could start serving clients and working through program design considerations while straightening out the financial details.

Austin/Travis County, Anchorage, and Lane County secured philanthropic investments to fund their pilots: St. David’s Foundation in Austin, Rasmuson and the Alaska Mental Health Trust in Anchorage, and Meyer Memorial Trust in Oregon. These are grants; none of these entities are interested in being repaid as investors would. Stakeholders at these organizations who were interviewed seemed very interested in the projects themselves and supported them because they aligned with the mission and priorities of their organizations. Although these stakeholders were open to future financial involvement, the PFS mechanism was not the attracting feature. In Austin/Travis County and in Lane County, St. David’s and Meyer provided flexible funding, allowing the project to use the resources in E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

4 9

whatever way would be most useful. Stakeholders found this particularly valuable to the project given challenges in advancing the PFS contract and transitioning from the pilot to the full project. In Anchorage, one stakeholder expressed interest in recycling the capital back into the project, rather than being repaid. Staff at these organizations see themselves as somewhat removed from the project because they are not actively involved in design decisions or service delivery. However, staff at St.

David’s have become more involved in contract negotiations as that negotiation process has stalled.

They have used the role of the foundation with the pilot—continuing to fund it through months of stalled negotiations—as leverage to encourage end payors to sign a contract.

In contrast, the Rhode Island project has stalled each time attempts to secure the state as an end payor have fallen short. After the disappointing outcome in the Rhode Island legislature, the site briefly considered securing a match from alternative payors in the state (such as hospitals, insurers, or philanthropies). However, this did not yield any promising leads, and site progress remained contingent on securing the state as end payor.

Securing Housing Resources

When planning their projects, most sites decided the PFS investment resources would finance the service components of the project while housing would be secured by leveraging existing housing assistance in the community such as tenant-based vouchers or slots in existing housing projects. At the end of 2019, no sites were pursuing new construction funded by this project. Housing strategies for most sites were scattered-site approaches, meaning projects needed to secure tenant-based vouchers or units in the community. Securing these housing assistance resources was one of the most challenging factors for several sites operating pilots or in implementation. Even in the one site with dedicated project-based housing, Lane County, slow turnover required the project to rely more heavily on other local housing resources than planned.

One strategy that multiple sites pursued was accessing tenant-based vouchers through state or local housing authorities. In Austin/Travis County, the project sought to gain access through a homeless preference for vouchers, where people experiencing homelessness are prioritized on waiting lists when vouchers become available. A voucher freeze and competing preferences for other populations meant that vouchers were not available at the required pace. Anchorage pursued a similar strategy, requesting a preference for the project from state and local housing authorities, but was unable to secure a commitment. The state housing authority encouraged the project to put participants in the queue for existing set-asides and preferences, instead of creating a dedicated pathway for project participants.

5 0

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

Additionally, the project was unsuccessful in trying to secure unused 811 vouchers held by the state housing authority. In Lane County, one of the primary partners from the inception of the project was Homes for Good, the local housing authority. Homes for Good provided the project with Section 8

vouchers and public housing units, but the project was reliant on turnover for access to vouchers and units, which meant that the flow of vouchers and units was unpredictable.

When vouchers were not accessible, sites pursued and brought on other housing resources as they were identified. This highlighted the importance of having backup plans for housing resources. In Austin/Travis County, the site secured funding from the Austin density bonus fund. Originally used for rental assistance for the pilot, the site secured an additional $950,000 from this fund which would cover the full anticipated voucher need for the project in 2020 while waiting for vouchers. In Anchorage, the project continued to negotiate with a community-based housing provider to access units and placed project participants in existing PSH projects. In Lane County, when turnover in the Oaks, the project-based housing dedicated to the project, slowed, the site considered placements in a new tiny homes development and increased the reliance on Section 8 vouchers for the project.

Los Angeles County did not face the challenge of securing local housing resources because the investor funding paid for rental assistance for clients enrolled in the project to be administered through the existing Flexible Housing Subsidy Pool in the county.22 But the housing assistance created for the project was in the form of tenant-based rental assistance, and multiple project partners indicated that project-based options would be beneficial, both for the clients (having crisis support and/or a clinician available on site) and for case managers (who would not need to crisscross Los Angeles County to see clients). Los Angeles County only had around a couple dozen project-based PSH beds available, but was looking to add this capacity in the future (this would not impact the PFS project, but future ODR clients).

Designing the Evaluation

Evaluation design during the transaction structuring process, and even into implementation, was an ongoing and frequently changing process often separated from conversations about defining success metrics, As noted in our report covering Year 2 of the demonstration (Liberman et al., 2020), none of the sites were planning to pay for impact. Instead, sites have considered metrics based on observed outcomes of program participants, such as retention in housing after 6 months. Sites may use an outside evaluator to validate the data analysis that determines how many participants met the metric, but 22 For more information on the Flexible Housing Subsidy Pool, see https://brilliantcorners.org/fhsp/.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

5 1

payment is not contingent on the results of a quasi-experimental study comparing the outcomes of participants with a comparison group.

Respondents across sites described shifts in evaluation methodologies and outcomes to be measured. In Austin/Travis County, the request for proposal seeking an evaluator listed three evaluations: a process study, an outcomes study, and a “learning evaluation.” In Anchorage, the evaluation team was primarily working with technical assistance provider Social Finance, and the evaluation design was primarily driven by what data were available. For example, Anchorage spent a lot of time discussing how to understand stability for people after they exit the program. Data systems for their programs are not designed to track people after exit, only during their time in a program unit. This presented a challenge for the evaluation team for understanding housing outcomes for participants who exit the program. The evaluator also played a large role in the SIPPRA application for the site; if successful, the site anticipated the requirements of that federal funding would need to be reflected in the final evaluation design (for example, measuring health outcomes and constructing a quasi-experimental comparison group). In Montgomery County and Prince George’s County, Maryland, the evaluator and the intermediary, an experienced program evaluation organization itself, engaged in multiple rounds of design review and iteration that was still in process at the end of 2019.

As in prior years, respondents in sites that remained in transaction structuring in 2019 expressed concerns about creating comparison groups. In Anchorage, respondents involved in evaluation design expressed concerns that there were not enough people to create a comparison group. In Austin/Travis County, the decision was made to go with a pre-post study, and the evaluation design team was clear with community stakeholders that the evaluation could not attribute any findings to the intervention.

Pilots Represented an Opportunity to Refine Program

Model and Demonstrate Effectiveness

Two sites, Austin/Travis County and Anchorage, operated pilots in 2019. At the end of 2019, Prince George’s County indicated that it was planning a pilot for 2020, but the launch would be contingent on funding. Prior to the start of implementation, Lane County and Los Angeles County completed pilots.

Respondents across these sites reflected that pilots offered opportunities to refine program design and operations and to demonstrate the effectiveness of the model. Pilots also helped communities start to improve outcomes for the target population and maintain momentum for the full project.

5 2

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

Pilots Helped Refine Program Design and Operations

Pilots also offered sites the chance to learn as they went, particularly around outreach and engagement pathways. In Anchorage, the project did not have an operational referral pathway when the pilot started. At the outset, the pilot used a mobile outreach team to identify potential participants. This created conflict with the local coordinated entry system; it did not always have these potential participants in their system, or if they were in the system, they would not be the people prioritized by the coordinated entry policies. As the pilot continued, the participant identification and referral process was refined, and the final process involved a prioritized list generated from administrative data.

Pilots also provided sites with an opportunity to revise program components. In Austin/Travis County, providers gained experience with a new population, one that was more vulnerable and needed more support than was expected. Providers had to rehouse people at high rates—for 29 participants, there were 53 housing placements. Austin/Travis County also changed its services model to ensure better access to care after realizing that participants needed and used services, particularly substance use services, at a much higher rate than was anticipated. Anchorage had a similar experience with services and revisited the services budgets for the project as a result.

The deaths of participants during pilots also informed program design and operations. The Austin/Travis County pilot experienced a high mortality rate among the target population, which negatively affected staff well-being and led to some turnover. The project made changes to operations while it was still small—24 participants were served in the pilot in 2019—compared with the 250 people the project is hoping to serve when full implementation launches. These changes included making sure staff had the right skills, training, and supports to handle the emotional stress of serving this population.

Project partners had previously considered using a less intensive Assertive Community Treatment model, but this experience of the pilot underscored for partners the need for clients to benefit from a fully-implemented Assertive Community Treatment model.

Pilots also provided sites with the opportunity to expand data collection capacity prior to full implementation. In Austin/Travis County, the pilot struggled with sharing client-level information across systems, and a dry run of data by the evaluator revealed the need for some technical assistance from the evaluator. In Anchorage, early work to create a target participant list with administrative data resulted in troubleshooting between project partners about how to create a prioritized list. It was beneficial to identify and refine these procedures when the participant pool was small, rather than it happening during implementation.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

5 3

Pilots Helped Demonstrate Ability to Implement

Pilots provide an opportunity for sites to demonstrate the effectiveness of the model, potentially creating a proof point for potential investors or end payors. In Austin/Travis County, respondents reflected that the pilot helped them prove that they could house “high barrier” people quickly. The pilot was able to house two to four people per month, making the case for a consistent enrollment pace. In Anchorage, the pilot was seen as an opportunity to persuade potential partners—including the state and local housing authorities and a local housing partners—who wanted to see that the project was working before committing a large number of units.

Pilots were also seen as an opportunity to test performance and ensure that outcome metrics for payment were achievable. In Austin/Travis County, respondents indicated that the pilot helped them set reasonable goals that would minimize risks for investors. In Anchorage, where pilot participants would be included in the evaluation findings but not in outcome measures, the pilot was seen as a way to set reasonable expectations for all project partners, including service providers. The project partners planned to use service provider performance to decide how to allocate the number of cases to providers during implementation.

But what PFS is doing from a performance outcome standpoint; it is already changing the rhetoric in the community around how PSH should be delivered. It’s really upping the ante because it’s based on firm outcomes and there’s not a lot of wiggle room for excuses, and that’s amazing. It’s affirming a whole new dialogue in the community around existing PSH and creating new PSH opportunities, or even looking at existing outflow and how we can be more aggressive about using the dollars well and expecting really high performance.

—Housing provider for target population

5 4

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

IV. Implementation Cross-Site

Themes

In the implementation phase, all partners perform the roles agreed upon in the Pay For Success (PFS) contract, including housing and service provision for the target population, measuring outcomes, completing success payments, and evaluating the project. In 2019, two sites—Lane County and Los Angeles County—continued to implement their projects. Additionally, two sites—Anchorage and Austin/Travis County—operated pilots, implementing Permanent Supportive Housing (PSH) for the target population. The themes discussed in this section are those identified in the two sites in the implementation phase, as well as those seen in the two sites with pilots.

Sites Added Coordination Mechanisms for Stakeholders

Involved in Permanent Supportive Housing

Implementation

Once implementing full projects or pilots, sites added another component of coordination: meetings (often weekly) between service providers and core project members. Sometimes set up like case conferencing meetings (Anchorage) or more general convenings (Los Angeles), these meetings allow people directly interacting with the target population to troubleshoot challenges and share best practices. They are also able to raise up any barriers or ideas to the core project management team. In Los Angeles County, this led the project to decrease caseloads and increase clinical support for clients.

This variation in project coordination structures is discussed in more detail in the time use section of the report.

Sites Reported Successful Lease-Ups and High Rates of

Retention

Across sites, respondents indicated that participants successfully enrolled, leased up, and, for those who accessed housing, remained housed. Los Angeles County completed full enrollment in the project during 2019. Lane County met enrollment goals for 2019 and helped 58 people enter housing. Austin/Travis County and Anchorage met pilot targets and were successful in moving participants into new housing.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

5 5

Austin/Travis County respondents noted high rates of re-housing but were eventually successful in housing participants despite it taking a couple of attempts.

Respondents across sites noted that they felt their programs were improving the lives of participants and producing positive outcomes for the community. One respondent summarized it, saying: “That’s the good thing about the end result—because when we see a person who has 18 arrests in 2 years, and then we had them for a year and they haven’t gone back, that’s like, wow.” Despite this, in the one instance that an evaluator made an observation of outcomes so far, the evaluator indicated that they had not seen a significant difference between the treatment and control groups. As noted previously, for most sites this would not make a difference in their PFS transactions because to date, contracts for sites have end payors paying for outcomes, as opposed to impact.

Implementation Challenges

Although all sites reported successes in housing the target population and said they observed improvements in the lives of participants, achieving these results did not happen without challenges.

Common challenges across sites included the need for greater capacity to serve “high-need”

participants, difficulty locating housing and working with landlords, developing skills in service and housing providers new to Housing First or serving the target population, and differentiating roles between service providers and criminal justice partners. As with the challenges faced in the transaction structuring phase, some of these challenges stem from or are heightened by PFS itself, and others could arise in any effort to expand PSH. These challenges are consistent with challenges experienced by PSH

projects not funded through PFS.

Building Capacity to Serve “High-Need” Participants

Multiple sites reported refining their program design or services model to reflect the higher-than-expected needs of program participants. In Austin/Travis County, Anchorage, and Los Angeles County, service providers added or increased clinical capacity and lowered caseloads to allow for sufficient service delivery for participants. This was the result of providers expressing that participants needed more support than was anticipated when the program was designed. Additionally, Austin/Travis County faced the challenge of building support for staff at service providers that had no experience with participant deaths. In Lane County, the project made the decision to hire a housing location specialist 5 6

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

after struggling to place participants with criminal histories in private market units with Section 8

vouchers.

Building Capacity in New Service Providers

In every site in implementation or operating a pilot in 2019, respondents indicated that the project engaged housing and service providers new to PSH or new to serving the target population. Anchorage, Austin/Travis County, Lane County, and Los Angeles County all had service providers new to Housing First and PSH and attempted to build provider capacity and skills. For example, the service provider in Lane County engaged a Housing First trainer and made the 1-day training available to the other project partners in the community, but noted more grant resources and staff support are needed at launch and throughout implementation for a seamless transition. Despite efforts to build Housing First and PSH

capacity across sites, respondents indicated that some providers new to the model fell short, requiring participants to be sober or to participate in services before housing them or expressing a desire to use the criminal justice partner to enforce compliance.

Finding Housing and Developing Relationships with Landlords

Respondents consistently identified locating housing, working with landlords, matching participants to units, and having to rehouse people multiple times as common housing challenges. As most sites were operating scattered-site programs, they were reliant on private-market housing.

Anchorage, Austin/Travis County, and Lane County reported challenges with finding housing and working with landlords. In Austin/Travis County, respondents indicated that the site struggled to place participants in units managed by large companies. Respondents also indicated that the units held by landlords willing to lease to participants with criminal histories were in neighborhoods where participants might struggle to be successful because of prevalent substance use by neighbors and guests. In Anchorage, the location of units was also raised as a challenge, with respondents worried about concentrating participants in one building. Project partners were not sure whether there were enough units held by landlords willing to work with the target population to meet project enrollment goals. These concerns were amplified by ongoing negotiations with a community-based housing partner.

This partner was considering setting aside 5 to 22 units for the project, but was new to serving the target population and was worried about the cost of damage to units and proposed a per unit per month

“service” fee.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

5 7

Strategies to address these concerns usually involved using or hiring staff dedicated to recruiting landlords and locating housing. In Austin/Travis County and Los Angeles, landlord recruitment and housing location were coordinated or centralized in one agency with dedicated housing navigation specialists. These specialists could recruit landlords across multiple programs and service providers. In Lane County, partners created a staff role for a full-time housing liaison at Sponsors. This person did outreach to home associations and community groups and worked with landlords to encourage them to consider renting to participants despite their histories.

Other housing challenges included matching participants with units and rehousing people if they left their unit, were evicted, or otherwise terminated their lease but remained enrolled in services.

Respondents in Austin/Travis County, Lane County, and Los Angeles identified matching participants with units as a challenge. Units in some neighborhoods were perceived as not being a good fit for participants because of the nature of participant criminal offenses, including sex offenses. In Austin/Travis County, respondents identified rehousing people as a primary challenge for the project, with 29 participants accounting for 53 housing placements. Most people had only one or two housing placements, but 34 percent had between three and six housing placements. Los Angeles County identified matching participants with a unit as a challenge during the rehousing process, causing some participants to return to interim housing (shelter) while awaiting a new placement.

Blending Service Provider and Criminal Justice Roles in PSH Implementation By design, the demonstration requires collaboration and coordination between local housing and criminal justice agencies to implement a program using a Housing First approach. Key to a successful collaboration and, ultimately, a successful program, is the establishment of clearly defined roles and responsibilities. In a Housing First model, participation in services is voluntary, and service providers do not compel participation in treatment programs. The two sites in the implementation phase, Los Angeles County and Lane County, are enrolling participants who are on probation and parole. In both projects, clients may exit supervision while still enrolled in the project, but all are connected for at least some time. Supervision requirements in these two sites have caused tension with fidelity to the Housing First model.

In Los Angeles County, clients are court-ordered to the program if they choose to accept it as an alternative to incarceration; if they leave housing or are not compliant with their service participation requirements, they are violating terms of their conditional release and must report to the judge in charge of their case. The Office of Diversion and Reentry Housing program partners with a designated 5 8

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

Department of the Superior Court and a set of Deputy Probation Officers who work with ODR clients differently than other people under supervision. People interviewed were very clear that the primary goal of the program is to stabilize clients in housing and operate within a harm-reduction approach—

including the probation officers who supervise clients, who even meet clients in housing rather than having the clients report to their office.

However, some traditional probation and supervision aspects remain. The judge orders participants to comply with their treatment and the terms of the program and remain in housing. Participants are subject to search and seizure without cause as probation clients. Service providers report to the court on client progress and treatment; there have been instances where case managers leverage court authority to compel compliance with program activities. The four providers differ somewhat in their approach to working with justice-involved people and the criminal justice requirements that accompany them. Similarly, in Lane County, even though the service provider shifted to a Housing First approach and the provider no longer requires service participation, respondents indicated that the parole officer can be the “stick” for the service provider when compliance is needed. In a PSH program that has fidelity to a Housing First model for building and maintaining participant-service provider relationships, the service provider reporting on service participation, as seen in Los Angeles County, or the ability to leverage court or probation officers for compliance, as occurs in both implementation sites, would not normally occur.

Classifying Participant Exits

During pilots and the implementation phase, projects faced several challenges related to data. One that stood out and had implications for measuring outcomes was how to categorize housing exits. Exits are typically defined in a PFS contract, but real-life circumstances can introduce ambiguity in their interpretation. This is especially true if there are circumstances under which a participant could leave housing but remain connected to services, or otherwise regain housing. In Los Angeles County, project partners discussed if participants who moved to interim housing23 between housing placements could not count as a “negative” exit. Ultimately, they decided that the contract was clear: even if participants who left housing were re-housed later, they were counted as a negative exit. In Lane County, however, the outcome measure is housing stability, so a participant exit can still be considered a positive outcome 23 Los Angeles County uses the term “interim housing” to describe transitional housing in the community. This is statutorily considered a homeless situation according to the HUD definition of homelessness. See:

https://files.hudexchange.info/resources/documents/HomelessDefinition_RecordkeepingRequirementsandCriteri

a.pdf.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

5 9

if the person is in any permanent living situation at the time of outcome measurement. This question of how best to deal with and classify short-term instability remains open in the PFS field.

6 0

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

V. Benefits of Pay for Success

Participation in the demonstration has provided sites with opportunities such as creating and improving partnerships and existing data systems. For projects that have moved into the implementation phase, additional benefits such as an increase in the number of people housed and an increased focus on the outcomes for those people have been realized.

Pay for Success Strengthened Partnerships and

Collaboration

Every site reported increased community collaboration in support of the target population as an impact of the Pay For Success (PFS) demonstration. All participating sites had existing PSH capacity, but participation in the demonstration allowed local actors to scale efforts and bring new partners to the table. This is consistent with reported findings from the first 2 years of the evaluation (see Liberman et al., 2019 and the Year 2 report: Liberman et al., 2020). Now midway through the demonstration, projects are solidifying working groups and decisionmaking structures and continuing to break down silos that affect other community work. We have found that who is at the table shapes how a project develops in important ways. Nearly all sites have struggled, however, to attract and sign on end payors and investors, impeding progress.

Some stakeholders attribute the increased community collaboration directly to the PFS project, which brought in new collaborators across sectors and heightened awareness of the target population and Permanent Supportive Housing (PSH). In Los Angeles County, however, some stakeholders believed that this level of collaboration was not new for the site and was already part of the work necessary to operate the broader ODR program. Still, even stakeholders in places with existing PSH

capacity reported benefits from this project.

Breaking Down Silos between Sectors

All sites reported that PFS planning processes led to new and/or improved relationships among local stakeholders. People met and formed connections at planning meetings, which in some cases facilitated relationships between leaders of different organizations or departments. Consistent with respondents in prior years, one respondent said that once they knew a person they could reach out to in a different department, they were more likely to seek opportunities to collaborate. In Prince George’s County, for E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

6 1

example, stakeholders reported that participating in planning processes for the PFS project strengthened relationships across county departments, from budget to corrections and social services.

In Anchorage, one stakeholder remarked that the collaborative work done for the PFS project was

“unlike anything else” they had witnessed.

Raising Awareness of the Target Population and Possible Cross-Sector Solutions Many stakeholders reported that one benefit of the PFS planning processes was education, an opportunity to learn new aspects of—and the depth of—the need for housing solutions in their communities. In both Montgomery and Prince George’s Counties, interview respondents discussed how the feasibility analysis and planning discussions shed light on the housing needs of the overlapping populations served by the criminal justice and homeless assistance sectors. One respondent noted that other issues surfaced during these discussions, including the need to ensure that people continue to receive their medication after being released from jail.

We’ve been given time and money to communicate with each other, create data agreements, and find vouchers for this population.

—Government partner

In Rhode Island, stakeholders felt that creating data-sharing agreements and finding housing resources for this population raised awareness of community needs and laid the foundation for other projects (such as connecting community mental health centers with homeless assistance providers). In Austin/Travis County, the PFS project brought new partners to the table to provide and fund PSH. In Los Angeles County, although public sector collaboration had existed previously, the PFS project brought private and non-profit organizations (namely, the two investors for the project, UnitedHealthcare and Hilton Foundation) into funding these solutions.

Strengthened community collaboration and data sharing were reflected in partnership survey responses as well. As shown in exhibit 19, survey respondents indicated that their organizations were aware of and committed to addressing the needs of the target population (see “organizational role and awareness” and “organizational level of involvement”). Most reported favorably on cross-system collaboration (including between law enforcement, the homeless assistance system, courts, jails, and 6 2

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

health and behavioral health services) and indicated that collaboration (including data sharing, formal agreements, and meetings) increased during 2019.

EXHIBIT 19

Partnership Survey Results on Partnerships and Collaboration

Mean

Index name

(1-4 scale)

Partnerships (roles, planning, and infrastructure)

Project roles and progress

3.4

Organizational role and awareness

3.7

Organizational level of involvement (self)

3.6

Vision alignment of partners

3.3

Community and provider support and capacity

3.2

Collaboration

PSH provider collaboration

3.2

Cross-system collaboration

3.3

Organizational collaboration frequency

3.0

Organization collaboration increase (scale is out of 3)

2.7

Collaboration challenges^

2.0^

PSH = permanent supportive housing.

Notes: The total possible number of observations is 104; number of observations per index ranges from 85 to 104 due to occasional missing data. Means are on a scale of 1 to 4 except where noted. Numbers closer to 4 are typically considered positive.

Means marked with ^ are on a reversed scale, however, and lower numbers are considered positive because they represent lower challenges. See appendix B for extended definitions of each index.

Source: 2019 annual partnership survey

On some indices, respondents at sites in PFS contract implementation reported better collaboration across systems and between PSH providers, as well as higher vision alignment among all partners, compared with respondents at sites in the transaction structuring phase (exhibit 20). This might indicate that the formalities that come with executing a project—such as contracts, regular meetings, and regular reporting—solidify mechanisms of community collaboration.

EXHIBIT 20

Selected Partnership Survey Results on Collaboration by Phase

Transaction

PFS contract

Index name

structuring

implementation

Difference

Vision alignment of partners

3.2

3.6

0.4***

Community and provider support and

capacity

3.1

3.4

0.3***

PSH provider collaboration

3.1

3.6

0.5***

Cross-system collaboration

3.1

3.6

0.5***

Collaboration challenges^

2.1^

1.8^

-0.3**

PFS = pay for success. PSH = permanent supportive housing.

Notes: The total possible number of observations is 104; number of observations per index ranges from 85 to 104 due to occasional missing data. The table is ordered by ascending level of difference between transaction structuring and PFS contract E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

6 3

implementation. Differences marked with *** are significant at P < 0.01, those with ** denotes significant at P < 0.05, and those with * are significant at P < 0.1. The total possible number of observations is 104, except for criminal justice data sharing frequency, which is 58 because the questions were answered only by people at organizations that directly serve the target population. Means are on a scale of 1 to 4. Numbers closer to 4 are generally considered positive. Low numbers are positive for the challenge-related indices (marked with ^) because a lower number indicates that respondents perceived few challenges. See Appendix B for extended definitions of each index.

Source: 2019 annual partnership survey

Strengthening Local Data Capacity and Cross-Sector Sharing

Survey respondents notably reported high levels of data sharing for PFS planning and among service providers (exhibit 21). Sites have made considerable efforts to share data across sectors since the beginning of the demonstration, as reported in our baseline implementation report (Liberman et al., 2019) and a brief, “Data Use and Challenges in Using Pay for Success to Implement Permanent Supportive Housing” (Gillespie, Batko, and Liberman, 2018). The results this year indicate that partners are continuing to use data to identify the target population and the level of services and housing needed to support them. Service and housing providers also indicated that their organizations frequently shared and received data from other organizations about clients in the target population. Data sharing related to the criminal justice history of a client was notably low in comparison, but this could be because of how projects are separating eligibility determinations from service provision (or the lower number of respondents working at criminal justice organizations). Sites in the implementation phase reported higher rates of data use, notably criminal justice data sharing and performance data use, than respondents at sites in the transaction structuring phase (exhibit 22).

EXHIBIT 21

Partnership Survey Results on Data Use and Sharing

Number of

Index name

Mean

observations

All respondents

Performance data use

3.1

99

Data sharing for PFS planning

3.6

102

Respondents from organizations that directly serve clients

Criminal justice data sharing frequency (self)

2.8

38^

Client data sharing frequency (self)

3.3

40^

Service provider data sharing frequency (self)

3.6

45^

PFS = pay for success. Notes: The total possible number of observations for “performance data use” and “data sharing for PFS

planning” is 104. The total possible number of observations for the rest (marked with ^) is 58 because the questions were answered only by people at organizations that directly serve the target population. Means are on a scale of 1 to 4. Numbers closer to 4 are considered positive. See appendix B for extended definitions of each index.

Source: 2019 annual partnership survey

6 4

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

EXHIBIT 22

Partnership Survey Results on Data Use and Sharing with Phase Variation Transaction

PFS contract

Number of

Index name

structuring

implementation

Difference

observations

All respondents

Performance data use

3.0

3.3

0.3**

99

Data sharing for PFS planning

3.5

3.7

0.2*

102

Data service challenges^

2.4^

2.0^

-0.4***

90

Respondents from organizations that

directly serve clients

Criminal justice data sharing frequency (self)

2.5

3.4

0.9**

38

PFS = pay for success; PSH = permanent supportive housing.

Notes: The table is ordered by ascending level of difference between transaction structuring and PFS contract implementation.

Differences marked with *** are significant at P < 0.01, those with ** denotes significant at P < 0.05, and those with * are significant at P < 0.1. The total possible number of observations is 104, except for criminal justice data sharing frequency, which is 58

because the questions were answered only by people at organizations that directly serve the target population. Means are on a scale of 1 to 4. Numbers closer to 4 are generally considered positive. Low numbers are positive for the challenge-related indices (marked with ^) because a lower number indicates that respondents perceived few challenges. See appendix B for extended definitions of each index.

Source: 2019 annual partnership survey

In interviews, stakeholders in every site except for Los Angeles County mentioned specific examples of how PFS planning or implementation had improved the ways they collect, share, and use data:

◼

The data analyses done to design the eligibility criteria in Anchorage complemented the CoC’s efforts to revamp coordinated entry.

◼

RICH in Rhode Island cited an increased ability to negotiate data-use agreements as a result of the PFS project.

◼

In both Anchorage and Rhode Island, data sharing was key for SIPPRA applications.

◼

Montgomery County Department of Corrections staff have been trained on and now have access to the CoC’s Homeless Management Information System.

◼

Three sites (Anchorage, Austin/Travis County, and Rhode Island) were successful in sharing and integrating homelessness, criminal justice, and healthcare data.

Some stakeholders involved with serving the target population, including government stakeholders, also noted an increased focus on outcomes. Although the data sharing in Los Angeles County is not new, stakeholders reported that monitoring specific client outcomes for repayment is unique to this project.

In Lane County, Sponsors uses a Community Data Link to track dosage and client outcomes.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

6 5

Increasing Local Capacity through External Technical Assistance Providers Technical advisers—particularly CSH, Social Finance, and Third Sector—continued to play an important role in supporting the intermediary or serving as the intermediary in all sites. In Anchorage, CSH and Social Finance participated as core team members, helping to make key project decisions. In Austin/Travis County, Social Finance serves as the financial intermediary for the project. Social Finance staff are paid through the HUD-DOJ grant to support many of the project management aspects of the project, such as coordination of meetings, processing of outcome payments, leading and managing governance meetings, analyzing provider data, and working with stakeholders to coordinate any changes to service delivery. After the full project launches, Social Finance will be funded through the PFS contract. In both Anchorage and Austin/Travis County, CSH played a role in the request for proposal for service and housing providers. In Montgomery County and Prince George’s County, MD, Third Sector advised intermediary AIR on finances and contracting and facilitated some peer-learning opportunities for site staff. Partners on the ground learned from the expertise of outside providers and hoped to increase their capacity for these types of projects.

Implementation and Pilots Increased Capacity to Serve

Target Population

An overall goal of using PFS to increase PSH through this project was to increase the capacity of the community to serve and create better outcomes for the target population. This increased capacity is not limited to the development of this project. In multiple sites, the development of the project resulted in an expansion of the service model of other projects, increased provider capacity and skills overall, and the adoption of Housing First by additional providers. In Los Angeles County, changes to the services component of the project—reducing caseloads and adding a clinical component—were made in not only the PFS project but in the broader PSH program for the target population.

In Austin/Travis County, Anchorage, Lane County, and Los Angeles County, implementation of a project or a pilot resulted in increased capacity or skills in services providers. In Austin/Travis County, clinical providers new to housing created staff positions and developed skills in locating housing and supporting participants as tenants. In Anchorage, providers increased clinical capacity within their agencies. In Los Angeles County and Lane County, service providers with a long history of experience in reentry services learned about and adapted to a Housing First model.

6 6

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

Compared with PSH implementation partners, PFS planning partners reported greater community and provider support and capacity, ability to bill Medicaid for services, and an increase in partners on local criminal justice coordinating committees (exhibit 23). However, PSH implementation partners felt that their organizational level of involvement was higher than that of PFS planning partners, so one possible explanation is that they were more acutely aware of challenges related to PSH provision for this population. This result highlights a discrepancy in the awareness of project details that may be related to the fact that PSH implementation partners feel they are more involved. While it is important to note that these results are about perceptions of involvement, they could indicate that PSH

implementation partners play a bigger role in the PFS demonstration. Given the systemic barriers faced by the target population to accessing housing and services in the community, it makes sense that PSH

implementation partners rated community and provider support and capacity lower than their PFS

planning counterparts did.

EXHIBIT 23

Partnership Survey Results with Significant Variation by Organizational Role PSH

PFS planning

implementation

Number of

Index name

partners

partners

Difference

observations

All respondents

Ability to bill Medicaid for services

2.8

2.4

-0.4**

88

Community and provider support and

capacity

3.3

3.0

-0.3***

95

Data service challenges^

2.4^

2.2^

-0.2*

84

Organizational level of involvement (self)

3.5

3.8

0.3**

90

Respondents from organizations that

directly serve clients

Increase in partners on local criminal

justice coordinating committee

(scale is out of 3)

2.5

1.8

-0.7***

32

PFS = pay for success. PSH = permanent supportive housing.

Note: The table is ordered by ascending level of difference between PFS planning partners and PSH implementation partners.

Differences marked with *** are significant at P < 0.01, those with ** denotes significant at P < 0.05, and those with * are significant at P < 0.1. The total possible number of observations is 104, except for respondents who directly service the target population, 58.

Means are on a scale of 1 to 4. Numbers closer to 4 are generally considered positive. Low numbers are positive for the challenge-related indices (marked with ^) because a lower number indicates that respondents perceived few challenges. See Appendix B for extended definitions of each index.

Source: 2019 annual partnership survey

For some indices, the responses from partners in the transaction structuring phase and the responses from partners in the implementation phase were very similar (see appendix B).

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

6 7

VI. Time Costs of Pay for Success

Conversations with sites about project challenges almost always included comments about the time-intensive nature of the work—both the total amount of time spent and the overall timeline. In Austin/Travis County, the intermediary noted that the long planning timeline creates time for change to occur before plans are executed, and one partner noted that it not only takes “an incredible amount of time” but that this extends to executives in the organization. In Lane County, a provider partner linked a heavy time burden of business-as-usual service delivery complexity. In Los Angeles County, the feasibility work, contract creation, and financial modeling were time-consuming for stakeholders. On the implementation side, a provider partner noted that the additional data and reporting requirements are time-intensive. In Montgomery County and Prince George’s County, the intermediary noted that the work required to inform stakeholders outside the core PFS planning team was a time cost driver.

Our evaluation is designed to quantify this time cost to individual sites by fiscal quarter and find answers to these questions: How much time do partners spend on developing the PFS PSH project in each phase? How does this vary by site? Which partners/stakeholders are spending the most time on the development of the project?

Time Spent on the Project Varied by Type of Stakeholder

The exhibits below illustrate how reported time spent varied by the type of project partner in each site. “Financial partner” refers to any organization, public or private, that has committed to being an end payor, investor, or loan provider. Government partners include any department or agency that is involved in planning or implementation but does not act (or has not yet acted) as an end payor.

Intermediaries drive project progress and play a role in the distribution of financial resources across partners. Service providers are organizations that work directly with program participants on service provision and/or housing navigation services. Technical assistance providers are organizations that bring their specialized expertise to help projects make progress.

Across all sites, intermediary organizations spend most of their time on these projects, accounting for 52 percent of total time spent (exhibit 24). Service providers are second, with 24 percent of the total time spent, and government partners accounted for the least, at 4 percent of total hours. It is important to note that while not all sites have service providers working on the projects (e.g., Prince George’s 6 8

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

County and Rhode Island), all the sites include intermediary organizations. Organizations involved as either investors or endpayors account for only 5 percent of total time spent.

EXHIBIT 24

Hours Spent by Partner Type, Q4 2018–Q3 2019

Site

Hours

Percentage

Financial partner

1,442

5

Government partner

973

4

Intermediary

13,691

52

Service provider

6,479

24

Technical assistance provider

3,987

15

Total

26,572

100

Q = quarter.

Source: Urban Institute Time Use Survey

The distribution of hours by organization type across all sites cannot account for the variation in total hours spent per site, as well as variation in approaches to PFS planning and implementation and progress made through Q3 2019. To assess time use in a more meaningful way, we illustrate the time spent by partner-type as a percentage of total site hours reported (exhibit 25). Reporting by percentage minimizes the impact of the variations previously identified.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

6 9

EXHIBIT 25

Time Use by Site and Role Type, Q4 2018–Q3 2019

TA = technical assistance.

Source: Urban Institute Time Use Survey

The variations reflect the different work types conducted by the sites at the end of 2018 and in the first three quarters of 2019. In Rhode Island and Prince George’s County, the two sites in structuring with no pilot program launched, the intermediary accounts for close to 80 percent of total time used; in the other sites, that figure ranges from 20 to 72 percent. In the two sites in the implementation phase, Los Angeles County and Lane County, the service provider accounted for 22 percent and 77 percent, respectively. In Los Angeles County, the actual amount of time spent by service providers is likely closer to the amount in Lane County than it seems. The four service providers serve all ODR clients, and case managers cannot distinguish which clients are part of the PFS project; consequently, they cannot easily estimate how much time they spend on the project. We can only report information by the organization that assists with housing navigation and placements. Two organizations in Anchorage, Social Finance and CSH, account for the large percentage of time spent on technical assistance. In other places, these organizations act as intermediaries; in Anchorage, they work alongside United Way, the intermediary, on a core project management team. The partnership survey captures how meetings can drive the time required for PFS planning and implementation (exhibit 26). In 2019, 57 percent of survey respondents reported attending 11 or more meetings, and 40 percent reported attending more than 20 meetings.

7 0

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

EXHIBIT 26

Number of Meetings Attended in 2019

Share of those who

Share of

N

completed survey

total sample

Number of meetings attended

0 meetings

2

2

1

1–4 meetings

21

20

15

5–10 meetings

21

20

15

11–20 meetings

18

17

13

More than 20 meetings

42

40

31

Total

104

100

76

Note: A total sample of 136 people received the survey.

Source: 2019 annual partnership Survey

Total Time Spent Varied by Phase and by Site

Respondents to our time use survey reported spending a total of 26,572 hours planning and implementing PSH with PFS from Q4 of 2018 through Q3 of 2019 across all demonstration sites (exhibit 27). This varied from just over 1,590 hours for partners in Los Angeles County to 7,886 hours for partners in Austin/Travis County, TX.

EXHIBIT 27

Time Use Totals by Site, Q4 2018–Q3 2019

Site

Total hours

Montgomery County/Prince George’s County, MD

2,655

Rhode Island

2,868

Anchorage, AK

5,229

Austin/Travis County, TX*

7,886

Lane County, OR

6,344

Los Angeles County, CA

1,591

Total

26,572

Q = quarter.

Source: Urban Institute Time Use Survey

Exhibit 28 illustrates trends in time use by quarter and by site. Total time use for the sites was lower in Q3

of 2019 than in Q4 of 2018, the first quarter of data collection. The biggest change was seen in Austin between Q1 and Q2 of 2019. Staff turnover in the intermediary organization likely contributed to this decrease. Rhode Island, Maryland, and Los Angeles exhibited little variation across this time period. None of these sites E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

7 1

exhibited large transitions during this period (such as launching pilots or changing phases). Lane County and Anchorage exhibited steady increases in time use.

EXHIBIT 28

Time Spent by Site in a 12-Month Period (Q4 2018–Q3 2019)

Q = quarter.

Source: Urban Institute Time Use Survey

Exhibit 29 illustrates how the total hours spent vary by site and by PFS phase, specifically transaction structuring, structuring with a pilot, and implementation. The combined hours for the two transaction-structuring sites that launched pilots exhibit the highest time cost. These sites have the combined responsibility of finalizing the transaction while launching service provision to test key assumptions made during PFS planning. The two implementation sites, Lane County and Los Angeles County, exhibit the largest percent difference between any sites in the same phase. This is likely because Lane County is still working to secure financing for the total cohort while Los Angeles County has been in full implementation and was concluding enrollment by the end of Q3 2019 (and, as noted previously, we are unable to report time use information directly from the service providers who work most closely with clients).

7 2

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

EXHIBIT 29

Total Hours by PFS Phase and by Site, Q4 2018–Q3 2019

Q = quarter.

Source: Urban Institute Time Use Survey

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

7 3

VII. Conclusion

In 2019, six sites in the demonstration continued toward increasing permanent supportive housing (PSH) capacity to serve the target population. Four sites—Anchorage, Alaska; Austin/Travis County, Texas; Prince George’s County, Maryland; and Rhode Island—began and ended the year in the transaction structuring phase. Two of these sites operated pilots during 2019—Anchorage and Austin/Travis County. Lane County, Oregon, and Los Angeles County, California, began and ended the year in the implementation phase. Los Angeles County completed full enrollment in the project in 2019.

To date, three jurisdictions initially included in the demonstration have withdrawn from participation—

Pima County, Arizona, Matanuska-Susitna Borough, Alaska, and Montgomery County, Maryland.

Projects continue in Alaska and Maryland but are now concentrated on the Municipality of Anchorage and Prince George’s County, respectively.

Participation in the demonstration and the activities required to develop and implement a Pay For Success (PFS) project provided benefits to participating communities. In all sites, stakeholders continue to report strengthened local collaboration and data sharing as a benefit, between homeless assistance providers and across multiple sectors, especially criminal justice and healthcare. For sites in implementation and operating pilots, community stakeholders reported success in helping the target population lease-up and remain successfully housed. Sites also reported increased provider capacity to serve the target population as the project increased the clinical capacity of providers and brought new service partners to the table.

Demonstration sites in the transaction structuring phase reported experiencing similar challenges.

Some sites had to redo data matching and analyses done in feasibility to come up with more concrete estimates of costs and eligibility criteria. Finding and securing financial partners, particularly state government and health end payors, proved challenging. Concerns raised by these potential financial partners were specific to PFS as a funding mechanism, namely the complexity of the financial model, the time required to negotiate and structure the deal, and the financial risk associated with relying on potential outcomes, particularly health outcomes. Time use survey findings confirmed that the time spent by end payors and investors was substantial. For sites that were attempting to secure a state government as an end payor, engaging in the legislative and appropriations processes was time-consuming and unsuccessful. Additionally, elections and political turnover proved to disrupt, and in some instances, undo site work, including the loss of state budget commitments. The PFS phases of feasibility analysis, transaction structuring, and contract implementation proved less meaningful as 7 4

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

distinct phases as sites triaged different workstreams to keep making progress on project goals without necessarily ticking all of the boxes usually associated with a particular phase.

Sites also struggled with securing housing resources. Several sites planned to rely on the PFS

transaction to pay for services and leverage federal or state vouchers through a local housing authority partnership. These efforts were not always successful, in large part because of local competition for vouchers. In one site, vouchers were needed for housing authority redevelopment and relocation activities. In another, the housing authority partner requested that the project plan to use the queue for existing set-asides instead of creating a set-aside specific to the project.

Anchorage and Austin/Travis County saw their pilots as an opportunity to test and refine participant identification, enrollment pathways, and program design. Sites also used the pilots to vet and build capacity in service providers. Challenges faced while implementing pilots were similar to implementation challenges.

The sites in the implementation phase—Lane County and Los Angeles County—reported successfully helping participants access and maintain housing. Providers continued to strengthen Housing First service provision to clients; Los Angeles County, in particular, was able to do so through additional county funding. Sites were mostly successful in meeting performance benchmarks to secure success payments. At the end of 2019, it was unclear whether the project evaluations would impact housing stability and justice outcomes for participants when compared to those receiving care as usual in the community.

Implementation and pilot sites reported implementation challenges consistent with those that are faced by PSH projects developed without a PFS financial mechanism. Sites that used tenant-based vouchers for housing struggled with locating housing and recruiting landlords, including matching participants to units and neighborhoods. Multiple sites indicated that participants required much more intensive services than had been expected and reported increasing capacity in service providers to ensure that participants were fully supported.

The central question of the demonstration is whether the PFS model can expand PSH to improve housing stability and other outcomes for the target population. To date, sites have shown mixed results.

Lane County and Los Angeles County successfully moved into the implementation phase, but Lane County has been unsuccessful in securing the full level of resources needed to implement the project through a PFS framework. The remaining sites struggled to secure end payors and other financial partners, who expressed concerns regarding the complexity, risks, and time commitment of the model.

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

7 5

In 2020, the national evaluation of demonstration sites will continue to monitor progress and process in sites and collect information on benefits, partnerships, and time commitment. This data collection will continue through years 4 and 5 of the demonstration. A final report summarizing demonstration success at sites in developing and implementing PSH with PFS will document their progress during those subsequent grant years.

7 6

E V A L U A T I O N O F T H E H U D -D O J P F S P S H D E M O N S T R A T I O N : Y E A R 3

Appendix A. Example Site Visit

Stakeholder List

The number and scope of stakeholders interviewed during a site visit depend on various project characteristics, such as the phase of work and existing services and government landscape. These factors determine which stakeholders are at the table and the extent to which they are involved. This list provides an example of the types of people we sought to interview.

◼

Intermediary, project coordinator.

◼

County Department of Health and Human Services, Housing, or Reentry, executive-level stakeholder.

◼

County Department of Health and Human Services, Housing, or Reentry, data or program manager.

◼

Other relevant local or state government stakeholders.

◼

Local political offices, such as mayor, city council, state legislature, or governor.

◼

Service and/or housing provider, executive director.

◼

Service and/or housing provider, case manager, or housing navigator.

◼

Other relevant stakeholders to homelessness, criminal justice, or health systems, such as a coordinated entry lead from a Continuum of Care or probation director at a probation and parole department.

◼

Foundation or investor, program officer.

◼

Technical assistance provider, staff working on this project.

◼

Evaluator, principal investigator of the project.

A P P EN D I X B

7 7

Appendix B. Detailed Partnership

Survey Methodology

The survey was launched in December 2019 with a sample of 9 to 16 active partner organizations on each project site, for a total of 136 potential respondents. The survey was intended to measure changes over time in the strength of the partnerships and the benefits at the population- and system-levels. The survey asked questions about respondent backgrounds and roles in the project, collaboration with partners, data sharing, and service provision barriers. The survey was administered using Qualtrics, a web-based software, and was designed to take 15 to 20 minutes to complete.

The survey was active until February 2020. We completed additional analyses to more clearly summarize the data and draw comparisons among subgroups of respondents. In the previous report, we reported on individual questions. Because the partnership survey is extensive, we reduced the number of items presented by combining them into thematic groups, or indices. These indices measure the broader themes in the partnership survey results, such as the strength of cross-system collaboration, performance data use, or the frequency of service provider data sharing. Creating indices and performing reliability analysis allowed us to reduce the number of items we are reporting on. The number associated with each index represents the mean of all responses to all subquestions that make up the index. Almost all the questions were asked on a 4-point scale; respondents were asked to rank their level of agreement with the statement, indicating the frequency of an action or the level to which a particular thing may have been challenging. Exhibit 30 shows which questions and sub-questions make up each index.

7 8

A P P EN D I X B

EXHIBIT 30

Partnership Survey Indices Detailed information

Survey

Questions

Index name

Index definition

in Index

Partnerships (roles, planning, and infrastructure)

Project roles and

The PFS demonstration is meeting its aims through data use and is led by 8

progress

organizations with distinct roles.

Organizational role and Partner organizations and staff members understand the problem the 9

awareness

project addresses, their role in the PFS demonstration, and how that role is related to supporting the target population.

Organizational level of

Respondent organization plays an active role in the PFS supportive

9

involvement (self)

housing project.

Vision alignment of

Project partners work well together and have a common vision for the

10

partners

project and understanding of PFS, PSH, and how they intersect.

Community and

The community supports the PFS demonstration by prioritizing and

11 & 25

provider support and

leveraging resources and expertise, and the community and service

capacity

providers reduce barriers to serving the target population by making

housing accessible and educating policymakers.

Ability to bill Medicaid

Supportive housing providers can bill Medicaid for covered services.

25

for services

Data use and sharing

Performance data use

Partners use data to identify the target population and the level of

18

services needed.

Data sharing for PFS

Partners use data to identify the target population and the level of

17

planning

services needed.

Service provider data-

How often service providers are sharing and receiving data about clients 20

sharing frequency (self)

in the target population.

Criminal justice data-

How often service providers are sharing criminal justice client data (such 21

sharing frequency (self)

as criminal history, jail/prison release, or probation/parole information).

Client data-sharing

How often service providers are making referrals to other organizations, 22

frequency (self)

as well as information about other services, assessments, or outcomes

obtained.

Collaboration

PSH provider

PSH providers are collaborating with other organizations on the PFS

12

collaboration

demonstration to provide housing stability and health services to the

target population.

Cross-system

Law enforcement, homeless assistance, courts, jails, and

13

collaboration

health/behavioral health services are collaborating.

Organizational

How often a respondent’s organization is sharing or leveraging resources, 16

collaboration

training, or service information.

frequency

Organizational

Change in overall collaboration—including data sharing, formal

14

collaboration increase

agreements, and meetings—over the past year (1=decrease, 2= no

change, 3=increase)

A P P EN D I X B

7 9

Survey

Questions

Index name

Index definition

in Index

Increase in partners on

Change in number of PFS partner organizations represented on the local

14

local criminal justice

criminal justice coordinating committee over the past year (1=decrease, coordinating

2= no change, 3=increase)

committee

Challenges

Collaboration

The project has faced challenges to collaboration, including distrust,

15

competition for resources, conflicting visions, lack of working

relationships, and skepticism about Housing First.

Data service

The project has faced challenges in providing services, including

23

coordination of entry for homeless services, data information sharing

across organizations, and limited time and/or resources.

Accessing clients in jail

The project has faced challenges accessing clients in jail.

23

PSH service

The project has faced challenges because of a lack of access to housing 24

and services, such as housing eligibility challenges or general lack of services.

PFS = pay for success. PSH = permanent supportive housing.

Notes: Scales are from 1 to 4 except where noted. The index definition corresponds clearly with the wording of each subquestion of the survey.

Source: 2019 annual partnership survey

We performed reliability checks on these groupings using Cronbach’s alpha tests. Cronbach’s alpha test measures internal consistency among groups of variables, and it creates an average of all the variables within the group. Where it was determined that one item within an index was inconsistent with the others in the index, we separated it out and reported on it on its own. We also confirmed the reliability of these indices by measuring how correlated they were.

Once these indices were created, we ran cross-tabulations with the organizational role to determine whether respondents involved in the Pay For Success (PFS) planning and those involved in (PSH)permanent supportive housing implementation ranked statements differently. PFS planning partners include intermediaries, technical assistance partners, end payors, investors, and other local partners, such as government stakeholders that served on planning committees but were not end payors. PSH implementation partners include direct service providers, housing partners, criminal justice referral partners, and other criminal justice partners. We exclude evaluators from this analysis because of their independent role outside of these categories. Similarly, we ran cross-tabulations by the PFS phase. Because not all respondents identified the phase of their site in the same way, we used determinations by HUD of the phases for each site. We ran regressions to test the significance of the 8 0

A P P EN D I X B

differences between different roles and phases on the indices we created. We report only differences that were determined to be significant using the regression.

We cleaned the data to ensure consistency among respondents at the same organization, and we made changes to how organizations identified themselves when the role they selected was not accurate. We also recognize that some partners play roles that fall into both the PFS planning and PSH

implementation phases; in those cases, we selected the primary role. We also expanded the definition of the intermediary to include organizations beyond the HUD grantee if those organizations were playing an intermediary role, such as serving as the financial host for the PFS transaction. Conversely, we still considered grantee organizations whose main mission is serving the target population as intermediaries.

Instead of reporting the average response for each item, we present the grouped indices we created that show how respondents ranked broader themes of partnerships, data use and sharing, collaboration, and challenges. We keep some items on their own if they could not easily be combined into an index. The averages represent the average response across whatever questions were combined into the index. The number of observations indicates the number of people whose responses are part of the average; it excludes people who selected “not applicable” for any question and “don’t know” for any nonagreement scale questions.

A P P EN D I X B

8 1

Appendix C. Evaluation

Methodology Limitations

All data collection and analysis methods have limitations. We document ours and the strategies taken to minimize them in this section.

Process study. We rely heavily on the project intermediaries to facilitate data collection for this study: we speak to them monthly, and they facilitate whom we contact while on site. Therefore, our findings could be biased by the perspective they present to us and limited in the reach of our interviews with the stakeholders they give us contact information for. We attempt to safeguard against this by fostering constructive relationships with intermediaries, doing our research about the project, and independently interviewing other stakeholders at the site. Data collected in these annual interviews are snapshots in time, and the accuracy of accounts from individuals is dependent on their recollections of events in the previous year. Because of scheduling conflicts or other reasons, we cannot always interview all the stakeholders we reach out to. We therefore might be leaving out the perspective of a certain role in a site.

Annual partnership survey. Although our response rate was strong and respondents rarely skipped questions, missing data for some questions limit the robustness of some results. For all questions, respondents could select “don’t know” if they did not know the answer to a question or “not applicable”

if the question did not apply to their organization. Survey results tables note the number of observations to show where more respondents may have selected those options. Lack of knowledge or applicability for certain questions limits the strength of the results.

Reporting on differences in responses by organization role and phase presents additional limitations. We recognize that some organizations serve as both Pay For Success (PFS) planning partners and permanent supportive housing (PSH) implementation partners, so we selected the role that best fit most of their work on the demonstration. Sometimes this meant categorizing organizations that provide housing as PFS planning partners because they are the grantees for the demonstration.

Even though they may also be providing some housing for the demonstration, their role is more heavily focused on PFS planning for the broader project. In terms of phase, we recognize that both Anchorage and Austin/Travis County were implementing pilots in 2019. We considered these sites to be in the transaction structuring phase but recognize that the presence of a pilot meant that we collected responses from stakeholders whose roles were primarily implementation-related. Thus, the lines 8 2

A P P EN D I X C

between PFS phases are blurry, and keeping that in mind is important when drawing conclusions about the differences reported between the phases.

Time use survey. The accuracy of the time use study depends on the ability and willingness of organization points of contact to respond to our request for data. Because PFS planning can be a time-intensive endeavor, taken on by people with multiple responsibilities in their own organizations, obtaining timely responses from some organizations has been a challenge. To reduce nonresponse, we designed the data collection to minimize burden. For example, we request quarterly data, not monthly, and leverage systems like the Disaster Recovery Grant Reporting System that intermediary organizations use to draw down grant funds from HUD and that include invoicing for some site organizations. We also use our monthly process study check-in calls to encourage response and to troubleshoot problems. Finally, for organizations and individuals who do not need to track their time regularly, human error in recalling time spent on a project over an entire quarter may lead to inaccuracies in estimation. Using invoicing data and encouraging respondents to review calendars or time cards to estimate totals mitigate potential for recall error.

A P P EN D I X C

8 3

Appendix D. Partnership Survey

Instrument

The Urban Institute, an independent research organization based in Washington, D.C., is conducting an Annual Partnership Survey as part of the Pay for Success (PFS) initiative in ${e://Field/Site}. Funded by the Departments of Justice and Housing and Urban Development, the PFS initiative aims to bring together cross-sector public and private partners to implement permanent supportive housing (PSH) in order to improve outcomes for chronically homeless individuals who are frequent users of jail and emergency medical services.

You are being asked to participate in this survey because of your role in an organization that is involved with ${e://Field/Site}’s Pay for Success Initiative. The survey asks questions about how different organizations in the initiative work together around performance management and evaluation, cross-sector collaboration, and innovation in service delivery. By conducting this survey annually, we hope to learn how the relationships among PFS partners and the broader community change over the life of the project. Please do not delegate or share this survey link with others, as it is unique to you.

The survey is designed to take 15 to 20 minutes to complete. You may stop at any time, and your answers will be saved so that you can finish at a later point. You can also navigate back and forth between pages without losing your work. Please answer every question as best as you can. If a question does not apply to you or your organization, or if you do not know the answer, please use the “not applicable” or “don’t know” response options.

Participation in the survey is voluntary. Only aggregate responses will be publicly reported—

never individual responses. Your name will not be used in any report about this survey, and nobody outside the Urban Institute research team will know how you answered a particular question. However, due to the nature of the project, a unique role or response could be potentially identifying.

If you have any questions about the survey or have technical difficulties completing the survey, please contact [NAME] at the Urban Institute at [PHONE NUMBER/EMAIL].

End of Block: Introduction

8 4

A P P EN D I X D

Start of Block: Background

1. What kind of organization do you work for? Please select the option that best matches your role on the PFS project.

Housing

Health and human services

Criminal justice

Executive or budget

Health or behavioral health provider

Housing provider

Other service provider

Evaluation organization

Technical assistance organization

Philanthropy/Foundation

For-profit investor

Other: please describe ________________________________________________

2. Which of the following best describes your organization's role in the PFS project?

Intermediary

Evaluator

Other technical assistance

Financial partner

Direct service for target population

Housing partner for target population

Criminal justice referral source for target population

Other criminal justice partner

Other role: please describe ________________________________________________

3. What is your title or position within your organization?

________________________________________________________________

4. How long have you been in this position? Please enter numeric values only (i.e., “0.5,” “4,” or “10.5”).

Years: ________________________________________________

5. How long have you been participating in the PFS supportive housing project? Please enter numeric values only (i.e., “8,” “18,” “48”).

Months: ________________________________________________

A P P EN D I X D

8 5

6. In 2019, how many meetings have you attended about the PFS supportive housing project?

0 meetings

1–4 meetings

5–10 meetings

11–20 meetings

More than 20 meetings

7. At the end of 2019, what was the current phase of this PFS project?

Feasibility assessment

Transaction structuring

PFS contract implementation

End of Block: Background

Start of Block: FYI

PLEASE NOTE

In the next section, please answer the questions based on your experiences working on the PFS project in ${e://Field/Site} in 2019.

In this survey, “target population” refers to people experiencing homelessness who are, or will be, served by the PFS-funded PSH program.

“PFS” refers to Pay for Success.

“PSH” refers to Permanent Supportive Housing.

If a question does not apply to you or your organization, please select “Not Applicable.” If you do not know the answer to any question, indicate “Don’t Know.” Please do not leave a question blank.

End of Block: FYI

Start of Block: Collaboration with Partners

8 6

A P P EN D I X D

8. PFS Project. To what extent would you agree with the following statements about your PFS project?

Disagree

Disagree

Agree

Agree

Don’t

Not

completely somewhat

somewhat

completely

Know

Applicable

(1)

(2)

(3)

(4)

I have a clear

understanding of my

organization’s roles

and responsibilities

within the PFS

project

I have a clear

understanding of

other organizations’

roles and

responsibilities

within the PFS

project

The PFS project is

meeting its aims and

objectives for its

current phase

The PFS project’s

work is being guided

by information and

data f

A P P EN D I X D

8 7

9. PFS Organization. To what extent would you agree with the following statements about your organization’s role in the project?

Disagree

Disagree

Agree

Agree

Don’t

Not

completely somewhat somewhat completely Know Applicable

(1)

(2)

(3)

(4)

My organization plays

an active role in the

PFS supportive

housing project

My organization has a

stake in increasing

supportive housing

for the target

population

Leaders in my

organization are

aware of the issues

surrounding the

target population

Leaders in my

organization are

committed to

addressing the needs

of the target

population

Staff members in my

organization are

aware of the issues

surrounding the

target population

8 8

A P P EN D I X D

10. PFS Partners. Would you agree with the following statements about the PFS partnership?

Disagree

Disagree

Agree

Agree

Don’t

Not

completely somewhat somewhat Completely Know Applicable

(1)

(2)

(3)

(4)

Partners share a

common vision

Partners have a

common

understanding of PFS,

and its role in

supporting PSH

Partners have a

shared understanding

that PSH is important

for target population

Partners share

understanding that

PFS is a promising

approach to establish

and/or support PSH

for target population

Partners are willing to

make changes to

achieve shared goals

Partners have mutual

trust and respect

Partners work well

together as a group

Partners make high

level decisions

through a

collaborative process

A P P EN D I X D

8 9

11. Community Support. Would you agree with the following statements about the community support for the project?

Disagree

Disagree

Agree

Agree

Don’t

Not

completely somewhat somewhat Completely Know Applicable

(1)

(2)

(3)

(4)

Housing the target

population is a high

priority for the

community

There are

organizations in the

community with

expertise and

experience in

providing

supportive housing

for the target

population

The PFS project is

bringing local

partners together

across sectors to

improve results for

the target

population

Over the past year,

the community has

acquired new

sources of funding

for rent, operating

subsidies, or

supportive services

for the target

population

9 0

A P P EN D I X D

12. Collaboration. Would you agree with the following statements about permanent supportive housing providers in your community?

Disagree

Disagree

Agree

Agree

Don’t

Not

completely somewhat somewhat Completely Know Applicable

(1)

(2)

(3)

(4)

Over the past year,

local supportive

housing providers

were able to

collaborate with

other agencies to

promote housing

stability and well-

being for the target

population

Over the past year,

local supportive

housing providers

have been working

with health and

behavioral health

providers to facilitate

access to benefits and

services

A P P EN D I X D

9 1

13. Collaboration. Over the past year, how much have the following systems collaborated with each other in working with the target population?

No

Little

Occasional

Extensive

Don’t

Collaboration Collaboration Collaboration Collaboration

N/A

Know

(1)

(2)

(3)

(4)

Law enforcement

with the homeless

assistance system

Courts with the

homeless

assistance system

Jail(s) with the

homeless

assistance system

Law enforcement

with health/

behavioral health

service providers

Courts with

health/behavioral

health service

providers

Law enforcement

with health/

behavioral health

service providers

Homeless

assistance with

health/behavioral

health service

providers

9 2

A P P EN D I X D

14. Collaboration. Over the past year, has collaboration increased or decreased?

Stayed

Don’t

Not

Decreased

the

Increased Know Applicable

Same

Data sharing between organizations

serving the target population

Number of formal agreements or

memoranda of understanding

between organizations serving the

target population

Frequency of meetings about

effectively working with the target

population

Number of PFS partner

organizations represented on the

local criminal justice coordinating

committee

15. Collaboration. Over the past year, did any of the following pose a serious problem for PFS partners collaborating?

Not a

Minor

Serious

Moderate

Don’t

Not

Problem

Problem

Problem

Problem (3)

Know

Applicable

(1)

(2)

(4)

Competition for

resources or ‘turf

issues’

Conflicting

priorities and

visions

Lack of trust among

agencies

Absence of

established working

relationships

Skepticism or

disagreement about

the Housing First

approach

A P P EN D I X D

9 3

16. Collaboration. Over the past year, how often has your organization engaged in the following activities with other organizations serving as partners in the PFS project?

Never

Rarely

Occasionally Frequently

Don’t

Not

(1)

(2)

(3)

(4)

Know Applicable

We collaborated with PFS

partners to leverage

resources (e.g., write grants

together)

We collaborated with PFS

partners to provide training

or educational

opportunities

We shared resources, such

as materials or equipment,

with PFS partners

We provided information

about our programs or

services to PFS partners

We met with PFS partners

to share information about

services, resources, or

clients

We collaborated with PFS

partners to provide direct

services to individuals

End of Block: Collaboration with Partners

Start of Block: Data Sharing and Focus on Outcomes

9 4

A P P EN D I X D

17. Data sharing. Would you agree with the following statements about data sharing, over the past year?

Disagree

Disagree

Agree

Agree

Don’t

Not

Completely Somewhat Somewhat Completely Know Applicable

(1)

(2)

(3)

(4)

PFS partners used data

to identify the target

population

PFS partners have used

data to understand the

level of housing and

services needed by the

target population

18. Performance data. Would you agree with the following statements about the use of evidence to manage supportive housing providers over the past year?

Disagree

Disagree

Agree

Agree

Don’t

Not

Completely Somewhat Somewhat Completely Know Applicable

(1)

(2)

(3)

(4)

Supportive housing

providers have used

performance data to

identify activities

needing improvement

Supportive housing

providers have

followed evidence-

based models of

service delivery

(housing first, harm

reduction,

motivational

interviewing, etc.)

Government agencies

have used

performance data to

assess whether

supportive housing

programs are

improving client

outcomes

A P P EN D I X D

9 5

End of Block: Data Sharing and Focus on Outcomes

Start of Block: Serve the Target Population

19. Does your organization directly serve the target population?

Yes (1)

No (0)

Skip To: End of Block If Does your organization directly serve the target population? = 0

20. Data sharing. Over the past year, how often has your organization shared or received information about the clients in the target population?

9 6

A P P EN D I X D

Never Rarely Occasionally Frequently Don’t

Not

(1)

(2)

(3)

(4)

Know Applicable

We have

shared

information

with other

organizations

about clients

in the target

population

We have

received

information

from other

organizations

about clients

in the target

population

21. Data sharing. Over the past year, has your organization shared criminal justice information with other organizations serving the target population?

Never Rarely Occasionally Frequently Don’t

Not

(1)

(2)

(3)

(4)

Know Applicable

A client’s criminal history

Whether a client was

recently released from jail

Whether a client was

recently released from

state or federal prison

Whether a client is under

community supervision

(probation, parole, etc.)

A P P EN D I X D

9 7

22. Data sharing. Over the past year, has your organization shared client information with other organizations serving the target population?

Never Rarely Occasionally Frequently Don’t

Not

(1)

(2)

(3)

(4)

Know Applicable

Client referrals

Whether a client is being

served by other

organizations

What services a client is

receiving from other

organizations

Client assessments

conducted by your

organization or another

organization

Client outcomes (e.g., days

in housing, connection to

health services, etc.)

End of Block: Serve the Target Population

Start of Block: Barriers to Service Provision for the Target Population 9 8

A P P EN D I X D

23. Service challenges. Over the past year, has your PFS project experienced any of the following as serious problems for providing services to the target population?

Not a

Minor

Moderate

Serious

Don’t

Not

problem problem

Problem

Problem Know Applicable

(1)

(2)

(3)

(4)

Access to clients in jail

Coordination of entry for

homeless services

Organizational policies about

data sharing

Obtaining client releases to

share information across

organizations

Accessing reliable assessment

information

Accessing other relevant data

Data systems compatibility

Other technological

limitations

Limited time and resources

A P P EN D I X D

9 9

24. Service challenges. Over the past year, have any of the following posed a serious problem for providing services to the target population for your PFS project?

Not a

Minor

Moderate

Serious

Don’t

Not

problem problem

Problem

Problem Know Applicable

(1)

(2)

(3)

(4)

Waiting lists for services for

the target population (e.g.,

housing, substance abuse

treatment, mental health

programs)

Lack of access to housing for

the target population

Lack of access to substance

abuse treatment for the

target population

Lack of access to mental

health programs for the

target population

Rigid eligibility

requirements for housing for

the target population

Policies excluding certain

kinds of offenders from

housing or services

1 0 0

A P P EN D I X D

25. Community infrastructure. Would you agree with the following statements about your community?

Disagree

Disagree

Agree

Agree

Don’t

Not

Completely Somewhat Somewhat Completely Know Applicable

(1)

(2)

(3)

(4)

The community is

committed to reducing

barriers to housing for

the target population,

including accepting

housing applications

for individuals with

criminal justice

histories

The community has a

process to prioritize

new and turnover

housing units for the

target population

Supportive housing

providers have staff

dedicated to landlord

recruitment and

relations

Supportive housing

providers are able to

bill Medicaid for

covered services

Partners are working

to educate

policymakers and local

elected officials about

the need for

supportive housing for

a chronically homeless

jail reentry population

End of Block: Barriers to Service Provision for the Target Population

Start of Block: Final comments

A P P EN D I X D

1 0 1

26. Are there other elements of the PFS project you would like to share?

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

End of Block: Final comments

Start of Block: Confirmation of submission

Thank you for filling out the annual PFS Partnership Survey. If you would like to go back and review or change your answers, please use the back arrow. If you would like to submit the survey, please hit the forward arrow.

Once you go to the next page, you will submit the survey and no longer be able to navigate back to the other questions.

1 0 2

A P P EN D I X D

References

Aidala, Angela A., William McAllister, Maiko Yomogida, and Virginia Shubert. 2014. Frequent Users Service

Enhancement ‘Fuse’ Initiative: New York City Fuse II Evaluation Report. New York, NY: Columbia University Mailman School of Public Health.

Cunningham, Mary, Sarah Gillespie, Devlin Hanson, Michael Pergamit, Alyse D. Oneto, Prasanna Rajasekaran, Tracey O’Brien, Liz Sweitzer, and Christine Velez. 2019. “Maintaining Housing Stability: Interim Lessons from

Denver’s Social Impact Bond Initiative.” Washington, DC: Urban Institute.

Flaming, Daniel, Patrick Burns, and Jane Carlen. 2018. Escape Routes: Meta-Analysis of Homelessness in L.A. Los Angeles, CA: Economic Roundtable.

Giantris, Kristin, and Bill Pinakiewicz. 2013. “Pay for Success: Understanding the Risk Trade-Offs.” San Francisco, CA: Federal Reserve Bank of San Francisco.

Gillespie, Sarah, Samantha Batko, and Akiva Liberman. 2018. “Data Use and Challenges in Using Pay for Success to

Implement Permanent Supportive Housing: Lessons from the HUD-DOJ Demonstration. ” Washington, DC: U.S.

Department of Housing and Urban Development, Office of Policy Development and Research.

Henry, Meghan, Rian Watt, Anna Mahathey, Jillian Ouellette, and Aubrey Sitler. 2020. The 2019 Annual Homeless

Assessment Report (AHAR) to Congress. Washington, DC: U.S. Department of Housing and Urban Development.

Kaeble, Danielle, and Mary Cowhig. 2018. Correctional Populations in the United States, 2016. Report NCJ

251211. Washington, DC: U.S. Department of Justice.

Lee, Christopher Thomas, David Guzman, Claudia Ponath, Lina Tieu, Elise Riley, and Margot Kushel. 2016.

“Residential Patterns in Older Homeless Adults: Results of a Cluster Analysis,” Social Science & Medicine 153: 131–40.

Liberman, Akiva, Mary Cunningham, Sarah Gillespie, Samantha Batko, Brian Bieretz, Matt Eldridge, Evelyn McCoy, Nicole DuBois, and Alexandra Ricks. 2019. Evaluation of the HUD-DOJ Pay for Success Permanent Supportive

Housing Demonstration: Baseline Report. Washington, DC: U.S. Department of Housing and Urban Development, Office of Policy Development and Research.

Liberman, Akiva, Mary Cunningham, Sarah Gillespie, Samantha Batko, Matt Eldridge, Nicole DuBois, Alexandra Ricks, and Kelly Walsh. 2020. Evaluation of the HUD-DOJ Pay for Success Permanent Supportive Housing

Demonstration: Year 2 Report. Washington, DC: U.S. Department of Housing and Urban Development, Office of Policy Development and Research.

Linton, Kristen Faye, and Michael S. Shafer. 2014. “Factors Associated with the Health Service Utilization of

Unsheltered, Chronically Homeless Adults,” Social Work in Public Health 29 (1): 73–80.

Mallik-Kane, Kamala, and Christy Visher. 2008. Health and Prisoner Reentry: How Physical, Mental, and Substance

Abuse Conditions Shape the Process of Reintegration. Washington, DC: Urban Institute.

Metraux, Stephen, Caterina G. Roman, and Richard S. Cho. 2007. Incarceration and Homelessness. Washington, DC: U.S. Department of Housing and Urban Development.

National Academies of Sciences, Engineering, and Medicine (NASEM). 2018. Permanent Supportive Housing:

Evaluating the Evidence for Improving Health Outcomes among People Experiencing Chronic Homelessness.

Washington, DC: National Academies Press.

Prince George’s County. 2012. Ten Year Plan to Prevent and End Homelessness in Prince George’s County 2012–

2021. Upper Marlboro, MD: Prince George’s County.

R E F ER E N CE S

1 0 3

Roman, John. 2015. “Solving the Wrong Pockets Problem: How Pay for Success Promotes Investment in Evidence-

Based Best Practices.” Washington, DC: Urban Institute.

Skopec, Laura. 2018. “Pay for Success in Health Care: Challenges and Opportunities.” Washington, DC: Urban Institute.

Technical Assistance Collaborative. 2018. Lane County Shelter Feasibility Study. Boston, MA: Technical Assistance Collaborative.

Walsh, Kelly, Brian Bieretz, Kimberly Walker, and Mayookha Mitra-Majumdar. 2017. “Why Governments and

Investors Choose Pay for Success: Perspectives on Motivations and Risks.” Washington, DC: Urban Institute.

Weber, Andrew. 2020. “With Automatic Release for Misdemeanor Cases, Travis County Takes a Step toward Bail

Reform,” KUT, February 7, 2020.

1 0 4

R E F ER E N CE S

U.S. Department of Housing and Urban Development

Office of Policy Development and Research

Washington, DC 20410-6000

August 2021

Document Outline

Evaluation of the HUD-DOJ Pay for Success Permanent Supportive Housing Demonstration

Foreword

Contents

Acknowledgments

Executive Summary

Background

Evaluation

Overview of 2019 Site Progress

Sites in Transaction Structuring Without Pilots

Sites in Transaction Structuring with Launched Pilots

Sites Implementing Permanent Supportive Housing Programming with Pay For Success

Transaction Structuring Cross-Site Themes

Implementation of Cross-Site Themes

Benefits of Pay for Success

Time Costs of Pay for Success

I. Introduction

The HUD-DOJ Pay for Success Permanent Supportive Housing Demonstration

Defining the Problem

Permanent Supportive Housing

The Pay for Success Model

The Demonstration Theory of Change

The Demonstration Evaluation in Year 3

Process Study Methodology

Annual Partnership Survey Methodology

Time Use Survey Methodology

II. Demonstration Site Overview and Progress in 2019

Transaction Structuring Sites without Pilots

Montgomery County and Prince George’s County, Maryland: Background

Montgomery County and Prince George’s County, Maryland: 2019 Progress

Rhode Island: Background

Rhode Island: 2019 Progress

Transaction Structuring Sites with Launched Pilots

Anchorage, Alaska: Background

Anchorage, Alaska: Progress in 2019

Austin/Travis County, Texas: Background

Austin/Travis County, Texas: Progress in 2019

Implementation Sites

Lane County, Oregon: Background

Lane County, Oregon: Progress in 2019

Los Angeles County, California: Background

Los Angeles County, California: Progress in 2019

III. Transaction Structuring Cross-Site Themes

Sites Employed Different Project Management Structures to Suit Current Needs

Sites Engaged Additional Technical Assistance Providers for Project Support

Feasibility and Transaction Structuring Phases Blended Together

Community Landscape and Stakeholders Shaped Project Design

Transaction Structuring Challenges

Finding and Solidifying Financial Partners

Securing Housing Resources

Designing the Evaluation

Pilots Represented an Opportunity to Refine Program Model and Demonstrate Effectiveness

Pilots Helped Refine Program Design and Operations

Pilots Helped Demonstrate Ability to Implement

IV. Implementation Cross-Site Themes

Sites Added Coordination Mechanisms for Stakeholders Involved in Permanent Supportive Housing Implementation

Sites Reported Successful Lease-Ups and High Rates of Retention

Implementation Challenges

Building Capacity to Serve “High-Need” Participants

Building Capacity in New Service Providers

Finding Housing and Developing Relationships with Landlords

Blending Service Provider and Criminal Justice Roles in PSH Implementation

Classifying Participant Exits

V. Benefits of Pay for Success

Pay for Success Strengthened Partnerships and Collaboration

Breaking Down Silos between Sectors

Raising Awareness of the Target Population and Possible Cross-Sector Solutions

Strengthening Local Data Capacity and Cross-Sector Sharing

Increasing Local Capacity through External Technical Assistance Providers

Implementation and Pilots Increased Capacity to Serve Target Population

VI. Time Costs of Pay for Success

Time Spent on the Project Varied by Type of Stakeholder

Total Time Spent Varied by Phase and by Site

VII. Conclusion

Appendix A. Example Site Visit Stakeholder List

Appendix B. Detailed Partnership Survey Methodology

Appendix C. Evaluation Methodology Limitations

Appendix D. Partnership Survey Instrument

References

Table of Contents

Evaluation of the HUD-DOJ Pay for Success Permanent Supportive Housing Demonstration

Foreword

Contents

Acknowledgments

Executive Summary

Background

Evaluation

Overview of 2019 Site Progress

Sites in Transaction Structuring Without Pilots

Sites in Transaction Structuring with Launched Pilots

Sites Implementing Permanent Supportive Housing Programming with Pay For Success

Transaction Structuring Cross-Site Themes

Implementation of Cross-Site Themes

Benefits of Pay for Success

Time Costs of Pay for Success

I. Introduction

The HUD-DOJ Pay for Success Permanent Supportive Housing Demonstration

Defining the Problem

Permanent Supportive Housing

The Pay for Success Model

The Demonstration Theory of Change

The Demonstration Evaluation in Year 3

Process Study Methodology

Annual Partnership Survey Methodology

Time Use Survey Methodology

II. Demonstration Site Overview and Progress in 2019

Transaction Structuring Sites without Pilots

Montgomery County and Prince George’s County, Maryland: Background

Montgomery County and Prince George’s County, Maryland: 2019 Progress

Rhode Island: Background

Rhode Island: 2019 Progress

Transaction Structuring Sites with Launched Pilots

Anchorage, Alaska: Background

Anchorage, Alaska: Progress in 2019

Austin/Travis County, Texas: Background

Austin/Travis County, Texas: Progress in 2019

Implementation Sites

Lane County, Oregon: Background

Lane County, Oregon: Progress in 2019

Los Angeles County, California: Background

Los Angeles County, California: Progress in 2019

III. Transaction Structuring Cross-Site Themes

Sites Employed Different Project Management Structures to Suit Current Needs

Sites Engaged Additional Technical Assistance Providers for Project Support

Feasibility and Transaction Structuring Phases Blended Together

Community Landscape and Stakeholders Shaped Project Design

Transaction Structuring Challenges

Finding and Solidifying Financial Partners

Securing Housing Resources

Designing the Evaluation

Pilots Represented an Opportunity to Refine Program Model and Demonstrate Effectiveness

Pilots Helped Refine Program Design and Operations

Pilots Helped Demonstrate Ability to Implement

IV. Implementation Cross-Site Themes

Sites Added Coordination Mechanisms for Stakeholders Involved in Permanent Supportive Housing Implementation

Sites Reported Successful Lease-Ups and High Rates of Retention

Implementation Challenges

Building Capacity to Serve “High-Need” Participants

Building Capacity in New Service Providers

Finding Housing and Developing Relationships with Landlords

Blending Service Provider and Criminal Justice Roles in PSH Implementation

Classifying Participant Exits

V. Benefits of Pay for Success

Pay for Success Strengthened Partnerships and Collaboration

Breaking Down Silos between Sectors

Raising Awareness of the Target Population and Possible Cross-Sector Solutions

Strengthening Local Data Capacity and Cross-Sector Sharing

Increasing Local Capacity through External Technical Assistance Providers

Implementation and Pilots Increased Capacity to Serve Target Population

VI. Time Costs of Pay for Success

Time Spent on the Project Varied by Type of Stakeholder

Total Time Spent Varied by Phase and by Site

VII. Conclusion

Appendix A. Example Site Visit Stakeholder List

Appendix B. Detailed Partnership Survey Methodology

Appendix C. Evaluation Methodology Limitations

Appendix D. Partnership Survey Instrument

References