Data Use and Challenges in

Using Pay for Success

to Implement Permanent

Supportive Housing:

Lessons From the HUD-DOJ

Demonstration

U.S. Department of Housing and Urban Development | Office of Policy Development and Research

Data Use and Chalenges in Using

Pay for Success to Implement

Permanent Supportive Housing:

Lessons from the HUD-DOJ Demonstration

BY SARAH GILLESPIE, SAMANTHA BATKO, AND AKIVA LIBERMAN

URBAN INSTITUTE

The U.S. Department of Housing and Urban Development (HUD) and Department of Justice (DOJ) have partnered to advance pay for success (PFS) as a promising model for financing permanent supportive housing for individuals who are the highest users of jails, shelters, emergency rooms, and other costly crisis services. Permanent supportive housing is an approach to breaking the cycle of criminal justice and homelessness for this population, with the potential for reducing the social costs of homelessness and recidivism, as wel as the financial costs these impose.

PFS is an approach to financing in which the government pays for outcomes achieved rather than directly for the services intended to achieve those outcomes. As described in the funding announcement for this Demonstration:

PFS contracting models involve payors and service providers who agree that al or some portion of payment will not be paid until an agreed-upon set of outcomes or level of impact has been verified. Such payments are known as Success Payments. Instead of being applied to the direct costs of housing or services, success payments are made based on the degree to which specific milestones agreed upon in advance are achieved, using payment rates also determined in advance. Achievement of outcomes is typical y verified by an independent evaluator... (HUD

Notice of Funding Availability, October 2015, p. 5) PFS typical y involves third-party social investors (usually private organizations or nonprofits) who initially fund a social program and are repaid (with a potential positive return) only if agreed-upon outcomes are achieved, as measured by an evaluation; these investors also assume the risk that the program may not demonstrate success and they may not be fully repaid. PFS is a good fit for social programs expected to decrease costs to government by achieving better outcomes than business as usual.

The PFS model creates a considerable demand for data in all phases of the project. Critical tasks—

such as using data to identify the right target population for the intervention and collecting accurate outcome data that will dictate success payments—often elevate data chal enges beyond what communities may experience in other types of projects. In the seven sites (Exhibit 1) funded by the HUD-DOJ Pay for Success (PFS) Permanent Supportive Housing (PSH) Demonstration across the country, communities are using the PFS model to provide permanent supportive housing to break the criminal justice-homelessness cycle among individuals with complex medical and behavioral health needs. These communities’ experiences offer early lessons about the chal enges of using data across the three domains of criminal justice, housing, and health in PFS projects.

EXHIBIT 1

HUD-DOJ PFS PSH Demonstration Sites

In June 2016, the Demonstration funded seven grantees across the country in Anchorage and Matanuska-Susitna Borough, Alaska; Pima County, Arizona; Los Angeles (LA) County, California; Montgomery County and Prince George’s County, Maryland; Lane County, Oregon; Rhode Island; and Austin/Travis County, Texas. Grantees were funded for up to three phases of the PFS lifecycle, from feasibility assessment to transaction structuring to project implementation (including evaluation and success payments). Most sites were funded beginning with feasibility analyses. LA County’s project was unique in that it built on an earlier pilot project and was close to a final PFS transaction at the time of the grant. Its HUD funding thus covered transaction structuring and PSH project implementation, and PSH implementation was initiated in 2017. Austin, too, was funded beginning with transaction structuring, which was ongoing at the end of 2017. The national evaluation of the demonstration is tracking the progress of each pay for success project and documenting the lessons and chal enges each community faces along the way.

Data Demands During the PFS Lifecycle

In the HUD-DOJ PFS PSH Demonstration, PFS is implemented in three phases: feasibility analysis, transaction structuring, and PFS project implementation. During the feasibility analysis phase, institutional and programmatic feasibility are examined through data analysis, and the resulting data demands are the focus of the present brief.

Feasibility analyses are also guided and influenced by informal and formal conversations with both implementation partners and possible end payors about what kind of project might be implemented.

The feasibility analysis is conducted by the project intermediary, and a report is then presented to the project leadership and partners, often through an executive committee or equivalent. If the feasibility report finds project implementation feasible, project partners and HUD can approve initiation of transaction structuring.

2

DATA US E AN D CHALLENGES IN US ING PAY FOR S UCCES S



In the feasibility analysis phase, data are needed to explore potential target populations and their size, to identify effective services and providers, to estimate anticipated outcomes, and to model costs and benefits for different government agencies that may make payments if outcomes are achieved.

In the transaction structuring phase, al details of the project must be specified in a legal contract.

Details may include considerable implementation detail beyond what was outlined in the feasibility report, such as identifying service providers and how they wil be contracted; housing options and how they wil be paid for; and the implementation plan, including when and where potential clients wil be identified, recruited, and enrol ed. The contract generally incorporates this with financial arrangements, evaluation requirements, and payment stipulations stating how success payments from the government to investors will be determined and paid.



In the transaction structuring phase, feasibility analyses might be refined in response to the needs and requirements of different government agencies or potential investors and to design evaluation metrics and determine performance thresholds for success payments.

Once a contract is finalized, the implementation phase begins. This phase includes identification and enrol ment of individuals into services, data col ection, and a formal evaluation that determines success payments based on agreed-upon outcomes. Considerable data are needed to complete each PFS phase successfully; in the HUD-DOJ PFS PSH Demonstration, this work is general y led by the PFS intermediary organization.



In the project implementation phase, real-time data may be necessary to identify potential clients and monitor their progress. Success payments wil depend on rigorous outcome and impact evaluation, which requires individual-level data for participating clients and comparison populations.

In the HUD-DOJ Demonstration, individuals are targeted for permanent supportive housing based on their interactions with the criminal justice, homelessness, and health sectors—factors that require negotiating data sharing across systems that have not traditionally done so in many communities.

Based on ongoing qualitative data col ection, including site visits and self-assessments of data use and chal enges completed by grantees in the fal of 2017, this brief examines the early experiences and lessons learned about data use and chal enges among the seven communities engaged in the PFS

Demonstration.

Data Use in the HUD-DOJ PFS Demonstration

Communities were funded for different phases of the PFS lifecycle and, therefore, were doing different work during the first year of the Demonstration. Depending on the phase of work for each project, we examined somewhat different questions about data use: DATA US E AN D CHALLENGES IN US ING PAY FOR S UCCES S

3



For sites working on feasibility analyses, which data did they attempt to use to identify the target population and understand baseline outcomes?



For sites without a completed transaction, which data did they anticipate using during project implementation and outcome evaluation?



For the one site that began implementation in 2017 (LA County), which data are being used during project implementation and outcome evaluation? (LA County’s HUD Demonstration grant fol owed an earlier pilot project and did not support feasibility analysis.) For feasibility analyses, most sites were planning to or had accessed data from three different systems: homelessness, criminal justice, and health.

Many of the sites began data analysis during feasibility with a focus on identifying a target population with a history of homelessness and other indicators of vulnerability. Sites often measured vulnerability through an assessment tool such as the VI-SPDAT1 (a combination of the Vulnerability Index and the Service Prioritization Decision Assistance Tool). At sites further along on developing a coordinated entry process consistent with HUD requirements2 that people with the greatest needs are prioritized for any assistance—including PSH, partners reported that homelessness data were a natural starting point for identifying the target population. For example, both Anchorage, Alaska and Austin, Texas already had experience creating a registry that documented their community's homeless population and needs (known as a by-name list). This information made it easier to match the homeless population with the data from the other systems to assess how many services they were using and at what cost.

Most sites used or were working to use both law enforcement and incarceration data to understand the anticipated criminal justice involvement among the target population and to understand the benefits and cost saving that would accrue from reducing recidivism. For feasibility analyses, sites most often used data on arrests and jail stays from local police departments; most sites were focused on jail reentry and return. Pima County, Arizona for example, used jail data that included homelessness status. Lane County, Oregon focused on prison stays, but as some individuals serve part of their felony convictions in jails, both systems are important sources of data. Incarceration data often involve using information from local sheriff’s departments (jail stays) and from state departments of correction (prison stays). In Alaska, however, both jail and prison data can be col ected 1 HUD mandates that communities establish a coordinated entry process that prioritizes homeless assistance services to those most in need. Many communities use the VI-SPDAT survey to determine risk and prioritization for services among individuals and families experiencing homelessness or at risk of homelessness. This standardized screening and assessment tool covers both medical and social risk factors that can contribute to homelessness.

2 The Homeless Emergency Assistance and Rapid Transition to Housing (HEARTH) Act requires coordinated entry. See https://www.hudexchange.info/resources/documents/Coordinated%20Assessment_3.20.12.pdf and

https://www.hudexchange.info/resources/documents/ImplementingCoC_7.24.pdf.

4

DATA US E AN D CHALLENGES IN US ING PAY FOR S UCCES S

from the Alaska Department of Corrections because of its unified correctional system and because jails are part of the state system. The municipality of Anchorage also has important local data, because it runs the local Anchorage Safety Center to temporarily house individuals incapacitated by alcohol or drugs in a public place. More generally, criminal justice data are organized differently in different jurisdictions, which depends in part on the local structure of criminal justice system. Other sources of data included court data and data on parole and probation.

Sites also worked to obtain and use data on high-cost health services for the target population, such as emergency room visits or inpatient behavioral health services. As discussed in the fol owing pages, this proved to be a chal enge in many communities. In Austin/Travis County, Texas, however, health data integration was a strength of the feasibility analysis and transaction structuring phases.

The community had previously established an Integrated Care Col aboration (ICC)3 consisting of several hospitals and other member healthcare providers. The ICC created a health information exchange to link data from multiple healthcare sites across multiple systems, in accordance with established laws and policies. This exchange facilitated linking available health data to data from the homelessness management information system (HMIS) and criminal justice system for the project.

In the fal of 2017, sites reported planning to measure outcomes across al three domains, but success payments were primarily linked to homelessness and criminal justice measures.

As part of the outcome evaluation, all sites anticipated measuring criminal justice, homelessness, and health outcomes. For success payments, however, most sites focused on housing and criminal justice outcomes. For housing-related payment outcomes, sites considered shelter stays and/or days in stable housing. Those sites planning to use criminal justice measures as a payment outcome focused on returns to jail or prison as indicators of recidivism.

When debating payment outcomes, some sites grappled with the more limited evidence base for the impact of permanent supportive housing on criminal justice outcomes. LA, for example, planned a quasi-experimental evaluation that structured success payments based simply on outcomes of program participants, without reference to a comparison group. Investors believed there was a stronger research base for setting housing benchmarks than for setting criminal justice benchmarks, and ultimately, 60 percent of success payments was based on participants’ housing retention, and 40

percent on participants not being arrested.

In part because of the difficulty of data access and the limited evidence base, health outcomes were the least commonly anticipated payment measures. Austin/Travis County is seriously exploring health-related success measures, given the project’s plans to include end payors from the nongovernmental health care sector. Exhibit 2 il ustrates the number of sites that reported using or 3 For more information on the Integrated Care Col aboration in Austin/Travis County, TX, see

http://communitycaretx.org/outreach/partnerships.html/title/integrated-care-col aboration and http://icc-

centex.org/.

DATA US E AN D CHALLENGES IN US ING PAY FOR S UCCES S

5

planning to use different types of data for feasibility analyses, project implementation, and outcome evaluation.

EXHIBIT 2

Data Use by PFS Lifecycle Phase

Number of Sites

0

1

2

3

4

5

6

7

ta

Arrests

e Da

Jail stays

ustic

Prison stays

al J

Health care in jail

in

rimC

Shelter stays/Homeless services

ata

nd

Housing retention/Stability

g a sness D

Vulnerability index

ousin esel

Homeless service referrals

H omH

Emergency room visits

ata

Ambulance rides

Behavioral health services

ealth D

Hospitalization/Inpatient days

H

General health services

Feasibility analyses

Project implementation

Outcome evaluation

Note: Bars represent the number of Demonstration sites reporting intended use of data among seven sites.

Source: Data Self-Assessments, September 2017

Data Chal enges in the HUD-DOJ PFS Demonstration

Based on each project’s progress so far, we examined the fol owing questions:



How successful are sites in involving stakeholders to help navigate data access?



How accessible are the needed types of data in each domain?



What privacy issues or other issues pose barriers to accessing and/or linking data in each PFS

phase?

6

DATA US E AN D CHALLENGES IN US ING PAY FOR S UCCES S

Data challenges arose mainly in four areas: stakeholders, data quality, privacy, and timeliness of data.

Stakeholders actively engaged in PFS working groups helped ease data access.

Most sites included stakeholders representing agencies that own data in each of the three data domains (criminal justice, housing and homelessness, and health) in their PFS working groups, often cal ed steering or governance committees. This representation was a factor in each project’s ability to access data; maintaining active partnerships helped ease data access across systems. Partners helped negotiate data sharing agreements or execute memoranda of understanding and helped allocate or prioritize staff time for data requests.

For example, in LA County, California, a long-standing relationship between the intermediary and the jails, though their pilot project, facilitated access to individual-level jail data. In Lane County, Oregon the three primary partners were funded as subgrantees (Sponsors, a service provider; Lane County Parole and Probation; and the Lane County housing authority) to support their time and work on the PFS project. In Rhode Island, project partners established a biweekly working group meeting of stakeholders, including organizations with necessary data on the target population and current housing and service inventory. The support of these organizational representatives—including from the HMIS system, Department of Corrections, and Executive Office of Health and Human Services—

were critical in helping match data for the feasibility analysis.

For sites lacking active stakeholders from each system, access to individual-level data presented a larger challenge. For example, in Alaska, while accessing individual-level data from Medicaid and corrections was helped by the PFS partnership, the site struggled with homeless data because the partners with access to that data were not as engaged. Several projects also required additional data from agencies beyond those involved with the project, and having partners from each sector helped provide introductions to or leverage relationships with those other agencies.

Data quality concerns surfaced in understanding homelessness in each community.

Data quality surfaced as a challenge across various domains. For some data, the quality concern focused on the lack of appropriate identifiers to link individuals across systems. Identifiers used in one system—such as names or social security numbers—might not be available in other systems or might be recorded incorrectly and introduce challenges for data matching. Data quality was also commonly reported as a challenge among sites using data from their local HMIS, a community data system used to col ect client-level data, and data on the provision of housing and services to individuals and families at risk of and experiencing homelessness. Gaps in coverage, different definitions and standards, and the difficulty of capturing data on individuals experiencing al forms of homelessness contributed to the HMIS data quality challenges. HMIS coverage rates, or the share of homeless service providers reporting data to HMIS, vary from community to community and among each of the Demonstration sites. In sites where some shelters do not report to HMIS, projects had to negotiate separate data sharing agreements with those service providers to build a ful picture of individuals DATA US E AN D CHALLENGES IN US ING PAY FOR S UCCES S

7

experiencing homelessness in the community. Other sites struggled to align the definitions and data standards for reporting individuals experiencing homelessness across different systems. For example, in at least one site, the local police department used a different definition of homelessness and captured a different catchment area than the local HMIS system, making data harder to match and compare.

Final y, sites reported that local HMIS systems did not necessarily include individuals experiencing al the types of homelessness their projects would be interested in serving. HMIS data can miss individuals who are experiencing homelessness but avoid shelters or services, instead opting for camps and other unsheltered situations. For example, in the Alaska site, a summer point-in-time count identified over 100 unsheltered persons who were a match for the other systems but were not in HMIS. In addition, individuals just returning from long periods of incarceration might be underrepresented in HMIS data, given their limited time in the community at that point, but might still be an appropriate target population. In Maryland, the Dept. of Corrections identified some individuals as homeless, but they were not found in HMIS due to definitional and eligibility issues. One site was working to identify this population by modifying an existing homeless assessment tool to include relevant criminal justice information.

Privacy concerns most commonly surfaced when accessing data on health services.

Data privacy concerns also surfaced across various domains. In one site, negotiating access to homelessness data held by a nongovernmental entity for feasibility analyses raised concerns about whether prior client releases of information covered this use of the data. Some sites also reported chal enges negotiating confidentiality issues with criminal justice partners. The most common privacy issues by far, however, were related to access to health data and concerns around Health Insurance Portability and Accountability Act (HIPAA) compliance. These privacy concerns made it difficult to negotiate access to individual-level data on health service use and link these data with other systems to explore the overlapping needs of individuals across sectors. For example, identifying information such as an individual’s name is often protected health information, so sites commonly reported chal enges with finding appropriate identifiers to link health data with other systems. For project implementation and outcome evaluation, some sites planned to obtain consent for access to health data from clients who participate in a supportive housing program but recognized that such consent would be hard to obtain for potential comparison groups or retroactively for the feasibility analysis tasks.

Approaches to Managing Privacy Issues

The Demonstration sites tried different strategies to address privacy issues associated with access to health data. One approach was to link data from the criminal justice and homelessness systems first and then send the identified, merged dataset to the data partner in the local health system. The health system then searched for the individuals in that merged dataset in its own data, attached the relevant health data, deidentified the final dataset to be analyzed by the PFS project, and sent it back. In Rhode

8

DATA US E AN D CHALLENGES IN US ING PAY FOR S UCCES S

Island, partners fol owed a similar matching process but then provided aggregate data back to the PFS

project for analysis rather than individual-level data. While this worked for feasibility analyses, the site recognized that agreements would need to be negotiated for individual-level data for the outcome evaluation.

Other sites worked to connect with local y established data hubs that could receive identified data from multiple sources, merge the data, and then provide deidentified data sets to authorized users such as evaluators or intermediary organizations. For example, Prince George’s County, Maryland worked to leverage its participation in the Data Driven Justice initiative led by the National Association of Counties, which aims to build local integrated data platforms. As mentioned earlier, Austin/Travis County, Texas leveraged the community’s existing health information exchange to match data on health services with other local systems.

Finally, one site reported exploring a HIPAA waiver for the data and analysis associated with its PFS project. Institutional Review Boards (IRBs) are tasked with reviewing data privacy and confidentiality concerns (among other issues) of covered research projects. When IRBs have been authorized by the U.S. Department of Health and Human Services to review multiple projects, they have the authority to review and waive the informed consent requirement for the release of HIPAA-protected data for research purposes, given the appropriate considerations. 4

Timeliness of data requests will be important in implementation and evaluation.

The last area in which many sites reported data challenges focused on the time it took to access and analyze data, which led to project delays in some cases. Sites faced delays while navigating multiple layers of approvals for data requests or while waiting for staff to have time to work on data requests.

Several sites also noted the long lag in Medicaid claim data. While such delays could be managed during the feasibility analysis phase for analyses based on historical data, sites recognized that challenges with more frequent or regular data pulls could be more problematic during project implementation and outcome evaluation, when service delivery and success payments wil depend on data access.

Exhibit 3 il ustrates the number of sites that reported data chal enges discussed in this section when accessing or analyzing data from the criminal justice, housing and homelessness, and health systems.

4 The IRB must determine that three criteria are satisfied: 1) The use or disclosure of protected health information involves no more than a minimal risk to the privacy of individuals. (Note that “minimal risk” is a technical term meaning approximately everyday risk.) 2) The research could not practicably be conducted without the waiver. 3) The research could not practicably be conducted without access to and use of the protected health information.

For more detail concerning these requirements, see https://www.hhs.gov/hipaa/for-professionals/special-

topics/research/index.html.

DATA US E AN D CHALLENGES IN US ING PAY FOR S UCCES S

9

EXHIBIT 3

Data Challenges by Domain

7

6

5

4

3

2

1

0

Data sharing

Data owner not in PFS

Data system not

Individual identifiers Data privacy issues to

agreements not yet in

working group

automated

not yet accessible in

navigate

place

data

Access chal enges

Quality chal enges

Privacy

Criminal justice data

Housing and homelessness data

Health data

Note: Bars represent the number of Demonstration sites reporting at least one chal enge within each domain among seven sites.

Source: Data Self-Assessments, September 2017

Conclusion

PFS places a premium on accessing and analyzing data to develop and implement an effective program and col ect outcome data that reliably measure outcomes for the purposes of success payments. All PFS projects face these types of data demands, even when focused on populations and outcomes in just one sector. Applied to a cross-sector project—as in the HUD-DOJ PFS Demonstration, which requires data concerning criminal justice, housing and homelessness, and health care utilization—those PFS data demands are raised considerably.

Data use can look different across the lifecycle phases for a PFS project. Although some feasibility analyses can be done with aggregate data, most projects seek individual-level data even in this early phase. PFS partners recognize the data challenges faced during feasibility analysis wil likely be even more critical during later phases when effective implementation and determination of success payments wil rely on quality and timely data access and analyses.

In the first year of the initiative, many of the seven sites in the HUD-DOJ PFS Demonstration made important progress bringing stakeholders to the table to support data access, negotiating privacy concerns and data sharing agreements, and problem-solving data quality issues or delays in data access. Even so, the process of accessing, linking, and analyzing data across three systems required

1 0

DATA US E AN D CHALLENGES IN US ING PAY FOR S UCCES S

considerable effort. Ongoing chal enges played a role in lengthening feasibility analyses beyond the anticipated 1-year timeline, as sites found ways to work through and around these issues to ensure their PFS projects are grounded in the best available data.

DATA US E AN D CHALLENGES IN US ING PAY FOR S UCCES S

1 1

U.S. Department of Housing and Urban Development Office of Policy Development and Research

Washington, DC 20410-6000

December 2018

Document Outline

- Blank Page

Table of Contents

1

3

4

Blank Page