5.3. What to Measure

5.3.1. Use of Validated Measures

Validated measures are questions that have been repeatedly tested and found to produce consistent results. Validated measures can enhance the quality of an evaluation and mean that the results are more likely to be regarded as accurate. Validated measures can include:

• Validated questions on mental and physical health.
• Validated questions on quality of life.
• Validated questions on social integration and social support.

Some validated measures are widely used at national level, but there are also examples of measures that are used internationally. Some examples of validated measures include (note this list is illustrative only):

• The SF-12((The SF-12 health questionnaire is available at: https://www.hss.edu/physician-files/huang/SF12-RCH.pdf)) and SF-36((Available at: http://www.shcdenver.com/Portals/902/web-content/files/JamesGenuario/JG-health%20questionnaire. pdf)) measures of health and well-being.
• Lehman’s Quality of Life Interview((Pleace, N. with Wallace, A. (2011) Demonstrating the Effectiveness of Housing Support Services for People with Mental Health Problems: A Review London: National Housing Federation)) (QoLI).
• Quality-Adjusted Life Years (QALYs) used in Health Economics.
• The Self-Sufficiency Matrix((Available at: http://www.selfsufficiencymatrix.org/zrm-int.aspx)) (SSM) developed in the US and adapted for use in the Netherlands.
• The SAMSHA (Substance Abuse and Mental Health Services Administration) scale((Pleace, N. with Wallace, A. (2011) Demonstrating the Effectiveness of Housing Support Services for People with Mental Health Problems: A Review London: National Housing Federation. Vide supra)).

5.3.2. Key Questions for Evaluation

• Housing First has three sets of interrelated goals (see Chapter 2):
• Promoting housing sustainment and a lasting exit from homelessness (also known as housing retention).
• Enhancing the health and well-being of Housing First service users, including:
  • Mental health.
  • Physical health.
  • Limiting illness and disability.
  • Drug and alcohol use (where this has been an issue for someone using Housing First services).
• Improving the social integration of Housing First service users, including:
  • Gains in social support and self-esteem.
  • Engagement in community and civic life.
  • Structured and productive activity and economic integration.
  • Working on nuisance, criminal or anti-social behaviour (where this has been an issue for someone using Housing First services).
• The cost-effectiveness of Housing First, which has two dimensions:
  • The cost-effectiveness of Housing First compared to other homelessness services.
  • The cost offsets, i.e. savings, that Housing First can potentially generate for other types of service, e.g. Housing First can produce reductions in expenditure for health services because it changes the ways in which long-term homeless people use health services and their level of use of medical services falls.

Measurement of outcomes of Housing First centres on these three sets of goals. Successful outcomes for Housing First rest on achieving as much as possible for each individual service user. Housing sustainment is an achievement, but it is a limited achievement if someone is isolated, bored, feels stigmatised or is not experiencing improvements in their health and well-being. Equally, gains in social support are an achievement, but Housing First is not working well if a service user is not sustaining their housing as well.

Overall effectiveness for Housing First rests on achieving successes across a range of outcomes, related to housing, health, well-being and social integration. When successes are achieved, it is important to understand those successes in terms of the overall well-being and situation of each Housing First service user.

Evaluations of Housing First may also need to include an assessment of cost-effectiveness. This element of evaluation looks at the relative cost-effectiveness of Housing First compared to other models of homelessness services. Evaluating cost effectiveness can also include assessing whether Housing First generates wider savings in public spending.

5.3.3. Housing Sustainment

Housing sustainment can be measured in three main ways:

• Length of time a Housing First service user has lived in the same home. This approach has some advantages:
  • It is a simple measure that is instantly understandable. If a Housing First service user has been living in their home for a year, this is a clear indication of housing sustainment.
  • The measure gives an idea of housing stability, i.e. if Housing First service users are typically remaining in the first apartment they are housed in for a year or more, this indicates that housing is being very effectively sustained.
• Time spent in an apartment compared to time spent sleeping and living in other situations. This approach:
  • Provides a night-by-night measure of where Housing First service users are and allows relative changes to be recorded. For example, if someone were living rough (on the street) for three nights a week prior to using Housing First and living rough drops to one or two nights a month, there is a clear gain.
  • Can be hard to interpret unless very carefully recorded. It needs to be clear whether nights in an apartment are within the same apartment or not, or whether there was a reason for someone not to be in their apartment for a given number of nights.
• Individuals’ feelings about their homes. This approach:
• Enables assessment of how a Housing First service user feels about their home and how settled they are.
• Looks at the success of housing in a wider sense, including:
  • whether someone feels physically safe in their home;
  • whether their home is affordable;
  • whether their home has all the facilities they need;
  • whether their home is of an adequate standard (damp, poor repair or poor space standards);
  • views on the neighbourhood where their home is located;
  • how happy a Housing First service user is with their home.

5.3.4. Health and Well-Being

There are three ways to measure health and well-being:

• Use very basic measures based on people’s own judgement about how their health is and whether there are any changes in drug/alcohol use (where this is relevant).
  • Using basic measures of whether someone feels they are getting better or worse, in terms of their physical health, mental health and drug/alcohol use, is very simple.
  • Answers will be subjective, i.e. they will be influenced by an individual’s interpretation of their health and well-being, which may be more positive, or more negative, than the view a medical professional would take.
  • Answers cannot be compared systematically, because the information being collected is not consistent (Housing First service users will not all interpret their health and well-being in the same way as each other).
• Use validated measures of health and well-being. A validated measure is one which has been repeatedly tested and found to be accurate in recording health and well-being. An example is the SF-12 health questionnaire, which has been widely used in surveys and statistical research, which establishes basic information on physical and mental health. This approach:
  • Allows the collection of data that can be compared over time and across Housing First service users, because questions and responses take place within a clearly-defined and consistent framework.
  • Collects data that may carry more influence in the outside world, because they use recognised standards of measurement that have been tested.
  • Will be more complex and expensive to administer than just asking very simple questions about health.
• Employ external evaluation of health and well-being. Medical teams and psychiatrists could be used to test health and well-being among Housing First service users over time. This is feasible and is likely to generate evidence that is taken seriously by external agencies, but may be difficult to fund.

5.3.5. Social Integration

In some respects, social integration is the hardest of the various outcomes to measure:

• Social support, participation in community and civic life and the nature and extent of structured activity are very subjective. When two individuals receive the same levels of social support, one may report that they are isolated and bored, and the other may feel supported and happy.
• Social integration can be interpreted in different ways for different groups of people. In Europe, it is quite common to talk about the lack of ‘community’ in poor areas as a social problem, but not to view the lack of ‘community’ in rich areas as being a social problem. It is important not to impose an ideal of what a ‘citizen’ should be on people using Housing First, when most other citizens do not match that same ideal.
• Validated measures of social support are available, but this is an area where qualitative outcome measurement, i.e. talking to Housing First service users about their lives and level of social integration, may be the most effective way to collect information.
• Measurement of social integration must take into account the other needs, characteristics and experiences of Housing First service users. If many people using a Housing First service have ongoing, limiting illnesses, this will influence how much success can be achieved with economic integration.

Measurements of social integration might include the following:

• Social support
  • Is the user in contact with their family?
  • Is the user in contact with friends?
  • Do they have a partner?
  • Do they have esteem support, a sense they are valued by others, and what is their level of self-esteem?
  • Do they have access to instrumental (practical) support from friends, family and/or a partner?
  • Do they have sufficient social companionship?
  • Are there people they can ask for advice and/or talk to?
• Community and civic participation
  • Does a Housing First service user participate in community events?
  • What are their relationships with their neighbours like?
  • Do they socialise within their community?
  • Do they participate in social media focused on their community?
  • Do they vote?
  • Do they volunteer in their community?
• Structured activity and paid work
  • Does a Housing First service user participate in the creative or performing arts?
  • Are they in education or receiving training?
  • Are they volunteering (in any capacity)?
  • Are they participating in a work placement/work experience scheme?
  • Are they in paid work?

5.3.6. Cost-Effectiveness

The measurement of the cost-effectiveness of Housing First services is heavily reliant on access to good quality, detailed, data. It is possible to produce estimations of cost-effectiveness, but these are less influential than detailed information that clearly shows Housing First delivering effective services. It is important to note that cost-benefit analysis is a distinctive, highly detailed and complex form of economic evaluation which should not be confused with evaluation of cost-effectiveness. There are two basic tests of cost-effectiveness which can be used for Housing First or other homelessness services((Pleace, N., Benjaminsen, L., Baptista, I. and Busch-Geertsema (2013) The Costs of Homelessness in Europe: An Assessment of the Current Evidence Base Brussels: FEANTSA http://housingfirstguide.eu/website/the-costs-of-homelessnessin-europe-an-assessment-of-the-current-evidence-base)):

• Is Housing First achieving better results than existing homelessness services for the same level of spending and/ or for a lower level of spending?
• Is Housing First producing cost offsets((Ibid.)), i.e. reductions in expenditure, for other publicly funded services? For example, by ending long-term and repeated homelessness, Housing First may produce savings for emergency health services, mental health services, drug and alcohol services, the criminal justice system, welfare systems and other homelessness services. It is important to explore whether these savings are realisable, i.e. the reductions in long-term and repeated homelessness delivered by Housing First really do allow publicly-funded services to reduce spending.