5.4. How to Measure

Planning an Evaluation

When designing an evaluation, it is useful to look at how other Housing First services (or programmes or strategies using Housing First) have been evaluated and also to look at any criticism of those evaluations. The Internet is a good source of information and resources such as Google Scholar can provide information on the evaluations that have been conducted, with access to some free resources. Major evaluations of Housing First, which tend to be supported by large, publicly-funded organisations, often produce reports which are freely available on the Internet. Some guidance on evaluation is also available on the Canadian Housing First Toolkit((http://www.housingfirsttoolkit.ca/evaluate)).

Evaluation can be comparative, which can include experimental or randomised control trials, in which two exactly matched groups (a minimum of 100 people in each group is desirable) are monitored. One group uses Housing First and the other uses existing homelessness services. Over the course of a year or more, outcomes for those using Housing First are compared with those for homeless people using existing homelessness services. These comparisons are expensive to conduct, but produce high quality evidence if they are carefully designed and precise. Randomised control trials (RCTs) of this sort have been used to test the French and the Canadian Housing First programmes and have generally reported very positive results (see Chapter 1).

Housing First has also been evaluated using comparison-group, or quasi-experimental, research. Again, these evaluations compare one group using Housing First with another group using existing homelessness services, but the groups are not precisely matched and can be smaller. This kind of evaluation can still be influential, but is generally viewed as being less accurate.

Many evaluations of Housing First are observational, which means looking at the people using a Housing First service and assessing how effective the service is in addressing their homelessness, improving their health and well-being and promoting social integration (e.g. being part of a community, having social support from friends, family and a partner, see 5.3). While this approach to evaluation can produce useful and persuasive evidence, the lack of a direct comparison with other homelessness services can mean the results are seen as less convincing than evaluations using RCTs or quasi-experimental approaches.

It is very important to consider the resources and objectives of an evaluation carefully. This includes thinking through what the evaluation is testing, what arguments it may be used to support, how much time and money are available and the potential criticisms that might be made of the results. While RCTs are often described as the best possible form of evaluation, they can still be the subject of criticism and their results may be rejected, particularly if there is seen to be a problem with design or a lack of precision. An RCT cannot be done cheaply and will involve a lot of resources if it is going to be truly persuasive. Equally, a much cheaper way of evaluating, an observational approach, while it has limitations, can still be highly persuasive.

Another consideration is who will be responsible for an evaluation. An evaluation is less likely to be influential if it is produced by the organisation providing a Housing First service, than if an evaluation uses independent researchers. This is not to suggest that an in-house evaluation (an evaluation of Housing First services by the people providing the Housing First service) has no value. The evidence from a good quality in-house evaluation can still be influential. Nevertheless, the argument that an in-house evaluation will be less likely to record or report problems may be used to question the results of an in-house evaluation.

Evaluations should always include feedback from Housing First service users. Giving service users a clear voice should enable any deficiencies in Housing First services to be identified and corrected. Equally, when Housing First is performing well, service users will have a detailed understanding of good practice that can be learned from and shared. Ensuring that the people using Housing First have a voice in evaluation is useful for the following reasons:

• Homeless people are experts by experience; they understand their own needs and what support they require better than anyone else does. The views of service users on how well a Housing First service is working are a very important part of an evaluation. Both the strengths and any limitations of Housing First are best understood by talking to the people using the service.
• The direct experience of homeless people using Housing First, when Housing First is working well, is a powerful way of conveying the effectiveness of Housing First. Statistics can be used to make the case for Housing First, but that case can be made more powerfully when positive opinion from service users is combined with statistical evidence.

Using qualitative methods, i.e. talking to people using Housing First in an open way, which allows and encourages them to express their opinions, is the best way to learn from their experience. It is also possible to understand opinion through statistical surveys, but it is important that surveys are not designed solely by researchers without any consultation with the people using Housing First, who are likely to have useful views on the kinds of questions that should be asked.

How an evaluation is done depends on what the wider goals of Housing First are. For example, if Housing First is being tested for the first time in a particular country, region or municipality, it will make sense to use experimental (RCT) or comparative approaches to research. When it has not been used before, Housing First needs to be tested to see how well it performs when compared to existing homelessness services. Depending on the results of that evaluation, Housing First may then be used on a larger scale.

If the existing evidence is strong enough, either based on a local evaluation or the international evidence base, it may be decided that there is no need to comparatively evaluate Housing First services. Instead, evaluation can be mainly about outcome monitoring, to ensure that the Housing First service is performing as expected and to look for any problems.

Evaluation also needs to be proportionate. A relatively expensive evaluation, such as an RCT evaluation, is only really practical when looking at a large Housing First service or Housing First programme, not for testing a single, small Housing First service. This is because, to be robust, an RCT should involve at least 200 people (100 using Housing First and 100 using other services). It can still be very valuable to look at single Housing First services comparatively, but smaller-scale services with, for example, 20 service users can also be evaluated using quasi-experimental or observational approaches.