FORUM on Corrections Research

The Probation Evaluation Project⁽¹⁾

This article focuses on evaluation, one of the numerous trends in current corrections research that appears to be gathering pace in many agencies. While one might assume that program evaluation is a standard feature of research in criminal justice, researchers and practitioners know that this is not the case.

There are probably several reasons for this. Customary ways of doing things are rarely evaluated because people simply take for granted that they work. Innovative action-research programs often involve considerable action but comparatively little research. Between the customary and the innovative is a range of programs in which the evaluation component is either forgotten, not covered in the costs or abandoned halfway because the evaluator moved to a permanent job.

The project described in this article is an evaluation of probation services in the city of Manchester, United Kingdom.

The Greater Manchester Metropolitan area, with a population of about 2.75 million people, is the third-largest urban area in the U.K. and has one of the country's highest crime rates.

Greater Manchester Probation Service has approximately 250 trained probation officers plus auxiliary and support staff. To illustrate the work it does, in 1990 the Service prepared a total of 15,571 Social Inquiry Reports (similar to Canada's presentence reports) for courts in the Manchester area.

Probation staff also provide social supervision to adult offenders allocated to them. This work may include aspects of welfare provision. The dominant model of service delivery, however, remains individually focused social casework. This could still be characterized as the "mainstream" mode of probation practice, though today it may be more likely to involve some analysis of offence behaviour and its causes.

Given its size and the diversity of the area it serves, the Greater Manchester Probation Service offers many specialist services designed to meet the specific needs of select groups of probation clients. The evaluation project focuses on some of these services. Converging Trends In recent years, observers from a variety of standpoints have converged in their opinion that probation departments need to become more actively involved in evaluation. Probation work, is at one and the same time, a form of social service, a kind of applied social science and a publicly funded agency. For these rather different reasons, some have argued that a detailed evaluation of probation work is long overdue.

In general terms, evaluating correctional programs can serve two principal purposes: one is actuarial (i.e., based on actual numbers) and the other is scientific. Successive British governments have shown steadily increasing interest in some form of actuarial approach to criminal justice agencies.

Probation, though a small part of the total criminal justice budget, has not been excluded from this process. There has recently been systematic pressure for a more detailed appraisal of the nature of probation work. This concern has been expressed most clearly by a report of the British government's Audit Commission, published in 1989. (This is a central government body which monitors budgets of city and county authorities.) Turning its attention to probation, the Commission declared that: ...while there is a striking variety of probation schemes in operation involving much vision, creativity and imagination, these schemes must be evaluated and their impact on offending behaviour assessed. It is unsatisfactory that at present considerable sums are spent with relatively little understanding of the effects achieved.⁽²⁾
Probation departments were therefore urged to evaluate their work to determine its effectiveness and disseminate this information to agency staff.

But parallel to this politically or economically driven desire to look more closely at probation activity, there is a quite separate reason for wishing to conduct an evaluation. As most researchers who have surveyed the fields of criminology and penology have discovered, a very large proportion of programs - even of those that are evaluated - are not evaluated well.

Going back to the 1970s when Martinson⁽³⁾ in the United States and Brody⁽⁴⁾ in the United Kingdom published their major literature reviews, a foremost issue was that a majority of existing studies were plagued with methodological problems, which made it difficult to draw any clear conclusions on effectiveness. Caveats reinforcing this point have been issued routinely through the years, most recently by Palmer: Without scientifically sound research to independently determine if and with whom programs have worked, interventions that receive even strong testimonials and high acclaim will probably fade after several years.⁽⁵⁾ Other commentators, such as Petersilia,⁽⁶⁾ have in a similar vein endorsed the view that without systematic, well-conducted evaluation, no clear evidence of effective intervention can be obtained for either scientific or policy-making purposes. The Probation Evaluation Project The Probation Evaluation Project was founded on these two complementary sets of ideas. Evaluation of interventions simply makes good sense, whether seen from a fiscal or a scientific point of view. The Project was a joint one undertaken by the Greater Manchester Probation Service in conjunction with the Department of Clinical Psychology at the University of Liverpool. The work was undertaken between October 1991 and September 1992.

Our research aims were to: evaluate the effectiveness of a range of probation programs, identify what works in them, and pass this information on to probation staff and to courts.

In selecting programs to evaluate, we examined a wide range. The Service operates 11 day centres plus a variety of other specialized activities ranging from drop-in units for homeless clients and alcohol education courses, to cognitive training programs. Only projects whose staff agreed to participate were considered.

Six programs were selected for inclusion in the research. They comprised:

• the DIAL project, an 8-week group program for individuals convicted of drunk-driving offences;
• the STOP program, focused specifically on auto crime;
• an offence-focused group program employing a cognitive-behavioural approach⁽⁷⁾ ;
• a more traditional form of day centre incorporating life skills and recreational activities in addition to more focused groups;
• a Reasoning and Rehabilitation program following the manual designed by Ross, Fabiano and Ross⁽⁸⁾ ; and
• a process-oriented therapeutic program aimed exclusively at women offenders.

In terms of the evaluative methods used in this research, we were eager to combine both internal and external criteria in the kinds of data that we collected. It seemed fruitless to apply the same battery of measuring instruments to such a diverse set of programs. We therefore chose to use a composite approach in which there were some common measures applied to all programs, while others were specially selected as uniquely suited to each particular program.

We also intended to relate the outcome of a program to its declared aims, and to look inside each program and get a sense of how its staff members perceived their roles. This method could be typified as being closest to the goal-oriented approach to evaluation as characterized by Stecher and Davis.⁽⁹⁾

Thus, several kinds of data were collected, though they could broadly be characterized as being of six principal sorts:

• general descriptive information on each unit and its resources;
• details of the aims of each program and the methods employed within it;
• criminological data on program participants and, where possible, on matched comparison groups;
• monitoring data concerning attendance and drop-out rates;
• "consumer feedback" from clients; and
• pre- and post-test measures on targetted psychological variables, tailored specifically to each project's aims.

The last of these ingredients allowed for the greatest flexibility in evaluation approaches. For example, for the Reasoning and Rehabilitation program, which emphasizes cognitive change, we used a series of measures of impulsiveness, social problem solving and locus of control to appraise levels of cognitive change. For the drunk-driving program, measures incorporated scales for assessing attitudes to driving and to alcohol.

In addition to such specific measures, risk of reconviction and self-esteem scales were used across all program samples, and it is proposed that standard criminological follow-up data be collected at a later stage. In this way, both the actuarial and scientific demands made of evaluation are combined to produce data that would be valuable from various standpoints including practice, research and policy formulation. Patterns of Outcome The results of this evaluation were generally positive regarding the short-term impact of probation programs on their clients. Though only a one-year research project, the evaluation generated considerable data, which are summarized in the research report.⁽¹⁰⁾

There were some difficulties, of course. Foremost among them was the very high attrition (drop-out) rate of offenders during the period between the court decision to place them on probation and the starting date of program work. At some sites, offenders also dropped out during the program itself.

Overall, however, for those attending programs, there were subjective reports obtained from interviews and rating scales that showed the activities had been beneficial to clients. Across all programs, there was a net gain in self-esteem and a reduction in numbers of perceived problems. Both of these changes were statistically significant in probation-client groups as compared with control groups.

The programs were also examined from a different perspective. They were compared with the findings of recent meta-analytic studies outlining some of the features that differentiate effective programs from ineffective ones across corrections. Reviews such as those of Andrews⁽¹²⁾ and Lipsey,⁽¹²⁾ surveying the results of numerous outcome studies, have indicated that interventions are more likely to lower recidivism if they:

• target offenders with a high risk of reoffending;
• focus on criminogenic behaviours;
• are located in the community;
• employ cognitive- or behaviourally based methods;
• are relatively more structured and directive in style; and
• possess high treatment integrity.

Scrutinizing the Manchester programs according to these criteria, we discovered that the more of these ingredients a program contained, the more positive outcome criteria were obtained from it. This relationship was by no means exact, but the general pattern was nevertheless clear. What we were able to conclude then, at the end of this probation evaluation, was that the factors found to be important in large-scale, wide-ranging literature surveys could be found operating at a local level within one county Probation Service.

We hope that such results will be of some interest to a wide spectrum of personnel, from practitioners and managers in agencies to researchers and policy makers alike.

---

⁽¹⁾Correspondence concerning this article should be addressed to: Dr. James McGuire, University of Liverpool, Department of Clinical Psychology, Ground Floor, Whelan Building, P.O. Box 147, Liverpool L69 3BX, United Kingdom. Tel:⁽⁴⁴⁾ 51 794-5529; fax: ⁽⁴⁴⁾51 794-5537.
⁽²⁾Audit Commission, The Probation Service: Promoting Value for Money (London: Her Majesty's Stationery Office, 1989), p.2.
⁽³⁾R. Martinson, "What Works? - Questions and Answers About Prison Reform," The Public Interest, 10(1975): 22-54.
⁽⁴⁾S.R. Brody, The Effectiveness of Sentencing: A Review of the Literature, Home Office Research Study no.35 (London: Her Majesty's Stationery Office, 1976).
⁽⁵⁾T. Palmer, The Re-emergence of Correctional Intervention (Newbury Park, Calif: Sage Publications, 1992), pp.174-175.
⁽⁶⁾J. Petersilia, "The Value of Corrections Research: Learning What Works," Federal Probation, June (1991): 24-26.
⁽⁷⁾J. McGuire and P. Priestley, Offending Behaviour: Skills and stratagems for Going Straight (London: Batsford, 1985).
⁽⁸⁾R. Ross, E. Fabiano and B. Ross, Reasoning and Rehabilitation: Trainer's Manual (Ottawa: Cognitive Station, 1990).
⁽⁹⁾B.M. Stecher and WA. Davis, How to Focus an Evaluation (Newbury Park, Calif: Sage Publications, 1987).
⁽¹⁰⁾J. McGuire, D. Broomfield, C. Robinson and B. Rowson, "Probation Evaluation Project: Research Report." Unpublished manuscript, University of Liverpool and Greater Manchester Probation Service, 1992.
⁽¹¹⁾D.A. Andrews, I. Zinger, R. Hoge, J. Bonta, P. Gendreau and F. Cullen, "Does Correctional Treatment Work? A Clinically Relevant and Psychologically Informed Meta-Analysis," Criminology, 28(1990): 369-404.
⁽¹²⁾M.W. Lipsey, "Juvenile Delinquency Treatment: A Meta-Analytic Inquiry into the Variability of Effects," in T.D. Cook, H. Cooper, D.S. Cordray, H. Hartmann, L.V. Hedges, R.J. Light, T.A. Louis and F. Mosteller (eds.), Meta-Analysis for Explanation: A Casebook (New York: Russell Sage Foundation, 1991).