FORUM on Corrections Research

Tried and true: Proof that the Custody Rating Scale is still reliable and valid

by Fred Luciani¹
Research Branch, Correctional Service of Canada

In the last decade, the Correctional Service of Canada has introduced a number of standardized assessment instruments and related protocols to guide correctional decisions on a rangeof areas from offender admission through to sentence expiry. Increasingly, managers and case management officers are asked to anchor their decisions in empirically derived, objective, risk assessment tools. Decisions governing initial security classification,² offender intake assessment³, recidivism potential⁴ psychological intake assessment⁵ and conditional release supervision strategies are now supported by standardized assessment protocols. Collectively, these measures represent an integrated, contiguous system of structured assessment relevant to the critical stages of incarceration and release.

Objective classification instruments minimize subjective bias,⁶ promote fair and equitable treatment and are helpful in planning accommodation needs and defining correctional strategies. They make public a correctional agency’s security classification norms and consequences for offender behaviour, provide authority for decisions and establish the basis for both personal and organizational accountability.

Standardized assessments are not intended to replace professional or clinical discretion but rather to supplement it. By publicly defining its protocols for making assessments, the Correctional Service of Canada takes responsibility for its risk criteria, leaving the primary responsibility for competently applying those protocols to case management staff.⁷ This is not to suggest that staff members do not have a vested interest in how these instruments are designed, developed and implemented, as many of these tools derive directly from case management experience and practice. Rather, it is important for case management officers to apply the assessment tools competently, and to do so requires an understanding of their theoretical framework and development.

Recently, the Correctional Service of Canada completed a validation study of the Custody Rating Scale.⁸ The study used many of the traditional tests of psychometric properties⁹ that standardized classification tools undergo before they are implemented. The results of these tests are summarized here, and it is hoped this will reassure staff members and improve their understanding of how standardized tools contribute to professional judgment.

Custody Rating Scale

Offender security classification is grounded in the belief that measurable differences exist among offenders. It is also supported by the growing evidence that offenders can be grouped into distinct categories according to their ability to adjust in institutions, their escape risk and their risk to public safety should they escape.¹⁰ Modern classification systems are often formulated on a two-tiered model in which an initial security rating, based on static factors, is made at admission followed by regular reassessments based on behaviour during incarceration. Classification systems often include a provision allowing for a security rating to be overridden for factors not related to risk (such as cell accommodation, protection or health needs) and for the cutoff values for security ratings to be adjusted. This gives considerable control over how offenders are distributed across security levels, contributes to the management and control of offenders,¹¹ and can play a major role in placing offenders to the least restrictive levels of confinement.¹²

The Custody Rating Scale (CRS) consists of two, independently scored subscales — a five-item Institutional Adjustment subscale and a seven-item Security Risk subscale. In most cases, scores on each item increase according to the frequency of incidents and, as scores increase on either subscale, the predicted security classification also increases. Security classification is determined by combining the total scores, in accordance with predetermined protocols that specify cutoff values for minimum and maximum security. If the score on one subscale indicates a level of security that differs from the other subscale, the overall CRS outcome is determined by the subscale that assigns the higher classification rating.

Operational research

The CRS was developed and validated in 1987 based on a retrospective sample of 600 male federal offenders. It was approved for national implementation in 1990. Two previous examinations of the scale were undertaken, but they involved pilot samples from only two regions of the Correctional Service of Canada and predated the 1991 automated electronic version found on the Offender Management System.

The study described in this article was intended to establish the current reliability and validity of the scale, determine the impact of the Offender Management System and analyze initial placement practices. In March 1995, a sample was drawn from the Offender Management System of all active offender files that contained a complete and accurate CRS report. This sample of 6,745 cases represented 48% of the incarcerated population at that time.

Reliability

The CRS is applied in all five administrative regions of the Correctional Service of Canada. While each region has its unique classification traditions, local perspectives and accommodation options, it is important to ensure the scale is applied consistently and meets acceptable reliability standards.

In earlier studies where the CRS was scored by hand,¹³ errors related to omissions, out-of-range responses and computation problems were found in as many as 40% of the files sampled. Since the automation of the scale and its inclusion in the Offender Intake Assessment process, these types of errors have been eliminated, suggesting its more consistent administration.

Scale reliability was also explored in terms of the internal consistency among items as measured by coefficient alpha tests. Alpha measures the average correlation between scores on each item of a scale, and where the alpha is high, it is assumed the consistency between scores is also high. The overall coefficient alpha was .39 for the Institutional Adjustment subscale, and all intercorrelations between items, with one exception, were significant (p<.005). The overall coefficient alpha was .10 for the Security Risk subscale, and for only three of the seven items were the intercorrelations found to be significant. Policy decisions to inflate the weighting for certain items may explain the poorer internal consistency for the Security Risk subscale.

Finally, the effectiveness of the CRS in grouping offenders into security classification categories that are discrete, exclusive and comprehensive was explored. The sample was grouped according to the security level designation given by the CRS and the average (mean) scores for each of the 12 items on the scale were analyzed. The average scores of the maximum-, medium- and minimum-rated groups were found to be significantly different (p<.001) on all 12 items. This suggests that the CRS is quite capable of establishing an institutional-adjustment and security-risk continuum that effectively distinguishes between security classification groups.

Validity

It is of little value to develop an instrument that is reliable but does not measure the behaviour it was intended to measure or fails to classify offenders according to anticipated behaviour. Therefore, the concurrent and predictive validity of the CRS was tested.

Tests of concurrent validity measure the extent to which ratings from the CRS are in accordance with ratings from an alternate method of security classification. In this case, the actual penitentiary placement decisions were used as an alternate method of security classification. The extent and nature of the agreement can be illustrated with a concordance table which also provides a rich source of information about placement patterns.

The frequencies and percentages in the cells on the diagonal as marked in Table 1 represent those cases where the CRS designation and the penitentiary placement decision agree on the security classification. The figures in the cells to the right of the diagonal represent cases where the CRS designation was overridden and a placement decision was made to a higher level of security. The figures in the cells to the left of the diagonal represent cases where the CRS designation was overridden by a placement decision to a lower level of security.

The overall concordance rate, as represented by the sum of the diagonal, was 74%. (Based on previous reviews, when the effects of legitimate overrides to the scale, such as protection and medical considerations, are accounted for, the actual concordance rate may reach as high as 84%.) Most disagreements with the scale (16%) were in the form of overrides to higher security levels, while the remaining disagreements (10%) were overrides to lower security.

Table 1

Concordance Between the Custody Rating Scale and the Penitentiary Placement Decision
CustodyRating Scale Designation	Penitentiary Placement Decisions
Security	Minimum	Medium	Maximum	Total
Minimum	16.3% (1,078)	10.7% (707)	0.3% (21)	27.3% (1,806)
Medium	7.7% (508)	54.7% (3,629)	5.3% (349)	67.7% (4,486)
Maximum	0.1% (4)	2.1% (142)	2.9% (195)	5.1% (341)
Total	24.0% (1,590)	67.5% (4,478)	8.5% (545)

These results suggest a high level of agreement between the security designations given by the CRS and the actual penitentiary decisions made. A closer examination of override patterns is revealing. For example, of 1,806 offenders rated as minimum security by the CRS, almost 60% (1,078) were actually placed to minimum security; 707 were placed to medium security. Similarly, 508 cases placed to minimum security were actually overrides of medium security ratings by the CRS.

It is interesting to note that the overall base rates for institutional incident (16%) and escape (4%) of offenders rated by the CRS as minimum security risks was lower than the incident (18%) and escape (6%) rates for all offenders initially placed to minimum security. These higher rates result from medium rated offenders placed to minimum whose substantially higher incident (26%) and escape (8%) rates inflated the base rates of all minimum placed offenders. A similar effect was noted with respect to base rates for violence and drug and alcohol incidents. The results suggest that placement of higher risk offenders to the least restrictive level of confinement is not without costs.

Tests for predictive validity assess the extent to which initial classification ratings are confirmed by future institutional behaviour. A number of indices of predictive validity were examined using data gathered after the CRS had been completed and the penitentiary placement decision had been made. Table 2 provides the rates of overall institutional incidents, violent incidents and escapes from minimum security among offenders classified as minimum, medium and maximum security by the CRS.

Table 2

As expected, there are significant differences in the rates of misbehaviour across the various security ratings of the CRS. The overall incident rate and the violent incident rate for minimum rated offenders (15.6% and 3.1% respectively) are lower than those of offenders rated as medium security (35% and 8.1% respectively) and markedly lower than those of offenders rated as maximum security (51% and 14.3% respectively). Similarly, the escape rate of offenders designated by the CRS as minimum security is significantly lower than that of offenders designated higher security by the CRS but placed to minimum security (4.6% versus 7.7%). Similar results can be demonstrated for a variety of other predictive indices including likelihood of drug and alcohol offences, discretionary versus non-discretionary release and conditional release adjustment.

Overall, then, the CRS performed very well in terms of categorizing offenders according to their relative risk for escape, disruptive or violent behaviour and drug and alcohol involvement, as well as according to their potential for discretionary release and behaviour on conditional release.

Practical utility

Finally, the CRS’s usefulness in promoting the values and meeting the objectives of the Correctional Service of Canada was examined. One way was by looking at the effect of initial placement on release potential.

Effective classification should encourage the placement of offenders at the least restrictive level of confinement and, in so doing, maximize offenders’ potential for discretionary release (that is, release on full parole as opposed to statutory release). Where an offender is initially placed has an important bearing on if and how quickly the offender is released. Offenders placed at lower security institutions have better opportunities to establish their release credibility than offenders with similar classification ratings who are placed at higher security institutions. Table 3 shows the discretionary release rates and average number of days of incarceration before release for offenders rated and/or placed at minimum and medium security levels.

Eighty-five percent of the offenders rated (by the CRS) and placed to minimum security were awarded a discretionary release after an average of 379 days of incarceration. This compares with a 68% release rate and an average of 462 days of incarceration for offenders rated as minimum but initially placed to medium security. Medium-security rated offenders placed to minimum security, on the other hand, enjoyed higher release rates (78%) and shorter incarceration periods (423 days) than offenders rated and placed to medium security (63% and 529 days) or even the minimum-rated, medium-placed offenders. (Statistical Information on Recidivism Scale scores were examined, and nothing was found to suggest the risk to recidivate may have influenced the placement and release potential.)

Table 3

Discretionary Release Rates and Average Days to Release by Rating and Placement Decision
Custody Rating Scale Designation	Penitentiary Placement Decision
	Minimum		Medium
Minimum	Release Rate	85 %	Release Rate	68 %
	Days to Release	379 days	Days to Release	462 days
Medium	Release Rate	78 %	Release Rate	63 %
	Days to Release	423 days	Days to Release	529 days

While medium-security rated offenders placed to minimum security enjoyed higher release rates and shorter incarceration periods than offenders rated at lower security levels, they also had substantially higher rates of institutional incidents, escapes and conditional release suspensions. It is clear that initial placement to minimum security, regardless of risk, has a dramatic effect on release potential. It is also clear that there are costs associated with overriding the CRS ratings: placement to higher security impedes release potential, while placement to lower security is associated with higher rates of institutional and conditional release maladjustment.

Conclusion

The Custody Rating Scale performed well in assigning discrete security classification ratings to newly admitted offenders and also in terms of its concordance with actual placement decisions. The scale also proved effective in assigning ratings that correlated with institutional adjustment patterns, escape risk, discretionary release potential and conditional release adjustment. An analysis of overrides of the scale illustrated the impact of initial placement on release potential.

The CRS provides the Correctional Service of Canada with an effective and objective measure of security classification, is a valuable resource to management and guides case management staff consistently in initial placement decisions.

---

1. Research Associate, Correctional Service of Canada, RHQ Ontario, 440 King Street West, Kingston, Ontario K7L 4Y8.

2. Solicitor General of Canada, Development of a Security Classification Model for Canadian Federal Offenders: A Report to the Offender Management Division (Ottawa: Correctional Service of Canada, 1987).

3. L.L. Motiuk, Case Management Manual, Part Two: Offender Intake Assessment and Placement (Ottawa: Correctional Service of Canada, 1996).

4. J. Nuffield,Parole Decision-making in Canada: Research towards Decision Guidelines (Ottawa: Ministry of Supply and Services Canada, 1982).

5. R. Serin “Psychological Intake Assessment: Contributing to Contemporary Offender Classification,” this issue.

6. J. Bonta, “Risk/Needs Assessment and Treatment,” in A.T. Harland (ed.), Choosing Correctional Options That Work: Defining the Demand and Evaluating the Supply (Thousand Oakes, California: Sage, 1996).

7. J. Alexander, “Classification of Objectives and Practices,” Crime & Delinquency, 32 (1986): 323-338.

8. F.P. Luciani, L.L. Motiuk and M. Nafekh, An Operational Review of the Custody Rating Scale: Reliability, Validity and Practical Utility (Ottawa: Correctional Service of Canada, 1996).

9. E.I. Megargee, “The Need For A New Classification System,” Criminal Justice and Behavior, 4 (1977): 107-113. See also L.L. Motiuk, “Predictions and Classification in Corrections,” in T.A. Leis, L.L. Motiuk and J.R.P. Ogloff (eds.), Forensic Psychology: Policy and Practice in Corrections (Correctional Service of Canada, 1995).

10. D.A. Andrews, J. Bonta and R.D. Hoge, “Classification of Effective Rehabilitation: Rediscovering Psychology,” Criminal Justice and Behavior, 17 (1990): 19-52. See also C.B. Clements, “Offender Classification: Two Decades of Progress,” Criminal Justice and Behavior, 23 (1996): 121-143. See also D.M. Gottfredson and M.H. Tony, Prediction and Classification: Criminal Justice Decision Making (Chicago: University of Chicago Press, 1987).

11. Report of the Auditor General, Canada (Supply and Services: Canada, 1994).

12. R.L. Levinson, “Security Designation Systems. Preliminary Results,” Federal Probation, 44 (1980): 26-30.

13. F.J. Porporino, F.P. Luciani, L.L. Motiuk, M. Johnston and B. Mainwaring,Pilot Implementation of a Custody Rating Scale (Ottawa: Solicitor General of Canada, 1989). See also F.P. Luciani, L.L. Motiuk and B. Mainwaring, Field Tests of the Custody Rating Scale (Ottawa: Solicitor General of Canada, in press).