Research Reports

Measuring Performance Towards Corporate Objective #1

Prepared by:

Frank J. Porporino
and
David Robinson

Research and Statistics Branch
Communications and Corporate Development

February 1991

This document attempts to develop a framework for arriving at a common understanding, and an agreed upon set of procedures, for ongoing monitoring and reporting of performance related to Corporate Objective #1.

More specifically, the paper has three aims:

1. To propose, and to explain the underlying rationale for, a set of summary "performance indicators" relevant to Corporate Objective #1.
2. To affirm the value of our corporate automated data sources, and to outline a broad-based quality control process to improve on the timeliness and integrity of data capture.
3. To suggest an approach wherein the Regions and the NHQ Programs and Operations, Corporate Management, and Communications and Corporate Development Sectors can make their respective and appropriate contributions to the reporting and analysis of our Corporate Performance.

CONTEXT: SOME BASIC ASSUMPTIONS

There are a number of essential qualities that should characterize corporate performance indicators as commonly understood in the management and organizational effectiveness literature.

Performance indicators should be:

• valid, reliable and sensitive measures of "outcomes that would be anticipated" if an objective is being achieved;
• "normative" and therefore able to be interpreted against a clear benchmark or reference point (e.g., a point in time);
• "integrative" and therefore relevant for executive level, higher-order decision-making (i.e., providing simple and easily interpretable quantification that permits comparative assessment of performance).

The need for performance indicators of organizational objectives to be normative and integrative is self-evident. The indicators should be conceptualized in a fashion that can lead to decision-making "action", and not to an ever-ending search for more data to explain trends. They should measure key and important facets of performance, clearly and unambiguously, at the appropriate level of abstraction for executive decision-making.

Performance indicators are valid to the degree that they actually measure, in a relevant and meaningful way, the outcome(s) implied by an objective. The indicators should measure those central and important outcomes that directly reflect the achievement of an objective ( and not outcomes that may be influenced by extraneous or uncontrollable factors ). Since the complete assessment of a complex objective will often demand analysis of sub-components, all facets of a multicomponent objective should be measured if performance measurement is to b considered valid.

Reliability concerns the extent to which measurements are repeatable. It is an index of our level of confidence that there is true variation, from one occasion to another, not caused by errors of measurement. Within measurement theory, there is a precise mathematical relationship between the reliability and validity of a measure ( i.e., the upper limit of the validity of a measure is constrained by the square root of the reliability ). To the extent that there is error of measurement, therefore, the validity of a statistical indicator is limited.

Depending on the types of outcomes we are attempting to measure, a valid and reliable indicator may still be inherently insensitive to small but significant differences in performance. For example, important process factors may be changing but there is a lag before outcome is affected. The sensitivity of a statistical performance indicator can only be determined over time as it is compared and contrasted with other "performance information" ( i.e., more qualitative, narrative, case-study information collected through a range of appropriate methods to supplement and enrich the interpretation of statistical indices).

DERIVING SUMMARY PERFORMANCE INDICATORS FOR CORPORATE OBJECTIVE # 1

A clearly stated objective is a necessary first step in deriving valid, reliable, sensitive, and easily interpretable "summary performance indicators". Corporate Objective #1 is a clear and unambiguous statement of correctional purpose. We wish to be held accountable to the degree that we :

ENHANCE THE SERVICE'S CONTRIBUTION TO THE PROTECTION OF SOCIETY BY SAFELY REINTEGRATING A SIGNIFICANTLY LARGE NUMBER OF OFFENDERS AS LAW-ABIDING CITIZENS WHILE REDUCING THE RELATIVE USE OF INCARCERATION AS A MAJOR CORRECTIONAL INTERVENTION

One possible end-result of pursuing this objective is that we hope to change the distribution of our population, achieving relatively fewer numbers of offenders incarcerated in institutions, and proportionately more under supervision. This hinges, quite obviously, on not experiencing unexpected changes in sentencing patterns or any significant and unanticipated influx of admissions.

More to the point, however, is the fact that this proportional shift is a longer-term result that we expect, quite likely somewhat unstable over the short-term, and perhaps more difficult to affect under some "structural" circumstances that characterize particular regions (e.g., a region may have a larger pool of offenders "uneligible" for release). Moreover, the sub-components of "safe reintegration" while concomitantly "reducing the relative use of incarceration as a major correctional intervention" can be conceptualized as measurable in a much more direct fashion that can be meaningfully tracked from month to month or quarter to quarter.

A simple example will illustrate the need to attend to both "temporal" and "frequency or count" dimensions in order to avoid a false impression of real performance changes over time, and furthermore, a false impression of regional variation.

• We begin with two regions, each with 300 offenders serving 6 years.
• The two regions release this cohort of cases as shown in Figure 1.

We note that after three years, region A has released 40 cases on full parole and region B 50 cases. Assuming that both regions had an equal number of cases under community supervision throughout the period, say another 100 each, then at the three year mark, the release patterns in the Figure would be translated as follows in terms of proportional shifts towards the community.

      FOR REGION A from 33% (100/300) to 54% (140/260)

      FOR REGION B from 33% (100/300) to 60% (150/250)

Looking only at proportional shifts, Region B would come out the winner. However, if we calculate "incarceration years" saved by the two Regions, then we get:

     FOR REGION A: 30 * 1 1/2 yrs + 10 * 1 yr = 55 years

     FOR REGION B: 50 * 1 yr = 50 years

Region A is now the better performer!

CONCEPTUAL MODEL: SUMMARY INDICATORS FOR CORPORATE OBJECTIVE #1

There are at least three significant aspects of Corporate Objective #1 that can be distinguished as performance dimensions we should monitor. Within each dimension, there are a number of potentially meaningful "summary indicators" that tap some aspect of our correctional performance.

DIFFERENTIATION:

A first and fundamental dimension is one of DIFFERENTIATION.... we can reduce the relative use of incarceration by selecting or differentiating a larger number of offenders for discretionary release "at the earliest possible time" (i.e., for release on full parole as close as possible to one-third eligibility).

The principle indicators we can subsume under a differentiation dimension are:

1. *quarter by quarter trends in the ratio of full parole to mandatory supervision releases (the ratio should show some gradual increase);

Over time, this should be reflected in a proportional shift towards more full parole, and fewer mandatory supervision cases under supervision. Therefore, it also becomes meaningful to monitor:

2. *quarter by quarter trends in the ratio of full parole versus mandatory supervision cases under supervision.¹

Differentiation also relates to the issue of how many offenders are prepared for release at, (or very close to), their parole eligibility dates. A useful indicator to monitor in this respect is:

3. *quarter by quarter trends in the proportion of offenders released within 30 days of parole eligibility (the proportion should be gradually increasing).²

---

¹Ideally, this should include only offenders with determinate sentences since life sentence and other indeterminate sentence offenders can only be under supervision on parole. In making regional comparisons, a disproportionate accumulation of indeterminate offenders in a given region would bias a ratio calculated on the basis of all FP to MS cases (particularly since indeterminate offenders can remain on parole reduced status for many years).

²Since offenders can be released on full parole for a second time on the same sentence (particularly longer-term offenders who may be returned on a technical violation and who may have long remnants of a sentence to serve), calculating this indicator for all full parole releases would introduce a systematic bias suggesting poorer performance (i.e., second time releases are by definition well past their original parole eligibility dates). Therefore, a proper conceptualization of this indicator is to examine only original Warrant of Committal admissions. Some regions may show superior performance on this indicator because of the phenomenon of "parole board dual decisions" (i.e., day parole followed by full parole, in principle, at one-third). In regions where dual decisions are not the norm, more cases may be kept on day parole well past their full parole eligibility.

TIMELINESS OF RELEASE:

A second key dimension of correctional performance that relates to Corporate Objective #1 is TIMELINESS OF RELEASE .... we can reduce the relative use of incarceration by shortening the amount of time served prior to release on full parole. A simple way to conceptualize this is to consider that all offenders have a portion of "eligible supervision time", which they can serve either incarcerated or under supervision (i.e., the two-thirds of the sentence after parole eligibility).

A meaningful indicator to monitor, therefore, is:

4. *for full parole releases, quarter by quarter changes in the average percent of eligible time served prior to release (this should be decreasing to a possible lower limit of 0 %).³

A more concrete, and perhaps more immediately relevant variation on this indicator would track the actual number of eligible days served prior to release ... thereby illustrating the actual number of days of incarceration that have had to be resourced for our offender population because of delays in time to release. Multiplying the average number of days, by the number of offenders it applies to and the average cost of accommodation per day, could give us an overall index of the incarceration costs incurred or saved over time. The indicator can be expressed simply as:

5. *average number of eligible days served prior to release on fun parole.

In any quarter, there will be a proportion of offenders who are eligible for release but who remain incarcerated. This population, in a sense, is the population for which we have lost the opportunity for early release. Over time, we should attempt to reduce, or at least keep constant, the average amount of eligible time served by this population of non-released offenders.

6. *for non-released offenders, the percent of eligible time served incarcerated.

There may be significant variation in the timeliness of release for different categories of offenders (e.g., short versus longer term offenders; natives versus non-natives; violent versus non-violent). Since this variation could focus our efforts on particular offender groups, it becomes important to monitor.

---

³The qualifier that only Warrant of Committal cases should be considered applied to each indicator relating to timeliness of release.

7. *quarter by quarter trends in the average percent of eligible time served, prior to release on full parole, for different categories of offenders.

REINTEGRATION:

The most significant challenge posed by Corporate Objective #1 is the challenge of reintegrating offenders safely in a way that contributes to the protection of society. An increase in rate of readmissions would suggest that we are neither meeting the challenge of safe reintegration nor the goal of reducing the relative use of incarceration.

This REINTEGRATION dimensionis the most difficult to conceptualize from the perspective of summary performance indicators since any single indicator will be inadequate in illustrating the quality and quantity of our reintegration performance.

For example, cases may be revoked and readmitted on technical violations because they are deemed at "high risk" to reoffend in a serious manner. From one perspective, a readmission is a failure, but from another, it is an effective way to ensure that we protect society.

Of more critical importance, however, is the fact that our reintegration performance does not lend itself easily to segmented, quarter by quarter analysis .... it is something we work towardsthroughout the sentence, from admission until Warrant Expiry. From this orientation, the only truly appropriate way to assess our "level of success" in reintegration is to follow a given group or cohort of offenders, from admission to Warrant of Expiry (and even beyond!). For the entire cohort, we would then examine such indices as:

• the total volume and proportion of time served incarcerated versus under supervision;
• the total volume and rate of offences committed, up to and for periods beyond Warrant Expiry;
• of the total volume of offences committed, the proportion that were violent or caused serious harm;
• the total number of individuals who were victimized;
• the total correctional cost in terms of volume of reincarceration days for those recidivists in this cohort of offenders.

The difficulty with this approach is the time lag that is required, by definition, to assess overall performance .... we would have to wait three to five years to assess our performance for any current cohort of admissions, and any cohort from the past would be more relevant in assessing our past rather than our current performance.

With these cautionary notes in mind respecting the complexity of theREINTEGRATION dimension, the following indicators are proposed as adequate for tracking the effectiveness of the community supervision phase of our correctional process.

8. *for the total supervision population at the beginning of each quarter, and for cases under full parole and mandatory supervision separately:
   • the percent readmitted that quarter, and
   • the percent readmitted for new offences or technical violations.

In order to obtain some index of trends in the severity of reoffending, we would track:

9. *the ratio of number of offenders returned for Category 1 offences versus other offences or technical violations.

Since we may be interested in the relative success rates under community supervision for different categories of offenders, we would examine:

10. *the proportions, for different categories of offenders under supervision, who are readmitted within a quarter.

Finally, for those cases who do recidivate, it becomes important to determine success in delaying the time period before readmission. If we take the portion of the sentence from release to Warrant Expiry and consider this as the "potential time under supervision", then a useful indicator is:

11. *the proportion of potential supervision time that is served successfully under supervision before readmission.

Appendix A details this entire set of summary performance indicators, nationally and regionally where appropriate, for the first three quarters of 1990-91. For a number of the indicators, a "rolling" national average over the first three quarters is illustrated, as a benchmark against which to make comparisons.

In instances where extreme outlying cases could distort the calculation of an average, the median rather than the mean is used in calculations so as to provide a more sensitive indicator.

IMPLICATIONS FOR DATA COLLECTION

To arrive at valid, reliable and sensitive indicators of performance, the process for gathering and reporting on performance indicator data has to satisfy a number of requirements.

• THERE MUST BE CORPORATE CONSENSUS AS TO THE DEFINITION OF THE DATA ELEMENTS TO BE USED IN CALCULATING OR DERIVING VARIOUS INDICATORS

Given the level of precision required in the specification of performance indicators, there must be uniformity in the use of nomenclature. A simple example of the difficulties that can be encountered is the confusion that can result when discussing even such seemingly straightforward data counts as "admissions" and "releases" .

At the institutional level, an "admission" might quite properly be regarded as any offender who enters the institution and who will be occupying an available cell or bed ..... regardless of whether that offender is on "temporary detention", is being accommodated in transit to another institution, is an escapee who has been returned and is already "on the register" of that institution ... etc. Because of the complexity in how we process offenders, aggregating admissions that might be recorded over a period of time at the institutional level will, by definition, result in significant over-counting at the regional or national level.

Clearly, for the purposes of calculating some indicator of "the volume of new cases we will be responsible for", we should track only "new admissions under sentence", either on warrant of committal or revocation.

The point to be made is that our "correctional terminologies" can have different meanings for different purposes, none of which are right or wrong, only appropriate or inappropriate depending on the measurement question to be addressed.

With reference to definition of key data elements related to Corporate Objective #1, agreement has now been reached within the CSC, among Regional Administrators Planning, Regional Administrators Operations and the CPO, CM and CCD Sectors at NHQ (see attached minutes of meeting held January 28, 1991). This should now establish a common frame of reference from which to work.

A related and equally important issue pertains to the sources of data for computation of performance indicators.

• THERE MUST BE AGREEMENT TO USE, AND COMMITMENT TO MAINTAIN, INFORMATION SYSTEM THAT WILL SERVE AS THE PRIMARY SOURCE FOR PARTICULAR DATA ELEMENTS

Our Sentence Administration System (SAS) and our Parole Supervision System (PSS), two major components of our Offender Information System (OIS), are corporate systems that are poorly understood both in terms of their capabilities and limitations.

The Service's Offender Information System, a conglomerate of fourteen distinct databases with specific types of information on all individual offenders incarcerated Federally since 1970, is a "live" interrelated automated system that is regularly updated e.g., as offenders enter or leave the jurisdiction of the Correctional Service of Canada. Since it is a "live" system with records of information constantly changing, and since information on the same offenders but in different database components may be updated by field staff at different times, inconsistencies are not uncommon (i.e., the same retrieval of specified data on separate occasions, or a slight variation to a question posed on the same occasion, may result in different answers). These problems inherent in a "live" operational system like the OIS are compounded by three other factors:

• data entry inaccuracies (e.g., an offender who is readmitted on a revocation with a new offence is not given a new "term identifier");
• data entry omissions (e.g., parole eligibility dates that may not be available at the time of penitentiary placement are not subsequently entered when the NPB officially informs the institution); and
• data entry delays (e.g., admissions and releases are backlogged and entered by field staff in batches).

Although arguably there has been documented evidence of problems in retrieving data from the OIS, the conclusion that it should be abandoned as a source of performance data because it is "too inaccurate" is unfounded.

As operational, offender-based, automated data storage systems, the SAS and PSS systems are quite adequate. The major limitation of these systems lies in the quality and timeliness of data input, a problem which can be corrected with a more vigilant and attentive approach to data-entry in the field. Moreover, there is no possible "self-reporting" alternative that could be implemented that would give us the level of detail required to meaningfully track and analyze performance related to Corporate Objective #1 (e.g., to make necessary distinctions at the individual offender level). To be at all useful, any "self-reporting" mechanism for operational units would have to duplicate much of what is already recorded on our O.I.S.

Until the full implementation of OMS, it has been agreed that our SAS and PSS will serve as the primary sources of data for measuring our achievement towards Corporate Objective #1.

The following standards have been agreed to by the Regions with reference to timeliness of data-entry:

• entry of all admissions and release data into SAS will be completed within five working days of month end;
• all updates to the PSS, including now releases, will be entered, as per citing standards, within two days of a change in offender status.

REPORTING ON CORPORATE OBJECTIVE #1

With agreement as to definition of terms and consensus as to the sources for basic data, a sensible process for reporting on Corporate Performance should be layered and interdependent:

• BEGINNING WITH A SET OF CLEAR. UNAMBIGUOUS, SUMMARY OR GLOBAL INDICATORS THAT CAN QUICKLY ALERT EXECUTIVE COMMITTEE TO:
  • HISTORICAL TRENDS AND SIGNIFICANT CHANGES IN PERFORMANCE THAT ARE DIRECTLY RELEVANT TO CORPORATE OBJECTIVE #1
  • VARIATIONS IN PERFORMANCE (E.G., AT THE REGIONAL LEVEL OR WITH PARTICULAR TARGET GROUPS OF OFFENDERS)

On certain occasions, there may be a need to proceed to a second phase of reporting on "PERFORMANCE INFORMATION"..... more specific and descriptive analysis of operational data that can EXPLAIN OR ACCOUNT FOR CHANCES AND VARIATIONS IN OVERALL PERFORMANCE.

For each step in the process, the Regions and Sectors at NHQ should have CLEARLY DELINEATED ROLES AND ACCOUNTABILITIES FOR EXPLAINING particular trends or differences.

For example, the Communications and Corporate Development Sector could report objectively on trends in SUMMARY PERFORMANCE INDICATORS.

The Corporate Management Sector should be explaining relativities in resource utilization (e.g., the "costs" of our performance in person year, overtime, or programming dollars).

Similarly, the Programs and Operations Sector should be examining differences in the case management process (e.g., % increase or decrease in number and types of cases presented to NPB), introduction of new programs and fluctuations in program participation rates or length of program waiting lists, in CCC or CRC bed utilization rates, community case-load sizes, suspension patterns and reasons, patterns of use of day-parole .... etc.

Regions, in turn, should be explaining particular and significant operational bottlenecks, outlining the strategies that have been put in place to unblock the process.

The process for reporting on Corporate Performance should be predictable and engaged in cooperatively. All sectors and the Regions should have as early access as possible to the same basic data to prepare appropriate analysis before discussion at Executive Committee.

With monthly updates of our Offender Population Profile System, and quarterly production of our Correctional Statistics Sourcebook, the necessary mechanisms will be in place to establish uniformity, consistency and rigour in our reporting on Corporate Objective #1.

APPENDIX A
SUMMARY OF PERFORMANCE INDICATORS

DIFFERENTIATION

APPENDIX B

RAPA(I) MEETING
JANUARY 28, 1991

Participants:
Regional Administrators, Planning, Administration (& Informatics) - all regions
Regional Administrators, Correctional Operations, all regions except Pacific
Director General, Correctional Policy, Program, & Interjurisdictional Affairs
Director General, Research & Statistics
Director, Program Planning
Director, Operational Planning & Resource Analysis
Senior Project Officer(s), Program Planning
Project Officers, Research & Statistics Branch

Subject: MONITORING & ASSESSMENT OF CORPORATE OBJECTIVE #1

Premise

The existing Automated Corporate Systems (example: Sentence Administration System (SAS) and Parole Supervision System (PSS) would be used to monitor and assess performance against Corporate Objective #1 (on-register populations). This data will be provided by the Research and Statistics Branch, CCD Sector.

Definitions and clarifications were required for the following performance indicators:

1. Incarcerated population and Supervised Populations
2. Admissions and Releases
3. Day Parole
4. Start Point/Targets

1. Incarcerated and supervised populations

1. Incarcerated Population

Definition: On-register population minus day parole population of any kind.

N.B. PSS system to be enhanced to identify the number of Day Parolees inside the institution.

Notes:

1. Federal cases only will be considered
2. The on-register population captures ESAs
3. Male and female offender populations will be captured separately
4. The on-register population excludes Temporary Detainees

Source Systems: Sentence Administration System (SAS) & Parole Supervision System (PSS)

Supervised Population

This population refers to offenders who have received grant of release by the National Parole Board.

Definition: This population includes day parolees, full parolees and offenders under mandatory supervision (includes temporary detainees and suspensions). Each category will be presented separately.

Notes:

1. Only federal offenders will be considered
2. Includes offenders under parole reduced
3. Clemency cases are excluded
4. The population captures Alberta federal cases
5. Clarification of issue is required regarding British Columbia Provincial Parole Board granting release to federal offenders. This population must be captured to reflect performance in this area.

Source System: Parole Supervision System

2. Additional data elements

In conjunction with the number of offenders under Day Parole, the number of admissions and releases will be used to explain changes to the level of incarcerated and supervised populations.

Admissions and Releases

Admissions

Definition: Offenders who are admitted under:

• warrant of committal
• revocation of full parole and revocation of mandatory supervision
• Lieutenant Governor warrants

Note: Revocation cases of Full Parole and Mandatory Supervision are counted as new admissions. Revocation cases of Day Parole are not (nor returns of Unlawfully at Large cases).

Releases

Definition: Offenders who are released under full parole mandatory supervision

Note:    Releases under day parole will be captured differently

Source System:     SAS, for both Admissions and Releases

Population under Day Parole:

This population will be captured by presenting a snapshot of the number of offenders under day parole at the end of the month.

Source System: PSS
PSS to be enhanced to capture offenders under day parole but who return to the institution on a daily basis.

3. Start Point & Target

The start point to assess performance is April 1, 1990.

For Long Range Accommodation Planning purposes, the level of the incarcerated population will refer to inmates in federal and provincial (ESAs) institutions as well as offenders under day parole who return to institutions on a daily basis. (Ref. August 31 ^st, 1990 planning worksheets).

Changes in populations will be measured by comparing the incarcerated and supervised populations against the forecasted offender population distribution, as indicated in the Long Range Accommodation Planning Worksheets.

For the purpose of comparison, the on-register population forecast (prepared by the Research & Statistics Branch in the Spring of 1991, without the impact of Corporate Objective #1); the anticipated on-register population (taking into account the regional commitments with respect to Corporate Objective #1); and the on-register population reflecting the current CSC situation (figures produced by the Research & Statistics Branch based on agreed-upon source systems) will be presented in a single graph.

Note: The Ontario, Atlantic and Pacific regions agreed that the distribution of the population over the 10 year planning period was accurate and reflected the regional commitment to Corporate Objective #1.

The Quebec and Prairie regions to clarify the offender population distribution with the Research and Statistics Branch.

4. Data Entry and Information Transmission

All regions will strive to ensure data accuracy by adhering to data entry procedures. The numbers presented at this meeting by the Research & Statistics Branch will be validated by the regional representatives and irregularities will be addressed.

Data entry: PSS data to be entered, as per existing procedures and standards, within 2 days of a change in offender status (including new releases).

SAS data entry should be completed within 5 working days of month end.

Data retrieval/transmissions to regions

The SAS data will be retrieved by the Research and Statistics Branch on the 8th day of each month, presenting data on the previous month. The data pulled on the 8th will be presented at EXCOM.

The Research and Statistics Branch will transmit electronically to DCCPO and CM sectors and regions (RA Planning and Informatics - E-Mail accounts) to enable regions to analyze the data and prepare for any discussions to be held at EXCOM (explanation of discrepancies).