Performance Assurance

Warning This Web page has been archived on the Web.

Formative Evaluation of the National Contribution Program and the National Aboriginal Contribution Program

394-2-31

Correctional Service of Canada
November 2004

3. KEY FINDINGS

3.1 RATIONALE/RELEVANCE

Interviews with key informants and file and document reviews constituted the main lines of evidence for assessing the main questions in terms of rationale and relevance. This section addresses the following questions:

  • How do the programs reflect current priorities and objectives of the Government of Canada and the Correctional Service of Canada?
  • Do the programs continue to fill a gap or need in offender reintegration?

In assessing the relevance and rationale for both the NCP and the NACP, the team drew information from the work done on the needs of offenders' families by the Canadian Families and Corrections Network (CFCN) entitled A Strategic Approach and Policy Document to Address the Needs of Families of Offenders: Safety - Respect and Dignity - For All. The report emphasizes the significance of family-friendly policies and practices in guiding correctional staff so that the effects of incarceration on the family relationship is minimized or mitigated. As stated by Lloyd Withers (2003), family-friendly policy and practice treats families with respect and dignity and engages the family as an asset and a factor in successful conditional release and crime prevention, while balancing safety concerns for the family, the offender, institutional staff, and the public.

The report indicates that a family affected by incarceration and reintegration may be a family in crisis. There are families that may not know how to access services or that may be hesitant to access support services. Some families who do access services often find it difficult to continue to participate in programs or treatment and may withdraw prematurely. By the time family members have reached the correctional jurisdictions, they may have already experienced significant disruptions in their family life, and for a significant duration. If the family relationship survives the arrest, trial, sentencing and conviction of the offending family member, the family faces the maintenance of a family relationship in a setting that may further stress the survival of the family. Reports from key informants during this evaluation indicated that the NCP and the NACP serve as a vehicle to fill the void outlined in the CFCN report.

Additionally, the CFCN report states that the emotional load for families in terms of feelings of grief, loneliness, abandonment, isolation, anger, and fear can be overwhelming. Also, as one family member said during the CFCN consultation, "the family is left holding the bag in the community" for the criminal behaviour of the offender. The offender may be in prison, but the family may reside at the same address or the same community in which the offence took place. "For All" in the title of the report includes communities. For example, the CFCN public consultation in Hamilton similarly identified the consequences of incarceration and support for families, particularly with respect to the social benefits of support during reintegration and the consequences for communities if support is not offered.

Jeremy Travis, Amy Solomon and Michelle Waul, in From Prison to Home: The dimensions and consequences of prisoner re-entry show the scope of difficulties facing families. While Travis, Solomon and Waul speak to the situation in the US, their comments are applicable to the Canadian experience. Many of the quality of life issues, including the effects and differences of paternal and maternal incarceration that these authors identified, arose at the CFCN public consultations. For example:

Incarceration and re-entry have substantial impacts on a large and growing number of families - ranging from the loss of financial and emotional support to the social stigma attached to having a family member in prison. These complex relationships, combined with the great distance between many prisons and their home communities, require creative management on the part of the families, government agencies, and community support systems to minimize the harm to children and families.

Travis, Solomon and Waul continue by discussing the destabilization of communities that occurs because of incarceration and reintegration, ways to increase the resilience of families and the community, and the need to address family and offender issues related to criminal behaviour, substance abuse, employment, housing and counselling. The impact of incarceration on the family and children, in turn, has an impact on the community. Providing services to families and children is a form of crime prevention. There is a social benefit in providing services to families and children affected by incarceration and reintegration.

Our evaluation findings underscored the importance of resources and programs such as those provided by the NCP and the NACP to offenders and offenders' families and the community at large.

3.1.1 ALIGNMENT WITH OBJECTIVES

File reviews, document reviews and key informant interviews provide evidence that show that the NCP and the NACP contribute to two of CSC's four strategic outcomes:

  1. offenders who are safely and effectively reintegrated, and
  2. corporate management services support the care, custody and reintegration of offenders, and partnerships to promote the achievement of CSC's Mandate and Mission.

Of the internal key informants interviewed, 100% believe that the program reflects the priorities and objectives of the Government of Canada in general and of CSC in particular. In the words of one of the respondents:

Reintegration is paramount, public safety is paramount.

The stated activities of recipients appear to be consistent with the findings that were espoused in the cited CFCN report on the strategic approach and policy document in terms of addressing the needs of offenders' families. The key informants interviewed emphasized the need for a program that provides support to offenders and their families.

3.1.2 OFFENDER REINTEGRATION

A review of files revealed that the stated objectives of the funded recipients were in line with the programs' objectives of contributing to the successful reintegration of offenders. For example, the file review indicates that 180 offenders with disabilities were provided community support to find housing, develop life skills, access transportation and locate literacy and educational supports. When asked if the contribution programs filled a gap or need in the offender's reintegration, 20 of 23 respondents felt that there was evidence that supports the achievement of that objective. The remaining 3 respondents did not disagree with the statement but emphasized the need to "wait and see" the impact of the programs. One external key informant said it best:

".without it, a barrier for Aboriginal people would be erected in regards to community follow-up and programs."

Responses to questions of rationale and relevance on questionnaires (Appendix A) are shown in the following graphs:

rationale and relevance on questionnaires

A significant number of internal key informants - 92% - indicated that the rationale for the NCP and NACP are based on clearly defined problems and/or needs and the contributions serve to fill a gap in the reintegration process.

internal key informants

The graph above illustrates that 92% of internal key informants believe that the NCP and the NACP are funding the appropriate projects/organizations in light of the components and criteria of the terms and conditions of the NCP and the NACP.

external key informants

The chart above demonstrates that 78% of recipients (external key informants) interviewed indicated that the NCP and NACP are relevant and are based on clearly defined problems or needs and that the activities are contributing to filling a gap in offender reintegration.

3.1.3 CONCLUSION ON RATIONALE AND RELEVANCE

The programs reviewed align with two of CSC's strategic objectives:

  • offenders who are safely and effectively reintegrated;
  • corporate management services support the care, custody and reintegration of offenders, and partnerships to promote the achievement of CSC's Mandate and Mission.

High numbers of key informants and other stakeholders agreed that the program is relevant both in engaging community in CSC's business and in creating support for offenders and offenders' families, which in turn enhances CSC's reintegration practices.

3.2 DESIGN, DELIVERY AND MANAGEMENT

Design, delivery and management issues refer to how the programs structure and administer their policies and procedures. Most of the evidence for responding to the following questions comes from the literature review, program documentation, and key informants.

  • Do the programs have effective and clear procedures for applying for funds, criteria for determining eligibility, and other management procedures for each of the categories?
  • Are the recipients satisfied with the services and support offered by the programs?
3.2.1 PROGRAM PROCEDURES AND PROCESSES

A review of documents, key informant interviews and file reviews revealed that the NCP and the NACP rely on the terms and conditions of the programs for implementation. These are considered the guidelines against which contribution applications are selected and assessed. At face value, this is a commendable endeavour because it ensures the programs' implementation is aligned with the terms and conditions. However, the terms and conditions do not include guidelines on some detailed processes as they pertain to timeframes for review of application and decision-making.

A comparison of the level of funding initially earmarked for the contribution programs with the actual level of expense revealed that the latter was more than 6 times higher in 2001-2002, more than 4 times higher in 2002-2003 and about 3.7 times higher in 2003-2004. These figures clearly show a planning pattern that estimates less than the actual funding requirements of these programs.

3.2.2 APPLICATION AND FUNDING APPROVAL PROCESS

The elaborate program processes outlined earlier in this report are in keeping with the fundamental requirements of modern comptrollership and stewardship in the Public Service. They ensure that checks and balances are in place to manage public money effectively and efficiently. The evaluation revealed, however, that there is a challenge in balancing the need for good comptrollership and timely approval of application and the disbursement of approved funds. Key informants, particularly external groups, reported significant delays in the application and approval process. These key informants, both external (79%) and internal, expressed dissatisfaction with the process and the extremely long period of time it takes to know whether funding will be received. On average, it takes 5 months into the new fiscal year (September) before any funding is received. Internal key informants validated the fact that it took an exceptionally long time between approval by the NCPC and the transfer of funds to recipients. For some recipients, this meant that many of the initial program activities identified could not be achieved in the timelines provided. In terms of the reasons for the delay, some of the respondents identified the lack of a national guideline that establishes a timeframe for responding to each phase of the application and approval process as a possible reason.

3.2.3 PERFORMANCE MEASUREMENT (RMAF)

In July 2004, an RMAF was developed. Until the development of this document, there were limited structured performance management tools in place. The RMAF represents a commitment by program management to the collection of performance data. This information is intended to be used to make improvements, prepare annual reports and guide future evaluations. The findings of this formative evaluation will provide program managers with the necessary information to improve the current RMAF in order to adequately respond to any risk and to reflect the complexity of the programs.

Responses to questions of design and delivery on questionnaires (Appendix A) are shown in the following graphs:

design and delivery of the programs

The above chart Q.1 depicts the responses to the question concerning whether the current design of the programs supports the achievement of outcomes. 85% of internal key informants indicated that the design of the programs indeed facilitated the achievement of its objectives. In looking at the graph above, one would ask why the number of key informants that did not respond to question 2.B appears significant (64%). These respondents did not provide any answer to the question because they stated that their responses would be in line with the answer to Q.1, which assessed whether the design and delivery of the programs facilitated the achievements of its objectives. The interpretation of this finding is that the 64% of the respondents also agreed with the premise of Q.2.B.

design and delivery of the programs

The chart above shows that 78% of external key informants responding to Q.1 and Q.2.B agreed that the current design of the programs facilitates the achievement of their objectives and simultaneously agreed that the objectives and desired results of the programs were clearly identified. However, the responses equally identified certain areas of the programs' delivery that require immediate attention. Such areas have been outlined in the conclusion section.

3.2.4 CONCLUSIONS ON DESIGN AND DELIVERY

One way of increasing the effectiveness and timeliness of program delivery is to collect data on performance. By monitoring key performance measures that reflect what is of greatest concern to the recipients of a program, program managers can link this understanding with the types of strategies and actions that best improve this performance. In addition, by providing meaningful program performance information at the national level, program management and operations could become an important element of the program for improving the application and approval process. Inherent in the use of performance measures is the importance of a "customer" perspective in planning and decision-making. This, in turn, necessarily entails knowing what the customer wants from the program design and delivery.

Key informants expressed a high degree of satisfaction with the program design as it relates to the desired outcomes. It is apparent from interviews, however, that the delivery approaches need to be strengthened to allow for a more effective and timely approval of contribution applications and the subsequent disbursement of approved funds based on the recipient's cash flow requirements.

The terms and conditions clearly outline the application and funding approval process. Nonetheless, they do not include a specific timeframe for responding to each of those phases. This formative evaluation will provide program managers with the necessary information to increase understanding of the program process among the program stakeholders and to build support for ongoing measurement and future evaluation.

As stated earlier, program delivery also addresses the efforts by the program managers to extend the reach of the programs so as to ensure maximum value for money. It was found that those organizations that have established relationships with CSC on contribution agreements essentially maintain their relationship. This is seen as a positive measure and program managers are encouraged to build on this strategy as an avenue for reaching out beyond the "prescribed" groups when call letters are sent out yearly.

Concerning the allocation of funds by CSC, the evaluation team found that the Department should better plan the contribution programs' requirements in order to reduce situations where funds from other envelopes have to be used.

3.3 SUCCESS/IMPACT

The NCP and the NACP successes were measured in light of four questions:

  • To what extent are the programs achieving their expected results?
  • Is there evidence that recipients would have undertaken the activity or initiative even without program support or funding?
  • What difference did the funds make to their activities?
  • Do recipients access funding from other government programs as well?

There was generally a strong sense that the NCP and the NACP contribute to enhancing opportunities for facilitating different programs that would otherwise not have not been implemented. Respondents were particularly pleased with the programs and believed that the programs have been successful in achieving expected outcomes.

According to most respondents, one of the key measures of success is the "reach" of the programs, the ability to provide support to offenders and community members. In general, respondents are positive about their ability to secure contribution agreements in their respective fields. This finding also underscores one of the recommendations (# 8.2) of the CFCN report that CSC, through its Community Engagement sector (now Communications and Community Engagement sector), includes the needs of families affected by criminal behaviour and by incarceration and reintegration in its community engagement and public education activities. Respondents agreed that the NCP and the NACP have had significant positive impacts on the life of some offenders and their families.

One common observation among some recipients, however, was that it would be more beneficial if the programs could provide additional administrative support such as personnel and other expertise. Given the small administrative framework and resources available for these programs, such expectations may be unreasonable.

In the course of the file review, several performance indicators evidently demonstrated that the NCP and the NACP are assisting CSC in creating awareness and knowledge about its services and that its ability to engage the community in its affairs is enhanced. The following indicators demonstrate the successes of the programs at both the "system" and the "organizational" levels.

  • Increased awareness of section 81 and 84 among Aboriginal peoples;
  • United Nations accreditation of one of the recipients receiving funding under the NACP;
  • Identification of issues facing Aboriginal communities in supporting the reintegration of offenders;
  • Creation of an accessible restorative justice database at a major Canadian university;
  • Creation of a certificate program in restorative justice at a major Canadian university;
  • International conference on restorative justice;
  • Community engagement enhances our strategies of building on what the public tell us about CSC's policy framework;
  • Importance of funding for organizations as a way of supporting CSC in achieving its mandates.

The majority of key informants agreed that public sector funding was critical to their continuation of activities in support of CSC's mission. Evidence indicates that recipients from larger organizations believe that a reduction of funding or no funding would result in the discontinuation of activities related to the reintegration agenda, but would not constitute a threat to their overall operations and activities in other similar areas.

Several respondents agreed that, since family and community factors are important to conditional release success, it would make sense to continue to allocate resources to programs and services for families and communities.

The evaluation also established that it is important for the NCP and the NACP to continue to enhance their performance measurement data collection strategies to ensure that those agencies that receive funding continue to be more accountable for the services that they deliver and establish a strategy to assess their programs' effects on the identified factors for successful reintegration. Responses to questions regarding success on the questionnaires (Appendix A) are shown in the following graphs:

success

The chart above shows that 79% of internal respondents believe that the NCP/NACP have been successful in achieving their expected results.

success

When the external key informants (Q.1.A) were asked about the need for and importance of federal funding for the implementation of their programs, 60% said that they would not have been able to undertake the activities or initiatives that were funded under the contribution programs without such funding.

Similarly, 60% indicated that they have accessed other government funding to implement other activities that were parallel to those covered under the contribution programs. In spite of the number of respondents that indicated they have access to other government funding, they underscored the importance of CSC funding for their programs and activities.

3.3.1 CONCLUSIONS ON SUCCESS/IMPACT

Although most recipients have a broad range of expected outcomes, support for offender reintegration remains common to all. A key challenge for the programs is to show the link between recipient activities and increased successful reintegration. Since linking intervention to change in human behaviour is similar to hitting a moving target, the linkages between program activities and successful reintegration will continue to pose a challenge. However, it is abundantly clear from file reviews and interviews with key informants that the disappearance of these programs and services would constitute an unbearable void in CSC's reintegration efforts.

3.4 COST-EFFECTIVENESS

A true cost-effectiveness analysis will not be possible until the programs produce data on outcomes that can be aggregated across funded contributions. The difficulty in conducting a cost-benefit analysis of programs of this nature is the challenge of assigning a monetary value to all of the costs and benefits of alternatives. This is not always plausible in a systematic and rigorous manner and it does not effectively speak to program outcomes. Since the evaluation has not assessed any alternatives for accomplishing the target goals of the NCP and the NACP, a common measure of effectiveness has been used to assess these programs.

Cost-effectiveness can also refer more generally to a range of organizational issues, for the program as well as for applicants and funded recipients. The questions for this section are as follows:

  • Is the current program design the most effective and efficient way to achieve outcomes?
  • Are the resources that have been allocated being used in the most efficient and effective way to deliver appropriate results?
  • Is it necessary for the Correctional Service of Canada/federal government to operate this program - could it be transferred in whole, or in part, to other levels of government, NGOs or to the voluntary sector?

Measuring the cost-effectiveness of the contribution programs entailed assessing the programs' effectiveness for participants, sponsors and the general public, which the Correctional Service of Canada represents. Some internal key informants indicated that contributions are not always an appropriate vehicle for furthering broad federal policy objectives, and CSC may want to consider delivering services nationally where services have been deemed essential to the offender reintegration strategy. This is not to minimize the significance of these programs. Any such national service will only serve to complement the services provided by the two contribution programs.

In the course of the file review and document review for the NACP, evidence indicated that there was some duplication in the activities of the ACDO and activities funded in the area of capacity building in Aboriginal communities. An earlier evaluation of the National Aboriginal Working Group (NAWG) had similar findings. There is a need to enhance the collaboration between the ACDO and recipient organizations that are engaged in capacity building within Aboriginal communities. The ACDO should by all accounts be the first line of contact and an invaluable bridge between CSC and those organizations.

3.4.1 PUBLIC SAFETY AND CSC

Research indicates that most Canadians overestimate the amount and severity of crime and underestimate the severity of penalties. A single incident can undermine a community's sense of security and, in turn, public confidence in Canada's approach to criminal justice. Traditionally, Canadians have always seen and continue to see the need for a continued role for CSC and the federal government in addressing crime and sanctions in Canada. Therefore, the NCP and the NACP provide a vehicle whereby the community can be active in helping to support the successful reintegration of offenders and educate communities on the reintegration process.

Responses to questions of cost-effectiveness and alternatives on questionnaires (Appendix A) are shown in the following graphs:

cost-effectiveness and alternatives

The above graph indicates that 71% of internal respondents believe that the NCP/NACP programs are cost-effective in achieving policy objectives and expected results.

cost-effectiveness and alternatives

The above graph shows that 67% of external respondents agree that the programs are cost-effective. Recognizing that 67% agree that the programs are cost-effective, the overwhelming majority cited the need to enhance and streamline the funding approval process.

3.4.2 CONCLUSIONS ON COST-EFFECTIVENESS

As is evident from the graphs above, the key informants and the program managers interviewed believe that the programs are effective in meeting their desired objectives. As was suggested earlier in the report (under Design and Delivery), program managers may use necessary performance measurement information from the evaluation to increase understanding of program processes among the program stakeholders and build support for ongoing measurement and future evaluation. The establishment of a national guideline that accentuates the approval process with specific timeframes for responding to each phase of application and approval may assist in enhancing communication between the program managers and those agencies that rely on the NCP and the NACP to provide services to their clients.

 

TABLE of CONTENTSNEXT