Performance Assurance

Warning This Web page has been archived on the Web.

Formative Evaluation of the National Contribution Program and the National Aboriginal Contribution Program

PDF

394-2-31

Correctional Service of Canada
November 2004

TABLE OF CONTENTS


EXECUTIVE SUMMARY

1. INTRODUCTION AND CONTEXT

2. EVALUATION DESIGN AND METHODOLOGY

3. KEY FINDINGS

4. CONCLUSIONS

5. RECOMMENDATIONS

APPENDICIES
APPENDIX A: Instruments
APPENDIX B: List of Key Informants
APPENDIX C: Files Reviewed
APPENDIX D: NCP and NACP Contributions by Year
SIGNATURES

EXECUTIVE SUMMARY

Transfer payment programs, including contributions, are programs used by Government of Canada departments to facilitate the implementation of their mandates and mission while providing results to Canadians. In the Correctional Service of Canada (CSC), the National Contribution Program (NCP) and the National Aboriginal Contribution Program (NACP) are used to complement the delivery of services to Canadians, particularly as it relates to inmates and parolees, community awareness and engagement, and capacity-building in Aboriginal communities.

The NCP and the NACP are intended to support the CSC mandate by providing conditional funding to eligible recipients for activities that encourage and support offenders in reintegrating into society and living as law-abiding citizens.

The activity for which the NCP provides contributions is divided into three main categories: activities intended for inmates and parolees; activities that promote community engagement and participation in CSC activities; and start-up projects for new community residential facilities that are related to corrections and aimed at inmates, parolees or agencies involved in corrections.

The NACP similarly provides funding and support for integrated community-based correctional services that are delivered by Aboriginal communities and organizations to federally sentenced Aboriginal offenders. It also supports the building of capacity in Aboriginal communities by promoting specific initiatives and programs that contribute to the successful reintegration of Aboriginal offenders.

In accordance with the Treasury Board Secretariat (TBS) Policy on Transfer Payments, the NCP and the NACP must renew their terms and conditions by March 31, 2005. Along with this renewal is a requirement for a formative evaluation that examines program processes and implementation. Formative evaluation compares the plan of action in a program proposal to what is actually taking place, using information collected from a variety of sources such as interviews and/or file reviews. The Department's Treasury Board submission must include results from an objective, evidence-based evaluation of the program rationale, relevance, impact and cost-effectiveness.

The evaluation assessed whether, in pursuing their objectives and expected outcomes, the NCP and the NACP efficiently address areas of control such as the activities, inputs and outputs of the programs' design, and the delivery of the programs. The team also reviewed the effectiveness of the programs by assessing their relevance, impact and cost-effectiveness, focussing on the period between 2001 and 2004.

Methodologies appropriate for conducting a formative evaluation were used. The methods consisted of document and file reviews, and a review of the literature pertaining to Government of Canada contribution agreements was conducted. Materials related to the contribution programs' objectives were also reviewed, along with data on funding applicants and recipients, and key informants (internal and external stakeholders) were interviewed. The file review primarily covered available documents that record the activities, procedures and processes of the programs. Interviewers were attentive to the subjective opinions espoused by program officials, stakeholders and representatives of the nationally funded programs.

KEY FINDINGS

Relevance/Rationale

Relevance is determined by establishing whether a program answers a verified need. The Canadian Families and Corrections Network report on the Needs of Families of Offenders demonstrated the need for services and programs that reflect the objectives of both the NCP and the NACP. The programs' objectives and goals were assessed in terms of their relevance to the Department's mission and the needs the programs intend to meet.

  • The findings in this area underscored the importance of the resources and programs such as those provided by the NCP and the NACP to offenders, their families and the community at large.
  • The file review and interviews with key informants revealed that the NCP and the NACP contribute to two of CSC's four strategic outcomes - offenders who are safely and effectively reintegrated, and corporate management support for the care, custody and reintegration of offenders.
Design, Delivery and Management

The evaluation team focused on understanding the practice of program design and delivery and their implications in terms of achieving the NCP and NACP objectives. This is a key component of the programs, because it refers to how the programs structure and administer their policies and procedures. It was determined that the programs rely on the terms and conditions of the program as a reference framework for implementation.

  • The terms and conditions of the program are regarded as the guidelines against which contribution applications are selected and assessed. This is a commendable practice because it ensures that program implementation is aligned with the terms and conditions of the program design.
  • Key informants expressed a high degree of satisfaction with the program design.
  • Interviews with key informants also revealed that delivery approaches needed to be strengthened to allow for a more effective and timely review.
  • The evaluation team found that, where an organization had an established relationship with CSC through contribution agreements, this generally meant that the sustainability of that relationship was ensured. The team sees this as a positive measure on which program managers are encouraged to build as an avenue of reaching out beyond the "prescribed" groups when yearly call letters are sent out.
  • Concerning the allocation of funds by CSC, a need for more accurate forecasting of contribution programs was identified with a view to reducing situations where funds from other envelopes have to be used.
Success/Impact

One of the impacts of the programs is the benefit Canadians acquire from the activities and services they provide. The successes of the NCP and the NACP were examined in strategic and cultural terms, pursuing three lines of inquiry: the extent to which the programs are achieving their expected objectives; whether there is evidence that recipients would be able to undertake the same activities and initiatives funded by the programs without such funding; and, finally, whether recipients access funding from other government programs for the same program or activity.

  • There was a strong sense among key informants that the NCP and the NACP facilitate various programs that would otherwise not have been implemented.
  • According to most respondents, the key measure of success is the fact that these programs reach both offenders and community members. Respondents were positive about their ability to secure contribution agreements and the impacts of the NCP and the NACP in their respective fields.
  • File reviews demonstrate that the NCP and the NACP are assisting CSC in building knowledge and creating awareness about CSC services, thereby systematically enhancing its ability to positively engage the community.
Cost-effectiveness

In measuring the cost-effectiveness of the contribution programs, the team took the approach of assessing the program's effectiveness for participants, sponsors and the general public. A true cost-effectiveness analysis will not be possible in this situation until the programs produce data on outcomes that can be aggregated across funded contributions. As well, the difficulty in conducting a cost-benefit analysis of a program of this nature is the challenge of assigning a monetary value to the costs and benefits of all alternatives. A systematic and rigorous analysis was not always possible because this evaluation did not include an assessment of any alternatives for accomplishing the results that the NCP and the NACP hope to achieve. Nonetheless, our evaluation found that:

  • A high number of key informants stated that the NCP and the NACP are cost-effective in achieving policy objectives and expected results.
  • Some internal key informants also stated that contribution agreements are not always an appropriate vehicle for furthering broad federal policy objectives and that CSC may want to consider delivering services nationally where services have been deemed essential to offender reintegration. Any national service will only serve to complement the services provided by the two contribution programs.
  • The NCP and the NACP provide a vehicle whereby the community can be active in supporting the successful reintegration of offenders and educate communities on the reintegration process.
  • It would be beneficial to enhance the collaboration between the Aboriginal Community Development Officer (ACDO) and recipient organizations engaged in capacity building within Aboriginal communities. ACDOs can be an invaluable bridge between CSC and these organizations.
Conclusions

This evaluation revealed that the programs are supporting a wide range of community activities and are clearly aligned with CSC's strategic objectives. The evaluation similarly discovers a range of accomplishments for the programs and identifies some implementation challenges. The main findings are as follows:

  • Community engagement is gradually being recognized as playing a critical role in creating safe communities. Sections 81 and 84 of the Corrections and Conditional Release Act (CCRA) epitomize the role a community could play in providing correctional services to offenders. Effective community engagement should be an essential benchmark against which the recipients of contribution agreements report their successes.
  • The programs' Results-based Management and Accountability Framework (RMAF) is a tool for defining program performance. It should be refined to precisely respond to expected outcomes and reflect the complexity of its measurement strategies, particularly in relation to measuring the cost-effectiveness of the programs. It must define long-term successes and identify how those successes could be sustained.
  • Regarding design and delivery, applicants need more guidance in terms of reporting on expected outcomes. Further, important lessons learned should continue to be integrated into the implementation of the programs and program processes, especially the application and approval process.
  • The most important challenge for the programs is to develop performance measurement practices, specifically a strategy to follow up with offenders and families so as to demonstrate the link between community support and successful reintegration.

RECOMMENDATIONS

Recommendation 1:

The Results-based Management and Accountability Framework (RMAF) needs to be refined to ensure sound planning in the design and delivery of the programs, enable strategic program management practices and ensure that it is responsive to the level of risk and complexity of the programs.

Recommendation 2:

Establish a timely and effective application review and approval process and enhance the disbursement of approved funds in a manner that supports an effective implementation of programs and services by the recipients of contribution funds.

Recommendation 3:

CSC should establish strategies to engage Aboriginal communities and agencies in the provision of services to Aboriginal offenders.

Recommendation 4:

The programs need to expand their reach to ensure equal access to funding and enhance the call letter as a vehicle for promoting CSC and engaging organizations beyond those approved for funding in previous years.

Recommendation 5:

Establish a consultation strategy that fosters regular consultation by program managers with Treasury Board in seeking guidance on reporting requirements based on the contribution's funding level.

Recommendation 6:

The NCP and NACP should consider establishing a formal strategic alliance and partnership with other federal contribution programs and exchange ideas on both the design and the delivery of the programs.

Recommendation 7:

The Department should better forecast the funding need of the contribution programs in order to allocate appropriate funding levels and reduce instances where funds from other envelopes have to be used.