Evaluation of the First Nations and Inuit Health Branch’s Health Services Integration Fund 2010-2011 to 2014-2015

Prepared by
Office of Audit and Evaluation
Health Canada and the Public Health Agency of Canada
March 2016

Table of Contents

• Executive Summary
• Management Response
• 1.0 Evaluation Purpose
• 2.0 Program Description
  • 2.1 Program Context
  • 2.2 Program Profile
  • 2.3 Program Narrative
  • 2.4 Program Alignment and Resources
• 3.0 Evaluation Description
  • 3.1 Evaluation Scope, Approach and Design
  • 3.2 Limitations and mitigation strategies
• 4.0 Findings
  • 4.1 Relevance: Issue #1 – Continued Need for the Program
  • 4.2 Relevance: Issue #2 – Alignment with Government Priorities
  • 4.3 Relevance: Issue #3 – Alignment with Federal Roles and Responsibilities
  • 4.4 Performance: Issue #4 – Achievement of Expected Outcomes (Effectiveness)
  • 4.5 Performance: Issue #5- Demonstration of Efficiency and Economy
• 5.0 Conclusions
  • 5.1 Relevance Conclusions
  • 5.2 Performance Conclusions
• 6.0 Recommendations
• Appendix 1 - References
• Appendix 2 - Health Services Integration Fund Logic Mode
• Appendix 3 - Summary of Findings
• Appendix 4 - Summary of Completed Objectives by Continuum Stage
• Appendix 5 - Evaluation Description
• List of Acronyms

List of Tables

• Table 1: Program Resources 2010-11 to 2014-15($)*
• Table 2: Limitations and Mitigation Strategies
• Table 3: HSIF - Variance between Planned Spending vs Expenditures*

Executive summary

The Health Services Integration Fund (HSIF) evaluation reviewed project activities for the period of April 2010 to March 2015. The assessment of the Program’s continued relevance and performance was in compliance with the Treasury Board of Canada’s Policy on Evaluation (2009). This was a scheduled evaluation for 2014-15 on the Health Canada Five-Year Departmental Evaluation Plan.

Evaluation Purpose and Scope

The evaluation assessed HSIF for its relevance and performance (effectiveness, economy and efficiency). HSIF was recently renewed (July 2015), as such evaluation findings will support decision-making for policy and program improvements. The evaluation covers program activities, excluding British Columbia (BC) projects.^{Footnote 1}

The methodology used in the evaluation included key informant interviews with First Nations and Inuit Health Branch (FNIHB) regional and national staff, provincial/territorial health department staff, and project coordinators who are often from Inuit and First Nation communities. A survey was also conducted with leads of implemented projects^{Footnote 2}, although the low number of responses received only allowed for survey findings to be used anecdotally. In addition, documents, data, and literature were reviewed.

The Assembly of First Nations (AFN) and Inuit Tapiriit Kanatami (ITK) were consulted during the development of the evaluation methodology, survey, key informant interview guides, and draft report.

Program Description

HSIF is an adapted replacement of the Aboriginal Health Transition Fund (AHTF) (2006-11) and the previous Health Integration Initiative (HII) launched in 2003. HSIF is a five-year initiative to support multi-year projects with a collaborative planning component which could include partners at a provincial/territorial and/or federal level, as well as First Nation/Inuit organizations and/or communities. The program’s stated objectives include:

• Improve the integration of federally-funded health services in First Nation and Inuit communities with those funded by the provinces and territories;
• Build multi-party partnerships to advance health service integration;
• Improve First Nations and Inuit access to health services; and
• Increase the participation of First Nations and Inuit in the design, delivery, and evaluation of health programs and services.

During the period of 2010-11 to 2014-15, HSIF expenditures totalled approximately $51.1 million. Of this amount, approximately $44.5 million was in the form of Grants and Contribution (G&C) spending provided to fund 77 HSIF projects (68 were in scope for this evaluation). HSIF projects varied in nature and ranged from theoretical-based projects (e.g., environmental scans, health model research) and tool development to coordinating and implementing new service arrangements, contributing to a five stage continuum of integration (e.g., planning, capacity building, coordinating services, partnering in the provision of services, and the transferring/merging/amalgamation of services).

CONCLUSIONS – RELEVANCE

There is a continued need for a federal role in establishing and ensuring a culturally relevant and responsive health care system for First Nations and Inuit is in place, which bridges the service gaps experienced across jurisdictions. HSIF integration activities can contribute to greater collaboration among stakeholders, providing the opportunity to lessen the existing fragmentation of health services that often causes disjointed health care planning and delivery priorities among stakeholders seeking to serve the same population. Work in the area of First Nations and Inuit health conducted under the HSIF program is consistent with the federal Indian Health Policy (1979), the Health Transfer Policy (1988) as well as the mandate and function of the First Nations and Inuit Health Branch. The HSIF design illustrates a concerted effort to bring multiple stakeholders together across jurisdictions as well as actively involve First Nation and Inuit communities and organizations in health service integration thereby working towards a more cohesive and responsive health service environment for First Nations and Inuit. Therefore, HSIF is consistent with Departmental and FNIHB strategic priorities, including: improve access to health care and increase quality of care; collaborate with provincial/territorial authorities to deliver services and improve the integration of health services; and increase the control of health care service development and delivery by First Nations and Inuit.  

CONCLUSIONS – PERFORMANCE

Achievement of Expected Outcomes (Effectiveness)

Project activities contributed to a better understanding of enabling factors and barriers impacting health services integration. The multi-jurisdictional nature of projects naturally brought multiple communities, stakeholders and partners together to collaborate on health services integration through partnership development, planning support, and capacity building activities (e.g., cultural sensitivity/awareness training, development of resource/reference materials, administrative and strategic planning skills development). These points are consistent with factors that facilitate health service integration.

HSIF projects were, for the most part, able to achieve their objectives thereby contributing to improved integration of health services for First Nations and Inuit. Evidence suggests that project activities relating to planning, capacity building, and the coordination of health services were most successful, while project activities contributing partnering in the provision of health services also demonstrated significant success in achieving their objectives. Activities feeding into the ultimate integration stage of the transferring/merging/amalgamation of health services also had a moderate degree of achieved objectives. There were a few cases of projects which engaged in beneficial work related to improving health outcomes but in some cases the projects were either not a part of, or did not link to an actual health service which ultimately means the work completed in these projects is not always able to directly impact HSIF’s final outcome of improving access to a quality health service.

Although it is too early to truly assess the depth of the impact HSIF projects have had on the intended long-term outcomes, positive developments have been identified in both the first long-term outcome and ultimate outcome (including both attributes of access and quality). HSIF projects addressed health service quality by improving the cultural awareness of service providers, contributing to a greater continuum of care, and employing a client-centered/community needs-based approach. Access was improved through enhanced awareness of services and service capacity, dedicated resources to facilitate patients’ access of relevant health services, improved service coordination, and a focus on establishing continuums of care.

Demonstration of Economy and Efficiency

Earlier years of HSIF exhibited funding surpluses, while later years demonstrated greater alignment between allocated funding and actual expenditures. Project coordinators reported that HSIF funding enabled them to leverage financial and/or other (in-kind) resources from partners in support of health integration initiatives. Project teams engaging in service mapping and needs assessment analyses became more aware of existing services, thereby extending their resource network and preventing the future duplication of services. In addition, several projects produced outcomes that resulted in more efficient healthcare service delivery due to incorporating the use of technology (e.g., the implementation of electronic medical records reduced the time required to establish a patient’s medical history prior to proceeding with treatment). Details relating to project management (e.g., lack of clarity around roles and responsibilities), in some cases, negatively impacted the efficiency with which project activities were carried out.

The program has a Performance Measurement Strategy in place. Project reporting is currently taking place, and opportunity exists to strengthen projects’ individual indicators by focusing on more outcome-driven indicators, ensuring indicators seeking to measure improvements are accompanied by baseline data, and accounting for a certain degree of specificity in indicators for accurate reporting. Project performance measurement indicators should continue to support program-level indicators while also establishing a distinct set of indicators to measure project-specific performance.

RECOMMENDATIONS

Recommendation 1

Screen proposals with the goal of selecting projects that directly contribute to HSIF’s long-term outcome of focusing on improved access to quality health services.

Potential project proposals should be screened with the intention of completing work with a direct impact on improving access to quality health services. Although that may exclude some projects which focus specifically on improving health outcomes and fall beyond the scope of the intended outcomes of HSIF, improvements to access and quality of health services have the potential to make impacts in those areas as a by-product of completed integration work.

Recommendation 2

Plan for the lifecycle of the project at the workplan stage, and actively consider measures for sustainability as the project progresses.

A common understanding and commitment of project partners at the outset of the project is key to the success of HSIF work. Confusion can arise and impede the timely progress of projects when roles and responsibilities of each participating member are not clear. Therefore, it is recommended that all projects define the roles and responsibilities of each partner at the workplan stage, as well as establish timelines for the achievable milestones for each project.  A discussion of tracking project success by identifying intended achievements, and the regularity of data collection which can feed into performance measurement, should also be established before project work is fully underway. Furthermore, ways to facilitate the sustainability of the work accomplished needs to be considered at all phases of the project; and should therefore be incorporated as a distinct field on project reporting templates.

Recommendation 3

Improve project-level performance measurement efforts by focusing on indicators that have greater clarity and are outcome-oriented. Ensure the program logic model is consistent with the recently updated direction of HSIF.  

Project indicators should focus on telling the story of the impacts of work completed. This is made possible through the use of indicators that are less focused on outputs and account more for what resulted from the processes, tools, and other activities implemented through HSIF projects. Identifying meaningful indicators can be a challenge, and it may be beneficial to host performance measurement capacity building sessions with project coordinators and regional staff at the outset of the project to ensure that the best possible data is collected throughout the lifespan of the project. In light of recent shifts towards more devolution focused projects, the program logic model should be reconsidered to account for long-term outcomes that are consistent with this new approach, thereby ensuring program activities continue to contribute to overall goals, and that future evaluations are assessing the proper criteria.

Management Response and Action Plan

1.0 Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance (effectiveness, economy and efficiency) of the FNIHB’s Health Services Integration Fund (HSIF) for the period of April 2010 to March 2015.

The evaluation was required by the Financial Administration Act (for grants and contributions) and the Treasury Board of Canada’s Policy on Evaluation (2009). This was a scheduled evaluation for 2014-15 identified on the Health Canada Five-Year Departmental Evaluation Plan. The evaluation will support Health Canada’s Deputy Minister and senior management in decision-making for policy and program improvements.

2.0 Program Description

2.1 Program Context

Health Canada’s involvement in health services integration was designed with the intent to promote innovation and multi-jurisdictional partnerships in First Nations and Inuit healthcare planning and delivery. HSIF’s focus on increased First Nations and Inuit participation in all aspects of healthcare delivery and design, seeks to create more responsive health services for community members. With this goal in mind came the implementation of three consecutive iterations of a fund to enable First Nations and Inuit health service integration. Integration takes many forms, and for the purpose of this evaluation it can be understood as taking place across a continuum with stages that often consist of: planning, capacity building, coordinating services, partnering in the provision of services, and the transferring/merging/amalgamation of services. Although this conceptualization captures the progression of greater degrees of integration, it is important to note that not all stages are relevant to each type of project. For example, a health assessment tool may contribute to capacity building and improved service coordination, but it would never evolve into an amalgamated service in and of itself. Ultimately, integration in the context of the HSIF is designed to improve the access to quality health services for First Nations and Inuit.

The first version of the Fund was launched in 2003 as the Health Integration Initiative (HII) and included eight pilot projects which sought to explore, develop and analyze models of integration for improved delivery of health care services and programs to First Nations and Inuit. The HII received $10.8 million over a three-year period from the Aboriginal Envelope of the Primary Health Care Transition Fund.

Building on this initiative, the Aboriginal Health Transition Fund (AHTF) was launched in 2006 and incorporated lessons learned from HII, including the need for capacity at all partner levels. As a result, AHTF provided capacity building for Aboriginal organizations in order to encourage participation at the national and regional levels. Unlike HII, approved AHTF project proposals addressed the integration and adaptation of existing services. AHTF had a $200 million budget over five years funding 311 projects, and was eventually extended by one year to allow for project completion and the gathering of lessons learned to inform the implementation of the HSIF.

HSIF was launched in 2010 with a focus on health system transformation (e.g., organization, structure, partnerships, and service delivery). The program built on the lessons learned from the previous HII and AHTF initiatives, and as a result more time was incorporated into the early phase of projects as a means to encourage joint planning and priority setting among partners.  HSIF received $59.4 million over a five year period,^{Footnote 3} funding 77 projects across all provinces and territories. The program has since been renewed with A-base funding of $15.8 million annually.

2.2 Program Profile

HSIF was designed as a five-year initiative to support multi-year projects with a collaborative planning component which could include partners at a provincial/territorial and/or federal level, as well as First Nation/Inuit organizations and/or communities. The program’s stated objectives are listed as:

• Improve the integration of federally-funded health services in First Nation and Inuit communities with those funded by the provinces and territories;
• Build multi-party partnerships to advance health service integration;
• Improve First Nations and Inuit access to health services; and
• Increase the participation of First Nations and Inuit in the design, delivery, and evaluation of health programs and services.

The 77 HSIF projects (68 were in scope for this evaluation ^{Footnote 4} ) varied in nature and ranged from theoretical research-based projects and tool development to implementing services, contributing to a continuum of integration:

• Planning: e.g., Developing community health plans/models, identifying gaps in services, developing co-strategic frameworks;
• Capacity Building: e.g., Cultural safety training courses, standardized screening tools;
• Coordinating Services: e.g., Hiring health system navigators to facilitate clients’ seamless access to health services, advocacy support for people with special needs, harmonization of practices across existing mental health and addictions services as a means of strengthening the continuum of care;
• Partnering in the Provision of Services: e.g., Multi-service one-day clinics hosted in communities, integrated primary care programs on-reserve, health service agreements; and
• Transferring/Merging/Amalgamation of Services: e.g., A supportive living residential model to support people with past hardships develop skills to enable them to live independently, and governance projects.

2.3 Program Narrative

HSIF has a logic model (see Appendix 2), and its components are discussed below.

The long-term expected outcomes for the program are (1) an increase in independent health services integration arrangements for First Nations and Inuit, and (2) improved access to quality health services for First Nations and Inuit. The thought is that integration arrangements created through HSIF partnerships will equip stakeholders with the skills and capacity to independently initiate and sustain new opportunities at a later date without reliance on start-up federal funding. Consequently, the first long-term outcome is expected to materialize beyond the life-cycle of the actual HSIF projects. These new arrangements are expected to continue addressing First Nations and Inuit health integration needs and will further contribute to HSIF’s ultimate outcome of improving access to quality health services for First Nations and Inuit.

The program lists one key intermediate outcome which is the heart of the initiative, greater integration of health services which serve First Nations and Inuit. Progress towards this outcome is estimated at 3-5 years from the point of initial funding, and therefore is achievable within the scope of outcomes reviewed for this evaluation. Increased integration could range from “coordination at the service level which improves communication and reduces service gaps to coordination at the governance level which clarifies roles and responsibilities all the way up to joint policy development and planning at the senior management and political levels”.^{Footnote 5} Evidence of progress in this outcome area could include the development, implementation or harmonization of either formal agreements, policies, tools or practices.

The immediate outcomes include: (1) increased capacity of key partners and stakeholders to collaborate on the integration of health services which serve First Nations and Inuit, and (2) increased knowledge of the concepts, barriers and enablers related to integrating the health services which serve First Nations and Inuit. Active partnership engagement which incorporates a balanced presence of partnering members on the project advisory committees would ultimately lead to jointly developed integration plans and program outputs that are informed by a greater understanding of prerequisites and risks to effective integration.

 The outcomes above are expected to be achieved through engagement in the following activity areas:

1. HSIF governance and planning
2. Building capacity for integration
3. Implementation of integration plans
4. Monitoring and evaluation of implementation
5. Policy development and knowledge sharing

The target group of HSIF’s activities is broad considering projects are initiated through needs based proposals.  The following groups are eligible to receive funding for projects that ultimately benefit First Nations and Inuit health services: First Nation communities, tribal councils and organizations, Inuit communities and land claims organizations, Aboriginal associations and organizations, provincial, territorial and regional health departments and authorities.

2.4 Program Alignment and Resources

The program’s financial data for 2010-11 through 2014-15 are presented below (Table 1).

Overall, the program was allotted a budget of $80 million over a five year period from 2010-11 to 2014-15, although part way through the budget was reduced to $59.4 million.^{Footnote 6} As highlighted in Table 1 of the budget available from 2010-11 to 2014-15, approximately three-quarters (74%) of total funding was allocated to project funding via Grants and Contribution (G&C) funding, with a further 12% of funding for Operations and Maintenance (O&M) accounting for overall expenditures. Approximately 14% of the budget was allocated to staff salaries, and Full-Time Equivalents (FTEs) over the length of HSIF range from 31.1 in the first year to an average of 21.73 over the following four years.

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The evaluation covered the period of April 1, 2010 to March 31, 2015 and included HSIF projects delivered in all regions except BC^{Footnote 7Footnote 8} . The 68 projects^{Footnote 9} varied greatly, contributing to different stages of the integration continuum including: planning, capacity building, coordinating services, partnering in the provision of services, and transferring/merging/amalgamation of services.

The evaluation issues were aligned with the Treasury Board of Canada’s Policy on Evaluation (2009) and considered the five core issues relating to relevance (continued need, federal role, alignment with government priorities) and performance (effectiveness, economy, efficiency), as shown in Appendix 5. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process. The Assembly of First Nations (AFN) and Inuit Tapiriit Kanatami (ITK) were also consulted during the development of the evaluation methodology, as well as survey and key informant interview guides.

Data was collected and triangulated from four lines of evidence to increase the reliability and credibility of the evaluation findings. Sources included:

Document and Data Review

The document review consisted of project-specific proposals as well as status reports drafted by Fund recipients and project evaluations. The program also supplied background material to shed light on the program’s evolution. Financial data and project tracking spreadsheets were also included in the review. The document review contributed to analyses for all of the evaluation questions.

Literature Review

The 2010 program-commissioned literature review continued to be relevant in discussing culturally sensitive forms of health service integration. As a result, this evaluation conducted a small supplemental literature review to inform outcome indicators. The literature review also contributed to the relevance section of the report. The combined literature review was comprised of information from peer-reviewed journals and grey literature.

Survey

An online survey was created to engage leads of implemented projects in order to gather lessons learned specific to implemented projects. The 37 leads from qualifying projects were invited to participate through email invites, followed by reminder phone calls, emails from the evaluation team, and FNIHB regional communication. The survey launched on May 12, 2015 and closed on May 26, 2015. In the end, 13 eligible participants completed the survey. Consequently, survey findings were used anecdotally.  

Key Informant Interviews

A total of 41 key informant interviews took place across three main groups: FNIHB staff at the national (2) and regional levels (10), provincial/ territorial health staff (12), as well as project coordinators mainly from First Nation and Inuit communities/organizations (17)^{Footnote 10} Regional representation was accounted for across all three groups. Participation was voluntary and key informants were provided with the opportunity to review the evaluators’ notes from their interview in order to ensure their feedback was accurately represented. Key informant interview data contributed to virtually all sections of the evaluation report.

3.2 Limitations and Mitigation Strategies

The following table outlines the limitations encountered in the conducting of the HSIF evaluation. Mitigation strategies put in place used to ensure the strength of analysis to support the findings are also noted below.

4.0 Findings

4.1 Relevance: Issue #1 – Continued Need for the Program

There is a continued need for the HSIF program as evidenced by challenges faced by Inuit and First Nation people in accessing a full range of health services. Often, the absence of quality health services located in close proximity to First Nation and Inuit communities negatively impacts the continuity and timeliness of care received. Such challenges are more likely to be addressed through greater health services integration which can facilitate the coordination of priorities across partners at all levels of health care planning and service delivery.

First Nation and Inuit clients seeking care in community often do not have access to a full range of health services. This is particularly relevant in the cases of remote or rural communities with low population density, and as a result do not have the developed infrastructure or capacity to support an array of health services (including treatment for complex diagnoses). A great majority of project coordinators interviewed identified there is a lack of First Nations and Inuit capacity to provide and/or coordinate services. As a result, community members often need to leave the community to access the needed health services.

Coordinating the receipt of health services outside of communities impacts the timeliness of care received. By not having easily accessible services, the frequency with which services are accessed decreases and can therefore impact the degree of care required, and the continuity of care received. Similar concerns about continuity of care persist in communities due to the high rates of staff turnover among health professionals traveling to communities.^{Footnote 11}

The lack of integrated services negatively impacts the ability to have coordinated health care priorities at all levels of health care planning and delivery. Most project coordinators and almost half of FNIHB regional staff interviewed identified a lack of coordination of available services. This lack of coordination often stems from the multijurisdictional nature of First Nations and Inuit healthcare and the associated confusion among government organizations as to who is responsible for the delivery of health services and the application of provincial standards determining eligibility criteria for on-reserve populations. This confusion paired with inconsistent health service priorities across jurisdictions creates greater challenges in addressing gaps in First Nations and Inuit health services. An integration literature review commissioned by the program in 2010 found that "the structure and organization of the [First Nations/Inuit federal system and the regional provincial health authorities] seem to have been going in different directions with the provincial system combining or integrating service providers into larger organizations in order to offer higher quality care and to reduce unnecessary costs while the federally funded First Nations health system has been devolving service delivery to smaller units usually at the band level".^{Footnote 12}

Without an integrated approach First Nations and Inuit input is not always prioritized in the design and delivery of health services. It is often presumed that this impacts the degree of cultural responsiveness incorporated into health services.^{Footnote 13} Approximately half of FNIHB regional and provincial/territorial staff interviewed identified that there is a lack of culturally appropriate services and cited this deficit as a barrier to health service integration. The availability of culturally relevant services feeds into the conversation about ensuring quality services exist, and ultimately contributes to better health outcomes for Inuit and First Nation people.^{Footnote 14}

It is worth noting that a shift to a more collaborative and inclusive approach to First Nations health services planning is evident in some provinces' cross-jurisdictional forums. Arrangements of note include:

• BC's Tripartite First Nations Health Plan (Kelly, 2011);
• The Saskatchewan Northern Health Strategy (Nilson, Jeffery & Hamilton, 2010); and
• The Manitoba Inter-Governmental Committee on First Nations Health (Government of Manitoba, 2013).

4.2 Relevance: Issue #2 – Alignment with Government Priorities

HSIF is aligned with federal priorities promoting healthy and self-sufficient First Nation communities, as well as the strategic priorities of both FNIHB and the Department. HSIF's health integration work is consistent with the 2015 Minister of Health Mandate Letter which underlines the importance of improved partnerships with Indigenous Peoples, provincial, territorial and municipal governments.

The HSIF objectives align with Health Canada's third organizational priority of strengthening First Nations and Inuit Health programming.^{Footnote 15} The Fund therefore contributes to the departmental strategic outcome^{Footnote 16} of First Nation and Inuit communities and individuals receive health services and benefits that are responsive to their needs so as to improve their health status.^{Footnote 17}This third priority outlines the Department's support of First Nations and Inuit programming, as stated in the 2013-14 Departmental Performance Report, which seeks to:

Improve access to health care and increase quality of care

HSIF's relevance has been highlighted in recent federal budgets that prioritized systemic change for the provision of healthcare for FNIHB's client population as a means to increase access to quality health services. The 2012 Budget brought an investment of $134 million to First Nations health service infrastructure. In addition, the 2013 Budget allocated $48 million over two years to contribute to the improvement of the quality of health services in First Nation communities. A specific commitment to services in the North was highlighted in the 2013 Speech from the Throne stating a continued commitment to the prosperity of Northerners through investments in health care, education and affordable housing. This priority is also consistent with FNIHB's first strategic goal of ensuring access to quality health services.  

Collaborate with provincial/territorial authorities to deliver services, and improve the integration of health services

HSIF aims to integrate provincial and federally funded health services as a means to address gaps in health service delivery for First Nation and Inuit communities. A review of 68 HSIF projects indicates that 60% (n=41) of projects involved provincial and territorial partners, and 56% (n=38)^{Footnote 18} have partnered with local health authorities. This priority is also consistent with FNIHB's second strategic goal of strengthening collaborative planning and relationships.

Increase the control of health care service development and delivery by First Nations and Inuit

HSIF's proposal-based design seeks to engage First Nations and Inuit in the planning and delivery of projects that contribute to sustained systemic change in health services. This may include, but is not limited to, greater service harmonization, increased aggregation of communities involved, and a broader scope of population served.^{Footnote 19}

First Nations and Inuit health services discussions have highlighted the importance of collaboration, joint priority development, and services that contribute to better health outcomes. Recent federal-level discussions specific to this priority have been captured in the following sources:

• The 2015 Minister of Health Mandate Letter commits to improved partnerships with Indigenous peoples, provincial, territorial and municipal governments.^{Footnote 20}
• The 2015 Budget prioritized providing targeted funding to support health system innovation.
• The 2014 Speech from the Throne highlighted the government's commitment to work with First Nation and Inuit communities to create healthy and self-sufficient communities.
• The 2013 Speech from the Throne emphasized the importance of working in partnership with Aboriginal communities to ensure health care services are responsive to community members' needs.
• The 2013 Budget prioritized the promotion of health and safety for Aboriginal families and communities.
• The 2012 Crown-First Nations Gathering resulted in shared priorities relating to the health of First Nations and Inuit.
• Health Canada's 2012 First Nations and Inuit Health Strategic Plan included priorities relating to responsive health services and the strengthening of health programming for First Nations and Inuit through innovative partnerships, as a means to increase access to, and integration of, health services.

4.3 Relevance: Issue #3 – Alignment with Federal Roles and Responsibilities

Work in the area of First Nations and Inuit health conducted under the HSIF program is consistent with the federal Indian Health Policy (1979), the Health Transfer Policy (1988) as well as the mandate and function of the First Nations and Inuit Health Branch.

Although there is no statutory framework for the provision of health care programs and services to First Nations and Inuit by the federal government, the following policies do outline the goals of the federal government with respect to Aboriginal health:

• The Indian Health Policy (1979) speaks to the federal importance of fostering a more responsive health system for First Nations and Inuit. This policy encompasses the improvement and enhancement of access to health care services for First Nations and Inuit, which is consistent with HSIF’s desired long-term outcome.
• The Health Transfer Policy (1988) supports federal efforts to engage First Nation and Inuit communities in all aspects of health service programming including planning, decision-making, delivery, and potential control of health services. The spirit of this policy seeks to nurture the relationship among the federal government and First Nation/Inuit communities. The engagement of communities in decision-making and planning for healthcare projects aligns with HSIF’s proposal-based design which often sees projects initiated at the First Nation and Inuit community/organization level with support from FNIHB’s regional staff and other partners.

As discussed in the previous section, the tenets outlined in the policies above are also consistent with the Branch’s priorities of: improving access to health care and increasing quality of care; collaborating with provincial/territorial authorities to deliver services, and improving the integration of health services; and increasing the control of health care service development and delivery by First Nations and Inuit.

4.4 Performance: Issue #4 – Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent have the immediate outcomes been achieved?

Immediate outcome #1:
Increased capacity of the key partners and stakeholders to collaborate on the integration of health services which serve First Nations and Inuit.

HSIF has increased the capacity of key partners and stakeholders to collaborate on the integration of health services which serve First Nations and Inuit through partnership development, planning support, and capacity building activities.

Partnership Development

There was significant evidence that projects funded through HSIF contributed to either the maintenance of existing partnerships, or supported the development of new partnerships. From 2010-15, HSIF funded 77 projects across all regions, and, in all but three cases, projects were comprised of multi-partner arrangements. Overall, data which includes BC projects indicates that partnerships fostered through HSIF have included the engagement of 518 communities, more than 70 First Nation and Inuit organizations, 47 Regional Health Authorities and 18 non-health government bodies at the federal, provincial and/or municipal levels.^{Footnote 21Footnote 22} There were at least 481 partnerships that fell within the scope of this evaluation. The majority of stakeholders interviewed felt that one of the greatest impacts of the project funding was the support/development of partnerships between First Nation and Inuit organizations and other federal/provincial health authorities. Provincial/territorial and FNIHB regional key informants also confirmed that the work completed through these projects led to an increase in multiple communities working together. Project data confirms that a proportional shift in community collaboration occurred across integration program iterations with 38% of AHTF projects having involved only a single community, whereas during HSIF this rate decreased to 4%.  

Increased networking attributable to partnership development produced significant benefits in improving relationships among all First Nations, Inuit, and provincial, territorial and federal health partners. In particular, development of partnerships was seen to improve overall communication and understanding between First Nation and Inuit organizations themselves, as well as with other federal, provincial/territorial and/or regional health organizations. The vast majority of provincial health key informants indicated that through HSIF activities there has been a greater integration of First Nations and Inuit community interests into government health services planning. This is supported by the presence of First Nation and Inuit stakeholders on provincial and territorial health committees, as indicated by approximately three-quarters of the provincial and territorial key informant interviews. Furthermore, the new iteration of HSIF born out of the July 2015 program renewal has a strong focus on supporting tripartite work across the various regions.

Planning Support

Most projects included the activity of conducting some type of environmental scan as a needs assessment, by reviewing which services were already available and identifying the gaps that may exist in the First Nations and Inuit health service landscape. These activities were instrumental in the strengthening of partnerships, informing project planning and to some degree, impacted the implementation of project activities at varying levels along the integration continuum by understanding the greatest areas of need and feasible points for integration based on service intersections.

Environmental scan activities allowed partners to truly understand the role each other played in First Nations and Inuit health service planning and delivery. Naturally, as indicated in the majority of FNIHB key informant interviews, this understanding was the result of improved communication and relationships of the partners from multiple jurisdictions that came together through HSIF projects. Improvements in communication went beyond forging relationships, and made progress in the area of language used from a cultural perspective versus a government-focused approach. In addressing these differences groups were able to better facilitate integration opportunities since the language and policies that hindered common understandings in fostering a seamless provision of health care to Inuit and First Nation people could be adapted to allow a better context for integration work.

The strides made through project planning and the greater understanding of respective roles in First Nations and Inuit health service delivery as well as the alignment of priorities had a more pronounced impact in the cases where protocols as well as the alignment and/or standardization of processes (14/18), and service agreements/MOUs (5/6) came to fruition. According to almost all interviewed project coordinators, formal and informal arrangements resulted from project partnerships in almost all cases. 

The involvement of First Nation and Inuit communities and organizations in project planning, development or implementation, and ultimately health services planning, helped bridge some of the disconnects existing between government-oriented planning and more holistic cultural approaches to health service planning and delivery. Gaps analyses and needs assessments informed the creation of community-based culturally relevant plans focusing on local needs and wants in the provision of health care services. This approach, along with the proposal-based nature of HSIF projects, positively contributed to the fostering of culturally relevant health services responsive to the needs of First Nations and Inuit.

Capacity Building

A great majority of project coordinators interviewed identified a lack of First Nation and Inuit community capacity to provide and coordinate services as a key barrier to integration. This clearly defined need for capacity building across First Nations and Inuit integration projects has been identified in previous health service integration initiatives. A review of AHTF found that the strength and quality of partnerships varied based on organizational capacity, and that low levels of capacity produced longer term impacts to integration progress by impeding knowledge transfer. A review of project reporting found that approximately half of projects were engaged in capacity building; a greater degree of capacity building took place among projects in the implementation phase. 

Capacity building was perceived by stakeholders as a prerequisite towards enhanced health care integration. A review of reporting from 68 projects^{Footnote 23} found partnership contributions related to capacity building typically included strategic planning (63%), knowledge sharing (67%) which included training opportunities, as well as linkages to other organizations and networks (65%). The expertise gathered through partnerships is consistent with the literature review findings that suggest organizations are better able to increase their capacity through the greater and more appropriate use of partner resources. These types of internal capacity building helped projects produce tangible plans and agreements while also instilling staff with new found skills that are beneficial for integration work. Project examples of this form of capacity building include:

• The Continuing Care project, which hosted a gap analysis workshop to assist the management team with the development of the analysis. As a result, the project management team developed an understanding of how to identify gaps, barriers, challenges, and successes in the project.
• The Minobiamaadizin project, which provided a workshop on the development of partnerships, MOUs and community service agreements. This resulted in the development of a draft MOU outline and service agreements.
• The Community Health and Wellness Training and Support project, which provided training to regional committees on a community development approach to identifying health priorities. This resulted in improved healthcare planning capacity within targeted communities.

In addition, projects developed and/or delivered at least 36 training sessions and or/training resources.^{Footnote 24} Providing technical, cultural or content area training helped ensure that staff had the appropriate skills to provide quality and/or culturally appropriate services to First Nations and Inuit. In the case of the project Partners for Culturally Adapted Health Care, through the development of cultural training, a strong relationship between Algonquin communities and provincial health and social services was created. As a result of their HSIF project, the evaluation report stated there was, "a better understanding of the unique cultural needs of the Algonquin communities, as well as the situation of health and social services within the communities". ^{Footnote 25}

Immediate outcome #2:
Increased knowledge of the concepts, barriers, and enablers related to integrating the health services which serve First Nations and Inuit.

The majority of key informants reported that as a result of HSIF, their knowledge of concepts, barriers, and enablers to the integration of health services serving First Nations and Inuit had increased. Knowledge transfer activities among partners were particularly effective in expanding their understanding of the factors impacting the integration of health services, as well as research-driven activities.

HSIF projects were instrumental in improving the communication among different stakeholders involved in health services planning and delivery. This extended to established partners within the project and linking with resources uncovered through service mapping/environmental scan exercises. By improving both the relationship and communication among partners, opportunities for knowledge exchange were made available. FNIHB regional key informants unanimously confirmed that they had shared project lessons with others. On the other hand, a strong majority of FNIHB regional key informants and project coordinators felt that more knowledge exchange activities were needed. One project coordinator suggested regular sharing across projects in the region could be facilitated by a website outlining the HSIF projects with links to project details as a starting point for efficiently sharing resources across projects and facilitating networking opportunities.

The sharing of lessons learned among project stakeholders, and projects' research conducted through literature reviews and varying forms of environmental scans, highlighted the enablers and barriers for integration which are discussed below.

 Interviews with some project coordinators suggested that a critical lesson learned was the importance of stakeholder readiness and involvement at the outset of the project. This was also supported in a review of FNIHB's health integration work taking place from 2003- 2013, which identified stakeholder readiness as a key factor to the success of health integration projects, and characterized it as, "all partners ready, willing and able to establish a relationship, with mutual understanding of the starting point and outcome".^{Footnote 26} In the case of some HSIF projects, as identified by provincial and territorial stakeholders, confusion at times materialized out of varying levels of readiness among stakeholders. Subsequently, projects engaged in capacity building endeavours such as sharing knowledge and expertise among partners, linkages to other organizations/networks, and strong strategic planning resources as a means to positively contribute to levels of preparedness.

Furthermore, a clear understanding of roles and responsibilities of partners at the outset of the project enables a positive progression towards completing project activities. The great majority of FNIHB regional staff felt that there was a range of at least a fair to strong understanding of roles and responsibilities among project stakeholders, which influences the effectiveness of implementation (e.g., greater frontline staff buy-in). Provincial and territorial representatives interviewed indicated only minor confusion in a few projects vis-à-vis roles and responsibilities, which typically arose in the initial stages of the project. Areas of confusion mentioned by either FNIHB staff and/or project coordinators included expectations surrounding the degree of involvement of stakeholders, staff turnover which at times resulted in role confusion, and in very few instances a lack of clarity surrounding roles and responsibilities was attributable to poor communication. Some of the best practices listed by key informants that contribute to a stronger sense of stakeholder roles and responsibilities included: the development of terms of reference, workplan definitions of roles and responsibilities, as well as formalized relationships/agreements. The vast majority of project coordinators listed regular meetings among stakeholders as the most effective approach for clearly defining roles and responsibilities, including in-depth discussions of project objectives for a clear common vision. This approach was also supported by provincial/territorial representatives. The importance of planning for the lifecycle of the project at the beginning stages was evidenced as a best practice to facilitate project sustainability.  

The involvement of First Nation and Inuit partners in decision-making processes was also found to be a key enabler for health services integration. The literature review notes that, "population health research has shown that improved health outcomes are associated with greater control by First Nations, by health programs that are more culturally appropriate and where the health programs take a more holistic approach and are integrated with other social programs."^{Footnote 27} Although improved health outcomes is not a specific long-term outcome of the program, as illustrated above there is a strong link to addressing quality and access (HSIF's ultimate outcome) which can be positively impacted by the involvement of First Nations and Inuit in the planning and implementation of health services. A significant majority of the project reports indicated that decision-making was collaborative.

Although general services may exist for possible integration initiatives, a barrier to effective service integration for First Nations and Inuit health services is a lack of culturally appropriate services, which limits opportunities for meaningful integration.  Half of FNIHB regional staff and almost half of provincial/territorial key informants found low cultural sensitivity or appropriateness within the services available to be a barrier to the integration of health services for First Nations and Inuit.

4.4.2 To what extent have the intermediate outcomes been achieved?

Intermediate Outcome #1:
Greater integration of health services which serve First Nations and Inuit.

Multiple lines of evidence confirmed that HSIF projects contributed to a greater degree of integration of health services which serve First Nations and Inuit. Projects relating to planning, capacity building, and coordinating services had high levels of success at achieving their stated objectives, while project activities relating to later stages of the integration continuum (partnering in the provision of services, and amalgamation of services) had moderate to strong degrees of success. ^{Footnote 28}

Health services integration can best be understood as taking place along a continuum comprised of five stages: planning, capacity building, coordinating services, partnering in the provision of services, and transferring/merging/amalgamating services. There is a certain fluidity among the stages and therefore a single project can consist of activities that contribute to multiple stages, or in some cases are not applicable to certain areas of the continuum. For example, a health assessment tool may facilitate the coordinating of services but would not lead to an amalgamated health service entity. HSIF projects spanned the continuum, varying in nature and scope, ranging from theoretical research-based projects and tool development to the alignment and implementation of service structures. All interviewed FNIHB staff indicated that projects had contributed to increased integration, and a majority of provincial/territorial staff interviewed were also in agreement. This section explores accomplishments and challenges specific to each phase in order to provide a better understanding of areas in which contributions to integration were made.

Planning

The planning stage of the continuum is important in terms of aligning partner priorities and visions to create a unified approach to integration. Theoretical activities such as research-driven projects like environmental scans of existing services/legislation and gap analyses can give partners a snapshot of the current state of health service delivery. Furthermore, environmental scans help in the identification of points of intersection where integration is feasible. Literature reviews and planning among parties with various areas of expertise can also help inform and strengthen integration planning. An inventory of planning activities beyond joint priority setting and planning included: 25 environmental scans (19 completed), 7 health models (e.g., service delivery models, health and wellness models (5 completed)), implementation of/participation in 7 symposiums or forums (6 completed), 5 strategic plans (4 completed), 3 business plans (1 completed), and 2 policy consultations (2 completed).

Research driven activities

Environmental scans, needs assessments, gap analyses and best practice reviews were methods used to better understand the health services landscape within which projects were operating. By performing these reviews, projects were able to target planning in areas of need and reach out to resources to create opportunities for referrals and the coordination of services, or simply discover services in the areas that staff did not know existed. In the case of the project Development of a System of Integrated Mental Health and Addictions Services Between First Nations, Aboriginal and Non-Aboriginal Services the environmental scan of services revealed that 38 mental health and addictions service providers in the surrounding area of the Iskatewizaagegan Independent First Nation in Kenora, Ontario were funded by the regional health authority and provincial government with a mandate to service First Nations clients, while only three of those providers were delivering services on reserve.^{Footnote 29} The environmental scan served as a starting point to clarify service provider assumptions that the First Nation reserve was beyond their catchment area. The project's evaluation report indicated that as a result of the HSIF project at least eight services in the surrounding area which were previously underutilized are now regularly being accessed by community members. A lack of clarity regarding service provider responsibility for services was listed by the majority of interviewed project coordinators as a barrier to service access. Although research activities can have broader impact as illustrated in the example above, the majority of the 25 projects with these components used the exercise to get acquainted with the health service landscape in their area and identify opportunities for integration. At the time of reporting, 19 projects had completed their research, one environmental scan was still in draft form while another project's environmental scan was still in progress. Three projects were unable to achieve their objective in this area, while one other project did not report on this task.

Joint planning/shared vision

Intended project outcomes often included First Nations/Inuit involvement in the design and delivery of projects/health services. Key informant interviews identified this involvement as an important and positive factor captured through HSIF. Furthermore, a review of the approved project proposals indicates that 90% were submitted by First Nation/Inuit communities and/or organizations

Symposiums and forums contributed to a better understanding of partners' vision for integrated services and a unified understanding of existing health needs. In the case of the Inuvialuit Regional Corporation's Regional Advocacy/Coordination Program project, a workshop to establish a unified vision for Beaufort-Delta Region service providers found through a survey of attendees that there was a "unanimous shift" from focusing on the past to working towards addressing barriers in order to foster a healthier future.^{Footnote 30} Five projects planned on offering a symposium or forum to foster a shared understanding of health service needs, and four were implemented. There was one case of a project that had an unintended outcome of developing a proposal for a youth suicide forum. Activities of a seventh project included youth from the community travelling to a national Truth and Reconciliation Commission event to learn about their past and the resounding impacts.

All key informant groups strongly believed that the improved communication and working relationships developed through HSIF contributed to the alignment of programs, goals and policies. An example of alignment activities achieved through projects included joint community health planning which often involved local committees and/or health partners at the provincial and territorial level. A strong majority of key informant interviews with provincial/territorial health representatives indicated that First Nation/Inuit representatives are a part of their health planning steering committees and that there has been improved integration of First Nation and Inuit interests into provincial/territorial health planning. In the case of the Community-Based Recovery from Addictions Programs project and the Nunavut Wellness Plan project, health planning goals fed into local health department initiatives including working with territorial-level health strategies and identifying common areas of interest, and establishing natural synergies for health service planning. In the case of some projects, shared priorities further established senior level buy-in, which could include a Chief's support of project goals/intended outcomes as a community priority or senior management direction at the provincial level. 

Developing health models/business plans/implementation plans

Fifteen projects included the activity of developing a type of health model, strategic plan, or business plan. Two additional projects focused on policy consultations and were able to complete the work they intended to do.

Only one of the three projects citing activities that could feed into a business plan were able to complete their predetermined tasks. The completed business plan in the case of Onion Lake First Nation was a supporting document for discussing the renewed integration relationship with the local public health authority and outlined a clear vision for the long-term health arrangement for the area. The activities of four out of five projects targeted at developing strategic plans and engagement strategies were completed.

With respect to the planned development of seven service delivery or general health and wellness models, five were fully developed, and two are still in the process of being drafted at the time of final reporting. Although deliverables were developed, project reporting demonstrates some challenges in the strength of the models and/or the details surrounding implementation. Capacity issues impacted the development and, in some cases, the ability to implement certain models. A lack of awareness on the parts of partners whether that be related to the specifics of content matter and/or the degree of involvement throughout the life cycle of the project were the primary capacity concerns impacting model development. In one particular case, the withdrawal of partners' committed resources impacted the rate with which the project could progress. In terms of implementation, the majority of projects did not develop an implementation plan in connection with their model or in the case of one project, the plan called for extensive work in reshaping the model with no financial or formalized support in place. As one project evaluation report aptly stated, model development without an implementation plan makes the project a theoretical exercise.

Capacity Building

FNIHB's former health service integration initiatives and the changing policy environment which has promoted the transfer of health service delivery to First Nations has highlighted the need for capacity building in order to support integration activities and encourage innovation. Since the introduction of the Health Transfer Policy in 1989, "Health Canada regional offices were as large as ever and the transfer of policy leadership and specialty services to First Nations had been thwarted by their small size and the lack of professional capacity in community health organizations".^{Footnote 31} The vast majority of HSIF project coordinator interviews indicated a lack of First Nations and Inuit community capacity to provide or coordinate health services as a barrier to health access. Approximately one-third of this same group of interviewees cited this lack of capacity as a factor impeding the increase in First Nations and Inuit ownership and control of health services. Close to half of Regional FNIHB staff interviewed indicated that projects' capacity building activities as a result of HSIF have provided First Nation and Inuit community members with the skills needed to direct their own health services. Lessons learned from both HII and AHTF demonstrated a need for capacity at all partner levels to engage in integration activities. In many cases, First Nations and Inuit's active involvement in the health service integration projects was hampered by a lack of capacity; consequently some AHTF funding was earmarked to support capacity development among National Aboriginal Organizations (NAO) and Provincial/Territorial Organizations (PTO). Funding to political organizations was impacted by budget cuts, and as a result, funds allocated under HSIF to support capacity in these organizations was eventually eliminated. The absence of HSIF financial support for PTO capacity may be felt more significantly in the latest iteration of HSIF (July 2015) which has a strong focus on the development of tripartite agreements.

HSIF capacity building project activities took many forms including training, mentoring, resource and tool development. Project reporting indicates that activities within this category demonstrated a strong degree of meeting the planned objectives.

Training

Training topics focused most prominently on improving cultural awareness and skills development in addressing mental health. As illustrated by the literature, these are two key areas of need within First Nations and Inuit health service delivery, further speaking to the relevance of HSIF projects.  

According to project progress and evaluation reports, 10 projects had objectives related to some form of cultural awareness training. Eight projects implemented their training sessions as planned, one developed a draft module, and another is still in the process of developing their training. There was also an additional project which developed and implemented a cultural training module without having listed it as a program objective. Each initiative found their training to have positively contributed to the cultural understanding of the participants.^{Footnote 32} The impact of these types of training initiatives included skill development among frontline service providers, a better idea of how to apply cultural understanding in a health care delivery context while also acknowledging the impacts of discrimination on the provision of services. Sessions also improved knowledge of First Nations worldviews, and different practices across the various First Nation and Inuit cultural groups including the role of language and history. Cultural awareness activities were seen to have impacted health service administration teams' commitment to culturally sensitive care in particular cases, thereby starting conversations which have the ability to bring about organizational change. In the case of Braiding First Nations' Culture into Vitalité and Horizon's Health Services project, a greater degree of cultural awareness resulting from their training sessions, led to increased access to cultural care by implementing policies which allow for smudging ceremonies in certain hospitals.^{Footnote 33} Some of the project leads who completed the survey indicated that their projects had contributed to the integration of Western and traditional healing practices. Improvements to the cultural responsiveness of health services were highlighted by interviewed project coordinators as a contributor to increasing community members' willingness to access services.    

The valuing of cultural competency across HSIF projects was clear. In the case of a few projects which embarked on integration initiatives in other areas, at the outset of their project they soon realized they were lacking a certain degree of cultural competency. As a result, they reshaped their project to provide cultural awareness/sensitivity training in order to ground future health service integration work within the appropriate cultural context.

Mental health training activities ranged from 'train the trainer' sessions, orientation to mental health assessment tools, and general training sessions relating to mental health, suicide, and substance abuse. Train the trainer programs lend an additional capacity building element by developing skilled trainers who can visit communities and engage in knowledge transfer activities, thereby empowering communities to be able to provide community-based responses in times of need, as opposed to relying on the availability of health service professionals often located outside of communities.^{Footnote 34} General training sessions and forums on mental health and/or suicide response were perceived by participants as allowing for open dialogue on topics that can at times be challenging to discuss and as a result, in some cases, were hindering effective response to mental health crises. For example, in the case of People's Strength in Health and Wellness project, their evaluation report stated that the awareness sessions and suicide intervention skills training sessions have fostered a greater awareness among community members of how to respond as opposed to 'freezing when the word suicide came up'. ^{Footnote 35}

Some projects included training in the areas of governance, developing service agreements/MOUs, as well as guidance in the areas of community-based planning and program development. Beyond knowledge gains, few projects outlined the tangible outputs of such activities but the impact was best described as enabling and strengthening efforts at a community level. This impact was supported by almost half of the FNIHB regional staff interviewees who felt that capacity building project activities have strengthened First Nation and Inuit community members' skills which would positively contribute to directing their own health services.

Resource/Tool Development

In most cases, resource materials were developed for training sessions, which could then be distributed to a wider audience thereby contributing to greater knowledge sharing while also producing a tangible deliverable enhancing the likelihood of a certain element of project sustainability if continually shared as team membership evolves.

Sixteen projects developed some form of implementation guide/manual (e.g., protocols, health model, standard operating procedures manual) or screening tools and assessment forms. An additional three projects were unable to complete their objectives in this area, while one other project was in the process of developing a guide. In certain cases this contributed to standardization of screening tools and reporting templates. As highlighted in the literature review, harmonizing practices among partners/programs often enables the following stage on the integration continuum (coordinating services) to occur. As articulated in one project report, partners often have similar health care goals and priorities but they often lack the framework and protocols to work together.

Thirty-seven percent of projects (25) engaged in environmental scans but only five projects sought to release a resource guide or online tool outlining available services for clients and other service providers. Almost a quarter of interviewed project coordinators felt an increased awareness of services among community members as a result of HSIF projects contributed to increasing their ability to access appropriate services. The distribution of online or hardcopy resource guides expands the knowledge sharing made possible by environmental scans and therefore presents a future opportunity for HSIF projects to broaden their reach. The impact of distributing this type of service listing to other service providers was illustrated in the case of two projects which distributed manuals to the Ontario Provincial Police. These guides provide authorities with more appropriate resources to connect community members with when picked up while under the influence of alcohol, drugs and/or other substances, providing a point of intersection for a more responsive approach to individuals with addictions coming in contact with the system. One project coordinator indicated, as a result of high staff turnover among medical professionals in First Nation and Inuit communities, their awareness of appropriate and available resources is often lacking, and the existence of a guide is an invaluable resource in familiarizing health workers new to the area. Although, it is important to note that constant staff turnover created challenges impacting the sustainability of tools and resource manuals if staff are not made aware of their existence.

A few projects engaged in database projects or the gathering of population and health care utilization data. In the projects which did more intensive data collection, it was seen as beneficial to have evidence to support responsive health care planning or to support future applications for funding.

Coordinating Services

Coordinating services can contribute to the seamless access of services, therefore supporting stronger degrees of integration along the health care continuum. This stage refers to the coordination of services and interactions between people and organizations/agencies. This would involve synchronizing certain approaches or transitions between the ways in which the two distinct systems/organizations operate, which can also facilitate the use of complementary health services. In many cases, this could result in establishing automatic coordination or negotiation between the health services at play. HSIF project activities that captured elements of service coordination included: staff navigator and advocacy positions (5/5 completed); the development of protocols, the alignment of services and policies, and/or standardization of processes (14/18 completed); strengthening continuums of care (4/4 completed); and the development of service agreements/MOUs (5/6 completed). 

The five projects dedicated to establishing staff positions serving a coordination or advocacy role in accessing services were all successfully implemented. The importance of these types of projects was supported in both the literature and key informant interviews where it was found that jurisdictional confusion contributed to challenges in patients' access to a continuum of health services. Specifically, half of interviewed project coordinators felt that jurisdictional confusion created barriers to system navigation. The Health Navigator from the Development of a System of Integrated Mental Health and Addictions Services Between First Nations, Aboriginal and Non-Aboriginal Services Project provided at least 32 clients with access to services they were not in contact with prior to the HSIF project.^{Footnote 36} Services were either culturally relevant or the project coordinator often worked with providers to enhance their cultural sensitivity.

Coordinator roles can often contribute to greater access of services due to a stronger degree of coordination between hospital and community-based services as well as an enhanced continuum of care by arranging care and/or appropriate services for clients upon discharge. Naskapi Nation of Kawawachikamach's HSIF project's Health Coordinator assisted 20 community members in accessing treatment facilities in 2014/15, by increasing their awareness and access to subsidized transportation. Other coordinator roles that materialized from HSIF projects focused on advocacy work for parenting rights and/or accessing appropriate services, coordinating appropriate services (e.g., cultural) and/or putting the proper supports in place to have meaningful health service appointments (e.g., translation services).

There were 18 projects seeking to establish a coordination of services by often focusing on the development of protocols as a means to create stronger linkages to complementary services (e.g., referral protocols) or establishing a certain degree of harmonization in administrative and operational practices (e.g., information sharing). Six of the eight projects focused on developing protocols were able to complete their objectives. Four projects also focused on aligning policies or services with those of health authorities and provinces, and all successfully completed activities related to these tasks. Four of the six projects working on standardization of care (e.g., pathways for navigating the health care system) were able to complete their planned deliverables. Among the projects aligning policies and services, their reporting demonstrated both improvements to the quality of, and access to, health services by enhancing a continuum of care through easing the transition between services through referral, operational and service provision protocols.

Continuums of care were a direct focus of four projects. Project objectives were all achieved, mainly through the development of service level agreements, liaising between complementary organizations, and improving service providers' capacity to provide culturally sensitive services. Projects of this nature call for a great degree of partnership engagement across various services that are often beyond the realm of health services. These projects in particular illustrated the importance of considering determinants of health and other points of entry into systems which have potential to link with health service providers. This was evident in four projects of partnering with schools, and two other projects that mentioned the importance of addressing housing in order to achieve their long-term outcomes. Another project expanded their scope in the continued development of MOUs, service agreements, capacity building and collaboration to include the areas of housing, education, social assistance, and employment programs servicing First Nations. The importance of acknowledging the interconnectedness of service areas beyond the health field is important in supporting health integration, which can have a meaningful impact on health outcomes. Multiple key informants interviewed for the People's Strength in Health and Wellness project evaluation echoed similar insights to those of one staff member who stated, "it is not realistic to focus on service integration when these basic necessities are missing for so many community members. Dealing with the negative consequences of poverty and injustices, such as the residential school system, at the community level keeps workers in a constant state of reacting to the immediate needs of the community".^{Footnote 37}

Partnering in the Provision of Services

Partnering in the provision of services can enhance capacity of individual service providers and in some cases strengthen the spectrum of health services available to clients. This type of integration could include adopting jointly supported common practices and structures to facilitate the smooth provision of health services. The literature review suggests organizations may even assess duplications across the integrated services and reallocate roles or practices accordingly in order to increase efficiencies.

The three larger scale service delivery model projects had limited implementation. In two of the three cases, there was limited implementation due to extenuating circumstances such as operating with less funds than anticipated or restrictions based on current service agreements that had not yet terminated. The model that was not implemented felt their activities have prepared them for future implementation and partner interest has since contributed to an expanded scope to include multi-sectoral services (outside of health care) in integration planning. 

Community-based health service teams (four) were all implemented and involved either bringing together resources from different fields for crisis response, improved continuums of care, or uniting health care service providers for screening initiatives. Also, other forms of programming which partnered with schools (three) were implemented with strong degrees of support from school staff. In the case of the First Nation Community-Based Screening to Improve Kidney Health and Prevent Dialysis project, the mobile mass screening initiative visited five remote communities and seven communities accessible by road to conduct kidney health screenings. The project's evaluation report indicates, "a mean screening rate of 21% of all community members eligible (aged 10-80) was achieved".^{Footnote 38}  The impact of these assessments was key in identifying community members in need of additional care, as over 25% of individuals screened through the project had some level of kidney disease. Six out of nine of the projects which included service agreements, were able to complete these planned activities. The two projects with cost-shared positions between health authorities/health departments/local health services were successful in completing their tasks. 

The three projects focusing on tele-health and the two projects seeking to implement electronic medical record databases all exhibited a positive progression towards their objectives. The intended medical record databases were developed, although of the two projects seeking to link to other stakeholders' data, one project put an agreement in place, while the other project was still in the process of developing data sharing agreements. Tele-health arrangements exhibited different rates of progress towards implementation. One HSIF project was able to implement the actual client sessions within the lifespan of the project, while another completed the development of a proposal for the service, and the third project had 2/3 of their identified tele-health sites ready for implementation. The feedback from one tele-health project evaluation report found the service to be effective without compromising the quality, in that eight out of nine clients felt the service was equivalent or better than the care they received during their in-person sessions.

Transferring/Merging/Amalgamating Services

This stage directly contributes to long-term policy objectives of a transfer of health services to First Nations and Inuit. There were two key theme areas for projects falling under this particular range of the continuum: governance projects and multi-service model projects.

The four governance projects exhibited varying degrees of progress towards their stated objectives. Two projects resulted in partnerships between First Nation and provincial/federal governments. The third project did not receive all the funding originally pledged by their partner and therefore was only able to accomplish the theoretical components of their project. The fourth project was unable to have meaningful negotiations at the provincial or federal levels, and their project resulted in a draft framework which key informants noted requires a greater degree of precision. Anecdotally, some project coordinators felt that HSIF projects positively contributed to First Nations and Inuit governance by allowing project participants to engage with national and provincial/territorial advisory committees on integration plans.  These types of tripartite agreements are seen as mechanisms to ensure a major role for First Nations/Inuit in the design and delivery of health care services while also benefitting from the coordination of provincial/territorial health services. Tripartite agreements can often set the stage for larger governance conversations, and in turn may lead to a greater degree of devolution of health services to First Nations and Inuit. Governance-related objectives take time to demonstrate impacts beyond the contribution to relationship development, and as such it is too early to discuss outcomes stemming from HSIF projects with governance-level objectives. As the renewed iteration of HSIF refocuses its emphasis on service devolution arrangements and tripartite agreements, the impact of these partnerships will need to be understood in order to truly capture the effectiveness of health service integration efforts across varying levels of government. This point will be expanded upon in the performance measurement section of this report.

There were four intentional multi-service model projects of which two were implemented, as well as one additional project that unexpectedly evolved into a proposed community clinic. The two models that were implemented demonstrated direct impacts to an increase in First Nations access of health services. In both cases, the development of new service models allowed for a greater ability to accommodate clients as a result of enhanced capacity due to the expansion of the service base (e.g., additional physician, more units to accommodate clients). One clinic model was able to double the number of patients seen per day by the physician, while also contributing to the quality of health services available by providing physician services as one of the ten complementary services offered at the health centre coordinated through the HSIF project. One Manitoba project unexpectedly evolved into working on the creating a community health clinic which led to the completion of a business plan approved by the board members representing all the First Nations in the area, and at the time of reporting the local health authority had been mandated to proceed with the implementation of the plan. Project reporting indicates that resource challenges related to capital investments and accessing shared staffing resources impeded the success of the two remaining projects within the final stage of the continuum.

Challenges

There were commonalities in challenges faced by projects spanning the integration continuum, including:

• Staff turnover: Staff turnover was common among health service professionals and in some cases representatives from partnering entities. This limited substantial capacity building in the areas of knowledge development (e.g., existing health services, cultural sensitivity) and impacted the continuity of the discussions at steering committee levels or the momentum of project activities at the project coordinator level.
• Ensuring the right partners are at the table: A few projects noted the impact of ensuring the decision-makers from partnering organizations are attending the project meetings. This was noted as a key to success in the cases where it did occur.   
• Engagement of First Nations: Although projects were found to have a strong degree of First Nations and Inuit involvement, in some cases the need for continued First Nations engagement beyond the initial consultation phase was identified. Also, the need for partners to build relationships with the First Nation community rather than a strong relationship with a First Nation individual was seen as necessary in order to be truly representative of community interests.

Overall, a greater degree of integration of health services for Inuit and First Nations people was achieved through HSIF projects. Furthermore, the majority of project coordinator key informants confirmed that projects materialized in the way they were originally presented in proposals.

4.4.3 To what extent have the long-term outcomes been achieved?

Long-term Outcome #1:
Increase in independent health services integration arrangements for First Nations and Inuit.

The development of new health service integration arrangements outside of HSIF is beyond the lifecycle of the projects and the timeframe for this evaluation. However, current projects highlighted early indications of plans for such arrangements, as well as key factors in sustaining integration arrangements.

This particular outcome of generating new health service integration arrangements is intended to take place beyond the life cycle of HSIF projects. The intention is that while gaining experience through HSIF projects, participants will have increased capacity to collaborate as well as plan and implement integration projects, and will therefore be able to do so independently in the future. In select cases, there are indications that these types of arrangements are likely to materialize based on information stated in final progress reports:

• The Clinique Minowe project included the implementation of the Minowe model in two Quebec Friendship Centres. The project evaluation served as a catalyst for collaboration in other program sectors to discuss further implementation of the model.^{Footnote 39}
• A combination of the Diabetes Integration Project/Kidney Health Project's kidney mobile health screenings are in the process of being replicated through the NorWest Mobile Diabetes Screening and Intervention Project with the support of a $200,000 grant from a private foundation. ^{Footnote 40}
• A steering committee member from the project Expanding Our Circle of Mental Wellness was also involved in a provincial pilot project for an integrated service delivery model of care for youth at risk, which created networking opportunities across initiatives and with Chiefs from various First Nations. The project's evaluation report states, "this project has allowed the partners to not only focus on the project's outcomes but also think about how integration of services could work in other areas, such as youth services, and how to involve First Nations meaningfully".^{Footnote 41}

This expected outcome not only focuses on the initiation of independent integration arrangements, but highlights the continued sustainability of these new arrangements without additional federal financial support. Although it is too early to assess whether new integration arrangements might materialize beyond the particular projects which ended March 31, 2015, the evaluation highlighted key factors in sustaining integration arrangements:

Clear planning at the outset of a project

Ensuring particular factors (e.g., stakeholder readiness, clearly defined roles and responsibilities, and planning for sustainability) are accounted for at the beginning of a project can have long-term benefits to the sustainability of integration initiatives. Stakeholder readiness at the outset of the project is invaluable. Readiness could include having the available resources in place, community/leadership support and understanding of the initiative, as well as having a mutual understanding among partners regarding the project scope and intended outcomes. Interviews with project coordinators and provincial/territorial representatives indicated that in some cases project confusion and the ease of progressing was impacted by stakeholders' capacity to begin planned work. It was also evident that clearly defined roles and responsibilities alleviated stakeholder confusion throughout the life of the project. Best practices outlined by provincial/territorial stakeholders and project coordinators included establishing terms of reference or having a clear description of protocols and commitments in the workplan. Continued communication through regular meetings, steering committee oversight, and the involvement of community stakeholders was also seen as contributing to a clearer understanding of roles and responsibilities. Finally, planning for sustainability at the beginning of the project and adapting those plans as the project evolves is pertinent to effective long-term planning. Examples of pre-planned sustainability courses of action included: formal agreements with sustainability provisions, sustainability sub-committees developed during the project implementation phase, as well as pro-actively planning for the supports and capacity building opportunities required at the completion of the project in order to sustain the work of the initiative.

Alignment of Partner Priorities

Partnerships developed through HSIF naturally created a greater awareness of stakeholder priorities. Understanding partners' priorities often showcased similar areas of focus and led to a greater connection between partners due to the alignment of their organizational/community goals. This compatibility of priorities more easily highlighted the opportunities and benefits for continued partnership through integration initiatives. One particular sustainability plan outlined the potential for access to an extended network of stakeholders with consistent priorities through their partners, thus continuing the progress achieved through their HSIF project.

Integrating project components into already established structures

Having been identified in a review of the HII and AHTF as well as the current HSIF evaluation, capacity building tends to be an area of concern for First Nation and Inuit projects. The ability to integrate into a pre-existing structure provides a strong knowledge base and system that can support implementation in the early stages of a project. In the case of the Nunavut Community Wellness Plan (NCWP), the development of community wellness plans for three regions was facilitated by leveraging the existing community development work through Nunavut's Public Health Strategy. Knowledge sharing greatly contributed to capacity building, and subsequent training opportunities shed light on successful approaches to project planning including project management, governance, and community engagement. Through the NCWP, planning was able to be guided by the larger territorial public health strategy as a point of reference, and was able to be expanded to meet specific regional needs.

Ongoing development of relationships and capacity building resources

Multiple lines of evidence confirmed that the implementation of health services integration initiatives is a complex and lengthy process. A review of the three iterations of Health Canada's First Nations and Inuit health integration initiatives identified a common factor for encouraging sustainability: empowering project leads as mentors and serving as a meaningful resource in knowledge transfer activities.^{Footnote 42} Some project coordinators also highlighted senior management buy-in as a substantial factor in the continued support of integration initiatives. Continued involvement of partnerships through formal agreements helped ensure ongoing opportunities with capacity growth. In contrast, less formal agreements created a greater risk to the sustainability of partnerships as the level of commitment could wane over time or in the face of high staff turnover in the partnering organizations. In any case, incorporating capacity building and skill development knowledge into training resources/tools greatly contributes to the longevity of accessing and applying best practices.

Sustainability of current projects was self-assessed by project coordinators through project progress reporting. Among the 64 projects with reports that listed sustainability ratings, all but one project cited likeliness for some degree of sustainability, with 'very likely' representing the most commonly selected category (50%). Although FNIHB staff expressed concern with the feasibility of sustainability plans without continued funding, the vast majority of regional staff acknowledged that plans did include elements that could be independently maintained without additional federal funds. Interviewed project coordinators at times listed the need for continued funding to acquire new staff or resources in order to continue the work of their HSIF project. Lessons Learned from Ten Years of Integration found that "an incubation period with dedicated resources is necessary; once partnerships are established and integrated systems are institutionalized funding levels can be reduced or eliminated."^{Footnote 43}  Common weaknesses in sustainability plans are consistent with the lessons learned listed above, most notably a lack of formal commitment by partners to the maintenance of project momentum and/or success. 

Long-term outcome #2:
Improved access to quality health services for First Nations and Inuit.

Although it is too early to determine the full extent of HSIF's impact on the access and quality of health services serving First Nations and Inuit, this evaluation has demonstrated contributions in both areas.

Access

Multi-jurisdictional confusion has often served as a barrier for Inuit and First Nations people needing to access health services. HSIF projects were effective in cultivating multi-jurisdictional collaboration, as well as served to increase the number of communities working together to address health service integration. The majority of key informants from all three groups (FNIHB staff, provincial/territorial health staff, and project coordinators) reported that HSIF projects had contributed to improving First Nations and Inuit access to health services. Project status and evaluation reports demonstrated examples of improved access which were most commonly seen in the following areas:

Strengthening a continuum of care

Projects with objectives of strengthening continuums of care typically worked to address challenges in continuity of care (e.g., developing or enhancing pre and post treatment supports). In many cases, projects facilitated liaising with health care professionals from various fields (e.g., psychiatrists, social workers) contributing to a more comprehensive health care experience comprised of access to multiple types of health care professionals. An Interdisciplinary and Integrated Culturally Safe Diabetes Care Clinic Health System for First Nationsproject in particular reported an additional 13 services offered in First Nation communities as a direct result of the one-day multi-service based clinics held in the community.^{Footnote 44}

Improved service coordination

Improved coordination of services contributed to improved access in many cases. Improved coordination was facilitated through a variety of project activities including the creation of dedicated positions to help clients navigate the health care system and arrange the appropriate health services. Interviews with project coordinators found that the majority of projects improved access to health services by increasing awareness of services. Improvements to referral processes and fostering greater alignment across services were also contributing factors to improving First Nations and Inuit access to health services. Examples of improved access in these areas include:

• Integrated Service Delivery Model for First Nations receiving Mental Health and Addictions Services in the Battleford Tribal Council Area project - The project reported an increased caseload of 40% over the previous year as a result of the improved intake and referral work completed through their HSIF project.^{Footnote 45}
• Minweyaanigoziwin Health Service Integration Project - The project reported 70 clients accessing new services with 17 referrals still ongoing as a result of the mental health and addictions integration work completed through HSIF which enabled the accessing of federal, provincial and LHIN funded services.^{Footnote 46}

Improving the ease of access

Projects which established tele-health services or brought screening initiatives/multi-service clinic fairs to communities, improved the ease of accessing services locally. This was particularly evident in the case of the First Nations Community-Based Screenings to Improve Kidney Health and Dialysis project which conducted kidney health screenings for over 1,900 community members from four remote and seven road-accessible communities. On average, 1/5 of community members between the ages of 10-80 years old received screenings, resulting in the identification of kidney disease among 25% of participants. Referral processes were in place, thereby contributing to a continuum of care with available support services. Intermediate and higher-risk clients were referred directly to nephrologists at the Manitoba Renal Program/Nephrology Child Health Program for follow-up treatment. All low risk clients were referred to primary care services available either in their community or within close proximity.^{Footnote 47}

Quality

Interviewed HSIF project coordinators most commonly cited the improvement in quality and coordination of services as the greatest impact of their projects. This evaluation identified the following factors as contributing to quality health services: cultural relevance, community-based/client-centered focus or the use of a holistic approach.^{Footnote 48}

Cultural relevance

Projects with components impacting cultural awareness/sensitivity, as indicated earlier, had a strong degree of completed objectives. These objectives most commonly related to developing and implementing cultural training for service providers. Project evaluation reports confirmed that participants found training to be informative and beneficial. Half of provincial/territorial key informants interviewed were involved in projects that increased the incorporation of traditional cultural practices into First Nations and Inuit health services or contributed to increasing health service providers' awareness and capacity to provide culturally appropriate services. The incorporation of traditional cultural practices constitutes a holistic factor contributing to quality health services.

Employing a client-centered/community needs-based approach

Community-based services contribute to both quality and access by offering a service in an environment that is more likely to understand the everyday realities/culture of the client (quality), and improving the ease of access by not requiring timely and often costly travel to be able to meet with service providers. As mentioned earlier, project coordinator key informants identified cultural responsiveness as a contributor to increasing community members' willingness to access services.

Context impacting the achievement of improved access to quality health services

The majority of projects had proposals that would seek to either improve access to, or quality of health services for First Nations and Inuit, but individual projects did not necessarily address both components of the long-term outcome. For example, more than half of the FNIHB regional key informants indicated improved quality was not an objective of all projects in their region.^{Footnote 49} In the case of access, it was confirmed by a small portion of project coordinator key informants that their projects were not designed with this intent in mind. Furthermore, there were a few examples of projects that were not designed to directly improve either access or the quality of health services, but rather focused on healthy behaviour initiatives.

As discussed in the intermediate outcome section, all stages of the integration continuum have value but their direct link to the long-term objectives of the program may need to be considered. Projects based on research-driven activities may be informative, but without development and implementation of plans to create linkages with other stakeholders, the exercise may set the stage for future activities aimed at increasing access and/or quality, but in and of itself it does not directly contribute to the program's long-term outcome. This finding also applies to projects with a stronger focus on improving health outcomes. For example, health promotion activities (e.g., healthy eating) that are informative but are either not a part of, or do not link to, an actual health service ultimately cannot feed into HSIF's final outcome of improving access to a quality health service. There is a clearly defined need for this work but in the end, the question becomes whether the projects are synonymous with the raison d'être of the program. A prioritization of certain types of projects could facilitate a more tangible progression towards achieving those outcomes, and as a result have the potential to impact future health outcomes as a by-product of HSIF's integration work.

4.5 Performance: Issue #5- Demonstration of Efficiency and Economy

Earlier years of HSIF program expenditures indicated funding surpluses, while later years demonstrated greater alignment between allocated funding and actual expenditures. The presence of in-kind contributions of HSIF project partners greatly contributed to project economy. The leveraging of technology (e.g., tele-health arrangements, health databases, etc.) contributed to efficiencies in the integration of First Nations and Inuit health services.  The program has demonstrated commitment to performance measurement by having a Performance Measurement Strategy in place. Project-level reporting could be strengthened to inform outcomes resulting from HSIF project activities by replacing output-oriented indicators with clear indicators of an outcome-oriented nature.  

Observations on Efficiency

Projects were able to exercise efficiencies through the leveraging of technology, and the use of project planning tools and resources.

Leveraging of Technology

The leveraging of technology through HSIF projects led to efficiencies in both the service planning and delivery domains. A few projects developed medical databases such as First Nation client registries, integrated health records, and population health data through either the digital transcription or coding of health records, creating the electronic infrastructure, and/or embarking on data sharing agreements. Such endeavours contributed to a better understanding of health needs among First Nation and Inuit populations. As a result, this data was able to inform health planning, thereby focusing strategic efforts on the most relevant areas of care for the population.

The tracking of First Nation and Inuit health data enhanced integration by fostering greater data exchange among health agencies. In some cases, databases developed as the project deliverable connected with regional health databases at either the health authority or provincial level, as well as with not-for-profit health organizations. In the case of Kenora Chiefs Advisory First Nation Client Registry (KCA FNCR), plans are to link their First Nation health database with Cancer Care Ontario in order to identify clients at risk of developing cancer. Clients would then be linked to resources as early as possible.^{Footnote 50}

The efficiencies created through the health databases developed in HSIF projects continued to impact on-site health service delivery. The ability to access the databases permitted medical staff to work more efficiently during community visits since they were able to access clients' health data immediately as opposed to sorting through various sources to understand patient history prior to being able to provide treatment.  

Health service efficiencies created through HSIF projects extended beyond systemic planning, and influenced the provision of health services as well. Some projects were able to develop tele-health arrangements, allowing for health care service delivery at a distance, thereby eliminating delays experienced by communities dependent on the travel schedules of medical staff from city centers. This was evident in the project reporting for Developing and Implementing a Telepsychiatry Collaborative Care Protocol for Miawpukek First Nation which indicated that through telepsychiatry arrangements, less travel will be required for psychiatrists and the patients, resulting in greater timeliness in receiving care as well as enhancing the quality and continuity of mental health care provided.  Through such developed resources as toll-free counselling lines, tele-health consultations with psychiatrists and online service maps to better highlight the availability of nearby resources, First Nation and Inuit clients were able to access more consistent care in a timelier manner. Furthermore, greater access to a range of services helps to strengthen the continuum of care available to First Nation and Inuit clients. In the case of the Give Us Wings project, they are in the process of exploring the use of a video link technology to connect First Nation inpatients residing in detox facilities off-reserve, with community workers, to facilitate the post-discharge aftercare. Creating efficiencies in health service planning and delivery contributes to HSIF's ultimate outcome of improving access to, and the quality of health services for First Nations and Inuit. In the survey, project coordinators found that projects had the ability to contribute to a safer, faster and more efficient health delivery system.

Project Management

At the project management level,interviews with project coordinators and provincial/territorial health staff identified the following strategies as being effective in keeping projects on track:

• Regular communication between partners;
• Project tracking tools;
• The use of web-based tools to share information and increase awareness; and
• Conducting research and service inventories at a regional level.

Some FNIHB regional staff noted that efficiency at times was negatively impacted in projects where there was a greater number of partners to manage. This often resulted in longer periods of time required to produce and finalize deliverables.

Observations on Economy

The program financial review includes data from 2011-12 to 2014-15.^{Footnote 51}  Although BC projects were out of scope for the evaluation, unfortunately the corresponding financial information was not able to be removed from the planned spending data. Since BC expenditures alone account for approximately $2.5 million, consequently, an amount of that nature still incorporated in the budgeted totals but removed from the expenditures, would create a certain degree of inflation in the variance data. As a result, for the purpose of this analysis, BC financial data has been included in both the planned spending and expenditure categories.

During the last four years of HSIF^{Footnote 52}, approximately 97% of budgeted funds for that time period were spent. The degree of planned budget spent ranged from 76% in 2011-12 and 94% in 2012-13, to the higher rates of 105% in 2013-14 and 107% in 2014-15. The largest area of unspent funds ($2.8 million) was connected to operations and maintenance (O&M). According to program officials, the 2011-12 O&M surplus was in part due to incorrect HSIF expenditure coding, as well as having a larger O&M budget than was required. As a result, the O&M budget was significantly reduced in the subsequent years of the program. Unspent funds in this area over 2012-13 and 2013-14 are reflective of direction given to restrict travel as a means to contain costs in anticipation of Department-wide funding reallocations. Lapsed grants and contribution (G&C) funding in 2011-12 ($1.1 million) occurred in particular regions that were slower to develop and acquire approval for their trilateral integration plans, which was required in order to receive their project funding. While planned and actual salaries were equivalent during 2011-12 and 2012-13, actual salary expenditures for the last two years were well above the planned amounts (2013-14: 143%; 2014-15: 241%). Salary expenditure reporting, as it relates to corporate costs, was impacted in 2014-15 with the introduction of proportionally allocating management salaries across all FNIHB programs accounting for $416,848 of the total expenditures within that category.^{Footnote 53} A decrease in planned salary spending starting in 2012-13 occurred as a result of budget cuts. At the time of the budget reduction, information provided in corporate documents identified impending salary deficits for 2013-14 (-$454,504) and 2014-15 (-$474,504), which is reflected in the chart below. As identified in the same documents, the salary deficits were a product of a revised salary budget which was lower than the amount of FTEs required and used to administer the program at headquarters and in the regions.

Service mapping and needs assessment analyses

Projects engaging in service mapping and needs assessment analyses became more aware of existing services, thereby extending their resource network and preventing the future duplication of services. More than one-third of projects listed this type of exercise as a project activity. This extended resource network which enables the spreading of resources across services provided through different funding sources was listed by FNIHB regional staff as one of the benefits of having multiple jurisdictions involved in health service delivery. Half of the provincial and territorial key informants were in agreement that multiple jurisdictions’ involvement in health service provision led to an increase in funding available for services. This finding can be understood in the context that the majority of FNIHB and provincial/territorial health staff stated that HSIF projects contributed to increased collaboration between multiple communities.

In-kind contributions

In-kind contributions greatly contributed to project cost-effectiveness generating a minimum estimated total value of $5.7 million. This is most likely a modest estimate considering not all projects listed values for their acquired in-kind resources. Of the 49 projects citing the receipt of an in-kind contribution, only 29 listed specific details, and 15 had no dollar value linked to the resources gained. In-kind contributions seen in HSIF projects most commonly equated to a range of 25-50% in supplemental funding for project activities. The most common form of in-kind contributions listed was staff time from partnering organizations and/or staff resources, office/meeting space and supplies, as well as training and knowledge services including support for evaluations, communication materials, technology support, and general capacity building.

Cost-drivers

Hindrances to cost containment of integration initiatives were linked to three main areas: geography, community engagement, and coordination needed for a larger number of partnerships in HSIF projects. The dispersed geography in which many projects operated typically meant significant travel budgets were required in order to ensure a sufficient degree of community engagement was captured. Although a comprehensive community engagement piece may contribute to greater project costs, the responsiveness of project planning is dependent on the incorporation of community perspectives. In fact, one-third of interviewed provincial/territorial health staff agreed that community engagement led to project results which were responsive to community needs. Presumably greater degrees of First Nations and Inuit involvement in project design and delivery would include a certain degree of community engagement and as indicated in the majority of project coordinator interviews, projects contributed to greater First Nations and Inuit control and ownership of health services. The coordination of a greater number of partners and jurisdictions involved led to a greater amount of staff time required in order to build and solidify relationships, which naturally has cost implications of its own although evaluation data could not confirm specific associated values.

Observations on the Adequacy and Use of Performance Measurement Data

The program has demonstrated a commitment to performance measurement. HSIF has a Performance Measurement Strategy in place. Key informant interviews with project coordinators indicated that project performance measurement data was used to inform project priorities and report on the project’s progress towards achieving objectives related to those areas. Tracking also included assessing the uptake and usage of project initiatives, and identifying areas posing challenges to health service delivery, as well as potential improvements. Finally, data was used for briefing purposes which included preparing reports and evaluations, as well as informing management or stakeholders on project progress.

The assessment of HSIF’s project performance measurement data is based on the review of progress reporting throughout the duration of the project at six month intervals, and the final project evaluation reports. Varying degrees in the quality and extent of performance measurement across projects hindered the assessment of project impacts. Through this review of project reporting, the following areas of performance measurement have been identified as areas to be strengthened:

Output-driven indicators

For a program like HSIF that exhibits a certain fluidity in order to achieve varying forms of integration, it is important to have outcome-driven indicators since it is not necessarily possible to create standardized indicators to report on across projects that would truly tell the story of change and the shaping of health service planning and provision. At this point in time, the common output-driven indicators used among projects to capture the ideas surrounding the strength of relationships have included the number of emails sent, phone calls made, and meetings scheduled, with no indication of what contribution this made to integration activities. Interviews conducted with project coordinators found that more than half of project coordinators felt there were no gaps in their performance data. Based on a review of progress reports, capacity building around performance measurement for project coordinators could introduce the value and approaches to collecting meaningful data to tell the story of project efforts which could inform project planning, implementation and assessment of outcomes.

Lack of specificity

Some project indicators involve a certain degree of subjectivity in their reporting. For example, indicators with qualifying terms such as “strong support” are used with no indication of what would constitute the different degrees of support for reporting purposes. This creates challenges in the consistency of reporting, and without a standardized understanding of how to report on indicators, over time that flexibility within the indicator will not necessarily capture the progress achieved across the lifespan of the project.  

Need for baseline data

Indicators that seek an improved status in a particular area have often not been accompanied by baseline data. As a result, the true degree of achievement will not be able to be measured. This speaks to the importance of preparing for measurement activities at the outset of the project in case opportunities exist to establish baselines relevant to the project’s work.

Program and project-specific indicators

Reporting at the project-level has in many cases linked project outcomes to the HSIF logic model. This approach supports informed program reporting. In the cases where projects are linking to the broader program outcomes, some projects continue to create additional indicators to complement the specifics of their project thereby giving more detailed insight into the activities completed with HSIF funding, and where applicable identifying some outcome data. This approach should be encouraged across projects.

With HSIF’s intended shift to eliminate the outcome reporting section on their progress reports and no longer conduct project-specific evaluation reports, it is imperative for projects to develop meaningful and reportable performance indicators in order to provide insights into project outcomes. Naturally, there are limitations to performance indicator development based on the information available, but as the program is in the process of revising their reporting template, opportunity exists to shape data collection to focus on impacts of project outputs (e.g., work accomplished through tripartite arrangements rather than simply documenting the arrangement’s existence). Without outcome-driven data, future impact and evaluation work to assess program outcomes at large will be hindered.

5.0 Conclusions

5.1 Relevance Conclusions

There is a continued need for a federal role in establishing and ensuring a culturally relevant and responsive health care system for First Nation and Inuit peoples is in place, which bridges the service gaps experienced across jurisdictions. HSIF integration activities can contribute to greater collaboration among stakeholders, providing the opportunity to lessen the existing fragmentation of health services that often causes disjointed health care planning and delivery priorities among stakeholders seeking to serve the same population. Work in the area of First Nations and Inuit health conducted under the HSIF program is consistent with the federal Indian Health Policy (1979), the Health Transfer Policy (1988) as well as the mandate and function of the First Nations and Inuit Health Branch. The HSIF design illustrates a concerted effort to bring multiple stakeholders together across jurisdictions as well as actively involve First Nation and Inuit communities and organizations in health service integration thereby working towards a more cohesive and responsive health service environment for First Nations and Inuit. Therefore, HSIF is consistent with Departmental and FNIHB strategic priorities, including: improve access to health care and increase quality of care; collaborate with provincial/territorial authorities to deliver services and improve the integration of health services; and increase the control of health care service development and delivery by First Nations and Inuit.

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes (Effectiveness)

Project activities contributed to a better understanding of enabling factors and barriers impacting health services integration. The multi-jurisdictional nature of projects led to better collaboration among multiple communities, which was supported by an increased capacity of key partners and stakeholders to collaborate on health services integration through partnership development, planning support, and capacity building activities. The factors discussed above are consistent with health service integration enablers.

HSIF projects were, for the most part, able to achieve their objectives thereby contributing to a greater degree of integration of health services for First Nations and Inuit. Project activities relating to planning, capacity building, and the coordination of health services demonstrated strong rates of achieved objectives. Project activities contributing to partnering in the provision of health services had moderate to strong rates of achieved objectives. Activities feeding into the ultimate integration stage of the transferring/merging/amalgamation of health services demonstrated achieved objectives to a moderate degree. There were a few cases of projects which engaged in beneficial work related to improving health outcomes but in some cases the projects were either not a part of, or did not link to an actual health service which ultimately means the work completed in these projects are not always able to directly impact HSIF’s final outcome of improving access to a quality health service.  

Although it is too early to truly assess the depth of the impact HSIF projects have had on the intended long-term outcomes, positive developments have been identified in the two outcomes at this level (including both attributes of access and quality). Most notably, individual projects have demonstrated improvements to the quality of health services impacted by their projects as well as instances of improved access within their projects’ catchment area. With time, a greater understanding of systemic changes possibly spurred by these integration projects could be explored.

5.2.2 Demonstration of Economy and Efficiency

Earlier years of HSIF exhibited funding surpluses, while later years demonstrated greater alignment between allocated funding and actual expenditures. The leveraging of technology positively contributed to efficiencies in projects’ work surrounding health services integration by facilitating tele-health arrangements and creating health record databases for efficient access to client information. Details relating to project management (e.g., lack of clarity around roles and responsibilities), in some cases, negatively impacted the efficiency with which project activities were carried out. Economy at the project-level was largely facilitated by in-kind contributions.

The program has a Performance Measurement Strategy in place. Project reporting is currently taking place, and opportunity exists to strengthen projects’ individual indicators by focusing on more outcome-driven indicators, ensuring indicators seeking to measure improvements are accompanied by baseline data, and accounting for a certain degree of specificity in indicators for accurate reporting. Project performance measurement indicators should continue to support program-level indicators while also establishing a distinct set of indicators to measure project-specific performance.

6.0 Recommendations

Recommendation 1

Screen proposals with the goal of selecting projects that directly contribute to HSIF’s long-term outcome of focusing on improved access to quality health services.

Potential project proposals should be screened with the intention of completing work with a direct impact on improving access to quality health services. Although that may exclude some projects which focus specifically on improving health outcomes and fall beyond the scope of the intended outcomes of HSIF, improvements to access and quality of health services have the potential to make impacts in those areas as a by-product of completed integration work.

Recommendation 2

Plan for the lifecycle of the project at the workplan stage, and actively consider measures for sustainability as the project progresses.

A common understanding and commitment of project partners at the outset of the project is key to the success of HSIF work. Confusion can arise and impede the timely progress of projects when roles and responsibilities of each participating member are not clear. Therefore, it is recommended that all projects define the roles and responsibilities of each partner at the workplan stage, as well as establish timelines for the achievable milestones for each project. A discussion of tracking project success by identifying intended achievements, and the regularity of data collection which can feed into performance measurement, should also be established before project work is fully underway. Furthermore, ways to facilitate the sustainability of the work accomplished needs to be considered at all phases of the project; and should therefore be incorporated as a distinct field on project reporting templates.

Recommendation 3

Improve project-level performance measurement efforts by focusing on indicators that have greater clarity and are outcome-oriented. Ensure the program logic model is consistent with the recently updated direction of HSIF.  

Project indicators should focus on telling the story of the impacts of work completed. This is made possible through the use of indicators that are less focused on outputs and account more for what resulted from the processes, tools, and other activities implemented through HSIF projects. Identifying meaningful indicators can be a challenge, and it may be beneficial to host performance measurement capacity building sessions with project coordinators and regional staff at the outset of the project to ensure that the best possible data is collected throughout the lifespan of the project. In light of recent shifts towards more devolution focused projects, the program logic model should be reconsidered to account for long-term outcomes that are consistent with this new approach, thereby ensuring program activities continue to contribute to overall goals, and that future evaluations are assessing the proper criteria.

Appendix 1 - References

• Battle River Treaty 6 Health Centre Inc. (2014). 2013-14 HSIF Final Progress Report: An Integrated Service Delivery Model for First Nations Receiving Mental Health and Addictions Services in the Battlefords and 6 First Nation Communities.

• Diabetes Integration Project Inc. (2014). 2014-15 HSIF Interim Progress Report: First Nation Community Based Screening to Improve Kidney Health and Prevention Dialysis.

• Diabetes Integration Project, Manitoba Renal Project, and Winnipeg Regional Health Authority. (2015). FINISHED Final Report 2015.

• Government of Canada. (2014). Budget 2015.

• Government of Canada. (1985). Canada Health Act

• Government of Canada. (2013). Economic Action Plan 2013

• Government of Canada. Office of the Prime Minister. (2015). Minister of Health Mandate Letter

• Government of Canada. (2013). Speech from the Throne.

• Government of Manitoba. (2013). Aboriginal and Northern Affairs: Annual Report, 2012-2013. Winnipeg: MB.

• Health Canada. (2013). 2013-14 Departmental Performance Report.

• Health Canada (2014). 2014-15 Departmental Performance Report.

• Health Canada. First Nations and Inuit Health Branch. (2010). HSIF Implementation Guide- A toolkit for Integration Projects 2010/11-2014/15.

• Health Canada. (2015). Renewal of Community-Based Aboriginal Health Promotion Programming.

• Health Canada, First Nations Inuit Health Branch. Strategic Policy, Planning & Information Directorate. (2014). Lessons Learned from Ten Years of Integration: Supporting and Building Partnerships in First Nations and Inuit Health.

• Health Canada. (2011). First Nations and Inuit Health Branch (FNIHB) Systems Integration- Health Services Integration Fund (HSIF) Evaluation Framework.

• Health Services Integration Fund Ontario Regional Meeting. March 11, 2015. Ottawa, Canada.
  Inuvialuit Regional Corporation. (2014). 2013-14 HSIF Final Report: RE: Regional Advocacy/ Coordination Program.

• Kelly, M.D. (2011). Toward a New Era of Policy: Health Care Service Delivery to First Nations. The International Indigenous Policy Journal, volume 2.

• LeBlanc, Denis. (2015). Braiding First Nations’ Culture into Vitalité and Horizon’s Health Services Final Project Evaluation Report.

• Lori Ann Roness Consulting. (2015). ‘Expanding Our Circle of Mental Wellness’ Project Final Evaluation Report.

• National Collaborating Centre for Aboriginal Health (2011). Access to Health Services as a Social Determinant of First Nations, Inuit and Métis Health.

• National Collaborating Centre for Aboriginal Health. (2011). Looking for Aboriginal health in legislation and policies, 1970 to 2008: The policy synthesis project.

• Nilson, C., Jeffery, N., & Hamilton, C. (2010). Understanding Cross-Jurisdictional Partnerships and Decision-Making in Northern Saskatchewan – Final Evaluation Report. Prince Albert, SK: Saskatchewan Population Health Evaluation Research Unit.

• Nipissing First Nation. (2014). 2013-14 Interim HSIF Progress Report: An Interdisciplinary and Integrated Culturally Safe Diabetes Care Clinic Health System for First Nations.

• Picard, Pierre. Groupe de recherche et d’interventions psychosociales en milieu autochtone (GRIPMA). (2015). Rapport d’évaluation du déploiement du modelé de la clinique minowé pour son implantation dance deux villes du Québec desservies par un centre d’amitié autochtone.

• Potter, I. (2010). First Nations Health: Governance and Organization Effectiveness, Integration and First Nation Control.

• Rohan, S. (2003). Opportunities for Cooperative Health Provision in Rural, Remote and Northern Aboriginal Communities. Government Affairs and Public Policy.

• Simon Management Services. Algonquin Nation Programs and Services Secretariat. (2015). Partners for Culturally Adapted Health Care: Health Services Integration Fund Final Evaluation Report 2014-15.

• Smith, Ross and Josee G. Lavoie. (2008). First Nations Health Networks: A Collaborative System Approach to Health Transfer. Health Care Policy, volume 4 (2).

• Sutherland, Mariette and Marion Maar. (2015). FINAL Evaluation Report for the “People’s strength in health and wellness” Mental Wellness Coordination Project.

• Williams Consulting. (2015). Health Services Integration Fund Final Evaluation Report: Minweyaanigoziwin Mental Health and Addictions Service Integration; A Project with Iskatewizaagegan Independent First Nation.

Appendix 2 – Health Services Integration Fund Logic Model

Appendix 3 – Summary of Findings

Rating of Findings
Ratings have been provided to indicate the degree to which each evaluation questions and issues have been addressed.

Relevance Rating Symbols and Significance:
A summary of Relevance ratings is presented in Table 1 below. A description of the Relevance Ratings Symbols and Significance can be found in the Legend.

Performance Rating Symbols and Significance:

A summary of Performance Ratings is presented in Table 2 below. A description of the Performance Ratings Symbols and Significance can be found in the Legend.

Appendix 4 – Summary of Completed Objectives by Continuum Stage

Appendix 5 – Evaluation Description

Evaluation Scope

The scope of the evaluation included an assessment of the relevance and performance of FNIHB’s Health Services Integration Fund from April 1, 2010 to March 31, 2015. Sixty-eight of the 77 HSIF projects’ activities were reviewed to determine their contribution to the program’s immediate, intermediate, and long-term outcomes. BC projects were excluded due to the British Columbia Tripartite Framework Agreement on First Nation Health Governance (October 1, 2013), which will be evaluated through a separate initiative.

Evaluation Issues

The specific evaluation questions used in this evaluation were based on the five core issues prescribed in the Treasury Board of Canada’s Policy on Evaluation (2009). These are noted in the table below. Corresponding to each of the core issues, evaluation questions were tailored to the program and guided the evaluation process.

List of Acronyms

AFN 

Assembly of First Nations

AHTF   

Aboriginal Health Transition Fund

BC

British Columbia

CA 

Contribution Agreement

DG

Director General

EBP

Employee Benefit Plan

FNIHB 

First Nations and Inuit Health Branch

G&C

Grants and Contribution

GOC 

Government of Canada

HII 

Health Integration Initiative

HQ

Headquarters

HSIF

Health Services Integration Fund

ITK 

Inuit Tapiriit Kanatami

LHIN 

Local Health Integration Network

MOU

Memorandum of Understanding

NAO 

National Aboriginal Organizations

O&M

Operations and Maintenance

PTO

Provincial/ Territorial Organizations

PWGSC

Public Works and Government Services Canada