CONSULTATION FOR SUMMATIVE EVALUATION OF UNICEF'S SUPPORTED INTERVENTIONS IN MOTHERS AND CHILDREN HEALTH

CAMEROON – UNICEF COOPERATION PROGRAMME 2018 – 2020

SUMMATIVE EVALUATION OF UNICEF’S SUPPORTED INTERVENTIONS IN MOTHERS AND CHILD HEALTH

TERMS OF REFERENCE

1.   Evaluation Object

UNICEF Cameroun is commissioning a summative evaluation of its health interventions implemented between 2013 and 2017. The conduct of this evaluation is envisaged in the 2018—2020 Costed Evaluation Plan developed as part of the UNICEF Country Planning Document (CPD). The CPD interventions to be evaluated are to be understood as UNICEF’s contribution to the reduction in the country’s neonatal, child and maternal mortality[1]. Though focused on curbing children and maternal deaths in the country, the Programme to be evaluated was also geared towards the strengthening of key components of the national health system.

After the CPD approval by the Executive Board, Cameroon’s Government and UNICEF signed the Country Programme Action Plan (CPAP) in April 3rd 2013 covering the period 2013-2017. The overall goal of this programme was to hasten Country’s progress towards achieving Millennium Development Goals targets. The Country Programme was anchored on the five goals of the then Strategic Plan. Each goal area was assigned three results at outcome level. The Child Survival and Development Programme Component was made-up of three sub-components, namely: Child Health, Child and lactating women Nutrition and Water, Hygiene and Sanitation[2].

As spelled out in UNICEF Theory of Change (annex 1), the Child health sub-component of the Programme was planned as UNICEF’s contribution to the national effort of curbing under-five and maternal mortalities. This sub-component was focused on the attainment of the following results:

(i)      Improving immunization among children and women;

(ii)    Controlling malaria (promotion, prevention and treatment);

(iii)  Improving the integrated management of childhood illnesses in children under-five Promoting maternal, neonatal, infant and reproductive health.

Key results expected assigned to the sub-component are listed below:

(i)      80% coverage rate of key high impact interventions among under-five children;

(ii)    50% coverage of high impact interventions (Tetanus vaccine and use of LLINs in pregnant women) among children in the intervention zones.

The implementation of the 2013 – 2017 country programme (box 1) focused on four priority regions, namely East, Adamawa, North and Far-North. There regions have the lowest level of indicators in almost all the domains of children rights. More than half of the children in the country are found there. All these initiatives were developed and implemented by UNICEF with the participation of the two zonal offices present in the four vulnerable targeted regions (Adamawa, East, Far-North and North).

Box 1. UNICEF Cameroun Health Programme Key Components

Overall, UNICEF-supported health interventions included the following components:

1)      Development and revision of policy, strategic, normative documents, including the Health Sector Strategic document, EPI (Expanded Program of Immunization), Malaria, Reproductive Maternal, Newborn and Child Health (RMNCH), Community Based Interventions, Integrated Management of Child Illnesses, Seasonal Malarial Chemo Prevention, RMNCAH strategic documents;

2)      Support to the country’s Government in developing an RMNCAH investment case;

3)      Strengthening of the Cold Chain and Logistic (CL&L) and equipment for the EPI, MNCH components in terms of Solar refrigerators, central and regional cold rooms, cold boxes, electronic devices for temperature monitoring, refrigerated truck for vaccines, current stabilizers, electric generators, motor bikes, moto-ambulances, vehicles, Emergence Obstetrical and Neonatal Care (EmONC), new-born care, ITC equipment;

4)      Vaccine supply and provision of medicines, especially for refugees and Internally Displaced Populations (IDPs);

5)      Human resource capacity building in Mid Level Management (MLM) course, EmONC, New-born care, Integrated Community Cases Management (ICCM), Cold Chain, Micro planning for immunization, Inter personal communication;

6)      Improvement of data quality on maternal, newborn and child health in collaboration with the health information Unit in the Ministry of Public Health; and

7)      Delivery of community health services (e.g. for the prevention and treatment of pneumonia, malaria and diarrhoea, increased immunisation coverage);

8)      Establishment of governor fora in 08 (eight) regions and one division of the Logone and Chari in the Far North region.

9)      Massive distribution of Long Lasting Insecticide Nets (LLINs) with provision of complementary LLINs to complete the Gaps; The promotion of the use of LLINS; Tetanus diphtheria Vaccine to pregnant women and Women of Child Bearing Age (WCBA); Vitamin A in post-partum; EmONC kits in some health facilities; Bikes-ambulances, blood banks for pregnant women.

These interventions were implemented in a changing and progressively unstable context. Contrary to the time when the programme was designed, emergency situations requiring response occurred.  For instance, the Programme needed to contribute to unplanned strong nationwide polio response action, once many cases of wild polio virus got confirmed in several health districts[3].  As per the response plan put in place in accordance with strategies advocated by the Global Polio Eradication Initiative (GPEI), 22 vaccination campaigns were conducted from the beginning of the program until December 2015.

In addition, three independent external evaluations were conducted in April and September 2014, as well as in April 2015 to assess the effectiveness of the response to stop or restrain the circulation of the virus and its transmission to the Central African sub region[4].  Furthermore, the socio-political crisis in the Central African Republic spilled-over and the number of Boko Haram attacks along the border with the Far North region of Cameroon arose. As a result, three of the country’s regions ended up hosting nearly 300,000 refugees who fled for their lives from the Central African Republic Internally Displaced Populations (IDP) arrived from Nigeria in search of security moved from borders to inlands in the country’s Far North region, as well in the northern regions, attacks perpetrated by Boko Haram activists claimed many lives, damaged basic infrastructures of social services, mainly in health and education. Hundreds of thousands were also forced to abandon their homes and properties in their strive to preserve their lives. A humanitarian response was put in place to meet the needs of all these IDPs.  

In order to respond to the egregious effects of this crisis, several strategic readjustments in UNICEF work were needed given that the then existing system was unable to provide an appropriate and adequate level of services to both refugees, IDP and host communities in the region.

Readjustments concerning program refocus on polio and humanitarian responses (Central African and Nigerian Refugees) were jointly decided by Government and UNICEF-authorized personnel, based on evidence gathered from the field of operations. One of the most important arguments was to fill the gaps existing in the health system’s technical, logistic and managerial capacity which hindered a timely and effective response to the key demands of the people affected (either directly or indirectly) by the emergency.

In addition, the implementation of the program was characterized by several other challenges, especially in the following areas: strengthening surveillance, cold chain management, ownership of vaccination activities by local communities, and reach of some populations (islander, Bororo, pygmies, nomads, refugees). In line with the "Polio End Game" strategy, Cameroon introduced the Inactivated Injectable Polio Vaccine in routine immunizations on July 15th, 2015 and successfully switched from trivalent Oral Polio Vaccine (tOPV) to bivalent Oral Polio Vaccine (bOPV) on April 20th, 2016.

UNICEF key Partners in the implementation of this programme included the following:

• International organizations: WHO;
• Civil Society: Community-based organizations, including Women’s groups, as well as National and international NGOS;
• Governmental agencies:  Ministry of Health, territorial administration, communication, social affairs, Women and Family promotion, Animal and husbandry, defense forces.

2.   Evaluation Purpose

The evaluation has two purposes: accountability and learning.

• With respect to accountability, this evaluation will yield evidence on whether UNICEF’s supported interventions have either succeeded or failed in their effort to contribute to the improvement of the health of children’s and women’s dwelling in Country Programme intervention zones.
• With respect to learning, this evaluation is expected to inform:

-          UNICEF management’ decision on how to increase the coverage, effectiveness, efficiency of the health sector interventions envisaged in the ongoing Country Programme, including emergency response and resilience;

-          Relevance analysis of strategies selected by the UNICEF Implementing partners, including the delivery to key population groups and the coordination with other organizations;

-          Confirmation/revision of management approaches of ongoing development, investments and researches in health sector.

3.   Evaluation Objectives

Evaluation is expected to:

·      provide robust evidences on the level of achievement of planned results;

·      provide insights into any unforeseen challenges and/or opportunities that happened and how it affected the delivery and/or strategic results;

·      identify lessons learned as well as potential good practices that can be taken to scale.

Specifically, the objective of this summative evaluation is to:

(a)   Independently and rigorously assess:

·         The quality of the health program component conception (strategic positioning – implicit/explicit Theory of Change – relevancy to needs of beneficiaries);

·         Appropriateness of implementation approaches and strategies used with special focus on synergies with major stakeholders of the health sector;

·         Program’s effects on the health system strengthening – communities – families – community health workers;

·         Effects (if any) of repeated campaigns on utilization of routine immunization services notably in the most deprived districts and communities.

(b)   Identify:

·         Lessons learned and good practices that can be proposed for scale-up;

·         Possible alternative approaches that would have permitted to achieve better results (children and mothers served) with the same amount of resources invested in the Program;

·         innovative approaches to encourage families and communities to use health services.

From a strategic perspective, UNICEF made resources available to support the development of several strategic documents aimed to enhance the children’s living conditions: the Strategic Plan to maintain the Maternal and Newborn Tetanus 2014-2020, the National policy of community health, the Guidelines on the delivery of community-led health services, The National Strategic Reproductive, Maternal, Newborn and child and Adolescent Plan 2014-2020, the newborn health Operational Plan, the Newborn care treatment guide. All such strategic documents were aimed to address some technical, procedural and legal systemic weaknesses at the national level.

From an operational perspective, UNICEF joined with partners and other key stakeholders in the national health arena to strengthen and scale-up cost-effective and high-impact interventions for the most deprived mothers and children. Such interventions included the following:  malaria prevention (LLINs and Seasonal Malaria Chemoprophylaxis (SMC), Integrated Management of Childhood Diseases (IMNCI), community-led health interventions that featured an integrated Community Case management (ICCM) component, newborn care, immunization with introduction of new vaccines, Supplementary Immunization Activities (SIAs), Cold Chain and Logistic capacity strengthening. UNICEF’s supported intervention improved quality, demand and use of services.

4.            Scope

Thematic scope

Since many partners are involved in the health sector, the evaluation will focus mainly on interventions supported at strategic level (revision of norms – policy formulation – evidence generation – policy/strategies formulation). Thematically, the evaluation will focus on two major domains:

• High impact interventions aimed at reducing neonatal mortality: with respect to immunization, the evaluation will analyze the results achieved by UNICEF’s support to cold chain, vaccine management, increase in demand and utilization uptake. Evaluators will also examine UNICEF’s contribution to the reinforcement of cold-chain, bottleneck-reduction and the quality of vaccines served to children;
• System strengthening in the most vulnerable districts of the priority intervention zones: the evaluation team will focus on the quality of services provided to the populations; as well as on the way UNICEF could address key systemic bottlenecks, including in the areas of partnership management and sector’s coordination platforms and committees.

Geographical scope

The evaluation will cover UNICEF support to the central services of the Ministry of Public Health as well as the formulation of norms, standards and policies in four Health Districts in the East – Adamawa – North – Far-North regions.

Chronological scope

The period to be scrutinized starts from January 1, 2013 through December 31, 2017.

5.   Evaluation Context

Though in continuous decrease, under-five mortality is still high in Cameroon. To accelerate reduction, UNICEF and the Ministry of public health would like to learn from past years’ interventions and adjust ongoing and future interventions. Additionally, UNICEF in West and Central Africa is committed to make the difference for the children of the region within the next four years. Immunization of among key areas where UNICEF wants to achieve significant progress. Evaluation results will permit UNICEF to revisit its strategic positioning in the health sector in Cameroun, notably in the domain of immunization. Evaluation results will be used to adjust implementation strategies and approaches to revamp performance of immunization in Cameroon.

Secondly, Cameroon has not reached vaccination autonomy and the Expanded Immunization Program is mainly sponsored by external resources. GAVI, one of the key contributors to immunization funding in Cameroon will be withdrawing very soon. It is very important to engage the Government to compensate the resulting funding gap. Evidences pooled from the evaluation will support UNICEF’s high-level advocacy to national (Government and Chambers of Parliament), local authorities and private sector in favor of domestic resources mobilization.

Thirdly, UNICEF and it partners would like to know how relevant were their strategies as well as their strategic choices articulated in the Health Component Theory of Change provided in Annex 1.

6.   Evaluation criteria

The evaluation will be conducted based on the following criteria: relevance, effectiveness, efficiency, sustainability, gender & equity, and partnerships.

7.   Evaluation questions

The evaluation team will be required to develop a strategy that could assist UNICEF in addressing all the following suggested questions:

1. Relevance

1.1.   To what extent were UNICEF CPD health interventions developed to respond to the key needs of the populations groups whom they were expected to serve?

1.2.   To what extent are program’s sub-components implemented in a coordinated approach?

1. Effectiveness

2.1.   To what extent have the different activities under UNICEF Child health program component achieved their expected results for the period 2013 – 2017?

2.2.   To what extent is the way children issues are discussed in these policies any different to what happened before implementation started in 2013?

2.3.   What have the Program’s comparative advantages and value-added been, in terms of contributing to national, regional and local policy processes and capacity-building?

1. Efficiency

3.1.   What was the cost of human resources and supply and how do they fare with the costs of similar interventions in the identified target areas?

3.2.   To what extent was the delivery of supplies timely? resources mobilized converted to outputs in a timely and the most cost-effective manner?

1. Sustainability

4.1.   To what extent was the program’s interventions designed so that their implementation could continue without UNICEF support once the program cycle ended?

4.2.   What benefits (if any) generated by UNICEF’s supported interventions could be sustained by Government, families, individual communities, community based organizations?

1. Gender and Equity

5.1.            How have gender equalities and other social health related iniquities been   

        addressed by the program?

5.2.            Have handicapped and other socially marginalized people been expressly in

              community participatory mechanisms?

1. Partnerships

6.1.            To what extent has the program built partnership with national, regional, local and international development counterparts, civil society organizations, UN family and private sector to support health system and service delivery?

6.2.            To what extent has the program facilitated the production, sharing and dissemination of new knowledge on implementation approaches of health interventions?

Throughout the evaluation process, equity and gender equality will be used as crosscutting concerns[5]. Each aspect of the evaluation will demonstrate how the poorest were targeted, served and participated.

1. Methodology

This evaluation will be a mixed methods theory-based evaluation with a strong participatory component and a distinct focus on equity. The evaluation will include a programmatic documents’ review and will require primary data collection, too.

The document review will include the following:

• A document review on all major interventions performed by the Country Programme in the health sector;
• A review of relevant performance evaluation done during the referred period;
• Review of consortium administration and other data;

The use of the monitoring data, which has been gathered to date and undertake additional data collection, will be maximised as much as possible.

Primary data collection will include the conduct of a survey and semi-structured interviews aimed to provide further insight into how UNICEF-Health is working with key stakeholders including:

·         The national Health sector coordination body;

·         Programme management staff (personnel working in the areas of Donor Partnerships, data, performance-based management, resources Administration, Financial management);

·         Key informants in communities selected for pilot and other demonstrative approaches

·         Representatives from the global health community in all intervention zones;

·         Staff of other related funding and technical organisations that are working in health

       sector in Cameroon; as well as peer organizations in health sector

·         Representatives of the government at central, decentralized, operational and grassroots levels

·         Leaders of CBO’s, NGO’s, CBO’s that participated to program implementation.

To gather data required to effectively provide meaningful answers to evaluation questions, the consultants will develop a mixed method oriented methodology. While quantitative data will inform about the magnitude of interventions conducted in health sector (e.g. the number of children and women served), qualitative methods will gather data that will tell the story of approached and strategies used by UNICEF to address major bottlenecks that were hampering demand and utilization and explain improvement of health conditions of populations. Qualitative approaches will also give the floor to key stakeholders, by gauging their perceptions of the overall Programme, including its effects and modalities of community engagement.

Last but not the least, the methodology should be designed robustly enough to permit the evaluation process to tell if some of the progress noted are sufficiently grounded in the communities to guarantee a minimum sustainability of gains produced by UNICEF’s supported interventions.

9.   Evaluation process

Maintaining the spirit of partnership and participation that guides UNICEF’s approach to managing interventions in Health sector, the Ministry of Public Health, UNICEF and other strategic programme partners agreed that:

• An Advisory Committee (composed of a representative from each of Ministry of Public Health, Ministry of Economy and Planning and UNICEF) will be set up to guide the evaluation and to secure quality and independence throughout the process. The Advisory Committee will be responsible for finalizing the evaluation TORs, providing all required information or documentation, facilitate access of evaluation team to and key informants.  The Advisory Committee will be responsible for approving the evaluator’s work plan and methodology, verifying preliminary findings, providing feedback on the draft evaluation report and approving the final version of the report;
• Technical stewardship will be ensured by dedicated staffs in UNICEF Country and Regional Offices;
• UNICEF will hire the evaluation team and take charge of all financial implications of the contract. As the evaluation manager, the UNICEF PM&E Chief will closely monitor the evaluation process in line with milestones articulated in these TORs, review data collection instruments, and liaise with and inform the Advisory Committee and UNICEF Regional Office on progress;
• Quality assurance of evaluation deliverables will be performed by Regional Evaluation Advisor in the UNICEF Regional Office, including the TORs, inception report (methodology - evaluation design), evaluation report and recommendations;
• The final report of the evaluation must be a publicly available document. Evaluation findings will be published in a the most suitable users’ friendly way. The Ministry of Public Health and UNICEF will prepare a dissemination plan for the evaluation report. A joint Government and UNICEF management response will be part of the final evaluation report. that includes, among others, a record of agreement/disagreement with the findings and recommendations from the evaluation that will be prepared for circulation within the consortium organisations and

10.     Evaluation Outputs

The expected outputs of this independent evaluation are as follows:

• An inception report (“note de cadrage” in French) including an Evaluation framework, a work plan, the recommended methodology, approach and data collection instruments;
• Presentation of preliminary findings + Power Point Presentation and draft outline of the structure of the evaluation report;
• Draft evaluation report;
• Final evaluation report by the evaluators of no more than 45 pages (excluding annexes) that responds to the questions outlined in these TORs, and incorporates feedback obtained on the draft report. The report should be formatted as per the guide on formatting evaluation reports.
• An user-friendly executive summary of no more than 2 pages.
• Annexes with details on the methodology, informants, and data collection instruments.
• Timeline and Milestones

Below are the timeline and milestones envisaged in the evaluation of UNICEF’s supported interventions in health sector. The specific details will need to be confirmed in negotiation with the evaluation team and the Advisory Committee to ensure timely completion of the evaluation and delivery of the evaluation report.

Activity

Date

Milestone

Targeted call for evaluators

End of June, 2018

 

Selection of evaluators

By July 12, 2018

 

Contract start-up

By July 25, 2018

·         Contract signature by both parties

Submission and presentation of evaluation work plan and methodology to Advisory Committee for feedback

By July 30, 2018

·         Work plan

·         Evaluation framework

Approval of work plan and evaluation design

By August 3, 2018

 

Document review and data collection

August 4-31, 2018

 

Presentation of preliminary findings and submission of detailed Evaluation Report Outline to Advisory Committee

Late August 2018

·         Progress report

·         Power Point on preliminary findings

·         Evaluation Report Table of Contents

Presentation of draft report and discussion in workshop with Advisory Committee

September 15, 2018

·         Draft evaluation report

Submission of final evaluation report

October 15, 2018

·         Final evaluation report

12.  Evaluation Team

The evaluation team will consist of five members of in four domains of expertise as follows (i) evaluation expert and team leader; (ii) expert in health system management; (iii) expert in immunization; (iv) expert in public health for child and newborn health; (v) expert in qualitative approaches. Academic backgrounds, technical skills and work experience required from team leader and members are detailed below.

i)                    Evaluator and team leader (international)

·      Graduated in social sciences or any other recognized equivalent field;

·      At least 10 years of relevant work experience in conducting evaluations and operational researches missions in developing countries;

·      Ability to work in an iterative, collaborative team approach; ability to give and receive constructive feedback;

·      Strong communication skills including negotiation skills;

·      Knowledge of the field of innovation and research in the context of global health.

·      Experience in the design and conduct of program-level evaluation of multi-partner initiatives, from a systems perspective;

·      Strong report writing and presentation skills, ability to communicate complex technical ideas using non-technical language to diverse audiences (an electronic copy of the most recent evaluation report authored by the consultant will need to be included in the application);

·      Ability to work in multi-organizational initiatives; a focus on innovation in health science and technology) research in the context of low and middle income countries;

·      Knowledge of evaluating initiatives that fund innovation research for development, particularly aspects relating to innovation derived from research (dissemination, commercialization and knowledge translation);

·      Fluency in written and spoken English; written and spoken French or a capacity to ensure that interview and other data collection can be undertaken in French with francophone recipients of DIF-Health.

ii)                  Expert in health system management (international)

·      Graduated in health or any other recognized equivalent field;

·      At least 10 years of relevant work experience in conducting evaluations and operational researches missions in developing countries;

·      Ability to work in an iterative, collaborative team approach; ability to give and receive constructive feedback;

·      Strong communication skills including negotiation skills;

·      Experience in being part of an evaluation team;

·      Very good report writing and presentation skills, ability to communicate complex technical ideas using non-technical language to diverse audiences (the electronic copy of a recent evaluation report whose the consultant has been a primary author will need to be included in the application);

·      Fluency in written and spoken English; written and spoken French or a capacity to ensure that interview and other data collection can be undertaken in French with francophone recipients of DIF-Health.

iii)                Expert in immunization (Cameroonian national)

·         Graduated in health or any other recognized equivalent field;

·         At least 10 years of relevant work experience in conducting evaluations and operational researches missions in developing countries;

·         Ability to work in an iterative, collaborative team approach; ability to give and receive constructive feedback;

·         Strong communication skills including negotiation skills;

·         Experience in being part of an evaluation team;

·         Very good report writing and presentation skills, ability to communicate complex technical ideas using non-technical language to diverse audiences (a copy of a recent evaluation report whose the consultant has been a primary author will need to be included in the application);

·         Fluency in written and spoken English; written and spoken French or a capacity to ensure that interview and other data collection can be undertaken in French with francophone recipients of DIF-Health.

iv)                Expert in newborn and child health (International)

• Graduated in medical sciences at doctorate level;
• Expert knowledge in one or more of the Grand Challenge areas (new talent for global health research, point of care diagnostics, maternal neo-natal and child health, and non-communicable diseases).

v)                  Expert in qualitative approach (Cameroonian national)

·         Graduated in anthropology, health or medical sociology or in any other equivalent field

·         Proven practical expertise in social development research activities;

·         Involvement in at least three researches/evaluations or studies over the past five years;

·         Good knowledge of social development issues on the East – Adamawa – North – Far-North.

·         Proven experience in design – collection – analysis – report writing of qualitative data.

13.  Contract arrangements:

Total contract period is for 90 days starting August, 2018 and ending October 2018.

• Evaluation team leader: 90 days
• Expert in health system management: 70 days
• Expert in immunization: 70 days
• Expert in newborn and child health: 70 days
• Expert in qualitative approach: 70 days

The detailed planning proposed by the team leader will clearly demonstrate how each of the team members will contribute to the process and to the final evaluation report.

14.     Selection criteria:

Applicants will be assessed in accordance with their ability to meet the following criteria:

·         Experience in working in complex institutional settings and with high-level officials.

·         Knowledge of consortium level and/or program level evaluation of research and innovation for development.

·         Knowledge of the field of global health and the Grand Challenge niche areas.

·         Knowledge and understanding of how health research impacts the lives of people living in the developing world.

Interested groups of individuals or evaluation firms should forward their CV`s, a cover letter (along with a sample of a recent evaluation report) to the hiring team in UNICEF Cameroon (xxxxx@unicef.org) fifteen days after the publication of the bidding.

Only short-listed candidates will be contacted by UNICEF hiring personnel for contracting purposes.

 

---