LRPS-2019-9150490 to conduct Evaluation of Multi sector Integrated Interventions for improved Nutritional Status.

 

 

 THEORY OF CHANGE:  IMPACT MODEL 

Multisector integrated Programme for Improved Nutrition in Urban areas

 

 

 

 

 

 

 

IMPACT

 

 

 

 

 

 

 

 

OUTCOMES

 

 

 

 

 

 

 

 

OUTPUTS

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

INPUTS

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

1. Purpose/Objective of the assignment

Primary objective: Measure the impact of the integrated intervention on child’s nutritional status with particular focus on improved complementary feeding practices and environmental enteropathy biomarkers as outcome at intermediate level and stunting reduction as an impact in the long run. Personal, household and community level WASH behaviours/practices, IYCF practices, access to routine health and nutrition service coverage etc. will be measured at the output level and more clearly elucidate all these potential pathway between interventions and child feeding practices leading to their growth potential.

 

Secondary objective: Evaluate the implementation process; documenting the lessons learned and knowledge/generated evidence sharing among different stakeholders so as to do the policy advocacy for at scale urban programme design, as a stand alone document. Temporal context of the programme should also be kept in mind to see how effective were the interventions when they were integrated for the common subject (children focused) which is to ensure that, the multisectoral and integrated intervention programme is designed well and on trach to achieve the expected results by 2020 in terms of:

1. Improved complementary feeding practices
2. Improved hygiene and Sanitation behaviours

and therefore: Improved health and nutritional outcomes

 

Finally, a third objective is to understand the cost-effectiveness of the intervention and undertake a thorough cost-effectiveness analysis as an independent and stand-alone piece.

 

1. Study design and proposed methodology:

A process evaluation and a cluster randomized pre-post study design with 2:1 treatment and control ratio is proposed for the impact evaluation of UNICEF’s multi-sector integrated programme for improved nutrition in urban slums. For evaluating overall programmatic public health, longitudinal panel surveys in both baseline and endline assessment will be conducted. For geographical homogeneity, the block randomization will be considered based on the intervention implementation area/location. Each block will constitute 3 geographical proximal slum clusters of approximately equal number of households. In total, 3 blocks will be considered. During the block randomization, in each block, two treatment and one control slum cluster will randomly be assigned. In total, 6 treatment slum clusters and 3 control slum clusters will be considered. According to the randomization, UNICEF and UNDP will provide the integrated intervention to the treatment slum cluster only. Given the geographical proximity of the clusters, the study design will need to take into account potential spill-over effects of the intervention, especially if mobility between slums is easy. Tools should allow to understand whether this is a potential risk to the study at baseline, and methods for minimizing the impact of spillover effects and for better measuring direct and indirect impacts of the intervention should be proposed.

 

Although cash is not randomized, the sample should allow to study the additional effect that cash brings to its beneficiaries, which will require different methods to account for the fact that cash only goes to the poorest.

 

Study participants: Improving nutritional status of children will be the main outcome of the intervention (See Theory of Change in the Annex of the TOR). Children less than 24 months of age and their mothers/caregivers will be selected as the study participants form both treatment and control slums during both Baseline and Endline. It is expected that intervention scale will be sufficient to cover the minimum sample size.

 

Sample size: Among the IYCF and child nutritional status indicators, minimum acceptable diet (MAD) is the main outcome of interest of UNICEF designed programme. According to the child wellbeing survey [3], 29.7% children received the minimum acceptable diet and during sample size calculation, 15% of absolute MAD proportion increment as per intervention impact is assumed. Considering 5% level of significance, 80% power, 0.02 inter cluster correlation (ICC), and one sided test for 2:1 unbalanced design; the calculated sample size will therefore be 867. Rounding the calculated sample, the proposed sample size is 900, of which 600 for 6 treatment slums and 300 for 3 control slums. Considerations should be made so that the sample can capture broad differences between different age groups of the mothers (adolescents (up to 19) and youth (up to 24) and older mothers (24 and above), mothers who are head of households, etc.

 

Primary impact indicators: WHO recommended indicators to measure IYCF practices [4,5] and the nutritional assessment indicators [6] comprising minimum dietary diversity, minimum meal frequency, minimum acceptable diet and consumption of iron-rich or iron-fortified foods will be used. In addition to that, length, weight, and mid-upper arm circumference (MUAC) using standardized measurement techniques will be assessed. Enumerators/anthropometric measurer will be trained and standardized in measurement techniques according to the FANTA and WHO guidelines.  Nutritional and Environmental Enteric Dysfunction (EED) (which is subclinical disorder of the small intestine characterized by the environmental enteropathy caused by poor WASH practices) biomarkers will be tested from one third sub-sample during both baseline and endline; Nutritional biomarkers including mean hemoglobin, iron, zinc, vitamin-A will be assessed [7]; as part of EED biomarkers, gut inflammation (myeloperoxidase, neopterin), permeability (alpha-1-antitrypsin, at two time points from the stool samples [8,9] will be assessed.

 

Secondary impact indicators: WASH indicators comprising both reported and observed indicators will be assessed as secondary impact indicators (See Theory of Change in Annex). The spot check/observed indicators, includes hygienic use of latrine, child potty observation, presence of soap and water at handwashing place and, absence of fecal and solid waste in compound will be assessed. As part of reported behaviours, handwashing practices at key times, water treatment practices, complimentary food preparation and preservation, usage of child potty, disposal of child feces and solid waste will be assessed at different time points of the study. This method will be conducted in real time, provide objective measures of hardware functionality and use indicating on the ground program operation, and any unexpected developments affecting the program and its objectives.

 

Unexpected or indirect impacts should also be explored. Special focus should be kept on gender specific indicators, especially those around dignity, empowerment of mothers, financial inclusion of recipients of cash, household power and decision making structures, self-esteem of recipients, among others.

 

Process evaluation: A program impact pathway (PIP) analysis will be followed in the process evaluation. PIP is an iterative process and for any intervention it enables a clear articulation of intervention activities, how they are implemented, and how they are expected to be linked with the immediate, intermediate, and final outcomes [10]. A primary goal of the process evaluation/documentation is to provide the team with the information necessary to ensure the intervention is implemented according to the prescribed strategy and to identify areas where additional communication and training would be helpful.   The process evaluation will comprise the explanatory mixed method design [11] and will be conducted at 3 different time points after the intervention implementation. The intervention fidelity of beneficiary households will be assessed from a randomly selected subset of study participants. In addition, the intermediary outcome of intervention will be also tracked by the quantitative survey of process evaluation. From each treatment slum, 25 mothers will be randomly selected for the quantitative fidelity assessment. Along with the quantitative investigation, anthropologists trained in qualitative investigation will conduct informal discussion with implementers [health and nutrition service providers (HNSP), community volunteers (CV) and community support group (CSG)], study participants [mothers or caregivers] to identify the barriers of uptake of the intervention both from demand and supply aspects. In depth interviews (IDI) and focus group discussion (FGD) will be conducted; based on data saturation, the sample size for the qualitative assessment will be identified. The process evaluation will also require a thorough desk review to understand the relevance and appropriateness of the interventions. The process evaluation will also include the service providers as respondents to get the views and the issues faced for actually delivering the services.

 

The process evaluation should also keep gender in the center. For example, whether services were designed, or not, to address gender concerns in terms of reception, privacy, polite treatment, etc.

 

The winning firm needs to provide the monitoring tools that will be used for monitoring the programme and which will help better analyse the impact results. The leading firm will be in charge of also undertaking the intermediate process monitoring checks and provide feedback to UNICEF on the findings.

 

Cost effectiveness analysis: The evaluation team will also carry out a cost effectiveness analysis and will ensure that the costs and investments incurred in the intervention are well captured and measured. The process evaluation will further allow to better understand the intensity of the human resources that are invested in the intervention, allowing for a more accurate estimation of costs. The planning of the cost data collection should start before implementation, and tools for real time collection of inputs should be embedded in the process evaluation tools. The Strategic Impact Evaluation Fund (SIEF) and other partners have started developing ideas on how to best do this, and their insights should guide this part of the assignment. Robust guidelines for doing a cost effectiveness analysis as part of an impact evaluation approach are available (for example here and here) and should guide the analysis.

 

Domain of data collection for baseline, process evaluation and endline: All the data will be captured using electronic tablet. The indicators of following domain will be collected during baseline, process evaluation and endline with standard and validated questionnaires and checklists.

 

 

Domain of data collection

Baseline

Process evaluation

Endline

1

Socio-economic and demographic status

√

√

2

Household’s food security

√

√

3

Child’s IYCF practices

√

√

√

4

Child’s nutritional status (anthropometry)

√

 

√

5

Child’s micronutrient biomarkers

 

 

 

 

Hemoglobin

√

 

√

 

Iron

√

 

√

 

Zinc

√

 

√

 

Vitamin A

√

 

√

6

Child’s Environmental enteropathy (EE) biomarkers

 

 

 

 

Fecal neopterin

√

 

√

 

Fecal myeloperoxidase

√

 

√

 

Fecal α-1 anti-trypsin

√

 

√

7

Personal hygiene and sanitation Practices

√

√

√

8

Household’s environmental hygiene condition

√

√

√

9

Household’s water treatment practices

√

√

√

10

Complimentary food safety

√

√

√

11

Child’s sanitation practices

√

√

√

12

Maternal health services (ANC, PNC pregnancy, postnatal and lactation)

 

√

 

13

Child health services (Immunization etc)

 

√

 

14

Maternal Nutrition Services (Dietary counselling, IFA, Pregnancy weight gain monitoring)

 

 

 

15

Child Nutrition Services (IYCF Counseling , GMP, SAM management, routine Vitamin A supplementation )

 

√

 

16

Nutrition Voucher/Cash usages for improving CF practices

 

 

 

17

HNSP, CV and CSG’s performance assessment (as per maternal report)

 

√

 

18

Facilitators and barriers of intervention and implementation/delivery (Qualitative)

 

√

 

 

 

Evaluation issues and questions: The indicative evaluation questions will be formulated using the key evaluation criteria of relevance/appropriateness, effectiveness and impact, efficiency, sustainability and scalability of the modelling.  Key cross-cutting issues include equity and gender equality, which need to be included across the evaluation.

The key evaluation questions that the evaluation seeks to answer are the following:

 

Relevance/appropriateness:

1. Are the interventions developed based on solid evidence on what may work to address the situation of women and children under 2 years of age in urban slums?

 

Effectiveness and impact:

1. How effective was the programme in achieving results at the outcome and impact levels of the ToC? Can these impacts be attributed to the joint efforts of UNICEF and UNDP? Are the impacts similar to boys and girls, or are gender differences apparent at this early age? Are the impacts similar for young mothers (adolescents and women up to 24) as opposed to older mothers?
2. Did the intervention lead to positive or negative unexpected outcomes?
3. What were external enabling factors for achieving results, and what were external constraining factors that may have prevented impacts to emerge?
4. How do implementation processes enable/constrain the achievement of results?
5. What role, if at all, did the C4D/SBCC components play in achieving the results?

 

Cash related impact questions:

1. What is the impact of providing cash in addition to the basic package? Is there evidence of a multiplier effect brought in by cash? (meaning, is the impact of cash greater than the amount of cash given)
2. Does the provision of cash lead to changed gender power relationships in the mother’s household? Does receiving the intervention package, and/or receiving cash lead to empowerment and increased decision making autonomy for the mothers who benefit from the programme? Does receiving cash lead to improved self-esteem of the recipients compared to those in the basic package and also to the control group?
3. Does cash have an effect on the time allocation females give to labour/rest & recreation /entrepreneurial activities?

 

Efficiency

1. The efficiency aspect of the evaluation will focus on a cost benefit analysis of the intervention as a whole, as mentioned in the study design. The main question is understanding the value for money of the programme. Cost-effectiveness should be done both for cash and non-cash scenarios and the cost-effectiveness of the package with cash and without cash should be estimated.

Sustainability

1. Have the interventions included sustainability criteria in their design? Special considerations around ease of ownership by the government and structures for its smooth scale up should be explored. Also, considerations for the sustainability of changed positive behaviors.
2. Are there challenges or barriers that have emerged that hamper sustainability at the level of sustaining results and also at the level of ensuring the sustainability of the approach?

 

In order to answer these questions and to align with the study design, the winning firm is expected to undertake: Formal Desk Review, Primary Data Collection for baseline and endline – Interviews or Focus Groups Primary Data Collection – Surveys with self-administered questionnaire with the mothers of the selected under-two children in the selected areas, anthropometry and biochemical data collection.. Special consideration needs to be paid to gender aspects, where applicable when answering the questions. The UNICEF ROSA gender toolkit should guide the analysis and gender considerations.

 

1. Evaluation norms and standards

The evaluation will abide by the following norms and standards:

• United Nations Evaluation Group (UNEG) Norms and Standards for Evaluation in the UN System, 2005 (updated 2016)
• Ethical Guidelines for UN Evaluations will guide the overall process
• The evaluation should incorporate the human rights-based and gender perspective and be based on Results Based Management principles and logical framework analysis

These guidance documents will be part of the contract of the evaluator/team.

 

The final report is expected to meet the UNICEF-adapted UNEG Evaluation reports standards as well as benchmarks used in UNICEF’s Global Evaluation Reports Oversight System (GEROS).

 

1. Duration, timeline and deliverables

Timeline:

Tasks

2019

2020

2021

2022

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Inception report

 

 

 

 

 

 

 

 

 

 

 

 

IRB approval

 

 

 

 

 

 

 

 

 

 

 

 

Baseline survey

 

 

 

 

 

 

 

 

 

 

 

 

Baseline report preparation (draft and final)

 

 

 

 

 

 

 

 

 

 

 

 

Process evaluation

 

 

 

 

 

 

 

 

 

 

 

 

Process evaluation report

 

 

 

 

 

 

 

 

 

 

 

 

Documenting the lesson learning

 

 

 

 

 

 

 

 

 

 

 

 

Endline survey

 

 

 

 

 

 

 

 

 

 

 

 

Endline report preparation (draft and final) and infographic of the summarized results, lessons learned and way forward

 

 

 

 

 

 

 

 

 

 

 

 

Cost-effectiveness analysis report

 

 

 

 

 

 

 

 

 

 

 

 

 

The deliverables attached to the above mentioned tasks are:

 

1. An inception report that presents the complete methodology approach to conducting the work, with all tools fully drafted. The inception report will also need to fully develop the data collection and analysis strategy and triangulation methodology. The inception report should cover the impact evaluation RCT approach, the process evaluation and the cost benefit analysis, as well as plans for a joint analysis of these three streams of evidence The tools and analytical methods used should explicitly consider gender, human rights and equity dimensions. An evaluation matrix that includes the evaluation questions and maps these to the tools and specific questions in the tools, as well as respondent groups is expected. Ethical considerations need to be included. The inception report should present the proposed content of the final report following the GEROS reporting standards. 

 

1. The inception report will need to be accompanied by a PPT, which needs to be presented and shared with the reference group.

 

1. Draft Baseline Report. A draft baseline report that describes the situation in the slums as observed in the data, that tests similarities between control and treatment slum blocks and that explores the threats of spillover effects to the evaluation design. The baseline report should establish baseline values of key indicators and ensure that with these values power calculations, sample sizes and precision of estimates remain as initially planned.

 

1. A process evaluation report. A first report on the process evaluation will be presented. It should already show how the process evaluation helps answer the evaluation questions and how the process evaluation will inform the sustainability and scale up strategy of the intervention. The process evaluation as a deliverable in itself needs to be self-contained and abide by the GEROS reporting standards for process evaluations.

 

1. Final Report. A final report that incorporates comments and inputs given to the draft final report and that adheres to the GEROS reporting standards. The report needs to show a clear flow from objectives and purpose of the evaluation, evaluation questions, methods and tools used to collect and gather information, analytical approach, findings, conclusions and recommendations.  The report should aim for conciseness, readability, and visual appeal. 

 

1. A reader-friendly policy brief that summarizes the key findings, conclusions and recommendations of the evaluation needs to be produced. The firm can choose the format, but it is expected that innovative formats such as infographics or an ebook are used for enhanced readability.

 

1. Data archive: Data gathered in the exercise, as well as the do-files used for its analysis are transferred in an organized archive that will permit follow-on users to replicate or extend the analysis. Suitable care to be taken in assuring the anonymity of respondents.

 

1. A cost-effectiveness analysis standalone report.

5. Data/Publication Authority

UNICEF will be the sole owner of the data and products produced, and that for academic publications co-authorship with UNICEF is necessary with prior permission from UNICEF.

6. Payment schedule:

First payment: 20% upon approval of the inception report and work plan

Second payment: 20% upon approval of baseline report

Third Payment: 20 % upon approval of draft final report and delivery of presentation

Fourth payment: 20% upon approval of the process evaluation report

Fifth payment: 20% upon approval of final report and infographic, and cost-effectiveness report

7. Qualifications or specialized knowledge/experience required for the assignment:

Institutional Consultancy:

Given the complexity of the assignment, it is anticipated that this evaluation is conducted by a reputable agency with experience in conducting similar evaluations on nutrition programmes or similar for organisations operating in the international development sector. The evaluation team should comprise a maximum of 5 team members, including at least the following: Lead Evaluator (international consultant), one subject matter expert (international or national consultant), one expert in quantitative data analysis and impact evaluation, one National Consultant (only in case the other team members are all international). Given the country context, it is important that the evaluation team be gender-balanced (also in leadership roles)

 

The Lead Evaluator will play a lead role during all phases of the evaluation and coordinate/supervise the work of the rest of the team. She/he will ensure the quality of the evaluation process, outputs, methodology and timely delivery of all products. The team leader will lead the inception phase including the conceptualization and design of the evaluation, guide the data collection phase, lead the analysis of key findings, lead the drafting of the final report and lead the validation process with stakeholders.

 

The key qualifications of the Lead Evaluator include:

• At least ten years of professional experience in evaluations with strong evidence of understanding global standards, theories, models and methods related to evaluation;
• Proven experience in designing, leading and conducting evaluations of similar scope in the context of developing programming, which involve critical analysis of organizational strategies and strategic positioning;
• Relevant working experience preferably in leading nutrition-related evaluations, not only on specific interventions but also on nutrition systems.;
• Strong experience in the design, management and implementation of development programmes and knowledge of programming principles: Human Rights Based Approach (HRBA), Gender Equality, Capacity Development and Results Based Management (RBM).
• Very strong quantitative and qualitative data collection and analysis skills;
• Knowledge of the UNICEF/UN programming;
• Excellent written and oral communication skills in English.
• Proven expertise doing cost-effectiveness analysis

 

The proposed Lead Evaluator of the bidding agencies should submit the report of the two most recent evaluations for which s/he served as a team leader.

 

The Subject Matter Expert will provide technical expertise on nutrition services and systems in the various stages of the evaluation. She/he will ensure that evaluation design, methods and tools are adapted to the nutrition system in Bangladesh; contribute to data collection as needed and ensure that the final evaluation report, including recommendations, accurately reflects the local governance context in Bangladesh.

The key qualifications of the Subject Matter Expert include:

• At least 7 years’ experience in conducting research on nutrition issues in the context of developing countries;
• Excellent understanding of nutrition issues in South Asia as well as in-depth knowledge of nutrition systems in Bangladesh;
• Good knowledge of nutrition sector programmes;
• Relevant working experience in Bangladesh and preferably experience in nutrition systems and the nutrition sector of the country;
• Very strong quantitative and qualitative data collection and analysis skills;
• Knowledge of the UNICEF/UN programming;
• Excellent written and oral communication skills in English

 

The National Consultant (only if the subject matter expert is international) will contribute to designing the evaluation, will provide inputs to the inception report and will be responsible for the collection of relevant data in the field. This consultant will work closely with the Team Leader and contribute substantively to the work of the team leader, providing advice regarding the context of Bangladesh. He/she will, under the overall supervision of the Team Leader, contribute to the preparation of the final report as necessary.

 

T   key qualifications of the National Consultant include:

• At least five years’ experience in conducting research and analysis on issues relating to nutrition (focusing on stunting and adolescent health and wellbeing in Bangladesh;
• Proven understanding of evaluation principles, methods, norms and standards – especially those of the United Nations Evaluation Group;
• Prior experience in evaluation and in supporting the conduct of evaluations;
• Ability to communicate with counterparts and stakeholders in Bangla;
• Proven ability to deliver high-quality written work in the English language and to engage effectively with stakeholders at all levels;
• Excellent written and oral communication skills in English and Bangla.

 

The quantitative expert needs to lead primary and secondary data analysis, support the development of quantitative tools of data collection and the design and calculation of quantitative indicators as needed. The quantitative expert needs to show the minimum following qualifications:

• At least 5 years experience in quantitative data collection and data analysis
• At least 5 years of experience in impact evaluations, ideally RCTs
• Professional formation in the areas of economics, statistics or comparable areas where quantitative analysis of data is core part of the foundation
• Proven expertise carrying out cost-effectiveness analysis
• Proven experience in working with specialised data analysis software such as R, Stata or SPSS

 

Given the diverse areas covered in the package, teams need to ensure expertise can be drawn from experts who have expertise in the following areas:

-Evaluation of cash transfer programmes

-Expertise in evaluating nutrition programmes, and in particular those related to maternal and child nutrition including health services aspects.

- Expertise in evaluating WASH related programmes

- Proven skills in gender responsive evaluations

 

8. EVALUATION CRITERIA FOR TECHNICAL PROPOSAL

 

CATEGORY

POINTS

OVERALL RESPONSE

* Understanding of, and responsiveness to, UNICEF Bangladesh Office requirements;

* Understanding of scope, objectives and completeness of response;

* Overall concord between UNICEF requirements and the proposal.

 

5

METHODOLOGY AND DETAILED TIMELINE

* Quality and suitability of the proposed approach and methodology (detailed description of overall approach, draft evaluation matrix, initial sampling design, initial proposed methods including those for an RCT based impact evaluation, a cost-effectiveness analysis, and a process evaluation, etc.);

* Quality of proposed implementation plan, i.e how the bidder will undertake each task and time-schedules;

*Risk assessment and ethical considerations - recognition of the risks/peripheral problems and methods to prevent and manage risks/peripheral problems, and inclusion of ethical considerations.

* Timelines proposed must be detailed and realistic;

(35)

20

 

 

 

5

 

5

 

5

 

 

ORGANISATIONAL CAPACITY and PROPOSED TEAM

* Professional expertise of the firm/company/organization, knowledge and experience with similar projects, contracts, clients and consulting assignments

* Team leader: Relevant experience, qualifications, and position with firm;

* Team members - Relevant experience, skills & competencies;

* Organization of the team and roles & responsibilities;

 

(30)

10

 

10

5

5

    TOTAL MARKS

70

 

For this RFP, the Technical Proposal has a total score of 70 points. Bidders must score minimum of 49 points to be considered technically compliant and in order, for the Financial Proposals to be opened. Financial proposal has a total score of 30 points.

 

The final selection of the bidder will be based on a quality and cost basis as specified in the RFP.