Evaluation of Social Cash Transfers Programmes, Oromia and SNNP regions, in Ethiopia - (3 years and 3 months)
) Evaluation of Social Cash Transfers Programmes, Oromia and SNNP regions, in Ethiopia - (3 years and 3 months)
2. Background
This ToR outlines the objectives, design, methods, timetable, instruments and indicators for the evaluation of the Social Cash Transfer (SCT) programme for Permanent Direct Support Clients of the Productive Safety Net Programme (PSNP) phase-IV in two regions of Ethiopia. These terms of reference should be looked at in conjunction with the design document and the logframe for the phase-IV of the PSNP, the draft document that outlines the design of the SCT in Oromia and SNNP, and the operations manual drafted in October 2014.
The overall purpose of the evaluation is to provide the PSNP Steering Committee and Donor Working Group, and Regional Social Protection Committee with the information they need to take decisions regarding programme implementation modifications and to provide information they will need to facilitate a discussion with the Ministry of Agriculture (MoA), the Ministry of Labour and Social Affairs (MoLSA), the Ministry of Health (MoH) and Regional Council regarding possible scale-up to MoLSA and BoLSA of Permanent Direct Support Clients of Social Cash Transfers, soft conditionalities linked to nutrition as part of the PSNP and regional social protection plan of action. The implementation modalities of the national Social Protection Strategy and the system approach for the Social Protection sector will also be informed.
To do so the evaluation will:
i) Assess the impacts of the SCT programme on the clients and the communities in which they live (- % of children < 2years with a height for age> -2 Z-score (WHO standards); - % of children under one year participate in at least 6 GMP sessions per year, - % of PW attendance to at least 4 ante-natal care visit; - % of adolescent girls dewormed); children’s dewormed and supplemented with vitamin A; guardians attended community conversation sessions
ii) Evaluate the operational linkages and coordination effectiveness for the system approach of the programme (the extent to which the programme operates in accordance with its design) including recording its capital and recurrent costs at kebele, woreda and regional levels;
a. Assess the capacity of BOLSA to expand their responsibilities in the two regions for the Permanent Direct Support Clients of the PSNP.
b. Assess the coordination between the implementing bodies and use of the MIS as case management tool (presence of an integrated PSNP and CBN M&E system).
c. Has the process been implemented as expected in the operations manual?
d. Have the transaction cost reduced? Or this system have just increased the transition cost of the programme?
iii) Assess the impacts of the soft conditionalities related to nutrition- Is the household dietary diversity of Direct Support Clients and Temporary support clients improved as a consequence of the exposure to the programme?;
iv) look at the effectiveness with which it reaches the target group and delivers the expected social outcomes in nutrition, health, education and child protection;
v) Identify challenges and lesson learned
The evaluation will employ several information collection methods: a quantitative household survey for welfare impact, soft conditionalities impact and operational performance, and quantitative community surveys (to understand issues such as supply side constraints); qualitative focus group discussions with clients, members of the community, programme managers and others; some in-depth interviews; an organisational review; and a cost analysis.
Evidence about social welfare impact, operational and coordination capacities, and soft conditionalities impact effectiveness is necessary to inform the decision on whether to scale up to regional/national levels. Information on operational effectiveness is also necessary in order to understand what changes to the design of the programme will be necessary if scaled up.
3. Objectives:
The evaluation questions address the widely-used five OECD-DAC criteria: efficiency, effectiveness, sustainability, relevance and impact. The overall objective of the evaluation is to answer the following main questions, which have been detailed in the programme document:
• What is the extent of impact on men and women participants attributable to the pilot programme?
• Has the SCT programme had a substantial impact on members of male and female HH’s welfare?
• Have welfare improvements in nutrition, education and child protection, if any, been in proportion with the investments made, or the value for money of the prorgramme?
• Have soft conditionalities additional impact on male and female HH’s welfare?
• What is the impact or incentive effect of imposing soft conditionalities on clients? What is the cost of doing so, for both households and the government? If households fail to comply with the soft conditions, why is this so?
• Has the coordination and operational modalities between the bureaus, i.e. social affairs, health and agriculture, involved in the implementation been efficient and effective to reach the targeted clients?
• Would be a similar region wide programme be affordable? And what would be the costs associated with a scale up at regional level?
• What have been the financial and human resources inputs from the regional government (bureaus/ woredas) into the program?
• On that basis, should the programme, or a variant of it, be scaled up to a regional/national level?
• If the programme is to be scaled up, which aspects of its operation and coordination mechanism must be modified or strengthened for it to operate effectively at a regional/national level?
• Which aspects of good practice should remain the same and be replicated in the PSNP Phase IV?
This evaluation distinguishes three types of information necessary to answer these questions: impact on participant households, especially impact on children and women; operational and coordination performance of the implementation, including costs broken down by capital and recurrent costs as well as identifying and addressing cross cutting issues like gender, and how they change in start-up and maintenance phase.
Four main activities will be undertaken: 1) a quantitative survey of households and communities; 2) an organisational review; 3) qualitative data collection; and 4) a cost analysis with a focus at the direct investment which will be required from the regional government in case of scaling up.
These activities will use a number of instruments: a) Quantitative surveys with anthropometric measure: Household and community surveys (baseline, follow-up, and endline); b) Qualitative: focus groups, with a range of different groups in beneficiary and comparison communities; and in-depth interviews; c) Cost analysis.
4. Background and Experience Required of the Firm/Institution.
The evaluation will be conducted by a registered institution or a team of freelance consultants. It is proposed that the evaluation team consist of one or two international experts in social cash transfer evaluation teamed with a national research firm experienced in quantitative and qualitative household surveys. The team leader will have the responsibility for all negotiations, decisions, and deliverables. The technical work is to be divided between the team leader and the team members.
Therefore, bidders for this contract are expected to provide:
One team leader:
•Extensive evaluation expertise and experience (at least 8 to 10 years) and a strong commitment to undertake the evaluation.
•Knowledge of institutional issues related to development programming (including funding, administration, the role of the UN system, partnerships, human rights and sustainable development issues)
•Familiarity with social protection policies and programmes with a focus on social cash transfer programmes and their evaluation