Evaluation Facts

Project Title: Integrated Humanitarian Assistance for displacement affected communities in Somali, Oromia, SNNP and Gambela regions of Ethiopia

Background and project context

DRC has been operational in Ethiopia since 2009. Currently, the organization works in five different regions

(Gambella, Tigray, SNNP, Oromia and Somali) and one administrative town (Addis Ababa) in a range of activities with a focus on providing humanitarian assistance and creation and promotion of protective environment for displacement affected communities.

DRC implemented multi-sector response to address needs in WASH, Shelter, livelihood, multipurpose cash assistance and protection for refugees, IDPs and host communities, to provide life-saving assistance and enhance community protection mechanisms. DRC have been providing live-saving interventions for the most in-need internally displacement people communities in Somali and Oromia region, while continuing to respond to the needs of South Sudanese refugees in Gambella region and Eritrean refugees in Tigray region.

These displacements affected populations continue to require humanitarian assistance to meet their basic and protection needs. In response, and considering the needs and gaps, the project, “Iintegrated Humanitarian Assistance for Displacement Affected Communities in Somali, Oromia, SNNP and Gambela regions of Ethiopia” was designed to respond multi sector humanitarian needs for IDPs, refugees and host community in the four regions. In particular, project was implemented in Liben and Fafan zones of Somali region, West Guji zone in Oromia, Gedio zone in SNNP, Kule, Tierkidi, Nguenyyiel and Pamdom camps in Gambella region. The project responded multi-sector namely WASH, shelter, multipurpose cash and protection assistance for IDPs, refugees and host communities in coordination with the relevant humanitarian clusters, working groups and government duty-bearers.

The action was formulated to achieve the objective of addressing critical humanitarian emergence needs and promote community protection and protection programming for IDPs, refugees and host communities in project environment locations. DRC implemented the project for a period of 14 months from 1st April 2020 to 31st May 2021.

Project’s Objectives and Expected Results

Principal objective (outcome): to contribute to the delivery and strengthening of coordinated humanitarian assistance for displacement affected communities in the Somali, Oromia, SNNP and Gambella regions, while continuing to respond to persistent and critical gaps in basic needs.

Specific objective: through the proposed integrated intervention, DRC will create and promote protective environments, increase access to WASH and shelter services, provide immediate emergency responses to displacement affected communities in the above stated regions. The following specific results are expected to be realised at the end of the project.

Result 1 (WASH): Affected people have access to safe, sanitary, and hygienic living environment through provision of improved WASH services that are safe, sanitary, user friendly and gender appropriate.

Result 2 (Protection):  Protection concerns faced by refugees, asylum seekers and IDPs are monitored and addressed through the provision of protection response services, community engagement and evidence-based advocacy. 

Result 3 (Shelter and Infrastructure): Access to shelter is improved in underserved and vulnerable host and refugee communities in Gambella.

Result 4 (Multipurpose cash for basic Needs): Crisis and displacement affected populations in Somali, Oromia and SNNP regions receive timely humanitarian assistance which meets their immediate basic needs and promotes safety and dignity.

Target Beneficiaries

Overall project targeted 240,360 individual beneficiaries. The programme sought to work with young people, women, youth groups, community leaders, underrepresented groups, local communities, local government authorities, humanitarian clusters and working groups in the above-mentioned project locations. Specifically, the project targeted vulnerable communities and individuals those at need of emergence humanitarian assistance.

The project has been implemented in the target areas since April 2020 and is planning to conduct an end of project evaluation in May 2021. The evaluation will help document key achievements, gaps, challenges, lessons, and recommendations for future programming in volatile context.

Timing of Evaluation: 1st May 2021 to 31st May 2021. Detailed timelines are outlined below:

Activity - Deadline - Responsible

Inception and data collection May 2021 Consultants

Data analysis and report writing - June 2021 - Consultants

Validation workshop and submission of final evaluation report - June 2021 - Consultants / MEAL team 

Evaluation Trigger: requirement of the donor and project commitment as set out in the project document

Objectives of the evaluation

• The primary objective of this evaluation is to assess achievement of results (intended and unintended) of the integrated protection-livelihoods approach over one-year implementation period. The evaluation will assess performance of the project against set objectives and result targets as defined in the project logical framework.
• The evaluation will also seek to document any lessons learned and changes within and outside the project that impacted project delivery, impact and sustainability of benefits to targeted beneficiaries.

Intended use of the evaluation findings and recommendations

• Intended users of the evaluation findings include ECHO, DRC programs in Ethiopia and globally, and protection and cash sector working groups interested in integrated programming in the project locations. The findings will inform inter alia:
• Adaptation and scaleup of the integrated humanitarian assistance approach in other DRC country programs.
• Development of detailed integrated programming guidance to inform piloting of the approach for other country programs / organizations that are interested.
• Reflection and peer learning at validation forum(s) bringing together ECHO, DRC staff and other key stakeholders. Findings from the evaluation will also be widely disseminated internally through webinars and through an evaluation summary posted on DRC’s website. A Lessons Learned Note will be documented from the evaluation and submitted to DRC MEAL HQ for input to meta-evaluations in DRC.

Scope of the evaluation

• Project time frame: The evaluation will cover the 14 months of implementation funded by ECHO, that is from 1st April 2020 to 31st May, 2021. The study may also refer to other interventions implemented within the project locations for comparison.
• Thematic areas: The evaluation scope will cover the key interventions implemented over Protection, WASH, Shelter and infrastructure and Multi-purpose cash assistance sectors. However, the service provider will focus on the relevance of the specific results of the project implemented by DRC.
• Evaluation time frame: The evaluation is expected to be conducted from 1st May 2021 to 31st May 2021. 
• Geographical areas: Fafan and Liben zones in Somale, West Guji in Oromia, Gedio in SNNP, Kule, Tierkidi, Nguenyyiel and Pamdom camps in Gambella region.

Evaluation Criteria and Key Evaluation Questions

•  The evaluation will use the OECD DAC evaluation criteria adapted to humanitarian action by ALNAP.
• The evaluation questions to be answered under each evaluation criterion below:

Relevance/Appropriateness

• To what extent were the intervention responses able to address the community and individual beneficiaries’ assistance needs? (compare needs of different project target groups including children). How was feedback collected and used to adapt the interventions? Did beneficiaries feel that their feedback was considered in design / adaptation of the interventions during project period?
• Are the ECHO project response strategies (and intervention) appropriate to the current needs and the consortium partners mandate? Looks at appropriateness of conceptualization (there is a need?) and the appropriateness of design.
• To what extent did protection monitoring reports contribute to design / adaptation of interventions for DRC and/or other actors in the project locations? Was the information useful in understanding the protection trends and informing advocacy priorities (for DRC, for other agencies)?
• Were the protection messages relevant to the information needs of targeted community groups (women, girls, boys, men, persons with disability)? Which dissemination strategies were most appropriate for the targeted groups including children?
• How was capacity building on protection, WASH, shelter and multi-purpose cash assistance tailored to meet the needs and capacities of the different target groups in the community?
• Were selection criteria and processes adequate in identifying individuals for the integrated interventions? Were the right people, living in the right area targeted at the right time? What worked? What requires further review / adaptation?
• How well did the project plan for and/or adapt strategies to address both anticipated and unexpected internal and external challenges and constraints? What lessons can be adapted for future similar interventions to improve project planning and management processes?
• To what extent does the project respond to priority issues?
• Did anything about the ECHO program contradict or conflict with other existing programs (especially longer-term plans) both of the country and of other donor programs?

Effectiveness and Efficiency

• What are the major achievements of the project to date in relation to its stated objectives and intended results? What were the contribution factors for success / failure?
• Has the program achieved the expected outcomes/result?
• To what extent are the target groups and beneficiaries reached?
• To what extent were protection actors and community structures able to use the skills they gained from capacity building initiatives? Did the training help the actors deliver services more effectively and/or efficiently?
• To what extent did selected activities match the preference and capacities of individuals participating in the project? What strategies were employed to adapt the protection, WASH, shelter, multi-purpose cash and coordination interventions to the specific needs of persons with protection concerns?
• What is the level of satisfaction among direct beneficiaries both Refugee and host communities on the project? 
• If the program has not achieved some results, investigate why not and the barriers to success?
• What internal and external factors affected (positively or negatively) the effectiveness of the project?
• Have activities and results been cost efficient compared with results? Could the results and activities have been attained with fewer resources?
• To what extent has the project design, planning and implementation took gender integration into consideration.

Coordination

•  How did DRC project teams collaborate in selection and support to beneficiaries to promote achievement of both outcomes for the targeted beneficiaries?
• To what extent have different components operated by different actors reinforced each other to contribute to the project overall objective?
• What were the strengths and weaknesses of coordination and collaboration?
• How actively were other organizations involved in the coordination? What effects did the coordination have on the project? Which factors have restricted the coordination, and which factors have supported it?

Outcome and impact

• Has the project achieved its planned impact? What situations have been changed / impacted most by the project? How have they changed/impacted? What are the evidence/justification? How well were the indicators and measurement methods able to discern changes in the protection situation for beneficiaries?
• Did impact vary for different individuals (consider age, gender, IDP, refugee, host),in the target locations? If so, how and why?
• What changes in knowledge, attitudes and practices towards Protection has the project contributed to? (In different areas and target populations). What factors facilitated or hindered achievement of the expected changes?
• What changes (intended/unintended) has the mentorship program contributed to for mentees and their families? Did the approach contribute to improved coping skills for mentees, and which ones if at all?
• To what extent has the integrated humanitarian approach contributed to changes in the protective environment for targeted beneficiaries and their households? Which activity(ies) contributed the most / least to achievement of protective outcomes and why? What factors facilitated or hindered achievement of expected results?
• How does an integrated humanitarian assistance approach compare with standalone interventions in addressing protection concerns while achieving basic needs?

Sustainability

• What specific institutional capacity needs did the project address to foster greater local engagement and accountability? What factors need to be taken into consideration to sustain project results and outcomes?
• To what extent were local capacities developed or strengthened through the project intervention
• To what extent are the changes in outcomes of individuals under the integrated humanitarian intervention still being felt/observed after exit from direct support? What factors have facilitated or hindered sustainability of protective outcomes?
• How has DRC structured support to beneficiaries under the integrated approach achieve self-reliance? How have beneficiaries responded to shocks/stresses that affected their livelihood, protection, WASH, shelter and multi-purpose cash situation (drought, floods, COVID-19)? Have the beneficiaries required additional support to cushion them from shocks and/or recover from shocks to their WASH, livelihoods or protection situation (if so, with what results)?
• To what extent is the sustainability of protective outcomes, increased access to WASH and shelter services and immediate emergency responses contingent on economic self-reliance (or vice versa)?
• How have the approaches applied contributed to the sustainability of the project achievements?

Evaluation deliverables

For all evaluations, a Lessons Learned Note should be completed and sent to mel@drc.ngo for input to meta-evaluation and peer learning processes in DRC. (template to be provided by DRC)

• An inception report detailing methodologies to be used and representative sample size calculations, a detailed execution plan, data collection tools, clearly defined research/assessment questions and corresponding or feeding sub-questions (incorporating the summary of the introductory meetings)
• Draft Report: Draft summary of findings and recommendations for review.
• A presentation of the key findings in a validation workshop(s) with key stakeholders.
• Final Report: A comprehensive evaluation report that highlights the findings, lessons learnt, best practices and practical recommendations.
• A presentation of the key findings in a validation workshop.
• DRC Lessons Learned Note.
• Evaluation datasets.
• Outputs of statistical analysis of quantitative results in MS Word.
• Content analysis or other analytical output of the qualitative data, including key themes identified and frequencies of those themes.
• 2-page standalone brief of the evaluation findings and recommendations for external dissemination.
• The final report and all deliverables will be due on or before June 15th 2021.

Methodology

The evaluation methodology will be further developed by the consultant / evaluator(s) and should be well articulated in the response to the ToR. The methodology should demonstrate robustness and rigour in addressing the evaluation objectives and questions. Interested consultants will be expected to propose the design and methodology to be used in this evaluation as part of their proposals (keeping in mind adaptations due to COVID-19). The methodology proposed by the successful consultant shall be discussed, finalized and approved by DRC evaluation committee before commencing data collection as part of the quality assurance process. It is however expected that the methodology will use a mixed-methods approach including both qualitative and quantitative components. The data collection methodologies should use a participatory approach engaging all relevant stakeholders, including community leaders, government counterparts and partners. The consultant should demonstrate how they will conduct document and system review, household surveys, FGDs, KIIs and Field visits/observations as appropriate. More over the evaluation team will assess integration of crosscutting issues in particular gender.

Data protection and confidentiality

While executing this assignment, the consultant and all the parties involved shall ensure effective protection of confidential and sensitive data and information in conformity with the humanitarian and protection principles and to applicable legal data protection standards[1]. All data collection and processing activities shall be executed in accordance with the following principles:

• Safeguarding individuals’ personal data is a crucial part of humanitarian mission to protect the lives, integrity and dignity of beneficiaries and participants and is fundamental in the provision of protection response and humanitarian aid.
• People-centred and inclusive: Evaluation activities will respect the interests and well-being of the population, in all relevant phases of the evaluation and which activities must be sensitive to age, gender, and other issues of diversity.
• Do No Harm: Evaluation activities must include a risk assessment and take steps, if necessary, to mitigate identified risks. The risk assessment must look at negative consequences that may result from data collection and subsequent actions.
• Defined purpose and proportionality: The purpose must be clearly defined and explained to the participants in the data collection process.
• Informed consent and confidentiality: Personal information may be collected only after informed consent has been provided by the individual in question and that individual must be aware of the purpose of the collection. Further, confidentiality must be clearly explained to the individual before the information may be collected. Consent must be genuine, based on the data subject’s voluntary and informed decision.
• Data protection and security: The evaluation process must adhere to international standards of data protection and data security.
• Competency and capacity: Actors engaging in this evaluation are accountable for ensuring that evaluation activities are carried out by a competent team who have been trained appropriately.
• Impartiality: All steps of the evaluation cycle must be undertaken in an objective, impartial, and transparent manner while identifying and minimizing bias.

Recommended documentation

As a minimum the evaluator will be made aware of and have access to the following documentation;

• Project proposal, LFA, publications and relevant monitoring and donor reports and data
• Reports from evaluations conducted on similar or related projects
• DG-ECHO protection mainstreaming indicator guidanc
• Evaluation report template
• Lessons learned note template

Job Requirements

The evaluation consultant or firm should meet the following requirements:

• Extensive expertise in evaluations of complex programs in humanitarian contexts and minimum of 5+ years of experience in conducting evaluation studies with particular experience in refugees, IDPs and host context for integrated programs that include protection components (protection monitoring, case management, GBV and child protection response and PSS etc), WASH, shelter and multi-purpose cash assistance sectors.
• Proven experience in conducting evaluation studies with international humanitarian organizations and preferably EU projects.
• Advanced degree in Social sciences, Statistics, Economics or another relevant academic discipline with extensive experience in evaluating projects in these fields
• Proven experience of using participatory methods for data collection and analysis.
• Knowledge of ICT tools for mobile data collection
• Excellent spoken and written communication skills in English. Knowledge of languages of the local communities is an asset
• Excellent skills and ability to articulate ideas in a clear and accurate manner including the ability to prepare quality reports. Good data presentation and visualization skills.
• Strong interpersonal skills, analytical skills and ability to establish and maintain effective working relations
• Practical experience on gender issues and gender integration analysis
• The consultancy firm / individual consultant is registered under the law of the country
• Working experience in the study locations is an added advantage

How to Apply

The key stakeholders and their specific roles in the evaluation process are:

No - Stakeholder - Role in the evaluation

1. DRC Ethiopia Head of Programs - Commission the evaluation
2. DRC MEAL Manger - Evaluation planning and coordination
3. Evaluation Team of Advisors: - HoP, Team Leaders, Regional Coordinators (MEAL, Protection) Provide technical support and guidance to the Evaluation Manager and team. Review key evaluation documents to shape the quality of the evaluation processes and products
4. Consultant TBD - Implement evaluation and deliver agreed deliverables within schedule
5. Other key stakeholders: Donor, partners, local government officials and beneficiaries’ communities. Project staff

Provide feedback and experiences on the project. Validation of findings.

Follow up

A signed management response to the evaluation findings will be shared with the GSL on Monitoring and Evaluation in OPSU by the manager commissioning the evaluation.

Contact information for manager commissioning the evaluation: megerssa.tadesse@drc.ngo 

Technical and financial evaluation criteria

Technical criteria # - Technical criteria (70%) Weighting

• Quality of methodology presented by consultant:

1. Work plan with realistic deadlines. (In line with ToR requirements)
2. Key considerations and proposed methodology outline
3. Approach in evaluation and data collection with sampling strategy - 40%

• Previous experience in fulfilment of similar assignments:

1. Proven record of conducting evaluations studies related to integrated protection programs
2. Quality level of at least two samples of reports from previous projects relevant to this evaluation study
3. Relevant feedback/reference from at least two previous clients -20%

• CV quality for staff performing the task:

1. Academic/professional qualifications in areas relevant to the study -10%

 Technical Evaluation Sub-Total - 70%

 Financial evaluation - 30%

 Total - 100%

Practical implementation of evaluation

Bid by the consultant(s) should include the following information:

• Evaluator’s capacity statement describing how they meet required qualifications and competencies including summary of similar studies previously conducted
• Detailed evaluation plan that includes a timeline of activities and level of effort required for each activity in tabular summary, including deliverable submission dates.
• An expression of interest detailing their understanding of consultancy assignment
• Description of methodology, sampling strategy, approach to primary data collection and Plan for data analysis and approach to data quality management 
• A detailed financial proposal that includes professional fees and all logistics expenses.
• Skills and experience of the evaluator(s), including key members list with core qualifications and their role in evaluation (Attach the CVs of these key team members involved in the evaluation)
• Two to three samples of evaluation reports from previous evaluations relevant to this study.

The interested consultant must submit both technical and detailed financial proposals separately sealed to DRC in ten working days from the date the announcement. All applications shall be submitted to email: “RFQ.ETH@DRC.NGO” latest by 15/04/2021.