TERM OF REFERENCE

EMERGENCY RESPONSE AND CMDRR 

TO EARTHQUAKES AND TSUNAMI IN CENTRAL SULAWESI

(EA 06/2021/INDONESIA)

EXTERNAL EVALUATION

INTRODUCTION

On Friday, 28 September 2018 at 17.02.44 WITA (Central Indonesia Time), a tectonic earthquake of 7.4 on the Richter scale struck 4 districts/cities in Central Sulawesi Province. This triggered a tsunami on the coast of West Donggala and Palu Bay. It also triggers liquefaction in 4 areas: Balaroa and Potobo in Palu and Jono Oge and Langaleso / Sibalaya in Sigi District. Meanwhile, Bowa Village, which is in Banawa District, Donggala Regency, sank into the sea at a depth of 23 meters.

Based on data published by the Kogasgabpad Post for the Central Sulawesi Earthquake and Tsunami Disaster Management on February 28, 2019, there were 4,402 victims died, to which the death toll was 2,685 recorded in Palu Regency: 2,132, Sigi Regency: 289, Donggala Regency: 249 and Parimo : 15. A total of 1,106 unidentified victims were buried in the graves of Poboya and Pantoloan. There were 4,438 injured victims, of which the total victims in Palu Regency: 1,549, Sigi Regency: 1,112, Donggala Regency: 1,750 and Parimo : 27. There are still 1,309 missing people and 122 points are still being used as temporary shelters located in Palu Regency: 80,034, Sigi Regency: 84,888, Donggala Regency: 41,019 and Parigi Moutong Regency: 553. Buildings destroyed included 65,733 houses, 327 houses of worship, 265 schools, 3 hospitals, 7 bridges, Airport: Towers and terminals, 168 road crack points at the Anjungan Talise, 78 office building units and 362 shops.

In response to the disaster, several organizations took an initiative to conduct joint assessments (JNA). The aim of this JNA was to determine the impact of the earthquake and tsunami during the first weeks which included primary data collection and community needs assessment in the shelter. Hence, NGOs’ work to support affected families was carried out strategically based on the JNA findings. The JNA targetted sectors of shelter, WASH, health, food security, protection, education and access to information and markets, and the vulnerability of people living in shelter centres. Caritas Manado, one of the members of Caritas Indonesia National Network, in which Palu Regency is one of its service coverage, has been actively helping survivors since the first day after the disaster hit the city. It opened a soup kitchen at the Church of St. Maria, Palu, to provide food for 500 survivors every day.  The kitchen reached 8,664 families equivalent to 38,110 people from 47 villages, and 17 sub-districts in Palu, Sigi and Donggala districts.

During the recovery transition, with the full support of Caritas Indonesia and Facilitating Partners, Caritas PSE of Manado Diocese has implemented the EA27 / 2018 project funded by the Caritas Internationalist network by activating the Emergency Appeal Protocol. The main activities of this project are NFI, MPCA / CTP, Livelihoods, DRR, Psychosocial and Protection. The number of beneficiaries supported by this project through the distribution of NFI supports 2,000 households, MPCA / CTP supports 3,648 households, DRR supports 12 villages and psychosocial supports 12 villages. This project was officially closed in December 2019.

In 2020, with the approval of CI, Caritas Indonesia and Caritas PSE of Manado Diocese developed the EA04 / 2020 project. This project was developed based on the sum of the remaining budget of the EA27/2018 Project and based on the real conditions in the area. There were many survivors who had not received proper housing. The project was officially closed in February 2021. Since decent housings were in need by the disaster-affected families, especially in Donggala Regency, where people had not received housing assistance either from the government or other humanitarian agencies, and some amount of funds were left over from CIMOS donors for the implementation of the EA project 27/2018 and EA 04/2020, Caritas Indonesia and Caritas PSE of Manado Diocese have intended to request approval from CI and CIMOS donors to use the remaining budget to build decent shelters for those unfortunate disaster-affected families.

This request was approved by Caritas Internationalis to carry over the project of EA27/2018 and EA04/2020, under the name of EA06/2021. The main activities of EA 06/2021 were the construction of shelters and latrines, dissemination of the lessons learned derived from the shelter programs post-emergency response, to 37 Dioceses,  members of Caritas Indonesia’s Network in Indonesia. The implementation of building shelters and latrine programs is a valuable experience for Caritas Indonesia and Caritas PSE of Manado Diocese. This experience needs to be shared with all Caritas networks in Indonesia in the form of training, workshops, or book publication. Since Indonesia lies above the Pacific Ring of Fire and is vulnerable to many natural disasters, it is inevitable that the same situation may occur in 37 dioceses. The last important activity is to ensure that Caritas PSE of Manado Diocese has a stronger organization and structure with sustainable program management that is measurable, structured, directed and well documented.

In general, the project implementation is beneficial to the beneficiaries with regard to safety, comfort, and privacy. However, to ensure that the assessment of project benefits is not biased, it is necessary to carry out an external evaluation conducted by an independent person or institution.

OBJECTIVES OF THE EVALUATION

The objectives of this external evaluation are:

1. 1. 1. Assessing the Relevance, Effectiveness, Efficiency, Participation, Impact and sustainability of the project implementation for 14 months carried out by Caritas PSE of Manado Diocese.
      2. Obtaining lessons learned and recommendations for planning, implementing, evaluating and monitoring for future projects.

MAIN POINTS OF EVALUATION

• Relevance

The key questions:

1. 1. 1. 1. 1. To what extent did the interventions for building houses, toilets and CMDRR support address the problems faced by the survivors of the earthquake, tsunami, and liquefaction in Donggala?
            2. To what extent did the constructed houses and toilets provide a sense of security and comfort to the beneficiaries?
            3. To what extent is the flexibility of the project implementation able to respond to changing contexts that occur in the communities served?

• Impact/Outcome

The key questions:

1. 1. 1. 1. 1. To what extent has project implementation resulted in positive and negative changes for all people served?
            2. To what extent does the project implementation benefit the assisted communities? As an example; behaviour change, knowledge, and skills.
            3. Behaviour Change: To what extent do housekeeping and toilet aids lead to a change in attitude or behaviour? What is the evidence of change? It is better if given real examples.
            4. To what extent are the field facilitators of Caritas-PSE of Manado Diocese who have been trained able to facilitate CMDRR training at the assisted community level?
            5. To what extent are the benefits of the project enjoyed equally by men and women? How can that be proven?
            6. Were there any unplanned impacts that occurred during project implementation? If so, how could that be?
            7. To what extent has working with network partners helped achieve project objectives?
            8. How large is the area (hamlet or village) that will enjoy the impact of the project activity? What impact did they enjoy from this project?
            9. What important lessons did Caritas Indonesia and Caritas-PSE of Manado Diocese gain during project implementation? Which important lessons need to be documented specifically to serve as future references for implementing projects with the same intervention? Has there been a process in terms of documenting it?

• Effectiveness

It includes all efforts or actions to achieve goals or objectives.

The key questions:

1. 1. 1. 1. 1. Have the mentoring activities been conducted realistic enough?
            2. Was the implementation of all activities conducted easily?
            3. Have all the implementation of activities been planned for a measurable and beneficial to the assisted community?
            4. To what extent have the objectives been achieved? What information is available on the indicators set at the start of the project design that will help answer this question? What other information is available regarding the achievement of goals?
            5. What are the main factors that influence the achievement or unachieved project objectives?
            6. Are the objectives set at the time of project design realistic?

• Efficiency

It is the accuracy of the method and the ability to do a job well. In other words, minimal resources to achieve optimal results.

The key questions:

1. 1. 1. 1. 1. What evidence shows that the project was conducted efficiently under certain circumstances? Is the project conducted in an economical and cost-conscious manner?
            2. What are the parameters on which the assessment will be based (e.g., cost per project output: cost per training course or trainee, cost per beneficiary…)? Are there benchmarks for these parameters available from other projects or agencies?
            3. How well is the organization’s performance (Caritas Indonesia and Caritas PSE Manado)? Possible areas to consider include, for example, management systems, administration, and communication structures.
            4. How well can the organization monitor and reflect on all project activities? How do you assess the PME (Planning, Monitoring and Evaluation) system, and the effectiveness of performance at the diocesan (Caritas-PSE Manado) and national (Caritas Indonesia) levels? How useful are the monitoring formats and meetings held?
            5. What is the relationship between the observed impacts and the resources used? Are the financial resources being used as planned or are they experiencing weaknesses and or excesses? Are there activities that have had excess or lack of funds? If so, why do you think it happened that way?
            6. Is the project implementation running on time?
            7. Does the capacity of the diocesan staff (Caritas-PSE Manado) match the needs?
            8. What has been done to achieve optimal efficiency?

• Participation

The main stakeholders in project implementation are all people served. They are the goal of the project. Therefore, their participation in project implementation is necessary.

The key questions:

1. 1. 1. 1. 1. To what extent were all the people being served actively involved at all stages of project implementation? For example: in all phases of housing and toilet construction and CMDRR activities.
            2. To what extent did those who were the most vulnerable actively participate in decision making?
            3. To what extent did women and men participate equally in this project?

• Sustainability

The ability of all people served to continue and manage project benefits after project implementation ends.

The key questions:

1. 1. 1. 1. 1. To what extent are project benefits likely to continue after project implementation ends?
            2. What are the main factors affecting the achievement or failure of the project sustainability?
            3. Does everyone served remain committed to maintaining the house and toilets and using them? In the judgment of everyone served, will the houses and toilets last or not last long? What factors make houses and toilets last or not last long?
            4. Does the Uetua PRB Community named Mosi Kabelo Pura Uetuah remain committed to continuing the activities they have planned? What factors make them continue to conduct activities or not continue the planned activities? Is there a possibility that they can access village funds to support their activities?

EVALUATION TARGETS

The target group for this external evaluation are 60 families served by the EA 06/2021 Project located in Ape Maliko Village, Donggala Regency.

EVALUATION STAGES

• Preparation

The preparation stage for this external evaluation includes a preparatory meeting between the evaluators and the Caritas Indonesia team (Director of Caritas Indonesia, Director of CPSE Manado, Program Consultant, Project Coordinator, Office Manager and Caritas Indonesia Finance Coordinator), a review of documents related to project planning, implementation, and evaluation EA06/ 2021.

• Field

Activities in the field include socialization of evaluation objectives and targets to all staff, collecting related data and information; relevance, effectiveness, efficiency, impact, participation, and sustainability of Project EA06 / 2021.

• Debriefing Session

Evaluators will conduct debriefing as soon as they dig up data and information in the field. This provision is conducted in two stages, among others; The first stage will be conducted at the Caritas-PSE of Manado Diocese level, with participants consisting of the Director of Caritas PSE Manado and the staff, and key figures as representatives of the people being served. The aim of this activity is for the evaluators to get feedback from Caritas PSE of Manado Diocese and representatives from the people served on the findings and recommendations from the field. It is important to give this feedback before the evaluator starts writing the report. The second stage will be conducted at the Caritas Indonesia Office after all fieldwork is completed.

• Final Report

The final report includes an analysis of the data and information collected related to the Relevance, Effectiveness, Efficiency, Participation, Impact and Sustainability of the EA06/2021 Project.

EVALUATION METHOD

1. 1. 1. Reviewing documents related to the planning, implementation, and evaluation of the EA06/2021 Project.
      2. Collecting stories and qualitative data relating to the EA06/2021 Project Relevance, Effectiveness, Efficiency, Participation, Impact, and Sustainability in assisted communities through interviews with key figures, FGDs and observations.
      3. Analyzing data and information to be able to provide an assessment of the Relevance, Effectiveness, Efficiency, Participation, Impact and Sustainability of the EA06/2021 Project.

EVALUATOR RESPONSIBILITY

In general, the responsibilities of the evaluator are as follows:

1. 1. 1. Reviewing documents related to the planning, implementation, and evaluation of the EA06/2021 Project.
      2. Developing the right methods and tools to collect data and information related to EA06 / 2021 Project Relevance, Effectiveness, Efficiency, Participation, Impact and Sustainability.
      3. Coordinating all activities and supplying updates on the progress of the evaluation process to Caritas Indonesia.
      4. Discussing the final draft of the evaluation report with Caritas Indonesia.
      5. Submit the final evaluation report in Indonesian and English to Caritas Indonesia on time.

EVALUATION SCHEDULE AND EVALUATOR CAPACITY REQUIRED

• Implementation Schedule

This external evaluation activity will be conducted between 10 June – 9 July 2022. The final report will be submitted on 15 July 2022.

• Evaluator

To obtain optimal results, this project evaluation requires the following personnel:

1. 1. 1. 1. 1. 1. Having experience, knowledge, and skills to evaluate empowerment programs and programs of disaster emergency response especially focuses on shelter construction and CMDRR activities for affected communities.
               2. At a minimum level, know and understand issues and community strengthening programs, community management programs, community resilience, and disaster risk reduction.
               3. Know and understand the specifics of Caritas’ work and services.

EVALUATION PRODUCT

The result expected from the evaluator is a final report written in Indonesian and English which has detailed findings and conclusions from the evaluation, accompanied by realistic recommendations that are useful for future learning for Diocese Caritas and Caritas Indonesia. The report should include both quantitative and qualitative information including the following structure.

REPORT STRUCTURE

1. 1. 1. Cover page (1 page)
      2. Abbreviation (1 page)
      3. Acknowledgement (1 page)
      4. Executive summary (1-2 pages)
      5. List of content (1 page)
      6. List of figures (1 page)
      7. Introduction: (1-2 pages)
      8. Evaluation objective (1 page)
      9. Evaluation methodology (1 page)
      10. Project setting (1-2 pages):
          • Project context and framework conditions
          • Project goals and objectives
          • Project strategy/approach
          • Description of project activities. Was there a difference between the planned activities and those implemented? Were there any lobbying or networking activities?
          • Description of target groups. How many were reached through the activities?
          • Organisational structure. How does project administration work?
      11. Description of the cooperation with Caritas Internationalis and CIMOs members.
      12. Evaluation findings (max 20 pages)
      13. Relevance
      14. Impacts
          • Effectiveness
          • Efficiency
          • Participation
          • Sustainability
      15. Recommendation (1-2 pages)
      16. Evaluator’s biography

APPLICATION PROCEDURE

Interested candidates /parties should submit their application material by 30^th May 2022:

• • • A cover letter briefly outlining relevant experience with this type of work.
    • A curriculum vitae with details of two professional references
    • An example of a relevant piece of written work.
    • A proposal outlining.
    • The proposed approach, methodology and tools for the consultancy,
    • Proposed Budget – including daily rate and all costs in relation to international and domestic travel, meals, accommodation, and other expenses related to the consultancy, and
    • Timeline and Availability to undertake the consultancy.
    • The application shall be submitted to the following email: hr-admin@karina.or.id cc dini21@karina.or.id before the deadline