16 Sep [VACANCY]: External Evaluator
TERM OF REFERENCE EXTERNAL EVALUATION
|Program Name, Location||:||The Resilience of the Local Communities in the West Kalimantan Sub Region. (Umbrella Sub Region West Kalimantan 3rd Year 2021)|
|Program Duration||:||1 June 2019 – 31 Dec 2021|
|Program Sector/s||:||Income Development and Disaster Risk Management|
|Implementing Agency||:||Caritas Indonesia|
|Funding Source||:||Caritas Australia (CPC)|
|Total Budget||:||AUD 310,000|
|Local Partners||:||Archdiocese of Pontianak, Diocese of Sintang, Diocese of Ketapang|
In 2019 the Archdiocese of Pontianak, Sintang Diocese and Ketapang Diocese undertook a process to design a joint project in the West Kalimantan Sub-region to address threats posed by deforestation which make inland communities more vulnerable.
The main objectives of the project are as follows:
- Strengthen the capacity of local governments and communities in DRR through facilitating community-level DRR
- Assisted community members have improved income through sustainable income-generating activities
- Local governments and communities adopt new behaviours to improve preparedness and response to natural disasters caused
The project focuses on strengthening capacity the capacity and function of local communities and government to develop and adopt DRR strategies so that communities are able to respond to shocks. Another primary outcome is to improve the income of community members through sustainable income-generating activities. The project aims to improve livelihoods by relying on farming as an income-generating activity while also diversifying livelihood activities by, for example, establishing small businesses such as village eco- or agro-tourism.
The total number of beneficiaries in this program is 3363 people (949 direct beneficiaries, 2414 indirect beneficiaries) located in 3 Caritas Sub Regions of West Kalimantan, Caritas PSE Sintang Diocese, Caritas Ketapang, Caritas Archdiocese Pontianak. This program is implemented in 2 assisted areas of the Caritas Sub Region, West Kalimantan. Caritas PSE Sintang Diocese assisted 2 communities in Tibu Mandau & Pelimping villages, Caritas Ketapang Diocese assisted 2 Communities in Kualan Hulu & Balai Pinang villages, Caritas PSE Archdiocese Pontianak assisted 2 Communities in Angan Tembawang & Sungai Bening villages.
OBJECTIVES OF THE EVALUATION
The purpose of this evaluation is to assess the performance of the project and capture project achievements, challenges, and best practices to inform future similar programming. The evaluation will offer a learning aspect for all stakeholders, and identify key lessons learned, challenges and the flexibility of the programme to adapt and respond to the changes and sustainability of Technical Vocational Education and Training in Lakes state.
The specific objectives of this external evaluation are:
- To assess and document the impact and changes that the project has brought about in the lives of the participants and their communities.
- To identify and assess key lessons learned, challenges, and draw recommendations for the future programming
KEY EVALUATION QUESTIONS
A detailed Evaluation Plan will be prepared by the Evaluation Team in consultation/ collaboration with Caritas Australia and Caritas Indonesia. The Evaluation Plan will set out a participatory process that will enable assessment of the impact of specific project objectives, using the indicators from the project document. In order to investigate the key questions for the evaluation, the Evaluation Plan will detail specific research questions, data sources, and data collection methods and tools.
- How relevant were the objectives and activities, implemented by the project? How do beneficiaries perceive the relevance of the project?
- To what extent was the project able to adapt and provide an appropriate response to context changes and emerging local needs, and the priorities of beneficiaries?
- What significant changes, both positive and negative, intended and unintended, have occurred, and what has been the contribution of the project towards these?
- To what extent have the objectives been achieved?
- What are the main factors influencing the achievement or failure of the project objectives?
- Was the program implemented in an economical and cost-conscious manner? What is the evidence to show that the program was implemented efficiently?
- To what extent was everyone served actively involved in all stages of program implementation?
- To what extent are the benefits of the program likely to continue after implementation ends?
- What are the main factors that affect the achievement or not achievement of program sustainability?
- Caritas-PSE Archdiocese of Pontianak:
- The village of Sungai Bening, sub-district of Sajingan Barat, district of Sambas, province of West Kalimantan
- The village of Angan Tembawang, sub-district of Jelimpo, district of Landak, province of West Kalimantan
- Caritas Diocese of Ketapang:
- The hamlets of Kayu Bunga and Balai Berkuak, in the village of Balai Pinang, sub-district of Simpang Hulu, district of Ketapang, province of West Kalimantan.
- The hamlets of Setontong and Sekantak, the village of Kualan Hilir, sub-district of Simpang Hulu, district of Ketapang, province of West Kalimantan.
- Caritas-PSE Diocese of Sintang:
- The hamlet of Tibu Mandau, village of Air Nyuruk, sub-district of Ketunggau, district of Sintang, province of West Kalimantan.
- The hamlets of Bubur Nyala and Pelimping Baru, village of Pelimping, sub-district of Kelam Permai, district of Sintang, province of West Kalimantan
- The preparation stage includes preparatory meetings between evaluators, Caritas Australia and Caritas Indonesia (program team), preparation of the Evaluation Plan, review of documents related to planning, implementation and evaluation of the West Kalimantan Sub-Regional Umbrella Program.
- Activities in the field include Dissemination of evaluation objectives and targets to all staff and former staff of Caritas-PSE Sintang, Caritas Diocese of Ketapang and Caritas PSE of Pontianak Archdiocese who will be involved, collecting related data and information, relevance, effectiveness, efficiency, impact, participation and sustainability of the Program.
- Debriefing Session
- Evaluators will hold briefings as soon as they collect data and information in the field. This debriefing is carried out in two levels, among others; The first level will be carried out at the Caritas-PSE Sintang level, Caritas Diocese of Ketapang and Caritas PSE Archdiocese of Pontianak, with participants consisting of the Director of Caritas-PSE Sintang, Caritas Diocese of Ketapang and Caritas PSE Archdiocese of Pontianak and their staff (or former staff) and key figures as a representative of the person being served.
- The goal is for evaluators to get feedback from Caritas-PSE Sintang, Caritas Keuskupan Ketapang and Caritas PSE Archdiocese of Pontianak and representatives of people served on findings and recommendations from the field. It is important to provide this feedback before the evaluator starts writing the report. The second level will be conducted at the Caritas Indonesia Office after all fieldwork is completed.
- Final Report
- The final report includes an analysis of the collected data and information related to the Relevance, Effectiveness, Efficiency, Participation, Impact and Sustainability of the Program.
The external evaluator, in discussion with relevant Caritas Australia and Caritas Indonesia staff, will determine appropriate data sources and methods/tools of assessment. Generally, evaluations will comprise the following discrete research and analysis components
- Desk Assessment. Review project documents of the Umbrella Sub Regio Program in West Kalimantan.
- Field data collection. The external evaluator will visit program sites to collect data and analyse evidence. A variety of methodologies will be employed to collect quantitative and qualitative data, including interviews and focus group discussions with program participants, staff and volunteers. Interview with partner office can be done online.
- Finding analysis and testing. Analyse data and information to be able to provide an assessment of the Relevance, Effectiveness, Efficiency, Participation, Impact and Sustainability of the Umbrella Sub Regio Program Project in West Kalimantan
EVALUATION SCHEDULE AND EVALUATOR CAPACITY REQUIRED
1. Implementation Schedule
The intended time frame for the entire consultancy for evaluation purposes is expected to run 17 days. The desk assessment and completion of the evaluation plan should be completed and shared with Caritas Australia and Caritas Indonesia before deployment to collect field data.
A total of 6 project areas will be visited, out of a total of 6 to have a range of responses. The evaluator will be expected to share their preliminary findings with the implementing partner staff (following field data collection) and the relevant Caritas Australia staff, before completion of their report.
- 2 days desk review of all project documents
- 2 days formation of evaluation plan submitted to Caritas Indonesia and Caritas Australia
- 10 days field data collection in Pontianak, Sintang, and Ketapang
- 3 days written final report, including responding to Caritas Indonesia and Caritas Australia comments and revision.
2. Evaluator required knowledge
The evaluation will primarily be contracted by Caritas Indonesia. Appropriate Caritas Australia and Caritas Indonesia staff will also be part of the evaluation. The external evaluator will be selected based on the following skills, experience, and qualifications.
- Have the experience, knowledge, and skills to evaluate empowerment programs and programs during disaster emergency response, particularly in shelter development and economic recovery programs of affected communities.
- An understanding of community development programs, resilience, and disaster risk reduction issue
- Extensive knowledge of Monitoring, Evaluation and Learning (MEL) theory and practice.
- Demonstrated experience evaluating international development programs using quantitative and qualitative methods and participatory methodologies.
- The ability to communicate with diverse groups of people respectfully and effectively.
- Strong interpersonal and facilitation skills, and cultural and gender sensitivity in working with local communities and a range of stakeholders.
- An understanding of the peculiarities of Caritas’ work and services.
- Fluency in Bahasa Indonesian
- Excellent oral communication skills and the ability to deliver high quality written reports in English.
The following support will be provided to the evaluator:
- Project and partner documentation (including program designs, progress reports, monitoring trip reports) will be provided to the external evaluator by Caritas Indonesia.
- Transport, accommodation, meals, and other logistical considerations are to be coordinated by Caritas Indonesia and the dioceses in consultation with the external evaluator and Caritas Australia.
- Interviews and meetings with program participants, staff, and volunteers will be coordinated by Caritas Indonesia and the diocese in consultation with the external evaluator and Caritas Australia.
EVALUATION EXPECTED DELIVERABLES
An evaluation plan and methodologies will be determined together with Caritas Australia and Caritas Indonesia staff and the consultant by 30 Sept 2021. The evaluation plan will usually include:
- Key considerations
- Qualitative/Quantitative evaluation methods
- Evaluation schedule
- Data Analysis methodology
- Evaluation Criteria, Questions and simplified for the field
- Evaluation Questions, Methodologies, Sources and Tools
- Questions for each stakeholder group
- Other key information specific to this evaluation
A final report of the evaluation submitted to Caritas Indonesia and Caritas Australia for comments and review by 30 Nov 2021, with a final report submitted by 15 Dec 2021. The final report will usually include:
- An executive summary and overall recommendations
- Key findings and analysis of these findings
- Successes/challenges and the reasons for these
- Lessons learned
- Specific recommendations on any aspect of the program, its planning, implementation, management, coordination, and other related factors.
- Where the evaluation covers multiple program partners: Specific and exclusive recommendations for each partner and program (i.e., separate reports or clearly distinct sections)
- Appendices of statistical data, participant interviews, case studies, photos, Videos etc.
- Lists of key informants including partners (PNGOs/Coops/staffs/local governments representatives)
- The length of the report will be determined by the evaluation scope, however should, as a rule, be no longer than 25 pages, excluding appendices.
Importantly the executive summary needs to be of a standard to share across the agency and potentially externally. Note that the executive summary will also be translated for dissemination to program partners and evaluation participants – please state who will be responsible for this translation.
Interested candidates should submit their application material by 25 September 2021. The application should include the following components:
- Updated Curriculum Vitae (CV) including 3 professional references.
- A copy of a valid Identity Card.
- Cover letter transparently summarizing relevant experience in Resilience of The Local Communities for income development and Disaster Risk Management as well as practical experience in planning and conducting project evaluations.
- Track record of conducted studies, research, publication, and references.
- A technical proposal for the evaluation, including the proposal evaluation methodology and work plan.
- Financial proposal underlying the evaluation, including the proposed fee.
- A sample of recent writing (report or similar) relevant to the terms of reference stated above.
- The application shall be submitted to the following email: firstname.lastname@example.org cc email@example.com before the deadline.