Evaluation

Evaluation at UN-Habitat   

  1. Introduction  

United Nations reform initiatives undertaken during the 2018-2019 period related to the evaluation function included the revision of the Regulations and Rules that govern programme planning, budget aspects, monitoring and evaluation (PPBME, ST/SGB/2018/3 – Regulation 7.4); and the establishment of the United Nations Sustainable Development Group System-wide Evaluation Office to coordinate system-wide evaluations.  The intent of these initiatives is to strengthen the evaluation function of programmes in the UN Secretariat as a learning tool and a management tool to better inform programme planning and reporting on the programme performance and to create roster with qualified internal and external evaluation experts evaluating United Nations’ work (A/72/492, Paragraph 61). 

In line with the UN Secretariat initiatives, UN-Habitat’s ongoing reforms emphasize evaluation for transparency and accountability, demonstrating the results the organization achieves, identifying where improvements are needed and informing evidence-based decision-making.  The following evaluation guidelines are targeted to UN-Habitat Staff and consultants involved in planning, approving UN-Habitat programmes and projects to prepare for evaluation as well as those responsible for managing, conducting, and using UN-Habitat evaluations. 

  1. Policy and Organizational Framework of at UN-Habitat 

Evaluation is integral to the work of UN-Habitat in all its substantive areas and a component of project cycle management.  It is led and coordinated by UN-Habitat’s Independent Evaluation Unit (IEU), established in 2012, as result of separating the monitoring and evaluation functions.  

The UN-Habitat Evaluation Policy (2013), the Revised UN-Habitat Evaluation Framework (2016), and UN-Habitat Evaluation Manual (2018) are the main documents that set forth principles, organizational framework and practice for evaluation planning, preparation, management, conduct and follow-up and use in UN-Habitat. These UN-Habitat tools are aligned with internationally accepted evaluation standards, including UNEG Norms and Standards for Evaluation and OECD-DAC Network on Development Evaluation Evaluating Development Co-operation: Summary of Key Norms and Standards. They are also aligned to operating procedures for the evaluation function within the United Nations Secretariat as outlined in the Secretary-General’s bulletin entitled “Regulations and Rules Governing Programme Planning, the Programme Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation” (PPBME, ST/SGB/2018/3).”   

  1. Purpose and Objectives of the Evaluation Function 

In line with the UNEG Norms and Standards for Evaluation, the following definition of Evaluation is used: 

An assessment, conducted as systematically and impartially as possible, of an activity, project, programme, strategy, policy, topic, theme, sector, operational area or institutional performance. It analyses the level of achievement of both expected and unexpected results by examining the results chain, processes, contextual factors and causality using appropriate criteria such as relevance, effectiveness, efficiency, impact and sustainability. An evaluation should provide credible, useful evidence-based information that enables the timely incorporation of its findings, recommendations and lessons into the decision-making processes of organizations and stakeholders. 

(United Nations Evaluation Group, Norms and Standards for Evaluation,                                                                               (2016, 2017), page 10) 

The main purpose of the UN-Habitat Evaluation function is to enhance development effectiveness, promote proactive engagement in evaluation partnerships and knowledge sharing activities. To support the organization in achieving development effectiveness, the evaluation function has adopted three main objectives: 

  1. To provide a basis for accountability by reporting on use of resources and results achieved to UN-Habitat governing bodies, Member States, donors, implementing partners and beneficiaries;  

  1. To enhance organizational learning where evaluation is a source of evidence about what works, what does not and why. Through incorporation of recommendations and lessons learned into decision-making processes, evaluation improves on-going and future policies, strategies, programmes, projects and processes.    

  1. To promote evaluation culture in organization, where knowledge generated is used by UN-Habitat project and programme managers to effectively plan objectives, outcomes, outputs, activities and allocation of resources. 

In addition, evaluations are: 

  • An important source of evidence of achievement of results, UN-Habitat’s performance and maximizing the impact of UN-Habitat’s contribution. UN-Habitat relies on evaluations to assess performance, identify results achieved, measure effectiveness and determine alternative ways to meet its objectives. 

  • Demonstrate accountability and transparency by sharing evaluation results with key audiences. By building a greater understanding about what UN-Habitat is intending to achieve and how it will achieve it; support is generated from donors, governing bodies and Member States. 

  • Credible and reliable evidence for decision-making on project design, allocation of resources, implementation and improving knowledge of projects. 

  • An agent of change and promote, defend or oppose specific approaches or projects and help shaping opinions. 

2.2   Types of evaluations conducted by UN-Habitat 

UN-Habitat carries out different types of evaluation that can be categorized according to: 

  • When the evaluation is undertaken (timing): ex-ante, mid-term, terminal/ end-term, ex-post evaluation. 

  • For what purpose the evaluation is conducted formative, summative, outcome, impact, meta, real-time evaluation, etc. 

  • What is evaluated: project, thematic, subprogramme, strategy, policy, institutional, country portfolio, programme, project, cluster evaluation, etc. 

  • Who conducts the evaluation: self-evaluation, external, internal, joint-evaluation, etc. 

  • Who manages the evaluation: centralized or decentralized evaluation. 

  • The scope of evaluation: In-depth, review, etc. 

Source: UN-Habitat Evaluation Manual, page 14, section 2.1 Types of evaluations conducted by UN-Habitat,

Regardless of type of evaluation the planning, preparation, management, conduct and use of norms and standards for evaluation and evaluation criteria are the same.  

  1. Financial Resources for UN-Habitat Evaluation Function 

The present revised guidelines require that projects with a budget value of USD300,000 and above should have a budget provision that will enable the project to be evaluated. Guidance on budgeting for evaluation is provided in paragraph 18 of the UN-Habitat Revised Evaluation Framework (2016) and the UN-Habitat Evaluation Manual (UN-Habitat Evaluation Manual, section 3.3 Budgeting for evaluation, page 27)   There is no funding sent aside for corporate evaluation of strategic plans, policies, projects nor programmes that do not have evaluation provisions and budget lines, and include charge for the running of the evaluation function. Evaluation costs cover the following: 

  • Human resources in the IEU 

  • External evaluators/Consultants’ fees  

  • Travel of external evaluator and IEU staff  

  • Editing and translation of evaluation reports  

  • Corporate evaluation training needs  

To enhance the evaluation function of UN-Habitat to include programmatic in addition to project level evaluations and to enhance opportunities for learning on results, the funding modality for the evaluation function should be revised in-line with the UN Secretariat’s efforts to strengthen the evaluation function.   

  1. Consideration of evaluation in design of projects  

Programmes and projects must be designed with provision for evaluation as part of the intervention’s life cycle. When designing a new project/programme, the Programme/project Manager should: (1) review evaluation results from previous interventions in order to inform the design of the new programme/project; (2) plan the timing of the evaluation (availability, mid-term, final evaluation or impact evaluation) and ensure sufficient budget provision for evaluation; and 3) review evaluation plans for coordination and coherence of evaluation activities. It is important that planning for evaluation takes place at the project design stage because: 

  • The design of an intervention affects how it can be evaluated; 

  • SMART programme and project outcomes and indicators are fundamental to assessing performance of an intervention in an evaluation; 

  • Monitoring throughout implementation of an intervention is critical to having valid data available for an evaluation. 

Mid-term evaluation is recommended for projects with a budget over USD 1 million and with a duration of four years or more (UN-Habitat Evaluation Manual, page 27) 

Ideally, all programmes and projects should be designed to identify baselines and enable robust monitoring and evaluation. At the approval stage of the project, the Programme Review Committee should ensure the intervention has the following, before it is approved for implementation: 

  • Enough information on background and context; 

  • Clearly identifies problem and solutions to the problem; 

  • Clear and SMART objectives, outcomes and indicators of achievements; 

  • Theory of Change (Section 1.6 Understanding theory of change in results-based evaluation, page 9, UN-Habitat Evaluation Manual) ; 

  • Monitoring and evaluation frameworks; 

  • Adequate budget provision to undertake evaluation.  

  1. Evaluation Process  

The evaluation process follows four phases of:  

  • Planning and preparation phase;  

  • Inception phase;  

  • Data collection phase; and  

  • Report writing and dissemination phase.  

Figure 1 shows key activities and outputs of the evaluation process. Depending the purpose, focus and scope of the evaluation, an evaluation process usually takes 3-6 months, from drafting of the Terms of Reference to the Management Response.   

  1. Planning and Preparation Phase 

Developing the Evaluation Plan 

The evaluation plan of UN-Habitat should be developed annually along with programme budgets and shared with the Independent Evaluation Unit (IEU). This plan should indicate the topic, type of evaluation, estimated cost and time frame of each evaluation. Evaluations of programmes and projects are required and will be scheduled based on their planned closure dates, while thematic and strategic evaluations are proposed by IEU in consultation with UN-Habitat substantive Divisions, Branches and Regional Offices.  The evaluation plans submitted as part of the programme budgets should to be considered estimates and adjustments could be made to respond to changing conditions. Evaluation focal points at branch level and in regional offices as well as the Independent Evaluation Unit can offer advice on how to prepare the evaluation plan (Template 1: Evaluation Plan of Regional Office or Branch, page 24, UN-Habitat Evaluation Manual) 

Preparing for Evaluation  

Evaluation requires careful preparation to ensure it will be of high quality, credible and useful.  Important tasks of the evaluation manager are to: 

  1. Inform and involve interested stakeholders, 

  1. Define the purpose of the evaluation, and 

  1. Establish an evaluation reference group. 

  1. Developing the Evaluation Terms of Reference (TOR) 

Involving stakeholders 

Involving internal and external stakeholders in evaluations is a policy requirement. UN-Habitat implementation partners and other key external stakeholders should be encouraged to actively participate in UN-Habitat evaluations.  

Defining the purpose of the evaluation 

Defining the purpose of an evaluation is one of the most important tasks in the early stage of the evaluation process. A clear purpose is necessary to formulate evaluation objectives and relevant evaluation questions and makes it easier for external evaluators to conduct a useful evaluation. 

The purpose of the evaluation should be formulated in a way that specifies how the information produced by the evaluation is to be used. For example, “The evaluation will be both summative and formative, serving purposes of accountability and enhancing learning. It will support reporting on resources used, results achieved and the way they were achieved by the project to key stakeholders and partners, and enhance their learning, and knowledge for decision-making on future programming and design of projects, up-scaling or replication.”     

CHECKLIST   

 

defining the evaluation purpose 

■ 

Identify possible users of the evaluation, i.e., those groups that are expected to make use of the evaluation process and its results. 

■ 

Ensure that the evaluation purpose is defined through a participatory process engaging interested users of the evaluation. 

■ 

Do not proceed with the evaluation unless a clear purpose is defined.  

 

Establishing an evaluation reference group  

It is good practice to involve key internal stakeholders to ensure that evaluations are relevant to their intended users within UN-Habitat. Such a set up can facilitate institutional support for the evaluation and subsequent ownership, credibility, and quality of the evaluation.  In complex evaluations where several external stakeholders might have significant and varied interests in the intervention and the outcome of the evaluation, formation of a reference group with internal and external members is recommended  would typically include representatives of the donor and implementing partners. Key tasks for a reference group include: 

  • Providing inputs to the Terms of Reference; 

  • Approving the Terms of Reference; 

  • Providing feedback to inception and draft report; 

  • Endorsing the inception report and final report and; 

  • Facilitating the dissemination and use of the results and other follow-up actions. 

Preparing Terms of Reference (TOR)  

The terms of reference (TOR) is an important document in preparing for an evaluation (Section 4.3 Developing the Evaluation Terms of Reference, page 34, UN-Habitat Evaluation Manual). The TOR defines why the evaluation is being undertaken (purpose and objectives), what will be assessed (Scope), how ( design and methods), when it will be conducted (time frame), who is to use it (intended users and how it will be used when completed. Time and effort spent in preparing a good TOR has big returns in terms of quality, relevance, and usefulness of the evaluation to be conducted.  

Box: What Goes into the Evaluation Terms of Reference 

Ambiguous TOR, resulting in misunderstandings between the evaluation manager and the consultant, is a common problem that tends to be difficult to solve if not tackled during the planning and preparation phase.  

CHECKLIST 

for the formulation of the TOR 

   

■ 

Useunhabitat.org/evaluation “evaluator reports”as a general source of ideas for the formulation of the TOR.All UN-Habitat evaluation reports include the TOR as Annex 1. 

■ 

Review of the TOR is mandatory for all external evaluations of UN-Habitat. Make sure that theIEU isprovided sufficient time to review the TOR.  

Selection and recruitment of evaluation consultants 

UN-Habitat contracts out its evaluations. Evaluators are selected through an impartial process that is intended to identify candidates best suited to deliver a credible and evidence-based quality evaluation report. This process is guided by UN rules of procurement.   

Evaluators can be national or international, individual(s) or firms. When an evaluation is conducted by a team, one member should be designated team leader, and will be overall responsible for the deliverables. The skills and other qualifications needed by the evaluators vary from case to case, but experience in results-based management and evaluation are essential. 

The recruitment of consultant should be initiated well before the evaluation expected to start. The supply of experienced and qualified evaluators is limited, and evaluators should be contracted well in advance. Recruiting evaluators at the last minute may considerably reduce prospects for obtaining the best qualified candidate and a good quality evaluation report. 

 

CHECKLIST 

 

for recruiting consultants 

 

■ 

Formulate required skills and qualifications of evaluators in the TOR. 

■ 

Make sure that evaluators have the required skills and that they satisfy the requirements of independence. 

■ 

 

 

■ 

Should be registered in the Inspira Roster of Consultants 

  

All evaluation consultants should be provided with a copy of the Code of Conduct for conducting evaluations in the UN system, sign an agreement form, as well as complete a declaration of interest form.   

 

4.2 Inception Phase: Preparation of the inception report 

The inception phase is where the TOR made operational through an inception report prepared by the evaluator. As good practice, UN-Habitat normally requires that the inception report is endorsed by the Evaluation Reference Group before the evaluation can proceed to the next phase. 

The scope and focus of the inception report vary from case to case, but the following are standard topics (Table 6: Checklist for Proposed outline of the Inception Report, page 49, UN-Habitat Evaluation Manual): 

• Proposed methods and analysis frameworks, including theory of change and interpretation of evaluation questions 

• Data collection procedures and sources that considers all stakeholders, including gender, youth and persons with disabilities 

• Results of an evaluability assessment or scanning 

• Review of documentation and scoping conducted 

• A work schedule with associated activities, deliverables, timetables, roles and responsibilities, as well as travel and logistical arrangements for the evaluation. 

4.3 Data collection phase 

Upon approval of the inception report, the evaluation team can begin collecting data. Collection of data collection should follow the approach outlined in the inception report. The project manager should provide logistical support to the evaluator to facilitate data collection.  

The project manager should ensure the evaluator takes the following into account: 

• If a theory of change or baseline does not exist, the evaluator can reconstruct these through stakeholder workshops. 

• Cultural aspects that could impact the collection of data should be analysed and integrated into data collection methods and tools 

• There should be adequate time for testing data collection tools 

• The limitations of the data should be understood and generalizing findings should be avoided unless a strong random sample was taken 

• Use multiple methods of data collection and analysis (triangulation), which allows for validation across the multiple methods, from a wide range of stakeholders (donor, partners, beneficiaries) and other sources 

• Validate findings through engagement. 

4.4 Report Writing and Dissemination Phase 

The main aim of the evaluation report is to convey the results of the evaluation in a way that corresponds to the information needs. Important tasks of the evaluation manager are to:  

  1. Examine the draft report against the contractual obligations; 

  1. Make sure that the draft report fulfils the agreed presentation format; 

  1. Examine the content in terms of quality and response to the TOR; 

  1. Ensure that the language and style is appropriate; 

  1. Make sure that key stakeholders are invited to review the draft report; 

  1. Make sure comments are considered integrated by the evaluator as appropriate; 

  1. Disseminate the results of the evaluation to all interested parties, in particular the intended users. 

Checking the format of the draft report 

UN-Habitat has standard format for evaluation reports. The format should be used unless there is good reason for doing otherwise. The evaluators should consult with UN-Habitat’s evaluation manager before adopting a different framework. The format is summarized in Box 1.  

Box 1Standard format for evaluation reports 

Title and Opening page 

 

Executive Summary 

Summary of the evaluation, with particular emphasis on main findings, conclusions, lessons learned and recommendations. 

Introduction and purpose, objectives and scope of evaluation 

Concise information about intervention background and context, mandate for the evaluation, overall objectives, purpose and scope of the evaluation. It should specify in brief how and by whom the evaluation is intended to be used, and describe the outline of the report. 

Description of intervention to be evaluated and context

Description of the evaluated intervention, and its purpose, logic model, history, organisation and stakeholders. 

Evaluation Methodology and limitations 

Methodology for data gathering and analysis 

Findings 

Factual evidence, data and observations that are relevant to the specific questions asked by the evaluation on achievement of outcomes and assessment of performance by evaluation criteria including social inclusion issues. 

Conclusions 

Assessment of the intervention and its results against given evaluation criteria, standards of performance and policy issues. 

Lessons learned 

Generalizations based on evaluation experience that are likely to have a potential for wider application and use. 

Recommendations 

Actionable proposals to the evaluation’s users for improved intervention cycle management and policy. 

Annexes 

Terms of reference, list of persons interviewed, project budget, bibliography, etc. 

Evaluation managers should try to make sure that the evaluation report conforms to the TOR and generally accepted quality standards for evaluations, yet respecting the evaluators’ independence (Table 8: Checklist for Outline of UN-Habitat Evaluation Report, page 53, UN-Habitat Evaluation Manual).  

The draft evaluation report should be shared with Evaluation Reference Group for review and comments received forwarded to the evaluator for consideration and revision of the evaluation report. The Evaluation Reference Group will endorse the final draft of the evaluation report.  

CHECKLIST 

 

examination of the draft report 

 

■ 

Check that the report meets the formal requirements stipulated by the TOR and inception report.  

■ 

Ensure that the report conforms to the agreed format for evaluation reports.  

■ 

Check that the report provides a satisfactory response to the evaluation questions in terms of clear account of evaluation methods, sources of information, evidence-based findings, justified conclusions, recommendations supported by findings and conclusions and impartiality. 

Dissemination of evaluation reports 

The dissemination of the evaluation report must be tailored to the needs and interest of different audiences. Some users may require information specifically targeted evaluation findings for decision-making with regard to the evaluated intervention. Others may seek to apply the lessons learned from the evaluation to other interventions and circumstances. Still others merely have an overall policy interest in the evaluation.  

 

CHECKLIST 

 

for dissemination of evaluation 

reports 

 

■ 

Discuss with the Evaluation Reference Group and relevant stakeholders to whom, when and how the results of the evaluation should be disseminated, and implement the dissemination plan accordingly. 

■ 

Consider a range of dissemination options, including meetings, seminars, workshops, social media, and any other kind of relevant and effective communications modalities. 

■ 

Make sure that the dissemination is tailored to the specific interests, information requirements and local context of audiences. 

■ 

Focus dissemination efforts on the intended users of the evaluation, and other groups that can be expected to make effective use of the evaluation. 

■ 

 

■ 

Make a list of people and institutions that should receive the evaluation report, and dispatch printed or electronic copies as soon as possible. 

Have the evaluation report published on UN-Habitat’s evaluation website: www.unhabitat.org/evaluation   

6.Use of evaluations  

Evaluations provide information that, when internalized, becomes knowledge and promotes learning and influence management. When the evaluation report is finalized, the evaluation manager should:  

  1. Make sure that a management response to the evaluation repot is prepared by Management; 

  1. Facilitate uptake of lessons learned. 

Formulation of Management Response 

The management response is the responsibility of the management team for the evaluated intervention (Template 4: Management Response and Action Plan, page 67, UN-Habitat Evaluation Manual). A focal point to coordinate the preparation of the management response and action plan should be appointed. The Independent Evaluation Unit facilitates the preparation of management responses of all corporate and centralized evaluations conducted by UN-Habitat. It also tracks implementation of recommendations on six-monthly basis.   

Key Documents (Can we provide examples of good evaluation?)

UN-Habitat Evaluation Policy (2013)

Revised UN-Habitat Evaluation Framework (2016)

UN-Habitat Evaluation Manual (2018)

UNEG Norms and Standards for Evaluation

OECD-DAC Network on Development Evaluation Evaluating Development Co-operation: Summary of Key Norms and Standards

Regulations and Rules Governing Programme Planning, the Programme Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation” (PPBME, ST/SGB/2018/3)

UN-Habitat’s evaluation website

Focal Points