Workshop 8: The evaluation of poverty reduction projects By Sylvia I. Bergh Associate Professor in Development Management and Governance, ISS/EUR Session as part of the minor ‘Global Poverty, Local Solutions’ ISS, 26 October 2017

Topics Nature of evaluation – what do we mean? Functions and use of evaluation Practice of evaluation Methods of evaluation Points of measurement of a program / project A practical exercise

What is evaluation? Discuss with your neighbour, using some examples from your own experience or course topics – 5 minutes

Nature of evaluation Setting standards Making comparisons Measuring outcomes Putting value to experiences / events  We do it all day! “… what distinguishes evaluators from other researchers is that evaluators must place value on their findings” (Alkin 2004:32)

A basic definition What - did the program do and achieve? So what - what difference did it make and for whom? Now what - what lessons did we learn and what’s next?

Formal definition The systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact and sustainability. (OECD/DAC 2002). (pp )

Functions and use of evaluation The question before the evaluation questions: what is the evaluation for? 1. Judgment-oriented evaluation 2. Improvement-oriented evaluation 3. Knowledge-oriented evaluation (Michael Quinn Patton, 1997)

Judgment-oriented evaluation Did the program work? Should it be continued or ended? Were desired outcomes achieved? Accountability – report on the program not to it Criteria of merit to be set; standards of performance Users: funders, external decision makers “Instrumental use” decision or action to follow

Improvement-oriented evaluation What are strengths and weaknesses? What progress towards desired outcomes? What problems and how have these been addressed? What do staff and participants think of the program? Aim: to improve the program Users: program managers, staff, participants Instrumental use: decision or action to follow

Knowledge-oriented evaluation Findings contribute to increasing knowledge Clarification of a model / testing of a theory Generation of lessons learned Input to policy making Aim: Enlightenment or generalization of evaluation findings about principles of effective programs Conceptual use of findings: no direct action or decision expected

Monitoring vs Evaluation Routine, continuous Program objectives and indicators taken as given Tracks progress in inputs and outputs Quantitative methods Does not address causal links Internal management tool to improve implementation Now & then, one shot Objectives assessed vis-à- vis higher goal Tracks results & impacts - looks at relevance Qualitative and quantitative methods Addresses causal links Usually initiated and performed externally, guidance for policy decisions

Practice of evaluation Practical choices needed about: When - mid term, end, ex-post Why - function, purpose What - program/policy What - ‘product’ - process/relations How - methods, indicators, questions Who - external, internal, stakeholders

Methods of evaluation Wide range: from natural science-based to participatory methods Natural science-based: comparison with-without, before-after the programme – counterfactual – qualitative – product first Participatory: large stakeholder / beneficiary involvement - evaluator as facilitator - qualitative – process first

Results chain Insert p 25 fig 2.1 From: Gertler et al, World Bank 2011:25

Step-by-step Evaluation along the results chain: moving gradually from inputs to outcomes/impact Statements about policy effects are not possible without a detailed understanding of results at operational level

Points of measurement of a program / project Impact - long-term sustainable change Outcomes – effect* of program, what beneficiaries do with it Outputs - efforts and activities, initiative on the side of the program Inputs – resources *Effect: tangible and observable change to be verifiable and measurable more than total sum of efforts (Oakley et al 1998:35)

Measurement in evaluation based on Fowler 1997 Points of measurement What is measured? Indicators ImpactChangeDifference from initial situation OutcomesEffectUse of outputs, initiative by beneficiaries OutputsEffortImplementation / activities by project organisation

Example: micro-credit for the rural poor LevelType of indicatorIndicator Goal: Enhance poor people’s participation in rural econ life & improve income and wellbeing Impact indicator Change Purpose: More employment for rural poor through small enterprises Outcome indicator Attitudinal change Activities: Group formation, saving&credit Training product improvement, bookkeeping, marketing Output indicator Resources: (Group) credit Technical assistance to village groups, poor people Input indicator

Example: micro-credit for the rural poor LevelType of indicatorIndicator Goal: Enhance poor people’s participation in rural econ life & improve income and wellbeing Impact indicator Change Decrease of poverty in area: higher incomes, better social services, etc. Purpose: More employment for rural poor through small enterprises Outcome indicator Attitudinal change Rural poor are in control of new resources and use credit and skills for their enterprise Activities: Group formation, saving&credit Training product improvement, bookkeeping, marketing Output indicatorCredit provided to X nr of groups / people X nr of people trained Resources: (Group) credit Technical assistance to village groups, poor people Input indicatorX amount available X TA available

Ruxton diagram

Depicting the U-turn: from problem situation to positive development Claims about attribution to be built up systematically and from below (Oakley et al: 1998:29, 32)

Social Action Program Project Pakistan Social Action Program ( ) to improve social services in Pakistan: Elementary education Primary health care Population welfare Rural water supply and sanitation Multiple interventions, multiple locations Multi-donor assistance - 10% conditioned SAPP = Social Action Program Project

Evaluation of SAPP Focus: Education component Gender equity 2 provinces Situation on education in Pakistan Objectives and activities of the project, see case study

Class exercise on SAPP 1.Read case study (individually) 2.Break down programme information following Ruxton diagram (in small groups): 3.Start at top left and work down the left column 4.Then work your way up through the right column, deciding what to measure in terms of outputs, outcomes, impact