Tracking jobs created under the U.S. Recovery Act - when should the attempt at measurement be abandoned? June 16, 2009
Posted by Paul Duignan in : Impact evaluation, Outcomes systems architecture, Attribution, Reporting systems, Outcomes theory & politics, Indicators, Accountability, Using the approach, Doing evaluation more efficiently, Measurement, Outcomes theory & the news, Evaluation planning , 1 comment so farThe default expectation in at least some sections of the U.S. public sector seems to be that it should always be feasible and affordable to both measure and attribute the results of interventions. This is using the term attribution to mean being able to actually demonstrate that a change in an outcome has been caused by a particular intervention rather than being the result of other factors (see here for more on attribution). The recent U.S. Recovery Act is a case in point. While it’s reasonable to start from the position that you should routinely assess the possibility of measuring and attributing changes in outcomes of particular interventions, you can’t start by just assuming that it will always be feasible or affordable to do this. Clinging to such an assumption, where it is untrue, can result in you either measuring an outcome when the data you are collecting is not accurate, or acting as though what you are measuring (even if it is an accurate measurement of a change in an outcome) is demonstrably attributable to a particular program, when in fact it may not be. (more…)
Impact evaluation on full program roll-out versus just on piloting - two paradigms June 10, 2009
Posted by Paul Duignan in : Impact evaluation, Attribution, Outcomes systems architecture, Doing evaluation more efficiently, Using the approach, Easy Outcomes , add a commentI’ve just posted an article on the two paradigms in regard to impact/outcome evaluation and full program roll-out. What this is about is making a distinction between designing an evaluation which can provide impact/outcome evaluation information about full program roll-out versus a paradigm where you do impact/outcome evaluation just on piloting and then in regard to full program roll-out you just make sure that best practice is implemented. I once was involved in the evaluation of an overall program which had over 900 component programs. The way that we went about evaluating it was, in my view, wrong. (more…)
The error of limiting focus to only the attributable June 8, 2009
Posted by Paul Duignan in : Outcomes systems architecture, Attribution, Reporting systems, Indicators, Accountability, Using the approach, Measurement, Easy Outcomes , add a commentI am continuing to develop a set of articles which outline various problems which are often built into the outcomes systems which I see. The one I have just put up is on the Error of Limiting Focus to Only the Attributable. This is where the whole emphasis of a performance management or other outcomes system is just on holding a provider to account for a list of demonstrably attributable indicators (often called outputs, deliverables, or key performance indicators). This often creates problems. (more…)
Over-simplifications in outcomes, monitoring and evaluation June 3, 2009
Posted by Paul Duignan in : Outcomes systems architecture, Attribution, Reporting systems, Outcomes theory, Accountability, Evaluation planning, Using the approach, Doing evaluation more efficiently, Easy Outcomes , add a commentAn evaluation colleague Patrica Rogers commented on an earlier blog posting of mine in which I was claiming that what I am trying to do it to make outcomes, monitoring and evaluation work ‘easier’. She challenged me on that idea and pointed out that often what we are having to deal with is over-simplification in the way people are working with outcomes, monitoring and evaluation. Her comment inspired me to work up an article on over-simplification in outcomes and evaluation and after getting underway with it I realized all of the different ways in which people approach outcomes, monitoring and evaluation with over-simplified approaches and the problems which these cause. (more…)
Unalterable deliverables and program inflexibility June 2, 2009
Posted by Paul Duignan in : Outcomes systems architecture, Reporting systems, Indicators, Accountability, Measurement, Using the approach , add a commentBack blogging now after having been on holiday. Recently I ran into the problem of unalterable deliverables in a project I am involved in. This problem was also mentioned in the UN report on its results-based management system that I blogged about a couple of postings ago. The problem arises where a project is set up and deliverables are set, but where ideally there needs to be some flexibility regarding deliverables as the program develops over time. Sometimes the problem is just a result of the difficulty of changing deliverables. (more…)
Being sector-centric not program-centric in deciding on program evaluation priorities May 22, 2009
Posted by Paul Duignan in : Impact evaluation, Doing evaluation more efficiently, Evaluation planning , 1 comment so farI have been blogging less in the last week or so due to going on holiday - blogging will still be less frequent for a week or so. I was recently involved in assessing a set of projects summaries to help determine which of them should be selected for more intensive evaluation input. This was not being done directly to determine project evaluation priorities, however the exercise reminded me once again of the general issue of how we determine what types of evaluation should be undertaken for particular projects. My blog comments below are about the general issue rather than the particular exercise of project selection I was recently involved in. The set of information we typically use to work out what type of evaluation should be undertaken for a project is information such as the nature of the project, the proposed evaluation questions, and the proposed evaluation budget. I think that we need more than this when determining evaluation priorities. (more…)
Intense analysis of the U.N. Results-Based Management System May 5, 2009
Posted by Paul Duignan in : Outcomes theory, Outcomes systems architecture, Attribution, Reporting systems, Indicators, Accountability, Standards, Using the approach, Doing evaluation more efficiently, Measurement, Outcomes models , add a commentI have just put up an Outcomes Theory Knowledge Base article which is an intense analysis of the United Nation Results-Based Management System. (Its obscure work, but someone has to do it!). The exciting part is that it has let me road-test my new Outcomes Systems Checklist. This now provides a common framework for analyzing any outcomes system - outcomes systems being any system which attempts to identify, measure, attribute or hold parties to account for outcomes or the steps which it is thought lead to them. A 2008 report from the U.N. itself on its Results-Based Management System said that the system was: ‘an administrative chore of little value to accountability and decision-making”.
Visual model of what I’m trying to do with my outcomes work April 28, 2009
Posted by Paul Duignan in : Communicating outcomes models, Outcomes theory, Using the approach, Outcomes models, DoView, Easy Outcomes, Blog info , 2commentsThought that I would apply a taste of my own medicine to my own work, so I drew a visual outcomes model of what it is that I’m trying to do with my work in the outcomes area. It is here. At the top is my high level outcome ‘Make working with outcomes, monitoring, evaluation etc. easier’ and below that is all of the lower-level steps I am using to get to this high-level outcome. I have included hyperlinks out to the various web sites where I am attempting to do the things listed in the lower-level steps.
Paul Duignan, PhD
Outcomes and Evaluation Blog (OutcomesBlog.org)
The single list of indicators problem April 27, 2009
Posted by Paul Duignan in : Attribution, Reporting systems, Outcomes systems architecture, Indicators, Measurement, Accountability, Using the approach , 2commentsMany results management, performance management and monitoring systems suffer from what is called the ’single list of indicators’ problem. I have just put up an article on the Outcomes Theory Knowledge Base regarding this problem (the URL of the article is at the bottom of this blog posting). It arises in situations where there is a demand that an indicator list be high-level (i.e. not at the output level) but at the same time that the list be able to be used to hold a program, organization or other intervention to account. Often one list cannot be used to do both of these jobs. There are four things that can happen in regard to single list approaches, all four create problems and can lead to undermining the credibility of the outcomes system in which they occur. (more…)
Don’t assume that impact evaluation should always be done April 26, 2009
Posted by Paul Duignan in : Impact evaluation, Attribution, Outcomes systems architecture, Doing evaluation more efficiently, Evaluation planning, Easy Outcomes , add a commentImpact evaluation - evaluation which looks at whether changes in high-level outcomes can be attributed to a particular program, organization or other intervention - is a particularly useful type of evaluation when done properly. It clearly tells us what works, and what doesn’t work, and this information can be used in decision-making about which programs should, and should not, be funded in the future. However, particularly at the present time, with all of the enthusiasm for evidence-based practice, many people mistakenly assume that impact evaluation should always be attempted in regard to any program, organization or other intervention. Assuming this is a serious mistake. I’ve just put up an article in the Outcomes Theory Knowledge Base which sets out in detail why it is and the way we should approach assessing when impact evaluation should be done. (more…)