A Systematic Outcomes Analysis framework for psychotherapy evaluation February 12, 2008
Posted by Paul Duignan in : Doing evaluation more efficiently, Outcomes systems architecture, Systematic Outcomes Analysis, Outcomes models, Easy Outcomes, Evaluation planning, DoView , 1 comment so far
In my last blog posting (which you should read before this one) I talked about using Systematic Outcomes Analysis to define the basic tasks one needs to do in quality assurance, monitoring and evaluation and how this can avoid the need for a protracted theoretical discussion about the difference between quality assurance and program evaluation. I was using the example of an illustrative Systematic Outcomes Analysis framework I set up based on an outcomes logic model in regard to psychotherapy which I’ve posted on the Outcomes Models site. Here’s the PDF of the DoView file. Using the Systematic Outcomes Analysis approach, indicators and evaluation questions are mapped onto the outcomes logic model (indicators are marked with a yellow icon and evaluation questions with a green circular icon). This blog posting looks in more detail at ways stakeholders can use such a framework once it’s been developed. (more…)
Avoiding the question: Defining quality assurance versus program evaluation February 12, 2008
Posted by Paul Duignan in : Using the approach, Doing evaluation more efficiently, Outcomes systems architecture, Systematic Outcomes Analysis, Outcomes models, DoView, Easy Outcomes, Uncategorized , 1 comment so far
Sometimes it’s more useful to avoid initially answering a question that’s posed in a particular way because there’s a better way of addressing the concern that lies behind the question. Such is the case if you’re ever asked to define the difference between quality assurance (or monitoring) and program evaluation.
Seeing the question as a theoretical one and attempting to find a definition which works has some similarities to the situation where you’re building a house and someone keeps wanting you to stop and define, from a theoretical point of view, the difference between the kitchen and the dining room. Now, some people do stuff in the dining room that others do in the kitchen, and some do stuff in the kitchen that others do in the dining room. Still other people don’t really have any theoretical problems because they have a kitchen/dining area where they do both kitchen and dining room stuff.
A more fruitful way of working with the question of the difference between quality assurance (or monitoring) and program evaluation is to attempt to identify all of the stuff (tasks) that you would do under each of these. Once you’ve done that, you can then decide whether or not you need to spend a lot of time defining the difference between the two if everybody concerned is clear about which of the underlying tasks are, and are not, being done by whom. (more…)
Reporting on outcomes to multiple bodies with different outcomes structures November 14, 2007
Posted by Paul Duignan in : Accountability, Outcomes systems architecture, Using the approach, Outcomes models, Easy Outcomes, DoView , add a commentIn a workshop the other day the issue arose of how you deal with the situation where you have to report to a number of different outside organizations on your outcomes. Now, this is not much of a problem where the outside organizations don’t make any implicit or explicit demands on how you report. The way to proceed in such cases is to simply develop your outcomes model and report back to them on it.
However, with more and more organizations thinking in terms of outcomes, they are starting to have outcomes structures of various types themselves. (more…)
New outcomes models (program logics) put up on outcomesmodels.org October 17, 2007
Posted by Paul Duignan in : Using the approach, Outcomes models, DoView , 1 comment so far
We’ve just put a number of new outcomes models (program logics) up on the outcomesmodels.org site. We’re planning to continue adding to the outcomes model collection at outcomesmodels.org for two reasons. The first is to give examples of how models can be drawn in DoView. The more examples people see, the easier they find it becomes to build DoView models. The second is to allow people to grab either the whole of one of these models or just some of the slices from one or more models to use as a start in building their own models. DoView’s unique modular approach using multiple diagram slices makes it very easy to do this. (more…)
Beware lumpers, splitters and slice globblers when you’re building outcomes models October 15, 2007
Posted by Paul Duignan in : Using the approach, Standards, Outcomes models, DoView , add a comment
When you’re drawing outcomes models (program logics, strategy maps, means-ends diagrams, results chains etc), using DoView or other software there’re a few things which will come up on a regular basis. The first of these is the personality difference between lumpers and splitters who are present in the room. Lumpers, obviously want to lump and splitters, well, they just want to split. So if you’re working on an outcomes model which at a high level includes, say, social and economic outcomes, a lumper will want a single outcome which goes ‘complete social and economic whatever’. A splitter, on the other hand, will want to have two separate outcomes - a ’social whatever’ and an ‘economic whatever’ outcome. Neither of them is right or wrong, although in the first instance I often let the splitter have their way because outcomes or steps can always be combined at a later stage if needed. (more…)
Clarifying program outcomes to reduce ‘talking past each other’ about program effectiveness September 27, 2007
Posted by Paul Duignan in : Outcomes theory & the news, Outcomes models, DoView , 1 comment so far
This morning when casually looking through old Washington Post articles which make reference to the General Accountability Office (GAO). (Hey: people have all sorts of hobbies and no one challenges them about it - beer tab collecting etc.). I came across an article about DARE - the Drug Abuse Resistance Education program. This is a program which puts law enforcement personnel into schools to talk to school children about drugs. The wisdom of funding DARE and other drug education in schools is the source of endless debate in evaluation and public health circles as important summaries of the evidence (such as the 2003 GAO report) referred to in the article fail to find evidence of the program having an effect on illicit drug use. (more…)
Avoid being an outcomes model ‘Go-Between’ September 26, 2007
Posted by Paul Duignan in : Doing evaluation more efficiently, Using the approach, Standards, Outcomes models, DoView , add a comment
A while ago a colleague recounted to me how they’d ended up pulling out their hair because they found themselves in a ‘Go-Between’ role when drawing an outcomes model (also called program logics, results chains, strategy maps, ends-means diagrams). You need to try to avoid this at all costs, although when dealing with high level stakeholders it’s often not easy to do so. I found myself in this role on a major project a while ago and I certainly didn’t enjoy it.
(more…)
Slicing up the world September 25, 2007
Posted by Paul Duignan in : Outcomes models, DoView , 2comments
The other day I was talking to Laurie Porima, an evaluation colleague, about building outcomes models with DoView outcomes software. We started discussing the best way of dividing an outcomes model (program logic, results chain, strategy map, ends-means diagram) up into sections. Obviously the real world of causality is not divided up into tidy conceptual pieces suitable for humans to quickly grasp. Many traditional approaches to outcomes models do not have to grapple with this problem because they limit themselves to a single printed page. There’s no question of having to divide the model up in any particular way. However if you allow your models to be larger than a single page - which seems to me to be essential if we are to develop models which can even start to do justice to the causal complexity of the world we’re trying to operate in - you need to think about how you can visualize this. (more…)
What’s the Easy Outcomes site all about? September 20, 2007
Posted by Paul Duignan in : Doing evaluation more efficiently, Evaluation planning, Easy Outcomes, DoView , add a commentI’ve realized that I’ve not put up a post about the Easy Outcomes site even though it’s been up for a few weeks. Easy Outcomes takes the Systematic Outcomes Analysis approach and applies it using DoView outcomes software. We developed it in response to feedback that the Systematic Outcomes Analysis site is a bit intense for those who just want to work out how to develop a good evaluation plan, but don’t want to get buried in the detail of the approach too soon. (more…)
New Systematic Outcomes Analysis case study - evaluating an academic research rating scheme September 20, 2007
Posted by Paul Duignan in : Doing evaluation more efficiently, Systematic Outcomes Analysis, Evaluation planning, DoView , 1 comment so farI’ve recently put up the link to another case study on the Systematic Outcomes Analysis site. This is a report detailing an evaluation and monitoring plan for the Performance Based Research Fund (PBRF). The PBRF is a national academic research output assessment system and the monitoring and evaluation framework was produced for the government organization responsible for the tertiary sector - the Tertiary Education Commission.
The report is a good example of a comprehensive use of Systematic Outcomes Analysis (in the report the approach went by one of its earlier names - REMLogic). One of the interesting things about the PBRF is that it’s an evaluation system in its own right and developing an evaluation framework for an evaluation system presents an interesting technical challenge. I’ll put up a blog post about that challenge some time in the future. (more…)