Checklists and evaluation
Often in evaluation, we are asked to evaluate projects and programmes from several different perspectives: the end user, the implementer or that of an external specialist or “expert”. I always favour the idea that evaluation is representing the *target audiences* point of view – as is often the case in evaluating training or communications programmes – we are trying to explain the effects of a given programme or project on target audiences. However, often a complementary point of view from an “expert” can be useful. A simple example – imagine if you making an assessment of a company website – a useful comparison would be comparing the feedback from site visitors with that of an “expert” who examines the the website and gives his/her opinion.
However, often opinions of “experts” are mixed in with feedback from audiences and comes across as unstructured opinions and impressions. A way of avoiding this is for “experts” to use checklists – a structured way to assess the overall merit, worth or importance of something.
Now many would consider checklists as being a simple tool not worthy of discussion. But actually a checklist is often a representation of a huge body of knowledge or experience: e.g. how do you determine and describe the key criteria for a successful website?
Most checklists used in evaluation are criteria of merit checklists – where a series of criteria are established and given a standard scale (e.g. very poor to excellent) and are weighed equally or not (e.g. one criteria is equal or more crucial than the next one). Here are several examples where checklists could be useful in evaluation:
- Evaluating an event: you determine “success criteria” for the event and have several experts use a checklist and then compare results.
- Project implementation: a team of evaluators are interviewing staff/partners on how a project is being implemented. The evaluators use a checklist to assess the progress themselves.
- Evaluating services/products: commonly used, where a checklist is used by a selection panel to determine the most appropriate product/services for their needs.
This post by Rick Davies actually got me thinking about this subject and discusses the use of checklists in assessing the functioning of health centres.
Glenn
Relationship measurement – hype?
There is an interesting debate on measuring relationships going on Don Bartholomew’s Metricsman blog – Don is saying that he thinks relationship measurement will not be the “next big thing” in measurement – and there are some interesting comments by K D Paine, M Weiner and J Grunig – all well known in this area.
I think it’s interesting to note that the financial world is advocating the measurement of relationships (an “intangible asset” as they call it) and would like to see this included in future company reporting as I’ve written about before. “Next big thing” or not – relationship measurement will grow in importance – in my humble opinion.
Glenn
Cause and effect – goal?
I recently came across a short article in a London newspaper which I summarise as following:
On Friday 21 September, Jose Mourinho, coach of the UK Chelsea football team resigned. On this day, a jump was seen in the number of people visiting a certain job vacancies website. It was proposed by the website marketing team that this jump was due to people learning of Mr Mourinho’s resignation and prompting them to think about changing careers and looking for a new opportunity – thus the rise in number of visits to the job vacancies website.
Doesn’t that strike you as a slightly spurious claim of cause and effect? They appear to be connecting the unrelated and proposing a series of changes that seem slightly unlikely: 1) a change to knowledge (knowing that Chelsea’s coach resigns) to 2) change to attitude (I should change jobs) to 3) behaviour (I actively look for a new job). Possibly of even more concern is that there are no alternative explanations offered as to the jump in website visitors. We can imagine several alternative explanations:
– On Fridays there is always a jump in website visitors (like patterns often seen in purchasing or visits to museums).
– A publicity campaign, a change is site referencing or link campaign drove more visitors to the website on that day.
– Or the more mundane, a technician installed a new monitoring software on the website that led to a more accurate or inaccurate reporting of website visitors.
In addition, no comparison was given to traffic on other job vacancies websites or even global statistics of web traffic (perhaps all sites experienced a jump on 21 September?)
Thinking it through, I think it’s safe to say that there is not enough evidence to point to an association between the resignation of Chelsea’s coach and the jump in visitors to a job vacancies website. The cause (resignation) does seem very distant from the effect (people seek to change jobs). We can point to other cases where an association does seem to exist between a public event and a particular change in behaviour as I’ve written about before: the effect of a reality TV program on the number of people willing to donate an organ and the effect of Kylie Minogue’s treatment for breast cancer on the number of young women taking a scan appointment. In both cases, an association between the cause and effect could be demonstrated and alternative causes were ruled out.
Glenn
Sharpening the focus on measurement
It is often difficult to get organisations away from simply measuring “outputs” – what is produced – to measuring “outcomes” – what are the effects of outputs.
Funny enough, many organisations want to go from the very superficial measuring of output (e.g. how many news articles did we generate) to the very in-depth measuring of impact (e.g. the long term effect of our media visibility on audiences). Impact is feasible but difficult to measure, as I’ve written about before. However, instead of focusing on the two ends of the measurement scale, organisations would perhaps be wise to focus on “outcome” measurement.
I think this quote from a UN Development Programme Evaluation Manual (pdf) sums up why outcome is an appropriate level to measure for most organisations:
“Today, the focus of UNDP evaluations is on outcomes, because this level of results reveals more about how effective UNDP’s actions are in achieving real development changes. A focus on outcomes also promises a shorter timeframe and more credible linkages between UNDP action and an eventual effect than does a focus on the level of overall improvement in people’s lives, which represent much longer-term and diffuse impacts .”
The notion of the shorter timeframe and more credible linkages is certainly appealing for many organisations considering their focus of evaluation.
Glenn
Impact – how feasible for evaluation?
As I mentioned in an earlier post, people often confuse “impact” with “results”. Is it possible to measure “long term impact” of projects? It is, however for most projects it is unrealistic to do so for two reasons: time and cost.
To evaluate impact, you would usually need to wait some 12 months after the major elements of a project have been implemented. Many organisations cannot simply wait that long. In term of costs, an impact study requires a triangulation methodology that uses various quantitative and qualitative research methods which could be costly. However, if time and cost are not issues, an impact evaluation is possible, keeping in mind the following points:
Was the impact desired defined at the beginning of the project?
For example, greater organisation efficiency; change in the way a target audience and/or an organisation behaves; or improvements in how services for a given audience are managed?
What have been the other elements influencing the impact you want to measure?
Your project cannot be viewed in isolation; there must have been other factors influencing the changes being observed. Identifying these factors will help you to assess the level of influence of your project compared to other factors.
Do you have a mandate to measure impact?
When assessing impact, you will be looking at long term effects that probably go outside of your own responsibilities and into the realms of other projects and units – you are looking at an area of the wider effects of your organisation’s activities and this needs to be taken into consideration. For example, if you are looking at the longer term effects of a training program, you would want to look at how individuals and the organisation as a whole are more efficent as a result of the training. Do you have the political mandate to do so? – As you may discover effects that go way beyond your own responsibilities.
Evaluating impact is a daunting but not impossible task. For most projects, it would be more realistic to focus on measuring outputs and preferably outcomes – and think of short term outcomes as I have written about previously.
Glenn
Impact or results?
When speaking of achieving objectives for a project, I’ve heard a lot of people speak of the “intended impact” and I’ve read quite some “impact reports”. I know it’s a question of language, but often people use the word “impact” when in fact they should use the word “results”. Impact in the evaluation field has a specific application to long term effects of a project. The DAC Glossary of Key Terms in Evaluation and Results Based Management (pdf) produced by the OECD contains the most widely accepted definitions in this field. Impact is defined as:
“Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended”.
And “results” is defined as
“The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention”.
Consequently I believe that when we produce a report that shows media visibility generated by a project, this is a short term output and should be called “results” rather than “impact” which applies to more long-term effects.
Glenn
New Blog: Measurement Standard
Bill Paarlberg and K. D. Paine have a relatively new blog online: the measurement standard blog – it’s a handy summary of their regular newsletter of the same name which focuses on PR measurement. There are plenty of interesting posts on various topics such how does PR measurement take news timing into account and measuring success of small organisations.
Glenn
Changing behaviour – immediate responses
Adding to what I wrote about last week concerning measuring behaviour changes that result from communication campaigns – and why I recommend to consider looking at immediate responses (or “outtakes”) as an alternative to long-term changes – I can see parallels in areas other than in campaigns.
As you may know, a favourite of mine is measuring the impact of conferences and meetings. Industry conferences are traditionally sold as being great places to learn something and network, network – and network. But I’m always surprised when attending such conferences at how organisers, if they measure something, focus on measuring the reactions to the conferences, usually in terms of satisfaction. No attempt is made to measure immediate changes to behaviour (such as extending a network) or longer term behaviour or impact in general.
But it is certainly possible, this diagram (pdf) illustrates what I did to measure immediate and mid-term changes to behaviour following a conference (LIFT). Despite the limitations of the research as I explain here, I was able to track some responses following the conference that could be largely contributed to participating in the conference – such as meeting new people or using new social media in their work. One year after the conference, participants also provided us with types of actions that they believed were influenced largely by their participation. Actions included:
– launching a new project
– launching a new service/product
– establishing a new partnership
– Initiating a career change
– Invitations for speaking engagements
Some of these actions were anticipated by the conference organisers – but many were not. It shows that it can be done and is certainly worth thinking about in conference evaluation.
Glenn
Changing behaviour – takes a long time?
In an interesting post by Tom Watson on the Dummyspit blog, he writes about the difficulty of changing behaviour through PR campaigns – in this case to encourage better eating habits in UK schools (staying away from hamburgers for example). He also notes the rather depressing statistic that PR campaigns normally have only a 0.04% success rate of changing behaviour! In setting campaign objectives, I encourage organisations to be realistic about the targets they set – for example, a modest 2 – 10% behaviour change for public health campaigns is cited as being a realistic target (Rice & Paisley 1982).
But before you abandon your PR activities in despair, we have to be clear about what we define as “behaviour change”. Changing eating habits is a long-term behaviour change. PR campaigns often focus on trying to change long term behaviour and don’t consider setting goals in terms of immediate responses – what can people be expected to do when reached by the campaign? In PR terminology this is called “outtakes” (different from long-term behaviour which would be “outcomes”).
Following are some practical examples of “outtakes” or immediate responses (which are short term behaviours) that can be measured. I believe that the percentage of change desired amongst a target audience can can be set at higher than 10% for many of these:
– % of people that sign a petition
– % of people that refer a web page to a friend
– % people that go online to participate in an online discussion
– % of organisations/individuals that publicly support a campaign
– % of people who report undertaking a new initiative as a result of a campaign
Glenn
Output or outcome?
I did appreciate the following quote from Alberto Gonzales, US Attorney General who when defending the work of his department said:
“Good, if you look at the output”
Regardless of what you think of Mr Gonzales and his department’s performance, I find it interesting the use of the word output – it has sneaked in from management-by-objective speak… but output is usually a poor measure for performance, as it represents the products or services produced. It is just like..
A press officer judges her performance by the number of press releases she writes
A training office judges his performance by the number of people that attends his training sessions
What is far more important are outcomes – the effects and changes that are a result of the outputs:
A press officer should judge her performance by how her press activities change the knowledge and attitudes of audiences
A training officer should judge his performance by how the people he trains uses what they have learnt
Like Mr Gonzales, most people prefer to look at outputs to judge performance as they are much easier to control and monitor compared to outcomes, which I’ve written about previously. But increasingly activities are assessed on what they achieve (outcome) rather than what they produce (output).
Glenn
intelligent measurement 