Evaluation, Proof and the Kylie effect
February 6, 2006 at 10:24 pm 5 comments
A question often asked by those commissioning an evaluation is how can we “prove” that a program or activities have caused a change we are observing. How can we be sure that a training program is responsible for the rise in productivity? That an awareness campaign has changed attitudes about a company? In most cases you simply cannot get 100% proof. But what you can do is collect evidence that indicates that a program / activity did play a major role in the change we are seeing. As one pundit put it:
“The key to winning a trial is evidence not proof”
Following are some strategies to tackle this issue:
- Set up a control group that were not exposed to the program or activity
- Use pre- and post measures to show the changes occuring over time
- Don’t only rely on survey or quantitative data – testimonies and anecdotes can be convincing evidence
- Identify any other possible factors that could have caused the change being observed.
Of course, setting up a control group is always difficult in a real-world environment. But my experience has shown that it can bring forward very useful results, if we are honest about limitations and other possible influences.
It is important to be transparent and recognise any other factors that could have caused the change being observed. Take for example, breast cancer awareness in Australia. Health educators have been working hard for years to get more young women to undertake a mammogram (breast screening). As if detected early, the disease can be treated successfully. So for health educators, a clear impact indicator would be the number of appointments taken for mammograms. In August 2005, appointments for women aged 40 to 69 in Australia jumped by 101%. Was this the result of a very successful awareness campaign? No, in fact what we were seeing is what has been labelled as the “Kylie effect”. In May 2005, Australian pop singer Kylie Minogue was diagnosed with breast cancer resulting in mass media coverage about the issue – and consequent awareness of breast cancer and its detection. Studies have shown that there is a direct link between the jump in screening appointments and Kylie Minogue’s illness. If interested, you can read further about the “Kylie effect” on the BBC website.
Glenn
Entry filed under: Evaluation methodology, PR evaluation, Training evaluation.
1. intelligent measurement » Media monitoring - what is it worth? | July 5, 2006 at 8:12 am
[…] This can work particularly well with targeted campaigns; my colleagues at Benchpoint in the UK recently did a joint media analysis (with Mantra International) on the retail sector linked with surveys of the target audience (customers). The problem of isolating the effect of a given media campaign/event/activity is not simple as I have written about previously but at least – at a minimum – you will have indications as to the influence of the different media and be able to make reasonable assumptions – with supporting evidence – about your media work and its impact. […]
2. Kidneys, Kylie and effects « intelligent measurement | July 25, 2007 at 4:21 pm
[…] Which reminds me of another example – in August 2005, the number of appointments made for manograms (to detect breast cancer) jumped by 101% in Australia. Was this the result of a successful communications campaign? No, in fact that month, pop singer and fellow Melburnian Kylie Minogue was diagnosed with breast cancer resulting in mass media coverage about the issue which I’ve written about previously. […]
3. Cause and effect - goal? « intelligent measurement | October 23, 2007 at 7:43 am
[…] the effect of a reality TV program on the number of people willing to donate an organ and the effect of Kylie Minogue’s treatment for breast cancer on the number of young women taking a sc…. In both cases, an association between the cause and effect could be demonstrated and alternative […]
4.
Shelli Rapier | September 16, 2010 at 7:46 am
My idol, Kylie Minogue. She is very kind of doing favors for cancer charity and also understands the need of cancer patients as she was diagnosed with breast cancer in 2005. She is a strong woman.
5.
Lonny Munda | September 25, 2010 at 11:32 pm
Love Kylie Minogue!! She was brilliant on the Paul O’Grady show, and her and Jake are such good friends; I loved her response when she realized it was him under the costume.