Metrics: You are what you measure!
An interesting post from the Metrics Man on the “Media Measurement Catch-22” as he puts it. His main point is that “you are what you measure”, in other words, you will focus you efforts on achieving the metrics you set, and further:
If all you measure is media relations (primarily clip tonnage), that is how the PR profession will be valued.
Read the full post and if you are interested to learn more about the concept of metrics and how they influence our work efforts, consult the Hauser and Katz paper “Metrics: You are what you measure!” (pdf)
Glenn
Geneva Roundtable: Future of Corporate Communications
For those of you in the Geneva (Switzerland) here is an event I’m helping organise that you may be interested to attend:
As part of the Future Series, the International University in Geneva and Geneva Women in International Trade present a roundtable:
“The Future of Corporate Communications”
Tuesday 14 November 2006, 6 p.m.
Movenpick Hotel, ICC, Rte de Bois 20, 1215 GenevaPanel members:
– Cary Adams, Managing Director, Lloyds TSB International Private Banking
– Charlotte Lindsey, Deputy Director of Communications, International Committee of the Red Cross
– Peter Warne, Senior External Communications Manager, Nestle“Through blogs, wikis and podcasts, publics are conversing amongst themselves and building influential sources of information as alternatives to traditional media. Are organisations ready and willing to converse with publics? Can corporate communications cope with competing sources of influence and use social media to build relationships with their key audiences?”
Entry fee: 25 CHF (15 CHF for GWIT members and IUN Alumni)Agenda:
6.00 p.m. Welcome
6.30 p.m. Presentations followed by discussions
7.30 p.m. Cocktail and networkingRegister online for the event.
Glenn
Why aren’t we measuring?
A real gem of a paper here (.doc) by Jim Macnamara, a well-know PR evaluation specialist from Australia. He provides an interesting response to the question “why don’t communication professionals measure more?”:
This is the real reason for lack of commitment to measurement. Most PR practitioners do not proactively use research to measure, either for planning or for evaluation, because in their worldview, it is not relevant. When one focuses on and sees one’s job as producing outputs such as publicity, publications and events, measurement of effects that those outputs might or might not cause is an inconsequential downstream issue – it’s someone else’s concern.
A very interesting conclusion – the focus on production is something I’ve seen a lot – I think there is certainly some truth in what he says.
Read the full article here (.doc).
And thanks to K D Paine for sharing this paper with us.
Glenn
It’s Official: Harold Burson says lack of PR Measurement is no. 1 Obstacle
Last night, I attended a communications forum in Geneva where Harold Burson, the founder of the PR agency Burson & Marsteller spoke (in the photo above, he is on the right and Keith Rockwell from WTO on the left).
In responding to a question from a member of the audience (none other than the public affairs representative from the US Mission) – as to how can communicators measure the effectiveness of their programmes, Mr Burson responded:
“The lack of research by communication professionals is the number one obstacle in the PR field today – people don’t do enough research to evaluate the impact of their activities..”
I agree fully. Then he went on to explain the reason “why”. For Mr Burson, the reason is cost – PR research and measurement is too expensive, he mentioned that often research to evaluate can often cost as much as the activities itself. And that’s where I disagree – PR measurement does not have to be expensive. Most capable communication managers should be able to manage measurement tasks themselves through using low cost media monitoring services, easy-to-use online surveys and innovative methods such as case studies and tracking mechanisms. To get started, check out the guidelines from the Institute for PR. And there are certainly other reasons why communication professionals don’t evaluate.
You can read more about the forum on the Geneva Communicators blog. And you can read more about Mr Burson’s thoughts on his blog (is he the oldest PR blogger at 84 years old..?)
Glenn
Assumptions, Evaluation and Development
At the international conference of the European Evaluation Society, I attended an interesting workshop on Assumptions Based Comprehensive Development Evaluation Framework (ABCDEF) presented by Professor Osvaldo Feinstein, evaluation consultant. The main thrust of his workshop was to challenge us to consider fully the assumptions that are made in development projects – and consequently the impact on evaluation. He has created a guide to what we should consider when exploring assumptions, namely: Incentives, Capacities, Adoption, Risk, Uncertainty and Sustainability. Cleverly, it makes the acronym Icarus, whom we all know flew too close to the sun which melted the wax holding together his wings. And that is the underlying theory of Professor Feinstein, as he put it:
“An unexamined assumption can be very dangerous!”
More information on ABCDEF can be found in the Sage Handbook of Evaluation.
Glenn
Evaluation: to Prove or Improve?
The International Conference of the European Evaluation Society is underway in London and I just participated in several of the pre-conference workshops.
In one workshop, Elliot Stern, a senior Evaluation Consultant made a very poignant point – is the aim of evaluation “to prove” or “improve”?
A simple but interesting distinction – traditionally he pointed out most evaluation aimed to “prove” if an intervention changed anything – and that’s it (a gasp when through the room…)
It’s only a rather recent development that evaluation has been asked to focus also on “improve” – how can an intervention be more effective.
Most likely you are thinking that it can’t be so – how can an evaluation not make the step from “prove” to “improve” ? But often the terms of reference for an evaluator is only to “prove” – in other words, evaluate but please no recommendations.
Glenn
The “Before” Aspect of evaluation
Evaluation is often thought of as a “concluding” activity – something that is done once a programme or project is finished. But evaluation has its role “before” and “during” an activity. A recent experience highlighted for me the importance that evaluation can play in the “before” phase.
I have been involved in setting-up a pan-European e-learning platform and prior to its launch, we decided to test the platform with a select group of users. In the learning or communications field that would be a standard procedure – to pre-test material before it is used with its target audiences. But I am amazed at how many organisations don’t pre-test material – a “before” evaluation activity.
The feedback we received from the test users was incredibly informative – they identified issues that we did not even think about; access, usablity and broader issues on motivation and incentives for using the platform. User tests for online websites/platforms do not have to be complicated and costly – Jakob Nielsen, the specialist in this field explains well why usability is not necessarily expensive.
The “before” evaluation phase is much broader than simply pre-testing material. The establishment of baseline data (e.g. attitude levels on issues), the gathering of existing research on a subject, benchmarking with comparable projects and ensuring that a project’s objectives are clear and measurable are some of the components of this phase.
Glenn
A Post-Modern Tale on Evaluation
A tale of an organisation concerned with “perception”: Due to some negative press, the organisation was convinced that this was causing a drop in their reputation and affecting their relationship with key government and political stakeholders. Consequently the organisation was pushing to re-orientate their communication activities to lobbying and campaigning activities aimed at government and political circles.
But before going ahead, the organisation did have some input from others (no, not me – but a “good friend”) who suggested evaluating the perception of the organisation amongst stakeholders. A methodology was drawn up and a survey conducted of the major stakeholders. Lo and behold, the organisation was shocked upon seeing the findings. The results showed that government officials and politicians actually had a very good perception of the organisation. But that other important target groups, notably key partners and staff had a negative perception of the organisation. So now the organisation is re-re-orientating activities towards staff and partners.
This simple but true tale illustrates two points that I consider important for image and evaluation:
1) Your view of how your organisation is perceived is probably false. Your stakeholders do not necessarily have access nor are influenced by the same media as you are.
2) The only way to determine how stakeholders perceive your organisations is by asking them. Don’t base your ideas on “feelings” or what the media are reporting. Go to the source.
And like all tales, it has a moral: Our intuition can often be wrong. We base our decisions on biased information formed by our own “world view”. An objective evaluation can be a solution and can alter, sometimes radically, what we thought of as the “truth”.
Glenn
New Methodologies in Evaluation
The Uk-based Overseas Development Institute have published a very comprehensive guide “Tools for Knowledge and Learning: A guide for development and humanitarian organisations” (pdf) which contains descriptions of 30 knowledge and learning tools and techniques.
It contains guidelines for several relatively new methodologies useful for evaluation, notably Social Network Analysis, Most Significant Change and Outcome Mapping. I believe these new methodologies could be useful in a lot fields, not only for the development / humanitarian sector.
Glenn
Communications Event in Geneva
For communication professionals in the Geneva region, Switzerland, the following event may be of interest to you:
“The Executive MScom Program (University of Lugano) in collaboration with the Société Romande de Relations Publiques (SRRP) and the International Public Relations Association (IPRA) is launching the MScom Geneva Communications Forum. The conference series, to take place twice yearly, will provide expert perspectives on the fast-changing field of communication management. The theme of the 1st Geneva communications forum will be:
“The changing communications landscape in the age of global business”
Keynote speakers:
Harold Burson, Founder and Chairman of Burson-Marsteller
Keith Rockwell, Spokesperson of the World Trade OrganizationDate and location:
Wednesday, October 11, 2006, 18:00 hrs
Swissôtel Métropole GenevaCost:
CHF 75 – members of the public
CHF 50 – members of SRRP, IPRA & MScom Alumni Association”
More information can be found in the invitation (pdf) and online registration is possible on the mscom website.
Disclaimer: I am a graduate of the Executive MScom Program.
Hope to see some of you there.
Glenn
intelligent measurement 