Top Ten Excuses for not Evaluating
This post at the IABC measurement blog caught my attention, as its author asks the question:
“So why don’t we measure more? Is it budget, competencies, time or the risk of accountability?”
People usually don’t evaluate for various reasons, but the most common excuses I’ve heard are the following:
- “It’s too expensive”.
With the amount of free advice, excellent guidelines and cheap research solutions available, this doesn’t pass anymore. - “I don’t know how to”.
Fair enough, but you can learn a lot yourself without having to engage expensive consultants. - “I’m too busy “doing” to be bothered with measuring”.
Frightening. People love doing things, it’s natural. But sometimes you have to stop and take a step back to see what you have achieved. - “What I’m doing couldn’t possibly be measured”.
Often heard from the Creative Type. People who create their own fonts, too clever campaigns and beautiful artwork that impresses other Creative Types. But my question is – what did you change? - “I don’t see the value of it”.
How else can you judge the value of your work if you don’t attempt to analyse and assess it? - “I’m scared what of what I will find out”.
But I think it will be scarier for you if you don’t evaluate and someone else does. - “People are fed up with giving their opinion”.
I don’t think people are – as I’ve written about before. - “My gut feeling tells me I’m doing a good job”.
There is a certain vogue that says out intuition is often our best call. But research often brings out issues that were not even on your radar. - “All my work is vetoed by the CEO, if s/he’s happy so am I”.
The CEO sees the organisation through the same rose-colored glasses as you do. In PR, it’s your public’s perception of your communication that counts. - “You can’t prove anything anyway”.
You can rarely obtain 100% proof that your programme caused the change seen. But what you do is collect evidence that indicates the role your activity played, as I further explained in this post.
Glenn
PR Guru Grunig on intensive research PR
Professor James Grunig, a leading thinker in the field of communications / PR evaluation was in Switzerland last week. While here, he gave an interview where he spoke about the notion of PR being "research intensive" amongst a wide range of subjects in the PR field.
Glenn
PR Measurement: are publics fed up?
People (read: potential clients) often tell me that their publics are fed up filling out online survey, being interview or queried about their opinions. Rubbish I say. People are willing to spend time giving their opinion about a website, service or issues that is important to them.
In a recent online survey of external audiences for an international organisation, 25% of the potential audience responded to the survey. And of those who responded, 44% requested a copy of the results. In an evaluation of an international event, 55% of the participants responded to survey. And of those who responded, 60% requested a copy of the results.
In the survey for the international organisation, 80% of respondents said that the organisation’s website was very important or important for their work. Wouldn’t you like to give an opinion on something that important to you? I would.
My experience is that people are willing to participate in research if it is something that is important to them and if they believe something will be done with the results.
In this article on the simply-communicate.com site, the author points out that:
“In a recent survey of reasons for non-response to employee questionnaires, the biggest driver of non-participation was found to be ‘Nothing would happen as a result'”.
So people do want to see that their opinion is valued. The fact that such a large percentage of people want to receive the evaluation results indicate that they are interested in the subject in question. In addition, I imagine that people are interested to see if what they think corresponds to the norm – and to judge what the organisation will do as a consequence of the evaluation.
Glenn
Evaluation – going beyond your own focus
If your focus is on evaluating PR, training or another business competency, it is sometimes helpful to learn more about evaluation by looking further than your particular focus. Look at international aid. I just read a review of a new book The White Man’s Burden – Why the West’s Efforts to Aid the Rest Have Done So Much Ill and So Little Good By William Easterly. Based on the book review, he raises two interesting points about evaluation:
Planners vs. Searchers: he says most aid projects have either one of these approaches. He writes “A Planner thinks he already knows the answer. A Searcher admits he doesn’t know the answers in advance”. He explains that Searchers treat problem-solving as an incremental discovery process, relying on competition and feedback to figure out what works.
Measuring Success: Easterly argues that aid projects rarely get enough feedback whether from competition or complaint. Instead of introducing outcome evaluation where results are relatively easy to measure (e.g. public health, school attendance, etc), advocates of aid measure success by looking at how much money rich countries spend. He says this is like reviewing movies based on their budgets.
You can read the full review of Easterly’s book on the IHT website.
These are some excellent points that we can apply to evaluation across the board. Evaluators can probably classify many projects they have evaluated as either of a Planner or Searcher nature. The Searcher approach integrates feedback and evaluation throughout the whole process – look at PR activities, we may not know what communication works best with our target audience to begin with, but by integrating feedback and evaluation we can soon find out.
His analogy about reviewing movies also strikes a chord. How many projects are evaluated on expenditure alone?
Aside from the evaluation aspects, his thoughts on international aid are interesting. I spent my formative years as an aid worker in Africa, Eastern Europe and Asia and a lot of what he says confirms my own experiences and conclusions.
Glenn
Favourite Quotes on Evaluation and Measurement
In presentations on evaluation and measurement, I find the use of quotes from experts or well-known personalities assists in clarifying the importance of the field for the listeners. Here are some of my favourite quotes:
“True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information”
Winston Churchill
“Fear cannot be banished, but it can be calm and without panic; it can be mitigated by reason and evaluation”
Vannevar Bush
“The only man who behaves sensibly is my tailor; he takes my measurements anew every time he sees me, while all the rest go on with their old measurements and expect me to fit them”
George Bernard Shaw
And a favorite from Monitoring and Evaluation News:
Friend to Groucho Marx: “Life is difficult!”
Marx to Friend: “Compared to what?”
Combining Qualitative and Quantitative Methods for Evaluation
In evaluation, we often choose between using qualitative (e.g. focus group) and quantitative (e.g. survey) methods. In fact, we should always try and use both approaches. This is what is referred to as triangulation: the combination of several research methods in the study of the same phenomenon. My experience has been that a combination of research methods helps provide more data to work with and ultimately a more accurate evaluation. In a recent project, I was able to use interviews combined with surveys to assess participant reaction to training. I found that the information we could draw from the interviews was complimentary – and of added value – to what we discovered through the surveys.
Even if you are only conducting online surveys, the inclusion of open questions (where respondents put in comments in a free text field) is not quite triangulation but will provide you with insight into the phenomenon being evaluated. In a recent online survey project, we were able to clarify important issues by sorting and classifying the comments made in open questions. This proved invaluable information and gave the evaluation heightened status within the organisation.
Glenn
PR Measurement – New Dictionary
The USA’s Institute for Public Relations, which offers all of its research and publications free on the web, has published a new edition of the Dictionary of Public Relations Measurement and Research (pdf – 198kb), edited by Dr. Don W. Stacks of the University of Miami.
We all measure different things and we measure them all differently (and not always intelligently). Not a good idea if you want to benchmark or compare data.
Don Stacks and his colleagues have done a magnificent job in assembling, codifying and defining most of the common terms we come across when talking about measurement, from “Algorithm” to “Z-score”.
It’s a giant step forward in defining some standard methodologies which everyone can use.
The Institute’s website, http://www.instituteforpr.org also contains a large number of downloadable papers on every aspect of communications measurement and evaluation. Well worth a visit.
Richard
Measuring the “Soft Issues” for Investors
One company that is a good example of “intelligent measurement” is Innovest Strategic Value Advisors, Inc. an investment research and advisory firm (disclosure: there are no links between the authors of this blog and Innovest). This company specializes in analyzing “non traditional” drivers of risk and shareholder value, including companies’ performance on environmental, social and strategic governance issues.
Innovest’s research is focused on those factors which contribute most heavily to financial performance. Environmental and social performance measures are used as leading indicators for management quality and long-term financial performance, not as commentaries on the intrinsic ethical worth of the companies. At the heart of Innovest’s analytical model is the attempt to balance the level of environmentally and socially driven investment risk with the companies’ managerial and financial capacity to manage that risk successfully and profitably into the future.
Environmental assessment criteria:
In total, the Innovest EcoValue‘21™ model synthesizes over 60 data points and performance metrics, grouped together under six key value drivers – Historical contingent liabilities, Financial Risk assessment, Operating risk exposure, Sustainability risk, Strategic management capability and Sustainable profit opportunities.
Social assessment criteria:
Over 50 individual performance indicators are addressed in Innovest’s IVA™ rating model. The principal value drivers are Sustainable Governance, Stakeholder management, Human Capital management, Products and Services, and behaviour in relation to oppressive regimes or exploitative labour markets in emerging markets.
The measurement process is complex, intensive and time consuming. Once the interview/data gathering process is completed, each company is rated relative to its industry competitors. Companies are rated against the Innovest performance criteria, and given a weighted score, as well as a letter grade (AAA, BB etc.). Each of the factors has an industry-specific weighting, based in part on a regression-based factor attribution analysis examining recent (5 year) stock market performance.
This approach is intelligent measurement at its finest. The methodology is rigorous, and the outputs have enormous strategic value to investors, and management of any capital intensive company trying to understand the correlation between investment in traditionally regarded “soft” management issues and shareholder value.
Which why Innovest commands a high price for its reports and is generally regarded as the leader in its field.
Richard
New PR Measurement Blog from IABC
The International Association of Business Communicators (IABC) has launched a new blog with three orientations: branding, employee and measurement. Interesting to note that PR measurement has been given a high profile by the IABC – an indication of its importance to communicators today.
The measurement blog has already some interesting posts and comments on measuring employee communications and context and evaluation, something I’ve written about before.
Glenn
Intelligent Measurement – what and how?
Intelligent Measurement is about using innovative methodologies and technologies to measure things which are (a) important to measure, and (b) notoriously difficult – normally because of the intangibility of the subject and the subjectivity of measurers.
Don’t get me wrong. As a declared enemy of bureaucracy, I have an aversion to all forms of obsessive micro management and control freakery.
But I have seen many organisations blunder along in the dark as they attempt to improve their communications or embark on change management or environmental performance programmes. If only they had some objective measurements at the start of the programme- their strategy would be informed by actualities, not anecdote. They could have measured outcomes, not output. They could have used the data to frame a sensible budget, and also measure which parts of the budget were wasted, and which returned the investment.
I have been to a number of conferences on PR measurement, notably Katie Paine’s “Measurement Summits”. The cry from the floor is “how can I obtain evidence to justify/keep my job/budget/team?” One solution is measurement. But what? And how?
Over the next months, I will be posting on this blog examples of good “Intelligent Measurement”. (please feel free to post any suggestions on the blog).
Richard
intelligent measurement 