Posts filed under ‘Communication evaluation’
PR Measurement – Catch 22
PR Week has recently published their Marketing Management Survey (pdf). Amongst other subjects, the survey asked US-based marketing excutives about measurement and their activities. Ed Moed of the Measuring Up blog points out that the survey “rehashes many of the same weary and misguided perceptions” on PR measurement, notably:
“PR is a difficult one to measure.”
“ Clients value PR, but question its value because they think it can’t be measured”
“In an economic climate where budgets are tight, research and measurement are very often the first portions of a PR budget to be cut. Yet measurement is necessary to prove ROI, which can help increase budgets, providing a Catch-22 situation…”
As Ed points out “People, public relations can be measured.” I couldn’t agree more, read Ed’s full post on his blog.
Glenn
Evaluating communication products
Organizations spend millions on communication products every year. Brochures, annual reports, corporate videos and promotional materials are produced and distributed to a wide variety of audiences as part of broader communication programmes or as “stand alone” products.
However, working with many different types of organizations, I’ve noticed that little systematic follow-up is undertaken to evaluate how these products are used and what is their contribution to achieving communication or organizational goals.
I recently worked on a project where did just that – we evaluated specific communication products and attempted to answer the following questions:
- Is the product considered to be of high quality in terms of design and content?
- Is the product targeted to the right audiences?
- Is the product available, accessible and distributed to the intended target audiences?
- Is the product used in the manner for which it was intended – and for what other unintended purposes?
- What has the product contributed to broader communication and organizational goals?
- What lessons can be learnt for improving future editions of the product and design, distribution and promotion in general?
The results were quite interesting and surprising. We were also able to map out the use of a given product, like in this example:
You can read more about this approach in this fact sheet (pdf) >>
Glenn
Internal communications and measurement
For those interested in measuring internal communications (communicating with staff within an organisation), here is an interesting article by Susan Walker from the Handbook of Internal Communication:
“Measurement is not just an optional extra for communicators, but an essential part of their professional tool kit. It has been seen sometimes as a threat (will they cut my budget? will they cut me?) It can be, however, an exciting opportunity to evaluate, guide and direct communication initiatives and investment.”
Glenn
Evaluating advocacy campaigns – No. 2
I’ve written previously about work that others and myself have done on evaluating communication and advocacy campaigns, particulary concerning campaigns that aim for both changes in individual behaviour and government/private sector policies.
In this area, here is an interesting article from the Journal of Multidisciplinary Evaluation, “Advocacy Impact Evaluation” (pdf) by Michael Q. Patton. The article explains how an evaluation was undertaken to evaluate the impact of an advocacy campaign to influence a decision of the US Supreme Court.
What I find interesting is how the evaluation was done – what is called the “General Elimination Method”.
This is where there is an effect (the Supreme Court decision) and an intervention (the advocacy campaign) and they search for connections between the two. They tried to eliminate alternative or rival explanations until the most compelling explanation remained. They did this through interviews, analysis of news, documents and the Court’s decision. The article explains all of this and makes for interesting reading, you can read the article here (pdf).
Glenn
Measuring social media
An interesting post from the Buzz Bin which provides a good summary of current thoughts on how to measure social media such as blogs, social networks and podcasts. Well worth a read…
Glenn
Relationship measurement – hype?
There is an interesting debate on measuring relationships going on Don Bartholomew’s Metricsman blog – Don is saying that he thinks relationship measurement will not be the “next big thing” in measurement – and there are some interesting comments by K D Paine, M Weiner and J Grunig – all well known in this area.
I think it’s interesting to note that the financial world is advocating the measurement of relationships (an “intangible asset” as they call it) and would like to see this included in future company reporting as I’ve written about before. “Next big thing” or not – relationship measurement will grow in importance – in my humble opinion.
Glenn
Impact – how feasible for evaluation?
As I mentioned in an earlier post, people often confuse “impact” with “results”. Is it possible to measure “long term impact” of projects? It is, however for most projects it is unrealistic to do so for two reasons: time and cost.
To evaluate impact, you would usually need to wait some 12 months after the major elements of a project have been implemented. Many organisations cannot simply wait that long. In term of costs, an impact study requires a triangulation methodology that uses various quantitative and qualitative research methods which could be costly. However, if time and cost are not issues, an impact evaluation is possible, keeping in mind the following points:
Was the impact desired defined at the beginning of the project?
For example, greater organisation efficiency; change in the way a target audience and/or an organisation behaves; or improvements in how services for a given audience are managed?
What have been the other elements influencing the impact you want to measure?
Your project cannot be viewed in isolation; there must have been other factors influencing the changes being observed. Identifying these factors will help you to assess the level of influence of your project compared to other factors.
Do you have a mandate to measure impact?
When assessing impact, you will be looking at long term effects that probably go outside of your own responsibilities and into the realms of other projects and units – you are looking at an area of the wider effects of your organisation’s activities and this needs to be taken into consideration. For example, if you are looking at the longer term effects of a training program, you would want to look at how individuals and the organisation as a whole are more efficent as a result of the training. Do you have the political mandate to do so? – As you may discover effects that go way beyond your own responsibilities.
Evaluating impact is a daunting but not impossible task. For most projects, it would be more realistic to focus on measuring outputs and preferably outcomes – and think of short term outcomes as I have written about previously.
Glenn
New Blog: Measurement Standard
Bill Paarlberg and K. D. Paine have a relatively new blog online: the measurement standard blog – it’s a handy summary of their regular newsletter of the same name which focuses on PR measurement. There are plenty of interesting posts on various topics such how does PR measurement take news timing into account and measuring success of small organisations.
Glenn
Changing behaviour – immediate responses
Adding to what I wrote about last week concerning measuring behaviour changes that result from communication campaigns – and why I recommend to consider looking at immediate responses (or “outtakes”) as an alternative to long-term changes – I can see parallels in areas other than in campaigns.
As you may know, a favourite of mine is measuring the impact of conferences and meetings. Industry conferences are traditionally sold as being great places to learn something and network, network – and network. But I’m always surprised when attending such conferences at how organisers, if they measure something, focus on measuring the reactions to the conferences, usually in terms of satisfaction. No attempt is made to measure immediate changes to behaviour (such as extending a network) or longer term behaviour or impact in general.
But it is certainly possible, this diagram (pdf) illustrates what I did to measure immediate and mid-term changes to behaviour following a conference (LIFT). Despite the limitations of the research as I explain here, I was able to track some responses following the conference that could be largely contributed to participating in the conference – such as meeting new people or using new social media in their work. One year after the conference, participants also provided us with types of actions that they believed were influenced largely by their participation. Actions included:
– launching a new project
– launching a new service/product
– establishing a new partnership
– Initiating a career change
– Invitations for speaking engagements
Some of these actions were anticipated by the conference organisers – but many were not. It shows that it can be done and is certainly worth thinking about in conference evaluation.
Glenn
Changing behaviour – takes a long time?
In an interesting post by Tom Watson on the Dummyspit blog, he writes about the difficulty of changing behaviour through PR campaigns – in this case to encourage better eating habits in UK schools (staying away from hamburgers for example). He also notes the rather depressing statistic that PR campaigns normally have only a 0.04% success rate of changing behaviour! In setting campaign objectives, I encourage organisations to be realistic about the targets they set – for example, a modest 2 – 10% behaviour change for public health campaigns is cited as being a realistic target (Rice & Paisley 1982).
But before you abandon your PR activities in despair, we have to be clear about what we define as “behaviour change”. Changing eating habits is a long-term behaviour change. PR campaigns often focus on trying to change long term behaviour and don’t consider setting goals in terms of immediate responses – what can people be expected to do when reached by the campaign? In PR terminology this is called “outtakes” (different from long-term behaviour which would be “outcomes”).
Following are some practical examples of “outtakes” or immediate responses (which are short term behaviours) that can be measured. I believe that the percentage of change desired amongst a target audience can can be set at higher than 10% for many of these:
– % of people that sign a petition
– % of people that refer a web page to a friend
– % people that go online to participate in an online discussion
– % of organisations/individuals that publicly support a campaign
– % of people who report undertaking a new initiative as a result of a campaign
Glenn
intelligent measurement 