Posts filed under ‘Campaign evaluation’
Communication evaluation at WHO
Here is an excellent article from Prof. Jim Macnamara on communication evaluation at WHO. He was part of a team which worked with WHO over a number of years to improve their communication evaluation, also over the period of the COVID-19 pandemic. it provides useful insights for anyone interested in communication evaluation and the challenges (and solutions) in this area for large global organisations today.
View the article here>>
AI and campaign evaluation
I asked ChatGPT, the artificial intelligence tool a simple question on communication evaluation: “How to evaluate a communication campaign?“. The response is copied in below.
It is interesting in that covers many good practices of campaign evaluation, such as setting KPIs, carrying out a baseline, ongoing monitoring and learning from the results. But it mixes up methods and measures and does not address one important aspect – assessing contribution/attribution of any results found. Also it fails to distinguish between the different levels of results, from output to outcomes to impact. Although it does end with a nice statement that I fully agree with:
“Remember that the evaluation process should be ongoing and iterative. It’s not just about measuring the success of a single campaign but also about improving your overall communication efforts over time.“
My conclusion is that AI has not yet eliminated us evaluators!
Q. How to evaluation a communication campaign?
Answer from ChatGPT:
Evaluating a communication campaign is crucial to determine its effectiveness and make informed decisions for future campaigns. Here’s a step-by-step guide on how to evaluate a communication campaign:
Set Clear Objectives and Key Performance Indicators (KPIs):
Before launching a campaign, establish specific, measurable, and achievable objectives. Define KPIs that align with these objectives, such as increased website traffic, higher social media engagement, or improved brand awareness.
Pre-Campaign Benchmarking:
Gather baseline data related to your KPIs before starting the campaign. This will provide a reference point for measuring the campaign’s impact.
Real-time Monitoring:
Monitor the campaign’s progress in real-time. Track metrics such as website visits, social media mentions, email open rates, and any other relevant data. Tools like Google Analytics, social media analytics platforms, and email marketing software can be valuable for this purpose.
Audience Feedback:
Collect feedback from your target audience through surveys, focus groups, or social media comments. This helps gauge how well the campaign is resonating with the intended audience.
Media Analysis:
Evaluate the media coverage your campaign generates. Are there positive or negative mentions in the press? Analyze the sentiment and tone of media coverage to understand public perception.
ROI Analysis:
Calculate the return on investment (ROI) of your campaign. Compare the costs of running the campaign to the revenue or value it generated. This can include both quantitative (sales, revenue) and qualitative (brand perception) aspects.
Website and Social Media Analytics:
Analyze website traffic, bounce rates, conversion rates, and social media engagement metrics during the campaign period. Assess whether there’s a significant increase in these metrics compared to the pre-campaign period.
Content Performance:
Evaluate the performance of individual pieces of content (e.g., blog posts, videos, infographics). Determine which content formats and topics resonated most with your audience.
Message Recall and Recognition:
Test the audience’s recall and recognition of your campaign’s key messages. Conduct surveys or interviews to assess whether the intended messages are getting across effectively.
Competitive Analysis:
Compare your campaign’s performance with that of your competitors. Analyze how your campaign stands out and whether it’s achieving its objectives in comparison.
Post-Campaign Benchmarking:
After the campaign ends, gather data similar to what you collected before the campaign. This will allow you to compare the post-campaign results to the pre-campaign benchmarks.
Report and Analysis:
Compile all the data and insights into a comprehensive report. Summarize the key findings, including successes, challenges, and areas for improvement.
Learn and Adjust:
Use the insights gained from the evaluation to inform future campaigns. Adjust your strategies, messaging, and targeting based on what worked and what didn’t.
Iterate and Optimize:
Apply the lessons learned from the evaluation to refine your communication strategies continuously. Implement improvements to maximize the effectiveness of future campaigns.
Share Results Internally and Externally:
Communicate the results of the evaluation both within your organization and, if relevant, to external stakeholders. Transparency can build trust and help secure support for future campaigns.
Remember that the evaluation process should be ongoing and iterative. It’s not just about measuring the success of a single campaign but also about improving your overall communication efforts over time.
New resource: The communicators guide to research, analysis and evaluation
The Institute for Public Relations has produced an excellent new guide: “The communicators guide to research, analysis and evaluation“.
The guide highlights:
“The importance of communication research, analysis, and evaluation continues to grow due to the pace of change in the marketplace and the corresponding need to make smarter and better decisions at the speed of business. The ability for an organization to successfully transform during this time of dynamic change requires the active leadership of the Chief Communication Officer and the application of talent and technology to perform the work, measure its impact, and improve at each stage of the public relations process.”
View the guide here>>
Campaign evaluation case study
Here is an interesting case study presenting an evaluation of a European-wide smoking cessation campaign (in other words, trying to get people to stop smoking) that I was involved with. Some interesting findings on campaigns and results in this challenging area.
Evaluating Advocacy: Challenges, Methodologies and Solutions
Today I was very happy to be with the European Centre for Public Affairs for a training on advocacy evaluation. For those who are interested, you can see my presentation slides below and I’ve also listed a range of advocacy evaluation and related resources.
Advocacy evaluation an related resources
Theory of change:
Julia Coffman & Tanya Beer (2015), The Advocacy Strategy Framework; A tool for articulating an advocacy theory of change
UNICEF (2014), Theory of change; methodological briefs
Advocacy monitoring and evaluation methods/approaches:
Robin Kane et al (2017), Contribution Analysis in Policy Work; Assessing Advocacy’s Influence
Harvard Family Research Project (2009), A User’s Guide to Advocacy Evaluation Planning
Julia Coffman (2019), Current Advocacy Evaluation Practice, Center for Evaluation Innovation.
Julia Coffman and Ehren Reed (2019), Unique Methods in Advocacy Evaluation
Oxfam, Process Tracing – Draft Protocol
Guidelines:
CARE (2018), Advocacy and Influencing MEL Guidance
UNICEF, Monitoring and evaluating advocacy; companion to the advocacy toolkit
ODI (2014), Monitoring and evaluation of policy influence and advocacy
Communicating monitoring and evaluation results:
Glenn O’Neil (2017), A Guide: Integrating Communication in Evaluation
CDC (2013), Evaluation Reporting: A Guide to Help Ensure Use of Evaluation Findings
“Think pieces” on advocacy evaluation:
Examples of advocacy evaluation reports:
UNHCR (2019), Evaluation of Effectiveness and Relevance of Advocacy Approaches in Europe
Oxfam International (2013), Evaluation of global communications campaign GROW
Measuring people power
MobLab has carried out a global study on how to measure people power – mobilising publics for campaigns and other initiatives. Here is an extract of the key takeaways
:
” key takeaways
For changemakers, if you’re not already measuring depth of supporter relationships and engagement (vs total numbers, or breadth), this report highlights a few ways to get started. Whether in a grassroots group, social movement, or an established NGO, tracking how committed people are to your mission (and to one another) and facilitating deeper relationships can be a big step forward.
For organisational leaders, this report is an urgent call to support cultures that are curious, innovative, and supporter- or people-centred. The results also clarified the importance of investing in staff and resources needed to build relationships with volunteers, grassroots, and community groups. The majority of the surveyed individuals found the very effort of measuring people power shifted and informed both tactics and strategy, guided day-to-day decision making, and was used to evaluate the past; essentially, to show them what was working and what wasn’t.
For funders, the results underscore the need to support organisations, groups and movements by investing in organising and power building, which don’t come with the same attractive metrics that we typically see in fundraising and digital advocacy. Funders should support both the implementation of sustained organising and supporter relationship building (outside of short term programme objectives) and ongoing development and experimentation in power building or “depth” metrics.”
Advocacy evaluation resources
Here is a non-exhaustive list of some advocacy evaluation resources available online (n.b. most links will take you directly to a PDF document).
Theory of change:
Julia Coffman & Tanya Beer (2015), The Advocacy Strategy Framework; A tool for articulating an advocacy theory of change
UNICEF (2014), Theory of change; methodological briefs
Advocacy monitoring and evaluation methods/approaches:
Robin Kane et al (2017), Contribution Analysis in Policy Work; Assessing Advocacy’s Influence
Harvard Family Research Project (2009), A User’s Guide to Advocacy Evaluation Planning
Julia Coffman (2019), Current Advocacy Evaluation Practice, Center for Evaluation Innovation.
Julia Coffman and Ehren Reed (2019), Unique Methods in Advocacy Evaluation
Oxfam, Process Tracing – Draft Protocol
Guidelines:
CARE (2018), Advocacy and Influencing MEL Guidance
UNICEF, Monitoring and evaluating advocacy; companion to the advocacy toolkit
ODI (2014), Monitoring and evaluation of policy influence and advocacy
Communicating monitoring and evaluation results:
Glenn O’Neil (2017), A Guide: Integrating Communication in Evaluation
CDC (2013), Evaluation Reporting: A Guide to Help Ensure Use of Evaluation Findings
“Think pieces” on advocacy evaluation:
Examples of advocacy evaluation reports:
UNHCR (2019), Evaluation of Effectiveness and Relevance of Advocacy Approaches in Europe
Oxfam International (2013), Evaluation of global communications campaign GROW
Webinar: How do Not-For-Profit organizations measure the results of their advocacy?
Join me for this webinar on Thursday 20th February 2020 at 15:00 GMT / 16:00 CET / 10:00 EST to learn how Not-For-Profits are measuring the results of their advocacy and what your organization can learn and do in this direction.
The webinar will be moderated by AMEC NFP Group co-chair Natalia Vaccarezza of UNICEF and feature Chris Stalker, Senior Advisor Policy & Campaigns at Oxfam America and myself – Glenn O’Neil, Founder of Owl RE research and evaluation consultancy.
Participants will gain insight using practical examples on:
-Challenges commonly faced in advocacy evaluation and possible solutions
-Trends in the field of advocacy evaluation
-Tools and tactics for effectively measuring advocacy results
Recommended actions for integrating evaluation and measurement in advocacy of NFPs
New resource: No Royal Road: Finding and Following the Natural Pathways in Advocacy Evaluation
Jim Coe and Rhonda Schlangen have published a very interesting publication on advocacy evaluation.
The highlight six factors that they believe should change for monitoring and evaluation of advocacy:
1. Better factor in uncertainty.
2. Plan for unpredictability.
3. Redefine contribution as combinational and dispositional.
4. Parse outcomes and their significance.
5. Break down barriers to engaging advocates in monitoring and evaluation.
6. Think differently about how we evaluate more transformational advocacy.
New article: Pathways to use of campaigns’ evaluation findings
I’m very happy to share with you an article I co-authored – for those interested in campaigning, evaluation and evaluation use:
Highlights
- A categorization of six pathways to evaluation use is proposed.
- Evaluation use occurred slightly more non-linear than linear.
- Evaluation use was mostly unexpected.
- Overall meaning assigned to evaluation was linked to frequency of evaluation occurring.
- Evaluation use contributing to change was often indirect and not simple.
Abstract
This article presents a study on the pathways and processes regarding the use of evaluation findings of communication campaigns from two international organizations, the Office of the High Commissioner for Human Rights (OHCHR) and the International Committee of the Red Cross (ICRC). Several years after the completion of the campaigns and their evaluations, our research identified 28 instances of use and six instances of non-use of the evaluation results, of which the large majority being surprising in nature. Results showed that evaluation use facilitated formal and informal changes at the individual and the organizational level; and, this pattern occurred in a predominantly non-linear fashion, interconnected and overlapping, while gradually decreasing in time and space. Evaluation use was mostly unpredictable, which reflected how meanings are constructed by staff members, as they adjusted and interpreted the findings in opportunistic ways.
intelligent measurement 