Implications of COVID-19 on evaluation
The ILO have published useful guidelines on “Implications of COVID-19 on evaluations in the ILO: Practical tips on adapting to the situation“. The guidelines are well worth a read as they can provide guidance for many of us carrying out evaluations remotely these days.
Barcelona Principles 3.0 – communications evaluation
The Barcelona Principles – which set out the main principles for communication evaluation – have been updated – version 3.0.
Following is an explanation provided by AMEC.
1. Setting goals is an absolute prerequisite to communications planning, measurement, and evaluation.
The founding principle of SMART (specific, measurable, actionable, relevant, and time-bound) goals as a foundation for communications planning has been promoted to an essential prerequisite. It pushes measurement and evaluation as a core component of the planning process, articulating target outcomes and how progress towards these will be assessed.
2. Measurement and evaluation should identify outputs, outcomes, and potential impact.
Previously, the Principles recommended measuring outcomes, rather than simply counting outputs. The updated principles extend this to consider longer term impact of communications strategy.
3. Outcomes and impact should be identified for stakeholders, society, and the organization.
From the original focus on business metrics, such as sales and revenue, the 2020 update embraces a more holistic view of performance. It allows the model to be more inclusive of a broader range of organisations and communications roles that are not necessarily profit-driven.
4. Communication measurement and evaluation should include both qualitative and quantitative analysis.
“To understand the full impact of your work, it is crucial that you use the full suite of methods to measure those outcomes,” summarised Ben Levine AMEC Board Director in describing the evolution of this principle to not just quantify but also understand how messages are being received, believed and interpreted.
5. AVEs are not the value of communication.
The message remains consistent and clear; “we continue to believe that AVEs do not demonstrate the value of our work.” It is important that communications measurement and evaluation employs a richer, more nuanced, and multi-faceted approach to understand the impact of communications.
6.Holistic communication measurement and evaluation includes all relevant online and offline channels.
Our founding principle that social media can and should be measured is so obvious today. The 2020 iteration reflects the game-changing shift in social communications’ capabilities, opportunities, and influence, such that all relevant online and offline channels should be measured and evaluated equally. The AMEC measurement framework promotes clarity across earned, owned, shared, and paid channels to ensure consistency in approach towards a common goal.
7. Communication measurement and evaluation are rooted in integrity and transparency to drive learning and insights.
Sound, consistent, and sustained measurement calls for integrity and transparency in recognition of today’s attention to data privacy and stewardship as organisations comply with new regulations, such as GDPR. This is also a statement that measurement isn’t simply about data collection and tracking, but about learning from evaluation and applying insight back into communications planning. It recognises the need to be transparent about the context in which programmes are run and being aware of any bias that may exist in the tools, methodologies and interpretations applied.
AMEC global virtual summit on communication measurement 2020
AMEC has announced their global virtual summit on communication measurement for 2020:
20/20 FORESIGHT: Measurement Frameworks, Myths and New Horizons
The virtual summit will take place on 8-9 July 2020
Keynote speaker will be Distinguished Professor Jim Macnamara of University of Technology Sydney & LSE – Prof. Macnamara is a specialist in communication evaluation and his research is always very interesting.
Jim will present his latest research that inform his new book Beyond Post-Communication and an evaluation project that showed a €24 million return on investment from communication. In two contrasting case studies, Jim will show the negative impact of disinformation and post-truth and the key causes, compared with a ground-breaking case study in evaluation of corporate communication.
Conference fees:
-Standard attendee £350 + VAT
-AMEC Member £250 +VAT
Industry Partner Member (CIPR, ICCO, IPR, PRCA, PRSA) £250 +VAT
-Government/academics/not-for-profit organisations £200 +VAT
-Enterprise delegate offer (up to 15 delegates) £2000 +VAT
New resource: Measuring Narrative Change
ORSImpact have produced an in-depth guide on measuring narrative change. The resource will be useful in communication and advocacy evaluation where we are often endeavouring to track and measure changes to the narratives of the pertinent issues. View the guide here>>
Measuring people power
MobLab has carried out a global study on how to measure people power – mobilising publics for campaigns and other initiatives. Here is an extract of the key takeaways
:
” key takeaways
For changemakers, if you’re not already measuring depth of supporter relationships and engagement (vs total numbers, or breadth), this report highlights a few ways to get started. Whether in a grassroots group, social movement, or an established NGO, tracking how committed people are to your mission (and to one another) and facilitating deeper relationships can be a big step forward.
For organisational leaders, this report is an urgent call to support cultures that are curious, innovative, and supporter- or people-centred. The results also clarified the importance of investing in staff and resources needed to build relationships with volunteers, grassroots, and community groups. The majority of the surveyed individuals found the very effort of measuring people power shifted and informed both tactics and strategy, guided day-to-day decision making, and was used to evaluate the past; essentially, to show them what was working and what wasn’t.
For funders, the results underscore the need to support organisations, groups and movements by investing in organising and power building, which don’t come with the same attractive metrics that we typically see in fundraising and digital advocacy. Funders should support both the implementation of sustained organising and supporter relationship building (outside of short term programme objectives) and ongoing development and experimentation in power building or “depth” metrics.”
CART principles for monitoring
Innovations for Poverty Action have developed some useful guidance on activity monitoring and evaluation based on their own CART principles: credible, actionable, responsible, and transportable (see summary graphic below).
Particularly useful for those interested in monitoring which is an issue many organisations find challenging. Read more here>>
Evaluation during COVID19 – infographic
Here an informative infographic from UNDP independent evaluation office on evaluation during COVID19 (view the pdf version here).
Evaluation and COVID19
There have been many useful and interesting articles on how evaluation can adapt and cope with the current COVID19 pandemic. Here is a collection of what I’ve found to date:
Practical tools/advice:
Conducting phone-based surveys during COVID 2019
A quick primer on running online events and meetings
Covid-19 crisis: how to adapt your data collection for monitoring and accountability
Think pieces:
Zenda Ofir: Evaluation in times of COVID19 outbreak
World Bank: Conducting evaluations in times of COVID-19
Chris Lysy: The Evaluation Mindset: Evaluation in a Crisis
M Q Patton: Evaluation Implications of the Coronavirus Global Health Pandemic Emergency
Integrating communications in evaluation – presentation slides
Earlier this year I gave a presentation on “integrating communications in evaluation” and I am now happy to share the presentation slides of the event:
Advocacy evaluation resources
Here is a non-exhaustive list of some advocacy evaluation resources available online (n.b. most links will take you directly to a PDF document).
Theory of change:
Julia Coffman & Tanya Beer (2015), The Advocacy Strategy Framework; A tool for articulating an advocacy theory of change
UNICEF (2014), Theory of change; methodological briefs
Advocacy monitoring and evaluation methods/approaches:
Robin Kane et al (2017), Contribution Analysis in Policy Work; Assessing Advocacy’s Influence
Harvard Family Research Project (2009), A User’s Guide to Advocacy Evaluation Planning
Julia Coffman (2019), Current Advocacy Evaluation Practice, Center for Evaluation Innovation.
Julia Coffman and Ehren Reed (2019), Unique Methods in Advocacy Evaluation
Oxfam, Process Tracing – Draft Protocol
Guidelines:
CARE (2018), Advocacy and Influencing MEL Guidance
UNICEF, Monitoring and evaluating advocacy; companion to the advocacy toolkit
ODI (2014), Monitoring and evaluation of policy influence and advocacy
Communicating monitoring and evaluation results:
Glenn O’Neil (2017), A Guide: Integrating Communication in Evaluation
CDC (2013), Evaluation Reporting: A Guide to Help Ensure Use of Evaluation Findings
“Think pieces” on advocacy evaluation:
Examples of advocacy evaluation reports:
UNHCR (2019), Evaluation of Effectiveness and Relevance of Advocacy Approaches in Europe
Oxfam International (2013), Evaluation of global communications campaign GROW
