Beyond online vanity metrics

Here is a very interesting study (pdf) from the Mobilisation Lab on what counts and doesn’t for online metrics and campaigns.

The study looks at what they call “vanity metrics” for online campaigns that they define as “data that are easily manipulated, are biased toward the short-term, often paint a rosy picture of program success, or do not help campaigners make wise strategic decisions”. Examples of vanity metrics include: number of petition signatures; web traffic, number of “opens” (of emails I guess).

So what do they recommend campaigns should be measuring?

They have plenty of good suggestions and insights. Here are some of the metrics they mentioned that could be more significant (and possible to measure online):

  • Monthly members returning for action
  • Actions per member (rather than size of lists)
  • Number of members actively part of a campaign

View the study here (pdf)>>

July 28, 2015 at 8:28 am 1 comment

10 elements of success for advocacy

The US-based Media Impact Funders have produced a ten point list on elements of success for policy change (what I’ve labeled “advocacy”), as reproduced below in this post.

Although it is looking from a US perspective of influence on policy through advocacy, what struck me was that many of the points are relevant to advocacy done globally or in other countries/regions.  For example:

Point 1: Solutions – when evaluating advocacy initiatives and talking with policy-makers, a common complaint I have heard is that advocacy is not “solution focused”, i.e. it makes valid points about the given issues (that often policy-makers are also aware of) but don’t necessarily put forward possible solutions to these issues.

Point 3: Agility – the ability to be flexible is so important – to take advantage of opportunities that arise, that were not necessarily included in the original advocacy plan. That I saw recently in an evaluation I carried out for Oxfam on global development policy – where they had enough flexibility to move resources as the issues peaked and new opportunities emerged.

Point 5: Humanity – often advocacy focuses on the “facts” but what can also make an impact is the “human factor”. One effective example of this was in the creation of the Arms Trade Treaty where state representatives drafting the treaty were directly confronted by survivors of armed violence; certainly bringing a human face to the dry legalistic treaty process and language.



July 14, 2015 at 11:28 am 1 comment

What sort of evaluator are you?

From the folks at ImpactReady, a fun quiz to determine what sort of evaluator are you:
Positivist, Constructivist or Transformative?

Take the quiz now!

p.s. I came out as a  Constructivist Evaluator…

June 22, 2015 at 11:57 am 1 comment

Practical Advice for Selecting Sample Sizes

A new publication has been released by Donor Committee for Enterprise Development providing practical advice for selecting sample sizes (pdf).  The publication is particularly useful for considering sampling issues for online surveys and provides a lot of good advice and tips.

June 12, 2015 at 7:03 am Leave a comment

Visual summaries of evaluation reports

In commuleapnicating evaluation findings, challenges are often seen with the key product to do so – the evaluation report.  Often evaluation reports suffer from being long, wordy and just plain boring! Therefore, we have to find new ways to communication evaluation findings.


One way I find interesting is the use of a visual summary of findings – that summarises the evaluation findings graphically  and in a limited number of pages.  I recently had the opportunity to use this format for an evaluation for Oxfam – as seen in the image in this post. You can view the full summary by clicking on the image (it brings you to a pdf file).



June 4, 2015 at 8:34 am 2 comments

Monitoring and Evaluating Advocacy toolkit

I recently came across this resource from UNICEF, which is their  guide to monitoring and evaluating advocacy (pdf).  It’s a companion to their larger advocacy toolkit (pdf).

It’s a very comprehensive guide and I’d add it to my previous list of best resources for advocacy evaluation>>

May 21, 2015 at 6:20 am Leave a comment

New paper: Beneficiary feedback in evaluation

DFID have released a new paper on the practice of beneficiary feedback in evaluation (pdf).

The paper highlights five key messages (listed below), with a main point being that beneficiaries are often only seen as a provider of data and aren’t given a broader role in the evaluation process – a point I can confirm from having been involved in many evaluations.

Rather ironically, the DFID study on beneficiary feedback includes no feedback from beneficiaries on the study…

Key Message 1: Lack of definitional clarity has led to a situation where the term beneficiary feedback is subject to vastly differing interpretations and levels of ambition within evaluation.

Key Message 2: There is a shared, normative value that it is important to hear from those who are affected by an intervention about their experiences. However, in practice this has been translated into beneficiary as data provider, rather than beneficiary as having a role to play in design, data validation and analysis and dissemination and communication.

Key Message 3: It is possible to adopt a meaningful, appropriate and robust approach to beneficiary feedback at key stages of the evaluation process, if not in all of them.

Key Message 4: It is recommended that a minimum standard is put in place. This minimum standard would require that evaluation commissioners and evaluators give due consideration to applying a beneficiary feedback approach at each of the four key stages of the evaluation process.

Key Message 5: A beneficiary feedback approach to evaluation does not in any way negate the need to give due consideration to the best combination of methods for collecting reliable data from beneficiaries and sourcing evidence from other sources.

View the full paper here (pdf)>>

May 13, 2015 at 6:35 am 1 comment

Older Posts

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 985 other followers




Get every new post delivered to your Inbox.

Join 985 other followers