Conference – building for the future: evaluation in governance, development & progress
The European Evaluation Society has it’s biennial conference (full title above) scheduled for 1-3 October 2008 in Lisbon, Portugal. The conference kicks off with a series of hands-on workshops, that I attended at the last conference and found very useful. Read more on the conference website.
I’ll be presenting a paper at the conference on “Evaluating Public Information and Advocacy Campaigns”, so if you are going, please drop me a line…
Glenn
Social network analysis and evaluation
Measuring networks can have many applications: how influence works, how change happens within a community, how people meet, etc. I’m interested in measuring networks as indicator of how contacts are established amongst people, particularly in events and conferences, as I’ve written about previously.
In this area, there is a new resource page available on social network analysis and evaluation from M&E news. The page contains many useful resources and examples of network analysis and evaluation for non-profit organisations, education, events and research and development – including one from myself.
(Above image is from a network analysis of a conference, further information is available here>> )
Glenn
media evaluation – sentiment analysis
Monitoring of the media often involves assessing the “tone” of a media item – is it positive, negative or neutral on a given subject? This is often done manually and many of the large media monitoring services do this but it is a paid service not readily available to many.
Now there is a tool available to all to assess the “tone” of a media item (written):
This is what David Phillips of the LeverWealth blog had to say about it:
“This kind of development is useful for analysing sentiment of news articles, blogs and other content, which is its primary purpose but it also has applications in evaluating style and and bias all of which are very useful to the PR industry, regulators and watchers of political sentinemt on and off line.”
Glenn
Cultural issues in evaluation
Having spent the last week in the Congo – mostly in Kisangani (pictured above) for an evaluation project, I’ve been thinking about cultural issues and evaluation – in particular how evaluators are perceived in different societies as I’ve written about before.
Interestingly, when I was recently in Central Asia, it was explained to me that evaluation in the Soviet tradition was traditionally seen as an inspection-like function which would search for small mistakes for which people could then be punished for (demotion or worse..).
In Africa, the perception is quite different. People see you as coming to listen, investigate and relay what you have found. Those working with NGOs are now familiar with evaluation.
Of course, cultural issues and how you are percieved can affect your evaluation. I don’t believe there are any quick learning points except to understand as much as you can about the cultural you are working in – and to test your evaluation methodology and questions by discussing with local people prior to any gathering of data.
This article (pdf) has some interesting points on evaluating across cultures, for example, explaining local relevance and usefulness of the evaluation and to be careful in the use of questionnaire types (such as the Likert scale) which may be misunderstood in some contexts.
Glenn
Internal communications and measurement
For those interested in measuring internal communications (communicating with staff within an organisation), here is an interesting article by Susan Walker from the Handbook of Internal Communication:
“Measurement is not just an optional extra for communicators, but an essential part of their professional tool kit. It has been seen sometimes as a threat (will they cut my budget? will they cut me?) It can be, however, an exciting opportunity to evaluate, guide and direct communication initiatives and investment.”
Glenn
network mapping tool
As regular readers will now, I am interested in network mapping and have undertaken some projects where I have used network mapping to assess networks that have emerged as a result of conferences.
Here is quite an interesting tool, Net-Map, an interview-based mapping tool. The creators of this tool state that it is a “tool that helps people understand, visualize, discuss, and improve situations in which many different actors influence outcomes”.
Read further about the tool and view many of the illustrative images here>>
Glenn
Event scorecard
In the work I do to evaluate conferences and events, I have put together what I believe is a “neat” way of displaying the main results of an evaluation: an event scorecard. In the evaluation of a conference that occurs every year in Geneva, Switzerland, the LIFT conference, the scorecard summarises both qualitative and quantitative results taken from the survey of attendees. Above you can see a snapshot of the scorecard.
As I have evaluated the conference for three years now, we were also able to show some comparative data as you can see here:
If you are interested, you can view the full scorecard by clicking on the thumbnail image below:
And for the really keen, you can read the full evaluation report of the LIFT08 evaluation report (pdf).
Greetings from Tashkent, Uzbekistan from where I write this post. I’m here for an evaluation project and off to Bishkek, Kyrgyzstan now.
Glenn
Tonsils, run over dogs and comparisons
In evaluation, we often make judgements based on “feelings” or “gut reaction” without any proper inquiry or comparison with other data. That is why this story about Ludwig Wittgenstein, the Austrian philosopher appealed to me. Apparently he telephoned a friend in hospital, Fania Pascal, who told the following story:
“I had my tonsils out and was in the Evelyn Nurshing Home feeling sorry for myself. Wittgenstein called. I croaked: “I feel just like a dog that has been run over.” He was disgusted: “You don’t know what a dog that has been run over feels like.”
The point being that Fania Pascal (in the hospital) is making a comparison that she cannot possibly provide any support for – how could she know what it feels like to be a dog that has been run over?
In the same way, you often hear people saying “our results are terrible” – or “we are doing too much of XY”. But my first reaction is “How do you judge that – what are you comparing it to?” – often no real inquiry or comparative data are used (which reminds me of another quote from Groucho Marx).
For those interested, the above quote comes from the book “On Bullshit” by Harry G. Frankfurt – well worth a read.
Glenn
Perceptions of evaluation
I’ve just spent a week in Armenia and Georgia (pictured above) for an evaluation project where I interviewed people from a cross section of society. These are both fascinating countries, if you ever get the chance to visit… During my work there, I was wondering – what do people think about evaluators? For this type of in-site evaluation, we show up, ask some questions – and leave – and they may never see us again.
From this experience and others I’ve tried to interpret how people see evaluators – and I believe people see us in multiple ways including:
The auditor: you are here to check and control how things are running. Your findings will mean drastic changes for the organisation. Many people see us in this light.
The fixer: you are here to listen to the problems and come up with solutions. You will be instrumental in changing the organisation.
The messenger: you are simply channelling what you hear back to your commissioning organisation. But this is an effective way to pass a message or an opinion to the organisation via a third party.
The researcher: you are interested in knowing what works and what doesn’t. You are looking at what causes what. This is for the greater science and not for anyone in particular.
The tourist: you are simply visiting on a “meet and greet” tour. People don’t really understanding why you are visiting and talking to them.
The teacher: you are here to tell people how to do things better. You listen and then tell them how they can improve.
We may have a clear idea of what we are trying to do as evaluators (e.g. to assess results of programmes and see how they can be improved), but we also have to be aware that people will see us in many different ways and from varied perspectives – which just makes the work more interesting….
Glenn
Evaluating advocacy campaigns – No. 2
I’ve written previously about work that others and myself have done on evaluating communication and advocacy campaigns, particulary concerning campaigns that aim for both changes in individual behaviour and government/private sector policies.
In this area, here is an interesting article from the Journal of Multidisciplinary Evaluation, “Advocacy Impact Evaluation” (pdf) by Michael Q. Patton. The article explains how an evaluation was undertaken to evaluate the impact of an advocacy campaign to influence a decision of the US Supreme Court.
What I find interesting is how the evaluation was done – what is called the “General Elimination Method”.
This is where there is an effect (the Supreme Court decision) and an intervention (the advocacy campaign) and they search for connections between the two. They tried to eliminate alternative or rival explanations until the most compelling explanation remained. They did this through interviews, analysis of news, documents and the Court’s decision. The article explains all of this and makes for interesting reading, you can read the article here (pdf).
Glenn
intelligent measurement 