Analyzing open-ended questions
December 18, 2007 at 9:43 am 35 comments
In an earlier post, I wrote about the advantages of using open-ended questions in surveys. The challenge is once you have 100s (or 1000s) of responses from your target audience – how do you analyze the answers to open-ended questions?
Basically, we draw on techniques developed for analyzing qualitative data – we are looking for patterns and trends in the responses so we can reach some conclusions as to what we are seeing. I summarise the main steps that I would usually undertake:
1. Read through the responses.Yes, as laborious as it may seem, you must read through each response to get a feeling for the data. As you read through the responses, you will probably see some common themes emerging.
2. Create response categories. The second step is to develop categories for the different themes you are seeing. For example, with a question asking for people’s feedback on a website, you will probably be able to group comments into categories such as “content”, “design”, “features”, “service”, etc.
3. Label each comment with one or several categories. As you read through the comments, assign at least one category to each response. This is what is called “coding” and best done in an excel sheet with responses in one column and your category (s) in the next column.
4. Look at what you have. In the example about feedback for a website, you might label half of your responses as “content”. You can then divide the responses on “content” into smaller categories, e.g. “corporate content”, “product content”, etc. By doing this you will start to see what are the trends in the data and the main issues raised by your respondents.
5. Think what are the responses about? Once you have categorised and coded data, it doesn’t do you much credit just to say “some half of people spoke about content; most of these people spoke about the corporate pages on the website”. You must be able to explain what is being said about the subject or theme. For example in the case of “content” – what were people saying about content? Imagine if a respondent said:
“I consult regularly the corporate pages. This information is well-presented but not up-to-date. I never seem to be able to find information on latest priorities and management profiles”
This example contains different comments on aspects related to design, site updating, navigation and missing content. Notice that the comment on navigation is actually not a “content” issue – but would be considered as a “design” issue and needs to be coded accordingly.
6. Identifying the patterns and trends: once the data has been studied and categories determined, the next step is to see what categories are related and where can trends and patterns be identified: are there common themes emerging? Or are there a series of unrelated points being mentioned?
7. Writing up the analysis: Once you have analyzed the data and identified the major patterns and trends your next step is to write a summary of what you have found. This would normally be a descriptive text incorporating comments directly from the respondents. For example:
“In providing feedback on the website, some half of the respondents spoke about content. The main issues raised included the inability to find content and the lack of up-to-date content on management themes. To a lesser extent, the high quality of the product information and the desire for more information on the management team were mentioned. The following comment from a respondent illustrates these points:
“I find the quality of the product information very good. However, the information is often difficult to find and is hidden on the website”.
As you see, I use terms such as “some half”, “main issues” and “to a lesser extent” to illustrate the magnitude of the trends identified. Some prefer to transfer such an analysis into quantifiable terms – such as “some 50%” or “under 30%”. I prefer not to – but if you are dealing with very few responses, it’s better to mention the precise numbers such as “5 out of 20 responses preferred…”.
Good luck with your analysis!
Glenn
Entry filed under: Evaluation tools (surveys, interviews..).
35 Comments Add your own
Leave a comment
Trackback this post | Subscribe to the comments via RSS Feed
intelligent measurement 
1.
lan | December 24, 2007 at 11:46 am
good
2.
Naveen | May 7, 2008 at 7:33 am
Good article on post coding … I tried looking elsewhere for this kind of summary but couldnt find it. Thanks
3.
Glenn | July 8, 2008 at 2:38 pm
thank you Naveen!
Glenn
4.
obigaba kenneth B | November 24, 2008 at 10:25 am
Thanks Naveen for your guide.Am surely releaved from the fear i had about analysing my data collected using open ended questionnaire.Need more of your guide on developing charts and tables.Other wise i will be great full if you send me more of your guideline.My research is about assessing wetlands degradation and their restoration potentials.
Obigaba kenneth Bigezikyi-Msc student -Makerere university,Kampala-Uganda
5.
Obwocha | November 26, 2009 at 11:01 am
Its an eye opener to me, detailed description of facts to aspiring scholars like me.
6.
Khyati | December 12, 2009 at 5:46 pm
Very good article. Thanks!
7.
Alice | December 22, 2009 at 10:03 am
thank you so much …..was really in need of this….total help..god bless..
8.
Katie Temple | March 17, 2010 at 3:57 pm
It seems like to have a good result one have to go through the steps to conduct a survey.
9.
Joy | April 29, 2010 at 4:31 pm
Wow Glenn!! You really helped me out with this article.
Thanks alot !
10.
william | July 12, 2010 at 5:52 am
Thanks alot Glen, you have been of much help to me and my research which involves open ended questions. i can now comfortably analyse them.
11.
Uma | July 14, 2010 at 6:47 am
thnx it is helpful to me
12.
Naomi | July 29, 2010 at 9:11 am
Waooh! This is very good. Thanks
13.
Samantha | August 15, 2010 at 1:58 pm
thanks! you just saved my life 🙂
14.
Dana C. | October 13, 2010 at 11:10 am
Hello, I have a question for you. On step 3 you said to label each comment with one or several categories. Do you happend to know other methods of labeling, that would cause less errors. This labeling method has certain disadvantages like: forgetting to code part of the answer or subjectivity.
Thank you,
D.
15.
Glenn | October 27, 2010 at 4:40 pm
Yes, one alternative to avoid errors is to “double code”. That means have two people do the coding to see if the same understanding is found. Both persons then compare the codes they have given and attempt to try and resolve differences seen.
16.
ceetrus | November 2, 2010 at 5:23 pm
I find this very helpful. One more question, is open ended question considered as thematic analysis?
I’m not so sure of qualitative research and attempting to do one. I have 2 questionnaires (open-ended) and a Likert scale questions, something that I am quite used to…well..I guess I am. 🙂
Thanks for your reply in advance.
17.
glenn | November 11, 2010 at 7:03 am
for open ended questions you would use the same techiques that you use for analysis of texts in general – qualitative methods (could also be called thematic analysis).
That you can use for the open-ended questions but obviously for the Likert questions you use quantiative analysis – qualitative is not applicable
Glenn
18.
Lebo | January 22, 2011 at 5:51 pm
Hi Glenn,
Fab job you’re doing here. Jah bless!
Lebo.
19.
me den | February 22, 2011 at 1:22 am
good work,dude!
20.
ketaki | May 23, 2011 at 7:23 am
it helped!!!!:)
21.
Masego Monare | February 25, 2012 at 10:09 am
hi,
you have really opened my eyes, i thought i wouldnt make on research, but all the comments and guides have helped. Stay blessed
Masego
22. How to Find Out Which Information Product to Sell – A 3-Step Guide | Internet Business Mastery | April 16, 2012 at 10:18 am
[…] Analyzing Open-Ended Questions by intelligentmeasurement […]
23.
eden | June 14, 2012 at 10:31 pm
Glenn, question – I’m finishing writing up a research paper. I have a couple of open-ended questions that I’ve analyzed exactly as you’ve described in this post. One of my professors asked me to site something that backs up the process that I used, and then asked if it was just a known process… So my question to you is – do you know if this process is well-known enough in academic circles so that I can just explain the process I went through, or do I need to come up with an article or two that justifies my process of coding? Any help you can give me would be really appreciated. Thanks!`
24.
Glenn | June 18, 2012 at 7:38 pm
Hi Eden, this is a well know process, but you may be best to cite some sources. Look for any sources on qualitiative content analysis and you will find more or less the steps described above.
Glenn
25.
eden | June 18, 2012 at 7:43 pm
thank you!
26.
gups deol | September 24, 2012 at 10:33 am
Hi, you could use research methods for business students by Saunders, Thornhill and Lewis (2009 ed). chapter 13 describes analysing data, and the categorisation process for qualititative data – should support this article post!
27.
Lynn | July 2, 2012 at 1:52 pm
This was really a great help. Thanks for this
28.
Open Surveys and their analysis | proventivesolutions | August 9, 2012 at 12:56 am
[…] open-ended questionsWebsite: https://intelligentmeasurement.wordpress.com/2007/12/18/analyzing-open-ended-questions/ Downloaded […]
29.
gups deol | September 24, 2012 at 10:29 am
really helpful for my dissertation thank you!
30.
Nurhussen | December 5, 2013 at 9:37 pm
Thank you very much.I think this is the only best article about analysing open-ended questions.
31.
Nurhussen | December 5, 2013 at 9:41 pm
Thank you very much. This is the only best explanation.
32.
Yana | March 23, 2014 at 9:32 am
Hi Glenn,
Would like to check which source did you obtain the above information from?:) Is there like theorist that came up with those like thematic analysis or content analysis? thanks:) hehe
33.
Glenn | March 26, 2014 at 11:19 am
HI Yana,
there are many textbooks which cover this topic, as one reader suggested, try:
research methods for business students by Saunders, Thornhill and Lewis (2009 ed). chapter 13 describes analysing data, and the categorisation process for qualititative data
34.
bheki | March 25, 2015 at 4:27 pm
very useful piece, am currently doing my MBA data analysis right now, am have google this intelligent measrument website. many thanks
35.
Aastha | December 19, 2019 at 9:46 am
Hi thanks for this information. Actually we do Open end coding , but we require a macro that will be able to code open ends by matching the code structure and assign codes accordingly, Can you please help