Developing Questionnaires: Practical Issues

Adherence to Measurement Principles
We are exposed to questionnaires so often in our lives, we tend to take them for granted. In fact, like with interviewing (where everybody is their own expert!) most people think that designing questionnaires is something that they could do rather well.

Well, maybe we can. But the field of questionnaire design has been extensively worked in the last 100 years or so, by thousands of extremely bright people. We have high standards of what to expect from questionnaires, so remember that whatever you create will be perceived in comparison with some truly great questionnaires - classics of our time - as well as some really awful tar-pools.

Here's a list of questions I have been asked from time to time about developing questionnaires and the answers I gave, should have given, or would have given if I knew as much then as I know now. But the most interesting thing is, that there is still scope for acquiring new knowledge. So the questions may remain the same, but the answers may get better.

Should I develop my own questionnaire?
If you have a lot of time, patience, and resources, then go right ahead. You are well advised to do a course in psychological measurement, including a heavy dose of statistics beforehand, and to gain experience with administering and interpreting questionnaires that have already been devised, for purposes outside usability engineering as well as for purposes within. You should ensure that your questionnaire has adequate reliability and validity and that you have an idea of what the expected values are. If this list of qualifications sounds ominous to you, then take the sensible option: use a questionnaire that has already been developed and standardised by someone else.

What's wrong with putting a quick-and-dirty questionnaire together?
The problem with a quick-and-dirty questionnaire is that you usually have no notion of how reliable or valid the questionnaire is. You may be lucky and have developed a very good questionnaire you may be unlucky. However, until you put your questionnaire through the intensive statistical and methodological procedure involved in creating a questionnaire, you just won't know.

A poor questionnaire will be insensitive to differences between versions of software, releases, etc. and will not show significant differences. You are then left in a quandary: does the questionnaire fail to show differences because they do not actually exist, or is it simply because your questionnaire is insensitive and unreliable? If your questionnaire does show differences, is this because it is biased, or is it because one version is actually better?

The crux of the matter is: you can't tell unless the questionnaire has been through the standard development and test process. This is an iterative process, and it takes time and effort. 'Quick' and 'dirty' are not two words that describe it at all well.

Factual-type questionnaires are easy to do, though, aren't they?
A factual, or 'survey' questionnaire is one that asks for relatively straightforward information and does not need personal interpretation to answer. Answers to factual questions can be proven right or wrong.

An opinion based questionnaire is one that asks the respondent what they think of something. An answer to an opinion question cannot be proven right or wrong: it is simply the opinion of the respondent and is inaccessible to independent verification.

Although it is important to check that the respondents understand the questions of both kinds of questionnaires clearly, the burden of checking is much greater with opinion style questionnaires because we cannot sanity check the answers against reality. However, it is important to check that factual questionnaires are being properly answered. Not everybody has the status of the Inland Revenue - if a respondent fails to reply to one of their questions correctly, the respondent can be put in jail!

What's the difference between a questionnaire which gives you numbers and one that gives you free text comments?
A closed-ended questionnaire is one that leaves no room for individual comments from the respondent. The respondent replies to a set of questions in terms of pre-set responses for each question. These responses can then be coded as numbers. An open-ended questionnaire requests the respondent to reply to the questions in their own words, maybe even to suggest topics to which replies may be given. The ultimate open-ended questionnaire is a 'critical incident' type of questionnaire in which respondents explain several good or bad experiences, and the circumstances which led up to them, and what happened after, all in their own words.

Closed-ended questionnaires are good if you are going to be processing massive quantities of data, or if your questionnaire is appropriately scaled to yield meaningful numeric data. If you are using a closed-ended questionnaire, however, encourage the respondents to leave their comments either in a special space provided on the page, or in the margins. You'll be surprised what this gives you.

Open ended questionnaires are good if you are in an exploratory phase of your research or you are looking for some very specific comments or answers that can't be summarised in a numeric code.

How do you analyse open-ended questionnaires?
The standard method is called 'content analysis' and is a subject all of its own. Content analysis usually lets you boil down responses into categories, and then you can count the frequency of occurrence of different categories of response. Although there are many tools for doing content analysis, it is easy to do this with the aid of a simple spreadsheet. Use one column for the items, and in the adjacent column, code the type of response, perhaps as a number or short alphabetic code. Analysis proceeds by categorizing, sorting by the codes, and further categorizing.

Conclusion
In questionnaire design, as most other avenues of life, it is true to say 'there is no such thing as a free lunch.' In general, the more attention is paid to the development of a questionnaire, the more usable it is. All the options - even the simple-sounding ones - have their own problems and questions. If you want to use questionnaires, you should be aware of them.

Dr Kirakowski is a Senior Lecturer in Applied Psychology and the Director of the Human Factors Research Group (hfrg.ucc.ie). He is the author of the Software Usability Measurement Inventory(sumi.ucc.ie) and the Website Analysis and Measurement Inventory (www.wammi.com).