A Simple Taxonomy of Question Types for Research

Adherence to Measurement Principles
Very often one hears "oh, let's do a questionnaire" as an answer to a problem that may (or may not!) have been well specified to start with. Soon after, one sees ideas for questions flying around the table. Although this presupposes a healthy spirit of inquiry, it is sobering to look a little more closely at the form of the questions being proposed. In fact, one finds there are three basic types of questions:

Factual-type questionnaires
Such questions ask about public, observable information that it would be tedious or inconvenient to get any other way. For instance, number of years that a respondent has been working with computers, or what kind of education did the respondent get. Or, how many times did the computer break down in a two-hour session, or how quickly did a user complete a certain task. If you are going to include such questions you should spend time and effort to ensure that the information you are collecting is accurate, or at least to determine the amount of bias in the answers you are getting.

Opinion-type questions
These ask the respondent what they think about something or someone. There's no right or wrong answer, all we have to do is give the strength of our feeling: do we like it or not, or which do we prefer? Will we vote for Mr A or Mr B? An opinion survey does not concern itself with subtleties of thought in the respondent, it is concerned with finding out how popular someone or something is. Opinion questions direct the thought of the respondent outwards, towards people or artefacts in the world out there. Responses to opinion questions can be checked against actual behaviour of people, usually, in retrospect ('Wow! It turned out that those soft, flexible keyboards were a lot less popular than we imagined they would be!')

Attitude questions
Attitude questions focus the respondent's attention to inside themselves, to their internal response to events and situations in their lives. There are a lot of questionnaires consisting of attitude questions about experiences with Information Technology, the Internet, Multi-media and so on. These tend to be of interest to the student of social science. Of more use to the HCI practitioner are questionnaires that ask the respondent what their attitudes are to working with a particular product the respondents have had some experience of. These are generally called satisfaction questionnaires.

In our research, we have found that user's attitudes to working with a particular computer system can be divided up into attitudes concerning:

• The user's feeling of being efficient
• The degree to which the user likes the system
• How helpful the user feels the system is
• To what extent the user feels in control of the interactions
• Does the user feel they can learn more about the system by using it.

We can't directly cross-check attitude results against behaviors in the way we can with factual and opinion type questions. However, we can check whether attitude results are internally consistent and this is an important consideration when developing attitude questionnaires.

Can you mix factual and opinion questions, closed and open ended questions?
It doesn't do to be too purist about this. It's a good idea to mix some open-ended questions in a closed-ended opinion questionnaire and it's also not a bad thing to have some factual questions at the start of an opinion questionnaire to find out who the respondents are, what they do, and so on. Some of your factual questions may need to be open-ended, for instance if you are asking respondents for the name of the hardware they are using. I have increasingly found the use of open-ended questions adds an extra dimension, but does not actually replace, a well-selected series of closed-ended questions.

This also means you can construct your own questionnaire 'booklets' (using the term generically - on the web this means a collection of web pages!) by putting together a reliable opinion questionnaire, for instance, and then add some factual questions at the front and maybe some open ended opinion questions at the end.

Different cultures have different expectations about the way a questionnaire booklet should be presented, so this is not the place to give hard-line advice about what goes where.

Conclusion: how do you go about putting a questionnaire together?
In short: iterate! From the very earliest beginnings, try your drafts and your first ideas out on the kind of people who may be replying to the questionnaire for real later. Watch them as they fill out your first paper-and-ink versions, and ask them afterwards what is good and what can be improved. Before you go public on the web, make sure you get at least a dozen people, not related to your project, to go through the 'final' version, and either watch them and hear what they have to say, or ask them the same questions: what is good, what can we improve?

If some questions prove difficult, it is often very illuminating to say to respondents: 'what do you think this question is asking?' Sit back and listen. This can sometimes become a very sobering experience as you realize that there are many ways in which to interpret what you thought was an extremely plain and simple question. In questionnaire design, Murphy's law can be stated thus: 'if a respondent can mis-interpret a question, they will mis-interpret that question.'

Dr Kirakowski is a Senior Lecturer in Applied Psychology and the Director of the Human Factors Research Group (hfrg.ucc.ie). He is the author of the Software Usability Measurement Inventory (sumi.ucc.ie) and the Website Analysis and Measurement Inventory (www.wammi.com).