Asking the right questions
When testing people in front of the computer screen, you need to guide them in some way through the process. The usability goals need to be operationalized in order to check if the product meets them. Operationalization means to take an abstract goal and describe it in terms that can be measured and observed. For example, a site for an airline operator has to be “easy to use”. There is no unit to measure easy to use. You can barely offer somebody the explanation “the site was 3 easytouses”. This has to become something more testable. How about “The site has to be easy for new users to book flights”? Not enough. Having this goal we can break it into little tasks.
How much time does it take for a newcomer to get to the page with the flights? How many clicks does it take for the user to book a flight or find information for a flight? You can invent some more…
But how to ask the right questions when testing if these tasks can be accomplished easily without the user being confused?
The questions must be on point and must be easily related to the tester. What I mean is that the task should vary from user to user. Don’t ask “Try to book a flight from Los Angeles’ LAX to Toronto’s Lester B Pearson on the 23rd of November.”, but “Try to book a flight from your current residence to a holiday location for the weekend”. Tasks must be something a user would do. The service will actually be used by these people, and they certainly won’t try to go someplace and do something pretending they’re somebody else.
Except if you’re targeting a service for stalkers or maniacs ![]()
Popularity: 100% [?]

Boiko is still asking himself "What was that question again?"
2 Comments, Comment or Ping
John Muller
I could not agree more. Making sure the tasks are “something a user would do” is of critical importance. For example, when testing the UI of the library on a developer support site .. it is not enough to bring in developers and ask them to find information on JSP and testing, if they have no interest in that area. Instead, I have found it more useful to ask developers to look up info on a topic of interest to them or to help solve some real world coding issue that they are having right now. That is how one can get to observe “real” customer behaviour . otherwise the folks I bring in are focused on what is being tested instead of acting normally and tyring to get something done which is of interest to them. Of course when trying to test very specific features or just a small section of a website, there has to be a balance between letting the customer/tester do whatever they want and leading them to the features you want to test through the questions you ask.
thanks for a great article, John
Jul 31st, 2007
Rostislav
Yes, balance is needed for some small and hard to target tasks, but apart from that we’re limiting “artificiality”.
Thanks for your feedback! I appreciate it!
Aug 1st, 2007
Reply to “Asking the right questions”