Why exams mean nothing out of context

One of the advantages of remote usability testing is that you are using participants in their real context.

Before we started Webnographer, James and I attended a lecture given by Robert Sternberg where he told the story about the maths ability of Brazilian street kids living in the in the favelas of Recife. This story helped both of us realise the importance of carrying out usability tests in context.

Three researchers (see: Carraher, Carraher, and Schliemann 1985) carried out research with children aged 9 to 15.  These kids had dropped out of school, and were selling sun screen, and chewing gum on the streets. The researchers worked out that they could set the kids questions by purchasing goods off them. For example, 1,000 minus 300  is the same as giving the kid a 1,000 Cruzeiros note for a product that costs 300 Cruzeiros. Multiplication can be done by asking the kids how much 3 of a product would cost.

In these tests the Brazilian street kids scored 98%. But when they were put into a formalised test setting, and asked instead of how much would 3 apples cost or what 3×9 is, the kids performance dropped to just 37%.

What is scary is that the researchers later tested middle class children in a private school. These kids did very well in the formal exam. But when they had to do transactions with real money in the street, using the same maths, they failed in being able to do the transactions.

This is one example of context and how it affects behaviour. We see it again and again with remote usability tests, that the context of the user really affects their performance. Test people where they normally use a website, and test people in an artificial environment like a lab, and behaviour will be very different.