User Testing is always good. It always yields some worthwhile and occasionally unexpected results. After the egos involved recover, they realize it is always better & cheaper to know sooner.

But User Testing is not perfect. There are different potential flaws in different contexts, but let’s take testing an existing web application as an example, as this is very common in a discovery process.

Testing an existing site is like watching life in a rear view mirror. It’s a reflection of a snippet of experience; a kind of shadow casting of real use.

Often, people evaluate what they have access to and make determinations on the basis of what they can see. This can be a very limited view of reality, and tends to reinforce that which has already been invented; to paraphrase a tautology: That which is good exists, that which exists may or may not be good.

So it has to be taken with a grain of salt. There are famous business cases of researching an existing context and making regrettable decisions. The research that Ford did as to whether or not they should have a drivers side rear door on the Windstar strongly indicated that consumers did not care.

Dodge ignored similar research and added one, exercising common sense and taking the market from Ford. What is the lesson? Once consumers saw it, and used it, they realized that it was a valuable addition. Another chapter was added to Ford's seemingly endless downward trajectory.

To suggest that the fault is the user testing is not quite the whole story. The user testing counted what was known to be countable; what it cannot do is invent, or count what it cannot quantify. That is the design and ideation part of the process, much more messy & out of control than testing should be.

However, a mix of interviews, done by objective mediators, can reveal trends that designers & developers may not have considered or may have de-emphasized in their process.