I just saw this article in my Twitter feed and it really illustrates the issue of what happens when executives come up with an answer and then try to validate it without really asking the right questions.

Ignore the customer experience, lose a billion dollars (Walmart case study)

The article states that Walmart asked customers in a survey if they wanted the Walmart stores to be less cluttered. This is such a common question and gets asked in all kinds of scenarios from stores to websites. How many times have I heard in a usability study that something looks too cluttered? Hundreds. The trick though, as Walmart found out, that that doesn’t mean anything should be removed. Walmart’s error was assuming removing something means less cluttered. The term “cluttered” has come too mean too much stuff, but does it mean really too much stuff or the appearance of too much stuff? If you had the same amount of stuff in a 9,000 square foot house and then had it in a 400 square foot studio apartment, would if feel like too much stuff in both scenarios? What if Space Bags were involved?  What if the stuff were diamonds? What if the stuff was chocolate?  The questions can go on and on, but the point is you can’t just ask “Is this cluttered?” because unless they are looking at some minimalist design, customers will answer yes, but that isn’t what they mean.

To find out what they mean, you have to watch what they do and ask a lot more questions, repeatedly. Or you could go through an exercise like Walmart has, but I wouldn’t suggest it.