Generalizing from Marketing Research: The Right Question and the Correct Analysis

The marketing researcher asks some version of the following question in every study, "Tell me what you want?" The rest is a summary of the notes taken during the ensuing conversation.



Steve Jobs' quote suggests that we might do better getting a reaction to an actual product. You tell me that price is not particularly important to you, yet this one here costs too much. You claim that design is not an issue, except you love the look of the product shown. In casual discussion color does not matter, but that shade of aqua is ugly and you will not buy it.

Although Steve Jobs was speaking of product design using focus groups, we are free to apply his rule to all decontextualized research. "Show it to them" provides the context for product design when we embed the showing within a usage occasion. On the other hand, if you seek incremental improvements to current products and services, you ask about problems experienced or extensions desired in concrete situations because that is the context within which these needs arise. Of course, we end up with a lot more variables in our datasets as soon as we start asking about the details of feature preference or product usage.

For example, instead of rating the importance of color in your next purchase of a car, suppose that you are shown a color array with numerous alternatives to which many of your responses are likely to be "no" or  marked "not applicable" because some colors are associated with options you are not buying. Yet, this is the context within which cars are purchased, and the manufacturer must be careful not to lose a customer when no color option is acceptable. In order to respond to the rating question, the car buyer searches memory for instances of "color problems" in the past. The manufacturer, on the other hand, is concerned about "color problems" in the future when only a handful of specific color combinations are available. Importance is simply the wrong question given the strategic issues.

Because the resulting data are high dimensional and sparse, it will be difficult to analyze with traditional multivariate techniques. This is where R makes it contribution by offering tools from machine and statistical learning designed for sparse and high dimensional data that are produced whenever we provide a context.

We find such analyses in the data from fragmented product categories, where diverse consumer segments shop within distinct distribution channels for non-overlapping products and features (e.g., music purchases by young teens and older retirees). We can turn to R packages for nonnegative matrix factorization (NMF) and matrix completion (softImpute) to exploit such fragmentation and explain the observed high-dimensional and sparse data in terms of a much smaller set of inferred benefits.

What does your car color say about you? It's a topic discussed in the media and among friends. It is a type of collaboration among purchasers who may have never met yet find themselves in similar situations and satisfy their needs in much the same manner. A particular pattern of color preferences has meaning only because it is shared by some community. Matrix factorization reveals that hidden structure by identifying the latent benefits responsible for the observed color choices.

I may be mistaken, but I imagine that Steve Jobs might find all of this helpful.

Looking for Preference in All the Wrong Places: Neuroscience Suggests Choice Model Misspecification

At its core, choice modeling is a utility estimating machine. Everything has a value reflected in the price that we are willing to pay in order to obtain it. Here are a collection of Smart Watches from a search of Google Shopping. You are free to click on any one, look for more, or opt out altogether and buy nothing.

Where is the utility? It is in the brand name, the price, the user ratings and any other feature that gets noticed. If you pick only the Apple Smartwatch at its lowest price, I conclude that brand and price have high utility. It is a somewhat circular definition: I know that you value the Apple brand because you choose it, and you pick the Apple watch because the brand has value. We seem to be willing to live with such circularity as long as utility measured in one setting can be generalized over occasions and conditions. However, context matters when modeling human judgment and choice, making generalization a difficult endeavor. Utility theory is upended when higher prices alter perceptions so that the same food tastes better when it costs more.

What does any of this have to do with neuroscience? Utility theory was never about brain functioning. Glimcher and Fehr make this point in their brief history of neuroeconomics. Choice modeling is an "as if" theory claiming only that decision makers behave as if they assigned values to features and selected the option with the optimal feature mix.

When the choice task has been reduced to a set of feature comparisons as is the common practice in most choice modeling, the process seems to work at least in the laboratory (i.e., care must be taken to mimic the purchase process and adjustment may be needed when making predictions about real-world market shares). Yet, does this describe what one does when looking at the above product display from Google Shopping? I might try to compare the ingredients listed on the back of two packages while shopping in the supermarket. However, most of us find this task quickly becomes too difficult as the number of features exceeds our short-term memory limits (paying attention is costly).

Incorporating Preference Construction

Neuroeconomics suggests how value is constructed on-the-fly in real world choice tasks. Specifically, reinforcement learning is supported by multiple systems within the brain: "dual controllers" for both the association of features and rewards (model-free utilities) and the active evaluation of possible futures (model-based search). Daw, Niv and Dayan identified the two corresponding regions of the brain and summarized the supporting evidence back in 2005.

Features can become directly tied to value so that the reward is inferred immediately from the presence of the feature. Moreover, if we think of choice modeling only as the final stages when we are deciding among a small set of alternatives in a competitive consideration set, we might mistakenly conclude that utility maximization describes decision making. As in the movies, we may wish to "flashback" to the beginning of the purchase process to discover the path that ended at the choice point where features seem to dominate the representation.

Perception, action and utility are all entangled in the wild, as shown by the work of Gershman and Daw. Attention focuses on the most or least desirable features in the context of the goals we wish to achieve. We simulate the essentials of the consumption experience and ignore the rest. Retrospection is remembering the past, and prospection is experiencing the future. The steak garners greater utility sizzling than raw because it is easier to imagine the joy of eating it.

While the cognitive scientist wants to model the details of this process, the marketer will be satisfied learning enough to make the sale and keep the customer happy. In particular, marketing tries to learn what attracts attention, engages interest and consideration, generates desire and perceived need, and drives purchase while retaining customers (i.e., the AIDA model). These are the building blocks from which value is constructed.

Choice modeling, unfortunately, can identify the impact of features only within the confines of a single study, but it encounters difficulties attempting to generalize any effects beyond the data collected. Many of us are troubled that even relatively minor changes can alter the framing of the choice task or direct attention toward a previously unnoticed aspect (Attention and Reference Dependence).

The issue is not one of data collection or statistical analysis. The R package support.BWS will assist with the experimental design, and other R packages such as bayesm, RSGHB and ChoiceModelR will estimate the parameters of a hierarchical Bayes model. No, the difficulty stems from needing to present each respondent with multiple choice scenarios. Even if we limit the number of choice sets that any one individual will evaluate, we are still forced to simplify the task in order to show all the features for all the alternatives in the same choice set. In addition, multiple choice sets impose some demands for consistency so that a choice strategy that can be used over and over again without a lot of effort is preferred by respondents just to get through the questionnaire. On the other hand, costly information search is eliminated, and there is no anticipatory regret or rethinking one's purchase since there is no actual transfer of money. In the end, our choice model is misspecified for two reasons: it does not include the variables that drive purchase in real markets and the reactive effects of the experimental arrangements create confounding effects that do not occur outside of the study.

Measuring the Forces Shaping Preference

Consumption is not random but structured by situational need and usage occasion. "How do you intend to use your Smartwatch?" is a good question to ask when you begin your shopping, although we will need to be specific because small differences in usage can make a large difference in what is purchased. To be clear, we are not looking for well-formed preferences, for instance, feature importance or contribution to purchase. Instead, we focus on attention, awareness and familiarity that might be precursors or early phases of preference formation. If you own an iPhone, you might never learn about Android Wear. What, if anything, can we learn from the apps on your Smartphone?

I have shown how the R package NMF for nonnegative matrix factorization can uncover these building blocks. We might wish to think of NMF as a form of collaborative filtering, not unlike a recommender system that partitions users into cliques or communities and products into genres or types (e.g., sci-fi enthusiasts and the fantasy/thrillers they watch). An individual pattern of awareness and familiarity is not very helpful unless it is shared by a larger community with similar needs arising from common situations. Product markets evolve over time by appealing to segments willing to pay more for differentiated offerings. In turn, the new offerings solidify customer differences. This separation of products and users into segregated communities forms the scaffolding used to construct preferences, and this is where we should begin our research and statistical analysis.

Statistical Models with a Point of View: First vs. Third Person

Marketing data can be collected in the first or third person, and we require different statistical models for each point of view.




Netflix encourages you to adopt a third-person perspective when it surveys your taste preferences by asking how often you watch different genres (e.g., action and adventures, comedies, dramas, horror, thrillers and more). Third-person remembering taps those regions of the brain responsible for semantic memory. We respond as we would in a conversation providing general information about ourselves. Is the Twilight Saga horror or romance? It does not matter since we answer about the genre without retrieving specific movies. Neither do we ponder the definition of the response categories: never, sometimes and often. I never watch horror films because they scare me and I avoid them, which I should not know because I never watch them, except for the horror movies that I do see and call thrillers. Such preference ratings are positioning statements about who we think we are and how we wish to present ourselves.

On the other hand, first-person recollection is needed to rate individual movies that we have seen. We answer by reliving the viewing experience, which is impacted by context (where, when, who we were with and what else we were doing while watching). We call this episodic memory, which is different from semantic memory, with its own region of the brain and its own retrieval process. Someone asks if you liked a particular movie and you cannot remember seeing it until they tell you the actors and describe aspects of the plot. Both of these are examples of episodic memory. First, the movie that was a blur suddenly becomes clear after some detail is mentioned and memories flood your mind. The second example is your first-person recollection that you have had such an episodic memory experience in the past.

We analyze data obtained from third-person remembering using the statistical methods that are most familiar to those with a social science background (e.g., regression, factor analysis and structural equation modeling). Semantic data is, well, semantic, and it has a factor structure that reflects the meaning of words. If you say that you like action films, then any genre associated with action will also be liked, where association is found in the way words are used by marketers, film critics and in everyday conversation. If I mention Jurassic Park, one can bring to mind one or more scenes from the film and perhaps even recall some details about your first viewing. You cannot do the same for the "science fiction" category, assuming that Jurassic Park is science fiction and not fantasy/thriller.

My point is that any questions about the genre or category will be answered by retrieving general semantic knowledge, including the way we have learned to talk about those genres. Thus, if I ask about usage, satisfaction or importance, I will be tapping the same semantic knowledge structures with all the relationships that you have learned over time by telling others what you think and feel and listening to others tell you what they think and feel. It will not matter whether my measurement is a rating or some type of tradeoff or ranking (e.g., MaxDiff). I am not denying that such information is useful to marketing simply that it is at least one step removed from remembered experiences.

This is not the case with first-person recollection that forces the respondent to relive the episode. You recently watched a specific movie and now you give it a rating by remembering how you felt and how much you liked it. Over time you can rate many movies, but only a very small fraction of all that is available. Your movie rating data is high-dimensional and sparse. Moreover, this tends to be the case for episodic data in general when the episodes are occasion-based combinations of many factors (e.g., who uses what for this or that purpose at a particular time in a specific place with or without others present).

In the third-person, we can ask everyone the same question and let them fill in the details. "How important was product quality when you made your purchase?" deliberatively leaves product quality open to interpretation. But in the first-person we ask about a series of specific events: knowing warranty protection and return policy, reviewing user comments, reading expert evaluations, trial usage in a store or through another user, familiarity with the brand, and so on. Of course, product quality is but one of many purchase criteria, so the list of specific events gets quite long and increasingly sparse since potential customers tend to focus their attention on a subset of all the items in our checklist.

As sparse data become more common, R adds more ways to handle it with both supervised (glmnet) and unsupervised (sparcl) packages. The new book by Hastie, Tibshirani and Wainwright, Statistical Learning with Sparsity, brings together all this work along with matrix decomposition and compressed sensing (which is where one would place nonnegative matrix factorization). High dimensionality ceases to be a curse and turns into a blessing when the additional data reveals an underlying structure that we could not observe until we began to ask in the first person.