Mutually Exclusive Clusters Are Boxes within Which Consumers No Longer Fit

Sometimes we force our categories to be mutually exclusive and exhaustive even as the boundaries are blurring rapidly.

Of course, I am speaking of cluster analysis and whether it makes sense to force everyone into one and only one of a set of discrete boxes. Diversity is diverse and requires a more expressive representation than possible in a game of twenty questions. "Is it this or that?" is inadequate when it is a little of this and a lot of that.

How do you classify a love seat? Is it a small sofa or a large chair for two people? Natural categories are not defined by all-or-none criteria. All birds do not possess the same degree of "birdness" (as shown below for the graded structure underlying the classification of birds). Some birds are more "bird" than other birds, and some mammals might be thought of as birds (bats) because they look and behave more like birds than typical mammals.

In an earlier post, I issued a warning that clusters may appear more separated in textbooks than in practice. I urged that we consider other representations for individual variation. Archetypes work because they reside at the periphery of the objects to be described with all the other species in-between (e.g., birds of prey, household pets in cages, winter migrants, evil dark birds, and white birds of peace). In marketing this is the realm of fans, fanatics and -philes in which it is so easy to visualize the extreme users and name everyone else as hybrid combinations of pure types. R makes the analysis doable. Moreover, with dimensions defined as contrasting ideals (liberal vs. conservative), archetypal analysis mimics hidden dimensions such as those from factor analysis and item response theory.

The Forces behind Diversity Do Not Yield Disjoint Clusters

I have found it best to begin with the forces generating diversity. Why doesn't one size fit all? As a marketer, I look toward consumer demand - whether it originates out of individual usage, preference or need or whether it is manufactured by providers introducing new products and services. I discover that demand is seldom contained within a single box. Some cable television viewers want to watch lots of sports, so let us place them in the sport segment. But they also want movies-on-demand, so we need four segments filling in the 2x2 for sports and movies-on-demand. Hopefully, they do not want to hear the business news because now we have 8 segments in our 2x2x2. As the consumer acquires greater control, the old segmentation scheme seems more and more forced as if we are holding onto a simpler world with everyone crammed into one of only a few silos.

Recommendation systems adopt a different metaphor with the marketplace partitioned by user collaboration and the chunking of offerings into micro-genres. Heterogeneity is seen as coevolving networks of consumers and what they buy. A handful of mutually exclusive boxes will not work in a market that is increasingly fragmenting.

When there are so many alternatives within easy reach of the internet, no consumer can attend to it all. The rows in our data matrix, each containing information from a single individual, become both longer with more options and sparser with limited attention. If we want to play in the marketplace of attention, we will need more than K-means or finite mixture models. Researchers will require even more - the type of easy access provided by R packages such as archetype and NMF.

With the appropriate statistical model, one can uncover such generating processes from the data matrix with consumers as rows and what they want or like in the columns. Using figurative language I have called this approach the "ecology of data matrices" and have suggested the need for biclustering. Yet, there is opposition since we are so accustomed to dividing the analysis of rows and columns into two separate procedures. Most cluster analyses input all the columns to calculate distances among the rows. Factor analysis starts with column correlations computed from data including every row. Biclustering, on the other hand, cares about sorting the cells into simultaneous groupings of row-column combinations. The data matrix gets divided into subspaces, possibly overlapping, with this community of consumers similar on only those groupings of variables.

The simplicity of boxes will not work with consumers in control and more available than any one buyer can attend to or know about. An underlying structure remains, but one defined by the joint interaction of rows and columns. Consumers with common needs and experiences are attract to the same purchase channels and learn about offerings from the same sources. This simultaneous clustering of rows and columns are the blocks from which consumers customized their own personal consumption patterns. Nothing forces the consumer to select only one building block. In fact, the opposite is more generally true for most of us play multiple roles (e.g., items purchased for work and play, self and others, necessities and gifts, and the list goes on). To capture such common practices, we need a clustering technique that does not impose a simplistic representation forcing consumers into boxes within which they no longer fit.