When Enron, which Fortune named “America’s Most Innovative Company” for six consecutive years, collapsed in 2001, there was no shortage of blame to go around. Many blamed corruption and greed, others hubris, and still others incompetence. But Nobel laureate Paul Krugman offered a different explanation.
In his New York Times column, Death By Guru, he identified the culprit as trendy management theories. He argued, essentially, that consultants selling compelling narratives persuade gullible CEOs to abandon fact-based analysis and good sense. Inevitably, reality catches up, and there is a price to be paid.
Yet there is a deeper problem. When a big consulting company sells a story, it’s not merely giving advice, but presenting its assertions as being backed by “research.” When they use the term, however, they rarely, if ever, mean that they have used the scientific method and employed appropriate controls for error and bias. These are the tricks of the trade.
1. The Case Study
Organizations are often inscrutable and hard to research. That’s why consultants tend to rely heavily on case study interviews. These purport to leverage consultants’ rare access to insiders in order to glean insight others cannot. These can sometimes be helpful, but they also have severe limitations that are often overlooked.
First, with shareholders and customers to please, managers are seldom eager to talk about failures. So we usually only hear about successes. Those, of course, are important but also subject to survivorship bias. For example, if a risky strategy results in 1% of the firms being wildly successful and 99% going out of business, then we’ll tend to hear glowing accounts of that lucky 1% and we’ll miss the vast majority that flamed out.
Another issue with the case study method is that it is necessarily limited. When researchers conducted a case study on a company I used to run, to take just one example, they interviewed insiders (including me) and did their best to interpret what they heard and what they could glean from background information about the market.
While I don’t think anything was inaccurate, it didn’t capture the reality either. Only a handful of people were interviewed, almost all of them from a single part of the business, and none of them, besides me, were involved in making decisions. The issues presented in the case study simply weren’t the ones we were actually wrestling with.
Finally, case studies rarely offer much in the way of documentation. In more rigorous fields like, say, sociology or psychology, researchers are expected to share their methods and data so that others can interpret them. Unfortunately, that’s rarely true in consulting research.
2. The Industry Survey
Besides case studies, another common technique for analyzing business trends and performance is the executive survey. Typically, a research company or consulting firm sends out questionnaires to executives. It then publishes a report with findings like 57% of companies are in the emerging stage of digital transformation and 49% of C-suite executives plan to diversify their supply chains.
This appearance of scientific rigor is largely a mirage. Yes, there are numbers, graphs, and pie charts, much as you would see in a scientific paper, but important information is often missing, such as how respondents were selected, how questions were phrased, how representative the sample was, and how the answers were interpreted.
Another problem is that these types of studies emphasize what a typical executive thinks about a particular business issue or trend. So what they really measure is the current zeitgeist, which may or may not reflect market reality. A great business strategy does not merely reflect what typical executives know, but exploits what they do not.
Perhaps most importantly, these types of surveys are generally not marketed as simple opinion surveys, but as sources of profound insight designed to help leaders gain an edge over their competitors. The numbers, graphs, and pie charts are specifically designed to look “scientific” in order to make them appear to be statements of empirical fact.
Surveys can tell you what executives think. They cannot tell you whether those executives are right or even well informed. As the mathematician G.H. Hardy once put it, “It is never worth a first-class man’s time to express a majority opinion. By definition, there are plenty of others to do that.”
3. The Industry Conference
A third weapon that consulting firms have in their arsenal is the industry conference, in which “industry experts” appear to speak along with the firm’s in-house gurus. The idea is that participants will get the opportunity to learn about the latest research and trends to help them make better decisions and get ahead of the competition.
Yet what is the nature of the expertise being presented? Often, the executives who provide case study interviews and respond to industry surveys are the same people who appear onstage and sit in the audience. Essentially, consulting firms are selling their clients’ ideas back to them.
So while these conferences are touted as promoting “out-of-the-box thinking,” what they really do is create echo chambers for the narratives that the consulting firm is already pushing. Executives are wined and dined. A consensus is forged, and ideas that don’t conform are subtly discredited in favor of the officially backed zeitgeist.
That doesn’t mean the ideas are bad, necessarily. And consultants often do outstanding work that benefits their clients. Yet it is also a salient fact that consulting firms have been behind the strategies involved in many major business scandals, whether it’s Enron, Purdue Pharma’s promotion of opioids, or Allstate’s questionable efforts to lowball claims.
It’s a simple truth that consultants’ main focus is on making their clients happy. Sometimes that means telling hard truths, and sometimes that means telling them what they want to hear.
Don’t believe everything you think
In Richard Feynman’s 1974 commencement speech at the California Institute of Technoloy, or Caltech, he recounted going to a new-age resort where people were learning reflexology. A man was sitting in a hot tub rubbing a woman’s big toe and asking the instructor, “Is this the pituitary?” Unable to contain himself, the great physicist blurted out, “You’re a hell of a long way from the pituitary, man!”
His point was that it’s relatively easy to make something appear “scientific” by, for example, presenting data in official-looking charts and tables, but that doesn’t really make it science. Consultants often boast that their “research” consists of hundreds of case study interviews and large databases containing data on thousands of firms. But without the proper methods and controls, those things are meaningless.
Another problem: While many top-tier consultants have blue-chip résumés coming from top schools, very few have had experience operating a business. Often they’re people who’ve been told they’re smart all their lives, have never failed, and expect others to be impressed by their ideas, not to examine them thoroughly.
The truth is that the world is a messy place. Many of the most important assets, like relationships and trust, don’t fit neatly into Excel spreadsheets. Ideas can only be validated forward, never backward. There are countless ways a seemingly sound, logical idea can fail in the marketplace.
So take ideas that are purported to be “scientifically backed” or “research-based” with a grain of salt. An idea is just an idea. The marketplace will tell you if it’s a good one or not.
