Don’t Always Believe the Hype: Ask for Sound Numbers (and Promises)
I was on my way into work one morning when I heard an ad on the radio. The company guaranteed that it could save its clients up to 50% on the cost of its product or service. Something about the ad made me uneasy, and then I realized that it was the “up to 50%” element of that guarantee. They are hedging on their guarantee. Essentially, the company promises to save its clients somewhere between 0% and 50% on their costs.
Yet, this seems to be the mode of today’s guarantee. Claiming that improved engagement can result in up to 20% improvement in productivity or that the use of social media can improve your customer contact by as much as 60% only implies that you will likely see some improvement, but there’s really no guarantee. Yet, these kinds of claims are made all the time about a variety of interventions.
Now, that is not to say that engagement initiatives and social media are bad ideas. The problem is that a whole lot of consultants, researchers, and companies are trying to convince you that you are guaranteed to see results by using their product, implementing their pet program, or relying on some magic bullet technology. Yet, if you read the fine print and pay attention to how the message is delivered, you can see what the seller is really “guaranteeing,” which often isn’t much.
There are lots of factors that influence the effectiveness of an organizational intervention, whether it is employee involvement, work-life balance, recognition, growth and development, or health and safety. In addition, some of the predictive power of interventions is actually really weak, but they don’t put that in the press release. Instead, they say things like, ‘our measure of engagement is significantly associated with organizational performance’ – even if their measure of engagement only accounts for .001% of organizational performance and is only (statistically) significant because it’s based on a sample of 30,000 people. In situations like that, it’s almost impossible NOT to find “significant” results.
What does this mean for organizations and managers interested in the latest “research” on healthy workplace practices? Well, it means a few things:
- Be skeptical of what you read on the web, especially when it comes in the form of a press release. A press release is not the most effective foundation for a decision. Read the full report with a critical eye.
- Become a better consumer of statistical results. Just because something is significant in a statistical sense does not mean it is meaningful in a practical sense. I iterated this point in a previous posting on health care consumers, but it deserves a mention here as well. Lots of things can lead to “significant” effects, with the most powerful driver being sample size. And, given that lots of consulting firms provide reports using sample sizes in the tens of thousands, statistical significance really takes a back seat to practical meaningfulness. Financially, it doesn’t make sense to spend $100,000 on an intervention that will result in only $20,000 in cost savings. Having an understanding of the real power of an effect is key to interpreting its practical meaning for your organization.
- Finally, be wary of advocates who push a particular type of intervention, regardless of the circumstances, especially when use of that practice will produce greater profit for that person or organization. Many of the reports and so-called studies released by consulting organizations are conducted and released strictly to sell their own products.
There are a lot of competing perspectives, interventions, and approaches for developing a healthy work environment. Many of them have true scientific and/or practical merit, but some are really a wolf in sheep’s clothing. Organizational decision makers would be wise to consider all of these programs with a healthy dose of skepticism, making informed, data-driven decisions. Most of all, organizations should avoid any “guarantee” that doesn’t come with a firm lower threshold of improvement.
We wouldn’t let academic institutions get away with guaranteeing that students with a degree will earn up to 200 times more money, or let doctors get away with guaranteeing that by using their services, people will live up to 50 years longer. Why do we let consultants and management fad salespeople get away with selling us a different fictional guarantee?
Listed below are links to blogs that reference this entry: Don’t Always Believe the Hype: Ask for Sound Numbers (and Promises).
TrackBack URL for this entry: