07 Apr. 2016 | Comments (0)
Managers use measurements every day to guide their analyses, decisions, and planning. But even the simplest measurements can mislead. Indeed, measurement is much more difficult than most managers appreciate. Managers must protect themselves by understanding weaknesses in measurements and taking these weaknesses into account as they use them.
Consider the following scenario involving one of the most basic units of measure: time. I have a clock in my office that synchronizes itself at 1:00 am every day using a signal sent out by the National Institute of Standards and Technology (NIST) in Fort Collins, Colorado, which has an advertised precision in the fraction-of-a-second range. It should be trustworthy, right? Not so — it can be wildly inaccurate. For example, on Thursday, October 29, at 4:44 PM (as verified three different ways), the clock read, “Friday, November 2, 4:52 PM.” The last time November 2 fell on a Friday was in 2012, and the next time is in 2018; essentially, the clock was off by about three years and eight minutes!
Of course, we could argue that this is just a flaw in the product. But in my personal life, I can also cite bad measurement stories involving my bathroom scale, smartphone, and GPS. While hardly the stuff of cocktail party banter, when people think about it for a moment, many recall similar instances when supposedly simple measurements were way off.
My point here is not to trash my devices or the work of NIST. Rather, it is to illustrate that, even under the best of circumstances, measurement is incredibly difficult. And the things of interest to managers, such as the size of a market, the effects of an advertising campaign, and the true costs associated with a poor quality product, are far more complex than time, weight, and distance. Because of this I urge managers to adopt a healthy skepticism of all measurements until they understand them deeply.
Here are a few steps to dig into your measurements and identify if they can be trusted:
1. Clarify what you want to know. Too many managers give this step short shrift. Suppose you need to want to know how long a process consisting of three steps (A, B, and C) takes. It is easy for people to interpret this: the time it takes to complete A plus the time it takes to complete B plus the time it takes to complete C. But here’s an alternative: the time it takes to complete A plus the time it takes to complete B plus the time it takes to complete C, plus the queue time between each step. Both have valid uses, but owing to the queue time components, the two can be quite different. So you must be clear about you really want.
2. Understand how actual measurements line up with what you want to know. The next step is to see how closely the measurements you’re getting line up with what you really want. Consider an internet advertising campaign aimed at increasing brand awareness. Since you can’t measure “increased brand awareness” directly, indirect measures like the number of page views and click-throughs may have to suffice. But are these good surrogates for what you really want? Perhaps, perhaps not. So strive to distinguish “pretty close” from “a good-enough indicator” from “not what I had in mind.”
Keep in mind, though, that some degree of compromise is often needed. Years ago, Dr. Lloyd Nelson observed, “The most important figures that one needs for management are unknown or unknowable.” So you may have to settle for a less-than-perfect measurement, but you must know exactly what you are settling on.
3. Account for weaknesses in the measurement process. All measurement devices have failure/error modes, and it is important that managers get to know them. People lie in surveys, countries manipulate numbers to look good, weather vanes fill up with sand, and so on. More complicated measurements involve a detailed definition of terms, sampling, remote data collection, and extensive analyses. And things can go wrong at any step along the way. Managers must understand the entire process, not just the points of actual data collection. If you’re measuring customer complaints, listen in to survey calls; if you’re measuring factory productivity, visit a factory; and so forth. There is no need to become an expert in all aspects of the process, but you must develop a feel for the weak links.
4. Subject results to the “smell test.” A fascinating New York Times article gives example after example of drivers ignoring facts right in front of them (such as “road closed” signs) in favor of GPS measurements. Managers make similar mistakes time and again. This month’s results come in far better (or worse) than expected, and they scurry around to explain why, rarely considering the possibility of a bad measurement. I advise managers to develop a keen sense of smell. When results just don’t smell right, dig deeper!
Then act on what you find. Eliminate bad data from critical analyses and, if a device proves itself untrustworthy, don’t hesitate to get rid of it altogether. I never trust my clock — indeed I only keep it around as a cautionary tale (and I’ve hidden it away so others aren’t victimized when it goes awry).
Measurement is increasingly important to all managers. Some, because they seek to advance data-driven cultures; others, because smart, connected devices are shining the light of measurement on the heretofore unmeasurable. But it is critical that managers appreciate that measurement — all measurement — is fraught. Good measurements enlighten, but bad ones mislead. Learn how to dig into the measurements more deeply, so you can protect yourself.
This blog first appeared on Harvard Business Review on 03/17/2016.
View our complete listing of Human Capital Analytics blogs.