Wednesday, February 22, 2012

10 Characteristics of a Good Measure and 7 Pitfalls to Avoid


It is not enough to simply create a numeric measure. The measure should accurately reflect the process. We use metrics to base decisions on and to focus our actions. It is not only important to measure the right indicators, it is important to measure them well.

To be effective and reliable, the metrics we choose to use need to have ten key characteristics. The following table was adapted from Keebler (1999) which suggest the qualitites to look for in indicators


A good measure:
Description:
Is quantitative
The measure can be expressed as an objective value
Is easy to understand
The measure conveys at a glance what it is measuring, and how it is derived
Encourages appropriate behavior
The measure is balanced to reward productive behavior and discourage “game playing”
Is visible
The effects of the measure are readily apparent to all involved in the process being measured
Is defined and mutually understood
The measure has been defined by and/or agreed to by all key process participants (internally and externally)
Encompasses both outputs and inputs
The measure integrates factors from all aspects of the process measured
Measures only what is important
The measure focuses on a key performance indicator that is of real value to managing the process
Is multidimensional
The measure is properly balanced between utilization, productivity, and performance, and shows the trade-offs
Uses economies of effort
The benefits of the measure outweigh the costs of collection and analysis
Facilitates trust
The measure validates the participation among the various parties

Choosing the right metrics is critical to success, but the road to good metrics is fraught with pitfalls. As your endeavors to become more metrics-driven, beware of errors in the design and use of metrics.

Common mistakes include:

  • Metrics for the sake of metrics (not aligned)
  • Too many metrics (no action)
  • Metrics not driving the intended action
  • Lack of follow up
  • No record of methodology
  • No benchmark
  • Underestimation of the data extraction
Although there may never be a single perfect measure, it is certainly possible to create a measure or even multiple measures which reflect the performance of your system. If the metrics are chosen carefully, then, in the process of achieving their metrics, managers and employees will make the right decisions and take the right actions that enable the organization to maximize its performance. These guidelines will make sure you pick the right indicators and measure them well.



Subscribe to my feed Subscribe via Email LinkedIn Group Facebook Page @TimALeanJourney YouTube Channel SlideShare

11 comments:

  1. Tim,

    Great reminder of the value of good metrics, and how to create them. Having spent what is now 14 years (egads!) in project planning & analysis, there's always those who use measurement to show what they want, not what is really happening. Although that solves a short-term problem, in the future, when looking at past data to determine future plans, using those bogus metrics only leads to terrible plans - which results in increasing variances and more and more need to "massage" the numbers.

    The process itself is a waste (rework, defects, overproduction) and the interpretations of the resulting data only lead to other wastes within the organizations, too.

    ReplyDelete
    Replies
    1. David, I believe that measures should be minimal. We need to apply lean thinking to the measures we use to capture the performance of our processes. The minimum set of measures possible to provide feedback on the system is desirable (and the goal).

      Delete
  2. Thank you for your good posts of a major subject. I am a process improvment guy in a manufacturing facility (now 23-1/2 years and counting) which rolled out "lean" back in 2008. We have improved in many areas and continue to drive for more improvement which leads to the increasing need for value-added accurate metrics. I've found it's difficult to measure performance related to 5S even though it's obvious that we have a safer work environment and we are more efficent on the shop floor. It would be fabulous to be able to capture some actual numbers for this activity. It's not difficult in our Energy and Waste efforts, VSM efforts or our TPM efforts. Any suggestions?

    Thanks,

    Bill

    ReplyDelete
    Replies
    1. Bill, 5S like many of the Lean tools and techniques are sometimes difficult to measure the full impact of the improvement. This is in part because of the intangible benefits that can come in the form of people development. Anyway, like any measure I like to start with the benefits of the initiative and then work out how we will see that improvement. If you think you will see 5S measures in the P&L I think you will be looking for a while. I would argue overtime you will see that improvement though.

      My friend Jon Miller wrote 2 good posts on this subject you should review:

      http://www.gembapantarei.com/2010/01/whats_the_roi_of_a_5s_program.html

      http://www.gembapantarei.com/2010/04/a_tracking_method_for_5s_programs.html

      This is probably a good topic for a post. Stay tuned for an update.

      Delete
  3. Thank you Tim. I'll review Jon's posts. Also, although we can completely prove it here at our company it's interesting to note that we have the lowest number of safety related lost work days since starting 5S in 2008 and we are working less overtime while shipping record numbers of product with a higher on-time average. All good stuff.

    ReplyDelete
  4. I really liked your 10 Characteristics of a Good Measure but you didn't define what you meant by "a measure" and I am wondering whether you would define it the same as I do. To me a measure is the charcteristic(s) by which performance will be judged or whaat we look for to see if our objective has been achieved. Thus a process might have, repeatability, safety, resource utlization, capacity etc as its measures. But to these one needs to choose units of measure and a method of measurement. I also think its how these are expressed that is important. e.g. in a hospital they often have waiting time as a measure but express in such a way that the administrators can cheat by permitting them to effect when a patient is put on the list and taken off.

    ReplyDelete
    Replies
    1. David you make a good point.

      A measure is a number or quantity that records a directly observable value or performance.

      All measures have a unit attached to them.

      In contrast, an indicator is an indirect measure or a predictor (such as a leading Indicator) of performance.

      Delete
  5. I prefer raw data, sent to cloud, which can later be re-analyzed in light of new interest, or events.
    The Flaw of Averages is good book to read before you start using stat methods of any kind.

    ReplyDelete
  6. I think you left off one crucial characteristic: it has to be readily and consistently measurable. In other words, if you measure the same thing several times by different people you should get the same answer each time. Lack of this characteristic is sometimes the downfall of measurement systems.

    ReplyDelete
    Replies
    1. Dennis, that is a good characteristic. I agree with you. Many places collect data but never bother to a) understand if it is any good and b)even analyze it. Consistency and readily measurable are important.

      Delete
  7. Great discussion on a topic near and dear to my heart. I often use as an example of "overproduction" (one of the lean wastes) the collection of data that are never used or analyzed.

    ReplyDelete