WCM Workshops, Part One:
The day before the opening of the Warranty Chain Management Conference next month, attendees have a chance to hear two experts talk about the pitfalls of relying upon seemingly simple statistical measures of warranty cost.
Those who can get to the Warranty Chain Management Conference in San Diego on Tuesday, March 4 have an excellent opportunity to take in a pair of half-day courses on warranty statistics.
In the morning, Dr. Vadim Kozyrkov, president of Aculocity LLC presents a three-hour course entitled "Lies, Damn Lies, and Warranty Statistics," while in the afternoon Fred Schenkelberg, a reliability consultant with FMS Reliability, presents a three-hour workshop entitled "How to Best Use Warranty and Field Data."
In phone chats this week, both men said they think that some warranty professionals are misleading themselves by relying on the wrong statistics. And both said that with a little training, warranty professionals could get it right more often.
Schenkelberg, who was with Hewlett-Packard Co. for almost eight years before striking out on his own in 2004, said he wants to explain the design and engineering side of warranty to the finance people who are frequently in charge of funding it.
He said he sees a huge and fundamental communications gap between the two groups, caused in part by their use of different units of measure. One deals in failures per thousand units, while the other deals in cost per thousand dollars of revenue. And they don't know how to easily translate between the two.
"I've been trying to influence product design in a wide range of different products, first at Hewlett-Packard and now across industries," Schenkelberg said. "By simply arguing that we want to make them better or more durable -- focusing on the failure rate -- really wasn't a compelling argument. People would agree, yet it wasn't in dollars. So the management decisions weren't supporting just making it better. The competing issues of time to market and bill of material cost would often win, because they had dollars and cents behind them.
Warranty Cost per Unit
"Converting the failure rate into dollars -- warranty is really my first cut at that. It's a simple question: what is the warranty cost per unit shipped? Often times, that gave it a figure of merit or magnitude that design managers and design teams could then grasp, that they could then understand.
"I got into it because I needed to have better arguments to make a product better, because I was competing with all these other things design teams were considering, and dollars and cents was the best way to do it.
"They can say, 'If I reduce the failure rate I can save X number of dollars.' And that savings might be larger than if they cost-reduced a component.
"I started to do quick estimates, and I found that I needed to use dollars. And then I got more refined and started working with financial organizations to get those warranty numbers.
"Giving the designers the capability to convert failure rates to dollars was the tipping point," Schenkelberg said.
When he was with Hewlett-Packard, he worked in the Electronic Systems Technology Center in the Bay Area, where he helped run the corporate reliability program. "That was where we conducted training, seminars and workshops, and where we did assessments -- very similar to the work I do now."
FMS Reliability is the name of his company, and reliability consulting is what he does. We guessed that his middle name starts with an M, and that FMS is his initials. What we didn't know is how he spends his time.
"About a third of my work is reliability engineering management," he said. "It's working with design teams and companies on how to improve their reliability engineering process, whether it's with the designers, across the whole organization, or with product testing. Where should they invest to make the best improvement?
"About a third of my time is specific activities such as facilitating risk assessment, failure modes and effects analysis, or doing the field data statistical analysis -- taking a look at failure returns and what they really mean. Basically, in these instances I'm just a hired hand to do something that they need expertise on.
"And then about a third of my time is teaching. I teach with the University of Maryland and with Stevens Institute of Technology, and also with seminars for specific companies." One school has him teach a two-week "short course" in the summer and the other has him do an occasional 40-hour course for working professionals over the span of a week. "But both of them are three credit-hour graduate level courses," he added.
Schenkelberg said he logged 130,000 miles on his frequent flyer account last year, and already this year he's past 30,000 miles. For the WCM workshop, though, he will face just a 440-mile trek south from San Jose.
Not Enough Time?
It's only a three-hour course, though, so the question is how much does he expect to teach? Schenkelberg said he's taken a look at the signups so far, and it appears to him from the job titles that he'll be hosting more finance and warranty accounting folks who are interested in hearing from him about the design and engineering side of the business. So he'll focus on teaching 1) the dollars of warranty per unit shipped, 2) how much does it cost an organization when something fails in the field, and 3) how to convert the dollars and cents in a way that designers can use.
"My first assumption is that the audience is familiar with warranty and collecting the dollars and cents side of warranty and warranty accruals. And then what I want to discuss with them is from a designer's point of view, what do they need in order to best use that information?" What he hopes to impress upon both the engineering types and the finance types is the magnitude of warranty cost, and how it's comparable to other factors that they already regularly consider.
"They find that the warranty per unit shipped is on the same order as the parts cost or the bill of materials cost. So it immediately gets an equal footing in the way they typically think about their design," Schenkelberg said.
But that's not all, as they say on those late-night TV offers. Knowing how much or how many is rarely enough. People want to know rates and ratios. How many per thousand? How much per dollar? And that's where he said the trouble usually starts.
"I've seen organizations collect field data, and they take, for example, the number of failures that come in the door, or the number of phone calls per month, and then they summarize that and strip it down, and so on, and collect that information and either they provide a pareto of the types of defects that are coming back or they have some kind of failure rate that they estimate on that."
Entering a "No MTBF" Zone
Schenkelberg said he's seen all sorts of assumptions and estimates attached to those charts that can throw off the calculations and make the data misleading. For instance, he said a lot of people try to calculate a figure for MTBF -- mean time between failures. But they do so in a way that assumes the failure rate is constant over time. "It very often isn't," he noted.
Yet he said he finds that companies frequently mislead themselves by calculating the number of failures divided by the number of products in use, or by the number of hours of operation. This doesn't allow for any measurement of the typical age of a product at the time of failure. Instead, it assumes that both old and new products are equally likely to fail at any given moment.
"Last year I did a little promotion for my business," he said. "I made these little buttons that said 'No MTBF.' Basically, it was the acronym MTBF in a red circle with a slash through it. I gave those out at conferences and to my clients, and quickly ran out of them. But I've been joking that I've been on a crusade to get rid of MTBF as a metric for reliability. It's so misleading, and so misunderstood, and it also simplifies your data unnecessarily."
In the past three years, since he launched his own company, Schenkelberg said he has worked with over 80 design teams. He said many do an excellent job of estimating failure rates over time, with different usage scenarios and different usage profiles. "But that's not well-communicated back to the financial organization in a way that they can then use in their financial models," he said. So the financial types do the best they can with the information they have and the techniques they know.
"What I've seen most often is the CFO will say 'Well, this is similar to another product and here's what our field history is.' It's not a terrible method, but it's fraught with error and uncertainty. And it generally isn't incorporating the engineering knowledge when it exists.
"So the communication, in my mind, is broken going both ways," he said. "Warranty professionals aren't giving the design teams great feedback after the product is shipping, and engineering teams often aren't providing coherent or understandable future predictions to the financial organizations and warranty organizations, so they can do the accruals correctly."
He said he knows a couple of organizations that are working not only on improving the communications, but also in the conveying of some sense of the uncertainty behind the information. How accurate is it likely to be? What are the probable upper and lower values? And what is the most likely outcome?
"They want to be predictable," he said. But with warranty accruals equal to 1%, 2% or even 3% or 4% of sales, there must be some allowance for the unpredictability. Frequently, a warranty surprise can turn into an earnings surprise -- over or under what was expected. So the finance people need to know not just the most likely prediction, but also the high and low estimates.
"Here's as good as we think it can be and here's as bad as we think it can be, because missing your target earnings per share is bad either way, whether you overshoot or undershoot it, as far as Wall Street is concerned," Schenkelberg said.
Another common problem he sees is how warranty professionals fail to account for the lag time between when a product is made or sold and when it fails or requires a repair. Occasionally, he said he's seen companies that track total returns in a given month and compare it to total shipments in the same given month. Unfortunately, this compares different batches of products made in different months, under different manufacturing conditions. And unless all products that are going to fail do so in their first month in the field, it gives a misleading picture of reliability.
When a new product first begins shipments, and there are few if any "old" units in the field, a comparison of new failures to new shipments would suggest a deceptively low failure rate. As sales rapidly increase, the expanding denominator in the fraction used to calculate the ratio would continue to mask the true failure rate. And then after sales peaked and the product line was getting towards the end of its run, the falling sales volumes would make the failure rate look like it was going through the roof, even if it was actually very stable.
Schenkelberg said the time lag between when a product is sold and when it needs a repair can be compensated for by software packages that use something called a "Nevada plot," to track the month a unit was sold and the month it failed. "I don't know why it's named that," he said, "other than if you turn your head sideways and squint, the shape kind of looks like the state of Nevada."
He said such a plot keeps track of how many units were shipped in a given month, and how many months of service they had when they failed, plus of course keeping count of all the units that haven't failed. Then when the calendar flips to a new month, everything is moved over one column, and all the surviving units are credited with another month of service.
"It gives me information on how many shipments I made three years ago," he said, "and how many units are operating for three years without failing. That's just as important as the ones I shipped last month that have been out there for only one month. It's not just total failures over total shipments that I'm interested in. The time to failure information is so much more informative to the design team."
Schenkelberg said it might actually be a good idea for people who plan to attend his workshop in the afternoon to spend their morning with Dr. Vadim Kozyrkov at his workshop. That workshop also will convey a double-edged message: that warranty statistics are very powerful when done correctly, but are very misleading when done incorrectly.
The Power of Numbers
Kozyrkov said he based the title for his workshop, "Lies, Damn Lies, and Warranty Statistics" on the quote variously attributed to Mark Twain, Benjamin Disraeli, and Winston Churchill: "There are three kinds of lies: lies, damned lies, and statistics."
Perhaps it's fitting that origin of one of the most famous quotes about falsification is itself in doubt. Whoever's quote it was [and as far as we can tell, Twain attributed it to Disraeli in his 1907 autobiography, but there's no date for Disraeli ever having actually said it], Kozyrkov said he wanted to invoke that association, so that people understand that statistics are very powerful tools, if used properly.
"Warranty statistics in particular," Kozyrkov said. "We all know they used to be neglected -- an afterthought and everything else. And recently, warranty has started to get the attention it deserves. But still, getting the attention is one thing. Having correct tools to act upon the information that a warranty system brings is a completely different thing. And I believe that industry is still quite a few years away from actually knowing how to utilize warranty statistics."
Before coming to the United States and joining Aculocity, Kozyrkov said he was a lecturer at the University of Port Elizabeth in South Africa for five years. Before that, he worked at the Moldova branch of the USSR's Academy of Science. He received a master's degree from Moscow State University and a Ph.D in physics from the University of Port Elizabeth.
Seven years ago, Aculocity was formed as the IT department of a company called the Grand Vehicle Works Holdings Corp., which at the time was a holding company that owned Workhorse Custom Chassis LLC, Uptime Parts LLC, Autocar LLC, and the Union City Body Company, among others. Kozyrkov and his programmers were basically providing the online parts sales and warranty claims management systems to GVW's companies, which were primarily in the automotive business, and more specifically, in trucks, trailers and recreational vehicles.
Then almost two years ago, in March 2006, Aculocity was spun off as an independent company, freeing it up to recruit additional customers, and enter additional industries. For instance, Kozyrkov said some of the latest new clients to sign up are Spirit Aero, which builds jetliner fuselages for Boeing, Shamash and Sons in the textile industry, and several clients that sell teaching tools and lab equipment to schools. But they still do plenty of work in the automotive industry. And GVW is still a client.
Besides the headquarters outside of Chicago, and smaller offices near Los Angeles and Indianapolis, Aculocity also has offices in Cape Town, South Africa and Chisinau, Moldova. Worldwide, it has around 20 full-time employees and perhaps another 15 working part time, Kozyrkov said, primarily students recruited from the University of Port Elizabeth where he used to study and teach.
Also Not Enough Time?
Kozyrkov said he's not sure if half a day is enough time to train people how to properly forecast their warranty costs, "but I'm willing to give it a try." In the future, he is hoping to help the statistics subcommittee of the Institute of Warranty Chain Management to put together an eight-hour course on the subject. So this workshop, he said, is something of a proving ground for that project.
"Even if I get it wrong, hopefully I will get it close enough that we can adjust and present a course at a later stage," he said. "But I like the challenge. I'll try to explain to people how to forecast it properly in three hours."
He readily concedes that he won't have enough time to comprehensively touch on all of these points in three hours. "However, my aim would be to create enough public interest that they'll start asking questions. Also, I want to show them some common pitfalls that they need to avoid. For instance, with early warning, I want to point out that they need to be very careful how they collect the data," he said. The data analysis tools are great, but the data itself isn't always in the best shape. So the results of the analysis may be somewhat misleading, because the data itself may be incomplete or inaccurate.
"Also, I want to discuss Weibull functions. If I ask people at the conference if they are using Weibull functions, everyone will probably say yes. I have no quarrel with Weibull. It's a great tool. But I suspect a lot of people use it because everyone else is using it," he said, and not because they decided that it's the best choice.
"I do realize that it's easy for me to see it, because I have a Ph.D and a master degree. So it's easy for me. The audience might not be ready. But my idea is to start with some simple mathematical games. Obviously, every game will have a purpose, because it will highlight some aspect of warranty statistics. But hopefully it will be lighthearted enough to explain serious stuff," he said.
Kozyrkov concedes that some of the largest manufacturers have both the tools and the staff in place to correctly process warranty data. But not all manufacturers can afford to employ Ph.D-holders on their staff or spend six figures on software licenses. "Those people don't look at warranty data properly," he said.
Different From Accounting
"It doesn't have to be sophisticated," he said. "But people need to realize that it has to be different from accounting. Accounting deals with a large number of linear and finite transactions. You issue an invoice and you receive money. If you don't receive money, you put it into a different account. So everything is linear and standard.
"In statistics you deal with probabilities. And that's what sets apart warranty statistics from everything else. And that's completely different from accounting. The moment you utilize a probability function in accounting, the IRS will become very interested. Warranty is different."
As does Schenkelberg, Kozyrkov acknowledges that one of the highest hurdles he faces is getting the financially-oriented warranty people to understand how the engineering-oriented people see warranty. And as did Schenkelberg, Kozyrkov said he's seen companies comparing warranty costs to the installed base, or worse, comparing current warranty costs to current production volume.
Hopefully, anyone who goes to either workshop will never do so again. And perhaps the finance people who go to both workshops will return to their companies with a new understanding of failure rates and warranty statistics, and how a heavy reliance on means and medians is not always the best idea.
It's not as if either Schenkelberg or Kozyrkov have anything against the use of simple ratios by the finance department. It's just that they want people to understand how such simple statistics can mislead their misuser. They're not saying that warranty analysis should be left to the experts. They're just saying that the simplest answer may not always be the best choice, and that even accurate statistics can be used to reach inaccurate conclusions.
"It could be all right, because financial people need to know their true costs," Kozyrkov said. And it's good to have the attention of a CFO, because they have the authority to change the rate at which money moves in or out of the warranty reserve. "Unfortunately, they still think that if they have a financial report that gives them simple ratios, they will have a proper answer for warranty. That's not the case."