- Email: [email protected]
The “trouble not identified” phenomenon in automotive electronics
Microelectronics Reliability 42 (2002) 641–651 www.elsevier.com/locate/microrel
The ‘‘trouble not identified’’ phenomenon in automotive electronics Dawn A. Thomas
a,1
, Ken Ayers b, Michael Pecht
c,*
a metaSENSORS, 358 Hungerford Drive, Rockville, MD 20850, USA Hancock, Rothert & Bunshoft LLP, 4 Embarcadero Center, San Francisco, CA 94111, USA CALCE Electronic Products and Systems Center, University of Maryland, College Park, MD 20742, USA b
c
Received 23 January 2002
Abstract In some cases, a failure occurs that cannot be verified, replicated at will, or attributed to a specific failure site, mode, and mechanism. Terms used to describe this phenomenon include trouble not identified (TNI), no trouble found (NTF), cannot duplicate (CND), ‘re-test ok’ (RTOK), no-fault found (NFF), and intermittent malfunctions. This paper discusses the concept, causes, and impact of the ‘‘trouble not identified’’ phenomenon on the automotive electronics industry. A case study is presented to clarify the issues. The key conclusion is that a manufacturer should assume that all field returns are field failures, unless some alternative reason can be verified. In fact, any company that produces a safety or emission regulated product should assume that every complaint or return of that product is a failure, and take on full responsibility for ascertaining the root cause. It must not be assumed that a returned module that passes tests associated with an engineering specification is good. 2002 Elsevier Science Ltd. All rights reserved.
1. What is trouble not identified? Product failure occurs when a product no longer performs its intended function in an application environment for the intended life of the product. Generally, the failure mode, root-cause failure mechanism, and failure site must be determined to eliminate product failure. The failure mode is the manifestation of a failure [1]. Examples of a failure mode include an electrical open or short and the change of an operating parameter outside a specification limit. Failure mechanisms are phenomena that either occur instantaneously or develop over time to induce failure, often due to thermal, mechanical, electromagnetic, or chemical loadings. The failure site is the location of the failure. In some cases, a failure occurs but cannot be verified, replicated at will, or attributed to a specific failure site, *
Corresponding author. Tel.: +1-301-314-9269. E-mail address: [email protected] (M. Pecht). URL: http://www.calce.umd.edu. 1 The work was performed when she was an MS student at University of Maryland.
mode, and mechanism [2]. Some of the terms used to describe this phenomenon include trouble not identified (TNI), no trouble found (NTF), cannot duplicate (CND), ‘re-test ok’ (RTOK), no-fault found (NFF), and intermittent malfunctions. TNIs pose a problem to almost everyone involved with the product, from customers to manufacturers and their suppliers. TNIs can range from being critical to the customer’s safety to being a mere nuisance. In most cases, customers are unaware of why the TNI occurred, and wonder if and when it will happen again, whether they imagined it, or if the TNI was just a ‘‘freak’’ event. Repair and/or service technicians often have limited information from the customer, and TNI failures thus can lead to the replacement of what may appear to be working products. If numerous products exhibit TNI problems, the manufacturer or supplier may receive an unfavorable reputation, and replacements can be costly to the manufacturer if the product is warranted. In this paper, the TNI phenomenon is presented in reference to a case study of a Ford ignition module (no longer used in new vehicles) and Ford’s automotive supply chain. The goal is to provide the reader with
0026-2714/02/$ - see front matter 2002 Elsevier Science Ltd. All rights reserved. PII: S 0 0 2 6 - 2 7 1 4 ( 0 2 ) 0 0 0 4 0 - 9
642
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
information to understand the concepts and the potential consequences of the TNI challenge and to make informed decisions when faced with products with warranty returns that exhibit the TNI phenomenon.
2. Case study background In the 1970s, Ford introduced various electronic ignition systems under the name Duraspark to replace the mechanical breaker point ignition system. However, the ‘‘Duraspark’’ system had reliability problems, numerous consumer complaints, 2 and was the focus of a classaction lawsuit and a subsequent recall [4]. In the 1982 model year, Ford began to replace the Duraspark system with a thick film 3 integrated (TFI) ignition module. 4 Ford used information generated by the Duraspark modules to project TFI module warranty return information [6]. The five year or 50 000 mile (5/ 50) warranty projections based on Duraspark were 10R/ 100, indicating that there would be 10 warranty returns per 100 modules (that is, 10%) before five years or 50 000 miles, whichever came first. In June 1986, Ford obtained 30 months of in-service TFI module data for California. The 5/50 TFI warranty projections indicated that returns exceeded previous projections fourfold. Actual data for the average model year 1984 car 5 projected warranty returns at 40R/100 [6], and as high as 99% for some car models [7]. This meant that, on average, 40% of the TFIs for the 1984 model year would ‘‘fail’’ within five years or 50 000 miles. This warranty return rate was suggestive of a major problem, but the problem was illusive in that many of the modules did not exhibit a failure mode when they were returned. The definition of TNI in this case describes warranty return and claims analysis wherein the cause of the customer concern cannot be identified [8]. That is, modules were removed from vehicles but they subsequently passed required engineering tests established to reflect the design intent [9]. A module marked TNI indicated that the cause of failure could not be identified initially and immediately. 2
In 1978, the National Highway Traffic Safety Administration (NHTSA) received over 50 reports of ignition system failure in 1975–1977 Ford vehicles [3]. This initiated an investigation that put focus on ignition module stalling. 3 ‘‘Thick film’’ denotes a ceramic substrate in which electronic circuitry is screen-printed. 4 Ford’s Engineering Specification for the TFI module stated that the module ‘‘must properly switch the ignition coil current off to produce a spark based on the input signal received’’, and that ‘‘the design life shall be equal to or greater than the life expectancy of the vehicle (100 000 miles minimum)’’ [5]. 5 Average car warranty projections are averaged over each model and engine type [6].
Some engineers suggested that the TFI modules had been replaced unnecessarily, because returned modules functioned properly when tested to engineering specifications. However, Ford identified failure mechanisms that could cause intermittent operation of the module 6 including substrate cracks, open bumps, excess leakage currents at high temperatures, connector–ramp mating problems, leadframes unsoldered from housing, shorted leadframes, open leadfoots, cracked solder, cracked resistors, cracked capacitors, and wire and wire bond lifts [10]. 7 The next sections of this paper give an overview of the challenges to uncovering the TNI phenomenon, such as how the vehicle repair dealer, the module manufacturer, and the vehicle manufacturer can address the problem. A summary and set of conclusions are then presented. 3. Problem identification at the dealer When customers (vehicle owners) believe their vehicles are not working properly, they generally take them to a dealer. A service writer greets the customer at the dealership and requests information about the nature of the problem. The customer typically leaves the dealership expecting to pick up the vehicle at some designated time. Technicians at the dealership generally try to assess and solve the problem quickly. Service writers are generally responsible for providing the necessary information about the customer’s vehicle complaints to the technicians. However, a study by Goldfarb Consultants 8 noted that . . . Technicians maintained that the service writers were not properly trained and, therefore, did not know what questions to ask of the customers 6 Failures were also found to include cracked capacitors, diode opens and shorts, Darlington transistor opens and shorts, opened flip-chip IC bumps, BeO separation due to poor attachment to the copper slug, leadframe opens and shorts, base and emitter die bond opens, mechanical damage, print voids, and cracked substrates. 7 One aspect of the intermittent problem was that when two components are soldered together and a crack develops in the solder connection, for whatever reason, and then the unit experiences expansion and contraction due to changes in temperatures, the connection may be closed at some times and open at other times. 8 Goldfarb Consultants is an international consulting firm that has been providing research services to Ford Motor Company for over 28 years. In March 1990, while Ford was still witnessing 80–90% TNIs of replaced TFIs, Goldfarb Consultants prepared ‘‘Perspectives on Intermittent Electrical Problems at the Dealership’’ for Ford. The purpose of the report was to investigate in detail the procedure at the dealership level for dealing with intermittent electrical problems and to determine what frustrations and/or obstacles inhibited the proper diagnosis and/or repair of an intermittent failure [11].
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
to pin down exactly what the problem was. As a result, many technicians reported having to frequently contact the customers themselves in the case of intermittent problems. This is time-consuming and ‘eats into’ the already limited diagnostic time allowed for in the shop manual [8]. Once a technician receives the service write-up, he or she must determine the state of non-operation of the system (e.g. failure to operate or intermittent operation), and the particular module that caused the problem [12]. Based on the problem, the technician will typically run through test procedures, often using specific diagnostic equipment, 9 manuals, previous training, and experience to locate the described problem. Upon completion, the technician documents what was faulty, which modules were replaced, and the number of hours required to perform the repair. If the module failed during its warranty period, the dealership usually sends the faulty module to a warranty return center for reimbursement. A survey of 18 Ford dealerships in 1987 noted that a top problem was intermittent ignition failure 10 and that more intermittent fault detection capability was needed, including self-test and diagnostic tools. Furthermore, 14 of the 18 dealerships indicated the TFI module was one of their top problems [14]. The following account appears in a Field Service Engineering Investigation Report from Ford’s Warranty Indicator System. The initial failure diagnosis was performed by a service technician and then repeated by a Ford field service engineer (FSE). The vehicle serviced was a Ford 5.0L van in which the TFI module was replaced. The customer described the problem as follows: ‘‘This van will stall after it gets hot and I’m in traffic. It also will not restart after I come out back from shopping’’ [15]. The FSE described the diagnosis of the problem:
9
Test equipment that could have proved helpful in identifying failures was often unavailable at dealerships. For example, Goldfarb Consultants noted: The monitor box [used to monitor the Ford vehicle while it is being driven] is a last resort due to the following: • there are few at any one dealership; • it is extremely difficult to hook-up due to the awkward location of the processor (GM’s Computerized Automotive Maintenance System, ‘‘CAMS,’’ by contrast, was seen as very easy to hook-up); and • number of technicians admitted to not being very comfortable with using the system due to lack of training [11]. 10 Ford considered a vehicle to be diagnosed with an intermittent fault when ‘‘the vehicle has been returned to the dealership with a fault that cannot be duplicated by the technician under normal driving conditions’’ [13].
643
Tech retold a detailed story of many attempted reports on this van. Originally, this superior tech did find ‘‘fuel related’’ codes in memory, but after performing the pin point tests for all codes {87, 95, and 96 continuous} he finally redirected his testing to the ignition system. He reported that when this van would not restart after hot soak, he would remove the starter solenoid ‘‘trigger’’ wire and the van would now restart. This was his only ‘‘hint’’ that the stalling concern was somehow ignition related in origin. Eventually, he hung on a ‘‘test remote TFI module’’ {As described in FoMoCo [Ford Motor Company] mag [magazine] ‘‘Service Life’’} and found that the stall/no restart concern would be resolved. FSE trusted the tech but wanted to reproduce the concern. FSE installed the returned TFI module on a 1987 Grd. Marq. [Grand Marquis]. Most of the concerns did indeed repeat; The engine would stall while in a queue, but would restart. The engine would sometimes start and stall after a short hot soak. When it would start, and the vehicle was driven, heavy detonation could be heard from the first acceleration. The most interesting discovery was the lack of ‘‘feedback’’ from all the experienced failure modes; Never did FSE receive any non-pass self test codes in KOEO [key on/engine off] or KOER [key on/engine running]. Furthermore, FSE was testing with the EEC system in ‘‘continuous monitor mode’’ and the ignition system having the ‘‘intermittent ignition monitor’’ in series with the TFI module. Neither of these tests revealed any problem during or after the different failure modes: The ‘‘beeper’’ of tester in ‘‘continuous monitor mode’’ never beeped and none of the intermittent ignition monitor LEDs ever blinked. FSE wondered just how would a tech know that this TFI module was defective. While visiting another dealership, a tech suggested that the TFI module be tapped at idle. Upon doing so, the engine stalled. This was repeated while in continuous monitor mode; The engine again stalled, but the beeper did not beep [15]. From the report one can see the TFI had an intermittent problem that was elusive to Ford’s experienced field engineers, as well as to the service technicians [16,17]. Sometimes the problem occurred and sometimes it did not; sometimes it could be detected and sometimes it could not. It is clear that the problem was extremely difficult to detect and that the diagnostic equipment was not always helpful. 3.1. The module swapping problem In some cases, a technician bypasses failure analysis and starts service with module swapping, using
644
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
experience to determine what needs to be replaced or replacing several modules at one time to fix the problem [9,18]. Ford reimbursed dealerships for replacing modules under warranty, but they did not cover labor costs. Thus, by module swapping, dealers could get a substantial markup on all modules they replaced, and the technicians could perform repairs faster, so more vehicles could be serviced [18,19]. The result of swapping is that TNIs can be generated. Goldfarb noted:
A–B–A, 11 which was not published in the diagnostics. 12 A–B–A testing reduces the possibility that the failure is a system, sub-system or other module problem, due to a loose connection, or connector, connector contamination, or connector corrosion. An example of A–B–A validation is summarized from the Ford 1992 Common Quality Information Service (CQIS) detail report. In this example the vehicle owner brought the vehicle in because it would stall intermittently when hot. The technician’s diagnosis is described:
. . . the Company [Ford] is seen as compensating more realistically for a part change versus proper diagnosis. As a result, the time allotted for a part replacement versus diagnosis could be a major factor inhibiting the correction of intermittent electrical problems [11].
Intermittently vehicle would stall on take off. EEC tested no codes. Checked fuel pressure when stalled and pressure held pressure. Hooked up ignition tester and problem was gone. Remove tester and problem came back. Then installed monitor and found loss of PIP signal. Replace TFI. Then tested OK. ABA old TFI and reconfirmed [8].
3.2. Assessing the TNI phenomenon data by additional tests and analysis
As noted in this diagnosis, A–B–A was valuable because the ignition tester failed to be useful. Yet, the technician could be confident that his diagnosis was accurate, since the vehicle functioned properly with the new module and still displayed symptoms when the old module was connected. The A–B–A technique showed that the TFI was the problem.
As TNI rates rose, Ford initiated a diagnostic data management system (DDMS) to help the product engineering community understand the causal factors that drove repairs, replacements, and service [8,20–22]. Sixteen of Ford’s best and larger dealerships were chosen to participate in the program to be monitored by Ford’s local FSEs, who could oversee diagnosis. The goals were to generate sufficient reports to be statistically significant and to have experienced technicians help make an accurate diagnosis. In the DDMS, modules returned under warranty were subjected to a parametric three-temperature test. If the module passed the initial test, it would be labeled TNI in the DDMS database. In some cases, additional testing of TNI modules was conducted, including a three-temperature test and thermal cycling. Failure of a module at this stage resulted in a note stating that the module failed the lab test, although the TNI classification was not changed. Pie charts for 1986–1989 model year vehicle DDMS data are shown in Figs. 1–4. The data include results for TNI modules subjected to additional lab tests. The charts show the various defects that were observed based on visual and three-temperature tests alone. They also show the number and percentage of defects that were not verified from these tests. The charts then show the results of ‘‘selected lab tests’’ (for example, thermal cycling and extended range threetemperature tests) in which defects were precipitated and the module failed. Unfortunately, this type of indepth analysis was not conducted on all the returned modules. Dealer participation in the DDMS program also required the technicians to use a procedure called
4. Problem identification by the module manufacturer Modules that are under warranty are binned at the warranty return center according to the module manufacturer’s name. The cause of failure, the failure mechanism, and the failure mode should be determined at this ‘‘plant’’ investigation level [12]. If the module manufacturer finds that the problem is with a component of 11 A–B–A testing is performed to assess whether the module, as opposed to the system, a connected sub-system, or some connection or connector, is causing the problem. The technician records the symptoms and the test results associated with the original module (module ‘‘A’’). Module A is then replaced with a known good module (module ‘‘B’’) and the vehicle is tested with module B in place. Finally module B is replaced by the original module A and vehicle testing is performed once again. If the vehicle symptoms occur only when module A is in place, module A is the candidate problem. If module B also fails or if module A functions properly after it is replaced, then there may be another reason for the vehicle problem. 12 The A–B–A was recommended for intelligent diagnosis with the predecessor Duraspark ignition module. Section 13F, Intermittent Diagnosis, page 13F, 29-02-2 instructed the technician to subject the module to a heat pump and tap the module to replicate the problem. ‘‘If this procedure results in ignition malfunction, substitute a known good module. If the malfunction is corrected by substitution, validate that the original module is at fault by reconnecting it to the vehicle.’’
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
Fig. 1. 1986 Model year Lansdale TFI DMS analysis (126 modules analyzed) (data obtained from [40]).
Fig. 2. 1987 Model year Lansdale TFI DMS analysis (237 modules analyzed) (data obtained from [40]).
645
646
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
Fig. 3. 1988 Model year Lansdale TFI DDMS analysis (70 modules analyzed) (data obtained from [40]).
Fig. 4. 1989 Model year Lansdale TFI DDMS analysis (75 modules analyzed) (data obtained from [40]).
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
the module, that component may be sent to the part manufacturer for analysis. 13 The testing conducted upon receiving the TFI module consisted of a bench test, which included a visual inspection and parametric three-temperature testing. 14 Visual inspection was hard because microscopic intermittent failures can be difficult, if not impossible, to detect, especially if the location of the failure is unknown. For example, ‘‘ceramic substrates can develop microcracks in the thick film substrate [of the TFI] which can cause intermittent problems. This intermittence had a major impact on the TNI percentage, possibly 20 percent’’ [23]. Ford North Penn Electronics Facility found that a module could pass the tests but fail in the field. A tester did not exist that could capture in real time every failure mode of the modules. It was also noted that parametric three-temperature testing ‘‘will not capture intermittents that could be caused by vibration and shock. It does not provide real-time monitoring of cracks as they develop’’ [23]. Further, validation tests were not correlated well with the field applications – packages passed lab tests but failed in proving grounds [24]. 15 In addition, tests conducted at Ford’s Lansdale and Motorola Automotive and Industrial Electronics Group (both TFI manufacturers [26]) yielded different results. For example, in a 1982 Ford memorandum [27] regarding 1865 units tested at Motorola and 1278 units at Lansdale, the following was observed:
647
mittent at Motorola’s 3-sec spark but subsequently passed Motorola’s bench test and Lansdale’s retest. Lansdale testing There were 44 NTFs [no trouble found] caused by test equipment malfunction. Finally, duplicating TNI problems was time consuming. A TFI failure analysis, in which nine days were required to recreate the failure, is described in a 1982 Ford memorandum [28], which states: During the course of PV [product validation] testing failures occurred which appeared to be related to thermal overstress. An extensive engineering study was conducted in an attempt to identify the root cause of this overstress. This effort included a PEO [Product Engineering Office] examination of FACC [Ford Aerospace and Communication Corporation] tester facilities from 4/5 to 4/9 [28]. Since no direct cause of the failures could be established a second PV sample was begun which included ICs from a second wafer lot. By 4/13 eight units failed catastrophically while under 140 C [284 F], 18v stress testing. . . . Armed with this FACC input PEO was able to duplicate conditions in the laboratory to recreate FACC’s observations. 5. Problem identification by the vehicle manufacturer
Motorola testing There were 89 NTFs [no trouble found] caused by testing difference for the stall mode shutdown test; relaxed deviated limits for fall time; minor output oscillations that pass Lansdale test criteria but fail Motorola testing; one unit that ‘‘sounded’’ inter-
13
When the part manufacturer receives the modules, it may also conduct some defect analysis, such as on an IC. If the IC functions during the IC test and is determined to have no physical defects, the IC manufacturer assumes that module failure is not caused by the IC, although the interconnection to the IC may have been the cause. 14 Parametric three-temperature testing was conducted to identify the condition of the module at three points in time during a hot, cold, and ambient temperature test. As a part of the DDMS program, modules were not thermal cycled until they passed the bench test and were already labeled TNI. The reader is referred to the charts of Figs. 1–4, which show that the additional test analysis did precipitate defective modules. 15 As an example, a Ford memorandum indicated that the ‘‘test equipment did not adequately measure the product performance’’ [25]. Thus, any correlation would be near impossible.
The vehicle manufacturer is ultimately responsible for the warranty returns and the TNI problem. The vehicle manufacturer collects warranty information from the dealers, determines the root cause of the returns, and answers to the customers and the regulatory agencies. With increased use of electronic components for driveability systems, a reliance on mere warranty data did not provide sufficient information for identifying driveability problems [18]. The automotive industry had been aware of the warranty return problems and the TNI phenomenon ever since they began introducing electronics into vehicles. One of the methods useful in tracking down TNI phenomena is the fishbone, or Isikawa, diagram. The success of this method depends on being able to identify as many of the potential failure modes and mechanisms as possible. Fig. 5 shows Ford’s fishbone diagram of some potential TFI module failure modes and mechanisms. To uncover the cause of TNIs, it is also helpful to determine all the potential load (stress) conditions that can arise throughout the lifecycle of the product. In some cases, links can be established to uncover a common cause. For example, one common link in the TFI
648
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
Fig. 5. Ford’s fishbone diagram of TFI module failure modes and mechanisms (reproduced from [41]). The * denotes a 50 000 mile concern, suggesting that Ford considered the failure mode or mechanism to be an issue for 5/50 warranty coverage.
failures was the severe underhood temperature environment to which the ignition module was exposed. High temperature and high temperature thermo-cycles were causal factors for TFI ignition module failures [29], and the TFI module warranty returns were highest on high temperature applications in warm weather areas [30]. Ford documented some of the reliability implications of high temperature [31] and thermal cycling: High temperature has several negative effects on the life of electronics. At high temperatures, many materials, including solder, weaken. Intermetallic reactions (chemical in nature) accelerate, sometimes forming compounds that reduce component life. A well-documented rule of thumb is that mean time to failure halves for each 10 C (18 F) rise in operating temperature. Thermal cycling causes failures by creating mechanical stress buildup in material systems due to differences in rates of expansion. Material fatigue and joint failures are the result [7]. Fig. 6 shows how temperature can affect the estimated 5/50 return rate. This kind of information can be used to suggest that field returns, even those listed as TNI, could have temperature-related failure mechanisms.
Vehicle testing performed by Ford in 1985 suggested that the TFI module was being exposed to temperatures higher than the module’s engineering specification temperature [32]. 16 In fact, Ford reported temperatures higher than the designed functional capability of the TFI module [30], and in one study, Ford found that of 16 1982–1986 model year vehicles tested, 12 had a TFI base plate temperature at or above the specified operating limit [34]. Furthermore, a Major/Major Risk 17 Assess-
16
The Society for Automotive Engineers (SAE) documents a temperature extreme of 150 C (300 F) for the underhoodengine location [33], with a fishbone diagram listing several potential failures due to temperature cycling and temperature extremes. 17 Ford defines ‘‘major risks’’ in two categories, major/minor and major/major. A major/minor risk applies to items that have failed to meet design or process standards or requirements, or reliability objectives, by the assessment date, but with corrective action, are projected to be a minor risk by job #1 (i.e. the production date). A major/major risk applies to items that have failed to meet design or process standards and requirements, or reliability objectives, and cannot be reduced to minor risk in time for job #1 or component production incorporation date [35].
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
649
Fig. 6. 1988 Estimated R/1000 versus TFI module temperature (reproduced from [34]).
ment of several 1985–1986 vehicle lines also indicated that temperatures had exceeded the TFI specification limit: Testing conducted at the Wind Tunnel (110 F ambient) [43.3 C] and/or Arizona Proving Ground identified temperatures, which exceeded the module design specification. The identified temperature could cause a thermal run away condition on the output transistor causing the vehicle to stall and not restart [36]. By 1986, ‘‘improvement proposals’’ were ‘‘planned’’ or ‘‘under investigation’’ to address the temperature issues and reduce TNIs. Potential solutions included: • Characterize IC at 140 C [284 F] vs. 125 C [257 F]. • Test IC at 140 C [284 F] vs. 125 C [257 F]. • Remote mount application 87 1/2 (3.8L Taurus) • Reduce underhood temperature. • Revise specification [37]. The manner in which Ford conducted warranty analysis was conducive to generating a large number of parts that tested TNI. First, not all the modules were tested, and those not tested were not considered warranty failures. Second, the tests were based on Ford engineering specifications, rather than on extensive failure analysis. Third, Ford Parts & Service Division (FPSD) had a policy whereby the manufacturer is not supposed to test returned parts to any greater extent
than the dealers are capable of if the manufacturer intends to chargeback those parts [38]. 18
6. Summary and conclusions: treating field returns as field failures It is good engineering practice to treat every field return (whether under warranty or not) as a field failure until the actual root cause of the return is known. In the auto industry, where the customer brings the vehicle to the dealer as a result of a problem, the dealer must assume that the customer did indeed have a problem. The dealer’s responsibility is to record the customer’s concerns and investigate the problem until a solution can be established. This investigation includes diagnostic monitoring, troubleshooting and A–B–A testing. All modules should be sent back to the manufacturer for root cause analysis even if the module is out of warranty. The module and the vehicle manufacturer
18
Ford chose not to chargeback dealers for returned ignition modules that were labeled TNI based on meeting the engineering specification. The reasoning behind that decision was, ‘‘due to the complexity of the electronics in the ignition module, there is no efficient way of ensuring with 100% confidence that a part is truly ‘‘TNI’’ [38].’’ In fact, in order to have charged back the dealers for TFIs labeled TNI, they would have had to show that their diagnosis of TNI was ‘‘bulletproof’’, which would be impossible to demonstrate, by definition.
650
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
have the responsibility to conduct both tests and teardowns of every module to determine the root cause of the problem or to determine why a ‘‘good’’ module has been returned under warranty (i.e. to rule out fraud by the dealer). Module testing must simulate actual and worst-case field conditions and will thus often require combining multiple and time varying load (stress) conditions for extended periods of time. A sequence and/or a combination of environmental and operating conditions can often trigger a fault; this is common in computer operation, when a unique and often complex sequence of keystrokes and operating conditions can initiate an unpredictable intermittent fault. Tear-downs should be conducted after non-destructive failure analysis and testing and must involve investigating every potential failure mode and failure site to assess degradation and damage to the module. If no defect can be determined and there is no other verifiable and auditable explanation, then the module must still be considered a field failure, due to the elusive and intermittent nature of electrical failures. A manufacturer should assume that all field returns are field failures, unless some alternative reason can be verified. In fact, any company that produces a safety or emissions regulated product should assume that every complaint or return of that product is a failure, and take on full responsibility for ascertaining the root cause. It must not be assumed that a returned module that passes tests associated with an engineering specification is good. The TNIs associated with Ford’s TFI involved several manufacturing, electrical, and thermal–mechanical factors that contributed to the existence of the failures, as opposed to the existence of a single Achilles’ heel. These factors included insufficient part analysis, a lack of systematic data accumulation, and a lack of understanding of how the component reacted within its system and environment [9]. Supina [18] wrote that ‘‘a large part of the problem of TNIs is what we do not know. With most failure modes, there is a clear direction to move in: something to attack to solve the problem. TNIs are not that clear cut’’ and ‘‘It is pretty safe to say that there is not one big, unknown problem out there accounting for our 90% TNI rate.’’ It is therefore critical to have a cause-and-effects (fishbone) diagram that has all possible failure modes and mechanisms, but at the same time be open to the discovery of something new. Furthermore, the necessary resources and best practices must be applied to solve the problem. A company should never categorize module warranty returns as TNI as a means to ignore, mitigate, or transfer problems. In 1987, a limited recall of TFI modules was conducted, consisting of 1.1 million 1984 and 1985 1.5L and 2.3L 4-cylinder engine vehicles [39]. New parts had to be manufactured for each replacement, and millions of dollars were spent on the recall. In 1997, a class action
suit was initiated. The case ultimately ended in a mistrial in November 1999 when, after approximately six months of testimony, the jury was unable to reach a decision. However, the trial continued in a second phase before the trial judge, which resulted in the first judicially mandated recall in US history. In a Statement of Decision issued in August 2000, the judge found that Ford had intentionally concealed information about the TFI module defect from millions of consumers, as well as NHTSA and other government agencies. Ford vowed to appeal, but ultimately agreed to settle the litigation on a nationwide basis before the damages issues were retried before another jury. The settlement required Ford to extend the TFI module’s warranty in each vehicle to 100 000 miles and to provide full reimbursement of all module replacement costs incurred before the odometer reached 100 000 miles. In addition, Ford agreed to contribute US $5 million for independent colleges and universities to conduct automotive safety research.
References [1] Lau JH, Pao Y. Solder joint reliability of BGA, CSP, flip chip, and fine pitch SMT assemblies. New York: McGrawHill; 1997. [2] Pecht M, Ramappan V. Are components still the major problem: a review of electronic system and device field failure returns. IEEE Trans Components Hybrids Manufact Technol 1992;15(6):1160–4. [3] Ford Motor Company (11). Lynn L. Bradford, Director Office of Defects Investigation Enforcement. 12 October 1978. [4] Ford Motor Company (10). Owner notification programs M04 and M05. 11 February 1981. [5] Ford Motor Company (1). Ford engineering specification for part number ES-E3EF-12A297-AA. 1986. [6] Ford Motor Company (5). TFI-IV thick film ignition module 5/50 warranty review. 5 November 1986. [7] Ford Motor Company (2). 1986 Ignition components warranty review. Ignition Systems EPME/EED, 20 March 1986. [8] Ford Motor Company (16). Powertrain quality plan. 27 August 1986. [9] Ford Motor Company (7). Electronic systems quality trouble-not-identified with field return components. Electrical & Electronics Division, 21 October 1986. [10] Russell R. ETS high mileage testing. Memorandum, 17 May 1989. [11] Goldfarb Consultants. Perspectives on intermittent electrical problems at the dealership. Ford Motor Company; 1990. March. [12] Ford Motor Company (9). Reliability engineering classification tree. 27 July 1994. [13] Ford Motor Company (12). Driveability intermittents. 1986. [14] Wikarski K, Morosky E, Tabbert E. California MIL pilot program and dealer visitation. 13 July 1987.
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651 [15] Ford Motor Company (8). Warranty indicator system: field service engineering investigation report. 18 May 1989. [16] Ford Motor Company (17). Perspectives on intermittent electrical problems at the dealership. San Francisco Area Focus Group, 21–22 February 1990. [17] Ford Motor Company (19). Vehicle electrical system: quality improvements & diagnosis of intermittents. Revised 22 November 1989. [18] Supina JG. TNIs. Inter-office memorandum, Electronics Division, Product Engineering Office, Ford Motor Company. 15 March 1991. [19] Ford Motor Company (18). TFI ignition MOL: 1991 MY 30% improvement shortfall. TFI4-5785. [20] Ford Motor Company (4). Lansdale TFI warranty analyses. 1988–1991. [21] Ford Motor Company (13). DDMS module analysis (TFIIV). TFI Product Quality Team, 20 April 1988. [22] Ford Motor Company (23). DDMS procedure. 5 September 1990. [23] Balok B. Requalification of TNI TFI-IV and EDIS modules. Inter Office Memorandum, Ford Motor Company, 3 September 1993. [24] Hu JM. Reliability validation of electronic packages. Presentation, Ford Motor Company, July 1996. [25] Davis DA. 1983 TFI-I push start test equipment issues. Inter-office memorandum, Electrical & Electronics Division, Product Engineering Office, Ford Motor Company, 30 April 1982. [26] Ford Motor Company (20). Ford and Motorola CS291 returns tracking log. 1990. [27] Simms H. T.F.I. 3 second spark test evaluation results. Inter-office memorandum, Ford Aerospace & Communications Corporation, Ford Motor Company, 6 January 1982. [28] Leszczynski SA. TFI-IV IC high voltage, high temperature concern. Handwritten letter, Ford Motor Company, 13 April 1982.
651
[29] Ford Motor Company (22). Case study: 1984/85 distributor ignition hall effect PIP sensor, analysis of field problem. TFI4-1364-1396, Ignition Components Department, Engine Engineering Office, revised 16 August 1985. [30] Ford Motor Company (21). Product problem and action status report: ignition components. TFI4-1395, 17 July 1985. [31] Jurosek ML, Macher FE. Underhood temperatures. Memorandum, Ford Electrical and Electronics Division, 31 July 1986. [32] Ford Motor Company (3). 1982/86 Distributor temperature tests EEC-IV vehicles—wind tunnel/APG testing. Ignition System EED, revised 1 and 14 August 1985. [33] SAE. Rev. Recommended environmental practices for electronic equipment design. SAE J1211, November 1978. [34] Ford Motor Company (15). 4.9/5.0/5.8L Remote mount TFI funding. Meeting announcement/agenda, 9 May 1989. [35] Ford Motor Company (6). Product engineering procedure no. 3-133. Vehicle program assessments, Attachment A, 19 March 1982. [36] Rivard JG. Major/major risk assessment on TFI-IV ignition module. Inter-office memorandum, Electrical & Electronics Division, Product Engineering Office, Ford Motor Company, 23 July 1985. [37] Gentilcore JA. Failure mode analysis: Part: Ign. module TFI-IV. 18 January 1986. [38] Lombardi J. Ignition System TNI Reduction Team 8D worksheet. Concern title: ignition system TNI reduction. 1 August 1990. [39] Center for Auto Safety. Center for Auto Safety action alerts. 1997. Available from: www.essential.org/orgs/cas/ action1.html. [40] Ford Motor Company (14). DDMS TFI warranty repair database. Model year 1986–1991. [41] Gentilcore JA. TFI ignition module failure mode Ishikawa diagram, 1985.
The ‘‘trouble not identified’’ phenomenon in automotive electronics Dawn A. Thomas
a,1
, Ken Ayers b, Michael Pecht
c,*
a metaSENSORS, 358 Hungerford Drive, Rockville, MD 20850, USA Hancock, Rothert & Bunshoft LLP, 4 Embarcadero Center, San Francisco, CA 94111, USA CALCE Electronic Products and Systems Center, University of Maryland, College Park, MD 20742, USA b
c
Received 23 January 2002
Abstract In some cases, a failure occurs that cannot be verified, replicated at will, or attributed to a specific failure site, mode, and mechanism. Terms used to describe this phenomenon include trouble not identified (TNI), no trouble found (NTF), cannot duplicate (CND), ‘re-test ok’ (RTOK), no-fault found (NFF), and intermittent malfunctions. This paper discusses the concept, causes, and impact of the ‘‘trouble not identified’’ phenomenon on the automotive electronics industry. A case study is presented to clarify the issues. The key conclusion is that a manufacturer should assume that all field returns are field failures, unless some alternative reason can be verified. In fact, any company that produces a safety or emission regulated product should assume that every complaint or return of that product is a failure, and take on full responsibility for ascertaining the root cause. It must not be assumed that a returned module that passes tests associated with an engineering specification is good. 2002 Elsevier Science Ltd. All rights reserved.
1. What is trouble not identified? Product failure occurs when a product no longer performs its intended function in an application environment for the intended life of the product. Generally, the failure mode, root-cause failure mechanism, and failure site must be determined to eliminate product failure. The failure mode is the manifestation of a failure [1]. Examples of a failure mode include an electrical open or short and the change of an operating parameter outside a specification limit. Failure mechanisms are phenomena that either occur instantaneously or develop over time to induce failure, often due to thermal, mechanical, electromagnetic, or chemical loadings. The failure site is the location of the failure. In some cases, a failure occurs but cannot be verified, replicated at will, or attributed to a specific failure site, *
Corresponding author. Tel.: +1-301-314-9269. E-mail address: [email protected] (M. Pecht). URL: http://www.calce.umd.edu. 1 The work was performed when she was an MS student at University of Maryland.
mode, and mechanism [2]. Some of the terms used to describe this phenomenon include trouble not identified (TNI), no trouble found (NTF), cannot duplicate (CND), ‘re-test ok’ (RTOK), no-fault found (NFF), and intermittent malfunctions. TNIs pose a problem to almost everyone involved with the product, from customers to manufacturers and their suppliers. TNIs can range from being critical to the customer’s safety to being a mere nuisance. In most cases, customers are unaware of why the TNI occurred, and wonder if and when it will happen again, whether they imagined it, or if the TNI was just a ‘‘freak’’ event. Repair and/or service technicians often have limited information from the customer, and TNI failures thus can lead to the replacement of what may appear to be working products. If numerous products exhibit TNI problems, the manufacturer or supplier may receive an unfavorable reputation, and replacements can be costly to the manufacturer if the product is warranted. In this paper, the TNI phenomenon is presented in reference to a case study of a Ford ignition module (no longer used in new vehicles) and Ford’s automotive supply chain. The goal is to provide the reader with
0026-2714/02/$ - see front matter 2002 Elsevier Science Ltd. All rights reserved. PII: S 0 0 2 6 - 2 7 1 4 ( 0 2 ) 0 0 0 4 0 - 9
642
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
information to understand the concepts and the potential consequences of the TNI challenge and to make informed decisions when faced with products with warranty returns that exhibit the TNI phenomenon.
2. Case study background In the 1970s, Ford introduced various electronic ignition systems under the name Duraspark to replace the mechanical breaker point ignition system. However, the ‘‘Duraspark’’ system had reliability problems, numerous consumer complaints, 2 and was the focus of a classaction lawsuit and a subsequent recall [4]. In the 1982 model year, Ford began to replace the Duraspark system with a thick film 3 integrated (TFI) ignition module. 4 Ford used information generated by the Duraspark modules to project TFI module warranty return information [6]. The five year or 50 000 mile (5/ 50) warranty projections based on Duraspark were 10R/ 100, indicating that there would be 10 warranty returns per 100 modules (that is, 10%) before five years or 50 000 miles, whichever came first. In June 1986, Ford obtained 30 months of in-service TFI module data for California. The 5/50 TFI warranty projections indicated that returns exceeded previous projections fourfold. Actual data for the average model year 1984 car 5 projected warranty returns at 40R/100 [6], and as high as 99% for some car models [7]. This meant that, on average, 40% of the TFIs for the 1984 model year would ‘‘fail’’ within five years or 50 000 miles. This warranty return rate was suggestive of a major problem, but the problem was illusive in that many of the modules did not exhibit a failure mode when they were returned. The definition of TNI in this case describes warranty return and claims analysis wherein the cause of the customer concern cannot be identified [8]. That is, modules were removed from vehicles but they subsequently passed required engineering tests established to reflect the design intent [9]. A module marked TNI indicated that the cause of failure could not be identified initially and immediately. 2
In 1978, the National Highway Traffic Safety Administration (NHTSA) received over 50 reports of ignition system failure in 1975–1977 Ford vehicles [3]. This initiated an investigation that put focus on ignition module stalling. 3 ‘‘Thick film’’ denotes a ceramic substrate in which electronic circuitry is screen-printed. 4 Ford’s Engineering Specification for the TFI module stated that the module ‘‘must properly switch the ignition coil current off to produce a spark based on the input signal received’’, and that ‘‘the design life shall be equal to or greater than the life expectancy of the vehicle (100 000 miles minimum)’’ [5]. 5 Average car warranty projections are averaged over each model and engine type [6].
Some engineers suggested that the TFI modules had been replaced unnecessarily, because returned modules functioned properly when tested to engineering specifications. However, Ford identified failure mechanisms that could cause intermittent operation of the module 6 including substrate cracks, open bumps, excess leakage currents at high temperatures, connector–ramp mating problems, leadframes unsoldered from housing, shorted leadframes, open leadfoots, cracked solder, cracked resistors, cracked capacitors, and wire and wire bond lifts [10]. 7 The next sections of this paper give an overview of the challenges to uncovering the TNI phenomenon, such as how the vehicle repair dealer, the module manufacturer, and the vehicle manufacturer can address the problem. A summary and set of conclusions are then presented. 3. Problem identification at the dealer When customers (vehicle owners) believe their vehicles are not working properly, they generally take them to a dealer. A service writer greets the customer at the dealership and requests information about the nature of the problem. The customer typically leaves the dealership expecting to pick up the vehicle at some designated time. Technicians at the dealership generally try to assess and solve the problem quickly. Service writers are generally responsible for providing the necessary information about the customer’s vehicle complaints to the technicians. However, a study by Goldfarb Consultants 8 noted that . . . Technicians maintained that the service writers were not properly trained and, therefore, did not know what questions to ask of the customers 6 Failures were also found to include cracked capacitors, diode opens and shorts, Darlington transistor opens and shorts, opened flip-chip IC bumps, BeO separation due to poor attachment to the copper slug, leadframe opens and shorts, base and emitter die bond opens, mechanical damage, print voids, and cracked substrates. 7 One aspect of the intermittent problem was that when two components are soldered together and a crack develops in the solder connection, for whatever reason, and then the unit experiences expansion and contraction due to changes in temperatures, the connection may be closed at some times and open at other times. 8 Goldfarb Consultants is an international consulting firm that has been providing research services to Ford Motor Company for over 28 years. In March 1990, while Ford was still witnessing 80–90% TNIs of replaced TFIs, Goldfarb Consultants prepared ‘‘Perspectives on Intermittent Electrical Problems at the Dealership’’ for Ford. The purpose of the report was to investigate in detail the procedure at the dealership level for dealing with intermittent electrical problems and to determine what frustrations and/or obstacles inhibited the proper diagnosis and/or repair of an intermittent failure [11].
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
to pin down exactly what the problem was. As a result, many technicians reported having to frequently contact the customers themselves in the case of intermittent problems. This is time-consuming and ‘eats into’ the already limited diagnostic time allowed for in the shop manual [8]. Once a technician receives the service write-up, he or she must determine the state of non-operation of the system (e.g. failure to operate or intermittent operation), and the particular module that caused the problem [12]. Based on the problem, the technician will typically run through test procedures, often using specific diagnostic equipment, 9 manuals, previous training, and experience to locate the described problem. Upon completion, the technician documents what was faulty, which modules were replaced, and the number of hours required to perform the repair. If the module failed during its warranty period, the dealership usually sends the faulty module to a warranty return center for reimbursement. A survey of 18 Ford dealerships in 1987 noted that a top problem was intermittent ignition failure 10 and that more intermittent fault detection capability was needed, including self-test and diagnostic tools. Furthermore, 14 of the 18 dealerships indicated the TFI module was one of their top problems [14]. The following account appears in a Field Service Engineering Investigation Report from Ford’s Warranty Indicator System. The initial failure diagnosis was performed by a service technician and then repeated by a Ford field service engineer (FSE). The vehicle serviced was a Ford 5.0L van in which the TFI module was replaced. The customer described the problem as follows: ‘‘This van will stall after it gets hot and I’m in traffic. It also will not restart after I come out back from shopping’’ [15]. The FSE described the diagnosis of the problem:
9
Test equipment that could have proved helpful in identifying failures was often unavailable at dealerships. For example, Goldfarb Consultants noted: The monitor box [used to monitor the Ford vehicle while it is being driven] is a last resort due to the following: • there are few at any one dealership; • it is extremely difficult to hook-up due to the awkward location of the processor (GM’s Computerized Automotive Maintenance System, ‘‘CAMS,’’ by contrast, was seen as very easy to hook-up); and • number of technicians admitted to not being very comfortable with using the system due to lack of training [11]. 10 Ford considered a vehicle to be diagnosed with an intermittent fault when ‘‘the vehicle has been returned to the dealership with a fault that cannot be duplicated by the technician under normal driving conditions’’ [13].
643
Tech retold a detailed story of many attempted reports on this van. Originally, this superior tech did find ‘‘fuel related’’ codes in memory, but after performing the pin point tests for all codes {87, 95, and 96 continuous} he finally redirected his testing to the ignition system. He reported that when this van would not restart after hot soak, he would remove the starter solenoid ‘‘trigger’’ wire and the van would now restart. This was his only ‘‘hint’’ that the stalling concern was somehow ignition related in origin. Eventually, he hung on a ‘‘test remote TFI module’’ {As described in FoMoCo [Ford Motor Company] mag [magazine] ‘‘Service Life’’} and found that the stall/no restart concern would be resolved. FSE trusted the tech but wanted to reproduce the concern. FSE installed the returned TFI module on a 1987 Grd. Marq. [Grand Marquis]. Most of the concerns did indeed repeat; The engine would stall while in a queue, but would restart. The engine would sometimes start and stall after a short hot soak. When it would start, and the vehicle was driven, heavy detonation could be heard from the first acceleration. The most interesting discovery was the lack of ‘‘feedback’’ from all the experienced failure modes; Never did FSE receive any non-pass self test codes in KOEO [key on/engine off] or KOER [key on/engine running]. Furthermore, FSE was testing with the EEC system in ‘‘continuous monitor mode’’ and the ignition system having the ‘‘intermittent ignition monitor’’ in series with the TFI module. Neither of these tests revealed any problem during or after the different failure modes: The ‘‘beeper’’ of tester in ‘‘continuous monitor mode’’ never beeped and none of the intermittent ignition monitor LEDs ever blinked. FSE wondered just how would a tech know that this TFI module was defective. While visiting another dealership, a tech suggested that the TFI module be tapped at idle. Upon doing so, the engine stalled. This was repeated while in continuous monitor mode; The engine again stalled, but the beeper did not beep [15]. From the report one can see the TFI had an intermittent problem that was elusive to Ford’s experienced field engineers, as well as to the service technicians [16,17]. Sometimes the problem occurred and sometimes it did not; sometimes it could be detected and sometimes it could not. It is clear that the problem was extremely difficult to detect and that the diagnostic equipment was not always helpful. 3.1. The module swapping problem In some cases, a technician bypasses failure analysis and starts service with module swapping, using
644
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
experience to determine what needs to be replaced or replacing several modules at one time to fix the problem [9,18]. Ford reimbursed dealerships for replacing modules under warranty, but they did not cover labor costs. Thus, by module swapping, dealers could get a substantial markup on all modules they replaced, and the technicians could perform repairs faster, so more vehicles could be serviced [18,19]. The result of swapping is that TNIs can be generated. Goldfarb noted:
A–B–A, 11 which was not published in the diagnostics. 12 A–B–A testing reduces the possibility that the failure is a system, sub-system or other module problem, due to a loose connection, or connector, connector contamination, or connector corrosion. An example of A–B–A validation is summarized from the Ford 1992 Common Quality Information Service (CQIS) detail report. In this example the vehicle owner brought the vehicle in because it would stall intermittently when hot. The technician’s diagnosis is described:
. . . the Company [Ford] is seen as compensating more realistically for a part change versus proper diagnosis. As a result, the time allotted for a part replacement versus diagnosis could be a major factor inhibiting the correction of intermittent electrical problems [11].
Intermittently vehicle would stall on take off. EEC tested no codes. Checked fuel pressure when stalled and pressure held pressure. Hooked up ignition tester and problem was gone. Remove tester and problem came back. Then installed monitor and found loss of PIP signal. Replace TFI. Then tested OK. ABA old TFI and reconfirmed [8].
3.2. Assessing the TNI phenomenon data by additional tests and analysis
As noted in this diagnosis, A–B–A was valuable because the ignition tester failed to be useful. Yet, the technician could be confident that his diagnosis was accurate, since the vehicle functioned properly with the new module and still displayed symptoms when the old module was connected. The A–B–A technique showed that the TFI was the problem.
As TNI rates rose, Ford initiated a diagnostic data management system (DDMS) to help the product engineering community understand the causal factors that drove repairs, replacements, and service [8,20–22]. Sixteen of Ford’s best and larger dealerships were chosen to participate in the program to be monitored by Ford’s local FSEs, who could oversee diagnosis. The goals were to generate sufficient reports to be statistically significant and to have experienced technicians help make an accurate diagnosis. In the DDMS, modules returned under warranty were subjected to a parametric three-temperature test. If the module passed the initial test, it would be labeled TNI in the DDMS database. In some cases, additional testing of TNI modules was conducted, including a three-temperature test and thermal cycling. Failure of a module at this stage resulted in a note stating that the module failed the lab test, although the TNI classification was not changed. Pie charts for 1986–1989 model year vehicle DDMS data are shown in Figs. 1–4. The data include results for TNI modules subjected to additional lab tests. The charts show the various defects that were observed based on visual and three-temperature tests alone. They also show the number and percentage of defects that were not verified from these tests. The charts then show the results of ‘‘selected lab tests’’ (for example, thermal cycling and extended range threetemperature tests) in which defects were precipitated and the module failed. Unfortunately, this type of indepth analysis was not conducted on all the returned modules. Dealer participation in the DDMS program also required the technicians to use a procedure called
4. Problem identification by the module manufacturer Modules that are under warranty are binned at the warranty return center according to the module manufacturer’s name. The cause of failure, the failure mechanism, and the failure mode should be determined at this ‘‘plant’’ investigation level [12]. If the module manufacturer finds that the problem is with a component of 11 A–B–A testing is performed to assess whether the module, as opposed to the system, a connected sub-system, or some connection or connector, is causing the problem. The technician records the symptoms and the test results associated with the original module (module ‘‘A’’). Module A is then replaced with a known good module (module ‘‘B’’) and the vehicle is tested with module B in place. Finally module B is replaced by the original module A and vehicle testing is performed once again. If the vehicle symptoms occur only when module A is in place, module A is the candidate problem. If module B also fails or if module A functions properly after it is replaced, then there may be another reason for the vehicle problem. 12 The A–B–A was recommended for intelligent diagnosis with the predecessor Duraspark ignition module. Section 13F, Intermittent Diagnosis, page 13F, 29-02-2 instructed the technician to subject the module to a heat pump and tap the module to replicate the problem. ‘‘If this procedure results in ignition malfunction, substitute a known good module. If the malfunction is corrected by substitution, validate that the original module is at fault by reconnecting it to the vehicle.’’
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
Fig. 1. 1986 Model year Lansdale TFI DMS analysis (126 modules analyzed) (data obtained from [40]).
Fig. 2. 1987 Model year Lansdale TFI DMS analysis (237 modules analyzed) (data obtained from [40]).
645
646
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
Fig. 3. 1988 Model year Lansdale TFI DDMS analysis (70 modules analyzed) (data obtained from [40]).
Fig. 4. 1989 Model year Lansdale TFI DDMS analysis (75 modules analyzed) (data obtained from [40]).
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
the module, that component may be sent to the part manufacturer for analysis. 13 The testing conducted upon receiving the TFI module consisted of a bench test, which included a visual inspection and parametric three-temperature testing. 14 Visual inspection was hard because microscopic intermittent failures can be difficult, if not impossible, to detect, especially if the location of the failure is unknown. For example, ‘‘ceramic substrates can develop microcracks in the thick film substrate [of the TFI] which can cause intermittent problems. This intermittence had a major impact on the TNI percentage, possibly 20 percent’’ [23]. Ford North Penn Electronics Facility found that a module could pass the tests but fail in the field. A tester did not exist that could capture in real time every failure mode of the modules. It was also noted that parametric three-temperature testing ‘‘will not capture intermittents that could be caused by vibration and shock. It does not provide real-time monitoring of cracks as they develop’’ [23]. Further, validation tests were not correlated well with the field applications – packages passed lab tests but failed in proving grounds [24]. 15 In addition, tests conducted at Ford’s Lansdale and Motorola Automotive and Industrial Electronics Group (both TFI manufacturers [26]) yielded different results. For example, in a 1982 Ford memorandum [27] regarding 1865 units tested at Motorola and 1278 units at Lansdale, the following was observed:
647
mittent at Motorola’s 3-sec spark but subsequently passed Motorola’s bench test and Lansdale’s retest. Lansdale testing There were 44 NTFs [no trouble found] caused by test equipment malfunction. Finally, duplicating TNI problems was time consuming. A TFI failure analysis, in which nine days were required to recreate the failure, is described in a 1982 Ford memorandum [28], which states: During the course of PV [product validation] testing failures occurred which appeared to be related to thermal overstress. An extensive engineering study was conducted in an attempt to identify the root cause of this overstress. This effort included a PEO [Product Engineering Office] examination of FACC [Ford Aerospace and Communication Corporation] tester facilities from 4/5 to 4/9 [28]. Since no direct cause of the failures could be established a second PV sample was begun which included ICs from a second wafer lot. By 4/13 eight units failed catastrophically while under 140 C [284 F], 18v stress testing. . . . Armed with this FACC input PEO was able to duplicate conditions in the laboratory to recreate FACC’s observations. 5. Problem identification by the vehicle manufacturer
Motorola testing There were 89 NTFs [no trouble found] caused by testing difference for the stall mode shutdown test; relaxed deviated limits for fall time; minor output oscillations that pass Lansdale test criteria but fail Motorola testing; one unit that ‘‘sounded’’ inter-
13
When the part manufacturer receives the modules, it may also conduct some defect analysis, such as on an IC. If the IC functions during the IC test and is determined to have no physical defects, the IC manufacturer assumes that module failure is not caused by the IC, although the interconnection to the IC may have been the cause. 14 Parametric three-temperature testing was conducted to identify the condition of the module at three points in time during a hot, cold, and ambient temperature test. As a part of the DDMS program, modules were not thermal cycled until they passed the bench test and were already labeled TNI. The reader is referred to the charts of Figs. 1–4, which show that the additional test analysis did precipitate defective modules. 15 As an example, a Ford memorandum indicated that the ‘‘test equipment did not adequately measure the product performance’’ [25]. Thus, any correlation would be near impossible.
The vehicle manufacturer is ultimately responsible for the warranty returns and the TNI problem. The vehicle manufacturer collects warranty information from the dealers, determines the root cause of the returns, and answers to the customers and the regulatory agencies. With increased use of electronic components for driveability systems, a reliance on mere warranty data did not provide sufficient information for identifying driveability problems [18]. The automotive industry had been aware of the warranty return problems and the TNI phenomenon ever since they began introducing electronics into vehicles. One of the methods useful in tracking down TNI phenomena is the fishbone, or Isikawa, diagram. The success of this method depends on being able to identify as many of the potential failure modes and mechanisms as possible. Fig. 5 shows Ford’s fishbone diagram of some potential TFI module failure modes and mechanisms. To uncover the cause of TNIs, it is also helpful to determine all the potential load (stress) conditions that can arise throughout the lifecycle of the product. In some cases, links can be established to uncover a common cause. For example, one common link in the TFI
648
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
Fig. 5. Ford’s fishbone diagram of TFI module failure modes and mechanisms (reproduced from [41]). The * denotes a 50 000 mile concern, suggesting that Ford considered the failure mode or mechanism to be an issue for 5/50 warranty coverage.
failures was the severe underhood temperature environment to which the ignition module was exposed. High temperature and high temperature thermo-cycles were causal factors for TFI ignition module failures [29], and the TFI module warranty returns were highest on high temperature applications in warm weather areas [30]. Ford documented some of the reliability implications of high temperature [31] and thermal cycling: High temperature has several negative effects on the life of electronics. At high temperatures, many materials, including solder, weaken. Intermetallic reactions (chemical in nature) accelerate, sometimes forming compounds that reduce component life. A well-documented rule of thumb is that mean time to failure halves for each 10 C (18 F) rise in operating temperature. Thermal cycling causes failures by creating mechanical stress buildup in material systems due to differences in rates of expansion. Material fatigue and joint failures are the result [7]. Fig. 6 shows how temperature can affect the estimated 5/50 return rate. This kind of information can be used to suggest that field returns, even those listed as TNI, could have temperature-related failure mechanisms.
Vehicle testing performed by Ford in 1985 suggested that the TFI module was being exposed to temperatures higher than the module’s engineering specification temperature [32]. 16 In fact, Ford reported temperatures higher than the designed functional capability of the TFI module [30], and in one study, Ford found that of 16 1982–1986 model year vehicles tested, 12 had a TFI base plate temperature at or above the specified operating limit [34]. Furthermore, a Major/Major Risk 17 Assess-
16
The Society for Automotive Engineers (SAE) documents a temperature extreme of 150 C (300 F) for the underhoodengine location [33], with a fishbone diagram listing several potential failures due to temperature cycling and temperature extremes. 17 Ford defines ‘‘major risks’’ in two categories, major/minor and major/major. A major/minor risk applies to items that have failed to meet design or process standards or requirements, or reliability objectives, by the assessment date, but with corrective action, are projected to be a minor risk by job #1 (i.e. the production date). A major/major risk applies to items that have failed to meet design or process standards and requirements, or reliability objectives, and cannot be reduced to minor risk in time for job #1 or component production incorporation date [35].
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
649
Fig. 6. 1988 Estimated R/1000 versus TFI module temperature (reproduced from [34]).
ment of several 1985–1986 vehicle lines also indicated that temperatures had exceeded the TFI specification limit: Testing conducted at the Wind Tunnel (110 F ambient) [43.3 C] and/or Arizona Proving Ground identified temperatures, which exceeded the module design specification. The identified temperature could cause a thermal run away condition on the output transistor causing the vehicle to stall and not restart [36]. By 1986, ‘‘improvement proposals’’ were ‘‘planned’’ or ‘‘under investigation’’ to address the temperature issues and reduce TNIs. Potential solutions included: • Characterize IC at 140 C [284 F] vs. 125 C [257 F]. • Test IC at 140 C [284 F] vs. 125 C [257 F]. • Remote mount application 87 1/2 (3.8L Taurus) • Reduce underhood temperature. • Revise specification [37]. The manner in which Ford conducted warranty analysis was conducive to generating a large number of parts that tested TNI. First, not all the modules were tested, and those not tested were not considered warranty failures. Second, the tests were based on Ford engineering specifications, rather than on extensive failure analysis. Third, Ford Parts & Service Division (FPSD) had a policy whereby the manufacturer is not supposed to test returned parts to any greater extent
than the dealers are capable of if the manufacturer intends to chargeback those parts [38]. 18
6. Summary and conclusions: treating field returns as field failures It is good engineering practice to treat every field return (whether under warranty or not) as a field failure until the actual root cause of the return is known. In the auto industry, where the customer brings the vehicle to the dealer as a result of a problem, the dealer must assume that the customer did indeed have a problem. The dealer’s responsibility is to record the customer’s concerns and investigate the problem until a solution can be established. This investigation includes diagnostic monitoring, troubleshooting and A–B–A testing. All modules should be sent back to the manufacturer for root cause analysis even if the module is out of warranty. The module and the vehicle manufacturer
18
Ford chose not to chargeback dealers for returned ignition modules that were labeled TNI based on meeting the engineering specification. The reasoning behind that decision was, ‘‘due to the complexity of the electronics in the ignition module, there is no efficient way of ensuring with 100% confidence that a part is truly ‘‘TNI’’ [38].’’ In fact, in order to have charged back the dealers for TFIs labeled TNI, they would have had to show that their diagnosis of TNI was ‘‘bulletproof’’, which would be impossible to demonstrate, by definition.
650
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651
have the responsibility to conduct both tests and teardowns of every module to determine the root cause of the problem or to determine why a ‘‘good’’ module has been returned under warranty (i.e. to rule out fraud by the dealer). Module testing must simulate actual and worst-case field conditions and will thus often require combining multiple and time varying load (stress) conditions for extended periods of time. A sequence and/or a combination of environmental and operating conditions can often trigger a fault; this is common in computer operation, when a unique and often complex sequence of keystrokes and operating conditions can initiate an unpredictable intermittent fault. Tear-downs should be conducted after non-destructive failure analysis and testing and must involve investigating every potential failure mode and failure site to assess degradation and damage to the module. If no defect can be determined and there is no other verifiable and auditable explanation, then the module must still be considered a field failure, due to the elusive and intermittent nature of electrical failures. A manufacturer should assume that all field returns are field failures, unless some alternative reason can be verified. In fact, any company that produces a safety or emissions regulated product should assume that every complaint or return of that product is a failure, and take on full responsibility for ascertaining the root cause. It must not be assumed that a returned module that passes tests associated with an engineering specification is good. The TNIs associated with Ford’s TFI involved several manufacturing, electrical, and thermal–mechanical factors that contributed to the existence of the failures, as opposed to the existence of a single Achilles’ heel. These factors included insufficient part analysis, a lack of systematic data accumulation, and a lack of understanding of how the component reacted within its system and environment [9]. Supina [18] wrote that ‘‘a large part of the problem of TNIs is what we do not know. With most failure modes, there is a clear direction to move in: something to attack to solve the problem. TNIs are not that clear cut’’ and ‘‘It is pretty safe to say that there is not one big, unknown problem out there accounting for our 90% TNI rate.’’ It is therefore critical to have a cause-and-effects (fishbone) diagram that has all possible failure modes and mechanisms, but at the same time be open to the discovery of something new. Furthermore, the necessary resources and best practices must be applied to solve the problem. A company should never categorize module warranty returns as TNI as a means to ignore, mitigate, or transfer problems. In 1987, a limited recall of TFI modules was conducted, consisting of 1.1 million 1984 and 1985 1.5L and 2.3L 4-cylinder engine vehicles [39]. New parts had to be manufactured for each replacement, and millions of dollars were spent on the recall. In 1997, a class action
suit was initiated. The case ultimately ended in a mistrial in November 1999 when, after approximately six months of testimony, the jury was unable to reach a decision. However, the trial continued in a second phase before the trial judge, which resulted in the first judicially mandated recall in US history. In a Statement of Decision issued in August 2000, the judge found that Ford had intentionally concealed information about the TFI module defect from millions of consumers, as well as NHTSA and other government agencies. Ford vowed to appeal, but ultimately agreed to settle the litigation on a nationwide basis before the damages issues were retried before another jury. The settlement required Ford to extend the TFI module’s warranty in each vehicle to 100 000 miles and to provide full reimbursement of all module replacement costs incurred before the odometer reached 100 000 miles. In addition, Ford agreed to contribute US $5 million for independent colleges and universities to conduct automotive safety research.
References [1] Lau JH, Pao Y. Solder joint reliability of BGA, CSP, flip chip, and fine pitch SMT assemblies. New York: McGrawHill; 1997. [2] Pecht M, Ramappan V. Are components still the major problem: a review of electronic system and device field failure returns. IEEE Trans Components Hybrids Manufact Technol 1992;15(6):1160–4. [3] Ford Motor Company (11). Lynn L. Bradford, Director Office of Defects Investigation Enforcement. 12 October 1978. [4] Ford Motor Company (10). Owner notification programs M04 and M05. 11 February 1981. [5] Ford Motor Company (1). Ford engineering specification for part number ES-E3EF-12A297-AA. 1986. [6] Ford Motor Company (5). TFI-IV thick film ignition module 5/50 warranty review. 5 November 1986. [7] Ford Motor Company (2). 1986 Ignition components warranty review. Ignition Systems EPME/EED, 20 March 1986. [8] Ford Motor Company (16). Powertrain quality plan. 27 August 1986. [9] Ford Motor Company (7). Electronic systems quality trouble-not-identified with field return components. Electrical & Electronics Division, 21 October 1986. [10] Russell R. ETS high mileage testing. Memorandum, 17 May 1989. [11] Goldfarb Consultants. Perspectives on intermittent electrical problems at the dealership. Ford Motor Company; 1990. March. [12] Ford Motor Company (9). Reliability engineering classification tree. 27 July 1994. [13] Ford Motor Company (12). Driveability intermittents. 1986. [14] Wikarski K, Morosky E, Tabbert E. California MIL pilot program and dealer visitation. 13 July 1987.
D.A. Thomas et al. / Microelectronics Reliability 42 (2002) 641–651 [15] Ford Motor Company (8). Warranty indicator system: field service engineering investigation report. 18 May 1989. [16] Ford Motor Company (17). Perspectives on intermittent electrical problems at the dealership. San Francisco Area Focus Group, 21–22 February 1990. [17] Ford Motor Company (19). Vehicle electrical system: quality improvements & diagnosis of intermittents. Revised 22 November 1989. [18] Supina JG. TNIs. Inter-office memorandum, Electronics Division, Product Engineering Office, Ford Motor Company. 15 March 1991. [19] Ford Motor Company (18). TFI ignition MOL: 1991 MY 30% improvement shortfall. TFI4-5785. [20] Ford Motor Company (4). Lansdale TFI warranty analyses. 1988–1991. [21] Ford Motor Company (13). DDMS module analysis (TFIIV). TFI Product Quality Team, 20 April 1988. [22] Ford Motor Company (23). DDMS procedure. 5 September 1990. [23] Balok B. Requalification of TNI TFI-IV and EDIS modules. Inter Office Memorandum, Ford Motor Company, 3 September 1993. [24] Hu JM. Reliability validation of electronic packages. Presentation, Ford Motor Company, July 1996. [25] Davis DA. 1983 TFI-I push start test equipment issues. Inter-office memorandum, Electrical & Electronics Division, Product Engineering Office, Ford Motor Company, 30 April 1982. [26] Ford Motor Company (20). Ford and Motorola CS291 returns tracking log. 1990. [27] Simms H. T.F.I. 3 second spark test evaluation results. Inter-office memorandum, Ford Aerospace & Communications Corporation, Ford Motor Company, 6 January 1982. [28] Leszczynski SA. TFI-IV IC high voltage, high temperature concern. Handwritten letter, Ford Motor Company, 13 April 1982.
651
[29] Ford Motor Company (22). Case study: 1984/85 distributor ignition hall effect PIP sensor, analysis of field problem. TFI4-1364-1396, Ignition Components Department, Engine Engineering Office, revised 16 August 1985. [30] Ford Motor Company (21). Product problem and action status report: ignition components. TFI4-1395, 17 July 1985. [31] Jurosek ML, Macher FE. Underhood temperatures. Memorandum, Ford Electrical and Electronics Division, 31 July 1986. [32] Ford Motor Company (3). 1982/86 Distributor temperature tests EEC-IV vehicles—wind tunnel/APG testing. Ignition System EED, revised 1 and 14 August 1985. [33] SAE. Rev. Recommended environmental practices for electronic equipment design. SAE J1211, November 1978. [34] Ford Motor Company (15). 4.9/5.0/5.8L Remote mount TFI funding. Meeting announcement/agenda, 9 May 1989. [35] Ford Motor Company (6). Product engineering procedure no. 3-133. Vehicle program assessments, Attachment A, 19 March 1982. [36] Rivard JG. Major/major risk assessment on TFI-IV ignition module. Inter-office memorandum, Electrical & Electronics Division, Product Engineering Office, Ford Motor Company, 23 July 1985. [37] Gentilcore JA. Failure mode analysis: Part: Ign. module TFI-IV. 18 January 1986. [38] Lombardi J. Ignition System TNI Reduction Team 8D worksheet. Concern title: ignition system TNI reduction. 1 August 1990. [39] Center for Auto Safety. Center for Auto Safety action alerts. 1997. Available from: www.essential.org/orgs/cas/ action1.html. [40] Ford Motor Company (14). DDMS TFI warranty repair database. Model year 1986–1991. [41] Gentilcore JA. TFI ignition module failure mode Ishikawa diagram, 1985.