- Email: [email protected]
Comparative evaluations of medical devices
Chapter 68
Comparative evaluations of medical devices James P. Keller Business Development Director, Emergo by UL, Austin, TX, United States
Evaluating medical device technology is a complex process. Many factors contribute to its complexity, ranging from the wide variety of devices purchased and used in hospitals, to the various settings and applications where devices are used, to the many different types of users of the equipment. Many hospitals do not have a good process in place to address these complexities effectively or to evaluate their choices for medical technology. They subsequently make inappropriate, and sometimes unsafe, equipment procurement choices. The purpose of this chapter is to present a process that healthcare organizations can use to effectively evaluate medical technology. Many of the steps in this process are modeled after a 50-year-old, comparative brand-name evaluation program and processes used by ECRI Institute, a research organization in Plymouth Meeting, Pennsylvania, to advise hospitals on medical device selection. Examples from ECRI’s evaluation program will be used throughout this chapter.
Complexities of hospital purchase of medical equipment Hospitals buy a large number of medical devices. Worldwide, $ 350 billion was spent on medical devices and $ 110 billion was spent in the United States in 2014 (Holtzman et al., 2015). The types of devices include capital equipment, computer-based systems, equipment integrated with computer-based systems, drug/device combinations, disposable/reusable products, implants, and supplies. Most applications of the devices are for use within the hospital. However, many hospitals also acquire equipment and supplies for distribution by a durable medical equipment (DME) provider; for placement directly in a home setting by a hospital (e.g., without using the DME as a “middle man”); for research or a combination of research and clinical use; and for use in clinics, doctors’ offices, nursing homes, patient-transport vehicles, same-day surgery facilities, and other settings. Depending on the setting, the hospital will Clinical Engineering Handbook. https://doi.org/10.1016/B978-0-12-813467-2.00069-9 Copyright © 2020 Elsevier Inc. All rights reserved.
have one user to a wide range of users for any given type of equipment. In most cases, the users of the equipment are healthcare professionals with some level of degree- or certificate-based training. However, many devices are used by untrained individuals and even by patients. There are often several different types of devices that can be used for the same, or some of the same, types of applications. Hospitals typically have many vendors to choose from when selecting a class of medical equipment to purchase. Often, one manufacturer will offer several choices of equipment within the same device class or several different configurations of the same device. Many hospitals rely on vendor information and specifications to make their purchasing decisions. However, vendor specifications are not presented in a standardized format from vendor to vendor. Specifications, product descriptions, and general product information (e.g., summaries of clinical studies) are also presented in the most positive light for each manufacturer. Clinical literature about the utility of products also might be biased (e.g., funded by manufacturers of the products being evaluated), incomplete, or of poor quality, and typically it is not comparative. This all results in a confusing, and often misleading, equipment purchasing data set for the hospital. Medical equipment purchasing is rarely centrally controlled within an institution. Purchasing typically takes place independently in many departments throughout the same institution. In many cases, the different departments are purchasing equipment for the same or similar purposes. This results in a wide range of criteria and purchasing practices within the same institution, often with significantly different standards of care. When one hospital is part of a larger healthcare system, the number of variables in system-wide purchasing can become even more overwhelming. Hospitals that participate in, or that are members of, group purchasing organizations (GPO) theoretically avoid some of the complexities described above. GPOs typically take advantage of the economies of scale provided by their
451
452 SECTION | 7 Medical devices: Design, manufacturing, evaluation, and control
membership, to negotiate substantial group-rate-type discounts for a large list of devices. GPO member hospitals that buy devices from the GPO lists will receive the negotiated pricing. The GPOs will list one-to-several choices for members within each device category on the list. However, GPO member hospitals usually have not had a direct say in the selection of the GPO-selected products. These products have not been selected with the unique needs of each GPO member in mind. In fact, they might not have been selected for their clinical utility, effectiveness, safety, or costeffectiveness. The GPO’s selection criteria sometimes can be solely based on the lowest negotiated price. Although the GPO member is not typically locked into only buying the products on the GPO contract list, it is in its best interest to strongly consider the discounted prices that are available. In fact, some GPO contracts are based on the GPO member committing to purchasing a certain number of devices from the GPO list in order to obtain the negotiated prices. However, similar or better bargains might be available elsewhere, and in order to be thorough, hospitals need to investigate whether the GPO-listed devices actually meet their specific needs. The regulatory status of devices can impact their purchase. Most devices used for clinical applications are clearly labeled as medical devices according to FDA definitions. However, some devices, such as clinical information systems, do not have a formal regulatory status or others are in the process of becoming approved or cleared for sale within certain jurisdictions. Hospitals that are interested in purchasing this type of equipment have an additional set of factors to consider before purchasing, relating to liability, insurance coverage, government reimbursement, the competence of users, patient consent, and other issues. The complexities discussed above are but a few of those that hospitals are confronted with when purchasing their wide range of medical devices. The best way to address these complexities is for hospitals to have a process in place to objectively evaluate its equipment needs and the true value of, and need for, the equipment that the institution is considering. This process will be discussed below and will be modeled, in large part, after a 50-year-old comparative brand-name evaluation program and processes used by ECRI Institute to advise hospitals on medical device selection.
Identifying equipment needs Medical devices are, in theory, purchased to meet one or more specific needs for the healthcare facility. Before any type of device evaluation takes place, however, those needs must be clearly defined. The need could be as simple as measuring a patient’s blood pressure. On the surface, the solution to the problem is to buy enough s phygmomanometers
and stethoscopes to meet the needs of the clinicians who will be taking the measurements. However, “taking blood pressure measurements” is a general criterion. The hospital’s true need, at least in some clinical departments, might be to record blood pressure readings automatically; to display the data in a way that can warn clinicians when blood pressure reaches dangerous levels; to store the readings in some form of database like in a patient’s electronic health record (EHR) for trending of blood pressure fluctuations; and to integrate the blood pressure data with other clinical parameters to manage patient care more effectively. Handheld sphygmomanometers and stethoscopes clearly cannot meet these needs. As with the blood pressure measurement example, most equipment purchase decisions are not as simple as the number and brand of device to buy. Each equipment purchase consideration must answer a series of questions before the purchasing process gets to the point of considering the number and brand to purchase. The first question to ask is whether you need equipment for an application. Sometimes, a better choice for an application might be drug therapy, rather than a device solution. If a device solution is appropriate, the next question concerns the type of equipment needed. Equipment evaluators should keep in mind that there might be several types of equipment choices, such as lasers, electrosurgical units, and harmonic scalpels for surgical dissection and coagulation. Each alternative must be considered with the specific needs of the institution in mind. Equipment evaluators also should determine the clinical specialties that would need the equipment or that have similar needs. For example, orthopedic surgery might request a certain type of laser for dissecting tissue during joint surgery. However, the laser might not be used often enough merely by orthopedic surgeons, to justify such a costly acquisition; lasers for orthopedic surgery typically cost well over $ 100,000 (ECRI, 2016). However, if it is determined that other surgical specialties will benefit from the use of the laser as well, the acquisition could be justified. Determining true clinical need is a critical step in the evaluation process. Unfortunately, many hospitals do not carefully consider this issue and have paid the high price of buying medical devices that are not really needed; the devices become expensive dust collectors. In some cases, the equipment is utilized, but not often enough for staff to become proficient with its use. Each time the device is used, the clinical staff must relearn how to use it. When this happens, the cost of poor purchase decisions also becomes a safety concern. Patients and staff are at risk of being injured by medical devices that operators do not know how to use proficiently. ECRI has operated a medical device problemreporting and investigation program for 50 years. It has investigated many cases of a serious patient or staff injury because use of medical devices that clinical staff did not fully understand how to use.
Comparative evaluations of medical devices Chapter | 68 453
Careful research is required, to determine the true clinical need for medical devices. It should include interviews with staff who will be using or affected by the equipment; review of relevant clinical and technical literature on the benefits and drawbacks of the technology; review of the hospital’s past history with similar equipment; surveys of other organizations’ experience with the equipment being considered; discussions with manufacturers; review of manufacturer literature; and consultation with outside experts. In some cases, there will be overwhelming evidence that a certain type of technology is a standard of care and totally appropriate for the application(s) being considered. However, with new and emerging technologies, the information gathered must be examined critically for bias and quality of data. ECRI has observed many examples of new and emerging technologies being prematurely implemented because of over-reliance on poor-quality clinical data or overly vigorous promotion of the technology by proponents who will receive financial gains from its success.
Establishing basic selection criteria Once it has been determined that a certain class of equipment is needed by the hospital, the actual device evaluation process can begin. The first step is to determine the features needed from the device. As mentioned above, there are often a wide variety of devices offered within one device class. Many of the features that are available for the different models are absolutely necessary for the devices to serve their main purpose. However, as with a car, many devices have features that might offer little more than cosmetic benefits that come at a high cost. These are the often-termed “bells and whistles.” Other features might be clinically useful, but they are not needed for the specific application being considered. Conversely, some devices lack a key feature in order to work as designed. One of the first steps of a device evaluation is to identify the “must have” features for the devices being considered for purchase. Once these features have been identified, the universe of devices available for purchase can be narrowed down to a manageable number. ECRI’s brand-name comparative evaluation program uses a rating system that is based on an essential-features analysis. It is based on a five-star rating system in which, in general a five-star device is a “preferred” choice, a threestar device is “acceptable,” and devices given one star are “not recommended.” ECRI Institute also has an “unacceptable” rating for devices that have one or more features that prevent them from performing their intended purposes; create an unacceptable risk, or are prohibitively expensive. For example, infusion pumps that do not have automated protection from free-flow delivery of medications and other intravenous fluids can deliver a fatal overdose. ECRI rates infusion pumps that lack this feature as “unacceptable” and recommends that hospitals remove these devices from their
purchase-consideration list. In fact, ECRI recommends that these devices be removed from use and replaced with the safer, free-flow protected devices. Other device features that have resulted in “unacceptable” ratings by ECRI have included needle-stick-prevention devices that lacked automated needle-stick protection (ECRI, 1999) and thoracic aspirators that lacked sufficient specified volumetric airflow to overcome large air leaks in the thoracic cavity and thus provide poor suction (ECRI, 1998a). In each example, the devices were not able to perform their intended purposes, at least according to ECRI’s expectations, and they were unsafe. The essential-features analysis from the ECRI evaluation examples covers generic problems or characteristics that would apply to any institution’s situation. However, some issues will come up in a hospital’s device selection that will be unique to its circumstances or at least a small sampling of similar hospitals. Part of the essential-features analysis is to identify the features that the devices under consideration must have, or should not have, in order to meet the hospital’s unique needs. For example, if the hospital is operating under a specific software platform, with which a medical device-based computer system is not compatible, the medical device should be ruled out of the hospital’s purchase consideration unless the hospital is in the process of also upgrading the software platform to a compatible system. In another example, many hospitals are beginning to integrate medical devices with their EHRs to support clinical decision support, centralize patient data, and generally improve patient safety. Technologies commonly being integrated with EHRs include infusion pumps, physiologic monitors, and dialysis machines. However, not all devices are compatible with all or even some EHRs. And, many require some type of middleware applications and devices in order for the integration to work. The selection process for technologies like infusion pumps, physiologic monitors, and dialysis machines has therefore evolved so that it needs to confirm their ability to transfer and receive data to and from the appropriate information systems in the hospital, such as the EHR.
Detailed comparative evaluation criteria After narrowing down the device-selection options using an essential features analysis, the real comparative nature of the evaluation process can begin. This process involves comparing the devices or systems being considered in a variety of general categories. Within each category, specific criteria must be established in order to effectively compare the systems to one another. The general categories include overall performance and safety, ease of use and other human factors issues, quality of construction, service and support, and cost. Depending on the type of device or system being evaluated, one category might need to receive more attention than others in the evaluation process. Another
454 SECTION | 7 Medical devices: Design, manufacturing, evaluation, and control
factor that determines the category that receives a higher level of attention will be the setting for the device under consideration. Devices intended for home use, especially those used and operated by patients, will have significant ease-of-use concerns. Devices intended for transport use, especially in helicopters, should receive a greater emphasis on quality-of-construction factors, such as the ability to withstand shock and vibration. In order to conduct a thorough evaluation, a comprehensive set of criteria must be established for each device being evaluated in each of the general categories mentioned above. This can be a daunting task for any hospital or other healthcare institution, especially considering the large number and variety of medical devices purchased every year. However, despite the confusing set of device-selection data with which hospitals are confronted, resources do exist to make this process much more manageable. One of the most well-known and respected resources is the Health Devices Evaluation program operated by ECRI Institute, a nonprofit corporation that engages in the independent evaluation of technology. For 50 years, ECRI has been conducting evaluations that are often compared to those of a similar institution, Consumers Union, publisher of Consumer Reports. ECRI has established rigorous conflict-of-interest rules and evaluation methods that support its unequaled reputation as a totally objective and comprehensive medical technology information source. Hospitals routinely refer to ECRI’s many comparative device evaluations for guidance on establishing their own evaluation criteria, or the ECRI criteria are fully adopted into the hospital’s process. Criteria for ECRI’s comparative evaluations are developed through a rigorous review of related clinical and technical literature, standards, and previous ECRI evaluations, and from ECRI’s experience in working with and investigating problems involving the evaluated technology. ECRI technical staff members also spend many hours interviewing experts on, and users of, the technology and observing the technology in clinical use. Manufacturers of the evaluated products are invited to ECRI’s laboratories to demonstrate their products and for questioning by ECRI Institute technical staff. Through this process, important performance characteristics are identified, user problems are noted, and other evaluation-related factors are established, from which the details of the criteria are developed. Examples of device areas where evaluation criteria are established include allowable turning rotation on control knobs, alarm volume thresholds on anesthesia machines, flow control accuracy on critical care ventilators, radiation exposure dose limits on CT scanners, puncture resistance for surgical gloves, and level of acceptable display-screen clutter on physiologic monitors. The ECRI criteria are packaged in a well-organized document and critically reviewed by ECRI technical staff, outside experts, users, and manufacturers whose products are
being evaluated. This results in a comprehensive, but practical, set of expectations for the evaluated device or system. Hospitals must examine each of ECRI’s criteria to verify that they apply to their unique situation. In most cases, the criteria will be completely applicable. Where a criterion does not apply or has not been established by ECRI, evaluations of similar technology by ECRI can be used to develop new or to revise existing criteria. The hospital conducting the evaluation can also use ECRI’s process of reviewing relevant clinical and technical literature and standards, and consultations with experts (including ECRI staff) and users to develop the criteria.
Critical evaluation criteria Because of the wide range of devices used in hospitals and other healthcare settings, it is impossible in the context of this chapter to describe critical-evaluation criteria for all of these devices. However, a number of examples based on recent ECRI evaluations and the experience of the author are presented below. Examples are provided for each of the general categories mentioned above.
Performance and safety Arthroscopic irrigation and distension systems are designed to distend a joint cavity with fluid to provide an orthopedic surgeon with a workable operating space within the joint during arthroscopic surgery. High pressures from the arthroscopic irrigation and distension systems are known to cause extravasation of the distension fluid into the tissue surrounding the joint cavity. At a minimum, extravasation can cause irritation and delayed healing in and around the surgically treated joint. More serious complications include nerve palsy, arterial compression, and even cardiac arrest. Clinical studies have shown that joint pressures of as low as 180 mmHg can cause extravasation. To minimize the risk of extravasation, arthroscopic irrigation and distension, systems should have a limit so that they cannot deliver fluid in the joint at pressures higher than 180 mmHg and joint pressure should be monitored by the arthroscopic irrigation and distension systems so that users can be alerted, should pressure levels reach the 180 mmHg limit (ECRI, 1998b).
Ease of use and other human factors issues Blood glucose monitors are used by diabetic patients and clinicians to monitor and to help control a patient’s blood glucose level. Most of these devices are used by patients outside of the hospital setting. Many of these patients are elderly, with limited manual dexterity and impaired vision. Blood glucose monitors have a reading port into which patients place a small calibrated test strip, onto which they have applied blood, for analysis. In order to accommodate
Comparative evaluations of medical devices Chapter | 68 455
users with limited manual dexterity, the test strips cannot be so small as to make the strip difficult for the user to handle. The display image must be large enough to present the glucose measurement in an easily legible format, clearly viewable under a variety of lighting conditions (ECRI, 1997).
Quality of construction A relatively unique type of electrosurgical return electrode uses a capacitive coupling to return electrosurgical energy used for cutting and coagulating of tissue to an electrosurgical unit from the surgical site on a patient. The return electrode is designed to disperse the electrosurgical current over its large surface area and to eliminate high-current densities in the electrosurgical current return path, which can cause a patient burn. This electrode is unique because of its large size (almost 2 ft × 3 ft) and is designed to be reused many times. The electrode consists of a large sheet of conductive fabric that is covered by a urethane insulating material. Because this device is reusable, it likely will be exposed many times to sharp objects and other abuse in the operating room over its expected useful life. If penetrated by sharp objects or otherwise damaged, the conductive fabric could be directly exposed to the patient’s skin, at which point electrosurgical current could concentrate and cause serious burns. This product, therefore, must be resistant to tearing and other damage that might occur during normal use in the operating room (ECRI, 2000).
Service and support It has been estimated that between 40% and 70% of medical devices in low- and middle-income countries are either broken, not being used, or unfit for their intended purpose. This is a major problem with health care in the developing world, which has been incomplete costing and inadequate consideration of maintenance services and user training during procurement planning (Diaconu et al., 2017). A critical criterion for an evaluation of any technology, especially one being considered in a middle- or low-income country, is that the manufacturer or supplier have the ability to provide comprehensive, responsive, and reliable support from installation, day-to-day operations, routine maintenance and repairs, system upgrades, and user training. And, many medical devices and systems include built-in software routine updating and patching to address security-related problems. Service and support evaluation criteria for these technologies need to address the manufacturer or supplier’s ability to provide timely and reliable software upgrades— for a reasonable price.
Cost Rarely is the cost of a medical device limited to the purchase price of the device itself. The additional costs could include accessories, service and support, installation fees,
additional staff needed to run the equipment, reagents, middleware, annual software license fees, and financing fees. In order to do a true comparative analysis of the costs for several choices of medical devices or systems, all of the applicable costs must be considered over the expected life of the equipment, typically for at least 5 or 7 years. When reagents or accessories are involved, evaluators must estimate the annual use of these elements in the equation and must come up with a projected annual cost for their use. The general cost criterion for any device evaluation is that a device should have a favorable life cycle cost as compared to other evaluated devices. The life cycle cost analysis must include all expected costs for the equipment, taking into consideration expected annual inflation rates and a discount factor to account for the changing value of money over the time of the analysis.
Testing for compliance with evaluation criteria Verifying compliance with essential criteria is relatively easy. Evaluators simply need to check with manufacturer specifications and to determine whether the devices being considered are specified to meet the essential criteria. If they do not, they can be immediately ruled out of the evaluation process. However, verifying the more specific performance and safety, ease of use or other criteria is not that easy. In order to get it right and to avoid inappropriate or more seriously ineffective, overly expensive, or unsafe choices, a comprehensive effort is necessary. In some cases, physical testing is required. In other cases, qualitative assessments and comparative analyses must be used. With life cycle analysis, all costs must be identified, and projected volumes must be carefully estimated. The data then must be entered into analytical tools such as life cycle cost calculators to obtain results for comparison. The staff who do all of the research, testing, and analysis depends on the technology being evaluated and the hospital’s previous history with the technology being considered. Older, more established, or simple technology might not require substantial hands-on analysis by the hospital. Consultation with an outside organization like ECRI for its experience with, and purchase advice for, the technology might be sufficient. Newer or more sophisticated (especially highly customized) technology like enterprise-wide and interconnected technologies or hospital information systems likely will need a dedicated team, including outside experts and nonbiased analyses, to determine how effectively the systems comply with the criteria, especially those that address the evaluating hospital’s unique circumstances or customization needs. It is important to include potential users on any assessment that occurs at the hospital. Users can provide valuable
456 SECTION | 7 Medical devices: Design, manufacturing, evaluation, and control
insights on any assessment of performance, device usability, or other characteristics and will be much more willing to embrace new technology that is eventually acquired if they believe that their opinion was sought and headed in the selection process. On many occasions, ECRI Institute has observed technology being thrust upon clinical staff without its input. These situations create ill will between equipment selectors and users and often result in poor utilization of the new technology. When ECRI conducts a comparative evaluation of a medical device, it uses detailed test methods and presents these and its findings in its online publication called health devices. Hospital equipment evaluators should look to this resource for guidance on ways to perform their analyses. In many cases, ECRI will have tested and presented results on the specific products being considered by the hospital. In this case, all that the hospital must do is to verify that ECRI’s findings apply to the hospitals’ unique circumstances. ECRI technical staff also can be used by the hospital evaluators for advice on ways to perform analysis and testing, which testing is most important to consider, and whether testing by an independent body is warranted for a particular device evaluation.
receiving a high weight in the device’s final analysis. As an example, consider an infusion device evaluation where the device was found to work well, except for over a range of infrequently used flow settings. The accuracy at those flow settings was such that patients could receive a dangerous overdose of medication. Despite doing well on all of the frequently used and important settings, the system should be given an overall low rating for the evaluation. How low the rating is set would depend on how likely the infrequently used flow settings actually would be used at the hospital. This also could depend on the type of instructions that would have to be provided to the clinical staff on the problem, how likely the staff would follow the instruction over time, and whether there are other acceptable alternatives to the device with the accuracy problem. Once the analysis is complete, evaluators should review their findings and conclusions carefully. The review should conclude that the hospital would end up acquiring a safe, effective, easy-to-use, clinically needed, and cost-effective device or system. If not, the appropriateness of the acquisition decision should be seriously considered.
Dealing with the findings
Evaluating medical technology is a complex process. Many factors make the process so complex, ranging from the wide variety of devices purchased and used in hospitals, to the various settings and applications where devices are used, to the many different types of users of the equipment. Newer applications of medical technology such as connectivity with hospital information systems add to this complexity. Many hospitals do not have good processes in place to effectively evaluate their choices for medical technology, and subsequently, they make inappropriate, and sometimes unsafe, equipment procurement choices. In order to have an effective evaluation program, a process must be put in place to identify the true need for medical equipment. Once a true need for equipment has been identified, the features required by the hospital for the equipment must be identified. Only equipment with the required features should be considered for further evaluation. Detailed criteria then must be established for the remaining equipment under consideration. Resources like ECRI’s Health Devices evaluations are valuable tools to use in establishing the evaluation criteria. ECRI Institute’s evaluation criteria often can be wholly incorporated into the hospital’s evaluation process. Once the evaluation criteria have been established, a careful process of assessing the equipment for compliance with the criteria can begin. ECRI and other outside resources can be utilized for this type of information. Hospitals likely will need to do some analysis on their own or with the assistance of outside experts, especially for highly customized equipment, such as clinical
No technology is perfect. Some devices or systems might work well but are somewhat difficult to use. Another might work well yet might have a safety problem that is correctable with routine user intervention. Another might be the best performer, might be the easiest to use, and might possess the most desirable feature, yet it might be significantly more costly than the alternatives. These are the types of findings that equipment evaluators are likely to be confronted with at the end of the evaluation process. The evaluator’s role is to assess the importance of each finding, then apply a weight or some other type of judgment scale to the finding, and tally the results to come to a final conclusion. The challenge for the evaluators is to determine the appropriate weight for each finding. And, unfortunately, one weighting system cannot be used for every class of device. For example, image quality is a critical factor when assessing digital radiography system or surgical video system. However, it is important, but not critical when assessing a physiologic monitoring system. Equipment evaluators must look at the individual features of each evaluated technology and determine the features that are most important. Findings relating to the more important features can be assigned greater weight in the analysis. Evaluators then need to look at individual results in every category to determine whether any finding could affect the safety or overall effectiveness of the equipment significantly. Depending on the nature of the problem, a finding relating to an infrequently used feature could end up
Summary
Comparative evaluations of medical devices Chapter | 68 457
information systems. Once results have been obtained, a weighting system must be established to help the evaluating committee come to a final conclusion. Factors such as safety, overall performance, ease of use, likelihood of the equipment being used, clinical utility of the equipment, and cost should be important considerations in establishing the weighting system.
References Diaconu, K., Chen, Y.F., et al., 2017. Methods for medical device and equipment procurement and prioritization within low- and middle-income countries: findings of a systematic literature review. Glob. Health 13, 59. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5563028/. (Accessed 21 December 2017). ECRI, 2016. Lasers, Ho:YAG, Surgical. Healthcare Product Comparison System. https://www.ecri.org/Components/HPCS/Pages/Lasers,-HoYAG,-Surgical.aspx. (Accessed 21 December 2017). Membership login required.
Holtzman, Y., Gorkhover, G., Ganz, M., 2015. The U.S. Medical Device Industry: Strengths, Weaknesses, Opportunities, and Threats. MDDI (Medical Device Diagnostic Industry). https://www.mddionline.com/ us-medical-device-industry-strengths-weaknesses-opportunities-andthreats. (Accessed 21 December 2017). ECRI, 1999. Needlestick prevention devices (update evaluation). Health Devices 28 (10), 395. ECRI, 1998a. Thoracic aspirators (evaluation). Health Devices 27 (12), 434. ECRI, 1998b. Arthroscopic irrigation/distension systems (evaluation). Health Devices 28 (7), 247–248. ECRI, 1997. Portable blood glucose monitors (evaluation). Health Devices 26 (9–10), 347. ECRI, 2000. Megadyne Mega 2000 return electrode (evaluation). Health Devices 29 (12), 451–453.
Further reading MHMG, 2000. Medical and Health Care Marketplace Guide 1999–2000. Dorland’s Biomedical, Philadelphia, pp. 1–653.
Comparative evaluations of medical devices James P. Keller Business Development Director, Emergo by UL, Austin, TX, United States
Evaluating medical device technology is a complex process. Many factors contribute to its complexity, ranging from the wide variety of devices purchased and used in hospitals, to the various settings and applications where devices are used, to the many different types of users of the equipment. Many hospitals do not have a good process in place to address these complexities effectively or to evaluate their choices for medical technology. They subsequently make inappropriate, and sometimes unsafe, equipment procurement choices. The purpose of this chapter is to present a process that healthcare organizations can use to effectively evaluate medical technology. Many of the steps in this process are modeled after a 50-year-old, comparative brand-name evaluation program and processes used by ECRI Institute, a research organization in Plymouth Meeting, Pennsylvania, to advise hospitals on medical device selection. Examples from ECRI’s evaluation program will be used throughout this chapter.
Complexities of hospital purchase of medical equipment Hospitals buy a large number of medical devices. Worldwide, $ 350 billion was spent on medical devices and $ 110 billion was spent in the United States in 2014 (Holtzman et al., 2015). The types of devices include capital equipment, computer-based systems, equipment integrated with computer-based systems, drug/device combinations, disposable/reusable products, implants, and supplies. Most applications of the devices are for use within the hospital. However, many hospitals also acquire equipment and supplies for distribution by a durable medical equipment (DME) provider; for placement directly in a home setting by a hospital (e.g., without using the DME as a “middle man”); for research or a combination of research and clinical use; and for use in clinics, doctors’ offices, nursing homes, patient-transport vehicles, same-day surgery facilities, and other settings. Depending on the setting, the hospital will Clinical Engineering Handbook. https://doi.org/10.1016/B978-0-12-813467-2.00069-9 Copyright © 2020 Elsevier Inc. All rights reserved.
have one user to a wide range of users for any given type of equipment. In most cases, the users of the equipment are healthcare professionals with some level of degree- or certificate-based training. However, many devices are used by untrained individuals and even by patients. There are often several different types of devices that can be used for the same, or some of the same, types of applications. Hospitals typically have many vendors to choose from when selecting a class of medical equipment to purchase. Often, one manufacturer will offer several choices of equipment within the same device class or several different configurations of the same device. Many hospitals rely on vendor information and specifications to make their purchasing decisions. However, vendor specifications are not presented in a standardized format from vendor to vendor. Specifications, product descriptions, and general product information (e.g., summaries of clinical studies) are also presented in the most positive light for each manufacturer. Clinical literature about the utility of products also might be biased (e.g., funded by manufacturers of the products being evaluated), incomplete, or of poor quality, and typically it is not comparative. This all results in a confusing, and often misleading, equipment purchasing data set for the hospital. Medical equipment purchasing is rarely centrally controlled within an institution. Purchasing typically takes place independently in many departments throughout the same institution. In many cases, the different departments are purchasing equipment for the same or similar purposes. This results in a wide range of criteria and purchasing practices within the same institution, often with significantly different standards of care. When one hospital is part of a larger healthcare system, the number of variables in system-wide purchasing can become even more overwhelming. Hospitals that participate in, or that are members of, group purchasing organizations (GPO) theoretically avoid some of the complexities described above. GPOs typically take advantage of the economies of scale provided by their
451
452 SECTION | 7 Medical devices: Design, manufacturing, evaluation, and control
membership, to negotiate substantial group-rate-type discounts for a large list of devices. GPO member hospitals that buy devices from the GPO lists will receive the negotiated pricing. The GPOs will list one-to-several choices for members within each device category on the list. However, GPO member hospitals usually have not had a direct say in the selection of the GPO-selected products. These products have not been selected with the unique needs of each GPO member in mind. In fact, they might not have been selected for their clinical utility, effectiveness, safety, or costeffectiveness. The GPO’s selection criteria sometimes can be solely based on the lowest negotiated price. Although the GPO member is not typically locked into only buying the products on the GPO contract list, it is in its best interest to strongly consider the discounted prices that are available. In fact, some GPO contracts are based on the GPO member committing to purchasing a certain number of devices from the GPO list in order to obtain the negotiated prices. However, similar or better bargains might be available elsewhere, and in order to be thorough, hospitals need to investigate whether the GPO-listed devices actually meet their specific needs. The regulatory status of devices can impact their purchase. Most devices used for clinical applications are clearly labeled as medical devices according to FDA definitions. However, some devices, such as clinical information systems, do not have a formal regulatory status or others are in the process of becoming approved or cleared for sale within certain jurisdictions. Hospitals that are interested in purchasing this type of equipment have an additional set of factors to consider before purchasing, relating to liability, insurance coverage, government reimbursement, the competence of users, patient consent, and other issues. The complexities discussed above are but a few of those that hospitals are confronted with when purchasing their wide range of medical devices. The best way to address these complexities is for hospitals to have a process in place to objectively evaluate its equipment needs and the true value of, and need for, the equipment that the institution is considering. This process will be discussed below and will be modeled, in large part, after a 50-year-old comparative brand-name evaluation program and processes used by ECRI Institute to advise hospitals on medical device selection.
Identifying equipment needs Medical devices are, in theory, purchased to meet one or more specific needs for the healthcare facility. Before any type of device evaluation takes place, however, those needs must be clearly defined. The need could be as simple as measuring a patient’s blood pressure. On the surface, the solution to the problem is to buy enough s phygmomanometers
and stethoscopes to meet the needs of the clinicians who will be taking the measurements. However, “taking blood pressure measurements” is a general criterion. The hospital’s true need, at least in some clinical departments, might be to record blood pressure readings automatically; to display the data in a way that can warn clinicians when blood pressure reaches dangerous levels; to store the readings in some form of database like in a patient’s electronic health record (EHR) for trending of blood pressure fluctuations; and to integrate the blood pressure data with other clinical parameters to manage patient care more effectively. Handheld sphygmomanometers and stethoscopes clearly cannot meet these needs. As with the blood pressure measurement example, most equipment purchase decisions are not as simple as the number and brand of device to buy. Each equipment purchase consideration must answer a series of questions before the purchasing process gets to the point of considering the number and brand to purchase. The first question to ask is whether you need equipment for an application. Sometimes, a better choice for an application might be drug therapy, rather than a device solution. If a device solution is appropriate, the next question concerns the type of equipment needed. Equipment evaluators should keep in mind that there might be several types of equipment choices, such as lasers, electrosurgical units, and harmonic scalpels for surgical dissection and coagulation. Each alternative must be considered with the specific needs of the institution in mind. Equipment evaluators also should determine the clinical specialties that would need the equipment or that have similar needs. For example, orthopedic surgery might request a certain type of laser for dissecting tissue during joint surgery. However, the laser might not be used often enough merely by orthopedic surgeons, to justify such a costly acquisition; lasers for orthopedic surgery typically cost well over $ 100,000 (ECRI, 2016). However, if it is determined that other surgical specialties will benefit from the use of the laser as well, the acquisition could be justified. Determining true clinical need is a critical step in the evaluation process. Unfortunately, many hospitals do not carefully consider this issue and have paid the high price of buying medical devices that are not really needed; the devices become expensive dust collectors. In some cases, the equipment is utilized, but not often enough for staff to become proficient with its use. Each time the device is used, the clinical staff must relearn how to use it. When this happens, the cost of poor purchase decisions also becomes a safety concern. Patients and staff are at risk of being injured by medical devices that operators do not know how to use proficiently. ECRI has operated a medical device problemreporting and investigation program for 50 years. It has investigated many cases of a serious patient or staff injury because use of medical devices that clinical staff did not fully understand how to use.
Comparative evaluations of medical devices Chapter | 68 453
Careful research is required, to determine the true clinical need for medical devices. It should include interviews with staff who will be using or affected by the equipment; review of relevant clinical and technical literature on the benefits and drawbacks of the technology; review of the hospital’s past history with similar equipment; surveys of other organizations’ experience with the equipment being considered; discussions with manufacturers; review of manufacturer literature; and consultation with outside experts. In some cases, there will be overwhelming evidence that a certain type of technology is a standard of care and totally appropriate for the application(s) being considered. However, with new and emerging technologies, the information gathered must be examined critically for bias and quality of data. ECRI has observed many examples of new and emerging technologies being prematurely implemented because of over-reliance on poor-quality clinical data or overly vigorous promotion of the technology by proponents who will receive financial gains from its success.
Establishing basic selection criteria Once it has been determined that a certain class of equipment is needed by the hospital, the actual device evaluation process can begin. The first step is to determine the features needed from the device. As mentioned above, there are often a wide variety of devices offered within one device class. Many of the features that are available for the different models are absolutely necessary for the devices to serve their main purpose. However, as with a car, many devices have features that might offer little more than cosmetic benefits that come at a high cost. These are the often-termed “bells and whistles.” Other features might be clinically useful, but they are not needed for the specific application being considered. Conversely, some devices lack a key feature in order to work as designed. One of the first steps of a device evaluation is to identify the “must have” features for the devices being considered for purchase. Once these features have been identified, the universe of devices available for purchase can be narrowed down to a manageable number. ECRI’s brand-name comparative evaluation program uses a rating system that is based on an essential-features analysis. It is based on a five-star rating system in which, in general a five-star device is a “preferred” choice, a threestar device is “acceptable,” and devices given one star are “not recommended.” ECRI Institute also has an “unacceptable” rating for devices that have one or more features that prevent them from performing their intended purposes; create an unacceptable risk, or are prohibitively expensive. For example, infusion pumps that do not have automated protection from free-flow delivery of medications and other intravenous fluids can deliver a fatal overdose. ECRI rates infusion pumps that lack this feature as “unacceptable” and recommends that hospitals remove these devices from their
purchase-consideration list. In fact, ECRI recommends that these devices be removed from use and replaced with the safer, free-flow protected devices. Other device features that have resulted in “unacceptable” ratings by ECRI have included needle-stick-prevention devices that lacked automated needle-stick protection (ECRI, 1999) and thoracic aspirators that lacked sufficient specified volumetric airflow to overcome large air leaks in the thoracic cavity and thus provide poor suction (ECRI, 1998a). In each example, the devices were not able to perform their intended purposes, at least according to ECRI’s expectations, and they were unsafe. The essential-features analysis from the ECRI evaluation examples covers generic problems or characteristics that would apply to any institution’s situation. However, some issues will come up in a hospital’s device selection that will be unique to its circumstances or at least a small sampling of similar hospitals. Part of the essential-features analysis is to identify the features that the devices under consideration must have, or should not have, in order to meet the hospital’s unique needs. For example, if the hospital is operating under a specific software platform, with which a medical device-based computer system is not compatible, the medical device should be ruled out of the hospital’s purchase consideration unless the hospital is in the process of also upgrading the software platform to a compatible system. In another example, many hospitals are beginning to integrate medical devices with their EHRs to support clinical decision support, centralize patient data, and generally improve patient safety. Technologies commonly being integrated with EHRs include infusion pumps, physiologic monitors, and dialysis machines. However, not all devices are compatible with all or even some EHRs. And, many require some type of middleware applications and devices in order for the integration to work. The selection process for technologies like infusion pumps, physiologic monitors, and dialysis machines has therefore evolved so that it needs to confirm their ability to transfer and receive data to and from the appropriate information systems in the hospital, such as the EHR.
Detailed comparative evaluation criteria After narrowing down the device-selection options using an essential features analysis, the real comparative nature of the evaluation process can begin. This process involves comparing the devices or systems being considered in a variety of general categories. Within each category, specific criteria must be established in order to effectively compare the systems to one another. The general categories include overall performance and safety, ease of use and other human factors issues, quality of construction, service and support, and cost. Depending on the type of device or system being evaluated, one category might need to receive more attention than others in the evaluation process. Another
454 SECTION | 7 Medical devices: Design, manufacturing, evaluation, and control
factor that determines the category that receives a higher level of attention will be the setting for the device under consideration. Devices intended for home use, especially those used and operated by patients, will have significant ease-of-use concerns. Devices intended for transport use, especially in helicopters, should receive a greater emphasis on quality-of-construction factors, such as the ability to withstand shock and vibration. In order to conduct a thorough evaluation, a comprehensive set of criteria must be established for each device being evaluated in each of the general categories mentioned above. This can be a daunting task for any hospital or other healthcare institution, especially considering the large number and variety of medical devices purchased every year. However, despite the confusing set of device-selection data with which hospitals are confronted, resources do exist to make this process much more manageable. One of the most well-known and respected resources is the Health Devices Evaluation program operated by ECRI Institute, a nonprofit corporation that engages in the independent evaluation of technology. For 50 years, ECRI has been conducting evaluations that are often compared to those of a similar institution, Consumers Union, publisher of Consumer Reports. ECRI has established rigorous conflict-of-interest rules and evaluation methods that support its unequaled reputation as a totally objective and comprehensive medical technology information source. Hospitals routinely refer to ECRI’s many comparative device evaluations for guidance on establishing their own evaluation criteria, or the ECRI criteria are fully adopted into the hospital’s process. Criteria for ECRI’s comparative evaluations are developed through a rigorous review of related clinical and technical literature, standards, and previous ECRI evaluations, and from ECRI’s experience in working with and investigating problems involving the evaluated technology. ECRI technical staff members also spend many hours interviewing experts on, and users of, the technology and observing the technology in clinical use. Manufacturers of the evaluated products are invited to ECRI’s laboratories to demonstrate their products and for questioning by ECRI Institute technical staff. Through this process, important performance characteristics are identified, user problems are noted, and other evaluation-related factors are established, from which the details of the criteria are developed. Examples of device areas where evaluation criteria are established include allowable turning rotation on control knobs, alarm volume thresholds on anesthesia machines, flow control accuracy on critical care ventilators, radiation exposure dose limits on CT scanners, puncture resistance for surgical gloves, and level of acceptable display-screen clutter on physiologic monitors. The ECRI criteria are packaged in a well-organized document and critically reviewed by ECRI technical staff, outside experts, users, and manufacturers whose products are
being evaluated. This results in a comprehensive, but practical, set of expectations for the evaluated device or system. Hospitals must examine each of ECRI’s criteria to verify that they apply to their unique situation. In most cases, the criteria will be completely applicable. Where a criterion does not apply or has not been established by ECRI, evaluations of similar technology by ECRI can be used to develop new or to revise existing criteria. The hospital conducting the evaluation can also use ECRI’s process of reviewing relevant clinical and technical literature and standards, and consultations with experts (including ECRI staff) and users to develop the criteria.
Critical evaluation criteria Because of the wide range of devices used in hospitals and other healthcare settings, it is impossible in the context of this chapter to describe critical-evaluation criteria for all of these devices. However, a number of examples based on recent ECRI evaluations and the experience of the author are presented below. Examples are provided for each of the general categories mentioned above.
Performance and safety Arthroscopic irrigation and distension systems are designed to distend a joint cavity with fluid to provide an orthopedic surgeon with a workable operating space within the joint during arthroscopic surgery. High pressures from the arthroscopic irrigation and distension systems are known to cause extravasation of the distension fluid into the tissue surrounding the joint cavity. At a minimum, extravasation can cause irritation and delayed healing in and around the surgically treated joint. More serious complications include nerve palsy, arterial compression, and even cardiac arrest. Clinical studies have shown that joint pressures of as low as 180 mmHg can cause extravasation. To minimize the risk of extravasation, arthroscopic irrigation and distension, systems should have a limit so that they cannot deliver fluid in the joint at pressures higher than 180 mmHg and joint pressure should be monitored by the arthroscopic irrigation and distension systems so that users can be alerted, should pressure levels reach the 180 mmHg limit (ECRI, 1998b).
Ease of use and other human factors issues Blood glucose monitors are used by diabetic patients and clinicians to monitor and to help control a patient’s blood glucose level. Most of these devices are used by patients outside of the hospital setting. Many of these patients are elderly, with limited manual dexterity and impaired vision. Blood glucose monitors have a reading port into which patients place a small calibrated test strip, onto which they have applied blood, for analysis. In order to accommodate
Comparative evaluations of medical devices Chapter | 68 455
users with limited manual dexterity, the test strips cannot be so small as to make the strip difficult for the user to handle. The display image must be large enough to present the glucose measurement in an easily legible format, clearly viewable under a variety of lighting conditions (ECRI, 1997).
Quality of construction A relatively unique type of electrosurgical return electrode uses a capacitive coupling to return electrosurgical energy used for cutting and coagulating of tissue to an electrosurgical unit from the surgical site on a patient. The return electrode is designed to disperse the electrosurgical current over its large surface area and to eliminate high-current densities in the electrosurgical current return path, which can cause a patient burn. This electrode is unique because of its large size (almost 2 ft × 3 ft) and is designed to be reused many times. The electrode consists of a large sheet of conductive fabric that is covered by a urethane insulating material. Because this device is reusable, it likely will be exposed many times to sharp objects and other abuse in the operating room over its expected useful life. If penetrated by sharp objects or otherwise damaged, the conductive fabric could be directly exposed to the patient’s skin, at which point electrosurgical current could concentrate and cause serious burns. This product, therefore, must be resistant to tearing and other damage that might occur during normal use in the operating room (ECRI, 2000).
Service and support It has been estimated that between 40% and 70% of medical devices in low- and middle-income countries are either broken, not being used, or unfit for their intended purpose. This is a major problem with health care in the developing world, which has been incomplete costing and inadequate consideration of maintenance services and user training during procurement planning (Diaconu et al., 2017). A critical criterion for an evaluation of any technology, especially one being considered in a middle- or low-income country, is that the manufacturer or supplier have the ability to provide comprehensive, responsive, and reliable support from installation, day-to-day operations, routine maintenance and repairs, system upgrades, and user training. And, many medical devices and systems include built-in software routine updating and patching to address security-related problems. Service and support evaluation criteria for these technologies need to address the manufacturer or supplier’s ability to provide timely and reliable software upgrades— for a reasonable price.
Cost Rarely is the cost of a medical device limited to the purchase price of the device itself. The additional costs could include accessories, service and support, installation fees,
additional staff needed to run the equipment, reagents, middleware, annual software license fees, and financing fees. In order to do a true comparative analysis of the costs for several choices of medical devices or systems, all of the applicable costs must be considered over the expected life of the equipment, typically for at least 5 or 7 years. When reagents or accessories are involved, evaluators must estimate the annual use of these elements in the equation and must come up with a projected annual cost for their use. The general cost criterion for any device evaluation is that a device should have a favorable life cycle cost as compared to other evaluated devices. The life cycle cost analysis must include all expected costs for the equipment, taking into consideration expected annual inflation rates and a discount factor to account for the changing value of money over the time of the analysis.
Testing for compliance with evaluation criteria Verifying compliance with essential criteria is relatively easy. Evaluators simply need to check with manufacturer specifications and to determine whether the devices being considered are specified to meet the essential criteria. If they do not, they can be immediately ruled out of the evaluation process. However, verifying the more specific performance and safety, ease of use or other criteria is not that easy. In order to get it right and to avoid inappropriate or more seriously ineffective, overly expensive, or unsafe choices, a comprehensive effort is necessary. In some cases, physical testing is required. In other cases, qualitative assessments and comparative analyses must be used. With life cycle analysis, all costs must be identified, and projected volumes must be carefully estimated. The data then must be entered into analytical tools such as life cycle cost calculators to obtain results for comparison. The staff who do all of the research, testing, and analysis depends on the technology being evaluated and the hospital’s previous history with the technology being considered. Older, more established, or simple technology might not require substantial hands-on analysis by the hospital. Consultation with an outside organization like ECRI for its experience with, and purchase advice for, the technology might be sufficient. Newer or more sophisticated (especially highly customized) technology like enterprise-wide and interconnected technologies or hospital information systems likely will need a dedicated team, including outside experts and nonbiased analyses, to determine how effectively the systems comply with the criteria, especially those that address the evaluating hospital’s unique circumstances or customization needs. It is important to include potential users on any assessment that occurs at the hospital. Users can provide valuable
456 SECTION | 7 Medical devices: Design, manufacturing, evaluation, and control
insights on any assessment of performance, device usability, or other characteristics and will be much more willing to embrace new technology that is eventually acquired if they believe that their opinion was sought and headed in the selection process. On many occasions, ECRI Institute has observed technology being thrust upon clinical staff without its input. These situations create ill will between equipment selectors and users and often result in poor utilization of the new technology. When ECRI conducts a comparative evaluation of a medical device, it uses detailed test methods and presents these and its findings in its online publication called health devices. Hospital equipment evaluators should look to this resource for guidance on ways to perform their analyses. In many cases, ECRI will have tested and presented results on the specific products being considered by the hospital. In this case, all that the hospital must do is to verify that ECRI’s findings apply to the hospitals’ unique circumstances. ECRI technical staff also can be used by the hospital evaluators for advice on ways to perform analysis and testing, which testing is most important to consider, and whether testing by an independent body is warranted for a particular device evaluation.
receiving a high weight in the device’s final analysis. As an example, consider an infusion device evaluation where the device was found to work well, except for over a range of infrequently used flow settings. The accuracy at those flow settings was such that patients could receive a dangerous overdose of medication. Despite doing well on all of the frequently used and important settings, the system should be given an overall low rating for the evaluation. How low the rating is set would depend on how likely the infrequently used flow settings actually would be used at the hospital. This also could depend on the type of instructions that would have to be provided to the clinical staff on the problem, how likely the staff would follow the instruction over time, and whether there are other acceptable alternatives to the device with the accuracy problem. Once the analysis is complete, evaluators should review their findings and conclusions carefully. The review should conclude that the hospital would end up acquiring a safe, effective, easy-to-use, clinically needed, and cost-effective device or system. If not, the appropriateness of the acquisition decision should be seriously considered.
Dealing with the findings
Evaluating medical technology is a complex process. Many factors make the process so complex, ranging from the wide variety of devices purchased and used in hospitals, to the various settings and applications where devices are used, to the many different types of users of the equipment. Newer applications of medical technology such as connectivity with hospital information systems add to this complexity. Many hospitals do not have good processes in place to effectively evaluate their choices for medical technology, and subsequently, they make inappropriate, and sometimes unsafe, equipment procurement choices. In order to have an effective evaluation program, a process must be put in place to identify the true need for medical equipment. Once a true need for equipment has been identified, the features required by the hospital for the equipment must be identified. Only equipment with the required features should be considered for further evaluation. Detailed criteria then must be established for the remaining equipment under consideration. Resources like ECRI’s Health Devices evaluations are valuable tools to use in establishing the evaluation criteria. ECRI Institute’s evaluation criteria often can be wholly incorporated into the hospital’s evaluation process. Once the evaluation criteria have been established, a careful process of assessing the equipment for compliance with the criteria can begin. ECRI and other outside resources can be utilized for this type of information. Hospitals likely will need to do some analysis on their own or with the assistance of outside experts, especially for highly customized equipment, such as clinical
No technology is perfect. Some devices or systems might work well but are somewhat difficult to use. Another might work well yet might have a safety problem that is correctable with routine user intervention. Another might be the best performer, might be the easiest to use, and might possess the most desirable feature, yet it might be significantly more costly than the alternatives. These are the types of findings that equipment evaluators are likely to be confronted with at the end of the evaluation process. The evaluator’s role is to assess the importance of each finding, then apply a weight or some other type of judgment scale to the finding, and tally the results to come to a final conclusion. The challenge for the evaluators is to determine the appropriate weight for each finding. And, unfortunately, one weighting system cannot be used for every class of device. For example, image quality is a critical factor when assessing digital radiography system or surgical video system. However, it is important, but not critical when assessing a physiologic monitoring system. Equipment evaluators must look at the individual features of each evaluated technology and determine the features that are most important. Findings relating to the more important features can be assigned greater weight in the analysis. Evaluators then need to look at individual results in every category to determine whether any finding could affect the safety or overall effectiveness of the equipment significantly. Depending on the nature of the problem, a finding relating to an infrequently used feature could end up
Summary
Comparative evaluations of medical devices Chapter | 68 457
information systems. Once results have been obtained, a weighting system must be established to help the evaluating committee come to a final conclusion. Factors such as safety, overall performance, ease of use, likelihood of the equipment being used, clinical utility of the equipment, and cost should be important considerations in establishing the weighting system.
References Diaconu, K., Chen, Y.F., et al., 2017. Methods for medical device and equipment procurement and prioritization within low- and middle-income countries: findings of a systematic literature review. Glob. Health 13, 59. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5563028/. (Accessed 21 December 2017). ECRI, 2016. Lasers, Ho:YAG, Surgical. Healthcare Product Comparison System. https://www.ecri.org/Components/HPCS/Pages/Lasers,-HoYAG,-Surgical.aspx. (Accessed 21 December 2017). Membership login required.
Holtzman, Y., Gorkhover, G., Ganz, M., 2015. The U.S. Medical Device Industry: Strengths, Weaknesses, Opportunities, and Threats. MDDI (Medical Device Diagnostic Industry). https://www.mddionline.com/ us-medical-device-industry-strengths-weaknesses-opportunities-andthreats. (Accessed 21 December 2017). ECRI, 1999. Needlestick prevention devices (update evaluation). Health Devices 28 (10), 395. ECRI, 1998a. Thoracic aspirators (evaluation). Health Devices 27 (12), 434. ECRI, 1998b. Arthroscopic irrigation/distension systems (evaluation). Health Devices 28 (7), 247–248. ECRI, 1997. Portable blood glucose monitors (evaluation). Health Devices 26 (9–10), 347. ECRI, 2000. Megadyne Mega 2000 return electrode (evaluation). Health Devices 29 (12), 451–453.
Further reading MHMG, 2000. Medical and Health Care Marketplace Guide 1999–2000. Dorland’s Biomedical, Philadelphia, pp. 1–653.