- Email: [email protected]
Impact of Quality Assurance Program: Providing Practice Assessment
P R
O
V I D
I N
G
P R
A C
T I C
E
A
S
S
E S
S
M
E N
T
R O B E R T A . S A P O R IT O , D .D .S .; C E C IL E A . F E L D M A N , D .M .D ., M .B .A .; D E N IC E C .l— S T E W A R T , D .D .S ., M .H .S .A .; H A F T O N E C H O L D T , P H .D .; R IC H A R D N. B U C H A N A N , D .M .D .
A
B
S
T
R
A
C
T
Participation in a selfadministered quality assessment (SAQA) program led to changes in New Jersey dentists’ perceptions of practice quality. Ninety-four percent indicated they discovered practice deficiencies. This study suggests that using a self administered quality assessment program, such as the SAQA program, can lead to a better understanding of a practice’s strengths and weaknesses.
his paper reports changes in dentists’ perceptions of practice quality before and after participation in a self-adm inistered quality assessm ent program . The quality assessm ent program was developed as p a rt of an 18-month project funded by th e Am erican Fund for D ental H ealth. The goal of the project, entitled “Development and V alidation of a Self-Assessment Q uality A ssurance Program ,” was to develop a self adm inistered quality assessm ent in stru m en t th a t can be used in a general d entist’s in-office quality assurance program .1The in stru m en t benefits dental practitioners by providing a non threaten in g quality assessm ent tool th a t practitioners can adm inister them selves. Such assessm ents are inexpensive and can have a significant im pact on the quality of care. D entists volunteered to participate in th e pilot study. Participation involved: ■■ answ ering a pre-assessm ent questionnaire regarding willingness to participate and perceived practice quality; *■ having a “peer-evaluator” dentist assess the volunteer’s office; ■■ perform ing a quality assessm ent by completing th e self adm inistered quality assessm ent questionnaire; ■* completing the post-assessm ent questionnaire regarding strengths and w eakness of the program and changes in th eir perception of practice quality. A comparison of th e dentists’ perception of practice quality before and after participation in the quality assessm ent program dem onstrates the in stru m en t’s ability to increase the den tists’ aw areness of quality im provem ent needs. M A T E R IA L S A N D M E T H O D S
In stru m en t d evelop m en t. The in stru m en t developed by the Development of Evaluation M ethods and Com puter Applications in D entistry project funded by the Kellogg Foundation was selected as 622
JADA, Vol. 125, May 1994
PRACTICE M ANAG EM ENT
the basis for the program .2 The DEMCAD instru m en t was selected because it has a scientific basis and provides a comprehensive evaluation of dental office structure, process and outcomes.2,3 The DEMCAD instru m en t distinguishes among dental practices th a t differ in th eir characteristics.2 The instru m en t also effectively evaluated rural, urban group and urban non-group practices located in all U.S. geographic regions.2 D entist peerevaluators can be trained to use the instrum ent in a standardized approach th a t produces comparable results when evaluating comparable practices.2,4 The Self-Assessment Quality Assurance program requires practitioners to self-assess their practice by completing the quality assessm ent question naire (the SAQA instrum ent), performing chart audits on about 12 selected patient records and asking eight patients to complete a patient satisfaction questionnaire. Like the DEMCAD instrum ent, the SAQA instru m en t is organized around the structure, process and outcome dimensions of dental and medical practice. The structure evaluation relates to a review of dental facilities, equipm ent, organiza tion, adm inistration and personnel. The process evaluation refers to w hat the dentist does in providing care. This includes the completeness of documentation, the tim eliness and appropriateness of care delivered and the technical skill of treatm ent. Outcomes evaluations are concerned with the health sta tu s of patients who have received treatm en t and consider disability 624
JADA, Vol. 125, May 1994
(reduction in a patient’s activity caused by acute or chronic dental conditions); social functioning (limits on patien t’s self-image caused by oral conditions); satisfaction (the reaction of the p atien t to overall dental care experiences), (Table 1 ).
W ith defined criteria, the SAQA instru m en t asks dentists to evaluate w hether significant change, some change or no changes are required for 120 item s related to practice structure and 65 item s related to practice process. The patient
The instrum ent benefits dental practitioners by providing a non threatening quality assess m ent tool that practi tioners can adm inister themselves. evaluation portion of the program requires dentists to evaluate restorations and p atien t oral hygiene. A patient questionnaire is included as p a rt of the SAQA program. The questionnaire asks patients to indicate th eir satisfaction w ith care received. O ffice recru itm en t. General dentists practicing in New Jersey were recruited to pilot-test the SAQA program. New Jersey D ental Association m em bers who are general dentists were divided into four age cohorts: practitioners who graduated before 1965, between 1965 and 1975, between 1975 and 1984, and after 1984. From each of the first three cohorts, questionnaires were sent to a random selection of 160 practi tioners. As m ost practitioners
who graduated betw een 1985 and 1990 would be either associates or in a postgraduate program , the 1985-90 cohort was not solicited. A lette r of introduction was drafted along w ith two question naires. The first questionnaire, titled W illingness to Participate, asked practitioners about th eir gender, practice setting, year of graduation, dental school, associateship/ partnership status, practice staffing and willingness to participate. The second, Perceptions of Practice Quality, asked practitioners to rate the level of im provem ents they believed were required for 16 components of their practices. The two questionnaires were adm inistered separately. As the W illingness to Participate Survey required the dentist to respond by nam e, we thought th a t dentists would be reluctant to indicate w hat areas of their practice needed improvement. In completing the Perceptions of Practice Quality questionnaire, dentists were not required to provide th eir name; however, they were asked to indicate if they were volunteer ing to participate. Collecting this willingness inform ation allowed the researchers to analyze differences in percep tions of practice quality based on willingness to participate. Letters and questionnaires were sent to the 480 practitioners, 160 practitioners in each cohort. A random sample of non respondents, individuals who had not retu rn ed the surveys, were contacted by telephone. Thus, survey respondents and non-respondents could be compared. P e e r-e v a lu a to r an d se lf a d m in ister e d a ssessm en t. To
PRACTICE M ANAGEM ENT
TA B LE
1
appointm ent. In most instances, th e peer-evaluator arranged w ith E q u ip m e n t the volunteer S te riliz a tio n /d isin fe c ta n t m a te ria ls practitioner to R adiog raph e q u ip m e n t arrive about 15 O p e ra to ry I n s tru m e n t/m a te r ia ls av ailab le m inutes before T re a tm e n t su p p o rt the first p atien t Office su p p o rt was scheduled. P a tie n t su p p o rt D uring this E n v iro n m e n t tim e, th e peerPersonnel evaluator introduced him self to the volunteer practitioner and office staff R a diog rap hic evaluation and reviewed A d m in istratio n the day’s R adiographic tech n iq u e activities. D iagnostic v alue/coverage Every effort was m ade to D iagnosis minimize disruptions to T re a tm e n t p la n norm al office T rea tm en t activities. On E ndodontic average, the Periodontic peer evaluation R e sto ra tiv e took five hours to complete. Before leaving, the evaluator left a copy of the SAQA P a tie n t education in stru m en t and instructions w ith th e volunteer dentist. At photocopies of p atien t records, p a tie n t radiographs and 35-mm no tim e during or after the site slides of p atien t dentitions. visit did th e peer-evaluator D entists volunteering to share results of th eir findings participate were scheduled for a w ith th e volunteer dentist. site visit w hen the peerA fter the site visit, the volun evaluator would complete the teer dentist completed the SAQA program . About three SAQA in stru m en t and returned weeks before th e site visit, a it to the U niversity of Medicine docum ent was m ailed and Dentistry-N ew Jersey describing how to prepare for D ental School. To determ ine the th e assessm ent. A day before validity and reliability of the th e peer-evaluator site visit, the SAQA instrum ent, we compared volunteer practitioner was resu lts of th e peer-evaluator called to confirm the assessm ent and th e self-assess-
COMPONENTS Of TOE SELF ADMINISTERED QUALITY ASSESSMENT INSTRUMENT. STRUCTURE
F acilities Office se ttin g R eception area/room B u sin e ss area/office R adiog raph a re a T r e a tm e n t rooms S u p p o rt roo m s/area Traffic flow
A d m inistratio n, P a t ie n t records A d m in istra tio n p a tie n t care su p p o rt A d m in istra tio n protocols or guides PROCESS
Practice m a na gem ent R e c e p tio n ist/a p p o in tm e n t control A p p o in tm en t book P e rso n n e l m a n a g e m e n t S te riliza tio n / infection control Staff* a n d p a tie n t pro tectio n S u rface disinfection D isposables I n s tr u m e n t ste riliz a tio n P ro sth o d o n tics a n d o rth o dontic app liance D a ta collection C o m pleteness of records O rg a n iz a tio n of p a tie n t records Legibility of records O UTCOM ES
P a tie n t satisfaction P a tie n t oral hygiene
determ ine th e reproducibility, accuracy and practicality of the SAQA in strum ent, a com parison of self-adm inistered evaluation results to a “gold sta n d a rd ” w as required. Two practitioners were recruited to serve as peer-evaluators. One h ad served as an evaluator in th e DEMCAD project. In four weeks, both exam iners were tra in e d and calibrated in use of th e SAQA instrum ent. They w ere tra in ed through 35-mm slides of dental offices, 626
JADA, Vol. 125, May 1994
PRACTICE M ANAG EM ENT
m ent. Overall, the SAQA in stru m e n t proved valid and reliable.' F in a l q u e stio n n a ir e . After we received th e self adm inistered instrum ent, the final questionnaire was m ailed to th e volunteer dentist w ith a $200 continuing education voucher. The final question naire allowed the volunteer dentists to assess th e peerevaluator visit, perceptions of th e SAQA in strum ent, tim e required to complete the self adm inistered SAQA in stru m en t and suggestions for improving the instrum ent. The la st page of th e questionnaire contained the sam e questions in the Perceptions of Practice Quality survey. P ractitioners were asked to ra te the level of
im provem ents required for 16 practice components so th a t pre- and post-study participation responses could be compared. V olunteer dentists were instructed to complete and re tu rn the questionnaire. RESULTS
O ffice re cr u itm e n t. Of the 480 Perceptions of Practice Q uality questionnaires sent to practitioners asking about th eir willingness to participate in the program , 30 indicated th eir willingness to participate and 68 th a t they were unwilling. The 98 responses am ounted to a 20 percent response ra te and a 6.25 percent willingness to participate. Sites of 24 practices were visited, and 22 dentists returned th eir self-adm inis
tered SAQA instrum ent. Eighty-three percent of th e participating dentists were m ale, 77 percent were sole proprietors and 83 percent were in suburban locations. Of the 384 practitioners not retu rn in g a survey, we attem pted to phone a random group of 146. Twenty-two were successfully contacted and completed the W illingness to P articipate and Perceptions of Practice Q uality surveys. Analysis of characteristics of questionnaire respondents and non-respondents showed no differences. Table 2 reports th e m ean level of im provem ent reported for each practice component. Practitioners rated, on a scale from 1 to 4 (1 = no im provem ent
TA B LE 2 P O
1 1 1
1—1— 1
1 —1 —1
COMPONENT
IF PRACTICE I M T ft
W IL L IN G T O P A R T IC IP A T E (N = 3 0 )
U N W IL L IN G T O P A R T IC IP A T E (N = 6 8 )
NON RESPONDENT D E N T IS T S (N = 2 1 )
P -V A L U E ANOVA
1.70 1.75 1.73 2.00
1.72 1.68 1.76 1.92
1.48 1.43 1.62 2.00
0.469 0.233 0.745 0.636
2.03 1.65
1.82 1.57
1.90 1.48
0.391 0.603
1.53
1.41
1.29
0.315
1.67 1.76 1.55
1.67 1.69 1.32
1.62 1.52 1.47
0.945 0.484 0.184
1.80 1.77 1.90 2.14
1.67 1.81 1.52 1.73 1.96
1.75 1.95 1.50 1.55 1.60
0.667 0.627 0.877 0.334 0.063
2.31
2.16
1.81
0.075
STRUCTURE
F a c ilitie s E q u ip m e n t P e rso n n e l A d m in istra tio n PROCESS
P ra c tic e m a n a g e m e n t S terilization /infection control R adiographic e v a lu a tio n D a ta collection T r e a tm e n t p la n P a tie n t m a n a g e m e n t OUTCO M ES
P a tie n t ed u cation P a tie n t o ral hygiene P a tie n t d isab ility P e rio d o n ta l disease C om pletion of* t r e a tm e n t Recall
1 .4 5
t 1 = No im provem ent required, 2 = m inor im provem ent required, 3 := m oderate im provem ent required, 4 = m ajor im provem ent required.
628
JADA, Vol. 125, May 1994
■PRACTICE M ANAG EM ENT
TA B LE
3
MEAN PERCEPTIONS Of PRACTICE QUALITY; PRE- VS. POSTEVALUATION COMPARISON. COMPONENT
MEAN P R E -S IT E V IS IT (N = 2 0 )
MEAN POST S IT E V IS IT (N = 1 8 )
NO. D E N T IS T S T H IN K W ORSE
NO. D E N T IS T S NO CHANGE
NO. D E N T IS T S T H IN K BETTER
1.90 1.95 1.70 2.00
2.00 1.50 1.61 2.11
8 1 2 7
3 8 9 4
4 6 4 4
2.05 1.75
1.78 1.83
1 5
9 5
4 4
1.70
1.67
4
6
5
1.70 1.75 1.70
1.83 1.78 2.50
5 4 4
7 8 7
2 3 3
1.75 1.70 1.42 1.90 2.21
1.39 1.41 1.17 1.71 2.06
3 4 1 2 3
9 6 IO 9 5
3 4 3 3 6
2.32
1.94
1
9
5
STRUCTURE
F a c ilitie s E q u ip m e n t P e rso n n e l A d m in istra tio n PROCESS
P ra c tic e m a n a g e m e n t S terilizatio n /in fectio n control R adiog raph ic e v a lu a tio n D a ta collection T r e a tm e n t p lan P a t ie n t m a n a g e m e n t OUTCO M ES
P a tie n t ed u cation P a tie n t o ra l hygiene P a t ie n t d isab ility P e rio d o n ta l disease C om pletion of tr e a tm e n t R ecall
t 1=No improvement required, 2=minor improvement required, 3 =moderateimprovement required, 4=major improvement required.
required, 2 = m inor improve m ent required, 3 = m oderate im provem ent required, and 4 = m ajor im provem ent required), th e level of im provem ent required for 16 components of dental practice. Overall, the greatest am ount of required im provem ent focused on practice m anagem ent. Most practitioners, however, thought th eir practices required little overall improvem ent. A reas requiring m ost im provem ent include patient recall, completion of tre a tm e n t and adm inistration. Com ponents requiring least im provem ent include patient m anagem ent, p atien t disability, radiographic evaluation and infection control. ANOVA revealed no significant 630
JADA, Vol. 125, May 1994
differences betw een willing and unw illing participants a t the 0.05 level of significance. A comparison of respondents to non-respondents also revealed no significant differences. F in a l q u e stio n n a ir e s o f v o lu n tee r p r a c titio n e r s. A questionnaire was provided to each d entist whose office was visited to g a th e r inform ation on th eir reaction to the peerevaluator visit and the self adm inistered assessm ent instrum ent. E ighteen dentists responded to th is questionnaire. The practitioner’s average tim e to complete th e self adm inistered assessm ent in stru m en t was 2.05 hours. The dentists had a positive reaction to the visits: 56 percent indicated it was “a stim ulating,
interesting experience” and an additional 39 percent reported a “generally positive experience.” Only 6 percent (one office) thought it was a “generally negative experience.” None thought it was “very traum atic.” W hen asked to note the results of answ ering th e self adm inistered assessm ent in strum ent, one office (6 percent) responded th a t the staff discovered “nothing they didn’t know before” completing the assessm ent. The rest indicated th a t they discovered either a few (65 percent) or some (29 percent) deficiencies. P e r c e p tio n s o f v o lu n tee r p r a c tic e q u a lity —b efore and a fte r th e se lf-a sse ssm e n ts. The Perceptions of Practice Q uality survey and the Final
«PRACTICE M ANAGEM ENT
Q uestionnaire asked volunteer den tists to indicate w hether 16 practice components required: no, m inor, m oderate or m ajor im provem ent. Twenty dentists who participated in the study signed th e Perceptions of Practice Q uality survey and 18 participating dentists returned th e final questionnaire. These two data sets allowed us to m atch and compare 15 pre- and post-study responses. We could th en determ ine if d e n tists’ perception of quality in th e ir office improved, stayed the sam e or declined (Table 3). The m ean response for each com ponent was calculated for th e pre- and post-participation surveys. For the 20 dentists com pleting the pre-partici pation survey, the m ean perception of quality for all com ponents ranged from a low of 1.42 for p atien t disability (outcomes) to a high of 2.32 for p a tie n t recall. Four responses had m eans equal to or greater th a n 2.00 (minor im provem ents required): adm inistration, practice m anagem ent, comple tion of treatm en t, recall. O ther th a n th e lowest m ean for p a tie n t disability, the rem aining component m eans w ere 1.70 to 1.95. For post participation surveys, m eans ranged from a low of 1.17 to a high of 2.50. Four responses had m eans equal to or greater th a n 2.00: facilities, adm ini stration, p atien t m anagem ent, completion of treatm ent. For six components, more th a n h a lf of th e dentists reported a change in the perception of quality in th eir office after participation. For th e practice structure assess m ent, th e components of facilities and adm inistration shifted to a more negative 632
JADA, Vol. 125, May 1994
perception. Eight dentists reported more im provem ents were required in facilities, and seven reported more improve m ent needed in adm inistration. Four dentists perceived these sam e components as requiring less im provem ent th a n they had reported. For facility percep tion, three dentists noted no change; four noted no change needed in adm inistration. Two process components resulted in a shift by more th an h a lf of the dentists. For sterilization/infection control, five dentists reported more im provem ent required in the
A large first step toward ensuring the public of a d en tist’s competency is the im plem entation of a voluntary quality assurance program such as the one developed. post-survey, five rem ained unchanged and four reported fewer im provem ents required. Sim ilarly, the radiographic evaluation component results show four dentists m aking a negative shift to more improve m ents, six unchanged and five to less im provem ent required. W ith the outcome compo n ents of patien t oral hygiene and completion of treatm ent, more th an h alf of the dentists noted a change in the am ount of im provem ent required. Although six dentists reported no change in the perception of p a tie n t oral hygiene, four noted more im provem ent and four indicated less improvement. A change in completion of tre a tm e n t perceptions was noted, as six dentists indicated
less im provem ent required th a n they had reported in the pre participation survey, three reported more im provem ent required and five rem ained unchanged. D IS C U S S IO N
As shown w ith previous studies, m ail solicitation to comm unity practitioners can recruit dentists to participate in quality assessm ent program s.5 The 6 percent willingness to participate ra te raises questions about practitioners’ commit m ent to participate in voluntary quality assurance program s. H ad the study not involved an in-office peer-evaluation, a higher positive response rate would m ost likely have been achieved. The instru m en t’s usefulness can be attested by the num ber of tim es dentists’ perceptions of practice quality changed. Comparisons of m ean responses from m atched groups of preand post-participation surveys indicated changes in volunteer dentists’ perceptions of practice quality. For all outcome components, the m ean response decreased. This decrease indicates the dentists thought th eir offices required less Dr. Saporito is associate dean, Clinical Affairs-D746; and acting chair, Department of General and Hospital Dentistry, New Jersey Dental
Dr. Feldman is
School, University of
director, Information
Medicine and
Services and Quality
Dentistry of New
Assurance; and
Jersey, 110 Bergen
associate professor,
St., Newark, N.J.
General and Hospital
07103. Address
Dentistry» New
reprint requests to
Jersey Dental
Dr. Saporito.
School.
PRACTICE M ANAG EM ENT)
Dr. Stewart is
Dr. Buchanan is
director. Quality
dean and professor,
Assurance; and
General and Hospital
assistant professor,
Dentistry, New
Dental Care
Jersey Dental
Systems, School of
School, University of
Dental Medicine,
Medicine and
University of
Dentistry of New
Pennsylvania.
Jersey.
im provem ent in this component before th eir participation. This shift could m ean th a t the criteria provided in the quality assessm ent in stru m en t were not clear enough or th a t the d entist had standards higher th a n those defined by the SAQA instrum ent. Shifts tow ard less improve m ent needed—as seen in the m eans for equipm ent, person nel, practice m anagem ent and radiographic evaluation—m ay also reflect th e delegation of duties associated w ith these components and reflect a change in knowledge level of w hat is actually going on in the d entist’s office. We noted positive shifts in m ean, indicating a need for more improvem ent, for facilities and adm inistration, steriliza tion/infection control, data collection, tre a tm e n t plan and p atien t m anagem ent. This shift in sterilization and infection
control is particularly interesting. W ith the recent, rapid changes in this Dr. Echoldt is aspect of research evaluator, dental practice New Jersey Medical School, University of (largely driven Medicine and by OSHA) it Dentistry of New would have Jersey. been expected th a t this would be the area w ith the highest aw areness. Again, perhaps the dentist has delegated these duties or m ade assum ptions about the level a t which office staff are perform ing these duties. C O N C L U S IO N S
This study indicates th a t use of an assessm ent instrum ent, such as SAQA, which requires practitioners to com prehen sively review his or h er practice can lead to a b e tte r under standing of a practice’s strengths and weaknesses. A large first step tow ard ensuring th e public of a dentist’s continuing competency is the im plem entation of a voluntary quality assurance program such as the one developed. A self adm inistered assessm ent m echanism is ideal for m ost practitioners as assessm ents can be performed a t a convenient tim e, in a non
th reaten in g environm ent and for little cost. As th e DEMCAD follow-up study suggested th a t dentists can modify th eir practice behavior as a resu lt of a quality assessm ent program ,4 program s like the SAQA in stru m en t can provide dentists w ith the necessary tools to assess and ultim ately improve th eir practice. ■ This project was supported by th e Am erican Fund for D ental H ealth. The evaluation in stru m en t w as modeled after th e assessm ent in stru m en t developed as p a rt of th e D ental E valuation M ethods and C om puter Applications in D entistry project funded by the W.K. Kellogg Foundation. The au th o rs th a n k the New Je rsey D ental Association for th eir assistance in identifying New Jersey practitioners; Drs. Ja m es Palm issano an d Anthony Vito, th e peerevaluators, for th e ir tim e, effort, evaluation skills and persistence; Ms. Michelle Fortunato for reviewing an d editing the in stru m en t drafts; and th e New Jersey practitioners who volunteered th e ir tim e and energy to participate in th is project. 1. Feldm an CA, Saporito RA, S tew art DCL, Buchanan RN. Developm ent and validation of a self-assessm ent quality assurance instrum ent. F inal report for th e Am erican Fund for D ental H ealth; 1992. 2. Morris AL, Bentley JM , Vito AA, Bomba MR. A ssessm ent of private den tal practice: report of study. JADA 1988;117(6):153-62. 3. Morris AL, K ephart BE, B ailit HL, Vito AA. A ssessm ent of dental practice. J Am Coll Dent 1982;49:53-9. 4. Morris AL, Vito AA, Bomba MR, Bentley JM . The im pact of a quality assessm ent program on the practice behavior of general practitioners: a follow-up study. JADA 1989;119(12):705-9. 5. Morris AL, Bentley JM , Bomba MR. Response of general practitioners to a national dental office evaluation program . JADA 1985;111(11):799-801.
JADA, Vol. 125, May 1994
633
O
V I D
I N
G
P R
A C
T I C
E
A
S
S
E S
S
M
E N
T
R O B E R T A . S A P O R IT O , D .D .S .; C E C IL E A . F E L D M A N , D .M .D ., M .B .A .; D E N IC E C .l— S T E W A R T , D .D .S ., M .H .S .A .; H A F T O N E C H O L D T , P H .D .; R IC H A R D N. B U C H A N A N , D .M .D .
A
B
S
T
R
A
C
T
Participation in a selfadministered quality assessment (SAQA) program led to changes in New Jersey dentists’ perceptions of practice quality. Ninety-four percent indicated they discovered practice deficiencies. This study suggests that using a self administered quality assessment program, such as the SAQA program, can lead to a better understanding of a practice’s strengths and weaknesses.
his paper reports changes in dentists’ perceptions of practice quality before and after participation in a self-adm inistered quality assessm ent program . The quality assessm ent program was developed as p a rt of an 18-month project funded by th e Am erican Fund for D ental H ealth. The goal of the project, entitled “Development and V alidation of a Self-Assessment Q uality A ssurance Program ,” was to develop a self adm inistered quality assessm ent in stru m en t th a t can be used in a general d entist’s in-office quality assurance program .1The in stru m en t benefits dental practitioners by providing a non threaten in g quality assessm ent tool th a t practitioners can adm inister them selves. Such assessm ents are inexpensive and can have a significant im pact on the quality of care. D entists volunteered to participate in th e pilot study. Participation involved: ■■ answ ering a pre-assessm ent questionnaire regarding willingness to participate and perceived practice quality; *■ having a “peer-evaluator” dentist assess the volunteer’s office; ■■ perform ing a quality assessm ent by completing th e self adm inistered quality assessm ent questionnaire; ■* completing the post-assessm ent questionnaire regarding strengths and w eakness of the program and changes in th eir perception of practice quality. A comparison of th e dentists’ perception of practice quality before and after participation in the quality assessm ent program dem onstrates the in stru m en t’s ability to increase the den tists’ aw areness of quality im provem ent needs. M A T E R IA L S A N D M E T H O D S
In stru m en t d evelop m en t. The in stru m en t developed by the Development of Evaluation M ethods and Com puter Applications in D entistry project funded by the Kellogg Foundation was selected as 622
JADA, Vol. 125, May 1994
PRACTICE M ANAG EM ENT
the basis for the program .2 The DEMCAD instru m en t was selected because it has a scientific basis and provides a comprehensive evaluation of dental office structure, process and outcomes.2,3 The DEMCAD instru m en t distinguishes among dental practices th a t differ in th eir characteristics.2 The instru m en t also effectively evaluated rural, urban group and urban non-group practices located in all U.S. geographic regions.2 D entist peerevaluators can be trained to use the instrum ent in a standardized approach th a t produces comparable results when evaluating comparable practices.2,4 The Self-Assessment Quality Assurance program requires practitioners to self-assess their practice by completing the quality assessm ent question naire (the SAQA instrum ent), performing chart audits on about 12 selected patient records and asking eight patients to complete a patient satisfaction questionnaire. Like the DEMCAD instrum ent, the SAQA instru m en t is organized around the structure, process and outcome dimensions of dental and medical practice. The structure evaluation relates to a review of dental facilities, equipm ent, organiza tion, adm inistration and personnel. The process evaluation refers to w hat the dentist does in providing care. This includes the completeness of documentation, the tim eliness and appropriateness of care delivered and the technical skill of treatm ent. Outcomes evaluations are concerned with the health sta tu s of patients who have received treatm en t and consider disability 624
JADA, Vol. 125, May 1994
(reduction in a patient’s activity caused by acute or chronic dental conditions); social functioning (limits on patien t’s self-image caused by oral conditions); satisfaction (the reaction of the p atien t to overall dental care experiences), (Table 1 ).
W ith defined criteria, the SAQA instru m en t asks dentists to evaluate w hether significant change, some change or no changes are required for 120 item s related to practice structure and 65 item s related to practice process. The patient
The instrum ent benefits dental practitioners by providing a non threatening quality assess m ent tool that practi tioners can adm inister themselves. evaluation portion of the program requires dentists to evaluate restorations and p atien t oral hygiene. A patient questionnaire is included as p a rt of the SAQA program. The questionnaire asks patients to indicate th eir satisfaction w ith care received. O ffice recru itm en t. General dentists practicing in New Jersey were recruited to pilot-test the SAQA program. New Jersey D ental Association m em bers who are general dentists were divided into four age cohorts: practitioners who graduated before 1965, between 1965 and 1975, between 1975 and 1984, and after 1984. From each of the first three cohorts, questionnaires were sent to a random selection of 160 practi tioners. As m ost practitioners
who graduated betw een 1985 and 1990 would be either associates or in a postgraduate program , the 1985-90 cohort was not solicited. A lette r of introduction was drafted along w ith two question naires. The first questionnaire, titled W illingness to Participate, asked practitioners about th eir gender, practice setting, year of graduation, dental school, associateship/ partnership status, practice staffing and willingness to participate. The second, Perceptions of Practice Quality, asked practitioners to rate the level of im provem ents they believed were required for 16 components of their practices. The two questionnaires were adm inistered separately. As the W illingness to Participate Survey required the dentist to respond by nam e, we thought th a t dentists would be reluctant to indicate w hat areas of their practice needed improvement. In completing the Perceptions of Practice Quality questionnaire, dentists were not required to provide th eir name; however, they were asked to indicate if they were volunteer ing to participate. Collecting this willingness inform ation allowed the researchers to analyze differences in percep tions of practice quality based on willingness to participate. Letters and questionnaires were sent to the 480 practitioners, 160 practitioners in each cohort. A random sample of non respondents, individuals who had not retu rn ed the surveys, were contacted by telephone. Thus, survey respondents and non-respondents could be compared. P e e r-e v a lu a to r an d se lf a d m in ister e d a ssessm en t. To
PRACTICE M ANAGEM ENT
TA B LE
1
appointm ent. In most instances, th e peer-evaluator arranged w ith E q u ip m e n t the volunteer S te riliz a tio n /d isin fe c ta n t m a te ria ls practitioner to R adiog raph e q u ip m e n t arrive about 15 O p e ra to ry I n s tru m e n t/m a te r ia ls av ailab le m inutes before T re a tm e n t su p p o rt the first p atien t Office su p p o rt was scheduled. P a tie n t su p p o rt D uring this E n v iro n m e n t tim e, th e peerPersonnel evaluator introduced him self to the volunteer practitioner and office staff R a diog rap hic evaluation and reviewed A d m in istratio n the day’s R adiographic tech n iq u e activities. D iagnostic v alue/coverage Every effort was m ade to D iagnosis minimize disruptions to T re a tm e n t p la n norm al office T rea tm en t activities. On E ndodontic average, the Periodontic peer evaluation R e sto ra tiv e took five hours to complete. Before leaving, the evaluator left a copy of the SAQA P a tie n t education in stru m en t and instructions w ith th e volunteer dentist. At photocopies of p atien t records, p a tie n t radiographs and 35-mm no tim e during or after the site slides of p atien t dentitions. visit did th e peer-evaluator D entists volunteering to share results of th eir findings participate were scheduled for a w ith th e volunteer dentist. site visit w hen the peerA fter the site visit, the volun evaluator would complete the teer dentist completed the SAQA program . About three SAQA in stru m en t and returned weeks before th e site visit, a it to the U niversity of Medicine docum ent was m ailed and Dentistry-N ew Jersey describing how to prepare for D ental School. To determ ine the th e assessm ent. A day before validity and reliability of the th e peer-evaluator site visit, the SAQA instrum ent, we compared volunteer practitioner was resu lts of th e peer-evaluator called to confirm the assessm ent and th e self-assess-
COMPONENTS Of TOE SELF ADMINISTERED QUALITY ASSESSMENT INSTRUMENT. STRUCTURE
F acilities Office se ttin g R eception area/room B u sin e ss area/office R adiog raph a re a T r e a tm e n t rooms S u p p o rt roo m s/area Traffic flow
A d m inistratio n, P a t ie n t records A d m in istra tio n p a tie n t care su p p o rt A d m in istra tio n protocols or guides PROCESS
Practice m a na gem ent R e c e p tio n ist/a p p o in tm e n t control A p p o in tm en t book P e rso n n e l m a n a g e m e n t S te riliza tio n / infection control Staff* a n d p a tie n t pro tectio n S u rface disinfection D isposables I n s tr u m e n t ste riliz a tio n P ro sth o d o n tics a n d o rth o dontic app liance D a ta collection C o m pleteness of records O rg a n iz a tio n of p a tie n t records Legibility of records O UTCOM ES
P a tie n t satisfaction P a tie n t oral hygiene
determ ine th e reproducibility, accuracy and practicality of the SAQA in strum ent, a com parison of self-adm inistered evaluation results to a “gold sta n d a rd ” w as required. Two practitioners were recruited to serve as peer-evaluators. One h ad served as an evaluator in th e DEMCAD project. In four weeks, both exam iners were tra in e d and calibrated in use of th e SAQA instrum ent. They w ere tra in ed through 35-mm slides of dental offices, 626
JADA, Vol. 125, May 1994
PRACTICE M ANAG EM ENT
m ent. Overall, the SAQA in stru m e n t proved valid and reliable.' F in a l q u e stio n n a ir e . After we received th e self adm inistered instrum ent, the final questionnaire was m ailed to th e volunteer dentist w ith a $200 continuing education voucher. The final question naire allowed the volunteer dentists to assess th e peerevaluator visit, perceptions of th e SAQA in strum ent, tim e required to complete the self adm inistered SAQA in stru m en t and suggestions for improving the instrum ent. The la st page of th e questionnaire contained the sam e questions in the Perceptions of Practice Quality survey. P ractitioners were asked to ra te the level of
im provem ents required for 16 practice components so th a t pre- and post-study participation responses could be compared. V olunteer dentists were instructed to complete and re tu rn the questionnaire. RESULTS
O ffice re cr u itm e n t. Of the 480 Perceptions of Practice Q uality questionnaires sent to practitioners asking about th eir willingness to participate in the program , 30 indicated th eir willingness to participate and 68 th a t they were unwilling. The 98 responses am ounted to a 20 percent response ra te and a 6.25 percent willingness to participate. Sites of 24 practices were visited, and 22 dentists returned th eir self-adm inis
tered SAQA instrum ent. Eighty-three percent of th e participating dentists were m ale, 77 percent were sole proprietors and 83 percent were in suburban locations. Of the 384 practitioners not retu rn in g a survey, we attem pted to phone a random group of 146. Twenty-two were successfully contacted and completed the W illingness to P articipate and Perceptions of Practice Q uality surveys. Analysis of characteristics of questionnaire respondents and non-respondents showed no differences. Table 2 reports th e m ean level of im provem ent reported for each practice component. Practitioners rated, on a scale from 1 to 4 (1 = no im provem ent
TA B LE 2 P O
1 1 1
1—1— 1
1 —1 —1
COMPONENT
IF PRACTICE I M T ft
W IL L IN G T O P A R T IC IP A T E (N = 3 0 )
U N W IL L IN G T O P A R T IC IP A T E (N = 6 8 )
NON RESPONDENT D E N T IS T S (N = 2 1 )
P -V A L U E ANOVA
1.70 1.75 1.73 2.00
1.72 1.68 1.76 1.92
1.48 1.43 1.62 2.00
0.469 0.233 0.745 0.636
2.03 1.65
1.82 1.57
1.90 1.48
0.391 0.603
1.53
1.41
1.29
0.315
1.67 1.76 1.55
1.67 1.69 1.32
1.62 1.52 1.47
0.945 0.484 0.184
1.80 1.77 1.90 2.14
1.67 1.81 1.52 1.73 1.96
1.75 1.95 1.50 1.55 1.60
0.667 0.627 0.877 0.334 0.063
2.31
2.16
1.81
0.075
STRUCTURE
F a c ilitie s E q u ip m e n t P e rso n n e l A d m in istra tio n PROCESS
P ra c tic e m a n a g e m e n t S terilization /infection control R adiographic e v a lu a tio n D a ta collection T r e a tm e n t p la n P a tie n t m a n a g e m e n t OUTCO M ES
P a tie n t ed u cation P a tie n t o ral hygiene P a tie n t d isab ility P e rio d o n ta l disease C om pletion of* t r e a tm e n t Recall
1 .4 5
t 1 = No im provem ent required, 2 = m inor im provem ent required, 3 := m oderate im provem ent required, 4 = m ajor im provem ent required.
628
JADA, Vol. 125, May 1994
■PRACTICE M ANAG EM ENT
TA B LE
3
MEAN PERCEPTIONS Of PRACTICE QUALITY; PRE- VS. POSTEVALUATION COMPARISON. COMPONENT
MEAN P R E -S IT E V IS IT (N = 2 0 )
MEAN POST S IT E V IS IT (N = 1 8 )
NO. D E N T IS T S T H IN K W ORSE
NO. D E N T IS T S NO CHANGE
NO. D E N T IS T S T H IN K BETTER
1.90 1.95 1.70 2.00
2.00 1.50 1.61 2.11
8 1 2 7
3 8 9 4
4 6 4 4
2.05 1.75
1.78 1.83
1 5
9 5
4 4
1.70
1.67
4
6
5
1.70 1.75 1.70
1.83 1.78 2.50
5 4 4
7 8 7
2 3 3
1.75 1.70 1.42 1.90 2.21
1.39 1.41 1.17 1.71 2.06
3 4 1 2 3
9 6 IO 9 5
3 4 3 3 6
2.32
1.94
1
9
5
STRUCTURE
F a c ilitie s E q u ip m e n t P e rso n n e l A d m in istra tio n PROCESS
P ra c tic e m a n a g e m e n t S terilizatio n /in fectio n control R adiog raph ic e v a lu a tio n D a ta collection T r e a tm e n t p lan P a t ie n t m a n a g e m e n t OUTCO M ES
P a tie n t ed u cation P a tie n t o ra l hygiene P a t ie n t d isab ility P e rio d o n ta l disease C om pletion of tr e a tm e n t R ecall
t 1=No improvement required, 2=minor improvement required, 3 =moderateimprovement required, 4=major improvement required.
required, 2 = m inor improve m ent required, 3 = m oderate im provem ent required, and 4 = m ajor im provem ent required), th e level of im provem ent required for 16 components of dental practice. Overall, the greatest am ount of required im provem ent focused on practice m anagem ent. Most practitioners, however, thought th eir practices required little overall improvem ent. A reas requiring m ost im provem ent include patient recall, completion of tre a tm e n t and adm inistration. Com ponents requiring least im provem ent include patient m anagem ent, p atien t disability, radiographic evaluation and infection control. ANOVA revealed no significant 630
JADA, Vol. 125, May 1994
differences betw een willing and unw illing participants a t the 0.05 level of significance. A comparison of respondents to non-respondents also revealed no significant differences. F in a l q u e stio n n a ir e s o f v o lu n tee r p r a c titio n e r s. A questionnaire was provided to each d entist whose office was visited to g a th e r inform ation on th eir reaction to the peerevaluator visit and the self adm inistered assessm ent instrum ent. E ighteen dentists responded to th is questionnaire. The practitioner’s average tim e to complete th e self adm inistered assessm ent in stru m en t was 2.05 hours. The dentists had a positive reaction to the visits: 56 percent indicated it was “a stim ulating,
interesting experience” and an additional 39 percent reported a “generally positive experience.” Only 6 percent (one office) thought it was a “generally negative experience.” None thought it was “very traum atic.” W hen asked to note the results of answ ering th e self adm inistered assessm ent in strum ent, one office (6 percent) responded th a t the staff discovered “nothing they didn’t know before” completing the assessm ent. The rest indicated th a t they discovered either a few (65 percent) or some (29 percent) deficiencies. P e r c e p tio n s o f v o lu n tee r p r a c tic e q u a lity —b efore and a fte r th e se lf-a sse ssm e n ts. The Perceptions of Practice Q uality survey and the Final
«PRACTICE M ANAGEM ENT
Q uestionnaire asked volunteer den tists to indicate w hether 16 practice components required: no, m inor, m oderate or m ajor im provem ent. Twenty dentists who participated in the study signed th e Perceptions of Practice Q uality survey and 18 participating dentists returned th e final questionnaire. These two data sets allowed us to m atch and compare 15 pre- and post-study responses. We could th en determ ine if d e n tists’ perception of quality in th e ir office improved, stayed the sam e or declined (Table 3). The m ean response for each com ponent was calculated for th e pre- and post-participation surveys. For the 20 dentists com pleting the pre-partici pation survey, the m ean perception of quality for all com ponents ranged from a low of 1.42 for p atien t disability (outcomes) to a high of 2.32 for p a tie n t recall. Four responses had m eans equal to or greater th a n 2.00 (minor im provem ents required): adm inistration, practice m anagem ent, comple tion of treatm en t, recall. O ther th a n th e lowest m ean for p a tie n t disability, the rem aining component m eans w ere 1.70 to 1.95. For post participation surveys, m eans ranged from a low of 1.17 to a high of 2.50. Four responses had m eans equal to or greater th a n 2.00: facilities, adm ini stration, p atien t m anagem ent, completion of treatm ent. For six components, more th a n h a lf of th e dentists reported a change in the perception of quality in th eir office after participation. For th e practice structure assess m ent, th e components of facilities and adm inistration shifted to a more negative 632
JADA, Vol. 125, May 1994
perception. Eight dentists reported more im provem ents were required in facilities, and seven reported more improve m ent needed in adm inistration. Four dentists perceived these sam e components as requiring less im provem ent th a n they had reported. For facility percep tion, three dentists noted no change; four noted no change needed in adm inistration. Two process components resulted in a shift by more th an h a lf of the dentists. For sterilization/infection control, five dentists reported more im provem ent required in the
A large first step toward ensuring the public of a d en tist’s competency is the im plem entation of a voluntary quality assurance program such as the one developed. post-survey, five rem ained unchanged and four reported fewer im provem ents required. Sim ilarly, the radiographic evaluation component results show four dentists m aking a negative shift to more improve m ents, six unchanged and five to less im provem ent required. W ith the outcome compo n ents of patien t oral hygiene and completion of treatm ent, more th an h alf of the dentists noted a change in the am ount of im provem ent required. Although six dentists reported no change in the perception of p a tie n t oral hygiene, four noted more im provem ent and four indicated less improvement. A change in completion of tre a tm e n t perceptions was noted, as six dentists indicated
less im provem ent required th a n they had reported in the pre participation survey, three reported more im provem ent required and five rem ained unchanged. D IS C U S S IO N
As shown w ith previous studies, m ail solicitation to comm unity practitioners can recruit dentists to participate in quality assessm ent program s.5 The 6 percent willingness to participate ra te raises questions about practitioners’ commit m ent to participate in voluntary quality assurance program s. H ad the study not involved an in-office peer-evaluation, a higher positive response rate would m ost likely have been achieved. The instru m en t’s usefulness can be attested by the num ber of tim es dentists’ perceptions of practice quality changed. Comparisons of m ean responses from m atched groups of preand post-participation surveys indicated changes in volunteer dentists’ perceptions of practice quality. For all outcome components, the m ean response decreased. This decrease indicates the dentists thought th eir offices required less Dr. Saporito is associate dean, Clinical Affairs-D746; and acting chair, Department of General and Hospital Dentistry, New Jersey Dental
Dr. Feldman is
School, University of
director, Information
Medicine and
Services and Quality
Dentistry of New
Assurance; and
Jersey, 110 Bergen
associate professor,
St., Newark, N.J.
General and Hospital
07103. Address
Dentistry» New
reprint requests to
Jersey Dental
Dr. Saporito.
School.
PRACTICE M ANAG EM ENT)
Dr. Stewart is
Dr. Buchanan is
director. Quality
dean and professor,
Assurance; and
General and Hospital
assistant professor,
Dentistry, New
Dental Care
Jersey Dental
Systems, School of
School, University of
Dental Medicine,
Medicine and
University of
Dentistry of New
Pennsylvania.
Jersey.
im provem ent in this component before th eir participation. This shift could m ean th a t the criteria provided in the quality assessm ent in stru m en t were not clear enough or th a t the d entist had standards higher th a n those defined by the SAQA instrum ent. Shifts tow ard less improve m ent needed—as seen in the m eans for equipm ent, person nel, practice m anagem ent and radiographic evaluation—m ay also reflect th e delegation of duties associated w ith these components and reflect a change in knowledge level of w hat is actually going on in the d entist’s office. We noted positive shifts in m ean, indicating a need for more improvem ent, for facilities and adm inistration, steriliza tion/infection control, data collection, tre a tm e n t plan and p atien t m anagem ent. This shift in sterilization and infection
control is particularly interesting. W ith the recent, rapid changes in this Dr. Echoldt is aspect of research evaluator, dental practice New Jersey Medical School, University of (largely driven Medicine and by OSHA) it Dentistry of New would have Jersey. been expected th a t this would be the area w ith the highest aw areness. Again, perhaps the dentist has delegated these duties or m ade assum ptions about the level a t which office staff are perform ing these duties. C O N C L U S IO N S
This study indicates th a t use of an assessm ent instrum ent, such as SAQA, which requires practitioners to com prehen sively review his or h er practice can lead to a b e tte r under standing of a practice’s strengths and weaknesses. A large first step tow ard ensuring th e public of a dentist’s continuing competency is the im plem entation of a voluntary quality assurance program such as the one developed. A self adm inistered assessm ent m echanism is ideal for m ost practitioners as assessm ents can be performed a t a convenient tim e, in a non
th reaten in g environm ent and for little cost. As th e DEMCAD follow-up study suggested th a t dentists can modify th eir practice behavior as a resu lt of a quality assessm ent program ,4 program s like the SAQA in stru m en t can provide dentists w ith the necessary tools to assess and ultim ately improve th eir practice. ■ This project was supported by th e Am erican Fund for D ental H ealth. The evaluation in stru m en t w as modeled after th e assessm ent in stru m en t developed as p a rt of th e D ental E valuation M ethods and C om puter Applications in D entistry project funded by the W.K. Kellogg Foundation. The au th o rs th a n k the New Je rsey D ental Association for th eir assistance in identifying New Jersey practitioners; Drs. Ja m es Palm issano an d Anthony Vito, th e peerevaluators, for th e ir tim e, effort, evaluation skills and persistence; Ms. Michelle Fortunato for reviewing an d editing the in stru m en t drafts; and th e New Jersey practitioners who volunteered th e ir tim e and energy to participate in th is project. 1. Feldm an CA, Saporito RA, S tew art DCL, Buchanan RN. Developm ent and validation of a self-assessm ent quality assurance instrum ent. F inal report for th e Am erican Fund for D ental H ealth; 1992. 2. Morris AL, Bentley JM , Vito AA, Bomba MR. A ssessm ent of private den tal practice: report of study. JADA 1988;117(6):153-62. 3. Morris AL, K ephart BE, B ailit HL, Vito AA. A ssessm ent of dental practice. J Am Coll Dent 1982;49:53-9. 4. Morris AL, Vito AA, Bomba MR, Bentley JM . The im pact of a quality assessm ent program on the practice behavior of general practitioners: a follow-up study. JADA 1989;119(12):705-9. 5. Morris AL, Bentley JM , Bomba MR. Response of general practitioners to a national dental office evaluation program . JADA 1985;111(11):799-801.
JADA, Vol. 125, May 1994
633