The relations between knowledge, search strategy, and performance in unaided and aided information search

Organizational Behavior and Human Decision Processes 90 (2003) 1–18

ORGANIZATIONAL BEHAVIOR AND HUMAN DECISION PROCESSES www.elsevier.com/locate/obhdp

The relations between knowledge, search strategy, and performance in unaided and aided information search John A. Barrick* and Brian C. Spilker School of Accountancy and Information Systems, Brigham Young University, Provo, UT, USA

Abstract Information search is a critical step in resolving complex issues in many different decisionmaking domains. This study examines the relations between knowledge, search strategy, and performance in both unaided and aided information search. The results of an experiment indicates that (a) task-relevant knowledge is directly related to performance in aided but not in unaided information search; (b) in unaided information search, search strategy mediates the relation between knowledge and performance; that is, knowledge indirectly affects performance through its effect on search strategy; and (c) an information search aid moderates both the relation between knowledge and search strategy and the relation between search strategy and performance. These findings highlight the importance and the roles of task-relevant technical knowledge and search strategy in explaining information search performance and are most likely to apply to complex, knowledge-intensive domains in which decision makers search large databases to resolve issues. Ó 2003 Elsevier Science (USA). All rights reserved. Keywords: Directed and sequential; Information search strategies; Knowledge; Search aids

1. Introduction Information search is an integral part of the decision-making process (e.g., Einhorn & Hogarth, 1981; Simon, 1977) and has been the object of study in many different decision-making contexts, including graduate school admissions (Johnson, 1988), judicial decision-making (Lawrence, 1988), financial analysis and forecasting (Hunton & McEwen, 1997), medicine (Hersh & Hickam, 1995; Hersh et al., 2000), and tax (Cloyd & Spilker, 1999). Information search performance (i.e., locating relevant information to resolve issues) and variables affecting search performance are important because information located during the search process can affect decision quality for better or worse (Lohse & Johnson, 1996). For example, Cloyd and Spilker find that when searching for information to resolve a tax issue, tax professionalsÕ performance is influenced by client preferences. They report that tax professionals tend to search for and attend to court cases with conclusions consistent with the clientÕs preferred position at the expense of attending to cases with conclusions inconsistent with the *

Corresponding author.

0749-5978/03/$ - see front matter Ó 2003 Elsevier Science (USA). All rights reserved. doi:10.1016/S0749-5978(03)00002-5

2

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

clientÕs preference. This biased search approach causes tax professionals to be overly optimistic about the support for a client-preferred tax position, causing them to make overly aggressive recommendations. Prior research indicates that knowledge is an important factor in explaining search performance in domains such as accounting, engineering, and medicine (e.g., Hersh et al., 2000; Lee, Herr, Kardes, & Kim, 1999; Spilker, 1995). This literature indicates that knowledge has a positive effect on search performance, but does not examine whether this effect is direct or whether knowledge affects performance indirectly through some other variable (e.g., search strategy). This study extends the information search literature by examining the strength of the direct and indirect relations between knowledge, search strategy, and performance. We further consider how and to what extent these relations change when decision makers are required to use an information search aid. Understanding how information search aids affect search strategy and performance is important because of the recent explosion of search technology in almost every discipline imaginable (e.g., law (LegalTrac and WestLaw), information science (ABI Inform and Lexis/Nexis), medicine (Medline), and the internet (Yahoo and Google)). We address the research issues with a computer-based experiment designed to simulate an actual tax information search task. A tax context is used because it provides a rich environment for examining information search and because tax professionals often use search aids to access large databases of information. The remainder of this paper is organized as follows. Section 2 provides background and develops hypotheses. Section 3 describes the research method. Section 4 presents the results, and Section 5 summarizes and concludes the paper.

2. Background and hypothesis development 2.1. Theoretical background Decision-making has been characterized as a multistage process in which the decision maker searches for and evaluates information before reaching a final decision (Einhorn & Hogarth, 1981; Simon, 1977). Thus the better the decision makerÕs search performance (i.e., the better the decision maker is at locating decision-relevant information), the more likely the decision maker will be to make an informed decision. Consistent with prior research, this study posits that search performance is influenced by the decision makerÕs task-relevant knowledge (Cloyd, 1997; Hersh et al., 2000; Spilker, 1995; Spink & Saracevic, 1997). This research indicates that high (low) levels of knowledge are associated with high (low) levels of performance. However, these studies are unclear about the nature of the relation. For example, it is not clear whether knowledge directly affects performance or whether knowledge affects performance through another variable that directly affects performance. This study proposes that, in addition to directly affecting search performance, knowledge affects performance indirectly through its effect on search strategy; that is, this study proposes that search strategy mediates the relation between knowledge and search performance. The ideas presented and addressed in this study are diagrammed in Fig. 1. As described in Fig. 1, knowledge is expected to influence search performance both directly (link 1) and indirectly (through links 2 and 3). 2.2. Direct effect of knowledge on performance during unaided information search (link 1) Knowledge should have a positive impact on decision makersÕ ability to locate relevant information because it allows decision makers to discriminate between

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

3

Fig. 1. Hypothesized relations between knowledge, search strategy, and search performance.

relevant and irrelevant information, thus enabling them to recognize and evaluate information as relevant when they encounter it during the search process. For example, a decision maker may need to do research to determine the rate at which gain from a sale of property is taxed. Knowledge that the gain from the sale of depreciable property is potentially taxed at a different rate than gain from the sale of nondepreciable property would facilitate the decision makerÕs ability to discriminate between relevant and irrelevant information during the search. If the gain is from the sale of depreciable property, the decision maker with knowledge can identify as relevant and attend to cues and information relating to gains from sales of depreciable property and identify as irrelevant cues and information relating to gains from sales of nondepreciable property. All else equal, a decision maker with more task-relevant knowledge should enjoy more success at identifying a given piece of information as relevant or irrelevant to the decision than should a decision maker with less knowledge (independent of the search strategy adopted). This suggests our first hypothesis as follows: Hypothesis 1. When performing unaided information search, knowledge has a positive direct effect on performance. 2.3. Effect of knowledge on search strategy during unaided information search (link 2) In describing information search strategies, the literature identifies a ‘‘directed’’ search strategy in which decision makers seek out specific decision-relevant information items and a ‘‘sequential’’ search strategy whereby decision makers examine information in the sequence in which it is presented (Biggs & Mock, 1983; Hershey, Walsh, Read, & Chulef, 1990; Johnson, 1988). Task demands inherent in many complex domains (e.g., engineering, law, medicine, and tax) suggest that knowledge should provide decision makers with access to more-directed, less-sequential search strategies. Task-relevant knowledge should help the decision maker (a) distinguish relevant from irrelevant information, (b) identify relevant issues and questions during the search by facilitating an understanding of interrelationships between and among various database cues (i.e., locating one cue may suggest another question or issue), and (c) accurately relate the

4

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

facts of the decision-making problem to potentially relevant information in the database. Thus knowledge should direct decision makers towards the most relevant information contained in the database. For example, knowledgeable decision makers searching the Internal Revenue Code to determine the rate at which gain from the sale of depreciable property is taxed, should be able to direct their search towards particular portions of the Internal Revenue Code that deal with the rate at which gain from the sale of different types of depreciable property are taxed (e.g., Secs. 1245, 1250, and 291). This will enhance the opportunity to locate relevant information. In contrast, when confronted with a complex search problem and a large database to search, less-knowledgeable researchers likely do not have the same level of understanding to guide them to areas of the database containing decisionrelevant information. Consequently, they will likely adopt a more-sequential, lessdirected search strategy, because this type of strategy provides some structure that simplifies the task by making it easier for the less-knowledgeable researcher to remember and track the information that has and has not been searched. For example, less-knowledgeable decision makers searching the Internal Revenue Code to determine the rate at which gain from the sale of depreciable property is taxed, are likely unable to initially rule out any Code Sections from consideration. Thus, they must choose how to search the database. One approach to locating relevant information would be to select Code Sections randomly. However, with a database of thousands of Code Sections, this approach is unlikely to be successful, because it is unstructured and does not allow decision makers to determine which areas of the database have and have not been previously searched. An alternative approach is to search the Code sequentially, beginning with Code Section 1 and continuing through the Code in sequence until the target information is located. This approach provides structure to the search and allows decision makers to keep track of the information they have previously searched so that they do not search the same areas of the database multiple times. Consistent with these ideas, research in judgment and decision-making suggests that more-knowledgeable decision makers should implement more-directed, lesssequential search strategies, while less-knowledgeable decision makers should implement more-sequential, less-directed search strategies (Chi, Glaser, & Farr, 1988; Yates, 1990). However, the bulk of this research involves small-sample, verbal protocol studies, and is therefore primarily descriptive in nature (e.g., Anderson, 1988; Bedard & Biggs, 1991; Bouwman, Frishkoff, & Frishkoff, 1987; Johnson, 1988). Furthermore, these descriptive studies provide inconsistent results on the issue. Results are also mixed in related hypothesis-testing studies. Hunton and McEwen (1997) find a marginally significant correlation between experience and financial analystsÕ search strategies. Bedard and Mock (1992) find that at a general search level (i.e., when choosing from among broad categories), experience is associated with less-sequential search approaches. However, when selecting from more specific, detailed information, both experienced and inexperienced decision makers search for information in a sequential manner. In summary, research examining search strategies provides inconclusive evidence on the effects of knowledge on search strategies. This may be caused by relatively low-powered tests resulting from small sample sizes or by the inherent noise created in using experience as a proxy for knowledge. This study contributes to the literature by directly testing the following hypothesis: Hypothesis 2. During unaided information search, more-knowledgeable decision makers adopt more-directed, less-sequential search strategies than do less-knowledgeable decision makers.

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

5

2.4. Effect of search strategy on performance during unaided information search (link 3) When searching a large database of information, more-directed, less-sequential search strategies should be associated with a greater opportunity to locate relevant information than should less-directed, more-sequential search strategies (Hunton & McEwen, 1997). A directed search focuses on specific areas of the database deemed most likely to contain relevant information. In contrast, a sequential search may or may not eventually guide the decision maker to relevant information, depending on where the relevant information is located in the database. When time and resource constraints prevent the decision maker from conducting a comprehensive search of the database, a sequential search is likely to result in a failure to locate or attend to important information cues in parts of the database that the decision maker is unable to cover or devote the necessary time to. For example, when searching the Internal Revenue Code to determine the rate at which gain from the sale of depreciable property is taxed, decision makers who are able to direct their search to areas of the Internal Revenue Code most likely to contain relevant information (e.g., Secs. 1245, 1250, and 291) significantly increase their chances of locating the relevant tax authority because they have been able to limit their search efforts to only a few Code Sections and, consequently, are able to rule out thousands of other Internal Revenue Code Sections. In contrast, decision makers adopting sequential search strategies are much less likely to locate the relevant information because they must consider numerous irrelevant Code Sections before locating the target information. In these situations, time and resource constraints may prevent the decision makers from ever encountering the relevant information during the search. This discussion suggests our third hypothesis: Hypothesis 3. During unaided information search, decision makers using moredirected, less-sequential search strategies will locate a greater amount of relevant information than do less-directed, more-sequential search strategies. 2.5. The moderating influence of search aids on search strategy and performance Decision aids are designed to facilitate the decision-making process by helping decision makers overcome knowledge deficiencies and to make more consistent and accurate decisions than they would otherwise be able to make (e.g., Dawes, Faust, & Meehl, 1989; Goldberg, 1965, 1968).1 While much of the decision aid research addresses decision quality and performance issues, and provides insights relating to various decision-making situations, little is understood about the effect of search aids on search strategy. This study is designed to provide relevant insights. A keyword or topical index is a common search aid designed to provide cues to help decision makers access the database and locate relevant information. These cues are likely to be particularly useful to decision makers with less task-relevant knowledge, because they may otherwise have difficulty selecting the most fruitful areas of the database to search. With a keyword index, less-knowledgeable researchers have the opportunity to match keywords in the search aid with information provided either explicitly or implicitly in the problem statement. 1 Several studies have documented sub-optimal by-products associated with decision aids. For example Glover, Prawitt, and Spilker (1997) find that reliance on decision aids can inhibit learning from experience; Whitecotton, Sanders, and Norris (1998) find that total reliance on a decision aid is inadequate when not all relevant information is captured by the decision aid; and Boatsman, Moeckel, and Pei (1997) report that decision makers may intentionally not rely on an accurate decision aid when they are attempting to outperform others who are relying on the decision aid.

6

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

For example, decision makers performing research to determine the rate at which gain from the sale of depreciable property is taxed could locate relevant authority directly by turning to Code Section 1245. The same researcher could also locate the relevant authority by using the keyword index. In the index, the keyword ‘‘Depreciation Recapture’’ would point the decision maker to Section 1245 (among other Sections such as Secs. 1250 and 291). While a researcher may not initially understand that the relevant authority is located in Section 1245, the cues in the keyword index may ultimately guide the decision maker to the relevant information. The inherent meaning of the words or topics in the index allows a researcher to consider certain keywords and to rule out other keywords. Thus, keywords provide a researcher, irrespective of knowledge level, with some guidance on where to look and where not to look for relevant tax authority. However, search aids should be of less help to a more-knowledgeable researcher who has access to a directed search strategy, which allows the researcher to go directly to relevant information in the database. The keyword index forces the researcher to access the database indirectly, introducing noise into the search process for at least two reasons. First, a particular keyword may be associated with, or refer to, multiple distinct pieces of information in the database—some of which may be irrelevant. Continuing the previous example, the keyword ‘‘Depreciation Recapture’’ will refer the decision maker to Code Sections 1250 and 291, both of which are irrelevant. The decision maker may decide to search these Sections, due to the keyword reference. However, the knowledgeable decision maker may not have searched these irrelevant Sections if she was not required to search through the keyword index. Second, editors of different databases may use different keywords to link to particular pieces of information, making it difficult for more-knowledgeable decision makers to determine whether the keyword will lead to target information. For example, one database service may use the keyword ‘‘Depreciation Recapture’’ to refer decision makers to Code Section 1245. A different database service may use the keyword ‘‘Sale or Exchange’’ to guide the decision maker to Section 1245. These types of potential inconsistencies make it more difficult for a knowledgeable decision maker to implement a directed search to locate relevant authority using a keyword index. In summary, a keyword index or search aid may help less-knowledgeable decision makers locate relevant information in a large database. A keyword index is likely to be of less help to more-knowledgeable researchers because it introduces noise into the search process, making a directed search strategy more difficult to implement. The search aid therefore likely works to weaken or moderate the strength of the link between knowledge and search strategy predicted in Hypothesis 2. This discussion suggests our fourth hypothesis as follows: Hypothesis 4. The link between knowledge and search strategy is stronger during unaided information search than it is during aided information search (i.e., a keyword decision aid moderates the knowledge–search strategy relation). As discussed above, a properly designed search aid can help decision makers overcome knowledge deficiencies to make better decisions. In information search settings, a keyword search aid provides cues to help less-knowledgeable researchers locate relevant information that they likely would have missed without the aid. This is likely true irrespective of search strategy. With the aid, a decision maker using a more-sequential, less-directed strategy can approach the effectiveness of a decision maker using more-directed, less-sequential strategy because the aid highlights cues that facilitate a successful search. Thus the aid should weaken or moderate the link between search strategy and search performance in the decision-making process. This suggests our fifth hypothesis as follows:

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

7

Hypothesis 5. The link between search strategy and performance is stronger during unaided information search than during aided information search (i.e., a keyword decision aid moderates the search strategy–performance relation).

3. Method 3.1. Subjects The sample consisted of 35 tax professionals from three ‘‘big-five’’ accounting firms and 40 graduate tax students from a large private university.2 The tax professionals had, on average, 32.00 months of tax work experience (SD ¼ 13.95 months). All of the graduate tax students were either about to complete a graduate tax research course at the time they participated in the study or had taken the same course one year earlier. On average, the graduate tax students had .85 months of work experience (SD ¼ 1.25 months). The professionals participated as either part of a continuing education course or at the request of a partner from their respective offices and the students participated at the request of their instructors. 3.2. Information search technology The experimental search technology simulates information search software available in tax practice. The experimental software was a custom programmed, internet-based database that included 112 selected Internal Revenue Code Sections, corresponding Treasury Regulations, and annotated explanations of the Code and the Regulations. The experimental software captured clickstream data which was a complete process trace of subjectsÕ information searches including each item examined and the time spent viewing the information. 3.3. Experimental procedures and task During each of the experimental sessions, subjects worked on similarly equipped personal computers in the presence of a researcher. Subjects began the experimental task by completing a ten-minute, task-relevant knowledge test and a thirteen-minute, eight-question multiple-choice ability test. Next, subjects completed a software tutorial designed to familiarize them with the information search software used in the search portion of the experiment. After the tutorial, subjects were informed that their objective was to search the database and save all of the relevant authority they would cite in a research memorandum summarizing their findings. During the information search stage of the experiment, subjects were given a set of facts, a research question, and a task objective. The facts detailed an installment sale of depreciable property to a related party (the facts and issue are included in Appendix A). The tax research objective was to find all the relevant authority necessary to determine the tax consequences of the proposed sale. Upon reading the facts and research question, subjects were reminded of their objective. Subjects were allowed 50 min to complete the research task, and they were informed that they would be asked to provide a numerical solution after completing the information search. Participants were able to shift between the facts and the database of tax authority during the search.

2

Thirty-eight professionals and 45 students completed the experiment. Five students were excluded from the analysis because they failed to follow directions. The data for three professionals were lost because of computer malfunctions.

8

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

After completing their search, subjects entered a post-search module in which they were instructed to review their notebooks and eliminate any authority they had originally saved but no longer deemed as relevant authority for purposes of writing a research memorandum. They were then asked to provide a numerical solution that indicated the amount and type of gain to be recognized. Finally, subjects completed a questionnaire about their background and experience. 3.4. Measures Knowledge. Prior to the information search task, subjects were given an eightquestion, multiple-choice test dealing with topics involved in the research issues including installment sales (1 question), related parties (1), depreciation recapture (2), sales of depreciable property (2), and sales of depreciable property to a related party (2). The questions were drawn from tax textbooks and prior CPA examinations.3 Because subjectsÕ knowledge scores were correlated with their ability scores (i.e., scores from a test with questions dealing with analogical reasoning, general problem solving, and deductive-reasoning abilities), we regressed subjectsÕ raw knowledge scores on their ability scores and used the residuals to represent KNOWLEDGE unexplained by ability. In the descriptive results section, we replace KNOWLEDGE with EXPERIENCE (months of tax work experience) to examine its relation with search strategy and performance. Search strategy. This variable determines whether a subjectÕs search process is more directed and less-sequential or vice-versa. STRATEGY was created using a factor analysis of four different process measures: SEQUENTIAL, TIME TO RELEVANT, TIME, and ITEMS. The analysis used the principal components method of extraction with varimax rotation and the four variables loaded on one factor explaining 46% of the variation. Each of the four individual measures that were used in computing the factor scores for the search STRATEGY measure is described below. The factor loading for each item is as follows: SEQUENTIAL (.69), TIME TO RELEVANT (.63), TIME (.73), and ITEMS (.65). SEQUENTIAL is a correlation coefficient indicating the correlation between the sequence that the Code Sections or keywords appear in their respective lists and the order in which subjects searched the Code Sections or keywords. For example, if a subject performing an unaided search selected the first Code Section in the list, followed by the 20th, 50th, and finally back to the 5th Code Section, the numbers 1, 20, 50, and 5 would be correlated with 1, 2, 3, and 4 to produce the correlation coefficient. A similar procedure was used for subjects performing an aided search (using a keyword index search aid). The higher the number, the more closely the subjectÕs search strategy paralleled the presentation order of the information (i.e., more sequential). TIME TO RELEVANT, TIME, and ITEMS were designed to capture the directedness of subjectsÕ search strategies. TIME TO RELEVANT is the amount of elapsed time (in minutes) from the beginning of the search until the subject initially located a relevant item. The more directed the search strategy, the less time required to initially locate relevant authority. TIME is the number of minutes elapsed from the time the subject began the search until the subject ended the search. The more directed the search, the less time the search should take to complete, because a directed search strategy suggests that the decision maker goes directly to the relevant information. Finally, ITEMS is the number of information items examined during the search process. The more directed the search, the fewer total items decision makers should consider. This measure is consistent with the literature that speculates 3 The reliability coefficient (CronbachÕs a ¼ .49) is moderately low but adequate for a multifaceted construct (Libby & Tan, 1994; Nunnally, 1982).

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

9

that more-knowledgeable individuals search for limited sets of information because they are better able to direct their search towards relevant information (Camerer & Johnson, 1991; Chi et al., 1988). Performance. PERFORMANCE represents performance at the information search task, defined as the number of relevant items saved in the subjectsÕ notebooks. A panel consisting of tax experts in the relevant area of the tax law, including four practice-office tax managers and a university tax professor, performed independent analyses of the research question and identified five Internal Revenue Code Sections (Secs. 267, 453, 707, 1239, and 1245) as relevant items a priori. 3.5. Manipulation The manner in which subjects accessed the database during the information search portion of the task was manipulated between subjects. Although the tax authority database was the same for all subjects, subjects were randomly assigned to the UNAIDED (Code) condition or AIDED (keyword index) condition. Subjects in the unaided condition received a list of Code Section numbers in ascending order. Subjects in the aided condition received a keyword index that listed 60 first-level keywords in alphabetical order. Each first-level keyword had an average of four second-level keywords (minimum of one and maximum of 18). Some form of keyword index search aid is commonly used by all professional tax services (Raabe, Whittenburg, Bost, & Sanders, 2000). Table 1 provides descriptive statistics across all subjects and also by condition (unaided or aided). This table also provides statistics showing that the aided and unaided subject groups are not significantly different in any demographic category.

Table 1 Demographic information for unaided and aided conditionsa Variables

Sample size Knowledge scoreb Ability scorec Months of experience Percent of time spent on research Installment sale in practiced Installment sale in educatione University creditsf a

Condition Overall

Unaided

Aided

t stat

p value

75 4.54 5.59 15.39 16.96 2.55 1.65 14.46

35 4.81 5.55 15.73 16.77 4.00 1.57 15.34

40 4.30 5.62 15.08 17.13 1.28 1.73 13.70

1.29 1.25 .15 .09 1.01 .34 .84

.20 .21 .88 .93 .32 .73 .40

This table compares the demographics of the subjects in unaided and aided conditions to determine whether any differences exist. The absence of any significant difference suggests that any differences in strategy or performance can be attributed to the hypothesized effects rather than differences between subjects. b This represents the number of correct responses subjects provided to an eight-question multiple-choice knowledge test. c This represents the number of correct responses subjects provided to an eight-question multiple-choice general problem solving ability test. d This represents the number of times subjects encountered an installment sale in practice. Although not a significant difference, subjects in the unaided condition have seen more installment sales in practice than those in the aided condition. This non-significant difference is due to one subject in the unaided condition that had dealt with an installment sale approximately 100 times. To test the robustness of the results this subject was removed from the sample and the results obtained are identical to those reported in the remainder of the study. e This represents the number of times subjects reported encountering an installment sale issue in a classroom exercise. f This represents the number of university tax credits accumulated by the subjects.

10

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

4. Results 4.1. Hypotheses The hypotheses focus on the links between variables modeled in Fig. 1 in both unaided and aided information search. Table 2 provides the Pearson correlations among these variables. Panel A presents the correlations for subjects in the unaided condition, and panel B presents correlations for the subjects in the aided condition. We use path analysis to test hypotheses by regressing each endogenous variable in the model (i.e., STRATEGY and PERFORMANCE) on the variables preceding it in the model. The standardized regression coefficients provide the path estimates. Fig. 2 presents the completed path diagram for subjects in the unaided condition, and Fig. 3 presents the diagram for subjects in the aided condition. Regression analyses for unaided search hypotheses (H1–H3). The results of the regression analyses supporting the path analysis for unaided subjects are presented in Table 3. Hypothesis 1 predicts that KNOWLEDGE will have a positive, direct effect on PERFORMANCE. As shown in panel A of Table 3, the effect of KNOWLEDGE on PERFORMANCE is positive but is not significant (std. b ¼ :19, tð32Þ ¼ 1:01, p ¼ :16). Contrary to Hypothesis 1, in the unaided condition, knowledge does not have a direct effect on performance. Table 2 Correlations among variables included in the path analysisa

Panel A: Subjects in unaided condition STRATEGY PERFORMANCE N ¼ 35 Panel B: Subjects in aided condition STRATEGY PERFORMANCE N ¼ 40

KNOWLEDGE

STRATEGY

).54 .36

).42

).12 .35

).05

a

See Fig. 1 for a description of the variables. p < :05. ** p < :01. *

Fig. 2. Path diagram of observed relations between knowledge, strategy, and performance for subjects performing an unaided search. See footnote a in figure.

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

11

Fig. 3. Path diagram of observed relations between knowledge, strategy, and performance for subjects performing an aided (Keyword Index) search. See footnote a in figure.

Hypothesis 2 predicts that knowledge and search strategy are related such that more-knowledgeable subjects will adopt more-directed, less-sequential search strategies than will less-knowledgeable subjects. As presented in Table 3, panel B, the link between KNOWLEDGE and STRATEGY is significant (std. b ¼ :54, tð33Þ ¼ 2:95, p ¼ :00). The negative coefficient is consistent with expectations and indicates that the higher the knowledge, the more-directed and less-sequential the search strategy. In terms of this study, more-knowledgeable decision makers were more likely to begin their search by turning directly to the relevant Code Sections, irrespective of the SectionsÕ location in the database, while less-knowledgeable decision makers were more likely to search the Code Sections in the order they were presented. This result supports Hypothesis 2. Hypothesis 3 posits that the more directed, less-sequential the search strategy, the greater number of relevant Code Sections the subject will locate. Table 3, panel A presents the results. Consistent with predictions, the link between STRATEGY and PERFORMANCE is significant and negative (std. b ¼ :32, tð32Þ ¼ 1:72, p ¼ :05). Subjects who searched the database in a more-directed manner were able to locate more relevant information than subjects who searched the database in a more-sequential manner. The overall indirect effect of KNOWLEDGE on PERFORMANCE is positive (i.e., multiplying the negative link between KNOWLEDGE and STRATEGY (std. b ¼ :54) and the negative link between STRATEGY and PERFORMANCE (std. b ¼ :32) results in a positive indirect effect ð:54  :32 ¼ :17Þ). This indirect effect indicates that the more the decision makers knew about determining the tax consequences of a sale of depreciable property, the more likely they were to search particular areas of the database likely to contain relevant information for resolving the research issue and thus the more relevant tax authority they were able to locate. Given the significant links between KNOWLEDGE and STRATEGY, and STRATEGY and PERFORMANCE, we conducted a mediation analysis to determine whether STRATEGY mediates the KNOWLEDGE–PERFORMANCE relation. Consistent with the procedures outlined by Baron and Kenny (1986), we use three regression equations to test for mediation. The first equation requires the independent variable (KNOWLEDGE) to affect the mediating variable (STRATEGY). This result is supported by Hypothesis 2 as shown in Table 3, panel B. The second equation requires the independent variable (KNOWLEDGE) to affect the dependent variable (PERFORMANCE) in the absence of the mediator variable. Results indicate that KNOWLEDGE has a significant effect on PERFORMANCE

12

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

in the absence of STRATEGY (std. (b ¼ :36, tð33Þ ¼ 2:22, p ¼ :02). Finally, the third equation requires the dependent variable (PERFORMANCE) to be regressed on both the independent variable (KNOWLEDGE) and the mediator (STRATEGY). As noted above, Table 3, panel A indicates that KNOWLEDGE was not significantly related to PERFORMANCE (std. b ¼ :19, tð32Þ ¼ 1:01, p ¼ :16) while STRATEGY was significantly related to PERFORMANCE (std. b ¼ :32, tð32Þ ¼ 1:72, p ¼ :05). The results support the idea that search STRATEGY mediates the KNOWLEDGE–PERFORMANCE relation because the contribution of KNOWLEDGE was reduced and became insignificant in the presence of STRATEGY (Baron & Kenny, 1986). This analysis provides further evidence that, in unaided search, knowledge improves search performance by allowing decision makers to adopt more-directed search strategies which, in turn, allow decision makers to locate more relevant authority. Regression analyses for aided search hypotheses (H4–H5). The regression results supporting the path analysis for the aided subjects are presented in Table 4. As shown in panel A, and opposite the results in the unaided condition, the direct effect of KNOWLEDGE on PERFORMANCE is positive and significant (std. b ¼ :36, tð37Þ ¼ 2:33, p ¼ :02). Having knowledge of tax rules relating to the tax consequences of sales of depreciable property had a direct impact on the amount of relevant tax authority the decision maker was able to locate, irrespective of the search strategy adopted. Hypothesis 4 predicts that the keyword search aid moderates or weakens the relation between knowledge and search strategy. Consistent with Hypothesis 4, as presented in panel B of Table 4, the KNOWLEDGE–STRATEGY link is not significant (std. b ¼ :12, tð38Þ ¼ :76, p ¼ :23). The finding of a significant link between KNOWLEDGE and STRATEGY in the unaided condition coupled with the lack of a significant link between the same two variables in the aided condition suggests that the keyword decision aid moderates the KNOWLEDGE–STRATEGY relation (Baron & Kenny, 1986). To test whether the link between KNOWLEDGE and STRATEGY is stronger in the unaided condition than the aided condition, we use an additional regression with STRATEGY as the dependent variable and KNOWLEDGE, AID, and KNOWLEDGE  AID interaction as independent variables. The KNOWLEDGE  AID interaction is significant (tð70Þ ¼ 2:33, p ¼ :01), providing additional support for Hypothesis 4. This result indicates that knowledge of the tax consequences of gains on the sale of depreciable property allows the decision maker to direct their search toward relevant authority to a greater extent when searching without a search aid than when searching with a search aid. Hypothesis 5 predicts that the information search aid will moderate the STRATEGY–PERFORMANCE relation. Table 4, panel A presents statistical tests indicating that STRATEGY is unrelated to PERFORMANCE (std. b ¼ :09, tð37Þ ¼ :60, p ¼ :28). When searching with the keyword information search aid, subjects who went directly to the areas of the database they believed contained relevant information were no more successful in locating information than were subjects who searched the database sequentially. The significant links between KNOWLEDGE and STRATEGY and STRATEGY and PERFORMANCE in the unaided condition, combined with the lack of significant links between the same variables for subjects in the aided condition, indicates that the search aid moderates the link between knowledge and performance (Baron & Kenny, 1986). To directly test whether the STRATEGY–PERFORMANCE link is stronger in the unaided condition than in the aided condition, we regress KNOWLEDGE, STRATEGY, AID, KNOWLEDGE  STRATEGY, and STRATEGY  AID on PERFORMANCE. The results indicate that the STRATEGY  AID interaction is significant (tð68Þ ¼ 1:62, p ¼ :05), supporting Hypothesis 5—that the manner in which subjects search for authority affects subjectsÕ ability to locate relevant authority more in the unaided than in the aided condition.

Unstandardized coefficient

SE

Standardized coefficient

t

p-value

Panel A: Regression results: PERFORMANCE ¼ a þ b1 KNOWLEDGE þ b2 STRATEGY þ e R2 ¼ :203, F ð1; 32Þ ¼ 4:080, p ¼ :02 KNOWLEDGE b1 .11 .10 .19 1.01 STRATEGY b2 ).28 .16 ).32 )1.72 Constant a 2.20 .16 Panel B: Regression results: STRATEGY ¼ a þ bKNOWLEDGE þ e R2 ¼ :287, F ð1; 33Þ ¼ 13:308, p ¼ :00 KNOWLEDGE b ).34 .09 ).54 Constant a .44 .15 N ¼ 35 a

2.95

.16 .05

H1 H3

.00

H2

See Fig. 1 for a description of the variables. All p-values are one-tailed.

Table 4 Regression results supporting the path analysis in the aided conditiona Unstandardized coefficient

SE

Standardized coefficient

Panel A: Regression results: PERFORMANCE ¼ a þ b1 KNOWLEDGE þ b2 STRATEGY þ e R2 ¼ :130, F ð1; 37Þ ¼ 2:756, p ¼ :08 .19 .08 .36 KNOWLEDGE b1 STRATEGY b2 .10 .17 .09 Constant a 2.32 .16 Panel B: Regression results for: STRATEGY ¼ a þ bKNOWLEDGE þ e R2 ¼ :015, F ð1; 38Þ ¼ :585, p ¼ :45 KNOWLEDGE b ).01 .08 ).12 Constant a .44 .15 N ¼ 40 a

See Fig. 1 for a description of the variables. All p-values are one-tailed.

t

p-value

2.33 .60

.02 .28

).76

.23

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

Table 3 Regression results supporting the path analysis in the unaided conditiona

13

14

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

4.2. Descriptive results Comparison of performance between conditions. The results described above demonstrate that KNOWLEDGE influences STRATEGY in the UNAIDED condition but does not affect STRATEGY in the AIDED condition. Nevertheless, overall PERFORMANCE did not differ between the AIDED and UNAIDED groups (tð73Þ ¼ :63, p ¼ :53). The direct effect of KNOWLEDGE on PERFORMANCE in the AIDED condition appears to have substituted for the impact of the indirect effect of KNOWLEDGE on PERFORMANCE in the UNAIDED condition. When searching with a keyword search aid in this study, knowledge may have had a relatively large direct effect because the number of available keywords was relatively small. Thus, subjects had the time and opportunity to evaluate each keyword for relevance. However, when searching actual databases for answers to complex issues in practice, the list of available keywords or other search aid cues are likely to be much longer. As a result, search strategy is likely to play a more important part in explaining performance than is the direct effect of knowledge. That is, the relative importance of being able to limit the scope of the search (i.e., conduct a directed search) is likely to increase in importance as the number of keywords or cues increases. With a directed search the number of keywords or cues becomes less important because the decision maker is able to go more directly to the relevant keywords or other cues and not sift through irrelevant cues. The direct effect of knowledge is likely to become relatively less important as the number of keywords or cues increase because of the impracticality of evaluating each keyword or cue for relevance. Replacing knowledge with experience. The current study considers the effects of knowledge measured via a pre-test on search strategy and performance. In contrast, prior research has used experience as a proxy for knowledge when considering performance issues relating to information search. Generally speaking, knowledge is likely correlated with experience. However, in this study, KNOWLEDGE and EXPERIENCE are uncorrelated (r ¼ :17, p ¼ :15). These variables are likely not correlated in this study because the students with minimal experience in practice had received classroom instruction on topics relevant to the research issue used in the study, making their task-relevant knowledge similar to professionals on this dimension. To provide descriptive insights on the effects of knowledge versus experience in this study, we conduct the same analyses we used to examine the hypotheses except that we substitute an EXPERIENCE variable (number of months of work experience) for the KNOWLEDGE variable. In this analysis, for subjects in the unaided condition, neither the EXPERIENCE–PERFORMANCE link ðp ¼ :82Þ nor the EXPERIENCE–STRATEGY link ðp ¼ :84Þ is significant. For subjects in the aided condition, the EXPERIENCE–PERFORMANCE link is positive and significant ðp ¼ :05Þ but the EXPERIENCE–STRATEGY link is not significant ðp ¼ :53Þ.4 These results highlight the idea that knowledge has more explanatory power than experience in explaining search strategies but that experience, independent of knowledge, may affect performance in certain situations. In this study, the experienced subjectsÕ exposure to similar keyword search aids may have given them an advantage over inexperienced subjects in recognizing the types of keywords that may lead to relevant information.

4 To determine the robustness of the experience results, we also analyze the data by adding EXPERIENCE to Fig. 1 model before entering KNOWLEDGE. This analysis indicates that EXPERIENCE is insignificant and supports the inferences provided in the original path analysis, but with slightly stronger results.

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

15

5. Summary and conclusions Information search is a critical step in the decision-making process. Consequently, variables affecting performance are important to study in order to understand ways to improve decision quality. Prior research suggests that task-relevant knowledge affects performance at information search tasks, without specifying the nature of the relation. This study contributes to the literature by providing evidence that, in unaided information search, search strategy mediates the knowledge–performance relation. That is, the effect of knowledge on performance is indirect; knowledge affects search strategy, and search strategy in turn affects the researcherÕs ability to locate relevant information. This finding highlights not only the importance of task-relevant knowledge in conducting successful information search but also the significance of the role that search strategy plays in the decision-making process. Understanding the influence of search strategy on performance, as noted by Lohse & Johnson (1996), is an important step toward improving the efficiency and effectiveness of search processes. This knowledge has important implications for both designing more-effective information search aids and training the users of these aids. For example, by examining the research strategies of experts we can learn techniques to improve search strategy effectiveness and efficiency. This study also provides evidence that an information search aid moderates the relations between knowledge and search strategy and search strategy and performance. These findings are important because they suggest that factors present in decision-making environments may mitigate the importance of knowledge when performing information search tasks. An implication of this finding for practice is that designers of information search technology need to provide mechanisms that allow researchers of all knowledge levels to optimize the extent to which they can direct their search strategies towards relevant information. While this study was conducted in a tax setting, results are likely to generalize to other contexts. Taxation is a knowledge-intensive professional domain that involves complex issues with many possible alternative solutions. Consequently the findings in this study are most likely to apply to other domains with similar characteristics such as accounting, engineering, finance, law, and medicine. When researching issues in these domains, professionals must navigate enormous databases of information to locate the information necessary to resolve issues. As the size of the database and complexity of issues increase, the nature of the professionalsÕ search strategy is likely to become increasingly important in explaining the professionalsÕ ability to locate relevant authority. With very large databases, professionals are likely to be unable to utilize their knowledge to resolve issues unless they are able to limit the search to areas of the database where the information is located. This is possible only with directed search strategies. A second issue about generalizability of the results relates to the nature of the search aid used in this study. The search aid was a keyword index consisting of a list of words that refer users to areas of the database that may contain relevant information. The search aid results from this study are likely to apply to aids that present the decision maker with a list of items to choose from to ultimately drill down to the relevant information. This would include: (a) a common keyword index provided to facilitate research in computerized information services and in paper information services, (b) menu-driven database interfaces, and (c) search aids that require the researcher to input keywords to generate a hit list of possible relevant information items that the decision maker must evaluate for relevance. When considering the results presented in this study, the following limitations should be considered. First, while the underlying theory suggests that the findings should relate to nontax contexts and other types of search aids, future research should examine empirically the generalizability of these results to other contexts and

16

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

search aids. Second, this task requires subjects to have or acquire significant technical knowledge to successfully complete the task. Thus, the results of this study may not generalize to less knowledge-based tasks. Third, this study uses proprietary software to capture clickstream data of subjectsÕ search processes. Future research could use the actual software that decision makers use in order to enhance task realism. For example, the recent explosion of Internet-based software that captures clickstream data should be a fruitful source of data for future research on information search strategy in many domains. Finally, although the information database used in this study contains several thousand pages of text information (which is many times larger than databases used in prior information search strategy studies), including all the relevant information needed for the experimental task, and the relevant to total information contained is relatively small, it still represents only a fraction of the information available to tax professionals. As discussed above, expanding the size of the database may work to increase the strength of the effect of information search strategy on performance. Future research could further increase task realism by including all of the information available to decision makers, which is particularly important in professional domains with large bodies of data or information.

Acknowledgments The authors thank Jean C. Bedard, C. Bryan Cloyd, Andy Cuccia, Shane Dikolli, Steve Glover, Scott Summers Jeff Wilks, Mark Zimbelman, three anonymous reviewers, and workshop participants at the 2001 American Accounting Association Annual Meeting, Brigham Young University, Northeastern University, and the University of Utah for their helpful comments.

Appendix A. Research fact pattern A.1. Research question John Eldredge is considering selling a piece of equipment to ERI, a partnership, for $800,000 ($100,000 cash and a $700,000 note receivable) on January 1, 1998. The note requires ERI to pay John Eldredge $100,000 per year beginning on January 1, 1999 and ending on January 1, 2005. In addition, the note bears interest at a rate of 10% per annum. John Eldredge purchased the equipment for $700,000 during 1995 and elected the straight-line method of depreciation. The equipment had a 7-year life. The adjusted basis of the property to John is $450,000. A principal reason for the sale is that ERI will receive a step-up in basis that will allow for additional depreciation deductions. ERI is owned equally by three individuals: John Eldredge, Christine Brown (JohnÕs sister), and Ken Brady (a friend). Each partner has a proportionate capital and profits interest. What would be the tax consequences to John Eldredge on the proposed sale? Specifically, what are the timing and the character of the gain recognized?

References Anderson, M. (1988). A comparative analysis of information search and evaluation behavior of professional and non-professional financial analysts. Accounting, Organizations and Society, 13(5), 431–446.

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

17

Baron, R., & Kenny, D. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173–1182. Bedard, J. C., & Biggs, S. (1991). Pattern recognition, hypothesis generation, and auditor performance on an analytical task. The Accounting Review, 66(2), 622–643. Bedard, J., & Mock, T. (1992). Expert and novice problem-solving behavior in audit planning. Auditing: A Journal of Practice and Theory, 11, 1–20. Biggs, S., & Mock, T. (1983). An investigation of auditor decision processes in the evaluation of internal controls and audit scope decisions. Journal of Accounting Research, 21(1), 234–255. Boatsman, J., Moeckel, C., & Pei, B. (1997). The effects of decision consequences on auditorsÕ reliance on decision aids in audit planning. Organization Behavior and Human Decision Processes, 71(2), 211– 247. Bouwman, M., Frishkoff, P., & Frishkoff, P. (1987). How do financial analysts make decisions? A process model of investment screening decisions. Accounting, Organizations and Society, 12(1), 1–30. Camerer, C., & Johnson, E. (1991). The process-performance paradox in expert judgment: How can experts know so much and predict so badly?. In K. Ericsson, & J. Smith (Eds.), Toward a general theory of expertise: Prospects and limits (pp. 195–217). New York, NY: Cambridge University Press. Chi, M., Glaser, R., & Farr, M. (1988). The nature of expertise. Hillsdale, NJ: Erlbaum. Cloyd, B. (1997). Performance in tax research tasks: The joint effect of knowledge and accountability. The Accounting Review, 72(1), 111–131. Cloyd, B., & Spilker, B. (1999). The influence of client preferences on tax professionalsÕ search for judicial precedents, subsequent judgments, and recommendations. The Accounting Review, 74(3), 299–322. Dawes, R., Faust, M., & Meehl, P. (1989). Clinical versus actuarial judgment. Science, 243, 1674– 1688. Einhorn, H., & Hogarth, R. (1981). Behavioral decision theory: Processes of judgment and choice. Journal of Accounting Research, 19(1), 1–31. Goldberg, L. (1965). Diagnosticians vs. diagnostic signs: The diagnosis of psychosis vs. neurosis from the MMPI. Psychological Monographs, 79(9, Whole No. 602). Goldberg, L. (1968). Simple model or simple processes? Some research on clinical judgment. American Psychologist, 23, 483–496. Glover, S., Prawitt, D., & Spilker, B. (1997). The influence of decision aids on user behavior: Implications for knowledge acquisition and inappropriate reliance. Organization Behavior and Human Decision Processes, 72(2), 232–255. Hersh, W., & Hickam, K. (1995). Information retrieval in medicine: The sapphire experience. Journal of the American Society for Information Science, 46(10), 743–747. Hersh, W., Crabtree, M., Hickam, D., Sacherek, L., Rose, L., & Friedman, C. (2000). Factors associated with successful answering of clinical questions using an information retrieval system. Bulletin of the Medical Library Association, 88(4), 323–331. Hershey, D., Walsh, D., Read, S., & Chulef, A. (1990). The effects of expertise on financial problem solving: Evidence for goal-directed, problem-solving scripts. Organization Behavior and Human Decision Processes, 46(1), 77–101. Hunton, J., & McEwen, R. (1997). An assessment of the relations between analystsÕ earnings forecast accuracy, motivational incentives, and cognitive information search strategy. The Accounting Review, 72(4), 497–515. Johnson, E. (1988). Expertise and decision under uncertainty: Performance and process. In M. Chi, R. Glaser, & M. Farr (Eds.), The nature of expertise (pp. 209–228). Hillsdale, NJ: Erlbaum. Lawrence, J. (1988). Expertise on the bench: Modeling magistratesÕ judicial decision-making. In M. Chi, R. Glaser, & M. Farr (Eds.), The nature of expertise (pp. 229–252). Hillsdale, NJ: Erlbaum. Lee, H., Herr, P., Kardes, F., & Kim, C. (1999). Motivated search: Effects of choice accountability, issue involvement, and prior knowledge on information acquisition and use. Journal of Business Research, 45, 75–88. Libby, R., & Tan, H. (1994). Modeling the determinants of auditing expertise. Accounting, Organizations and Society, 19(8), 701–716. Lohse, G., & Johnson, E. (1996). A comparison of two process tracing methods for choice tasks. Organization Behavior and Human Decision Processes, 68(1), 24–43. Nunnally, J. (1982). Psychometric theory (2nd ed). New York: McGraw Hill. Raabe, W., Whittenburg, G., Bost, J., & Sanders, D. (2000). WestÕs federal tax research (5th ed). St. Paul, MN: West Publishing. Simon, H. (1977). The new science of management decision. Englewood Cliffs, NJ: Prentice-Hall. Spilker, B. (1995). The effects of time pressure and knowledge on keyword selection behavior in tax research. The Accounting Review, 70(1), 49–70. Spink, A., & Saracevic, T. (1997). Interaction in information retrieval: Selection and effectiveness of search terms. Journal of the American Society for Information Science, 48(8), 741–761.

18

J.A. Barrick, B.C. Spilker / Organizational Behavior and Human Decision Processes 90 (2003) 1–18

Whitecotton, S., Sanders, D., & Norris, K. (1998). Improving predictive accuracy with a combination of human intuition and mechanical decision aids. Organization Behavior and Human Decision Processes, 76(3), 325–348. Yates, F. (1990). Judgment and decision making. Englewood Cliffs, NJ: Prentice-Hall. Received 5 January 2001