- Email: [email protected]
Identifying peer states to assess technology-based economic development
Technology in Society 39 (2014) 68e76
Contents lists available at ScienceDirect
Technology in Society journal homepage: www.elsevier.com/locate/techsoc
Identifying peer states to assess technology-based economic development David L. Schwarzkopf* Adamian 208, Bentley University, 175 Forest Street, Waltham, MA 02452, USA
a r t i c l e i n f o
a b s t r a c t
Article history: Received 4 June 2014 Received in revised form 11 August 2014 Accepted 18 August 2014 Available online
States often rely on 50-state “report cards” or indices to track their progress in technologybased economic development. Economic development agencies value these indices, published by independent consultancies, because they cut the costs of compiling data, compare states to one another and allow agencies to avoid charges of “cherry-picking” measures to serve their own purpose. The rankings of the states in these indices have appeal as they give policymakers and development agencies an idea of likely peer states and possible members of an aspirant group. Peers and aspirant groups provide a state with examples of alternative approaches to economic development, while allowing agencies to depict economic development activities in competitive terms for policymakers and legislators. Therefore, it is important that these comparisons be valid and, since the state's development policies affect the public, it is worthwhile for the citizenry to understand how agencies make these comparisons. Although rankings are easy to understand, manipulating multiple measures to produce a single number (the ranking) can distort important differences between states. The present research addresses whether these rankings provide a reliable source for peer and aspirant groups. Analysis of two popular scorecardsdthe State New Economy Index and the State Technology and Science Indexdshows that rankings provide only a broad picture of a state's relative standing. Clustering techniques based on self-organizing maps give a more refined view, better suited for policy analysis. © 2014 Elsevier Ltd. All rights reserved.
Keywords: Economic competition Economic development Science and technology reporting Scorecards Self-organizing maps
1. Introduction 1.1. The value of comparisons in technology-based economic development Sub-federal governmental units such as states, provinces or municipalities can look to other units in order to build a story of how they are doing in technology-based economic development (TBED). In particular, language in
* Tel.: þ1 781 891 2783; fax: þ1 781 891 2896. E-mail address: [email protected]. http://dx.doi.org/10.1016/j.techsoc.2014.08.004 0160-791X/© 2014 Elsevier Ltd. All rights reserved.
state reports on TBED shows that states try to identify peers while targeting other states as members of a group whose success in TBED they aspire to (see below for examples). Peers can foster a sense of real or implied competition that can be manipulated to motivate legislators and other policymakers to act in ways believed to increase the state's attractiveness to businesses, skilled workers, entrepreneurs and investors. Similarly, members of an aspirant group can serve as models to spur state agencies to increase political and business activity to improve economic development. Peers and aspirant groups also provide a state with examples of alternative approaches to economic development. Therefore, it is
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
important for a state to make valid comparisons to others in order to find peers and aspirant groups. Since the state's development policies affect the public, it is worthwhile for the citizenry to understand how state agencies go about making these comparisons. There are many ways that states can identify peersdfor example, geographic proximity. Over the past decade or two, however, various agencies have provided indices or “report cards” on the 50 states' TBED activities, resulting in state rankings. For reasons discussed below, these rankings hold the promise of an easy way to find peers using TBED data. But do rankings provide a reliable source for identifying peer and aspirant groups? Is there another reasonable way to identify what makes a good comparison for a given state? By addressing these questions, this research expands on academic inquiry into science and technology (S&T) reporting, which generally has examined national reporting issues (e.g., [1,2]), rather than state efforts (although [3,4] are exceptions). The discussion proceeds with background information on state TBED reporting, including the rise of 50-state indices. The methodology, findings and analysis of the study follow, before the paper concludes with suggestions for further academic research and practical action. 1.2. Background to state reporting on economic development Federal governments have measured their economic development status and, in particular, the effects of S&T on development, for at least the past 50 years [5e9]. In the U.S. more recently, states have tracked their progress in separate S&T reports and through 50-state scorecards or indices [10]. These indices include the State New Economy Index (published over various years by the Progressive Policy Institute, the Kauffman Foundation and the Information Technology and Innovation Foundation), the National Science Foundation's Science and Engineering Indicators, the Milken Institute's State Technology and Science Index, TechAmerica Foundation's Cyberstates, and the Development Report Card for the States, once published by the Corporation for Enterprise Development but now discontinued. Even though the extent to which S&T directly affect economic growth remains controversial and the role that a state government can play in fostering growth through S&T is at best indirect, these scorecards remain popular. This is understandable, since publication of such measures not only signals the value the state places on particular aspects of economic development [11], but also can lend credence to a state's claims to progress [see Ref. [12]]. However, this signaling effect means that those involved with these scorecards need to be aware of the state's intentions as a reporter and the audience of users [13,14]. In fact, among the reasons that 50-state indices published by independent agencies are valued is that they allow states to avoid charges of “cherry-picking” measures to serve the reporter's purpose. In addition, it is costly for a state to compile data on its own. Savings come from the use of publicly available data, such as those presented in these scorecards.
69
1.3. The promise and perils of rankings Beyond cost savings, these indices provide comparable data, which is not the norm in individual states' S&T reporting.1 To focus on this comparability, most 50-state scorecards provide rankings that appear easy to understand2done either is or is not in the top ten, the top quintile, or such. Ranking on multiple measures, such as those used in the scorecards, is complicated because a state can score “higher” or “lower” on any single given measure among the many used. To simplify the picture, scorecard issuers combine multiple measures into a final singlenumber ranking. While some authors provide detail on how to interpret rankings and the limitations of the same (e.g., [15]), it is the overall ranking that draws media and agency attention. That is, rankings tempt readers not to look at similarity of economic or environmental conditions, but to look at “who's ahead” or “who's catching up.” Evidence of this power of rankings is plentiful, from state reports (e.g., “In the Technology and Science Workforce Composite Index, Georgia fell 19 spots in 4 years”)3 to issuers' web sites (e.g. “Can anyone catch Massachusetts?”).4 The use of rankings is thus separate from the question of the degree to which the measures employed are valid, since the public and policymakers take the rankings as published. Their use raises concerns about whether they can tell a state who the state's peers are and who is a member of the state's aspirant group. By comparing rankings to another analytical method using the same data, this research addresses that issue.
2. Method The method of inquiry proceeded in two phases. In the first, data from two 50-state indices were used to produce self-organizing maps (SOMs)dvisualization tools that help one to judge similarities among multi-measure data sets. The second stage used these SOMs to generate clustersdgroupings of states with similar characteristics based on the TBED data. The 2012 edition of two popular scorecards provided the data for this study: the State New Economy Index (“SNEI 2012”) [16] and the Milken Institute's State Technology and Science Index (“STSI 2012”) [17]. SNEI began publishing in 1999; STSI in 2002. Both indices are used in individual state
1 A study of 31 reports issued between 2004 and 2008 by agencies in 24 states found 1160 different measures used, only one of which appeared in more than half of the reports. Less than one percent of the measures appeared in 10 or more reports, with 833 measures (71.8%) appearing only once. Yet 15 of the 24 states compared themselves to others on selected measures. See Flynn, PM and Schwarzkopf, DL. Assessing State Reports on Technology-Based Economic Development: Lessons for Benchmarking. Report to the National Science Foundation (award number 0617112); 2010. 2 The National Science Foundation's Science and Engineering Indicators groups states into quartiles, rather than rank them. 3 Technology Association of Georgia. Invigorating Georgia's Technology Industry, http://www.tagonline.org/files/marketing_collateral.pdf, accessed 11 August 2014. 4 State Technology and Science Index home page, http:// statetechandscience.org/, accessed 11 August 2014.
70
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
Table 1 State ranks by scorecard. Rank
SNEI 2012
STSI 2012
SNEI 2002
Rank
SNEI 2012
STSI 2012
SNEI 2002
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
MA DE WA CA MD VA CO UT CT NJ NY NH MN OR VT AZ TX GA MI IL FL PA RI ID NC
MA MD CO CA WA VA DE UT CT NH PA VT MN NY NJ AZ RI TX IL OR NM NC MI KS WI
MA WA CA CO MD NJ CT VA DE NY OR UT MN TX NH AZ IL FL PA ID RI GA MI MO ME
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
NV ME AK KS NM WI OH MO ND NE HI MT IA TN SC WY IN SD LA KY AL OK AR WV MS
GA IN MO AL OH IA ND NE ID HI MT TN FL ME OK AK SC SD LA WY KY NV WV AR MS
NC NM VT KS OH AK NV NE OK HI IN MT IA TN WI SC KY SD ND LA WY AL AR MS WV
Sources: Atkinson (2002, p.7) [23]; Atkinson and Stewart (2012, p. 11) [16]; Klowden and Wolfe (2013, p. 2, as adjusted to consider omitted variablesdsee Appendix 2) [17]. Note. States are identified by their two-letter postal code. For SNEI, Atkinson and Stewart (2012, p. 11) [16] caution, “Due to changes in methodology, changes in rank from previous editions may not positively reflect changes in economic structure.”
economic development reports, describe their methodology clearly and allow free public access to their raw data. SNEI 2012 ranks states based on a composite of 26 variables. The index's authors establish weights in an effort to compensate for the likelihood of high correlations among the variables. The analysis below preserves those weights. The STSI 2012 data set consists of 79 measures grouped into 5 composite categories that are further weighted to produce a single score.5 Because of missing state data, five measures were dropped for the present analysis. It was also necessary to adjust STSI 2012's weightings across the composites, which led to minor changes from the published rankings. See Appendix 1 to this paper for details. State rankings from each index appear in Table 1. The SOM Toolbox application [18] produced a selforganizing map [19] based on each index's data. The selforganizing principle behind the map uses a case's (here, a state's) multiple characteristics (measures) to form an input vector. The SOM chooses reference or codebook vectors based on the distribution of all the input cases, matching each case to the codebook vectors. This iterative process calculates proximity by Euclidean distance and results in an organization that has not relied on any prior sorting scheme, hence is “self-organizing.” Groups of similar cases and a sense of proximity between cases are visually displayed and
5
For details of the measures, see Refs. [16,17].
can be subjected to further analysis through clustering techniques, as described below. See Kohonen (2001) [19] for technical details and a lengthy list of references to academic research using SOMs. The approach used here is similar to that introduced by Vesanto and Alhoniemi (2000) [20]. The visual displays of the SOMs for SNEI 2012 and STSI 2012 appear in Figs. 1 and 2, respectively.6 3. Findings 3.1. What do the SOMs indicate? Both vector quantization error (Q) and topological error (te) enter into the assessment of the goodness-of-fit of an SOM. Q measures the overall difference between the given input vectors and their closest reference vector (Kohonen, 2001, section 1.5) [19]. Topological error measures the fit of overall groupings by showing the proportion of input vectors for which the best-matching reference vector and the second-best-matching reference vector are not neighbors (Kohonen, 2001, 161) [19]. For the SNEI 2012 mapping, Q ¼ 3.29 and te ¼ 0.00; for the STSI 2012 map, Q ¼ 5.63 and te ¼ 0.00. The topological error suggests a reasonable mapping; quantization errors enter into the stability analysis described in Appendix 2. Figs. 1 and 2 show the resulting SOM visualizations, called U-matrices, with and without labels. A U-matrix consists of two kinds of cells: placement cells (areas on the matrix where the cases are placed) and intervening cells (areas between placement cells). In the figures, placement cells are labeled on the right-hand side with the state's postal code and ranking for the given scorecard. Intervening cells appear only on the left-hand display in each figure. The shade of a placement cell can be interpreted as the average distance of that cell (here, the state or states in the cell) from adjacent placement cellsdthe lighter the shade, the closer the proximity. Intervening cells separate rows of placement cells as well as adjacent placement cells and thus provide a broad view of large-cluster boundaries. The shade of intervening cells gives a sense of the distance between states, based on the vector of measures, again with lighter shades suggesting more similarity. States sharing a given placement (labeled) cell are most similar to each other. While some placements are what one would expect from the rankings (e.g., for SNEI 2012, #12 New Hampshire and #13 Minnesota being paired, or the pairings of some top-ranked and bottom-ranked states), other results are unexpected in that the grouped states are distant on the rankings (e.g., SNEI 2012's #27 Maine with #38 Iowa or STSI 2012's #31 Iowa with #43 South Dakota). This suggests a limit to identifying a peer group based on rankings, which the analysis below explores. 3.2. Are the SOMs reliable? Observed differences can arise because of an index's scorekeeping method or because of problems with the
6 To conserve space, mappings for the individual variables from each scorecard are omitted. These are available from the author upon request.
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
Fig. 1. SOM based on SNEI 2012.
Fig. 2. SOM based on STSI 2012.
71
72
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
Table 2 SNEI 2012 and STSI 2012 clusters (with state rankings). Cluster
SNEI 2012
STSI 2012
1
Tennessee (39), South Carolina (40), Louisiana (44), Kentucky (45), Alabama (46), Arkansas (48), West Virginia (49), Mississippi (50) North Dakota (34), Montana (37), Wyoming (41), Indiana (42), South Dakota (43), Oklahoma (47) Nevada (26)
Tennessee (37), Oklahoma (40), Louisiana (44), Wyoming (45), Kentucky (46), West Virginia (48), Arkansas (49), Mississippi (50) Hawaii (35), Florida (38), Maine (39), Alaska (41), South Carolina (42), Nevada (47) Iowa (31), North Dakota (32), Nebraska (33), Montana (36), South Dakota (43) Wisconsin (25), Indiana (27), Missouri (28), Ohio (30)
2 3 4 5 6 7 8 9 10 11
North Carolina (25), Alaska (28), Kansas (29), Ohio (32), Missouri (33), Nebraska (35) Maine (27), Wisconsin (31), Hawaii (36), Iowa (38) Arizona (16), Texas (17), Georgia (18), Michigan (19), Florida (21), Pennsylvania (22), New Mexico (30) Oregon (14), Vermont (15), Illinois (20), Rhode Island (23), Idaho (24) [an empty cluster] Massachusetts (1), Washington (3), California (4), Maryland (5), Virginia (6), Colorado (7), Utah (8) New York (11), New Hampshire (12), Minnesota (13) Delaware (2), Connecticut (9), New Jersey (10)
Texas (18), Illinois (19), Oregon (20), Kansas (24), Georgia (26), Idaho (34) North Carolina (22), Michigan (23), Alabama (29) Pennsylvania (11), Vermont (12), New York (14), Rhode Island (17), New Mexico (21) Utah (8), New Hampshire (10), Minnesota (13), Arizona (16) Massachusetts (1), Maryland (2), Colorado (3) Virginia (6), Delaware (7) California (4), Washington (5), Connecticut (9), New Jersey (15)
Note. Clusters are numbered based on their position on the SOM (see Figs. 1 and 2), roughly from upper-left to lower-right. Cluster numbers do not imply “better” or “worse” status.
SOM's mapping. Tests using a bootstrap technique described by de Bodt, Cottrell and Verleysen (2002) [21] minimize the chance of the latter. For each of the two maps, tests were run on the placement of states that are close on the rankings but not close on the mapdthat is, not located in the same or an adjacent celldand between states close on the map but not close on the rankings. These stability tests, described in detail in Appendix 2, showed it was highly unlikely (p < 0.001) that the observed mapping outcomes are random. Thus the differences between the mapped positions and the rankings are highly unlikely to be caused by the mapping procedure, but represent a different relationship than that presented in the indices. 3.3. What clusters result from the SOMs? A k-means clustering technique helps derive a better sense of peer groups on each map. Clustering used 25 trials, each with 100 randomized vector-starts, since cluster designation is sensitive to the location of the starting vector, and with the maximum number of clusters set at 20. For each trial, the number of clusters that produced the lowest Davies-Bouldin (DB) index [22] was noted. Over the 25 trials on each map, 11 clusters most frequently produced the lowest DB index for each. Table 2 lists the clusters and the respective index's ranking of the states. 4. Analysis: do the clusters improve the notion of a state's peers? To compare the contents of the clusters with the notion of “peer” derived from the rankings, we can look at the inclusive range of rankings within the clusters, intervals between rankings for the states within each cluster, and discontinuities between clusters. The clusters show a wide average inclusive range of rankings. For example, using SNEI 2012's ranks, there are 12 places, inclusive of the endpoints, from #39 Tennessee to
#50 Mississippi in Cluster 1. Again, there are 17 inclusive places from STSI 2012's #18 Texas to #34 Idaho in that index's Cluster 5. The average of the (non-zero) clusters' inclusive ranges is 9.6 for SNEI 2012 and 9.8 for STSI 2012. Thus, a peer may be farther away from a given state than the ranked order may suggest. Looking at the intervening ranks for the states in a cluster provides another perspective on whether this clustering method improves on rankings in determining peers. Starting with the highest-ranked state in the cluster, one can count the intervening ranks to the next-highestranked state and continue this for consecutive pairs in the cluster. For example, in the STSI 2012 Cluster 1, there are two intervening ranks between #37 Tennessee and #40 Oklahoma (namely, #38 and #39), and three intervening ranks between Oklahoma and #44 Louisiana. Based on this metric, 28 of the 40 consecutive pairs (70%) in the SNEI 2012 clusters and 24 out of the 39 (62%) in the STSI 2012 have zero or one rank intervening. This suggests that the rankings seem to provide a reasonable, if broad, start on determining peers and aspirant groups. Next, note the substantial discontinuities between the rankings and the clusters. For example, for SNEI 2012, the “middle” clusters (numbers 4 through 7), contain the following range of rankings: #14e#24; #16e#30; #27e#38; #25e#35. For STSI 2012, the range in those clusters is: #11e#21; #22e#29; #18e#34; #25e#30. In summary, it appears that rankings provide a mixed picture for a state looking for peers or members of an aspirant group. While it is possible that a state can simply look one or two steps to either side of its ranking to find its peers, it is not necessarily the case that those states are similar based on TBED data and it is quite likely that it will have peers that its ranking will not reveal. At the same time, a state wanting to “climb the rankings” may find that even moving up as many as five or six ranks leaves it in the same peer group as before. This is not to minimize the effect that a single number has on public attention. “We're
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
73
Fig. 3. SOM based on SNEI 2002.
number 14” will sound better than “We're number 20.” But the reality may be that both 14 and 20 are highly similar. Rankings can also overshadow some more-nuanced differences. For example, in both mappings, the top ten ranked states are divided into two (SNEI 2012) or four (STSI 2012) smaller groupings. Similarly, the bottom ten states in both scorecards fall into two different clusters. Note also that the contents of the clusters are not consistent from index to index. Of the 122 possible pairings of states within all of the clusters generated by SNEI 2012, only 33 of those pairs (27%) appear in the clusters formed from STSI 2012.7 This underscores the sensitivity of the notion of “peer” to the kinds of measures used in the indices. Finally, it is possible that the observed relationship between clusters and rankings is a unique product of the 2012 indices. To test this, the same tests were run on the SNEI scorecard of ten years earlier (“SNEI 2002”) [23]. See Table 1 for SNEI 2002's state rankings. The resulting SOM appears in Fig. 3 (Q ¼ 2.72; te ¼ 0.00). Again, note the unexpected pairings (e.g., #18 Florida and #32 Nevada), as well as a
7
The 33 include the 15 pairs formed from the set {Arkansas, Kentucky, Louisiana, Mississippi, Tennessee, West Virginia}, the 3 formed from {Montana, North Dakota, South Dakota}, the 3 from {Idaho, Illinois, Oregon}, the 3 from {Colorado, Maryland, Massachusetts}, and the following 9 individual pairs: OklahomaeWyoming; MissourieOhio; HawaiieMaine; GeorgiaeTexas; New MexicoePennsylvania; Rhode IslandeVermont; CaliforniaeWashington; MinnesotaeNew Hampshire; and ConnecticuteNew Jersey.
number of groupings that are closer to what the rankings would lead one to expect.8 The k-means cluster technique suggested 12 clusters, as listed (with SNEI 2002 rankings) in Table 3.9 Here 18 of the 35 consecutive pairings (51%) have zero or one intervening rank within their cluster; the average inclusive range of the clusters is 10.5; and discontinuities similar to those seen in the 2012 indices appear in the middle clustersdall of which suggests the previous observations are not unique to the 2012 indices. Stability tests of the SOM (described in Appendix 2) again show it is highly unlikely (p < 0.001) that the SOM groupings are random. While it is tempting to trace a state's trajectory from 2002 to 2012 on the SNEI mappings, the validity of such a move is questionable because of changes over that period in the metrics and methods. This adds to the need for a way to map consistently over time, as mentioned in the following section. 5. Discussion and conclusion This work is not intended to criticize the measures used in the scorecards or to compare rankings across scorecards, but to see whether these rankings provide readers with a
8
The 2002 SNEI ranks were based on 21 measures [23]. The difference in the number of clusters between the 2002 and 2012 SNEI mappings is purely a function of the clustering algorithm. 9
74
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
Table 3 SNEI 2002 clusters (with state rankings). 1
Oklahoma (34), South Carolina (41), Kentucky (42), Louisiana (45), Alabama (47), Arkansas (48), Mississippi (49) 2 Alaska (31), Montana (37), Iowa (38), South Dakota (43), North Dakota (44), West Virginia (50) 3 Georgia (22), North Carolina (26), Hawaii (35) 4 Missouri (24), Maine (25), Kansas (29), Ohio (30), Nebraska (33), Indiana (36) 5 Florida (18), Nevada (32) 6 Pennsylvania (19), Michigan (23) 7 Utah (12), Texas (14), Arizona (16) 8 Oregon (11), Minnesota (13), Illinois (17) 9 Idaho (20), Rhode Island (21), New Mexico (27), Vermont (28) 10 Massachusetts (1), Washington (2), California (3), Colorado (4), Maryland (5), Virginia (8) 11 Connecticut (7), New Hampshire (15) 12 New Jersey (6), Delaware (9), New York (10) Note. Clusters are numbered based on their position on the SOM (see Fig. 3), roughly from upper-left to lower-right. Cluster numbers do not imply “better” or “worse” status.
reasonable sense of peer and aspirant states. While rankings can give a state's economic development agents a rough guide to peers and aspirants, SOMs offer a more refined approach. The mapping itself points to most-similar peers; dividing the map into clusters reveals other similar states while showing collections of “near” neighbors that states may look to as members of their aspirant group. Mapping, expanded by clustering, thus provides states with a better idea of group membership than the scorecards' presentations based on rankings. What explains the observed discrepancies? The scorecards' methodology appears to magnify the importance of small differences in individual measures. For example, for each of its five composites, STSI 2012 ranks the states on each measure, translates that rank into a score, averages these scores and then ranks these averaged scores. Thus, a small difference on a particular measure (e.g., Initial Public Offering Proceeds as a Percentage of Gross State Product) leads to a difference of the rank based on measure, but since the interval between successive scores is constant no matter the size of the interval between measures, the difference between the averaged converted composite scores may exaggerate the initial difference on the measure. When the composites are compiled, these differences distort rankings. The mapping and clustering, on the other hand, sort states by similarities using distance-algorithms and thus do not exaggerate small differences. Further, rather than reducing multiple measures down to one number for a ranking, the mapping preserves the complex n-item vector of measures. The trade-off, of course, is that it is easier to describe a state via rankings than to interpret the detail of one state's vector compared to another. A closer analysis of each cluster will provide a better understanding of what constitutes membership in that cluster, but this in turn will provoke a more detailed and nuanced discussion among policymakers than a simple call to “climb the rankings” does. This closer analysis is what states need in order to address the question of what policies or practices they should adapt from others to improve their own TBED activity. Identifying immediate peers is easy: look within a cluster. Understanding why these are peers is more difficult. Similarly, looking at
adjacent clusters provides a rough picture of aspirant (or “potential challenger”) groups. However, a look at the details of how a state stands on related measures (e.g., in technology-based labor force deployment or in research and development funding) is required to provide the guidance policymakers need to advance TBED.10 The attraction to compare oneself with others is strong. While this research has shown how to improve the static view of a state's economic development status, mapping also suggests ways to view changes in state relationships over time. However, the SNEI authors (Atkinson, 2002, p. 7; Atkinson & Stewart, 2012, p. 11) [16,23] warn against comparing state positions across years of their reports because of changes in reporting methodology. Thus, there is a question of how to design a simple, reasonable method for displaying a state's development in comparison to other states over time. That design forms the next phase of this research agenda. Most importantly, by shifting the focus from a single index number or rank, the more detailed groupings offered by SOMs allow states to resist the allure of considering economic development as a zero-sum game where one state's gains in the rankings come at the cost of another state's losses. For example, a state should feel the benefit of increased research and development spending within its borders, even if all states experience increases in such spending and whether the state's increase is enough to move it ahead in the rankings or not. In short, policymakers and legislators can then feel free to look at economic development as an endeavor in which all states can show improvement. The aim, after all, should not be to “catch Massachusetts” or “jump five places in the rankings,” but to improve citizens' living conditions within a state's economic context.
Acknowledgments This paper builds on the author's unpublished research with Patricia Flynn into state reports on technology-based economic development, funded by the National Science Foundation (“NSF”; award number 0617112). In particular, material in the introduction, technical descriptions of selforganizing maps and the analytical approach are derived from that study, although different data sources are used. The author thanks Patricia Flynn and two anonymous reviewers for their suggestions, Dominique Haughton for her insights into self-organizing maps, and the NSF and Bentley University for their financial support of the earlier study. Neither funding group influenced the research data collection, analysis or commentary. Appendix 1. Data adjustments to STSI 2012 The STSI 2012 scorecard gathers 79 measures over five areas: Research & Development Inputs; Risk Capital and
10 As an aside, it is worth considering the value of this technique to provide an approach to other areas where rankings are popular, but which require more in-depth study to determine policy direction, such as the status of universities, hospitals or nursing homes. Thanks to an anonymous reviewer for raising this point.
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
Entrepreneurial Infrastructure; Human Capital Investment; Technology & Science Workforce; and Technology Concentration & Dynamism [17]. For each measure, states are ranked based on their raw data, with the top state given a score of 100; the second state, 98; the third, 96; and so on to the fiftieth, which receives a score of 2. When states are missing data for a measure, they generally receive no score, but in some instances are given a score corresponding to the rank below the last-ranked state to report data on the measure. All non-zero scores within an area are averaged to produce five composite scores for each state. These are weighted to produce the final score; the overall ranking is based on this final score. The present analysis omitted the following five measures that had a substantial number of states not reporting data: Venture Capital Investment in Nanotechnology as a Percent of Gross State Product (GSP); Sum of Equity Invested in Green Technology per $100,000 GSP; Venture Capital Investment in Clean Technology as a Percent of GSP; Intensity of Agricultural Engineers; and Intensity of Biomedical Engineers. This left 74 measures for the analysis. Further, an examination of STSI 2012's method of computing scores based on ranks revealed an apparent inconsistency. In some instances, the raw data reported for a measure was not correctly transferred to STSI's summary for ranking. Where possible, these omissions were corrected and scores were assigned consistent with STSI 2012's overall methodology. These adjustments produced only small changes in the overall rankings, with no state moving more than two positions. Note that none of the analysis or commentary is sensitive to these differences. The present analysis uses the adjusted rankings. Appendix 2. SOM stability tests The aim of SOM stability tests is to minimize the chance that the observed SOMs show positions that are not stabledthat is, that are no better than chance happenings. As noted in the narrative, the tests examine the mapped positions of states that are “close” in the SOM but “distant” in rankings and states that are “distant” in the SOM but “close” in the rankings. The tests rely on a bootstrap technique developed by de Bodt, Cottrell and Verleysen (2002) [21] that has been used in other SOM-based research (e.g., [24]). A single test under this method uses a 50-state data set that includes selected, “fixed” states (the target of the comparisons). The remaining positions in the set are open, to be populated randomly with an equal likelihood by any of the “nonfixed” states. Each test re-populates the open positions in the data set. Each test results in a SOM. The analysis used 40 tests (mappings) with topological errors equal to zero. See the description of topological error (te) in the narrative. For each mapping, the shortest path between the target (fixed) states was counted, with target states mapped in the same cell having a distance of zero. As in de Bodt, Cottrell, and Verleysen (2002) [21], these distances are compared to those produced by chance by analyzing them in a binomial distribution, where the probability of two states being
75
“highly similar” is considered as the likelihood of the two being in the same (distance ¼ 0) or an adjacent (distance ¼ 1) cell. The probability is thus 7 divided by 36dthat is, the same cell plus six adjacent cells in the hexagonal map divided by the number of cells in the overall SOM. Finally, a measure similar to the coefficient of variation provides a sense of the stability of the mapping: the standard deviation of the 40 trials' quantization errors divided by the average of those quantization errors. A smaller number indicates greater stability. See the narrative for a description of quantization error (Q). A summary of the tests follows. To test the 2012 SNEI mapping, the comparison was among Maine (#27), Iowa (#38) and Tennessee (#39). Iowa and Tennessee are adjacently ranked but four cells removed in the SOM (see Fig. 1). Iowa and Maine are 12 inclusive ranks apart but mapped in the same cell. The test focused on the joint probability that Maine and Iowa are mapped in the same or an adjacent cell and that Iowa and Tennessee do not appear in the same or adjacent cell. Thus, pr ¼ (7/36) * (29/ 36) ¼ 0.157. The 40 trials produced 20 instances of the hypothesized arrangement. Using the Gaussian approximation suggested by de Bodt, Cottrell, and Verleysen (2002) [21], the likelihood of this is p < 0.001. The average quantization error was 2.86, with a standard deviation of 0.117, producing a coefficient of 0.041, suggesting reasonable stability. The STSI 2012 mapping test looked at Nebraska (#33), South Carolina (#42) and South Dakota (#43). Although South Carolina and South Dakota are adjacent in the rankings, they are three steps removed in the SOM (see Fig. 2). Nebraska and South Dakota are ten inclusive ranks apart but mapped in the same cell. Again, the test uses the joint probability that Nebraska and South Dakota appear in the same or adjacent cell and that South Carolina and South Dakota do not appear in the same or adjacent cell. In this case, the 40 trials resulted in 22 instances of the hypothesized relationship, with a likelihood of p < 0.001. The average quantization error was 4.99, with a standard deviation of 0.258, producing a coefficient of 0.052dagain suggesting reasonable stability. Finally, two separate tests examined the follow-up SNEI 2002 mapping: Florida (#18) and Nevada (#32), and Georgia (#22) and Michigan (#23). Florida and Nevada are mapped in the same cell despite being 15 inclusive ranks apart; Georgia and Michigan are adjacent in rankings but three steps apart in the SOM. In 21 of the 40 trials, Florida and Nevada appeared in the same cell (p < 0.001, where the probability of this observation is 7/36 ¼ 0.194). Georgia and Michigan did not appear in the same or adjacent cell in 33 of the 40 trials (p < 0.001). The quantization error, standard deviation and coefficient for the first test were 2.44, 0.128 and 0.052, respectively. For the second test, those figures were 2.44, 0.138 and 0.057, respectively. Each of the tests suggests that the observed mapping results are highly unlikely to be random events and that the SOMs are reasonably stable. References [1] Feller I, Gamota G, Valdez W. Developing science indicators for basic science offices within mission agencies. Res Eval 2003;12:71e9.
76
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
[2] Earl L, Gault F. National innovation, indicators and policy. Cheltenham, UK: Edward Elgar; 2006. [3] Burton L. Indicators for state science and technology programs. Policy Stud J 1989;18:164e75. [4] Jones M, Guston DH, Branscomb LM. Informed legislatures: coping with science in a democracy. Lanham, MD: University Press of America by arrangement with the Center for Science and International Affairs, Harvard University, 1996. [5] Narin F, Hamilton KS, Olivastro D. The development of science indicators in the United States. In: Cronin B, Atkins HB, editors. The web of knowledge: a festschrift in honor of Eugene Garfield. Medford, NJ: Information Today, Inc; 2000. p.337e60. [6] Hansen JA. Technology innovation indicator surveys. In: Jankowski JE, Link AN, Vonortas NS, editors. Strategic research partnerships: proceedings from an NSF workshop. National Science Foundation, Division of Science Resources Studies, NSF 01e336; 2001. [7] Godin B. The numbers makers: fifty years of science and technology official statistics. Minerva 2002;40:375e97. [8] Godin B. The emergence of S&T indicators: why did governments supplement statistics with indicators? Res Policy 2003a;32:679e91. [9] Godin B. The rise of innovation surveys: measuring a fuzzy concept. Working paper No. 16. Project on the history and sociology of S&T de Que bec: INRS/Observatoire des Sciences et statistics. Universite des Technologies; 2003b. [10] State Science and Technology Institute [SSTI]. Special issue: a look at innovation indices and report cards. SSTI Weekly Digest. 2002; November 1st. [11] Feldman MS, March JG. Information in organizations as signal and symbol. Adm Sci Q 1981;26:171e86. [12] Kurunmaki L, Lapsley I, Melia K. Accountingization v. legitimation: a comparative study of the use of accounting information in intensive care. Manag Account Res 2003;14:112e39.
[13] Gormley Jr WT, Weimer DL. Organizational report cards. Cambridge, MA: Harvard University Press; 1999. [14] Pages ER, Toft GS. Benchmarking innovation. Econ Dev J 2009;8: 22e7. [15] Atkinson RD. Measuring up. Econ Dev J 2007;6:5e12. [16] Atkinson RD, Stewart LA. The 2012 state new economy index: benchmarking economic transformation in the states. The Information Technology and Innovation Foundation; 2012. Accessed 11 August 2014 from: http://www.itif.org/publications/2012-statenew-economy-index. [17] Klowden K, Wolfe M. 2012 state technology and science index: enduring lessons for the intangible economy. Milken Institute; 2013. Accessed 11 August 2014 from: http://statetechandscience. org/. [18] Alhoniemi E, Himberg J, Parhankangas J, Vesanto J. SOM toolbox for Matlab 5 (version 2). 2000. Accessed 11 August 2014 from: http:// www.cis.hut.fi/somtoolbox. [19] Kohonen T. Self-organizing maps. 3rd ed. Berlin Heidelberg: Springer; 2001. [20] Vesanto J, Alhoniemi E. Clustering of the self-organizing map. IEEE Trans Neural Netw 2000;11:586e600. [21] de Bodt E, Cottrell M, Verleysen M. Statistical tools to assess the reliability of self-organizing maps. Neural Netw 2002;15:967e78. [22] Davies DL, Bouldin DW. A cluster separation measure. IEEE Trans Pattern Analysis Mach Intell PAMI-1 1979:224e7. [23] Atkinson RD. The 2002 state new economy index: benchmarking economic transformation in the states. Progressive Policy Institute; 2002. Accessed 11 August 2014 from: http://www.itif.org/ publications/2002-state-new-economy-index. [24] Nguyen P, Haughton D, Hudson I. Living standards of Vietnamese provinces: a Kohonen map. Case Studies Business Ind Gov Stat 2009;2:109e13.
Contents lists available at ScienceDirect
Technology in Society journal homepage: www.elsevier.com/locate/techsoc
Identifying peer states to assess technology-based economic development David L. Schwarzkopf* Adamian 208, Bentley University, 175 Forest Street, Waltham, MA 02452, USA
a r t i c l e i n f o
a b s t r a c t
Article history: Received 4 June 2014 Received in revised form 11 August 2014 Accepted 18 August 2014 Available online
States often rely on 50-state “report cards” or indices to track their progress in technologybased economic development. Economic development agencies value these indices, published by independent consultancies, because they cut the costs of compiling data, compare states to one another and allow agencies to avoid charges of “cherry-picking” measures to serve their own purpose. The rankings of the states in these indices have appeal as they give policymakers and development agencies an idea of likely peer states and possible members of an aspirant group. Peers and aspirant groups provide a state with examples of alternative approaches to economic development, while allowing agencies to depict economic development activities in competitive terms for policymakers and legislators. Therefore, it is important that these comparisons be valid and, since the state's development policies affect the public, it is worthwhile for the citizenry to understand how agencies make these comparisons. Although rankings are easy to understand, manipulating multiple measures to produce a single number (the ranking) can distort important differences between states. The present research addresses whether these rankings provide a reliable source for peer and aspirant groups. Analysis of two popular scorecardsdthe State New Economy Index and the State Technology and Science Indexdshows that rankings provide only a broad picture of a state's relative standing. Clustering techniques based on self-organizing maps give a more refined view, better suited for policy analysis. © 2014 Elsevier Ltd. All rights reserved.
Keywords: Economic competition Economic development Science and technology reporting Scorecards Self-organizing maps
1. Introduction 1.1. The value of comparisons in technology-based economic development Sub-federal governmental units such as states, provinces or municipalities can look to other units in order to build a story of how they are doing in technology-based economic development (TBED). In particular, language in
* Tel.: þ1 781 891 2783; fax: þ1 781 891 2896. E-mail address: [email protected]. http://dx.doi.org/10.1016/j.techsoc.2014.08.004 0160-791X/© 2014 Elsevier Ltd. All rights reserved.
state reports on TBED shows that states try to identify peers while targeting other states as members of a group whose success in TBED they aspire to (see below for examples). Peers can foster a sense of real or implied competition that can be manipulated to motivate legislators and other policymakers to act in ways believed to increase the state's attractiveness to businesses, skilled workers, entrepreneurs and investors. Similarly, members of an aspirant group can serve as models to spur state agencies to increase political and business activity to improve economic development. Peers and aspirant groups also provide a state with examples of alternative approaches to economic development. Therefore, it is
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
important for a state to make valid comparisons to others in order to find peers and aspirant groups. Since the state's development policies affect the public, it is worthwhile for the citizenry to understand how state agencies go about making these comparisons. There are many ways that states can identify peersdfor example, geographic proximity. Over the past decade or two, however, various agencies have provided indices or “report cards” on the 50 states' TBED activities, resulting in state rankings. For reasons discussed below, these rankings hold the promise of an easy way to find peers using TBED data. But do rankings provide a reliable source for identifying peer and aspirant groups? Is there another reasonable way to identify what makes a good comparison for a given state? By addressing these questions, this research expands on academic inquiry into science and technology (S&T) reporting, which generally has examined national reporting issues (e.g., [1,2]), rather than state efforts (although [3,4] are exceptions). The discussion proceeds with background information on state TBED reporting, including the rise of 50-state indices. The methodology, findings and analysis of the study follow, before the paper concludes with suggestions for further academic research and practical action. 1.2. Background to state reporting on economic development Federal governments have measured their economic development status and, in particular, the effects of S&T on development, for at least the past 50 years [5e9]. In the U.S. more recently, states have tracked their progress in separate S&T reports and through 50-state scorecards or indices [10]. These indices include the State New Economy Index (published over various years by the Progressive Policy Institute, the Kauffman Foundation and the Information Technology and Innovation Foundation), the National Science Foundation's Science and Engineering Indicators, the Milken Institute's State Technology and Science Index, TechAmerica Foundation's Cyberstates, and the Development Report Card for the States, once published by the Corporation for Enterprise Development but now discontinued. Even though the extent to which S&T directly affect economic growth remains controversial and the role that a state government can play in fostering growth through S&T is at best indirect, these scorecards remain popular. This is understandable, since publication of such measures not only signals the value the state places on particular aspects of economic development [11], but also can lend credence to a state's claims to progress [see Ref. [12]]. However, this signaling effect means that those involved with these scorecards need to be aware of the state's intentions as a reporter and the audience of users [13,14]. In fact, among the reasons that 50-state indices published by independent agencies are valued is that they allow states to avoid charges of “cherry-picking” measures to serve the reporter's purpose. In addition, it is costly for a state to compile data on its own. Savings come from the use of publicly available data, such as those presented in these scorecards.
69
1.3. The promise and perils of rankings Beyond cost savings, these indices provide comparable data, which is not the norm in individual states' S&T reporting.1 To focus on this comparability, most 50-state scorecards provide rankings that appear easy to understand2done either is or is not in the top ten, the top quintile, or such. Ranking on multiple measures, such as those used in the scorecards, is complicated because a state can score “higher” or “lower” on any single given measure among the many used. To simplify the picture, scorecard issuers combine multiple measures into a final singlenumber ranking. While some authors provide detail on how to interpret rankings and the limitations of the same (e.g., [15]), it is the overall ranking that draws media and agency attention. That is, rankings tempt readers not to look at similarity of economic or environmental conditions, but to look at “who's ahead” or “who's catching up.” Evidence of this power of rankings is plentiful, from state reports (e.g., “In the Technology and Science Workforce Composite Index, Georgia fell 19 spots in 4 years”)3 to issuers' web sites (e.g. “Can anyone catch Massachusetts?”).4 The use of rankings is thus separate from the question of the degree to which the measures employed are valid, since the public and policymakers take the rankings as published. Their use raises concerns about whether they can tell a state who the state's peers are and who is a member of the state's aspirant group. By comparing rankings to another analytical method using the same data, this research addresses that issue.
2. Method The method of inquiry proceeded in two phases. In the first, data from two 50-state indices were used to produce self-organizing maps (SOMs)dvisualization tools that help one to judge similarities among multi-measure data sets. The second stage used these SOMs to generate clustersdgroupings of states with similar characteristics based on the TBED data. The 2012 edition of two popular scorecards provided the data for this study: the State New Economy Index (“SNEI 2012”) [16] and the Milken Institute's State Technology and Science Index (“STSI 2012”) [17]. SNEI began publishing in 1999; STSI in 2002. Both indices are used in individual state
1 A study of 31 reports issued between 2004 and 2008 by agencies in 24 states found 1160 different measures used, only one of which appeared in more than half of the reports. Less than one percent of the measures appeared in 10 or more reports, with 833 measures (71.8%) appearing only once. Yet 15 of the 24 states compared themselves to others on selected measures. See Flynn, PM and Schwarzkopf, DL. Assessing State Reports on Technology-Based Economic Development: Lessons for Benchmarking. Report to the National Science Foundation (award number 0617112); 2010. 2 The National Science Foundation's Science and Engineering Indicators groups states into quartiles, rather than rank them. 3 Technology Association of Georgia. Invigorating Georgia's Technology Industry, http://www.tagonline.org/files/marketing_collateral.pdf, accessed 11 August 2014. 4 State Technology and Science Index home page, http:// statetechandscience.org/, accessed 11 August 2014.
70
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
Table 1 State ranks by scorecard. Rank
SNEI 2012
STSI 2012
SNEI 2002
Rank
SNEI 2012
STSI 2012
SNEI 2002
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
MA DE WA CA MD VA CO UT CT NJ NY NH MN OR VT AZ TX GA MI IL FL PA RI ID NC
MA MD CO CA WA VA DE UT CT NH PA VT MN NY NJ AZ RI TX IL OR NM NC MI KS WI
MA WA CA CO MD NJ CT VA DE NY OR UT MN TX NH AZ IL FL PA ID RI GA MI MO ME
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
NV ME AK KS NM WI OH MO ND NE HI MT IA TN SC WY IN SD LA KY AL OK AR WV MS
GA IN MO AL OH IA ND NE ID HI MT TN FL ME OK AK SC SD LA WY KY NV WV AR MS
NC NM VT KS OH AK NV NE OK HI IN MT IA TN WI SC KY SD ND LA WY AL AR MS WV
Sources: Atkinson (2002, p.7) [23]; Atkinson and Stewart (2012, p. 11) [16]; Klowden and Wolfe (2013, p. 2, as adjusted to consider omitted variablesdsee Appendix 2) [17]. Note. States are identified by their two-letter postal code. For SNEI, Atkinson and Stewart (2012, p. 11) [16] caution, “Due to changes in methodology, changes in rank from previous editions may not positively reflect changes in economic structure.”
economic development reports, describe their methodology clearly and allow free public access to their raw data. SNEI 2012 ranks states based on a composite of 26 variables. The index's authors establish weights in an effort to compensate for the likelihood of high correlations among the variables. The analysis below preserves those weights. The STSI 2012 data set consists of 79 measures grouped into 5 composite categories that are further weighted to produce a single score.5 Because of missing state data, five measures were dropped for the present analysis. It was also necessary to adjust STSI 2012's weightings across the composites, which led to minor changes from the published rankings. See Appendix 1 to this paper for details. State rankings from each index appear in Table 1. The SOM Toolbox application [18] produced a selforganizing map [19] based on each index's data. The selforganizing principle behind the map uses a case's (here, a state's) multiple characteristics (measures) to form an input vector. The SOM chooses reference or codebook vectors based on the distribution of all the input cases, matching each case to the codebook vectors. This iterative process calculates proximity by Euclidean distance and results in an organization that has not relied on any prior sorting scheme, hence is “self-organizing.” Groups of similar cases and a sense of proximity between cases are visually displayed and
5
For details of the measures, see Refs. [16,17].
can be subjected to further analysis through clustering techniques, as described below. See Kohonen (2001) [19] for technical details and a lengthy list of references to academic research using SOMs. The approach used here is similar to that introduced by Vesanto and Alhoniemi (2000) [20]. The visual displays of the SOMs for SNEI 2012 and STSI 2012 appear in Figs. 1 and 2, respectively.6 3. Findings 3.1. What do the SOMs indicate? Both vector quantization error (Q) and topological error (te) enter into the assessment of the goodness-of-fit of an SOM. Q measures the overall difference between the given input vectors and their closest reference vector (Kohonen, 2001, section 1.5) [19]. Topological error measures the fit of overall groupings by showing the proportion of input vectors for which the best-matching reference vector and the second-best-matching reference vector are not neighbors (Kohonen, 2001, 161) [19]. For the SNEI 2012 mapping, Q ¼ 3.29 and te ¼ 0.00; for the STSI 2012 map, Q ¼ 5.63 and te ¼ 0.00. The topological error suggests a reasonable mapping; quantization errors enter into the stability analysis described in Appendix 2. Figs. 1 and 2 show the resulting SOM visualizations, called U-matrices, with and without labels. A U-matrix consists of two kinds of cells: placement cells (areas on the matrix where the cases are placed) and intervening cells (areas between placement cells). In the figures, placement cells are labeled on the right-hand side with the state's postal code and ranking for the given scorecard. Intervening cells appear only on the left-hand display in each figure. The shade of a placement cell can be interpreted as the average distance of that cell (here, the state or states in the cell) from adjacent placement cellsdthe lighter the shade, the closer the proximity. Intervening cells separate rows of placement cells as well as adjacent placement cells and thus provide a broad view of large-cluster boundaries. The shade of intervening cells gives a sense of the distance between states, based on the vector of measures, again with lighter shades suggesting more similarity. States sharing a given placement (labeled) cell are most similar to each other. While some placements are what one would expect from the rankings (e.g., for SNEI 2012, #12 New Hampshire and #13 Minnesota being paired, or the pairings of some top-ranked and bottom-ranked states), other results are unexpected in that the grouped states are distant on the rankings (e.g., SNEI 2012's #27 Maine with #38 Iowa or STSI 2012's #31 Iowa with #43 South Dakota). This suggests a limit to identifying a peer group based on rankings, which the analysis below explores. 3.2. Are the SOMs reliable? Observed differences can arise because of an index's scorekeeping method or because of problems with the
6 To conserve space, mappings for the individual variables from each scorecard are omitted. These are available from the author upon request.
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
Fig. 1. SOM based on SNEI 2012.
Fig. 2. SOM based on STSI 2012.
71
72
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
Table 2 SNEI 2012 and STSI 2012 clusters (with state rankings). Cluster
SNEI 2012
STSI 2012
1
Tennessee (39), South Carolina (40), Louisiana (44), Kentucky (45), Alabama (46), Arkansas (48), West Virginia (49), Mississippi (50) North Dakota (34), Montana (37), Wyoming (41), Indiana (42), South Dakota (43), Oklahoma (47) Nevada (26)
Tennessee (37), Oklahoma (40), Louisiana (44), Wyoming (45), Kentucky (46), West Virginia (48), Arkansas (49), Mississippi (50) Hawaii (35), Florida (38), Maine (39), Alaska (41), South Carolina (42), Nevada (47) Iowa (31), North Dakota (32), Nebraska (33), Montana (36), South Dakota (43) Wisconsin (25), Indiana (27), Missouri (28), Ohio (30)
2 3 4 5 6 7 8 9 10 11
North Carolina (25), Alaska (28), Kansas (29), Ohio (32), Missouri (33), Nebraska (35) Maine (27), Wisconsin (31), Hawaii (36), Iowa (38) Arizona (16), Texas (17), Georgia (18), Michigan (19), Florida (21), Pennsylvania (22), New Mexico (30) Oregon (14), Vermont (15), Illinois (20), Rhode Island (23), Idaho (24) [an empty cluster] Massachusetts (1), Washington (3), California (4), Maryland (5), Virginia (6), Colorado (7), Utah (8) New York (11), New Hampshire (12), Minnesota (13) Delaware (2), Connecticut (9), New Jersey (10)
Texas (18), Illinois (19), Oregon (20), Kansas (24), Georgia (26), Idaho (34) North Carolina (22), Michigan (23), Alabama (29) Pennsylvania (11), Vermont (12), New York (14), Rhode Island (17), New Mexico (21) Utah (8), New Hampshire (10), Minnesota (13), Arizona (16) Massachusetts (1), Maryland (2), Colorado (3) Virginia (6), Delaware (7) California (4), Washington (5), Connecticut (9), New Jersey (15)
Note. Clusters are numbered based on their position on the SOM (see Figs. 1 and 2), roughly from upper-left to lower-right. Cluster numbers do not imply “better” or “worse” status.
SOM's mapping. Tests using a bootstrap technique described by de Bodt, Cottrell and Verleysen (2002) [21] minimize the chance of the latter. For each of the two maps, tests were run on the placement of states that are close on the rankings but not close on the mapdthat is, not located in the same or an adjacent celldand between states close on the map but not close on the rankings. These stability tests, described in detail in Appendix 2, showed it was highly unlikely (p < 0.001) that the observed mapping outcomes are random. Thus the differences between the mapped positions and the rankings are highly unlikely to be caused by the mapping procedure, but represent a different relationship than that presented in the indices. 3.3. What clusters result from the SOMs? A k-means clustering technique helps derive a better sense of peer groups on each map. Clustering used 25 trials, each with 100 randomized vector-starts, since cluster designation is sensitive to the location of the starting vector, and with the maximum number of clusters set at 20. For each trial, the number of clusters that produced the lowest Davies-Bouldin (DB) index [22] was noted. Over the 25 trials on each map, 11 clusters most frequently produced the lowest DB index for each. Table 2 lists the clusters and the respective index's ranking of the states. 4. Analysis: do the clusters improve the notion of a state's peers? To compare the contents of the clusters with the notion of “peer” derived from the rankings, we can look at the inclusive range of rankings within the clusters, intervals between rankings for the states within each cluster, and discontinuities between clusters. The clusters show a wide average inclusive range of rankings. For example, using SNEI 2012's ranks, there are 12 places, inclusive of the endpoints, from #39 Tennessee to
#50 Mississippi in Cluster 1. Again, there are 17 inclusive places from STSI 2012's #18 Texas to #34 Idaho in that index's Cluster 5. The average of the (non-zero) clusters' inclusive ranges is 9.6 for SNEI 2012 and 9.8 for STSI 2012. Thus, a peer may be farther away from a given state than the ranked order may suggest. Looking at the intervening ranks for the states in a cluster provides another perspective on whether this clustering method improves on rankings in determining peers. Starting with the highest-ranked state in the cluster, one can count the intervening ranks to the next-highestranked state and continue this for consecutive pairs in the cluster. For example, in the STSI 2012 Cluster 1, there are two intervening ranks between #37 Tennessee and #40 Oklahoma (namely, #38 and #39), and three intervening ranks between Oklahoma and #44 Louisiana. Based on this metric, 28 of the 40 consecutive pairs (70%) in the SNEI 2012 clusters and 24 out of the 39 (62%) in the STSI 2012 have zero or one rank intervening. This suggests that the rankings seem to provide a reasonable, if broad, start on determining peers and aspirant groups. Next, note the substantial discontinuities between the rankings and the clusters. For example, for SNEI 2012, the “middle” clusters (numbers 4 through 7), contain the following range of rankings: #14e#24; #16e#30; #27e#38; #25e#35. For STSI 2012, the range in those clusters is: #11e#21; #22e#29; #18e#34; #25e#30. In summary, it appears that rankings provide a mixed picture for a state looking for peers or members of an aspirant group. While it is possible that a state can simply look one or two steps to either side of its ranking to find its peers, it is not necessarily the case that those states are similar based on TBED data and it is quite likely that it will have peers that its ranking will not reveal. At the same time, a state wanting to “climb the rankings” may find that even moving up as many as five or six ranks leaves it in the same peer group as before. This is not to minimize the effect that a single number has on public attention. “We're
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
73
Fig. 3. SOM based on SNEI 2002.
number 14” will sound better than “We're number 20.” But the reality may be that both 14 and 20 are highly similar. Rankings can also overshadow some more-nuanced differences. For example, in both mappings, the top ten ranked states are divided into two (SNEI 2012) or four (STSI 2012) smaller groupings. Similarly, the bottom ten states in both scorecards fall into two different clusters. Note also that the contents of the clusters are not consistent from index to index. Of the 122 possible pairings of states within all of the clusters generated by SNEI 2012, only 33 of those pairs (27%) appear in the clusters formed from STSI 2012.7 This underscores the sensitivity of the notion of “peer” to the kinds of measures used in the indices. Finally, it is possible that the observed relationship between clusters and rankings is a unique product of the 2012 indices. To test this, the same tests were run on the SNEI scorecard of ten years earlier (“SNEI 2002”) [23]. See Table 1 for SNEI 2002's state rankings. The resulting SOM appears in Fig. 3 (Q ¼ 2.72; te ¼ 0.00). Again, note the unexpected pairings (e.g., #18 Florida and #32 Nevada), as well as a
7
The 33 include the 15 pairs formed from the set {Arkansas, Kentucky, Louisiana, Mississippi, Tennessee, West Virginia}, the 3 formed from {Montana, North Dakota, South Dakota}, the 3 from {Idaho, Illinois, Oregon}, the 3 from {Colorado, Maryland, Massachusetts}, and the following 9 individual pairs: OklahomaeWyoming; MissourieOhio; HawaiieMaine; GeorgiaeTexas; New MexicoePennsylvania; Rhode IslandeVermont; CaliforniaeWashington; MinnesotaeNew Hampshire; and ConnecticuteNew Jersey.
number of groupings that are closer to what the rankings would lead one to expect.8 The k-means cluster technique suggested 12 clusters, as listed (with SNEI 2002 rankings) in Table 3.9 Here 18 of the 35 consecutive pairings (51%) have zero or one intervening rank within their cluster; the average inclusive range of the clusters is 10.5; and discontinuities similar to those seen in the 2012 indices appear in the middle clustersdall of which suggests the previous observations are not unique to the 2012 indices. Stability tests of the SOM (described in Appendix 2) again show it is highly unlikely (p < 0.001) that the SOM groupings are random. While it is tempting to trace a state's trajectory from 2002 to 2012 on the SNEI mappings, the validity of such a move is questionable because of changes over that period in the metrics and methods. This adds to the need for a way to map consistently over time, as mentioned in the following section. 5. Discussion and conclusion This work is not intended to criticize the measures used in the scorecards or to compare rankings across scorecards, but to see whether these rankings provide readers with a
8
The 2002 SNEI ranks were based on 21 measures [23]. The difference in the number of clusters between the 2002 and 2012 SNEI mappings is purely a function of the clustering algorithm. 9
74
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
Table 3 SNEI 2002 clusters (with state rankings). 1
Oklahoma (34), South Carolina (41), Kentucky (42), Louisiana (45), Alabama (47), Arkansas (48), Mississippi (49) 2 Alaska (31), Montana (37), Iowa (38), South Dakota (43), North Dakota (44), West Virginia (50) 3 Georgia (22), North Carolina (26), Hawaii (35) 4 Missouri (24), Maine (25), Kansas (29), Ohio (30), Nebraska (33), Indiana (36) 5 Florida (18), Nevada (32) 6 Pennsylvania (19), Michigan (23) 7 Utah (12), Texas (14), Arizona (16) 8 Oregon (11), Minnesota (13), Illinois (17) 9 Idaho (20), Rhode Island (21), New Mexico (27), Vermont (28) 10 Massachusetts (1), Washington (2), California (3), Colorado (4), Maryland (5), Virginia (8) 11 Connecticut (7), New Hampshire (15) 12 New Jersey (6), Delaware (9), New York (10) Note. Clusters are numbered based on their position on the SOM (see Fig. 3), roughly from upper-left to lower-right. Cluster numbers do not imply “better” or “worse” status.
reasonable sense of peer and aspirant states. While rankings can give a state's economic development agents a rough guide to peers and aspirants, SOMs offer a more refined approach. The mapping itself points to most-similar peers; dividing the map into clusters reveals other similar states while showing collections of “near” neighbors that states may look to as members of their aspirant group. Mapping, expanded by clustering, thus provides states with a better idea of group membership than the scorecards' presentations based on rankings. What explains the observed discrepancies? The scorecards' methodology appears to magnify the importance of small differences in individual measures. For example, for each of its five composites, STSI 2012 ranks the states on each measure, translates that rank into a score, averages these scores and then ranks these averaged scores. Thus, a small difference on a particular measure (e.g., Initial Public Offering Proceeds as a Percentage of Gross State Product) leads to a difference of the rank based on measure, but since the interval between successive scores is constant no matter the size of the interval between measures, the difference between the averaged converted composite scores may exaggerate the initial difference on the measure. When the composites are compiled, these differences distort rankings. The mapping and clustering, on the other hand, sort states by similarities using distance-algorithms and thus do not exaggerate small differences. Further, rather than reducing multiple measures down to one number for a ranking, the mapping preserves the complex n-item vector of measures. The trade-off, of course, is that it is easier to describe a state via rankings than to interpret the detail of one state's vector compared to another. A closer analysis of each cluster will provide a better understanding of what constitutes membership in that cluster, but this in turn will provoke a more detailed and nuanced discussion among policymakers than a simple call to “climb the rankings” does. This closer analysis is what states need in order to address the question of what policies or practices they should adapt from others to improve their own TBED activity. Identifying immediate peers is easy: look within a cluster. Understanding why these are peers is more difficult. Similarly, looking at
adjacent clusters provides a rough picture of aspirant (or “potential challenger”) groups. However, a look at the details of how a state stands on related measures (e.g., in technology-based labor force deployment or in research and development funding) is required to provide the guidance policymakers need to advance TBED.10 The attraction to compare oneself with others is strong. While this research has shown how to improve the static view of a state's economic development status, mapping also suggests ways to view changes in state relationships over time. However, the SNEI authors (Atkinson, 2002, p. 7; Atkinson & Stewart, 2012, p. 11) [16,23] warn against comparing state positions across years of their reports because of changes in reporting methodology. Thus, there is a question of how to design a simple, reasonable method for displaying a state's development in comparison to other states over time. That design forms the next phase of this research agenda. Most importantly, by shifting the focus from a single index number or rank, the more detailed groupings offered by SOMs allow states to resist the allure of considering economic development as a zero-sum game where one state's gains in the rankings come at the cost of another state's losses. For example, a state should feel the benefit of increased research and development spending within its borders, even if all states experience increases in such spending and whether the state's increase is enough to move it ahead in the rankings or not. In short, policymakers and legislators can then feel free to look at economic development as an endeavor in which all states can show improvement. The aim, after all, should not be to “catch Massachusetts” or “jump five places in the rankings,” but to improve citizens' living conditions within a state's economic context.
Acknowledgments This paper builds on the author's unpublished research with Patricia Flynn into state reports on technology-based economic development, funded by the National Science Foundation (“NSF”; award number 0617112). In particular, material in the introduction, technical descriptions of selforganizing maps and the analytical approach are derived from that study, although different data sources are used. The author thanks Patricia Flynn and two anonymous reviewers for their suggestions, Dominique Haughton for her insights into self-organizing maps, and the NSF and Bentley University for their financial support of the earlier study. Neither funding group influenced the research data collection, analysis or commentary. Appendix 1. Data adjustments to STSI 2012 The STSI 2012 scorecard gathers 79 measures over five areas: Research & Development Inputs; Risk Capital and
10 As an aside, it is worth considering the value of this technique to provide an approach to other areas where rankings are popular, but which require more in-depth study to determine policy direction, such as the status of universities, hospitals or nursing homes. Thanks to an anonymous reviewer for raising this point.
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
Entrepreneurial Infrastructure; Human Capital Investment; Technology & Science Workforce; and Technology Concentration & Dynamism [17]. For each measure, states are ranked based on their raw data, with the top state given a score of 100; the second state, 98; the third, 96; and so on to the fiftieth, which receives a score of 2. When states are missing data for a measure, they generally receive no score, but in some instances are given a score corresponding to the rank below the last-ranked state to report data on the measure. All non-zero scores within an area are averaged to produce five composite scores for each state. These are weighted to produce the final score; the overall ranking is based on this final score. The present analysis omitted the following five measures that had a substantial number of states not reporting data: Venture Capital Investment in Nanotechnology as a Percent of Gross State Product (GSP); Sum of Equity Invested in Green Technology per $100,000 GSP; Venture Capital Investment in Clean Technology as a Percent of GSP; Intensity of Agricultural Engineers; and Intensity of Biomedical Engineers. This left 74 measures for the analysis. Further, an examination of STSI 2012's method of computing scores based on ranks revealed an apparent inconsistency. In some instances, the raw data reported for a measure was not correctly transferred to STSI's summary for ranking. Where possible, these omissions were corrected and scores were assigned consistent with STSI 2012's overall methodology. These adjustments produced only small changes in the overall rankings, with no state moving more than two positions. Note that none of the analysis or commentary is sensitive to these differences. The present analysis uses the adjusted rankings. Appendix 2. SOM stability tests The aim of SOM stability tests is to minimize the chance that the observed SOMs show positions that are not stabledthat is, that are no better than chance happenings. As noted in the narrative, the tests examine the mapped positions of states that are “close” in the SOM but “distant” in rankings and states that are “distant” in the SOM but “close” in the rankings. The tests rely on a bootstrap technique developed by de Bodt, Cottrell and Verleysen (2002) [21] that has been used in other SOM-based research (e.g., [24]). A single test under this method uses a 50-state data set that includes selected, “fixed” states (the target of the comparisons). The remaining positions in the set are open, to be populated randomly with an equal likelihood by any of the “nonfixed” states. Each test re-populates the open positions in the data set. Each test results in a SOM. The analysis used 40 tests (mappings) with topological errors equal to zero. See the description of topological error (te) in the narrative. For each mapping, the shortest path between the target (fixed) states was counted, with target states mapped in the same cell having a distance of zero. As in de Bodt, Cottrell, and Verleysen (2002) [21], these distances are compared to those produced by chance by analyzing them in a binomial distribution, where the probability of two states being
75
“highly similar” is considered as the likelihood of the two being in the same (distance ¼ 0) or an adjacent (distance ¼ 1) cell. The probability is thus 7 divided by 36dthat is, the same cell plus six adjacent cells in the hexagonal map divided by the number of cells in the overall SOM. Finally, a measure similar to the coefficient of variation provides a sense of the stability of the mapping: the standard deviation of the 40 trials' quantization errors divided by the average of those quantization errors. A smaller number indicates greater stability. See the narrative for a description of quantization error (Q). A summary of the tests follows. To test the 2012 SNEI mapping, the comparison was among Maine (#27), Iowa (#38) and Tennessee (#39). Iowa and Tennessee are adjacently ranked but four cells removed in the SOM (see Fig. 1). Iowa and Maine are 12 inclusive ranks apart but mapped in the same cell. The test focused on the joint probability that Maine and Iowa are mapped in the same or an adjacent cell and that Iowa and Tennessee do not appear in the same or adjacent cell. Thus, pr ¼ (7/36) * (29/ 36) ¼ 0.157. The 40 trials produced 20 instances of the hypothesized arrangement. Using the Gaussian approximation suggested by de Bodt, Cottrell, and Verleysen (2002) [21], the likelihood of this is p < 0.001. The average quantization error was 2.86, with a standard deviation of 0.117, producing a coefficient of 0.041, suggesting reasonable stability. The STSI 2012 mapping test looked at Nebraska (#33), South Carolina (#42) and South Dakota (#43). Although South Carolina and South Dakota are adjacent in the rankings, they are three steps removed in the SOM (see Fig. 2). Nebraska and South Dakota are ten inclusive ranks apart but mapped in the same cell. Again, the test uses the joint probability that Nebraska and South Dakota appear in the same or adjacent cell and that South Carolina and South Dakota do not appear in the same or adjacent cell. In this case, the 40 trials resulted in 22 instances of the hypothesized relationship, with a likelihood of p < 0.001. The average quantization error was 4.99, with a standard deviation of 0.258, producing a coefficient of 0.052dagain suggesting reasonable stability. Finally, two separate tests examined the follow-up SNEI 2002 mapping: Florida (#18) and Nevada (#32), and Georgia (#22) and Michigan (#23). Florida and Nevada are mapped in the same cell despite being 15 inclusive ranks apart; Georgia and Michigan are adjacent in rankings but three steps apart in the SOM. In 21 of the 40 trials, Florida and Nevada appeared in the same cell (p < 0.001, where the probability of this observation is 7/36 ¼ 0.194). Georgia and Michigan did not appear in the same or adjacent cell in 33 of the 40 trials (p < 0.001). The quantization error, standard deviation and coefficient for the first test were 2.44, 0.128 and 0.052, respectively. For the second test, those figures were 2.44, 0.138 and 0.057, respectively. Each of the tests suggests that the observed mapping results are highly unlikely to be random events and that the SOMs are reasonably stable. References [1] Feller I, Gamota G, Valdez W. Developing science indicators for basic science offices within mission agencies. Res Eval 2003;12:71e9.
76
D.L. Schwarzkopf / Technology in Society 39 (2014) 68e76
[2] Earl L, Gault F. National innovation, indicators and policy. Cheltenham, UK: Edward Elgar; 2006. [3] Burton L. Indicators for state science and technology programs. Policy Stud J 1989;18:164e75. [4] Jones M, Guston DH, Branscomb LM. Informed legislatures: coping with science in a democracy. Lanham, MD: University Press of America by arrangement with the Center for Science and International Affairs, Harvard University, 1996. [5] Narin F, Hamilton KS, Olivastro D. The development of science indicators in the United States. In: Cronin B, Atkins HB, editors. The web of knowledge: a festschrift in honor of Eugene Garfield. Medford, NJ: Information Today, Inc; 2000. p.337e60. [6] Hansen JA. Technology innovation indicator surveys. In: Jankowski JE, Link AN, Vonortas NS, editors. Strategic research partnerships: proceedings from an NSF workshop. National Science Foundation, Division of Science Resources Studies, NSF 01e336; 2001. [7] Godin B. The numbers makers: fifty years of science and technology official statistics. Minerva 2002;40:375e97. [8] Godin B. The emergence of S&T indicators: why did governments supplement statistics with indicators? Res Policy 2003a;32:679e91. [9] Godin B. The rise of innovation surveys: measuring a fuzzy concept. Working paper No. 16. Project on the history and sociology of S&T de Que bec: INRS/Observatoire des Sciences et statistics. Universite des Technologies; 2003b. [10] State Science and Technology Institute [SSTI]. Special issue: a look at innovation indices and report cards. SSTI Weekly Digest. 2002; November 1st. [11] Feldman MS, March JG. Information in organizations as signal and symbol. Adm Sci Q 1981;26:171e86. [12] Kurunmaki L, Lapsley I, Melia K. Accountingization v. legitimation: a comparative study of the use of accounting information in intensive care. Manag Account Res 2003;14:112e39.
[13] Gormley Jr WT, Weimer DL. Organizational report cards. Cambridge, MA: Harvard University Press; 1999. [14] Pages ER, Toft GS. Benchmarking innovation. Econ Dev J 2009;8: 22e7. [15] Atkinson RD. Measuring up. Econ Dev J 2007;6:5e12. [16] Atkinson RD, Stewart LA. The 2012 state new economy index: benchmarking economic transformation in the states. The Information Technology and Innovation Foundation; 2012. Accessed 11 August 2014 from: http://www.itif.org/publications/2012-statenew-economy-index. [17] Klowden K, Wolfe M. 2012 state technology and science index: enduring lessons for the intangible economy. Milken Institute; 2013. Accessed 11 August 2014 from: http://statetechandscience. org/. [18] Alhoniemi E, Himberg J, Parhankangas J, Vesanto J. SOM toolbox for Matlab 5 (version 2). 2000. Accessed 11 August 2014 from: http:// www.cis.hut.fi/somtoolbox. [19] Kohonen T. Self-organizing maps. 3rd ed. Berlin Heidelberg: Springer; 2001. [20] Vesanto J, Alhoniemi E. Clustering of the self-organizing map. IEEE Trans Neural Netw 2000;11:586e600. [21] de Bodt E, Cottrell M, Verleysen M. Statistical tools to assess the reliability of self-organizing maps. Neural Netw 2002;15:967e78. [22] Davies DL, Bouldin DW. A cluster separation measure. IEEE Trans Pattern Analysis Mach Intell PAMI-1 1979:224e7. [23] Atkinson RD. The 2002 state new economy index: benchmarking economic transformation in the states. Progressive Policy Institute; 2002. Accessed 11 August 2014 from: http://www.itif.org/ publications/2002-state-new-economy-index. [24] Nguyen P, Haughton D, Hudson I. Living standards of Vietnamese provinces: a Kohonen map. Case Studies Business Ind Gov Stat 2009;2:109e13.