- Email: [email protected]
Policy evaluation quality
Regional
Science and
Urban
Economics
23 (1993) 51-65.
Policy evaluation A quasi-experimental subsidies in Sweden
North-Holland
quality study of regional
employment
Peter Bohm and Hans Lind Stockholm
Received
Unicersity,
S-106 YI Stockholm.
Sweden
May 1990, final version received
April 1991
Policy evaluation methods differ as to the nature of their assumptions about the hypothetical policy-off case. We argue that policy evaluation quality can be raised by using a quasiexperimental method when feasible. In addition, the quality of evaluating policy changes can be raised by designing and publicizing the evaluation method, specified in detail, before any data on target variables are available. These policy evaluation properties characterize the study presented here. It concerns a signilicant reduction of payroll taxes in a depressed region of Sweden in order to bolster employment. The evaluation produced clear evidence of a complete lack of employment effects.
1. Introduction The purpose here is to evaluate the effects on the main target variable employment - of a change in regional policy towards subsidizing labour instead of capital, using an improved form of policy evaluation. Specifically: (1) We use a quasi-experimental evaluation method, which, as we shall argue, requires less questionable assumptions than traditional evaluation methods. (2) The evaluation method was designed and publicized before any data for the policy-on period were available [Bohm and Lind (1984)]. Any adjustment of the analytical approach that one may wish to make would then easily be detected and call for an explicit explanation. The reason for using this approach - and for advocating it whenever possible - is that a number of difficult choices have to be made when designing an evaluation; then, if data already exist, knowledge of the effects of these choices can influence the choice made by even the most serious evaluator. (3) By starting the work on evaluation at the same time as the policy is introduced, quality can be raised also in other respects. It is easier to check Correspondence
Stockholm,
to:
P. Bohm,
Department
of Economics,
Stockholm
University,
Sweden.
016Wl462/93/$06.00
(1) 1993-Elsevier
Science Publishers
B.V. All rights reserved
S-106
91
P. Bohm
52
and H. Lind.
PO/&~ rruluation
quality
data when they are new, and various events that may affect the target variables can more easily be investigated. In Sweden as in many other countries, regional policy has been dominated by capital subsidies. Economists have long argued in favour of using labour subsidies when the basic policy objective is to increase employment opportunities in depressed regions [e.g. Lind and Serck-Hansen (1972)]. In Sweden, a significant step in this direction was taken in 1984, when the payroll tax for Norrbotten (the northernmost county in Sweden) was reduced by about onethird, implying a labour-cost reduction by some 7:;. The employment effects of this new policy are investigated in this paper and the results show the extent to which the theory-based expectations of the effect of this type of policy actually materialized. The outline of the paper is as follows. In section 2 we give a short theoretical background, describe the experimental approach used and compare it with more traditional methods. In section 3 we describe the details of the method used, in particular how control groups and ‘twin-firms were chosen. The results are presented in section 4. Alternative interpretations of the results and policy implications are discussed in the final section. 2. The experimental
approach compared with other methods
If we look at a perfectly competitive product market, the effect on employment of a subsidy on labour is determined by the resulting change in output and the substitution of labour for capital. The change in output depends upon the slope of the marginal cost curve around the old equilibrium, which in turn depends upon technology and the change in input prices that input changes may give rise to. If the product is sold under less than perfectly competitive conditions, the output effect also depends upon how elastic the demand for the product is. A wage subsidy would have a substantial effect on employment if there is (1) a high elasticity of demand and a flat marginal cost curve, and/or (2) a high elasticity of substitution between labour and capital. Evaluating the actual employment effect of a wage subsidy in a region with a considerable number of different firms, we are rarely able to estimate the underlying factors now mentioned. Rather, we have to settle for a more direct estimate of employment changes. We may distinguish among three approaches here: (quasi-)experiments, shift-share analysis and multiple regression analysis.
2. I
The experimental
upproach
Controlled experiments, where a group of subjects (regions) are randomly allocated to an experiment group and a control group, are for various
P. Bohm and H. Lind, Policy evaluation
quality
53
reasons rare in the field of economic policy. In practice, the experimental approach to (regional) policy evaluation will focus on the possibility to identify, at least reasonably well, a control group after the policymaker has decided that a certain policy should be introduced in a certain region. Since some differences between the two groups are likely to remain, for example as regards distances to other regions, the use of a control period is a natural part of this type of experimental approach.’ This kind of experimental approach may seem paradoxical in the sense that if the policy is applied to a certain region and not to another, then the implications would seem to be that the two regions actually are significantly different and that therefore the second region cannot be used as a control group.2 In many situations, however, this problem may not be so severe: (1) The explanation why one region gets the policy while another does not may be wholly ‘political’. (2) The policy may be limited, for administrative reasons, to an area which is smaller than a rather homogeneous problem area. If so, parts of the problem area are left which fit the requirements for a control group. (This is true in our particular case, where the tax reduction was accorded to Norrbotten county. There is no county like the county of Norrbotten but for each part of Norrbotten there exist parts of other counties that could form a control group.) 2.2. Shift-share
analysis
The standard method for evaluating regional policy has been shift-share analysis. The basic idea is that without any change in policy, the region’s share of national employment would continue to develop as in the pre-policy period. If the development of the region’s share changes during the policy-on period, this change is taken to reveal the policy effect.3 The problematical aspect of shift-share analysis is use of the region’s share of national employment when estimating what would have happened if the policy had not been introduced. The development of this share can change for a number of other reasons. In Sweden, for example, there have been ‘As far as we know the only earlier attempt to use an experimental approach when evaluating regional policy is that of Isserman and Merritield (1982). Important ditTerences between their study and ours are: (a) the way in which the control group of regions is selected (we base the choice on similarities in certain characteristics while Isserman and Merrifield base it on similarities in growth rates in a pre-policy period); (b) in addition to comparisons for manufacturing industry as a whole and for large industries, we compared a group of ‘twin-firms in the two regions; and (c) as already indicated, we specified the evaluation method in detail before any data for the policy period were available. *Folmer (1986) uses this as one argument against experimental approaches. 3A description of the method, comments on its weaknesses and further references may be found in, for example, Moore and Rhodes (1973), Bartels et al. (1982), Diamond and Spence (1983). Tervo and Okko (1983) Armstrong and Taylor (1985) and Folmer (1986).
recurring periods of strong metropolitan growth during the post-war period. When using shift-share analysis, the measured effect of changes in regional policy may depend critically on the time at which such changes are made in relation to the time when the more general growth trend changes. The ‘trend’ of the control group used in the experimental approach is likely to be a better indicator of what would have happened in the no-policy case than the national trend. Shift-share analysis is usually carried out at the industry level. At this level, however, it is very likely that important differences is characteristics remain between the region subject to policy change and the nation as a whole. These differences in characteristics at the industry level can be expected to be smaller when a control group with similar basic characteristic is used. 2.3. Multiplt~ rrgression
unu1~~si.s
Using multiple regression analysis, the effect of a certain regional policy is measured by specifying the relation between the target variable and all the characteristics of the regions judged to be importance for the development of the target variable.4 The policy to be evaluated is then included as one of these characteristics. In the simplest case, assuming employment to be the target variable, we specify a linear equation such as ~f=aX:+ci
(,j= 1 . ..r regions,
r= 1 . ..n years),
where J+ =employment in region j (or in a certain sector in region j) at time t, a =a vector of coefficients. X/ = a vector of relevant characteristics for region j at time t (or at some earlier time if lags are believed to occur), ti{ = a stochastic disturbance term. Data for the r regions, usually all regions in a nation, and the n years, are then used to estimate the coefftcients, including the one relating the policy of interest to employment in the region under study. Central to a comparison between the experimental approach and this type of multiple regression analysis for evaluating policy effects is the question of the pros and cons of limiting the investigation to regions that are judged to have about the same characteristics as the region where the policy was introduced. The basic advantage of such a limitation is that the effect than can be measured with fewer assumptions. In this way we can circumvent “This method is used, for example. in Folmer and Nijkamp (1987) to analyse investment premiums. It is discussed more generally in Folmer (1986). Newman and Sullivan (1988) present a survey of attempts to use regression analysis to evaluate the impact of business taxes on location.
P. Bhn
and H. Lind, Policy evaluation quality
55
some of the specification and measurement problems that plague traditional regression analysis: (1) To avoid biased estimates we have to include all relevant characteristics in the regression equation. In many cases we are genuinely uncertain whether a characteristic is important or not and, as consequence, we are uncertain as to which estimate is closest to the truth. (2) In the regression analysis we have to take a stand on how various characteristics interact. What kind of functional form is correct? What is the misspecification loss from postulating a linear function? (3) Variables may be made precise and measured in different ways and we may be uncertain about which of these is the most relevant. The results may crucially depend on the exact choice made in these three respects. As the choice in many cases has to be made without a firm empirical foundation, it is difficult to know how much weight we should attach to the result of the analysis. The argument for using the experimental approach, when feasible, is that some of these choices do not have to be made. More specifically, we need to know that the experiment and the control regions are alike in all possibly relevant respects - certainly a demanding task - but we do not have to worry about biased estimates if we include factors that are irrelevant. We do not have to choose any functional form, given that it is only the policy effect on target variables that we want to find out. Nor do we have to worry about the choices among possible interpretations of certain variables, given that the experiment and control regions are alike for all of them.
3. An experimental study of employment Norrbotten: The method
effects of reduced payroll taxes in
The payroll tax in Norrbotten was reduced by 10 percentage points on 1 January 1984, from around 35% to around 2.5%. This amounted to a reduction of approximately SEK 10,000 ($1,700) per employee. The reduction was limited to mining, manufacturing industry, tourism and some minor service sectors5 It was stated that the reduction would be ‘long-lasting’. At the same time the existing marginal employment subsidy was raised by SEK 10,000 ($1,700) per employee. The state budget costs for the payroll tax reduction for the period 1984-1986 were around SEK 1,000 million ($170 million), while the rise in the marginal employment subsidy turned out to cost only about SEK 30 million ($5 million). The total cost amounted to approximately 20% of Swedish government expenditure for regional policy. In the remainder of the paper, the policy change is regarded as a reduction 50~r study was limited to manufacturing industry, employment point of view in the regions concerned.
which is the dominating
industry
from an
P. Bohm und H. Lind, Policy rouluation quality
56
in payroll of payroll
3.1.
taxes, although the evaluation refers to the simultaneous taxes and rise in marginal employment subsidies.
reduction
The control group
The starting point for identifying the control group was the ‘Regional Support Areas’ (RSA). The local government districts can belong to one of three RSAs: A, B and C. The classification is based primarily on the general labour market situation. RSA A is comprised of the inland parts of northern Sweden where unemployment is the highest. Firms in RSA A can receive more generous support, e.g. in terms of marginal employment and capital subsidies. Unemployment and the support offered are lower in RSA B and still lower in C. Our basic idea was to compared the development in each RSA in Norrbotten with the development in the same RSA in other parts of northern Sweden. As only two minor local government districts in Norrbotten belonged to RSA B, they were allocated between areas A and C on the basis of the area they resembled the most. Three municipalities in RSA C where excluded from the control group because they were judged to be located too close to a major town outside the RSA (Umea). The areas are shown in fig. 1. The areas to be compared thus shared the following two crucial characteristics: l the general situation in the labour market; l the possibilities of obtaining other types of regional subsidies. As can be seen from the map in fig. 1, the control groups are situated quite close to Norrbotten. Isserman and Merrifield (1982, p. 48) argue against the choice of adjacent regions as positive policy effects on the experiment group _ to the extent they arise - may to some degree occur at the expense of the control group, thus leading to overestimation of the policy effect. These possibilities have to be investigated if positive policy effects occur. An important advantage of choosing a control group adjacent to the experiment group is that the groups can be expected to be more similar, e.g. with respect to cultural characteristics, climate and distances to other regions.
3.2.
The control period
Even though adjacent areas can be expected to be similar, differences may still remain in many respects. The use of a control period has the role of controlling for differences that are constant over time. As the regions may be affected to different extents by the general business cycle, it was decided that the control period should cover the same phase in
P. Bohm and H. Lind, Policy evaluation quality
Group
Group
A
C
Experiment Control
group
group
County border Fig. 1. Location
of the experiment
and control
groups
the business cycle as the period 19841986. This was to be determined by the change in real GNP. The options for a control period, the eventual choice among which would be determined by the actual development 19841986, was made in the beginning of 1984. When the facts were known (in 1987), this led to the choice of 1979-1981 as the control period.
3.3. Comparisons
,for different levels qf aggregation
Comparisons were made on the following levels of aggregation: (1) Manufacturing industry in Norrbotten as a whole. The advantage of such a comparison is that it includes a large number of firms, hence reducing the risk that changes in a few enterprises heavily affect the result. The disadvantage, of course, is that the result can be affected by differences in industrial structure between Norrbotten and the control group. (2) Manufacturing industry in groups A and C, respectively. (3) Major branches of manufacturing industry, but only for those with a considerable number of firms and no single dominant firm. This selection of industries was made in 1984, before any data were available.
Branches in manufacturing industry are far from homogeneous. Therefore groups of ‘twin-firms were constructed. The basic idea here was to indentify a number of firms in Norrbotten for which a firm with similar characteristics could be found in the control group. The selection of twins was based on four characteristics: (1) size, (2) age, (3) type of ownership and (4) line of production (in 1983). This choice of characteristics was determined partly by the type of information available, but also by the argument that these factors should imply that the twins met similar market conditions [implied by (4)], had similar objectives [implied by (3)] and used similar production technologies [implied by (I), (2) and (4)]. The selection was made in the spring of 1985 when employment statistics for 1983 were available. Plants with more than 10 employees were included for group A and more than 25 for group C, where the number of firms is higher. The reason for these limits was that it is particularly difficult to obtain reliable information about small firms. Forty-four pairs of firms were identified. If we had required the twins to be identical ~ not only of the same approximate size, age, type of ownership and line of business ~ we would hardly have found any. We did not use any definite criterion for similarity concerning each separate characteristic. If the firms were quite similar in some respects, a somewhat larger difference could be accepted in other respects. The advantage of choosing twins prior to knowing how the firms have developed is obvious. In difficult cases it is otherwise hard not to be affected by such knowledge when judging whether the firms are similar enough. A successful identification of ‘twin-firms would significantly reduce the risk that factors other than the difference in the payroll tax caused differences in employment change (or counteracted a difference caused by the difference in the payroll tax). It should be noted, however, that the twin study neglects
P. Bohm and H. Lind,
Policy etlaluation
quality
59
possible effects of policy on the emergence of new firms. Such effects, if considerable, should however show up in the other parts of the evaluation.
3.5. The effi?cts of changes in the environment In the ideal experiment, the environment is kept under control and held constant. In a field experiment such as this, other factors may change which affect the regions differently. For example, other types of policy with regional effects may be changed. Information on such changes was collected through contacts with the regional authorities and local governments. A similar problem is created by indirect effects of changes in production and employment in certain large non-manufacturing plants, for example the mining company in Kiruna/Gallivare (LKAB). After consulting experts in these fields careful attempts were made to evaluate the likely effects of these factors on the relative development of the two groups. In two cases the probable effects were of a magnitude that required some adjustment of the data.’ 4. Results 4.1. Total manufacturing
and major manyfacturing
industries
The results for Norrbotten as a whole and for areas A and C are reported in table 1. The control group for Norrbotten as whole is a weighted sum of the control groups A and C, with weights determined by the size of these groups in Norrbotten.’ The table lists employment figures for the beginning and the end of the control and experimental periods. The table also states the percentage change in each period and the difference between the regions in each period. The change in this difference is the crucial figure for judging the effects of the reduction in the payroll tax. A positive change means that Norrbotten underwent a more positive development in the period when the payroll tax was reduced, relative to the development in the control period. The results presented in table 1 indicate that there has not been any positive change in the relative developrment of employment in Norrbotten during the period of reduced payroll taxes. The development in the major industries are presented in table 2, where the pattern for manufacturing industry as a whole recurs: no positive effects of the reduced payroll tax can be found. To judge whether these results could be caused by random events, not ‘See Bohm and Lind (1988). ‘This explains why the number same as in Norrbotten.
of employed
in the control
group
in 1978 and
1983 are the
P. Bohm and H. Lind, Policy evaluation
60
Table Employment 1978
Area
changes
I
in manufacturing ‘x change
1981
I. Norrbotten
quality
industry.
1983
1986
“/:, change
as a whole
0.0 Norrbotten 15,500 15,496 14,441 15,500 15,095 -2.6 14,441 Control group Difference between the areas in the control period: DiITerence between the areas in the experiment period: Change in relative development in Norrbotten:
15,244 15,207 + 2.67; + 0.3% -2.3%
+5.6 + 5.3
3,693 2,188 +3.1”/, + 3.5% + 0.42,
+5.8 +2.3
11,551 24,131 +2.3”/, - 2.2% -4.5%
+5.5 +7.6
II. Group A 3,554 +3.2 Norrbotten 3,443 3,490 2,606 2,609 +O.l 2,139 Control group Difference between the areas in the control period: Difference between the areas in the experiment period: Change in relative development in Norrbotten: III. Group C Norrbotten 12,057 11,924 - 1.1 10,951 Control group 27,831 26,889 -3.4 22,708 Difference between the areas in the control period: Difference between the areas in the experiment period: Change in relative development on Norrbotten: Source: SCB (Statistics
Sweden).
Industrial
statistics,
adjusted
figures.
covered by the factors controlled for, we made the following assumptions about the role of chance events on the firm level:’ (a) For each firm these exists a certain probability distribution for the change in employment caused by chance events during a three-year period. The changes are assumed to be normally distributed and less than 10% of the employment in 1983 in 90% of the cases. (b) The probability distributions for the firms are independent of one another. Given these assumptions the probability that a certain change in the difference between the experiment and control groups would arise by chance can be calculated. In 907; of the cases this figure is smaller than the value stated in table 3. These results indicate that, given our assumptions, several of the results above are not significantly different from zero. They also indicate that if these assumptions are approximately correct, it is very unlikely that any large positive employment effect of the reduced payroll tax has been counteracted by a series of chance events. 4.2. The twin study For
each
pair
“It is easy to remake
of
twins
the calculations
we
have
calculated
with other assumptions.
the
relative
change
in
Source: See table
Group C Food Wood Paper/graphics Machinery
Group A Wood Machinery
Group/ industry
group (2)
1.
+4.8 + 8.3 + 4.7 + 10.5 -7.1 +0.6 -5.6 +0.4
+3.5 -6.6
Control
Exper.group (1) -3.1 +9.7
change
Employment 1978-1981
change
+ 11.9 + 7.7 + 10.3 + 10.1
- 6.6 + 16.4
Difference (3)=(l)-(2)
Employment
2
+4.1 - 1.7 -0.8 + 20.9
- 8.0 + 12.6
group (4)
Exper.-
Employment 1983-1986
on the industry
Table
+ 5.3 -5.3 + 19.8 + 14.6
-1.0 +9.1
group (5)
Control
change
level (%).
- 1.2 + 3.6 -20.5 +6.3
- 7.0 + 3.5
Difference (6)=(5)-(4)
-13.1 -4.1 - 30.8 -3.8
-0.4 - 12.9
(7)=(6)-(3)
Relative change
P. Bohm and H. Lind, Policy eoaluution quality
62
Table 3 Possible
effects of chance
events (‘I,,)
Subgroup/industry Group A Manufacturing Wood Machinery Group C Manufacturing Food Wood Paper,‘graphics Machinery
Interval as a whole
k3.1 k4.8 * 5.1
as a whole
f 2.5 k5.1 f 5.5 + 6.0 i3.9
Table 4 Difference
in employment
change
in the twin lirms
Number Relative (“0)
of twins
change Group
> +50 +30-+50 +10-+30 0++10 -10-o -3o- - IO -5om-30 < -50 Number of positive changes Number of negative changes Sourcc~ See table
A
Group
C
Total
I 2 2 I
3 3 3 3
4 5 5 4
I 5 3 4 6 I3
5 3 I 4 I2 I3
6 8 4 8 1x 26
I.
employment in the way described in the preceding section. The results are shown in table 4 where a positive figure indicates that the Norrbotten firm exhibited a more positive relative development in the period of reduced payroll taxes, as compared with the development during the control period. Once again, there is no sign of a more positive development of employment in the Norrbotten firms. In fact, out of the 44 twins, the Norrbotten firms underwent a more negative relative development in 26 cases. A sign test was made to check the likelihood that a positive effect has been counteracted by chance events. The question asked was this: Assuming that, due to the payroll tax reduction, there was a 10% effect of the Norrbotten firms as compared with their twins during the three-year period, what is the probability of observing a distribution between positive and negative cases
P. Bohm and H. Lind, Policy evaluation quality
like the one in table 4? The answer is that this happens 100 for group A and once out of 10 for group C.
63
three
times out of
5. Concluding remarks 5.1. Why no employment
effect?
There are several possible explanations why no employment effects of the reduced payroll tax could be detected for the period under study. The three most interesting seem to be the following. (1) The reduction in the payroll tax has been neutralized by rising wages and other input prices.’ The Swedish Board of Industry has tried to measure the effect of the reduced payroll tax on the wage level in the twinfirms. The available, but far from perfect, wage data suggest, however, that although wages have risen somewhat more in Norrbotten, the rise was clearly insufficient to neutralize the reduction in the payroll tax. (2) The elasticity of demand is too low. The reduction of the payroll tax leads to a cost reduction of around 4%, given that labour accounts for some 50% of production cost in the industry concerned. A price reduction on that order of magnitude may be too small to affect sales other than marginally. (3) Sluggishness and uncertainty. It takes time to find new customers or to increase capacity. Moreover, the firms may have believed that the reduction would not last (in spite of the fact that the government said that it would be ‘long-lasting’). The Swedish debate came to focus on the argument about sluggishness, and that a three-year period may be too short for any employment effects to materialize. This argument is doubtful, however, especially since one advantage claimed for the reduced payroll tax, as compared with capital subsidies, was that it would reduce the otherwise frequent lay-offs in non-expanding firms in these areas. It is also hard to believe that a three-year period, with an additional half-year alert, is so short that sluggishness alone can explain that nothing happened to employment. It should also be noted that, if an employment effect of the reduced pay-roll tax eventually will show up, the absence of any effect for as long a period as three years or more implies that the budgetary costs for each new job become very high.
5.2. Policy implications The result that no noticeable employment budget cost was SEK 1 billion motivates a design of the policy. If neutralizing changes payroll tax could be made conditional on the 91.e. a low elasticity
of supply
of inputs,
implying
effects arose even though the discussion of changes in the in wages occur, the reduced wage level in the firm, say, in
a steeply rising marginal
cost curve
64
P. Bohm and H. Lind, Policy eualuation quality
relation to some national average. Furthermore, what makes the present policy design costly is that the reduction in payroll taxes is general, i.e. not marginal only. The argument in favour of this design is that such a reduction would stimulate not only expanding firms but also firms that may be planning to lay off employees. A less expensive way of reaching most firms in the latter group is to introduce a marginal employment subsidy only and let it start at, say, 80% of the employment in a base year. As no subsidy would be paid for the first 8074 of employment, the marginal subsidy rate could be set rather high and hence be more effective, without exceeding the budget constraint.
5.3. Methodological
conclusions
In this study of employment effects of locally reduced payroll taxes, we have illustrated the practicability ~ under certain conditions ~ of an evaluation method that tries to come close to a classical experiment. Our study was made possible by the fact that the reduced payroll tax cut across existing support areas, defined so as to be internally homogeneous, thereby allowing us to identify comparable areas. We have argued that the experimental approach requires less questionable assumptions than standard evaluation methods: shift-share analysis and multiple regression analysis. Shift-share analysis presupposes that, in the absence of policy action, the relation between the development in the region and the nation would remain constant. The experimental approach, on the other hand, uses comparisons between similar regions.” Traditional regression analysis presupposes specifying a complete theory of what determines the development of the target variable in the region. In the experimental approach we do not have to specify the role of and interactions among those possible determinants which are similar for the experiment region and the control region. In order to raise the quality of the evaluation even further, firms in the experiment region, for which a ‘twin’ could be found in the control region, were subjected to a separate study. Such pairs are likely to constitute a case where non-policy differences between the two regions are minimized. It is argued here that it is important to raise the quality of evaluating a policy change not only by choosing an appropriate evaluation approach but also, as was done in this study, by specifying the evaluation method in detail before any data on the target variables after the policy change are available. “‘11is noteworthy that superficial, but quite common types of evaluation give a signilicantly different picture of the effects of the policy change discussed here. Thus, employment in the manufacturing industry in Norrbotten grew by 5.6”,,, from 1983 to 1986 and not at all during the ‘control period’, the preceding similar business-cycle period, 197%1981 (see table I). Similarly, using a simplified version of shift-share analysis, employment in the manufacturing industry in with 5.6”,;, for Sweden as a whole grew by 0.5”,, percent from 1983 10 1986 as compared Norrbotten.
P. Bohm and H. Lind, Policy evaluation quality
65
This specification involves several choices, the outcome of which may significantly influence the outcome of the evaluation. Therefore, an ex ante specification - although perhaps not time efficient - is clearly desirable both from a scientific point of view and from the point of view of credibility among the policymakers who are supposed to take the result of policy evaluations into account. References Armstrong, H. and J. Taylor, 1985, Regional economics and policy (Philip Allan Publishers, Oxford). Bartels, P.A., W.R. Nicol and J.J. van Duijn, 1982. Estimating the impact of regional policy: A review of applied research methods, Regional Science and Urban Economics 12, 3-41. Bohm, P. and H. Lind, 1984, Sysselsattningseffekter av sankt arbetsgivaravgift i Norrbotten/ Svappavaara: Metodbeskrivning (Employment effects from reduced payroll taxes in Norrbotten/Svappavaara: Specification of the evaluation method), SIND PM 1984:2, Stockholm. Bohm, P. and H. Lind, 1988, Sysselsittningseffekter av sankt arbetsgivaravgift i Norrbotten 19841986 (Employment effects of reduced payroll taxes in Norrbotten 19841986) Research paper in economics 1988: I, Department of Economics, Stockholm University. Diamond, D. and N. Spence, 1983, Regional policy evaluation: A methodological review and the Scottish example (Cower Publishing Company, Hampshire). Folmer, H., 1986, Regional economic policy: Measurement of its effect (Martinus Nijhoff Publishers, Dordrecht). Folmer, H. and P. Nijkamp, 1987, Investment premiums: Expensive but hardly effective, Kyklos 40, 43-72. Isserman, A. and J. Merrifteld, 1982, The use of control groups in evaluating regional economic policy, Regional Science and Urban Economics I2,43358. Lind, T. and J. Serck-Hansen. 1972, Regional subsidies on labour and capital, Swedish Journal of Economics 74, 68-84. Moore, B. and J. Rhodes, 1973, Evaluating the effects of British regional economic policy, Economic Journal 83. 87-1 IO. Newman, R. and D. Sullivan, 1988, Econometric analysis of business tax impacts on industrial location: What do we know, and how do we know it? Journal of Urban Economics 23, 2 155234. Tervo, H. and P. Okko, 1983, A note on shift-share analysis as a method of estimating the employment effects of regional economic policy, Journal of Regional Science 23, 115-l 21.
Science and
Urban
Economics
23 (1993) 51-65.
Policy evaluation A quasi-experimental subsidies in Sweden
North-Holland
quality study of regional
employment
Peter Bohm and Hans Lind Stockholm
Received
Unicersity,
S-106 YI Stockholm.
Sweden
May 1990, final version received
April 1991
Policy evaluation methods differ as to the nature of their assumptions about the hypothetical policy-off case. We argue that policy evaluation quality can be raised by using a quasiexperimental method when feasible. In addition, the quality of evaluating policy changes can be raised by designing and publicizing the evaluation method, specified in detail, before any data on target variables are available. These policy evaluation properties characterize the study presented here. It concerns a signilicant reduction of payroll taxes in a depressed region of Sweden in order to bolster employment. The evaluation produced clear evidence of a complete lack of employment effects.
1. Introduction The purpose here is to evaluate the effects on the main target variable employment - of a change in regional policy towards subsidizing labour instead of capital, using an improved form of policy evaluation. Specifically: (1) We use a quasi-experimental evaluation method, which, as we shall argue, requires less questionable assumptions than traditional evaluation methods. (2) The evaluation method was designed and publicized before any data for the policy-on period were available [Bohm and Lind (1984)]. Any adjustment of the analytical approach that one may wish to make would then easily be detected and call for an explicit explanation. The reason for using this approach - and for advocating it whenever possible - is that a number of difficult choices have to be made when designing an evaluation; then, if data already exist, knowledge of the effects of these choices can influence the choice made by even the most serious evaluator. (3) By starting the work on evaluation at the same time as the policy is introduced, quality can be raised also in other respects. It is easier to check Correspondence
Stockholm,
to:
P. Bohm,
Department
of Economics,
Stockholm
University,
Sweden.
016Wl462/93/$06.00
(1) 1993-Elsevier
Science Publishers
B.V. All rights reserved
S-106
91
P. Bohm
52
and H. Lind.
PO/&~ rruluation
quality
data when they are new, and various events that may affect the target variables can more easily be investigated. In Sweden as in many other countries, regional policy has been dominated by capital subsidies. Economists have long argued in favour of using labour subsidies when the basic policy objective is to increase employment opportunities in depressed regions [e.g. Lind and Serck-Hansen (1972)]. In Sweden, a significant step in this direction was taken in 1984, when the payroll tax for Norrbotten (the northernmost county in Sweden) was reduced by about onethird, implying a labour-cost reduction by some 7:;. The employment effects of this new policy are investigated in this paper and the results show the extent to which the theory-based expectations of the effect of this type of policy actually materialized. The outline of the paper is as follows. In section 2 we give a short theoretical background, describe the experimental approach used and compare it with more traditional methods. In section 3 we describe the details of the method used, in particular how control groups and ‘twin-firms were chosen. The results are presented in section 4. Alternative interpretations of the results and policy implications are discussed in the final section. 2. The experimental
approach compared with other methods
If we look at a perfectly competitive product market, the effect on employment of a subsidy on labour is determined by the resulting change in output and the substitution of labour for capital. The change in output depends upon the slope of the marginal cost curve around the old equilibrium, which in turn depends upon technology and the change in input prices that input changes may give rise to. If the product is sold under less than perfectly competitive conditions, the output effect also depends upon how elastic the demand for the product is. A wage subsidy would have a substantial effect on employment if there is (1) a high elasticity of demand and a flat marginal cost curve, and/or (2) a high elasticity of substitution between labour and capital. Evaluating the actual employment effect of a wage subsidy in a region with a considerable number of different firms, we are rarely able to estimate the underlying factors now mentioned. Rather, we have to settle for a more direct estimate of employment changes. We may distinguish among three approaches here: (quasi-)experiments, shift-share analysis and multiple regression analysis.
2. I
The experimental
upproach
Controlled experiments, where a group of subjects (regions) are randomly allocated to an experiment group and a control group, are for various
P. Bohm and H. Lind, Policy evaluation
quality
53
reasons rare in the field of economic policy. In practice, the experimental approach to (regional) policy evaluation will focus on the possibility to identify, at least reasonably well, a control group after the policymaker has decided that a certain policy should be introduced in a certain region. Since some differences between the two groups are likely to remain, for example as regards distances to other regions, the use of a control period is a natural part of this type of experimental approach.’ This kind of experimental approach may seem paradoxical in the sense that if the policy is applied to a certain region and not to another, then the implications would seem to be that the two regions actually are significantly different and that therefore the second region cannot be used as a control group.2 In many situations, however, this problem may not be so severe: (1) The explanation why one region gets the policy while another does not may be wholly ‘political’. (2) The policy may be limited, for administrative reasons, to an area which is smaller than a rather homogeneous problem area. If so, parts of the problem area are left which fit the requirements for a control group. (This is true in our particular case, where the tax reduction was accorded to Norrbotten county. There is no county like the county of Norrbotten but for each part of Norrbotten there exist parts of other counties that could form a control group.) 2.2. Shift-share
analysis
The standard method for evaluating regional policy has been shift-share analysis. The basic idea is that without any change in policy, the region’s share of national employment would continue to develop as in the pre-policy period. If the development of the region’s share changes during the policy-on period, this change is taken to reveal the policy effect.3 The problematical aspect of shift-share analysis is use of the region’s share of national employment when estimating what would have happened if the policy had not been introduced. The development of this share can change for a number of other reasons. In Sweden, for example, there have been ‘As far as we know the only earlier attempt to use an experimental approach when evaluating regional policy is that of Isserman and Merritield (1982). Important ditTerences between their study and ours are: (a) the way in which the control group of regions is selected (we base the choice on similarities in certain characteristics while Isserman and Merrifield base it on similarities in growth rates in a pre-policy period); (b) in addition to comparisons for manufacturing industry as a whole and for large industries, we compared a group of ‘twin-firms in the two regions; and (c) as already indicated, we specified the evaluation method in detail before any data for the policy period were available. *Folmer (1986) uses this as one argument against experimental approaches. 3A description of the method, comments on its weaknesses and further references may be found in, for example, Moore and Rhodes (1973), Bartels et al. (1982), Diamond and Spence (1983). Tervo and Okko (1983) Armstrong and Taylor (1985) and Folmer (1986).
recurring periods of strong metropolitan growth during the post-war period. When using shift-share analysis, the measured effect of changes in regional policy may depend critically on the time at which such changes are made in relation to the time when the more general growth trend changes. The ‘trend’ of the control group used in the experimental approach is likely to be a better indicator of what would have happened in the no-policy case than the national trend. Shift-share analysis is usually carried out at the industry level. At this level, however, it is very likely that important differences is characteristics remain between the region subject to policy change and the nation as a whole. These differences in characteristics at the industry level can be expected to be smaller when a control group with similar basic characteristic is used. 2.3. Multiplt~ rrgression
unu1~~si.s
Using multiple regression analysis, the effect of a certain regional policy is measured by specifying the relation between the target variable and all the characteristics of the regions judged to be importance for the development of the target variable.4 The policy to be evaluated is then included as one of these characteristics. In the simplest case, assuming employment to be the target variable, we specify a linear equation such as ~f=aX:+ci
(,j= 1 . ..r regions,
r= 1 . ..n years),
where J+ =employment in region j (or in a certain sector in region j) at time t, a =a vector of coefficients. X/ = a vector of relevant characteristics for region j at time t (or at some earlier time if lags are believed to occur), ti{ = a stochastic disturbance term. Data for the r regions, usually all regions in a nation, and the n years, are then used to estimate the coefftcients, including the one relating the policy of interest to employment in the region under study. Central to a comparison between the experimental approach and this type of multiple regression analysis for evaluating policy effects is the question of the pros and cons of limiting the investigation to regions that are judged to have about the same characteristics as the region where the policy was introduced. The basic advantage of such a limitation is that the effect than can be measured with fewer assumptions. In this way we can circumvent “This method is used, for example. in Folmer and Nijkamp (1987) to analyse investment premiums. It is discussed more generally in Folmer (1986). Newman and Sullivan (1988) present a survey of attempts to use regression analysis to evaluate the impact of business taxes on location.
P. Bhn
and H. Lind, Policy evaluation quality
55
some of the specification and measurement problems that plague traditional regression analysis: (1) To avoid biased estimates we have to include all relevant characteristics in the regression equation. In many cases we are genuinely uncertain whether a characteristic is important or not and, as consequence, we are uncertain as to which estimate is closest to the truth. (2) In the regression analysis we have to take a stand on how various characteristics interact. What kind of functional form is correct? What is the misspecification loss from postulating a linear function? (3) Variables may be made precise and measured in different ways and we may be uncertain about which of these is the most relevant. The results may crucially depend on the exact choice made in these three respects. As the choice in many cases has to be made without a firm empirical foundation, it is difficult to know how much weight we should attach to the result of the analysis. The argument for using the experimental approach, when feasible, is that some of these choices do not have to be made. More specifically, we need to know that the experiment and the control regions are alike in all possibly relevant respects - certainly a demanding task - but we do not have to worry about biased estimates if we include factors that are irrelevant. We do not have to choose any functional form, given that it is only the policy effect on target variables that we want to find out. Nor do we have to worry about the choices among possible interpretations of certain variables, given that the experiment and control regions are alike for all of them.
3. An experimental study of employment Norrbotten: The method
effects of reduced payroll taxes in
The payroll tax in Norrbotten was reduced by 10 percentage points on 1 January 1984, from around 35% to around 2.5%. This amounted to a reduction of approximately SEK 10,000 ($1,700) per employee. The reduction was limited to mining, manufacturing industry, tourism and some minor service sectors5 It was stated that the reduction would be ‘long-lasting’. At the same time the existing marginal employment subsidy was raised by SEK 10,000 ($1,700) per employee. The state budget costs for the payroll tax reduction for the period 1984-1986 were around SEK 1,000 million ($170 million), while the rise in the marginal employment subsidy turned out to cost only about SEK 30 million ($5 million). The total cost amounted to approximately 20% of Swedish government expenditure for regional policy. In the remainder of the paper, the policy change is regarded as a reduction 50~r study was limited to manufacturing industry, employment point of view in the regions concerned.
which is the dominating
industry
from an
P. Bohm und H. Lind, Policy rouluation quality
56
in payroll of payroll
3.1.
taxes, although the evaluation refers to the simultaneous taxes and rise in marginal employment subsidies.
reduction
The control group
The starting point for identifying the control group was the ‘Regional Support Areas’ (RSA). The local government districts can belong to one of three RSAs: A, B and C. The classification is based primarily on the general labour market situation. RSA A is comprised of the inland parts of northern Sweden where unemployment is the highest. Firms in RSA A can receive more generous support, e.g. in terms of marginal employment and capital subsidies. Unemployment and the support offered are lower in RSA B and still lower in C. Our basic idea was to compared the development in each RSA in Norrbotten with the development in the same RSA in other parts of northern Sweden. As only two minor local government districts in Norrbotten belonged to RSA B, they were allocated between areas A and C on the basis of the area they resembled the most. Three municipalities in RSA C where excluded from the control group because they were judged to be located too close to a major town outside the RSA (Umea). The areas are shown in fig. 1. The areas to be compared thus shared the following two crucial characteristics: l the general situation in the labour market; l the possibilities of obtaining other types of regional subsidies. As can be seen from the map in fig. 1, the control groups are situated quite close to Norrbotten. Isserman and Merrifield (1982, p. 48) argue against the choice of adjacent regions as positive policy effects on the experiment group _ to the extent they arise - may to some degree occur at the expense of the control group, thus leading to overestimation of the policy effect. These possibilities have to be investigated if positive policy effects occur. An important advantage of choosing a control group adjacent to the experiment group is that the groups can be expected to be more similar, e.g. with respect to cultural characteristics, climate and distances to other regions.
3.2.
The control period
Even though adjacent areas can be expected to be similar, differences may still remain in many respects. The use of a control period has the role of controlling for differences that are constant over time. As the regions may be affected to different extents by the general business cycle, it was decided that the control period should cover the same phase in
P. Bohm and H. Lind, Policy evaluation quality
Group
Group
A
C
Experiment Control
group
group
County border Fig. 1. Location
of the experiment
and control
groups
the business cycle as the period 19841986. This was to be determined by the change in real GNP. The options for a control period, the eventual choice among which would be determined by the actual development 19841986, was made in the beginning of 1984. When the facts were known (in 1987), this led to the choice of 1979-1981 as the control period.
3.3. Comparisons
,for different levels qf aggregation
Comparisons were made on the following levels of aggregation: (1) Manufacturing industry in Norrbotten as a whole. The advantage of such a comparison is that it includes a large number of firms, hence reducing the risk that changes in a few enterprises heavily affect the result. The disadvantage, of course, is that the result can be affected by differences in industrial structure between Norrbotten and the control group. (2) Manufacturing industry in groups A and C, respectively. (3) Major branches of manufacturing industry, but only for those with a considerable number of firms and no single dominant firm. This selection of industries was made in 1984, before any data were available.
Branches in manufacturing industry are far from homogeneous. Therefore groups of ‘twin-firms were constructed. The basic idea here was to indentify a number of firms in Norrbotten for which a firm with similar characteristics could be found in the control group. The selection of twins was based on four characteristics: (1) size, (2) age, (3) type of ownership and (4) line of production (in 1983). This choice of characteristics was determined partly by the type of information available, but also by the argument that these factors should imply that the twins met similar market conditions [implied by (4)], had similar objectives [implied by (3)] and used similar production technologies [implied by (I), (2) and (4)]. The selection was made in the spring of 1985 when employment statistics for 1983 were available. Plants with more than 10 employees were included for group A and more than 25 for group C, where the number of firms is higher. The reason for these limits was that it is particularly difficult to obtain reliable information about small firms. Forty-four pairs of firms were identified. If we had required the twins to be identical ~ not only of the same approximate size, age, type of ownership and line of business ~ we would hardly have found any. We did not use any definite criterion for similarity concerning each separate characteristic. If the firms were quite similar in some respects, a somewhat larger difference could be accepted in other respects. The advantage of choosing twins prior to knowing how the firms have developed is obvious. In difficult cases it is otherwise hard not to be affected by such knowledge when judging whether the firms are similar enough. A successful identification of ‘twin-firms would significantly reduce the risk that factors other than the difference in the payroll tax caused differences in employment change (or counteracted a difference caused by the difference in the payroll tax). It should be noted, however, that the twin study neglects
P. Bohm and H. Lind,
Policy etlaluation
quality
59
possible effects of policy on the emergence of new firms. Such effects, if considerable, should however show up in the other parts of the evaluation.
3.5. The effi?cts of changes in the environment In the ideal experiment, the environment is kept under control and held constant. In a field experiment such as this, other factors may change which affect the regions differently. For example, other types of policy with regional effects may be changed. Information on such changes was collected through contacts with the regional authorities and local governments. A similar problem is created by indirect effects of changes in production and employment in certain large non-manufacturing plants, for example the mining company in Kiruna/Gallivare (LKAB). After consulting experts in these fields careful attempts were made to evaluate the likely effects of these factors on the relative development of the two groups. In two cases the probable effects were of a magnitude that required some adjustment of the data.’ 4. Results 4.1. Total manufacturing
and major manyfacturing
industries
The results for Norrbotten as a whole and for areas A and C are reported in table 1. The control group for Norrbotten as whole is a weighted sum of the control groups A and C, with weights determined by the size of these groups in Norrbotten.’ The table lists employment figures for the beginning and the end of the control and experimental periods. The table also states the percentage change in each period and the difference between the regions in each period. The change in this difference is the crucial figure for judging the effects of the reduction in the payroll tax. A positive change means that Norrbotten underwent a more positive development in the period when the payroll tax was reduced, relative to the development in the control period. The results presented in table 1 indicate that there has not been any positive change in the relative developrment of employment in Norrbotten during the period of reduced payroll taxes. The development in the major industries are presented in table 2, where the pattern for manufacturing industry as a whole recurs: no positive effects of the reduced payroll tax can be found. To judge whether these results could be caused by random events, not ‘See Bohm and Lind (1988). ‘This explains why the number same as in Norrbotten.
of employed
in the control
group
in 1978 and
1983 are the
P. Bohm and H. Lind, Policy evaluation
60
Table Employment 1978
Area
changes
I
in manufacturing ‘x change
1981
I. Norrbotten
quality
industry.
1983
1986
“/:, change
as a whole
0.0 Norrbotten 15,500 15,496 14,441 15,500 15,095 -2.6 14,441 Control group Difference between the areas in the control period: DiITerence between the areas in the experiment period: Change in relative development in Norrbotten:
15,244 15,207 + 2.67; + 0.3% -2.3%
+5.6 + 5.3
3,693 2,188 +3.1”/, + 3.5% + 0.42,
+5.8 +2.3
11,551 24,131 +2.3”/, - 2.2% -4.5%
+5.5 +7.6
II. Group A 3,554 +3.2 Norrbotten 3,443 3,490 2,606 2,609 +O.l 2,139 Control group Difference between the areas in the control period: Difference between the areas in the experiment period: Change in relative development in Norrbotten: III. Group C Norrbotten 12,057 11,924 - 1.1 10,951 Control group 27,831 26,889 -3.4 22,708 Difference between the areas in the control period: Difference between the areas in the experiment period: Change in relative development on Norrbotten: Source: SCB (Statistics
Sweden).
Industrial
statistics,
adjusted
figures.
covered by the factors controlled for, we made the following assumptions about the role of chance events on the firm level:’ (a) For each firm these exists a certain probability distribution for the change in employment caused by chance events during a three-year period. The changes are assumed to be normally distributed and less than 10% of the employment in 1983 in 90% of the cases. (b) The probability distributions for the firms are independent of one another. Given these assumptions the probability that a certain change in the difference between the experiment and control groups would arise by chance can be calculated. In 907; of the cases this figure is smaller than the value stated in table 3. These results indicate that, given our assumptions, several of the results above are not significantly different from zero. They also indicate that if these assumptions are approximately correct, it is very unlikely that any large positive employment effect of the reduced payroll tax has been counteracted by a series of chance events. 4.2. The twin study For
each
pair
“It is easy to remake
of
twins
the calculations
we
have
calculated
with other assumptions.
the
relative
change
in
Source: See table
Group C Food Wood Paper/graphics Machinery
Group A Wood Machinery
Group/ industry
group (2)
1.
+4.8 + 8.3 + 4.7 + 10.5 -7.1 +0.6 -5.6 +0.4
+3.5 -6.6
Control
Exper.group (1) -3.1 +9.7
change
Employment 1978-1981
change
+ 11.9 + 7.7 + 10.3 + 10.1
- 6.6 + 16.4
Difference (3)=(l)-(2)
Employment
2
+4.1 - 1.7 -0.8 + 20.9
- 8.0 + 12.6
group (4)
Exper.-
Employment 1983-1986
on the industry
Table
+ 5.3 -5.3 + 19.8 + 14.6
-1.0 +9.1
group (5)
Control
change
level (%).
- 1.2 + 3.6 -20.5 +6.3
- 7.0 + 3.5
Difference (6)=(5)-(4)
-13.1 -4.1 - 30.8 -3.8
-0.4 - 12.9
(7)=(6)-(3)
Relative change
P. Bohm and H. Lind, Policy eoaluution quality
62
Table 3 Possible
effects of chance
events (‘I,,)
Subgroup/industry Group A Manufacturing Wood Machinery Group C Manufacturing Food Wood Paper,‘graphics Machinery
Interval as a whole
k3.1 k4.8 * 5.1
as a whole
f 2.5 k5.1 f 5.5 + 6.0 i3.9
Table 4 Difference
in employment
change
in the twin lirms
Number Relative (“0)
of twins
change Group
> +50 +30-+50 +10-+30 0++10 -10-o -3o- - IO -5om-30 < -50 Number of positive changes Number of negative changes Sourcc~ See table
A
Group
C
Total
I 2 2 I
3 3 3 3
4 5 5 4
I 5 3 4 6 I3
5 3 I 4 I2 I3
6 8 4 8 1x 26
I.
employment in the way described in the preceding section. The results are shown in table 4 where a positive figure indicates that the Norrbotten firm exhibited a more positive relative development in the period of reduced payroll taxes, as compared with the development during the control period. Once again, there is no sign of a more positive development of employment in the Norrbotten firms. In fact, out of the 44 twins, the Norrbotten firms underwent a more negative relative development in 26 cases. A sign test was made to check the likelihood that a positive effect has been counteracted by chance events. The question asked was this: Assuming that, due to the payroll tax reduction, there was a 10% effect of the Norrbotten firms as compared with their twins during the three-year period, what is the probability of observing a distribution between positive and negative cases
P. Bohm and H. Lind, Policy evaluation quality
like the one in table 4? The answer is that this happens 100 for group A and once out of 10 for group C.
63
three
times out of
5. Concluding remarks 5.1. Why no employment
effect?
There are several possible explanations why no employment effects of the reduced payroll tax could be detected for the period under study. The three most interesting seem to be the following. (1) The reduction in the payroll tax has been neutralized by rising wages and other input prices.’ The Swedish Board of Industry has tried to measure the effect of the reduced payroll tax on the wage level in the twinfirms. The available, but far from perfect, wage data suggest, however, that although wages have risen somewhat more in Norrbotten, the rise was clearly insufficient to neutralize the reduction in the payroll tax. (2) The elasticity of demand is too low. The reduction of the payroll tax leads to a cost reduction of around 4%, given that labour accounts for some 50% of production cost in the industry concerned. A price reduction on that order of magnitude may be too small to affect sales other than marginally. (3) Sluggishness and uncertainty. It takes time to find new customers or to increase capacity. Moreover, the firms may have believed that the reduction would not last (in spite of the fact that the government said that it would be ‘long-lasting’). The Swedish debate came to focus on the argument about sluggishness, and that a three-year period may be too short for any employment effects to materialize. This argument is doubtful, however, especially since one advantage claimed for the reduced payroll tax, as compared with capital subsidies, was that it would reduce the otherwise frequent lay-offs in non-expanding firms in these areas. It is also hard to believe that a three-year period, with an additional half-year alert, is so short that sluggishness alone can explain that nothing happened to employment. It should also be noted that, if an employment effect of the reduced pay-roll tax eventually will show up, the absence of any effect for as long a period as three years or more implies that the budgetary costs for each new job become very high.
5.2. Policy implications The result that no noticeable employment budget cost was SEK 1 billion motivates a design of the policy. If neutralizing changes payroll tax could be made conditional on the 91.e. a low elasticity
of supply
of inputs,
implying
effects arose even though the discussion of changes in the in wages occur, the reduced wage level in the firm, say, in
a steeply rising marginal
cost curve
64
P. Bohm and H. Lind, Policy eualuation quality
relation to some national average. Furthermore, what makes the present policy design costly is that the reduction in payroll taxes is general, i.e. not marginal only. The argument in favour of this design is that such a reduction would stimulate not only expanding firms but also firms that may be planning to lay off employees. A less expensive way of reaching most firms in the latter group is to introduce a marginal employment subsidy only and let it start at, say, 80% of the employment in a base year. As no subsidy would be paid for the first 8074 of employment, the marginal subsidy rate could be set rather high and hence be more effective, without exceeding the budget constraint.
5.3. Methodological
conclusions
In this study of employment effects of locally reduced payroll taxes, we have illustrated the practicability ~ under certain conditions ~ of an evaluation method that tries to come close to a classical experiment. Our study was made possible by the fact that the reduced payroll tax cut across existing support areas, defined so as to be internally homogeneous, thereby allowing us to identify comparable areas. We have argued that the experimental approach requires less questionable assumptions than standard evaluation methods: shift-share analysis and multiple regression analysis. Shift-share analysis presupposes that, in the absence of policy action, the relation between the development in the region and the nation would remain constant. The experimental approach, on the other hand, uses comparisons between similar regions.” Traditional regression analysis presupposes specifying a complete theory of what determines the development of the target variable in the region. In the experimental approach we do not have to specify the role of and interactions among those possible determinants which are similar for the experiment region and the control region. In order to raise the quality of the evaluation even further, firms in the experiment region, for which a ‘twin’ could be found in the control region, were subjected to a separate study. Such pairs are likely to constitute a case where non-policy differences between the two regions are minimized. It is argued here that it is important to raise the quality of evaluating a policy change not only by choosing an appropriate evaluation approach but also, as was done in this study, by specifying the evaluation method in detail before any data on the target variables after the policy change are available. “‘11is noteworthy that superficial, but quite common types of evaluation give a signilicantly different picture of the effects of the policy change discussed here. Thus, employment in the manufacturing industry in Norrbotten grew by 5.6”,,, from 1983 to 1986 and not at all during the ‘control period’, the preceding similar business-cycle period, 197%1981 (see table I). Similarly, using a simplified version of shift-share analysis, employment in the manufacturing industry in with 5.6”,;, for Sweden as a whole grew by 0.5”,, percent from 1983 10 1986 as compared Norrbotten.
P. Bohm and H. Lind, Policy evaluation quality
65
This specification involves several choices, the outcome of which may significantly influence the outcome of the evaluation. Therefore, an ex ante specification - although perhaps not time efficient - is clearly desirable both from a scientific point of view and from the point of view of credibility among the policymakers who are supposed to take the result of policy evaluations into account. References Armstrong, H. and J. Taylor, 1985, Regional economics and policy (Philip Allan Publishers, Oxford). Bartels, P.A., W.R. Nicol and J.J. van Duijn, 1982. Estimating the impact of regional policy: A review of applied research methods, Regional Science and Urban Economics 12, 3-41. Bohm, P. and H. Lind, 1984, Sysselsattningseffekter av sankt arbetsgivaravgift i Norrbotten/ Svappavaara: Metodbeskrivning (Employment effects from reduced payroll taxes in Norrbotten/Svappavaara: Specification of the evaluation method), SIND PM 1984:2, Stockholm. Bohm, P. and H. Lind, 1988, Sysselsittningseffekter av sankt arbetsgivaravgift i Norrbotten 19841986 (Employment effects of reduced payroll taxes in Norrbotten 19841986) Research paper in economics 1988: I, Department of Economics, Stockholm University. Diamond, D. and N. Spence, 1983, Regional policy evaluation: A methodological review and the Scottish example (Cower Publishing Company, Hampshire). Folmer, H., 1986, Regional economic policy: Measurement of its effect (Martinus Nijhoff Publishers, Dordrecht). Folmer, H. and P. Nijkamp, 1987, Investment premiums: Expensive but hardly effective, Kyklos 40, 43-72. Isserman, A. and J. Merrifteld, 1982, The use of control groups in evaluating regional economic policy, Regional Science and Urban Economics I2,43358. Lind, T. and J. Serck-Hansen. 1972, Regional subsidies on labour and capital, Swedish Journal of Economics 74, 68-84. Moore, B. and J. Rhodes, 1973, Evaluating the effects of British regional economic policy, Economic Journal 83. 87-1 IO. Newman, R. and D. Sullivan, 1988, Econometric analysis of business tax impacts on industrial location: What do we know, and how do we know it? Journal of Urban Economics 23, 2 155234. Tervo, H. and P. Okko, 1983, A note on shift-share analysis as a method of estimating the employment effects of regional economic policy, Journal of Regional Science 23, 115-l 21.