Generating coset representatives for permutation groups

JOURNAL

OF ALGORITHMS

Generating

2, 221-244 (1981)

Coset Representatives Groups

for Permutation

LLOYD ALLISON Department of Computer Science, University of Western Australia, Nedlanh, Western Australia 6009, Australia Received January 3, 1980; revised March IO, 1981 A group of symmetries can divide the group of all permutations of n marks into equivalence classes of isomorphs; each class is a coset of the group of symmetries. An algorithm for generating a set of &morph, or coset, representatives is discussed. It can be used when n! is too large for all permutations to be useful, but the number of equivalence classes is manageable. The group of symmetries can be varied to match many natural notions of similarity. A computation on permutations that is restricted to a set of representatives can then concentrate on the essentially different cases.

This paper describes a backtrack algorithm for isomorph rejection in permutation generation, or equivalently, an algorithm for generating coset representatives of a permutation group. The backtrack progr amming technique is one of the oldest programming paradigms. The term is attributed to Lehmer in the 1950’s and denotes a common principle upon which there are numerous variations. Walker [15] gave possibly the first problem independent description of the technique, Golomb and Baumert [3] showed it to be of very wide use and Gerhart and Yelowitz [2] have covered its “parameterization” more fully and more formally. Usually backtracking is formulated as a search in the direct product space x, x x, x * * - XX, for solution vectors (x,, x2,. . . , x,), satisfying some criteria. The search is carried out in such a way that the vectors are built up from the left, element by element. Figure 1 gives a simple recursive coding. There are potentially II;=, )Xi ] vectors to consider. The term backtracking derives from the fact that in many problems it is often clear rather early in the construction of a vector that the stem (x,, x2,. . . , xi) i < n cannot be the stem of any solution and that there are no valid choices xi+,; Si+l = +; 227 0196-6774/81/030227-18$02.00/O Copyright 0 1981 by Academic Press, Inc. All rights of reproduction‘in any form reserved.

228

LLOYD

ALLISON

procedure bt(m: integer; S,,,: choiceset); begin for each choice,,, in S,,, {if any} do if m = n then process else bt(m + 1, S,,,) (return IO levels < m) end (bf) initial cak br( 1, S, ) FIGURE

I

it was on the wrong track and different choices should be made for the ith, and possibly earlier, coordinates. The advantage is that up to n;=,+ I 1Xi ] fruitless branches are eliminated from the tree of calls and this can lead to an efficient algorithm. Variations in objective include finding any solution, the best solution, all solutions or all nonisomorphic solutions. Variations in coding include recursion or iteration, and the use of parameters or global variables to pass information. In many problems the sets Xi are all the same, and the valid choices Si for the ith coordinate are rather easily calculated from S,- , . From the first, it was realized that solutions to many problems fall into classes of isomorphs in the presence of a group of symmetries, all members of a class being thought equivalent. It can be just as wasteful to generate all isomorphic solutions as to investigate the subtree beyond a fruitless or invalid stem. Swift [ 121 discusses some techniques for isomorph rejection in particular problems, but also makes the important general observation that if isomorphs can be recognized as a stem then they can be rejected in the same way as fruitless branches and with the same gains. Indeed, isomorph rejection can be considered backtracking with the modified criteria function: Accept solutions but only nonisomorphic ones. Perhnan [7] gives a rather general backtrack algorithm for rejecting isomorphs on power sets which operates in this way. This paper examines isomorph rejection in permutation generation. Being an instance, it is possible to completely specify an algorithm. It is hoped that the problem is still general enough to be of wide use, and it is at least general to the extent of leaving the group of symmetries quite arbitrary. Throughout, a permutation will be considered a l- 1 function on some set of marks. Permutations can be naturally extended to operate on sets of marks and also on permutations. S, denotes the group of all permutations of n marks. Z,, denotes the cyclic group of order n. 1.

PERMUTATION

GENERATION

Since the output of an exhaustive permutation generator rises exponentially with the number of marks, n, its runtime can hardly behave better.

GENERATING

COSET

REPRESENTATIONS

229

The fastest algorithms are iterative, but they are only fast in having a small constant of proportionality; their complexity is still 0( n !). The rapid growth of n! means that it probably will never be possible to enumerate, let alone to use, all of S,, for an n much over 15. However, depending on the symmetries in a problem, there are between one and n! isomorph classes: A computation restricted to a set of representatives will usually save a great deal of time! Sedgewick [lo] has reviewed the considerable body of work on exhaustive permutation generation, covering many ingenious methods of enumerating permutations quickly or in special orders. A simple recursive permutation generator which is adapted from Rohl[8] is given in Fig. 2. By various additions it can be made to generate only the representatives mentioned before. A permutation is a l- 1 function on the set of marks. If the marks are given some arbitrary order, the algorithm of Fig. 2 builds a permutation by choosing images for the first mark under the permutation, and calls itself recursively to choose images for the succeeding marks. It is an example of a backtrack algorithm (although there always is a valid choice to make at each call) and by restricting the range of its choices, as described below, it can be made to generate only the representatives.

2.

SYMMETRIES

Symmetries do not only arise in geometrical problems. For example, certain functions of n variables, which occur when investigating the solution of polynomials by radicals, exhibit symmetries between their arguments. In fact the problem studied here was originally posed in this connection [5]. For example, $x2 + x22x3+ x$x, is invariant under a cyclic interchange of its arguments and, permuting those arguments, it takes only two values each of which can be written in three ways. Supplying actual arguments a, b, c these are a2b + b2c + c2a = b2c + c2u + a*b = c2u + a2b + b2c pmcedure choose (m: integer; freemarks: markset); begin for each mark in freemarks do begin choice [ml: = mark; if m = n then process else choose ( m + 1, freemarks-[ mark]) end

{choose} initial call: choose ( 1, allmarks) end

FIGURE 2

230

LLOYD

ALLISON

and u2c + c2b + b2a = b2u + u2c + c2b = c2b + b2u + u2c.

Consider the function P( x1,

X2,

X3,

X4,

X5,

X6)

=

X1X2X3

+

X4X5X6.

(2.1)

Although there are 6! = 720 ways of permutingp’s arguments, doing so only produces 10 distinct values as the 720 arrangements fall into 10 classes each of 72 equal valued members. Supplying actual arguments a, b, c, d, e, f the 10 values are ubc ube ucd ucf udf

+ + + + +

def cdf bef bde bee

ubd + cef ubf + cde ace + bdf ude + bcf uef + bed.

p is the sum of two terms, each term being the product of three arguments. p is invariant under a cyclic permutation of the arguments in the first term such as

a denoted (2,3,1,4,5,6) or under an interchange of x, and x2 P:(2,1,3,4,5,6).

Similar remarks apply to the second term giving y:(1,2,3,%64)

&(1,2,3,5,4,6) and finally, interchanging

the two terms also leaves the value of p unaltered

Any sequence of the operations a, fi, y, 6, e, applied in any order, will also leave p’s value invariant, as will any inverse operation. If we are interested in the distinct values that can be got by permutingp ‘s arguments, it is clearly best to produce 10 representative permutations that yield distinct-valued arrangements rather than exhaustively generating all 720. Strictly by the way, note that in this example the terms within the 10 distinct valued arrangements enumerate the combinations of u to f taken

GENERATING

COSET REPRJZSENTATIONS

231

three at a time. An algorithm to solve this problem can also be used as a (rather complex) combination generator. Here, permutations are considered to be l- 1 functions on the set of marks, and they may be composed: (any mark) (a.p)

= ((any mark)a)p.

S, forms a group under composition, with identity the identity permutation. Under composition the permutations are extended to act on permutations in a natural way. In the example, (Y,/3, y, S, c generate a subgroup G of S, which consists of the 72 symmetries of p (2.1), each of which can be expressed in terms of the five generators: G = (a, P, Y, 6, c) < S,. In general, if H < K are groups, we can define two elements of K as equivalent under H if there is an element of H carrying one onto the other. The equivalence classes are clearly left cosets: If (J E K then aH = { ag 1v E H} is a left coset of H in K. It is a basic result of group theory that the cosets are disjoint, all of equal size, and partition K. The number of cosets equals the index of K in H: IH: KI = IKI/IHI.

(In a similar way Ha = {qalq E H} is a right cosetof H. Right cosets are in general distinct from left ones, the simplest example of this being (isomorphic to) (p ) < ((Y, p ) .) In the example, I G : S, I = 10 and all the elements (permutations) in a given left coset of G correspond to the same value of p. This can be seen by applying any u E S, to p and following this by any q E G. 1) necessarily does not alter the value any further. An algebraic statement of the subject of this paper is to find a set of left coset representatives for a given subgroup of S,. Note that group theorists have long used the Todd-Coxeter algorithm [ 1,141 to enumerate coset representatives of groups presented as generators and defining relations, but there does not seem to be a straightforward way of applying it to groups presented in terms of permutations. 3. CHOOSING

REPRESENTATIVES

The algorithm of Fig. 2 generates each of the n! permutations in S,, by a succession of choices: for the images of the first mark and of subsequent marks by deeper recursive calls. If G is a subgroup of S,, the algorithm can

232

LLOYD

ALLISON

be modified to generate only coset representatives as follows. (It is convenient to represent G by a set of generators X, which will be much smaller than G. G = (X).) Let G be the symmetries of p (2.1) as before. Consider where to place the first actual parameter “u” in the function. It may as well be placed as the first element of the first term because there is some symmetry in G that will carry it to any other position (G is transitive). Doing this removes much of the symmetry of the function. Let P’&, x3, x 4, x5, x6) = ux2x3 + x4x5x6. The second term of p’ can be permuted as before but the two terms cannot now be interchanged. There are two representative positions at which to place “b”: abx, + x4x5x6

and

ux2x3 + bx,x,.

For each of these two choices we can continue in a similar way and, as Fig. 3 indicates, arrive at the 10 distinct values; positions with a common tag (o or *) are equivalent. More formally, G partitions the arguments/marks into orbits orbit(m,G)

= {(m)qln

E G}.

It is sufficient to let the first choice range over a set of representatives of the orbits, as there is some symmetry in G to carry each representative onto any other mark in its orbit. (The orbits are equivalence classes.) A similar

FIGURE

3

GENERATING type orbittype

generators

= record end; = record

COSET

REPRESENTATIONS

norbits: 0.. n; orb: array [mark]

233

of markset

ngenerators: 0.. maxgen; perm: array [ 1.. maxgen]

of permutation

end; procedure choose (m: integer; freemarks: markset; X: generators); ((X) leaues choice [I], . . , choice [m - I] fixed} begin calculate orbits of marks under ( X) ; for each orbit do if orbit IT freemarks{) empty then begin choice [ml: = first (orbit fl freemarks); if m = n then process else begin calculate Xk a generator set for stabilizer of choice [m] in (X); choose (m + I, freemarks-[ choice [ml], Xk); end end end (choose) initial call: choose (I, allmarks,

giuen generators) FIGURE

4

argument applies at deeper levels in the recursion except that the subgroup of G which fixes or stabilizes all current choices must replace G. Since Fig. 2 produces (all) permutations in lexicographic order, if orbits are ordered by their least member then the modified algorithm, given in Fig. 4, will produce representatives in this order. Figure 4 is essentially Perlman’s [7] algorithm in recursive form and made particular to permutations. It is also a permutation generator employing Swift’s [12] observation that only nonisomorphic stems need to be pursued for that is what the orbit representatives construct. What remains to be provided are algorithms to calculate orbits and (generators for) stabilizer subgroups. These follow in the next two sections.

4.

ORBITS

In the example (2.1) G is transitive-there is only one orbit consisting of all arguments/marksalthough this is not generally so. The set of orbits under a group of permutations is a partition of the set of marks. If a transitive group is one extreme, the other is the trivial group {id} which gives rise to n orbits each of one mark. An algorithm for calculating orbits in the general case is described below.

234

LLOYD

Each permutation

ALLISON

consists of a product of disjoint cycles

m, (mh, (m)v2,. . . , (m)$ = m,

k?

0,

as the common cycle notation [4] clearly illustrates. The orbits due to a group of one generator are easily found by following the disjoint cycles, each one forming one orbit. But in the presence of several generators the effects of all combinations of them must be taken into account. We may define a graph in which the nodes represent marks. For each generator p E X and each mark m we connect m and (m)p. These arcs can be calculated in O( n * ) XI). The orbit of a mark is the set of nodes connected to it in the closure of this graph. Figure 5 shows that although no single generator of G is transitive, the group as a whole is and so there is only one orbit and only one representative first choice, x,, say. The subgroup G, leaving this first choice fixed can be generated by three permutations

G, = (4, Y, 6) and it partitions the remaining marks into two orbits {x2, x3} and {x4, x5, x,}-see Fig. 6. Representatives are x2 and x4, and G,,, = (y, S) are the respective stabilizers for each of these second

G acting

on

~x~,x~,x~,x~,x~,x~~

FIGURE

5

GENERATING

COSET

REPRESENTATIONS

235

FIGURE 6

Warshall’s algorithm can be used to calculate the closure of the graph described above. It is convenient to represent the graph as a collection of sets, {(m)plp

E X} for each mark m.

As noted by Rohl [9] the closure can then be calculated in O(n2) provided that set operations are fast. If n is less than a “small” multiple of the machine word length, as assumed here, this can be guaranteed. Finally, the distinct sets in the closure can be extracted in O(n2), at worst, and these are the distinct orbits. A procedure to calculate orbits on these lines is given in Fig. 7.

5.

STABILIZERS

Each call of choose (Fig. 4) is passed a set of generators for the subgroup H of symmetries that fixes all existing choices. Using the orbits under H, choose can make representative current choices of marks. For each current choice k, choose must calculate generators X, for the stabilizer Hk of that choice (X,)=

Hk< H=

(X).

As before, let G = (a, p, y, 6, E) . Following one particular tree in Fig. 3 we get the following chain of subgroups: Gl G 1.4

= (44 Y, a>,

G 1,4,2

= (YS),

= (44 YS>,

G 1,4,2,5 = {id) = G1,4,2,5,3

- G1,4,2,5,3,6,

branch of the

236

LLOYD ALLISON procedure

findorbits

(var orbits:

orbitype;

var X: generutors);

begin witb orbits do

begin for each murk do begin orb [murk]: = [murk]; for each perm in X do (connections} orb [murk]:

= orb[ murk]

+ [ perm

[murk]]

end;

If x.ngenerators > 0 then begin for each murk, do (closure) foreachmarkldo If orb[mark,] rl orb[ murk,]( )[ ] then beginorb[mark,]:= orb[murk,] + orb[mark2]; orb[mark2]:= orb[mark,] end; norbits: = 0; for each murk

beginpt:= while (pt

do (remooe

duplicates)

1; < norbits)

and(orb[pt]()orb[murk]) pt: = suqpt);

do

If pt > norbits then begin norbits: = pt; orb[ norbits]: = orb[ murk] endend end else (no generators} norbits: = n end; end (findorbits}

FIGURE I

on the way to ace + bu”. That these are correctly generated can be easily checked by hand. Tarjan has noted [13] that a group H has some generator set of size log,(] HI). However, quickly finding such a set without all of the group to peruse seems to be rather difficult. If such a set can be found, the space requirements for a stabilizer are bound by n log(n!) I n2 log n integers.

5.1. Calculating Stabilizers

Hall [4, Lemma 7.2.21 shows that a generator set for Hk can be formed as

where u runs over members of X, and .$runs over right coset representatives of Hk in H, and 9(n) is the (right) coset representative of the coset of Hk containing 9.

GENERATING

COSET REPRESENTATIONS

237

For any right coset H,J and any 8 E H,& 8 = At for some A E Hk. Then (k)e = (k)U = (k)E, i.e., all members of Hk[ have the same effect on k. To find right coset representatives of Hk in H it is sufficient to find permutations in H that carry k onto each member of its orbit under H. (Note that whatever (k)ta is, cp(&~)-’ will carry it back to k so that, as defined, (X, ) I Hk. The equality is a little harder.) Each member of k’s orbit corresponds to a right coset of Hk in H. k itself corresponds to H,id, whose elements have no effect on k, and the identity permutation is a suitable representative of this coset. Returning to p, there is a single orbit under G consisting of all arguments/marks. One set of right coset representatives for G, in G is {id, a, a*, /3, a/?, a*/?} as these carry x, onto all members of its orbit. In calculating such a set of representatives, a start can be made by considering the effect of each generator in X, and its powers, on k. So a and a* carry x, to x2 and x3, and b carries it to x4. This is not all of the orbit but the process can be repeated on these k-images just produced until the whole orbit is covered. Since each permutation in H can be expressed in terms of the generators X, the process must terminate. Each time that a new k-image is discovered a new coset representative of Hk is available. The time complexity of this step is highly dependent on the generators X. At best it is O(lorbit (k)!) when one generator includes a cycle around the entire orbit. Worse cases can be constructed (at least). Given the coset representatives of Hk, X, is calculated according to (5.1). A severe hazard is that in ranging over members of X and over coset representatives, it is likely that 1X, 1 >> 1XI. Since the process is carried out at each level in the recursion, the size of generator sets can “explode,” increasing both space and time requirements. In practice, simply checking that neither the identity permutation nor duplicates are added to X, often keeps I X, ] in reasonable limits-see Section 16. In this case the process is roughly O(\ XI . I X, ) . )Hk : H 1) and in favorable circumstances the calculation of a stabilizer does not exceed O(n3). Such a stabilizer procedure is given in Fig. 8. (A more expensive test- that powers were not being added to X, -did not prove worthwhile.) In some problem instances most symmetries move “most” marks or they incorporate a “long” cycle of marks. Once one of these marks has been chosen, the stabilizers become rather small and so too do their generator sets. This can be seen withp once x, is chosen. It may happen that several stabilizers in sequence are identical; this is so if orbit (k, H) = {k} and X, can be set to X immediately. Again, withp for example, G1.2 = G1.2.3 = (Y, 6). In general however, there is so much freedom in the choice of the symmetries G that any simple expedient is likely to be outwitted eventually and produce bad generator sets for stabilizers.

238

LLOYD

ALLISON

procedure stabilizer (k: mark; var X, Xk: generators); var rep: array [mark] of permutution; notcompleted, kimgs, thekorbit: markset: begin

if X. ngenerators

= 0 then Xk. ngenerators: = 0 else begin calculate the k orbit; rep [k]: = identity; kimgs: = [k]; notcompleted: = [k]; while true do begin mark: = first (notcompleted); notcompleted: = notcompleted - [mark]; for each perm in X do begin thep: = perm; while thep[ mark]()mark do begin if not (thep[ mark] in kimgs) then {new

representative}

begin kimgs: = kimgs + [ thep[ mark]]; notcompleted: = notcompleted + [ thep[ mark]]; rep [ thep [ mark]]: = rep [ mark] o thep end; if kimgs = the k orbit then calculate Xk; {else) thep : = thep o perm end (while}

end {for} end (while] ; calculate Xk: Xk. ngenerators: = 0; for each mark in the k orbit do for each perm in Xdo beginp:= rep[mark] operm; thep: = p o inverse(rep[ p[ k]]); if thep () identity then begin Xk.perm[ ngenerutors + I]: = thep(sentinel); pt:= I; while thep()Xk.perm[pt] dopt:= succ(pt) if pt > Xk. ngenerutors then Xk. ngenerators: = pt end end end {stabilizer)

FIGURE 8

GENERATING

5.2. Generator

COSET

REPRESENTATIONS

239

Sets

A possible way to always produce good generator sets is to test each new candidate for X, to see if it is generated by existing members, discarding it if it is. The empty set generates the trivial group of one member. If a set Y generates a group J and p @ J, then if (p} U Y generates J’, J’ > J and J’#J, so IJ’I L 2/J]. c onsequently such a process would accept at most log, I Hk I members for X,. Unfortunately this “permutation generation problem” (PGP)-is pE (Y ) -is a difficult test. Sims [ 1l] describes a sophisticated algorithm for it that is usually fast for very large values of n. However, the best known bound is exponential. The permutation generation problem has not been proved hard although the more general “finite-function generation problem” has been shown PSpace complete by Kozen [6]. If Q is the quantity of output of choose (Fig. 4), or the number of representatives, Q = ]G : S, I = n!/]G I. The number of calls on choose and stabilizer is 1 n and I nQ. At each call the number of candidates for X, is the number of cosets I Hk : H ] times the number of generators of H; under the method sketched above this is 5 n log] H 1 I n logIG I. Through all calls, the total number of candidates is I n2Qlog]G], although this is probably a generous limit. The number of candidates per final output is I n2 log] G I I n3 log n. The larger G is, the more complex are its subgroups (on average) and the smaller is Q. So that for large G the running time of the algorithm would become dominated by the detection of redundant candidates which has as complexity a polynomial in n, times the complexity of PGP, for each output. A better solution would be a more direct method than (5.1) of calculating good generator sets for stabilizers- but this might not be possible. It is interesting to note that Sim’s PGP algorithm can also be used to enumerate a group, given its generators, and is in a sense the converse of generating coset representatives. It employs a particular descending chain of stabilizer subgroups and various techniques are used to keep the chain short and the stabilizers economically generated. These techniques do not seem to be open to coset generation directly because the chain of stabilizers is not fixed, but varies, depending on the choices made and the choices depend on the stabilizers; a tree of stabilizers is employed and those in a subtree cannot be formed until the stabilizer at its root is known.

6.

PERFORMANCE

The algorithm was coded in Pascal and run on a CDC Cyber 72. All runtime checks were left on and no attempt was made to fine tune the code.

240

LLOYD

ALLISON

The simple method of Section 5.1 for calculating fast PGP algorithm was not available.

stabilizers was used, as a

6.1. Time There are two extreme cases of subgroups G < S,,; one is G = {id} when no two members of S,, are equivalent, and the other extreme is G = S,, when all members are equivalent to the identity permutation. (It would of course be possible to include specific tests for these special cases and employ a different algorithm for them, but this was not done.) In the first case the algorithm behaves exponentially like any exhaustive permutation generator but with a poor constant of proportionality due to its extra complexity; it is about seven times slower than the simple algorithm of Fig. 2. In the G = S,, case the time complexity is O(n3’5) over the range examined (n = 2 to 15).

slope

FIGURE

9

= -71

GENERATING

COSET

241

REPRESENTATIONS

The reason for this latter behavior is that although the “tree” of recursive calls is extremely thin, consisting of one branch at each level, it is still n deep and the amount of work in calculating stabilizers (always isomorphic to S,,, 1 I m I n) depends on n. S,,, is generated, for example, by m) and (2,3,4,. . . , m, 1) but the stabilizer algorithm of Sec(2,1,3,4,..., tion 5.1 is not clever enough to produce such a generator set. In this case the size of stabilizer generator sets rises rapidly with the level of call- to n + 1 as it happens-and then tails away in deeper calls. The computation times for other cases fall between the two extremes when the stabilizer subgroups are well behaved. Figures 9 and 10 show total time versus n for {id} < S, and S,, < S,, Figure 11 shows the times for some subgroups of S,. (Z,,, is a cyclic group of order m, e.g., ((2,3,. . . , m, 1)) .) In general it is not possible to predict time just from n and the subgroup’s index, although patterns are evident for certain classes of subgroup.

n=15 loglO(time

msec)

/ 3.0'

2.0'

1.0

slope

, 0.0 ^ ^ “.”

= 3.5

/ loglo

I’ ^ 0.2

0.4

0.8

0.6 FIGURE

10

1.0

(n)

242

LLOYD

ALLISON

3.5

3-c

2.5

'6 log 1o (index) I

2.c 0.0

-

.

-

.

1.0

I

.

.

I

.

2.0

FIGURE

It is quite feasible to “enumerate” S,,--it takes a few seconds.

11

the one coset representative of S,, -=I

6.2. Space The depth of recursion for the algorithm is exactly n. The space required is bounded by n times that needed for one call of choose (Fig. 4). The orbits can be represented compactly by at most n sets and it is space for generators of stabilizers that dominates. Unfortunately, as mentioned in Sections 5.1 and 6.1, this is not predictable. Various tests were run using the stabilizer procedure of Section 5.1, Fig. 8, which uses the simple test of not adding duplicates or the identity to X,. For G = S,, < S,,, G generated by a transposition and a lo-cycle, the sizes of generator sets at descending levels in the recursion were 10,ll. 10,9,8,6,4,1,0. This was independent of the order of the generators or the choice of the particular transposition. Generating S,, by 10 transposi-

LLOYD ALLISON

243

tions the sizes were 11,12,11, 10,9,7,&l, 0 at descending levels. As the stabilizers are S,, Ss, S,, . . . this is quite satisfactory. These results are typical of S, < S,. Performance was also good for groups which are simple products of symmetric groups. For example, S, X S, X Z, < S,, gave rise to sets up to size 10,9,8,6,5,4,1,0 although not necessarily on the same trunk of the tree. If the group of symmetries is large and the generators interact in a complex way, the simple test is easily beaten. (2,3,4,5,1,6,7,8,9,10,11,12) (1,2,3,4,5,6,8,9,10,11,7,12) (1,2,7,3,5,6,12,8,9,4,11,10) and (1,7,3,6,5,2,11,8,9,10,4,12) generate a transitive group G < S,,. The algorithm calculated 31 generators for G, and more than 100 for the next level stabilizers before running out of space. If this method of generating nonisomorphic permutations is to be used with large groups of symmetries with arbitrary structure, then a better method of calculating stabilizers must be found. Using a crude PGP algorithm in the way suggested in Section 5.2 did produce small generator sets, regularly finding two or three generators for S,,. However, its time complexity was very poor and conclusions could not be drawn for that parameter.

7. CONCLUSIONS An algorithm has been given which generates a set of permutations that differ from each other in a useful and general way: A set of coset representatives for G < S,,. Many problems have a group of symmetries G, and such sets of representatives are generally much smaller than all of S,,. If called upon to generate all n! permutations the algorithm is a constant factor worse than existing permutation generators. When generating the one representative for S, < S,, it is O(n3.5). Other computations have intermediate times if the group of symmetries has a simple structure. Space and time needs may become excessive for large groups of complex structure. In cases when processing time due to using the permutations dominates, an obvious saving results from only generating and then processing the essentially different representatives.

244

LLOYD

ALLISON

REFERENCES

I. J. J. CANON, L. A. DIMINO, G. HARVAS AND J. M. WATSON, Implementation and analysis of the Todd-Coxeter algorithm, Math. Comp. 27, No. 123 (July 1973), 463-490. 2. S. L. GERHART AND L. YELOWITZ, Control structure abstraction of the backtracking programming technique, IEEE Trans. Software Engrg. SE2, No. 4 (Dec. 1976), 285-292. 3. S. W. GOLOMB AND L. D. BAUMERT, Backtrack programming, J. Assoc. Comput. Mach. 12, No. 4 (Oct. 1965), 516-524. 4. M. HALL, “The Theory of Groups,” MacMillan, New York, 1959. 5. A. K. HEAD, The Galois unsolvability of the sextic equation of anisotropic elasticity, J. Elasticity 9, No. 1 (Jan. 1979), 9-20. 6. D. KOZEN, Lower bounds for natural proof systems, in “Proceedings, 18th Annual Symposium on Foundations of Computer Science,” pp. 254-266, IEEE, New York, 1977. 7. 0. M. PERLMAN, Isomorph rejection on power sets, SIAM J. Compur. 3, No. 3 (Sept. 1974), 177- 183.

8. J. S. ROHL, Generating permutations by choosing, Compur.

J. 21, No.

4 (Nov.

1978).

302- 305.

J. S. ROHL, On sets in programming, in “Proceedings, Australian Computer Science Conference, Hobart (Feb. 1979).” IO. R. SEDGEWICK, Permutation generation methods, Comput. Surueys 9, No. 2 (June 1977), 9.

137-164.

I I. C. C. SIMS, Some group-theoretic algorithms, in “Topics in Algebra,” Lecture Notes in Mathematics No. 697, pp. 108- 124, Springer-Verlag. Berlin/New York, 1978. 12. J. D. SWIFT, Isomorph rejection in exhaustive search techniques, in “Proceedings, 10th Symposium in Applied Mathematics of the A.M.S. (April 1958);’ pp. 195-200. 13. R. E. TARJAN, Private communication to G. L. Miller. 14. J. A. TODD AND H. S. M. COXETER, A practical method for enumerating cosets of a finite abstract group, Proc. Edinburgh Math. Sot. (2) 5, (1936). IS. R. J. WALKER, An enumerative technique for a class of combinatorial problems, in “Proceedings, 10th Symposium in Applied Mathematics of the A.M.S. (April 1958),” pp. 91-94.