Transitive permutation groups where nontrivial elements have at most two fixed points

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.1 (1-31) Journal of Pure and Applied Algebra ••• (••••) •••–•••

Contents lists available at ScienceDirect

Journal of Pure and Applied Algebra www.elsevier.com/locate/jpaa

Transitive permutation groups where nontrivial elements have at most two fixed points Kay Magaard a,∗ , Rebecca Waldecker b a b

University of Birmingham, Birmingham, B15 2TT, UK Martin-Luther-Universität Halle–Wittenberg, 06099 Halle (Saale), Germany

a r t i c l e

i n f o

Article history: Received 5 March 2013 Received in revised form 25 March 2014 Available online xxxx Communicated by E.M. Friedlander

a b s t r a c t Motivated by a question on Riemann surfaces, we consider permutation groups that act nonregularly, such that every nontrivial element has at most two fixed points. We describe the permutation groups with these properties and give a complete, detailed classification when the group is simple. © 2014 Elsevier B.V. All rights reserved.

1. Introduction From its inception the theory of permutation groups has been concerned with the question of how fixed point sets of elements influence the structure of a permutation group. For example a celebrated result by Jordan is the fact that a primitive permutation group which contains a transposition is the full symmetric group. The special case where the permutation group has degree 5 was used by Galois to prove the unsolvability of the general quintic. Jordan’s result was strengthened and generalized in several ways. For example the maximal subgroups of Sn which contain transpositions are either intransitive of the form Sk × Sn−k or imprimitive of the form Sk  Sn/k . Other results concern primitive permutation groups in which some nonidentity element fixes many points. For example in [16] one can find the explicit list of exceptions to the statement that a primitive permutation group where some nonidentity element fixes at least half the points of the permutation domain contains the full alternating group. We would like to determine the group theoretic structure of a transitive permutation group where all nontrivial elements fix at most a bounded number of, say k, points. Of course the case k = 0 is when the action of the group on its permutation domain is regular. The case where k = 1 is the situation that Frobenius characterized; i.e. Frobenius groups. Motivated by an application to the theory of compact Riemann surfaces, see Corollary 1.5, we investigate the case k = 2. Although we do not impose any hypothesis on primitivity or higher transitivity, we would like to mention the related work of Zassenhaus and Suzuki, see for example [26,25] and Theorem 2.9 in [22]. From this point of view the series of groups that we encounter are no * Corresponding author. E-mail addresses: [email protected] (K. Magaard), [email protected] (R. Waldecker). http://dx.doi.org/10.1016/j.jpaa.2014.04.027 0022-4049/© 2014 Elsevier B.V. All rights reserved.

JID:JPAA 2

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.2 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

surprise. In fact our results show that groups satisfying our more general hypotheses can have arbitrarily large permutation rank. For details, see the remarks at the end of this article. From now on we operate under the following Hypothesis 1.1. Suppose that G is a finite, transitive, nonregular permutation group with permutation domain Ω. Suppose further that |Ω| ≥ 4 and that every element g ∈ G# has at most two fixed points. For simple groups we prove Theorem 1.2. Suppose that (G, Ω) satisfies Hypothesis 1.1 and that G is simple. Then either G is isomorphic to PSL3 (4) or there exists a prime power q such that G is isomorphic to PSL2 (q) or to Sz(q). We will show in the last section that there are no quasisimple, nonsimple examples. For the almost simple groups we have Theorem 1.3. Suppose that G is almost simple, but not simple. Let E := F ∗ (G) and suppose that (G, Ω) satisfies Hypothesis 1.1. Then there exists a prime power q such that E ∼ = PSL2 (q) and one of the following holds: (a) G ∼ = PGL2 (q). (b) q is a power of 2, moreover |G : E| is prime, and there exists an element g ∈ G\E such that g induces a field automorphism on E and CE (g) ∼ = PSL2 (2). (c) q is odd, |G : E| = 2 and the elements of G \ E act as diagonal-field automorphisms of order 2. This includes the case where E ∼ = A6 and G ∼ = M10 . = PSL2 (9) ∼ Our most general result is Theorem 1.4. Suppose that (G, Ω) satisfies Hypothesis 1.1. Then one of the following holds: (1) G has a subgroup of index at most 2 that is a Frobenius group. (2) |Z(G)| = 2 and G/Z(G) is a Frobenius group. (3) The point stabilizers are metacyclic of odd order. If H is a nontrivial two point stabilizer, then |NG (H) : H| = 2, G is solvable and H or NG (H) has a normal complement K in G such that K is nilpotent and (|K|, |H|) = 1. (4) The point stabilizers are metacyclic of odd order. Moreover G has normal subgroups N , M such that N < M < G, N is nilpotent, M/N is simple and isomorphic to PSL2 (q), to Sz(q) or to PSL3 (4), and G/M is metacyclic of odd order. (5) The point stabilizers have twice odd order and G has a subgroup M of index 2 such that either (3) or (4) holds for M or M acts regularly on Ω.  (6) The point stabilizers have even order and G has a normal subgroup N of odd order such that O2 (G)/N is either a dihedral or semidihedral 2-group or there exists a prime power q such that it is isomorphic to Sz(q) or to a subgroup of P Γ L2 (q) that contains PSL2 (q). In part our result was motivated by its application to the theory of Riemann surfaces. A Weierstrass point of a compact Riemann surface X of genus g > 1 is a point x ∈ X such that there exists a holomorphic function which has a pole of order at most g at x and is holomorphic on X \ {x}. Apart from their analytic significance Weierstrass points also influence the structure of the automorphism group. For example, by considering the action on the Weierstrass points, Schwarz showed that the automorphism group of a compact Riemann surface X of genus g > 1 is finite. (See for example page 258 in [9].) Going in the opposite direction Schoeneberg showed that if an automorphism of a compact Riemann surface of genus g ≥ 2 fixes at least five

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.3 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

3

points, then every one of its fixed points is a Weierstrass point (page 264 in [9]). One can define the concept of a q-Weierstrass point (or higher Weierstrass point) by relaxing the condition “holomorphic function” to “holomorphic q-differential”, see for example page 87 in [9]. Now let X be a compact, connected Riemann surface of genus at least 2 with automorphism group G. If X → X/G is a ramified covering, then some g ∈ G# has fixed points on X. If x ∈ X is a fixed point of g ∈ G# and if Ω = xG , then Ω is a transitive non-regular G-set. Suppose that some element h ∈ G# has three or more fixed points on Ω. Then every fixed point of h is a higher Weierstrass point of X, as can be seen for example on page 284 in [9]. Thus all the fixed points of hk for k ∈ G are also higher Weierstrass points. Now the transitivity of G on Ω implies that all points of Ω are higher Weierstrass points. Therefore an application of our main theorem to the pair (G, Ω) proves the following. Corollary 1.5. Let X be a compact, connected Riemann surface of genus at least 2 with automorphism group G. If G is not one of the groups in the conclusion of Theorem 1.4 and if X → X/G is ramified, then for all g ∈ G# every fixed point of g on X is a higher Weierstrass point. This means that, under fairly mild group theoretic hypotheses, we can show that the fixed points of the automorphisms of a Riemann surface are analytically distinguished. We would like to point out however that there are situations when not every fixed point of an automorphism is a Weierstrass point, the Klein and the MacBeath curves being examples of this (see [21]). This paper is organized as follows. In Section 2 we present some small examples of groups satisfying Hypothesis 1.1 and collect preliminary lemmas before we start working towards our first structural result, Theorem 2.23. In Section 3 we determine which alternating or Lie type groups allow for examples and in each case we describe the actions in detail. In Section 4 we show that no sporadic simple group appears as an example. In Section 5 we prove Theorems 1.2 and 1.3 and then collect a few more specialized results towards the proof of Theorem 1.4. Finally we argue by looking at a minimal counter example and then exploiting Theorem 2.23 from Section 2 and our results on simple and almost simple groups. Finally we provide a more detailed version of our main result with additional information. 2. Preliminaries In this paper, by “group” we always mean a finite group, and by “permutation group” we always mean a group that acts faithfully. In this chapter let Ω denote a finite set and let G be a permutation group on Ω. Notation: Let ω ∈ Ω and g ∈ G, and moreover let Λ ⊆ Ω and H ≤ G. Then Hω := {h ∈ H | ω h = ω} denotes the stabilizer of ω in H, fixΛ (H) := {ω ∈ Λ | ω h = ω for all h ∈ H} denotes the fixed point set of H in Λ and we write fixΛ (g) instead of fixΛ ( g). We write ω H for the H-orbit in Ω that contains ω. For all n ∈ N, we denote the cyclic group of order n by Zn . 2.1. A characterization of Frobenius groups In this subsection we prove a little result that will strengthen some of our statements based on Hypothesis 1.1. While the content of the following lemma can be extracted from the proof of Proposition 16.17 in [14], we include a proof so as to make our arguments more self contained. Lemma 2.1. Suppose that G has a nontrivial proper subgroup H such that the following holds: Whenever 1 = X ≤ H, then NG (X) ≤ H. Then G is a Frobenius group with Frobenius complement H.

JID:JPAA 4

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.4 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

Proof. Let p ∈ π(H). We begin by proving that H contains a Sylow p-subgroup of G. For this let P ∈ Sylp (H). Then P = 1 and therefore NG (P ) ≤ H, so in particular P ∈ Sylp (NG (P )). This forces P ∈ Sylp (G) and we keep this notation. Next we show that H is strongly p-embedded in G. Let g ∈ G and suppose that x ∈ H ∩H g is a nontrivial p-element. Without loss x ∈ P . Then by hypothesis it follows first that Z(P ) ≤ H ∩ H g and then that −1 −1 P ≤ H ∩ H g . In particular P and P g are Sylow p-subgroups of H. Let h ∈ H be such that P h = P g . Then P hg = P whence hg ∈ NG (P ) ≤ H by hypothesis. We deduce that g ∈ H, so H = H g . This means that H is strongly p-embedded in G. But now, for all g ∈ G\H, we see that H ∩ H g = 1. This means that G is a Frobenius group with H as Frobenius complement. 2 2.2. Examples The smallest nontrivial examples for our situation are provided by S3 and A3 in their natural action on {1, 2, 3}. The natural action of A4 on {1, 2, 3, 4} also gives an example. In these cases S3 and A4 act as Frobenius groups. Lemma 2.2. The group G := S4 provides examples satisfying Hypothesis 1.1. The size of the set Ω that G is acting on is 4, 6, 8 or 12. In the action on six points, the point stabilizers have structure Z22 or Z4 , and in the other cases the point stabilizers are S3 , Z3 or Z2 , respectively. Proof. We use GAP [24] to extract the permutation characters from the table of marks, which yields our list. The first example is the natural action of G on {1, 2, 3, 4} where the point stabilizers are isomorphic to S3 . The second example is on the set of cosets of a non-normal fours subgroup or of a cyclic group of order 4 and the point stabilizer of the last example is cyclic generated by a transposition. 2 The natural action of S4 on {1, 2, 3, 4} generalizes to a series of examples where the point stabilizers are Frobenius groups. Lemma 2.3. Let r be a prime and let K be a finite field of order 2r . Let A denote the additive group of K, let M denote the multiplicative group of K and let H denote the Galois group of K over its prime field. Let F be the semidirect product of M and H. Then the semidirect product G := A · F acting on the set Ω := G/F of right cosets of F gives an example satisfying Hypothesis 1.1. Proof. Our choice of Ω implies that |Ω| ≥ 4 and that the subgroup A is a regular normal subgroup. The point stabilizer of the zero element in K is evidently F , showing that our G-action is not regular. Now let x ∈ F . Then |fixΩ (x)| = |CA (x)|. If x ∈ M , then |CA (x)| = 1 whereas, if x ∈ / M , then x is conjugate to a nontrivial element of H and hence CA (x) is a subfield of K. The hypothesis that r is prime forces |CA (x)| = 2. Thus all elements in F # have at most two fixed points. 2 Remark 2.4. The groups G in this series of examples are also known as AΓ1 (2r ), that is the group of affine semilinear maps of a 1-dimensional vector space over the field of 2r elements. If |K| = 4 in the lemma above, then G = S4 and F = S3 . If we drop the hypothesis that r > 1 is prime and a is a proper divisor of r, then the Galois correspondence implies that H possesses an element which centralizes the subfield of order 2a . As a ≥ 2, this violates our hypothesis about the number of fixed points. Also note that, if we drop the hypothesis that the characteristic p of K is even, then we can still construct G and F . However |K| = pr now, with p ≥ 3, and the possible fixed point sets of elements of G on the set of cosets of F will now have sizes 0, 1 or p. Again this violates Hypothesis 1.1.

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.5 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

5

As an example motivating the next lemma, we look at G := (12)(45), (23)(56), (14)(25)(36) ≤ S6 , acting naturally on Ω := {1, 2, 3, 4, 5, 6}. Then the pair (G, Ω) satisfies Hypothesis 1.1, and this idea can be generalized: Lemma 2.5. Suppose that F is a group that acts transitively on a set Δ of size at least two as a Frobenius group. Let Δ1 and Δ2 be disjoint sets such that F acts on both of them exactly as it acts on Δ and let t be an involution that centralizes F and maps Δ1 to Δ2 bijectively. Let G := F × t and Ω := Δ1 ∪ Δ2 . Then (G, Ω) satisfies Hypothesis 1.1 and the point stabilizers are the Frobenius complements of F . Proof. By hypothesis |Ω| ≥ 4. If ω ∈ Ω, then without loss ω ∈ Δ1 and hence Fω = 1. This implies that Gω = 1 and hence G does not act regularly on Ω. Moreover G is transitive on Ω because F is transitive on Δ1 and Δ2 and t interchanges these two sets. Let ω ∈ Ω and g ∈ Gω . Without loss ω ∈ Δ1 . Then g stabilizes Δ1 and hence g ∈ F . It follows that ω is the unique fixed point of g on Δ1 and that g also has a unique fixed point on Δ2 . Hence all nontrivial elements in G have either zero or two fixed points on Ω, and this means that Hypothesis 1.1 is satisfied. 2 Lemma 2.6. Let p, r be primes and let K be a finite field of order p2r . Let A denote the additive group of K, let M denote the multiplicative group of K and let H denote a subgroup of order 2 of the Galois group of K over its prime field. Let F be the semidirect product of M and H. Then the semidirect product G := A · F acting on the set Ω := G/M of right cosets of M gives an example satisfying Hypothesis 1.1. Proof. Our choice of Ω implies that G is transitive on Ω, and that |Ω| ≥ 4. The subgroup A has exactly two orbits on Ω and both of these are regular. Our choice of M implies that M fixes one of the two A-orbits, hence both. Moreover AM acts as a Frobenius group on each orbit, therefore every nontrivial element of M fixes two points. 2 Lemma 2.7. Suppose that G has a subgroup M of index 2 and that t ∈ G \ M is an involution such that |CM (t)| = 2. Suppose further that N is a t-invariant normal subgroup of M of even index and as large as possible subject to these constraints. Then t inverts N , and if |M : N | = 2, then N = Z(M ) and M/N ∼ = A4 . If |G| ≥ 8 and if we let Ω := G/Sα , then (G, Ω) with G acting by right multiplication satisfies Hypothesis 1.1. Proof. The first statements are proved in Satz 4.8 in [5], where Bender attributes them to Zassenhaus. The hypotheses yield that |Ω| ≥ 4 and that every element of G has zero or two fixed points in the action on Ω, hence Hypothesis 1.1 is satisfied. 2 2.3. Properties of groups satisfying Hypothesis 1.1 Throughout, we suppose that Hypothesis 1.1 holds. Lemma 2.8. Let α ∈ Ω and let 1 = X ≤ Gα . Then Gα contains a subgroup of NG (X) of index at most 2. If α is the unique fixed point of x ∈ G on Ω, then CG (x) ≤ Gα . Proof. As NG (X) acts on the set of fixed points of X on Ω, it is contained in Gα or it induces the symmetric group on two points on the two fixed points of X. Therefore NGα (X) has index at most 2 in NG (X). The second statement is immediate. 2 Corollary 2.9. If G has odd order, then G is a Frobenius group.

JID:JPAA 6

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.6 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

Proof. This follows from Lemmas 2.8 and 2.1.

2

Lemma 2.10. |Z(G)| ≤ 2. Proof. Let x ∈ Z(G)# and assume that x has a fixed point ω ∈ Ω. Then x ∈ Gω and the transitivity of G forces x to fix every point in Ω. This is a contradiction because |Ω| ≥ 4. Therefore x has no fixed points on Ω. Let α ∈ Ω and 1 = x ∈ Gα . Then Z(G) ≤ CG (x), but Z(G) ∩ Gα = 1 by the previous paragraph. Therefore |Z(G)| = |Z(G) : Z(G) ∩ Gα | ≤ 2 by Lemma 2.8. 2 Lemma 2.11. Let ω ∈ Ω. (a) If p is odd and p ∈ π(Gω ), then Gω contains a Sylow p-subgroup of G. (b) Suppose that S ∈ Syl2 (G) is such that Sω = 1. Then S is dihedral or semidihedral and |Sω | = 2 or Gω contains a subgroup of index at most 2 of S. In the second case, if S  Gω , then there exists δ ∈ Ω such that ω = δ, Sω = Sδ and some element in S interchanges ω and δ. Proof. For (a) suppose that p is odd and let P ∈ Sylp (G) be such that Pω = 1. Then Z(P ) ≤ Gω by Lemma 2.8 and, with the same lemma, also P ≤ NG (Z(P )) ≤ Gω . We turn to (b). Let Δ := ω S and let n, m ∈ N0 be such that |Sω | = 2n and |S : Sω | = 2m . First suppose that m ≥ 2. Let d denote the number of fixed points of Sω on Δ. Note that d is even, but d = 0, and thus Hypothesis 1.1 implies that Sω acts semiregularly on Δ \ fixΩ (Sω ). So now choose a ∈ N0 such that |Δ| = d + a · 2n . As n ≥ 1 and |Δ| = 2m ≥ 4, we see that d = 2 and hence 2m = 2 · (1 + a · 2n−1 ). This implies that a · 2n−1 is odd, in particular n = 1. Therefore Lemma 2.8 forces CS (Sω ) to be of order at most 4, and it follows that S has maximal class. Then 11.9 in [17] yields that S is quaternion, dihedral or semidihedral. But now we recall that |Sω | = 2, so in the quaternion case it contains the central involution of S and this contradicts the fact that |CS (Sω )| ≤ 4. Next suppose that m ≤ 1. If S  Gω , then |Ω| is even and hence Sω must fix a second point δ on Ω. As S does not fix ω, but normalizes Sω , there must be an element in S that interchanges ω and δ. 2 Lemma 2.12. Let S ∈ Syl2 (G) and α ∈ Ω. Then one of the following holds: (1) Gα has odd order. (2) S is dihedral or semidihedral and |Sα | = 2. (3) |S| ≥ 4, there is a unique S-orbit on Ω of length 2, and all other S-orbits have length |S|. In this case G has a normal subgroup of index 2 that is a Frobenius group. (4) |Ω| is odd. Proof. Suppose that neither (1) nor (4) holds. Then with Sylow’s Theorem we may suppose that Sα = 1, but S  Gα . Lemma 2.11(b) implies that (2) holds or that there exists β ∈ Ω such that α = β, Sα = Sβ has index 2 in S and some element in S interchanges α and β. (In fact all elements in S\Sα interchange α and β.) As Sα already has two fixed points, this subgroup must have regular orbits on the remaining points of Ω. It follows that {α, β} is the unique S-orbit of length 2 and all other orbits have length |S|. In particular |S| ≥ 4 and together with Hypothesis 1.1 this implies that |Ω| ≥ 6. For the last assertion in (3) let x ∈ S\Sα . If we view x as an element of the symmetric group on Ω, then there are two possibilities: Either x induces a single cycle on Ω \ {α, β} or an even number of cycles of 2-power length. In the first case x has order |S| and therefore S is cyclic. Then Burnside’s p-Complement Theorem yields that G has

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.7 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

7

a normal 2-complement and in particular G has a subgroup N of index 2. In the second case it follows that, viewed as a permutation on Ω \ {α, β}, the element x is an even permutation. But x interchanges α and β and therefore x is an odd permutation in its action on Ω. It follows that G is not contained in the alternating group on Ω, so again G has a normal subgroup N of index 2. We note that G = N · x whence Gα ≤ N . In particular N does not act regularly and it has two orbits of equal size on Ω that are interchanged by x. We denote the N -orbit that contains α by Γ and the orbit containing β by Λ. Suppose now that y ∈ Gα is such that |fixΓ (y)| = 2. Then Hypothesis 1.1 implies that |fixΛ (y)| = 0. Let a, b ∈ N0 be such that   |Γ | = fixΓ (y) + a · o(y) = 2 + a · o(y) and |Γ | = |Λ| = b · o(y). Then we deduce that 0 ≡ |Γ | ≡ 2 modulo o(y) and consequently o(y) = 2. So without loss y ∈ T , but then α and β are the unique fixed points of y. This is impossible. We conclude that nontrivial elements of N fix at most one point of Γ and thus N is a Frobenius group as claimed. 2 Lemma 2.13. Suppose that N  G. Let Ω denote the set of N -orbits on Ω and let G := G/N . Then for all ω ∈ Ω we have that Gω = Gω . Proof. Let g ∈ Gω . As N  G, we see that g stabilizes ω and hence Gω ≤ Gω . Conversely let N ≤ H ≤ G be such that H = Gω . Then Gω ≤ H and hence N Gω ≤ H. Moreover H acts transitively on ω and therefore |ω| = |H : Gω |. We also know that |ω| = |N : N ∩ Gω | and this yields that |N | · |Gω | = |H| · |N ∩ Gω |. |·|Gω | It follows that |H| = |N |N ∩Gω | = |Gω N | and therefore H = Gω . 2 Lemma 2.14. Suppose that |Ω| ≥ 5 and that |Z(G)| = 2. Let Ω denote the set of Z(G)-orbits on Ω and let G := G/Z(G). Then (G, Ω) satisfies Hypothesis 1.1. If the point stabilizers in G have odd order, then G is a Frobenius group. Proof. Let z ∈ Z # . Then z interchanges the points in every Z(G)-orbit, so by Hypothesis 1.1 it is the only element in G that acts this way. Now |Ω| is even, by hypothesis it has at least six elements now and Z(G) is the kernel of the action of G on Ω. In particular G acts faithfully on Ω. By Lemma 2.13 this action is not regular. Let α ∈ Ω and H := Gα . Then H fixes αZ(G) element-wise and hence it is a two point stabilizer. If H has odd order, then no element of H \ Z(G) interchanges two points of Ω and hence H fixes no other element of Ω. In particular every element of G# has at most one fixed point on Ω in this case, so G is a Frobenius group. In the general case we suppose that x ∈ G is such that x = 1 and that x has two fixed points α and β on Ω. By Lemma 2.13 we may choose x ∈ Gα and we have seen that this means that x fixes the two points in α. Now x must interchange the two points in β. This shows that x does not have a third fixed point on Ω, for otherwise it interchanges the two points in this third Z(G)-orbit and then xz has four fixed points (contrary to Hypothesis 1.1). This shows that (G, Ω) satisfies Hypothesis 1.1. 2 We remark that the dihedral group D := D24 provides an example with point stabilizers of order 2 and center of order 2 where the factor group D/Z(D) is not a Frobenius group.

JID:JPAA 8

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.8 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

Lemma 2.15. Let α, β ∈ Ω. If Gα ∩ Gβ is nontrivial and properly contained in Gα , then Gα is a Frobenius group with complement Gα ∩ Gβ . In particular, all Sylow subgroups of Gα ∩ Gβ are cyclic or quaternion. Proof. Let H := Gα and H0 := H ∩ Gβ . Now we look at the orbit Δ := β H . Then H acts transitively on Δ and nonregularly, because H0 = 1 by hypothesis. Every h ∈ H # has at most one fixed point in Δ because all these elements already fix α. Thus H is a Frobenius group with Frobenius complement H0 and it follows, for example from [17, Hauptsatz 8.7], that the Sylow subgroups of H0 for odd primes are cyclic and that the Sylow 2-subgroups of H0 are cyclic or quaternion. 2 Lemma 2.16. Let α ∈ Ω. Suppose that p ∈ π(G) is such that Op (G) = 1 and that |Gα | is odd. Then Gα is a Frobenius complement (hence metacyclic) or p = 2, Op (G) = Z(G) and G/Z(G) is a Frobenius group. Proof. First we observe that Op (G)Gα acts transitively and nonregularly on αOp (G) . If |αOp (G) | ≤ 2, then p = 2 and O2 (G) = Z(G). Then Lemma 2.14 implies that G/Z(G) is a Frobenius group. Hence we may now suppose that (Op (G)Gα , αOp (G) ) satisfies Hypothesis 1.1. If p is odd, then Lemma 2.11(a) yields that p does not divide |Gα |. Thus COp (G) (Gα ) = 1 by Lemma 2.8, which means that Op (G)Gα is a Frobenius group with Frobenius complement Gα . Hence suppose that p = 2. If Z := Z(O2 (G)Gα ) = 1, then O2 (G)Gα /Z is a Frobenius group with complement Gα by Lemma 2.14. If Z = 1, then Gα acts fixed point freely on O2 (G) and thus O2 (G)Gα is a Frobenius group with complement Gα . In both cases [17, Satz 8.18] yields that Gα is metacyclic. 2 Lemma 2.17. Let α ∈ Ω, and suppose that |Ω| is odd and that |Gα | is even. Suppose further that G has cyclic or quaternion Sylow 2-subgroups. Then G is a Frobenius group with Frobenius complement Gα . Proof. By hypothesis G has even order and Gα contains a full Sylow 2-subgroup S of G. As |Ω| is odd, Hypothesis 1.1 yields that S fixes exactly one point in Ω, hence only α. Let s ∈ S denote the unique involution. Then α is the only fixed point of s and hence CG (s) ≤ Gα by Lemma 2.8. Now the theorems of Burnside (see 7.2.1 in [19]) and of Brauer and Suzuki [6] give that G = Gα · O(G). Let 1 = X ∈ Gα and assume that X has a second fixed point β ∈ Ω. In particular 1 = Gα ∩ Gβ = Gα and Lemma 2.15 implies that Gα is a Frobenius group with complement Gα ∩ Gβ . As S has the unique fixed point α, it lies in the Frobenius kernel of this group. Now without loss X normalizes S (by coprime action) and hence centralizes s, which is impossible. Then Lemma 2.1 forces G to be a Frobenius group with Frobenius complement Gα . 2 Lemma 2.18. Suppose that |Ω| is odd. Then G is a Frobenius group or G has normal subgroups M and N such that N ≤ M , that N and G/M have odd order and that there exists a power q ≥ 4 of 2 with M/N ∼ = PSL2 (q), Sz(q) or PSU 3 (q). Proof. Let ω ∈ Ω. Then the transitivity of G on Ω yields that |Ω| = |ω G | = |G : Gω | and hence |G| = |Ω| · |Gω |. Suppose that Gω has odd order. Then G has odd order and it is a Frobenius group by Corollary 2.9. This is the first possibility. Next suppose that Gω has even order and let t ∈ Gω be an involution. Then the orbits of t on Ω have length 1 or 2 and therefore, as |Ω| is odd, t must have an odd number of fixed points. Hypothesis 1.1 forces |fixΩ (t)| = 1. Let α ∈ Ω denote the unique fixed point of t and let H := Gα . Then H contains t and hence it has even order. Let g ∈ G\H and suppose that x ∈ H ∩ H g is a 2-element. Then x has at least two fixed points on Ω, namely α and αg . But |Ω| is odd and the orbit lengths of x on Ω are 1 or even, therefore x must have at least one more fixed point on Ω. This forces x = 1 by our hypothesis. It follows that H ∩ H g has odd order and hence H is a strongly 2-embedded subgroup of G. If the second statement in the lemma

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.9 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

9

does not hold, then the main result in [4] yields that G has cyclic or quaternion Sylow 2-subgroups. Then Lemma 2.17 applies and the proof is finished. 2 Next we consider components of groups satisfying Hypothesis 1.1. Lemma 2.19. Let ω ∈ Ω and suppose that E(G) = 1. Then E(G) ∩ Gω = 1. Proof. Assume that E(G) ∩ Gω = 1 and let x ∈ Gω be an element of prime order p. Let E be a component of G. We recall that, by Lemma 2.8, the subgroup CGω (x) has index at most 2 in CG (x). In particular we have for all y ∈ CG (x) that y 2 ∈ Gω . If x does not normalize E, then let E := E1 , E2 , . . . , Ep be components of G that are moved transitively p−1 is an element of E(G) by x. As different components commute, we have for all e ∈ E that y := eex · · · ex that is centralized by x. Therefore y 2 ∈ Gω ∩ E(G) = 1. It follows that e is a 2-element and hence E is a 2-group. But this is impossible. If x normalizes E, then we recall that, for all y ∈ CE (x), we have that y 2 ∈ E ∩ Gω = 1. Hence all elements in CE (x) have order at most 2, so it follows that CE (x) is an elementary abelian 2-group. Then the main theorem in [11] forces E to be solvable, which is again a contradiction. 2 Corollary 2.20. Suppose that G is almost simple, but not simple. Let E := F ∗ (G) and let α ∈ Ω. Then (E, αE ) satisfies Hypothesis 1.1. Proof. Lemma 2.19 yields that E ∩ Gα = 1, so the action on αE is transitive and not regular. Then Hypothesis 1.1 is satisfied. 2 Lemma 2.21. Let ω ∈ Ω. Then Gω does not contain a component of G. Proof. Assume otherwise and let E be a component of G that is contained in H := Gω . Then E(G) = E because otherwise E fixes every point in Ω, which is impossible. Hence let K be another component of G. Let x ∈ E # . Then Lemma 2.8 yields that H ∩ K has index at most 2 in K because K ≤ CG (x). As K is quasi-simple, it follows that K ≤ H. This argument shows that all components of G are contained in H and therefore E(G) ≤ H, which is a contradiction. 2 Lemma 2.22. If E(G) = 1, then E(G) is a single component. Proof. Suppose that E(G) = 1 and let ω ∈ Ω. Then E(G) ∩ Gω = 1 by Lemma 2.19. Let E denote a component of G and suppose first that E ∩ Gω = 1. Then let x ∈ E ∩ Gω be such that x = 1. Assume that there exists another component K of G. Then K ≤ CG (x) and K has no subgroup of index 2, hence Lemma 2.8 implies that K ≤ Gω . This contradicts Lemma 2.21, so this case is finished. Now we suppose that E ∩ Gω = 1. Let L ≤ E(G) be a product of components such that E(G) = E · L and let y ∈ E(G) ∩ Gω be such that y = 1. Let y1 ∈ E and a ∈ L be such that y = y1 a. Then CE (y) has a subgroup of index at most 2 that lies in Gω , but E ∩ Gω = 1 in the current case. Therefore CE (y) has order at most 2 and in particular y has order 2. If y1 = 1, then y1 is an involution and CE (y) = CE (y1 ) has order at least 4, which is a contradiction. Hence y1 = 1 and y = a ∈ L. But then E ≤ CE (y), which is again impossible. 2 We collect the information that we have so far in a single result:

JID:JPAA 10

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.10 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

Theorem 2.23. Let α ∈ Ω. Then one of the following holds: G has a normal subgroup of index at most 2 that is a Frobenius group. |Z(G)| = 2 and G/Z(G) is a Frobenius group. Gα is metacyclic of odd order. G has dihedral or semidihedral Sylow 2-subgroups and the point stabilizers have twice odd order. G has normal subgroups M and N such that N ≤ M , that N and G/M have odd order and that there exists a power q ≥ 4 of 2 with M/N ∼ = PSL2 (q), Sz(q) or PSU 3 (q). (6) F ∗ (G) is simple. (1) (2) (3) (4) (5)

Moreover, if G is simple, then Gα is metacyclic of odd order or G is isomorphic to one of the following groups: PSL2 (q), Sz(q), PSL3 (q) or PSU3 (q) with a suitable prime power q, A7 or M11 . Proof. We go through the cases in Lemma 2.12, starting at (1). Then Lemma 2.16 is applicable. If F (G) = 1, then this leads to (2) or (3), and otherwise E(G) = 1 and Lemma 2.22 gives the statement in (6). Lemma 2.12(2) immediately gives Case (4) and Lemma 2.12(3) leads to Case (1). So it is left to look at Lemma 2.12(4). Then Lemma 2.18 is applicable and yields (1) or (5). The last statement is a combination of the possibilities from Lemma 2.12, Lemma 2.18 and the theorems of Gorenstein and Walter [13] and Alperin, Brauer and Gorenstein [1]. 2 For future reference, we record an additional detail in one of the cases from the last statement in Theorem 2.23. Remark 2.24. Let q be a prime power such that G ∼ = PSL3 (q) and let α ∈ Ω. If Gα has even order, then the Sylow 2-subgroups of G are dihedral or semidihedral. In our analysis of simple groups of Lie type we also need some more information about centralizers of involutions. Lemma 2.25. Let α, β ∈ Ω be distinct and such that U := Gα ∩ Gβ = 1. Suppose that |Gα | is odd and that |Ω| is even. Suppose further that G has no subgroup of index at most 2 that is a Frobenius group. Then there exists an involution x ∈ NG (U ) \ U that interchanges α and β and such that one of the following is true: (1) U is abelian. Moreover if G is simple, then all involutions in G are conjugate. (2) Z(CG (x)) = x and CG (x)/ x is a Frobenius group. Proof. Our hypothesis that Gα is of odd order together with Lemma 2.11 implies that (|Gα |, |Ω|) = 1. Lemma 2.8 and our hypothesis that G does not contain a Frobenius group of index at most 2 yield that |NG (U ) : U | = 2. Thus there exists an involution x ∈ NG (U ) \ U . If CU (x) = 1, then U is abelian (see for example 8.1.10 in [19]). Thus (1) holds in this case and in particular CG (u) ≤ U for all u ∈ U # . If G is simple, then 2.6 of [7] implies that all involutions in G are conjugate. If CU (x) = 1, then we define C := CG (x) and we consider the action of C on αC . If |αC | ≤ 2, then C = x × U and thus the Sylow 2-subgroups of G are cyclic. Then G has a normal 2-complement G1 . But now G1 has odd order and (G1 , αG1 ) satisfies Hypothesis 1.1 which implies that G1 is a Frobenius group; contradicting our hypothesis. Thus we see that |αC | ≥ 3 and that C is not regular on αC . Therefore (C, αC ) satisfies Hypothesis 1.1. Applying Lemma 2.10 together with the fact that x ∈ Z(C) yields that x = Z(C). Thus Lemma 2.14 implies that C/Z(C) is a Frobenius group as stated in (2). 2

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.11 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

11

In the next two sections we describe all pairs (G, Ω) satisfying Hypothesis 1.1 where G is simple, using the Classification of Finite Simple Groups. As some of the arguments based on Lemmas 2.8 and 2.11 become repetitive, we decided to adapt Aschbacher’s notation from Section 9 of [2] at least for some connections between prime divisors of |G|. This notation will be used throughout Sections 3 and 4. Definition 2.26. Suppose that p, q ∈ π(G) are prime numbers and let H ≤ G be a point stabilizer in G. • We write p  q if and only if one of the following holds: – q is odd and there exists a nontrivial p-subgroup X ≤ G such that q ∈ π(NG (X)). – q = 2 and there exists a nontrivial p-subgroup X ≤ G such that 4 divides |NG (X)|. • We write → for the transitive extension of . • We write 2  p if and only if p divides the order of every involution centralizer in G. • Suppose that π ⊆ π(G) is a set of primes. Then we say that π is connected if and only if the following hold: – If p, q ∈ π and p is odd, then p → q. – If 2 ∈ π and π = {2}, then there exists some odd r ∈ π such that 2  r. Lemma 2.27. Suppose that Hypothesis 1.1 holds and that H ≤ G is a point stabilizer. Suppose further that q ∈ π(G) and p ∈ π(H). (1) If p is odd and p  q, then q ∈ π(H). (2) If p = 2 and 2  q, then q ∈ π(H). (3) If π ⊆ π(G) is connected and p ∈ π, then π ⊆ π(H). Proof. In (1) Lemma 2.11(a) gives that H contains a Sylow p-subgroup of G. Then by Sylow’s Theorem there exists a nontrivial p-subgroup X of H such that q divides |NG (X)| and therefore Lemma 2.8 yields that q ∈ π(H). For (2) we just apply Lemma 2.8 to the centralizer of an involution from H. In (3) let q ∈ π be arbitrary. If p is odd, then p → q because π is connected, and hence (1) yields that q ∈ π(H). Next suppose that p = 2. If π = {2}, then we are done. Otherwise, by connectedness, let r ∈ π be odd and such that 2  r. Then r ∈ π(H) by (2) and hence the first part of the proof yields that all odd primes in π are also contained in π(H). 2 3. Alternating groups and Lie type groups 3.1. Alternating groups Lemma 3.1. The group G := PSL2 (5) provides examples satisfying Hypothesis 1.1. The size of the set Ω that G is acting on is 5, 6, 10, 12, 20 or 30 and there is an example for all these numbers. The point stabilizers have structure A4 , D10 , D6 , Z5 , Z3 or Z2 , respectively. Proof. As G  A5 , we see that the natural action of G on five points provides an example. In order to obtain an example for six points, we let Ω := {1, 2, 3, 4, 5, 6} and we keep viewing G as an alternating group with the corresponding notation. The double transpositions of G as well as the 5-cycles have one or two fixed points in their natural action. Now we define an action of the 3-cycles: Let g ∈ G be a 3-cycle and let a, b, c ∈ {1, 2, 3, 4, 5} be such that g = (a, b, c). Let x, y be the remaining two points in Ω and choose notation such that x < y. Then the

JID:JPAA 12

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.12 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

action of g on Ω is defined to be the action of the element (a, b, c)(x, y, 6) ∈ S6 . In particular all 3-cycles of G act fixed point freely on G and therefore (G, Ω) satisfies Hypothesis 1.1. Next we consider the action of G on the set of cosets of a subgroup U of order 6, for example U :=

(123), (12)(45). This is an action on a set with 10 points where every 5-cycle of G is fixed point free and every 3-cycle of G has exactly one fixed point. The involution (12)(45) has two fixed points in this action, namely U itself and the coset U (14)(25). As all involutions in G are conjugate, they all have exactly two fixed points. Hence G (with this action) satisfies Hypothesis 1.1. For the next example we let G act on the set of cosets of H := (12345) in G. This is an action on a set with 12 elements where every 5-cycle has two fixed points and all 3-cycles are fixed point free. In this case every involution is also fixed point free. The natural action of G on the set of cosets of a subgroup of order 2 or 3 leads to the last two examples, respectively. Now let the set Ω be such that (G, Ω) satisfies Hypothesis 1.1. Let n := |Ω|. As G is transitive, but not regular, and since G does not have subgroups of index 2 or 3, we have that n divides 60 and n ≥ 4. Moreover G does not have a subgroup of order 15, so this leads to the possibilities n ∈ {5, 6, 10, 12, 20, 30}. This gives exactly the numbers listed. We showed in the previous paragraph that all numbers lead to examples and thus the proof is finished. 2 Lemma 3.2. (S5 , Ω) satisfies Hypothesis 1.1 if and only if |Ω| ∈ {6, 10, 20, 30}. The point stabilizers are isomorphic to 5 : 4, D12 , Z6 or Z4 , respectively. Proof. Let n := |Ω| and let ω ∈ Ω. We first observe that n = 5 because 2-cycles have three fixed points in their natural action. If 2 is a divisor of |Gω |, then Lemma 2.8 implies that 4 or 6 is a divisor of |Gω |. If 6 is a divisor of |Gω |, but 4 is not, then Gω is cyclic of order 6 which yields an example for n = 20. If 8 divides |Gω |, then Gω contains a Klein fours group whose normalizer is S4 , which implies that Gω = S4 and n = 5. But this is impossible. Thus the Sylow 2-subgroup of Gω is either cyclic of order 4 or elementary abelian of order 4. First suppose that Gω is cyclic of order 4. As the proper overgroups of a 4-cycle are 5 : 4 and S4 , this allows for the possibility that Gω is cyclic of order 4 which implies that n = 30 or that Gω is a Frobenius group of order 20 and n = 6. Next suppose that Gω is elementary abelian of order 4. Then one involution is a transposition, which in turn implies by Lemma 2.8 that 3 is a divisor of |Gω |. This forces Gω ∼ = D12 or S4 . The first possibility yields an example, but the second possibility does not because otherwise n = 5. Next note that 3  2. Now if 5 is a divisor of |Gω |, then Gω ∼ = D10 or 5 : 4. The former case is ruled out by Lemma 2.8 whereas the second case leads to an example. 2 Lemma 3.3. If (A6 , Ω) satisfies Hypothesis 1.1, then |Ω| ∈ {10, 72, 90}. Proof. If 3 is a divisor of |Gω |, then Gω contains a Sylow 3-subgroup of G by Lemma 2.11(a). As 3  2, it follows that Gω contains an involution, which in turn implies that 36 divides |Gω |. Inspection of the maximal subgroups of G shows that 36 = |Gω |, which leads to the example |Ω| = 10. If Gω has even order, then it contains an involution. Since all involutions in G are conjugate, Lemma 2.8 implies that 4 divides |Gω |. Inspection of the maximal subgroups of G shows that the only subgroups of G whose order is not divisible by 3 but which contain a subgroup of order 4 are the Klein fours subgroups, the cyclic groups of order 4 and the dihedral subgroups of order 8. Lemma 2.8, applied to a Klein fours subgroup, only leaves the case of a cyclic subgroup of order 4. In fact this leads to an example where |Ω| = 90. The last possibility is that 5 = |Gω |. This is another example because the normalizer of a subgroup of order 5 in G is D10 . 2

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.13 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

13

Lemma 3.4. Suppose that G is almost simple, that G = F ∗ (G) = A6 and that (G, Ω) satisfies Hypothesis 1.1. Then one of the following is true: 1. |Ω| = 10 and G is either PGL2 (9)  S6 or G is M10 , the one point stabilizer of M11 in its action on 11 points. 2. |Ω| = 72 and G = PGL2 (9). 3. |Ω| = 90 and G is either PGL2 (9) or M10 . Proof. As in Lemma 3.3 we conclude that, if 3 divides |Gω |, then 36 divides |Gω |. Lemma 2.8, applied to an involution of Gω , implies that in fact 72 divides |Gω | if |G : F ∗ (G)| = 2, and 144 divides |Gω | if |G : F ∗ (G)| = 4. Thus |Ω| = 10. In the action on 10 points, the involutions in S6 \ A6 have 4 fixed points, which implies that |G : F ∗ (G)| = 2. Inspection shows that the two possibilities listed in our conclusion do indeed occur. Thus we now consider the case where Gω is a 3 -group. We note that 5  2 and that only the 2-central involution, which is contained in A6 , or the involution in PGL2 (9) have centralizer order coprime to 3. Thus applying Lemma 2.8 to an involution in Gω implies that either 20 · |G : F ∗ (G)| is a divisor of |Gω |, or that Gω is cyclic of order 10 and G = PGL2 (9). The latter case is an example for n = 72, whereas the former case does not occur because G does not possess proper subgroups of order divisible by 20 · |G : F ∗ (G)|. To conclude we consider the case that Gω is a 2-group, which by Lemma 2.8 implies that the normalizer of every subgroup of Gω is a 2-group. Thus only involutions from A6 can be elements of Gω . Also Gω cannot contain a Klein fours group all of whose involutions are from A6 , because otherwise Lemma 2.8 forces an element of order 3 into Gω . The upshot of this is that Gω must be cyclic or quaternion. Lemma 2.8 implies that the quaternion case is also impossible (the normalizer contains an element of order 3). Thus Gω is cyclic of order 8. (Note that order 4 is impossible by Lemma 2.8.) This rules out the possibility that G = S6 . Also Lemma 2.8 rules out the possibility that G = Aut(A6 ). This shows that |Ω| = 90, which leads to the listed examples. 2 Lemma 3.5. There is no set Ω such that (A7 , Ω) satisfies Hypothesis 1.1. Proof. Assume otherwise. Let G := A7 and let Ω be a finite set such that (G, Ω) satisfies Hypothesis 1.1. In particular G acts transitively and nonregularly on Ω. Let α ∈ Ω. Then H := Gα = 1. We begin by showing that H is a {2, 3}-group. Throughout, we use facts about the subgroup structure of A7 that can, for example, be found in the ATLAS. Assume that H is not a {2, 3}-group and first suppose that X ≤ H is a subgroup of order 7. As 7 → 2, Lemma 2.8 yields that H contains a subgroup isomorphic to A4 . However, G does not have a proper subgroup that contains a 7-cycle and a subgroup isomorphic to A4 . Next suppose that Q ≤ H is a subgroup of order 5. We know that 5  3 and 3  2, so H contains a subgroup isomorphic to A4 × 3 by Lemma 2.8. Hence a similar argument as before applies: G does not have a proper subgroup containing a 5-cycle as well as a subgroup isomorphic to A4 × 3. Now we know that H is a {2, 3}-group. Lemmas 2.8 and 2.11(a) force 2  3 and 3  2, so both primes occur in |H| and it follows that H contains a direct product of A4 with a subgroup of order 3. In particular H is not a Frobenius group. Lemma 2.15 then gives that there exists β ∈ Ω such that β = α and H = Gβ . There are two cases remaining: If |H| = 36, then Ω has 70 points and thus every element of order 3 has one fixed point whereas every involution has two fixed points. This is impossible. If |H| = 72, then Ω has 35 points and every element of order 3 has two fixed points whereas every involution has exactly one fixed point. Again this is impossible. 2

JID:JPAA 14

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.14 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

Now we consider Hypothesis 1.1 in the situation where G ∼ = An or Sn with n ≥ 8. Lemma 3.6. Suppose that n ≥ 8, that G ∼ = Sn or An and that Ω is a set such that (G, Ω) satisfies Hypothesis 1.1. Then the order of a point stabilizer in G is not divisible by 3. Proof. Assume otherwise and let ω ∈ Ω and x ∈ Gω be such that o(x) = 3. Then Lemma 2.11(a) implies that Gω contains a full Sylow 3-subgroup P of G. In particular, we find a pair x1 and x2 of commuting 3-cycles in Gω . Without loss, as G contains a subgroup isomorphic to A8 and is hence sixfold transitive, we may suppose that x1 = (1, 2, 3) and that x2 = (4, 5, 6). Lemma 2.8 implies that Gω contains CG (x1 )∞ , CG (x2 )∞ , where the superscript ∞ denotes the last term of the derived series. Note that CG (x1 )∞ ∼ = An−3 , which is a perfect group, as n ≥ 8. Also CG (x2 )∞ ∼ = An . This = An−3 and thus Gω ≥ CG (x1 )∞ , CG (x2 )∞  ∼ contradicts the fact that G acts faithfully on Ω. 2 Lemma 3.7. If n ∈ {8, 9} and G ∼ = An or Sn , then there is no set Ω such that (G, Ω) satisfies Hypothesis 1.1. Proof. Assume otherwise and let Ω be such a set. Let ω ∈ Ω. By Lemma 3.6 there is no element of order 3 in Gω . As 5  3 and 2  3, we see that |Gω | is coprime to 30 which implies that Gω is generated by a 7-cycle. But also 7  3, which is a contradiction. 2 Theorem 3.8. Suppose that n ≥ 10 and that G ∼ = An or Sn . Then there is no set Ω such that (G, Ω) satisfies Hypothesis 1.1. Proof. Assume otherwise, let Ω denote such a set and let ω ∈ Ω. Let g ∈ Gω be a p-element. Then g is a product of k cycles each of length p and g has exactly n − k · p fixed points. The centralizer CG (g) contains Zpk : Sk × An−pk if p = 2 and (Z2k : Sk × Sn−2k ) ∩ An otherwise. So if n − p · k ≥ 3, then Lemma 2.8 implies that CG (g) ∩ Gω contains a 3-cycle. This contradicts Lemma 3.6. Therefore n − p · k ≤ 2. Assume that p = 2. Then k > 2 and hence every index two subgroup of (Z2k : Sk ∩ An ) ≤ (CG (g) ∩ An ) contains some double 2-cycle. Thus Lemma 2.8 implies that Gω contains some double 2-cycle d. As n ≥ 10, we see that CG (d) contains a subgroup isomorphic to An−4 , which is a perfect group of order divisible by 3. This implies that |Gω | is divisible by 3, contradicting Lemma 3.6. Thus we see that 2 is also not a divisor of |Gω |. Hence p > 3 (using Lemma 3.6). Now Lemma 2.11(a) implies that Gω ∩ An contains a full Sylow p-subgroup P of G. Thus Gω ∩ An contains a p-cycle, say h. If n − p > 2, then 3 divides |CG (h)| which, by Lemma 2.8, implies that |Gω | is divisible by 3; again a contradiction. Therefore n − p ≤ 2. This property holds for all prime divisors p of |Gω |. As n ≥ 10 and p is prime, this forces p ≥ 11. In particular |NG ( g) : g| ≥ p−1 2 ≥ 5 and it follows that |Gω ∩ NG ( g)| is divisible by a prime r such that 2r ≤ p − 1 ≤ n. This implies r ≤ n − r. We know that r = 2 and r = 3, so 5 ≤ r ≤ n − r. But the previous paragraph shows that r satisfies n − r ≤ 2. This is impossible. 2 Corollary 3.9. Suppose that n ∈ N is such that n ≥ 5 and that Ω is a set such that (An , Ω) satisfies Hypothesis 1.1. Then n ∈ {5, 6} and the action is as described in Lemmas 3.1 and 3.11. Proof. It follows from Theorems 3.7 and 3.8 that n ∈ {5, 6}. As A5 ∼ = PSL2 (5) and A6 ∼ = PSL2 (9), the only possible actions are explained in Lemmas 3.1 and 3.11. 2

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.15 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

15

Lemma 3.10. Suppose that n ∈ N is such that n ≥ 5. Then there is no set Ω such that (2An , Ω) satisfies Hypothesis 1.1. Proof. Assume otherwise, let ω ∈ Ω and let G := 2An . Moreover we let G := G/Z(G) and we let Ω denote the set of Z(G)-orbits on Ω. Then (G, Ω) satisfies Hypothesis 1.1 by Lemma 2.14 and so we only need to consider the cases n = 5 and n = 6, by Corollary 3.9. This means that G ∼ = SL2 (5) or 2A6 . Let S ∈ Syl2 (G). Then S is a quaternion group and S  Gω because Z(G)  Gω . Moreover Lemma 2.14 forces Gω , and hence Gω , to have even order, and Case (3) of Lemma 2.12 does not hold. This contradicts Lemma 2.12. 2 3.2. Lie type groups Lemma 3.11. Suppose that r is prime and that m ∈ N. Let q := rm ≥ 7. If Ω is a set such that (PSL2 (q), Ω) satisfies Hypothesis 1.1, then one of the following holds: (a) |Ω| = q + 1 and G acts on Ω as on the set of cosets of the normalizer of a Sylow r-subgroup of G. (b) |Ω| = q(q − 1) and G acts on Ω as on the set of cosets of a cyclic subgroup of order q+1 2 (if r is odd) or q + 1 (if r = 2). (c) |Ω| = q(q + 1) and G acts on Ω as on the set of cosets of a cyclic subgroup of order q−1 2 (if r is odd) or q − 1 (if r = 2). (d) q = r = 7 and the point stabilizers are isomorphic to A4 . Proof. Let G := PSL2 (q) and suppose that Ω is a set such that (G, Ω) satisfies Hypothesis 1.1. For the subgroup structure of G we refer to Theorem 6.5.1 in [15]. Let α ∈ Ω. As G acts transitively and nonregularly on Ω, we have that H := Gα = 1. Case 1: q ≡ 1 modulo 4. If q = 9, then Lemma 3.3 yields our conclusion. So we may assume that q ≥ 13. Let p ∈ π(H). If p = r, then Lemma 2.11(a) yields that H contains a Sylow r-subgroup of G and hence a subgroup of index at most two of its normalizer (by Lemma 2.8). This means that H contains a subgroup of order q−1 4 . As q ≥ 7 and as q − 1 is divisible by 4, we know that q − 1 ≥ 8. Hence Lemma 2.8 implies that H contains a subgroup of order q−1 2 of G. Now the subgroup structure of G gives that H is the normalizer of a Sylow r-subgroup of G whence |Ω| = q + 1. Conversely, if we let Ω denote the set of cosets of the normalizer of a Sylow r-subgroup in G, then Ω has q + 1 elements and (G, Ω) satisfies Hypothesis 1.1 with the natural action of G on Ω by right multiplication. This yields (a). Next suppose that p divides q − 1. Then it follows with Lemma 2.8 that H contains a subgroup of order q−1 2 of G. If the previous case does not occur, then the subgroup structure of G yields that |H| has order q−1 2 or q − 1. In the first case G acts on q(q + 1) points. This is an example with the natural action of G on the set of cosets of a cyclic subgroup of G of order q−1 2 in G as described in (b). In the second case H contains a Klein fours group A because q − 1 is divisible by 4. As NG (A) contains a subgroup isomorphic to A4 , Lemma 2.8 implies that this subgroup lies in H. But it does not. If p divides q + 1, then we only consider the case where H has odd order and in particular p is odd (otherwise the previous paragraph applies). Then H contains a subgroup of G of order q+1 2 by Lemma 2.8. Now |H| = q+1 and G acts on q(q − 1) points. As an example for this, we see the natural action of G on 2 q+1 the set of cosets of a cyclic group of order 2 in G. This yields (c). Case 2: q ≡ 3 modulo 4.

JID:JPAA 16

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.16 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

Let p ∈ π(H). We argue as in the first case, with minimal differences. If p = r, then H contains a Sylow r-subgroup of G and hence a subgroup of index at most two of its is odd in this case, it follows that H is the normalizer of a Sylow normalizer (by Lemma 2.8). As q−1 2 r-subgroup of G and that (a) holds. If p divides q − 1, then H contains a subgroup of order q−1 2 of G. If the previous case does not occur, then the subgroup structure of G yields that |H| has order q−1 2 or q − 1, or q = 7 or 11 and |H| could also be 12 or 60. In the first case G acts on q(q + 1) points and (b) holds. In the second case H contains an involution and therefore an index two subgroup of an involution centralizer, i.e. a subgroup of order q+1 2 . This is impossible if q > 11. If q = 11, then H  A5 is possible, but then an involution fixes three points in the action of G on the cosets of H, ruling out this case. If q = 7 and H  A4 , then the action of G on the cosets of H gives an example. If p divides q + 1, then H contains a subgroup of G of order q+1 2 and hence an involution, because this time q + 1 is divisible by 4. Then it follows that |H| = q+1 and that G acts on q(q − 1) points as in (c). 2 Case 3: r = 2. If H has even order, then H contains an involution and hence a subgroup of a Sylow 2-subgroup of G of index at most two, by Lemma 2.8. As q ≥ 7, this implies that H contains a Klein fours group. Moreover, in the centralizer of an involution, we see an element of order dividing q − 1, hence an element of odd order (by Lemma 2.8). Then Lemma 2.8 and the fact that q − 1 is odd imply that H contains a subgroup of order q − 1 and then a Sylow 2-subgroup of G. This means that |H| = q(q − 1) and that Ω = q + 1 as stated in (a). If H has odd order, then it contains a subgroup of order q + 1 or q − 1 and hence coincides with it. This leads to the cases (b) and (c). This last case concludes the proof. 2 Lemma 3.12. Suppose that m ∈ N is odd and let q := 2m ≥ 8. If Ω is a set such that (Sz(q), Ω) satisfies Hypothesis 1.1, then one of the following holds: (a) |Ω| = q 2 + 1 and G acts 2-transitively in its natural action. (b) |Ω| = q 2 (q 2 + 1) and G acts on Ω as on the set of cosets of a subgroup of order q − 1. Proof. Let G := Sz(q) and suppose that Ω is a set such that (G, Ω) satisfies Hypothesis 1.1. For the subgroup structure of G we refer to Theorem 6.5.4 in [15]. Let α ∈ Ω. As G acts transitively and nonregularly on Ω, we have that H := Gα = 1. If 2 divides |Gω |, then Lemma 2.8 implies that an index 2 subgroup of some Sylow 2-subgroup S of G is contained in Gω . As every index 2-subgroup of S contains Z(S) we see that Z(S) ≤ Gω . Thus we see that an index 2 subgroup of NG (Z(S)) = S : (q − 1) is contained in Gω . As NG (Z(S)) is maximal in G we may assume that Gω = NG (Z(S)). Indeed G in its action on the cosets of Gω is an example of a Zassenhaus group and this is (a). Thus we may now assume that H has odd order. Hence for all prime divisors p of |H|, Lemma 2.11(a) yields that H contains a Sylow p-subgroup. Let p ∈ π(H). If p divides q − 1, then the subgroup structure of G yields that H is cyclic of order q − 1. Then |Ω| = q 2 (q 2 + 1) and G acts on Ω as on the set of cosets of a a+1 a+1 subgroup of order q − 1. This is (b). If p divides q + 2 2 + 1 or q − 2 2 + 1, then H contains a subgroup a+1 a+1 of order q + 2 2 + 1 or q − 2 2 + 1, respectively, and hence it has even order. But this is impossible. Therefore the proof is complete. 2 Lemma 3.13. Let G := PSL3 (4) and let H be a subgroup of G of order 5. Let Ω := G/H. Then (G, Ω) satisfies Hypothesis 1.1 with the natural action of G on Ω, and this is the only example for G satisfying this hypothesis.

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.17 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

17

Proof. Let Δ be a finite set such that (G, Δ) satisfies Hypothesis 1.1 and let δ ∈ Δ and H := Gδ . All information about the subgroup structure of G that we use comes from the ATLAS. The Sylow 2-subgroups of G are not dihedral or semi-dihedral and thus Remark 2.24 implies that H is a  / π(H), and we are left with 2 -group. As 3  2, we deduce that H is a 3 -group. Moreover 7  3 whence 7 ∈ the case that |H| = 5. Then |Δ| = 26 · 32 · 7 = |Ω| and G acts on Δ in the same way that it acts on Ω by right multiplication. 2 Lemma 3.14. Suppose that q ≥ 27 is a prime power and that G =2 G2 (q). Then there is no set Ω such that (G, Ω) satisfies Hypothesis 1.1. Proof. Let ω ∈ Ω. Firstly we assert that 3 does not divide |Gω |. Assume otherwise. Then Lemma 2.11(a) implies that a full Sylow 3-subgroup P of G lies in Gω . Applying Lemma 2.8 to P shows that Gω contains an element t of order (q − 1)/2. Now note that |NG ( t)| = 2(q − 1), which by Lemma 2.8 implies that Gω contains an involution s. But this is false by Theorem 2.23. If x ∈ Gω is of odd order coprime to 3, then CG (x) is a torus, see Theorem B in [18]. If |CG (x)| = (q − 1), then the argument from the first paragraph of this proof implies that Gω contains an involution, which is √ impossible. If |CG (x)| = (q ± 3q + 1), then |NG (CG (x)) : CG (x)| = 6 and thus Lemma 2.8 implies that 3 is a divisor of |Gω |. This is again a contradiction. 2 Lemma 3.15. If G = PSU3 (q), with pa = q = 2, then there is no set Ω such that (G, Ω) satisfies Hypothesis 1.1. Proof. Let ω ∈ Ω, let x ∈ G# ω and assume that q ≥ 7. Firstly we assert that the order of x is not a divisor of q + 1. To see this we assume otherwise and we work in the group SU3 (q) and observe that the action of any element y, that projects naturally onto x, is diagonalizable with orthonormal eigenbasis {e1 , e2 , e3 }. Hence CG (x) contains a torus of order (q + 1)2 /(3, q + 1). This, together with Lemma 2.8, implies that Gω contains elements x1 and x2 whose lifts y1 , y2 ∈ SU3 (q) are such that CSU3 (q) (yi ) = SU( ei , ei+1 ). Evidently SU3 (q) = CSU3 (q) (yi ) , CSU3 (q) (yi )  and thus G = CG (x1 ) , CG (x2 )  ≤ Gω . This is a contradiction. Next if o(x) is a divisor of q − 1, then (q 2 − 1)/2 divides |CG (x)|, and hence Lemma 2.8 implies that q + 1/2 is a divisor of Gω . We excluded this case in the previous paragraph. If o(x) is a divisor of q 2 − q + 1 and is coprime to q ± 1, then |CG (x)| = (q 2 − q + 1)/(3, q + 1) is odd, and hence Lemma 2.8 implies that CG (x) ≤ Gω . As |NG (CG (x)) : CG (x)| = 3, Lemma 2.8 implies that NG (CG (x)) ≤ Gω . Now 3 either divides (q 2 − 1) or p = 3. The former case implies Gω = G by the above, whereas the latter case implies the Gω contains a Sylow 3-subgroup of P of G, by Lemma 2.11(a). As |NG (P ) : P | = q 2 − 1/(3, q + 1), Lemma 2.8 then yields that Gω contains elements of order dividing q 2 − 1. But we excluded this case. Let G = PSU3 (5) and let ω ∈ Ω. Using the list of maximal subgroups in [8] we observe that no proper subgroup of G contains both a Sylow 3 and a Sylow 5-subgroup of G. We show first that Gω is a 3 -group. Suppose otherwise, then let x ∈ Gω be an element of order 3. From the ATLAS we infer that |CG (x)| = 36 which, by Lemma 2.8, implies that Gω contains an involution t. But GU2 (5) ≤ CG (t) which by Lemma 2.8 implies that Gω contains both a Sylow 3 and a Sylow 5-subgroup of G; a contradiction. As 5  2 and 7  3, there are no more possible prime divisors left for |Gω |. Let G = PSU3 (4) and let ω ∈ Ω. If Gω contains an element of order 5, then by Lemma 2.11(a) it contains a Sylow 5-subgroup of G. Thus by Lemma 2.8 it contains a subgroup A := Z5 × A5 . Now pick an involution s ∈ A5 and observe that its centralizer also lies in Gω . But now G = Gω = A, CG (s), which is impossible. So Gω must be a {2, 5} -group, which implies that Gω is cyclic of order 3 or 13. Both cases are impossible by Lemma 2.8.

JID:JPAA 18

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.18 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

Let G = PSU3 (3) and let ω ∈ Ω. Lemma 2.8 implies that if (|Gω |, 6) = 1, then 6 divides |Gω |. So Lemma 2.8 implies that 24 × 33 divides |Gω |. But G does not contain subgroups of index less than 14 which rules out that (|Gω |, 6) = 1. The only remaining possibility is that Gω is cyclic of order 7. But 7  3, so this is impossible, and the proof is complete. 2 Lemma 3.16. Let G = PSL3 (q) with q = rm odd. If G acts transitively on Ω and |Gω | is even, then (G, Ω) does not satisfy Hypothesis 1.1. Proof. G contains a unique class of involutions whose centralizer is GL2 (q). So if |Gω | is even, then Lemma 2.8 implies that SL2 (q) ≤ Gω and in particular Gω contains r-central elements. Applying Lemma 2.8 to an r-element of SL2 (q) shows that Gα contains a full Sylow r-subgroup of G as well as an opposite. Thus G = GΩ and our claim is established. 2 Note that the simple groups occurring at the end of Theorem 2.23 are of Lie rank 1 or PSL3 (q). The Sylow 2-subgroups of PSL3 (2n ) are special of order 2n+2n . They are of dihedral or semidihedral if and only if n = 1. But if n = 1, then PSL3 (2) ∼ = PSL2 (7). Thus at this point we have considered all the simple groups occurring at the end of Theorem 2.23 and all the groups of Lie rank 1. Thus we may now work under the following hypothesis: Hypothesis 3.17. G is simple of Lie type and of Lie rank at least 2 defined over a finite field GFq of characteristic p such that the following hold: 1. Ω is a set such that (G, Ω) satisfies Hypothesis 1.1 and 2. |Gα | is of odd order. We note that Hypothesis 3.17 implies that |Ω| is even because G has even order. Moreover the conclusion of Lemma 2.25 holds, that is there exist α, β ∈ Ω such that α = β and 1 = U := Gα ∩ Gβ and there is an involution x ∈ NG (U ) such that • either x inverts every nontrivial element of U and all involutions in G are conjugate, or • CG (x)/ x is a Frobenius group. Lemma 3.18. If (G, Ω) satisfies Hypothesis 3.17 and q is even, then G  PSL3 (q) with q ≤ 4. Before starting the proof of Lemma 3.18 we recall that in a group of Lie type an element is semisimple if its order is coprime to q and that a semisimple element is regular if the order of its centralizer is coprime to q. Proof of Lemma 3.18. Since q is even, all elements in Gα are semisimple. If an element g ∈ G is semisimple but not regular, then CG (g) contains a subgroup K isomorphic to PSL2 (q). If q = 2, then |K|2 > 2 and so Lemma 2.8 implies that all nontrivial elements of Gα are regular semisimple. If q = 2, then |K| = 6 which implies that if Gα contains a nonregular semisimple element, then 3 divides |Gα |. In turn this implies that Gα contains a Sylow 3-subgroup of G. If G  PSL3 (2), then Gα contains an elementary abelian subgroup of order 9 whose normalizer order is divisible by 4 and so Lemma 2.8 forces an involution into Gα , contrary to Hypothesis 3.17. Thus we have established that all nontrivial elements of Gα are regular semisimple. So for every subgroup U of Gα it is true that CG (U ) is of odd order. Thus if U is as in Hypothesis 3.17, then U is abelian and all involutions in G are conjugate. As G is of Lie type in characteristic 2, it follows from [3] that G  PSL3 (q).

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.19 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

19

The regular semisimple elements in PSL3 (q) are of order q 2 + q + 1/(q − 1, 3) and of order q 2 − 1/(q − 1, 3). If Gα contains an element of order q 2 − 1/(q − 1, 3), then Gα contains a nontrivial element of order (q − 1)/(q − 1, 3) if q > 4. But then Lemma 2.8 implies that Gα contains a split torus of G (a subgroup of order (q − 1)2 /(q − 1, 3)) and thus elements which are not regular and semisimple, a contradiction. If Gα contains an element of order (q 2 + q + 1)/(q − 1, 3), then again Lemma 2.8 implies that Gα contains an element of order 3 whose centralizer is of order (q 2 − 1)/(q − 1, 3) or (q − 1)2 /(q − 1, 3), contrary to our previous observation. Thus our claim follows. 2 Lemma 3.19. There do not exist pairs (G, Ω) which satisfy Hypothesis 3.17 with q odd. Proof. If p divides |Gα |, then Lemma 2.11 implies that Gα contains a Sylow p-subgroup of G because p is odd. Thus Gα contains a p-element whose centralizer contains a subgroup isomorphic to SL2 (q) which by Lemma 2.8 implies that Gα contains an involution; contradicting Hypothesis 3.17. Thus we may assume that every nonidentity element of Gα is semisimple. Moreover, arguing as in Lemma 3.18 we see that every nontrivial element in Gα is regular and semisimple. As noted right after Hypothesis 3.17 either all involutions in G are conjugate or some involution centralizer is a Frobenius group modulo its center. Table 4.5.1 in [15] shows that the groups satisfying Hypothesis 3.17 all of whose involutions are conjugate are PSL3 (q), G2 (q) and 3 D4 (q). Those that might have an involution centralizer which modulo its center is a Frobenius group are PSL3 (3), PSL4 (3), PSU4 (3), PSp4 (3), G2 (3), Ω7 (3) and Ω8± (3). The potential centralizers are normalizers of central products C of subgroups isomorphic to SL2 (3). The number of “SL2 (3)-factors” in C is at least two, except in case G  PSL3 (3), which implies that C possesses 3-elements which do not act fixed point freely on O2 (C)/Z(C). This leaves PSL3 (3). If G 3 D4 (q), or G2 (q) and q > 3, then we see in Table 5.2 in [20] that the normalizers of the maximal tori have orders divisible by 6 or 4 which implies by Lemma 2.8 that 2 or 3 divides |Gα |. Hypothesis 3.17 rules out the former. The second possibility forces a Sylow 3-subgroup into Gα and hence, as in the first paragraph of this proof, an involution into Gα which again is contrary to Hypothesis 3.17. For G2 (3) we see that 13  3 and 7  3 which forces a Sylow 3-subgroup into Gα , contradicting the fact that Gα does not contain unipotent elements. Finally we consider the groups PSL3 (q). As in Lemma 3.18 we note that the regular semisimple elements lie in tori of orders q 2 − 1/(q − 1, 3) and q 2 + q + 1/(q − 1, 3). Now Lemma 2.8 and the fact that q is odd imply that 3 divides |Gα | which in turn forces an involution into Gα , a contradiction. 2 Theorem 3.20. Let G be a finite simple group of Lie type and let Ω be a set such that (G, Ω) satisfies Hypothesis 1.1. Then one of the following is true: (1) G = PSL2 (q) where q = rm ≥ 7 and one of the following holds: (a) |Ω| = q + 1 and G acts on Ω as on the set of cosets of the normalizer of a Sylow r-subgroup of G. (b) |Ω| = q(q − 1) and G acts on Ω as on the set of cosets of a cyclic subgroup of order q+1 2 (if r is odd) or q + 1 (if r = 2). (c) |Ω| = q(q + 1) and G acts on Ω as on the set of cosets of a cyclic subgroup of order q−1 2 (if r is odd) or q − 1 (if r = 2). (d) |Ω| = 14 and G = PSL2 (7) acts on Ω as on the set of cosets of A4 . (2) G = PSL2 (4) ∼ = PSL2 (5) ∼ = A5 , the size of the set Ω that G is acting on is 5, 6, 10, 12, 20 or 30 and there is an example for all these numbers: (a) |Ω| = 5 = 4 + 1 and G acts on Ω as on the set of cosets of the normalizer of a Sylow 2-subgroup of G.

JID:JPAA 20

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.20 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

(b) |Ω| = 6 = 5 + 1 and G acts on Ω as on the set of cosets of the normalizer of a Sylow 5-subgroup of G. (c) |Ω| = 10 and Gω is the normalizer of a Sylow 3-subgroup of G. (d) |Ω| = 12 = 4 × 3 and Gω is cyclic of order 5. (e) |Ω| = 20 = 4 × 5 = 5 × 4 and Gω is cyclic of order 3. (f) |Ω| = 30 = 5 × 6 and Gω is cyclic of order 2. (3) G = Sz(q) and one of the following holds: (a) |Ω| = q 2 + 1 and G acts 2-transitively in its natural action. (b) |Ω| = q 2 (q 2 + 1) and G acts on Ω as on the set of cosets of a subgroup of order q − 1. (4) G = PSL3 (4) and Gω is cyclic of order 5. Proof. The simple groups of Lie rank 1 are PSL2 (q), PSU3 (q), 2 G2 (q) and Sz(q). Lemmas 3.15 and 3.14 eliminate the families PSU3 (q) and 2 G2 (q), whereas PSL2 (q) and Sz(q) lead to examples. These are described in Lemma 3.11 and Lemma 3.12. For the groups PSL3 (q) with q-odd we showed in Lemma 3.16 that |Gω | is odd and thus we may work under Hypothesis 3.17. If q is even, then the conclusion of Lemma 3.18 is that G  PSL3 (2)  PSL2 (7) or G  PSL3 (4). The first case was considered in Lemma 3.11 and gives rise to the examples in (1) of our conclusion whereas the second case was considered in Lemma 3.13 and gives rise to the example in (4). If q is odd, then Lemma 3.19 shows that there are no examples. 2 Lemma 3.21. Suppose that G is quasisimple of Lie type. Then there is no set Ω such that (G, Ω) satisfies Hypothesis 1.1. Proof. Assume otherwise and let G := G/Z(G). Note that |Z(G)| = 2 by Lemma 2.10 and let Ω denote the set of Z(G)-orbits on Ω. Then (G, Ω) satisfies Hypothesis 1.1 by Lemma 2.14 and hence we only need to consider the cases from Theorem 3.20. By the same lemma we only need to look at cases where point stabilizers in G have even order. Together with Lemma 2.12 we conclude that G has dihedral or semidihedral Sylow 2-subgroups. We go through the cases from Theorem 3.20. If there is a prime power q such that G ∼ = PSL2 (q), then G  SL2 (q) has quaternion Sylow 2-subgroups, which is impossible. The case where G  A5 has already been treated in Lemma 3.10. If G is a Suzuki group, then again G does not have dihedral or semidihedral Sylow 2-subgroups. Finally G is not PSL3 (4) because, in the only possible action there, the point stabilizers have order 5. 2 We close this section with two results on almost simple groups that will be used in the proof of Theorem 1.3. In the first lemma we look at the groups PGL2 (q), and in the second lemma we only consider odd prime powers q and consider groups that Gorenstein (in [12]) denotes by PGL∗2 (q). By this we mean the extension of PSL2 (q) by a diagonal-field automorphism. This is a nonsimple, almost simple group with a single component of index 2. Lemma 3.22. Suppose that q ≥ 7 is a prime power such that G  PGL2 (q). Then E := E(G) acts transitively on Ω with point stabilizers as described in Lemma 3.11. Moreover all point stabilizers in E have index two in the corresponding point stabilizer in G. Proof. Let α ∈ Ω and Δ := αE . We only need to show that Δ = Ω. Let g ∈ G\E be such that g induces a diagonal automorphism on E(G) and assume that Δg = Δ. If x ∈ CE (g)# fixes α, then x = xg fixes αg and hence x has two fixed points. This implies that x has a unique fixed point on Δ and hence that

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.21 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

21

NE ( x) ≤ Gα . However, with Lemma 2.8 and the subgroup structure of PSL2 (q) this forces Eα := E ∩ Gα to contain a dihedral subgroup of E. This contradicts Lemma 3.11. Therefore CE (g) acts fixed point freely on Δ and Case (b) from Lemma 3.11 must hold. Now Eα is cyclic and is inverted by an involution t in E, and if we set β := αg , then Eα = Eβ . Hence Eα has two fixed points on Δ and therefore no fixed point on Δg , because of Hypothesis 1.1. This is impossible. Hence G normalizes Δ and Δ = Ω. 2 Lemma 3.23. Let q be a power of an odd prime r, let G = PGL∗2 (q) and let S ∈ Syl2 (G). Then S is semidihedral and we denote by C, D and Q the maximal subgroups of S that are cyclic, dihedral and quaternion, respectively. Moreover let Ω be a set such that (G, Ω) satisfies Hypothesis 1.1 and let α ∈ Ω. Then one of the following is true: (1) Gα = NG (R) = RQ where R is a Sylow r-subgroup of G. (2) Gα = T C where T is a torus of order (q − 1)/2. (3) Gα = T Q where T is again a torus of order (q − 1)/2. Proof. Lemma 2.4 in [12] says that G has semidihedral Sylow 2-subgroups, so this is the first statement. Moreover q ≡ 1 modulo 8, and by Lemma 3.4 we may suppose that q ≥ 25 because q is a square. This implies that q − 1 as well as q + 1 is divisible by an odd prime. Let E := F ∗ (G) = PSL2 (q). We begin by showing that E acts transitively on Ω. Let Δ := αE . Then Lemma 2.20 gives that (E, Δ) satisfies Hypothesis 1.1. In particular we know the possible structure of Eα by Lemma 3.11. Keeping in mind that |Eα | is divisible by some odd prime p, we take a Sylow p-subgroup P of Eα . Then P ∈ Sylp (E) and by Frattini G = E · NG (P ). Moreover we note that |NE (P ) : P | is even and hence Lemma 2.8 forces |Gα : Eα | = 2. This means that Δ = Ω. After this preparation we look at Gα and we use the fact that Eα is as in Lemma 3.11(a)–(c). Here we note that Case (d) does not occur because 7 is a prime. We begin with Lemma 3.11(a), which in our notation means that Eα = NE (R). Then, since |Gα : Eα | = 2, we have that Gα = NG (R) = RQ. This is Case (a) in our lemma. Next we look at Lemma 3.11(b). Here Eα is a cyclic subgroup of E of order (q + 1)/2, and NG (Eα )/Eα is cyclic of order 4. Thus Lemma 2.8 implies that Gα contains an involution. Its centralizer contains a subgroup of index 2 of some conjugate of NG (Eα ), which together with Eα generates all of E. Hence Lemma 2.8 forces E ≤ Gα in this situation, which is impossible. Finally suppose that Lemma 3.11(c) holds, hence Eα is cyclic of order (q − 1)/2. Then NG (Eα ) contains a Sylow 2-subgroup of G, so without loss S ≤ NG (Eα ) and Lemma 2.8 forces a subgroup of index 2 of S to be contained in Gα . It cannot be D, so the only possibilities are C or Q, and this leads to the Cases (2) and (3) in our lemma. 2 Remark 3.24. All possibilities listed in Lemma 3.23 do indeed occur as examples. For this we first suppose that x ∈ Q# and let R ∈ Sylr (G). Lemma 2.4 in [12] says that whenever a subgroup U of G is normalized by a fours group in G, then it has order coprime to r. Therefore the maximal subgroups of G containing Q are NG (T ) and NG (R) = RQ. As NG (R) is a Frobenius group, we see that |CG (x)| is coprime to r, so CG (x) ≤ NG (T ). This holds for all x ∈ Q# and hence even NG ( x) ≤ NG (T ). If x ∈ C # , then similarly NG ( x) ≤ CG (x|C|/2 ) ≤ NG (T ). Finally we observe that Gα \ Eα consists entirely of elements whose order is divisible by 4 and divides (q − 1). So if y ∈ Gα \ Eα , then a suitable power y has order 4 and thus NG ( y) ≤ NG (T ) and hence |NG ( y) : NG ( y) ∩ Gα | ≤ 2.

JID:JPAA 22

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.22 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

4. The sporadic simple groups In this section we show that the sporadic simple groups do not appear as examples for Hypothesis 1.1. We usually adapt the notation in the ATLAS [8] for the names of the sporadic groups and we remind the reader that we use the notation introduced at the end of Section 2. Lemma 4.1. Suppose that G ∼ = M11 . Then there is no set Ω such that (G, Ω) satisfies Hypothesis 1.1. Proof. Assume otherwise. Hence let Ω be a set such that (G, Ω) satisfies Hypothesis 1.1. Let α ∈ Ω. As G acts transitively and non-regularly on Ω, we have that H := Gα = 1. For maximal subgroups of G and information about local subgroups we refer to Table 5.3a in [15]. Looking at the subgroup structure of G, we notice that 11  5 and 5  2, moreover 2  3 and 3  2. Thus, starting with an arbitrary prime divisor of |H|, Lemma 2.27 tells us that 2, 3 ∈ π(H). It follows with Lemma 2.11(a) that H contains a Sylow 3-subgroup of G and Lemma 2.8 implies that H contains a subgroup of index at most 2 of an involution centralizer, hence a subgroup isomorphic to SL2 (3). But then G does not have any maximal subgroup that could contain H. 2 Now that we have excluded M11 , a key observation is the following: Lemma 4.2. Suppose that G is a sporadic simple group and that Ω is such that (G, Ω) satisfies Hypothesis 1.1. Then the point stabilizers have odd order. In particular, if H is a point stabilizer and p ∈ π(H), then p  2. Proof. As G is simple, this follows immediately from Lemma 4.1, the last statement in Theorem 2.23 and Lemma 2.27. 2 Now we start to look at the remaining sporadic groups. In the first few lemmas we will give all the details, so that it becomes clear how the subgroup structure, the relations between prime divisors (Lemma 2.27) and Lemma 4.2 work together. But then the arguments become repetitive and for the larger sporadic groups it is very easy to extract the crucial information from the corresponding tables in [15]. Therefore we stop giving all the little arguments after the Conway groups and we only refer the reader to the tables where the necessary information can be found. Lemma 4.3. Suppose that G is a Mathieu Group. Then there is no set Ω such that (G, Ω) satisfies Hypothesis 1.1. Proof. Assume otherwise. Hence let Ω be a set such that (G, Ω) satisfies Hypothesis 1.1. Let α ∈ Ω and H := Gα = 1. By Lemma 4.1 it suffices to treat the groups M12 , M22 , M23 and M24 . For information about local subgroups we refer to Tables 5.3b–e in [15]. First we note that 5  2 and therefore 5 ∈ / π(H). As 11  5, Lemma 2.27(1) yields that H also has no subgroup of order 11. If 23 ∈ π(H), then the fact that 23  11 gives a contradiction. If 7 ∈ π(H), then we note that 7  3 and 3  2, so again we have a contradiction. But now, when looking at the group orders, there is no prime left that could divide |H|. 2 Lemma 4.4. Suppose that G is a Janko Group. Then there is no set Ω such that (G, Ω) satisfies Hypothesis 1.1. Proof. Assume that Ω is such a set and let α ∈ Ω. Then H := Gα = 1 by Hypothesis 1.1. For information about local subgroups of G we refer to Tables 5.3f–i in [15]. We show that 2 ∈ π(H), which contradicts Lemma 4.2.

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.23 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

23

Lemma 2.8 yields that {2, 3, 5} is connected. (For J2 this uses that, if 5 ∈ π(H), then H contains a Sylow 5-subgroup by Lemma 2.11(a) and hence 5-elements from both conjugacy classes.) Therefore, by Lemma 2.27(3), it is even sufficient that 3 or 5 is in π(H). To see this, we note: 37  3, 31  5, 43 → 3, 29 → 3, 23 → 5, 19  3 and 17  2. Hence all possible prime divisors of |H| eventually lead to 2, and this is our contradiction. 2 Lemma 4.5. Suppose that G is a Conway Group. Then there is no set Ω such that (G, Ω) satisfies Hypothesis 1.1. Proof. Assume otherwise, let Ω denote such a set and let α ∈ Ω. Then H := Gα = 1 by Hypothesis 1.1. For information about local subgroups of G we refer to Tables 5.3j–l in [15]. Again we begin by noticing that {2, 3, 5} is connected. We claim that |H| is divisible by 2, contrary to Lemma 4.2. As in the previous lemma we show that all possible prime divisors of |H| eventually lead to one of the numbers 2, 3, 5. So we note: 23 → 5, 7  3 and 13  3. This proves our claim. 2 Lemma 4.6. Suppose that G is one of the groups HS, McL, Suz, He, Ly, Ru, O  N , a Fischer sporadic simple group, HN , Th, BM or the Monster group. Then there is no set Ω such that (G, Ω) satisfies Hypothesis 1.1. Proof. As in the previous results the only information needed is that, for all prime divisors p of |G|, we have that p → 2. Then it follows that the point stabilizers have even order, contrary to Lemma 4.2. This can be observed in the corresponding Tables in [15], namely tables 5.3m up to 5.3z. 2 The conclusion of this chapter is: Theorem 4.7. Suppose that Hypothesis 1.1 holds. Then G is not a sporadic simple group. Corollary 4.8. Suppose that G is quasisimple with G/Z(G) isomorphic to a sporadic simple group. Then there is no set Ω such that (G, Ω) satisfies Hypothesis 1.1. Proof. This follows from Lemma 2.14 and Theorem 4.7.

2

5. Proofs of the main results For our theorem on simple groups, the work has been done in the previous two sections. Then our statements about quasisimple and almost simple groups follow without too much effort. However, some more preparation is required for Theorem 1.4, building on the general results in Section 2. Proof of Theorem 1.2. Suppose that (G, Ω) satisfies Hypothesis 1.1 and that G is simple. It follows from the Classification of Finite Simple Groups that G is an alternating group of rank at least 5, that G is of Lie type or that G is a sporadic group. (See for example the Appendix of [19] for a statement of the classification theorem.) Now Corollary 3.9, Theorem 3.20 and Theorem 4.7 give the result. 2 We also obtain Theorem 5.1. Suppose that G is quasisimple, but not simple. Then there is no set Ω such that Hypothesis 1.1 holds.

JID:JPAA 24

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.24 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

Proof. This follows immediately from the Classification of Finite Simple Groups and the corresponding results in Sections 3 and 4, namely Lemmas 3.10, 3.21 and 4.8. 2 Proof of Theorem 1.3. Let α ∈ Ω and let Δ := αE . We know by Lemma 2.20 that (E, Δ) satisfies Hypothesis 1.1 and hence we only need to consider the cases from Theorem 1.2. Throughout, let q denote a prime power. First assume that E  Sz(q). Then the elements of G\E induce field automorphisms of odd order on E and hence we let g ∈ G be such that g induces a nontrivial field automorphism of odd order on E. We begin by noticing that g is contained in a point stabilizer because g fixes at least five cosets of Eα in E (recall the possible orders of Eα from Lemma 3.12). Now we look at C := CE (g). This is a subfield subgroup of E, i.e. there exists a prime power q0 such that C  Sz(q0 ). In particular C has order divisible by 5 and hence so does Eα , using Lemma 2.8. But we know from Lemma 3.12 that Eα has order q 2 (q − 1) or q − 1, which is not divisible by 5 (recall that q is a power of 2 with odd exponent). This is a contradiction. Next assume that E  PSL3 (4). Then Lemma 3.13 yields that |Eα | = 5, in particular Eα ∈ Syl5 (E) and therefore G = ENG (Eα ) by a Frattini argument. We choose x ∈ NG (Eα )\E and note that x2 ∈ Gα by Lemma 2.8. Looking at the ATLAS [8] we see that the outer automorphism group of E is isomorphic to 2×S3 . If x induces an automorphism of order 3 on E, then x ∈ Gα and therefore CE (x) contains a subgroup isomorphic to PSL3 (2), hence PSL2 (7). This is false because |Eα | = 5. Thus x induces an automorphism of order 2 on E and normalizes Eα . Let t ∈ E denote an involution inverting Eα . Then Lemma 2.8 yields that x or xt is contained in Gα and the information about maximal subgroups of the automorphism group of PSL3 (4) from the ATLAS, together with Lemma 2.8 implies that Eα contains an involution. This is false. We are left with the case where E ∼ = PSL2 (q). Then Lemma 3.11 yields the possible orders for Eα , and Lemma 3.22 yields that Case (a) in Theorem 1.3 does in fact occur. Next we suppose that g ∈ G is such that g induces a nontrivial field automorphism on E. Again let C := CE (g). Then C is a subfield subgroup. Assume that C is almost simple. Then we may choose α such 2 that Eα ∩ C = 1. Again we deduce that α, αg and αg are not pairwise distinct, which means that g ∈ Gα or β := αg = α and g interchanges α and β. In any event g 2 ∈ Gα . If g 2 = 1, then Lemma 2.8 implies that a subgroup of index at most 2 of C is contained in Eα , which is impossible. Hence g 2 = 1 and in particular G induces a group of automorphisms on E that is an elementary abelian 2-group. This argument also shows that g ∈ / Gα , in fact g is not contained in any point stabilizer. As C is not a Frobenius group, there exists an element x ∈ C that fixes α and β. As g centralizes C, it centralizes Eα ∩ C and Eβ ∩ C, but also interchanges Eα and Eβ . This implies that Eα ∩ C = Eβ ∩ C. First suppose that Eα has even order. As Gα does not contain g, Lemma 2.8 implies that q is even and that Eα is the normalizer of a Sylow 2-subgroup of E. In particular |Ω| is odd and hence every involution in C fixes exactly one point. This point is then fixed by g, but we argued above that g is not contained in any point stabilizer. Hence Eα is cyclic of odd order, in particular C ∩ Eα is. Let t ∈ C be an involution such that t inverts C ∩ Eα . Then t acts fixed point freely on Ω because the point stabilizers have odd order, and hence t interchanges α and β. This implies that the involution gt centralizes α and β. But gt also centralizes a Sylow 2-subgroup of C and hence a subgroup of order at least 4. This implies (with Lemma 2.8) that Eα has even order, which is a contradiction. We conclude that C is not almost simple, so it is isomorphic to PSL2 (2), PSL2 (3) or PGL2 (3). In particular q is a power of 2 or 3 and g induces a field automorphism on E of prime order. We keep the above notation and prove that q is even. Assume that q is odd. Let r ∈ π(Eα ) be odd and let R ∈ Sylr (Eα ). Then R ∈ Sylr (E) and hence G = NG (E)H by a Frattini argument. Lemma 2.8 implies that a subgroup of index at most 2 of NG (R) lies in Hα , but we also know that NE (R)  Eα (because R has two fixed points that are interchanged by an

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.25 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

25

involution in NE (R)). Hence Hα contains an element from G\H and this means that |Gα : Eα | = |G : E|. In particular g lies in a point stabilizer. Now Lemma 2.8 implies that this point stabilizer contains a subgroup isomorphic to A4 , and this is impossible by Lemma 3.11. Hence this situation leads to Case (b) in the theorem. We finally consider the case where some g ∈ G\E induces a diagonal-field automorphism on E of order 2. We note that q is odd in this case and we let r denote the characteristic of the field. This is exactly the situation of Lemma 3.23, which shows that Case (c) of the theorem does also occur. There are no more cases to consider. 2 We will prove Theorem 1.4 by looking at a minimal counter example, and there it plays a role how Hypothesis 1.1 behaves with respect to factor groups. We have already seen an example of this in Lemma 2.14. Lemma 5.2. Suppose that N is a proper normal 2 -subgroup of G, and that α ∈ Ω. Suppose that p ∈ π(N ) is such that P := Op (N ) = 1. Then every x ∈ G# α acts fixed point freely on P . If we let Ω denote the set of P -orbits on Ω and G := G/P , then one of the following is true: (1) |Ω| = 1 and G = P Gα is a Frobenius group. (2) |Ω| = 2 and Gα is a Frobenius group of index 2 in G. (3) (G, Ω) satisfies Hypothesis 1.1. Proof. If p divides |Gα |, then Lemma 2.11(a) yields that P ≤ Gα and hence P fixes every point in Ω, because P  G. This is impossible and therefore p does not divide |Gα |. It follows that P has only regular orbits on Ω. As p is odd and Gα is a p -group now, it follows from Lemma 2.8 that Gα acts fixed point freely on P . Now we consider the action of G on Ω. If |Ω| = 1, then Ω = αP and G = P Gα . As Gα acts fixed point freely on P , it follows that G is a Frobenius group with complement Gα , proving (1). If |Ω| = 2, then Gα is of index 2 in G, hence normal in G. The action of Gα on αP is as above and thus Gα is a Frobenius group, and (2) follows. So we may now assume that |Ω| > 2. Let ω1 , ω2 ∈ Ω with representatives ω1 , ω2 ∈ Ω. Then there exists g ∈ G such that ω1g = ω2 , and ω1 g = ω2 . As Gα = 1, we may take x ∈ G# α and we see that x fixes α, so G does not act regularly on Ω. Let g ∈ G# and suppose that ω is a fixed point of g in Ω. By Lemma 2.13 we may suppose that g ∈ Gω . As P acts regularly on every element of Ω, it follows that g acts on ω in the same way as on P , hence with a unique fixed point. This means that g has a fixed point in every P -orbit that it stabilizes. Our hypothesis yields that g stabilizes at most two P -orbits and hence g fixes at most two elements of Ω. Thus (G, Ω) satisfies Hypothesis 1.1, so (3) follows. 2 We remark here that the examples in Lemmas 2.5 and 2.6 show that Case (2) in the conclusion of Lemma 5.2 really occurs. Lemma 5.3. Suppose that N is a normal 2 -subgroup of G. Let Ω denote the set of N -orbits of Ω and set G := G/N . Then (G, Ω) satisfies Hypothesis 1.1 or there exists a subgroup of G of index at most 2 that is a Frobenius group.  := G/Op (N ) and Proof. As N is solvable by [10], we choose an odd prime p such that Op (N ) = 1. Let G  denote the set of Op (N )-orbits of Ω. If one of the possibilities (1) and (2) in Lemma 5.2 holds for G,  let Ω then we use Lemma 2.8 to see that the point stabilizers act fixed point freely on Op (N ) and therefore G

JID:JPAA 26

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.26 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

 Ω)  satisfies Hypothesis 1.1 and we continue itself also satisfies (1) or (2) from Lemma 5.2. Otherwise (G,  of G  and again this leads with this argument. If Op (N ) = N , then we consider the normal 2 -subgroup N to the same possibilities. Once no more repetition is possible, we have reached one of the conclusions that are stated. 2 

Lemma 5.4. Suppose that Hypothesis 1.1 holds and that N is a normal 2 -subgroup of G such that O2 (G)/N  is simple. Then O2 (G)/N is isomorphic to one of the groups from Theorem 1.2. 

Proof. Let M := O2 (G) and G := G/N . Let Ω denote the set of N -orbits of Ω. Then the pair (G, Ω) satisfies Hypothesis 1.1 by Lemma 5.3. By hypothesis on M/N and by Lemma 2.19 the action of M on Ω is not regular, and moreover M is not a Frobenius group. Therefore (M , Ω) also satisfies Hypothesis 1.1 and we may apply Theorem 1.2. 2 Theorem 2.23 from Section 2 already played a role when we looked at which simple groups occur as examples. It will also play a role in the proof of Theorem 1.4, and there we need to know more details if the point stabilizers have odd order. We recall that a proper, nontrivial subgroup H of G is called a t.i. subgroup if and only if, for all g ∈ G, either H = H g or H ∩ H g = 1. Lemma 5.5. Suppose that G is not a Frobenius group. Let α, β ∈ Ω be distinct and such that H := Gα ∩ Gβ = 1. If |H| is odd, then H is a t.i. subgroup of G and |NG (H) : H| = 2. In particular, in its action on Λ := G/H, every element of G has either zero or two fixed points. Proof. Set Δ := {α, β} and note that |Ω| = 4, because H has odd order and acts semiregularly on Ω \ Δ. In particular, for all 1 = X ≤ H, we know that Δ = fixΩ (X) and hence NG (X) stabilizes the set Δ. Conversely, if y ∈ G stabilizes Δ, then it is contained in H or it interchanges α and β and still normalizes H. Let g ∈ G \ NG (H). Then H = H g and, as we just argued, also Δ = Δg . Thus if x ∈ H ∩ H g , then x fixes Δ ∪ Δg point-wise and hence |fixΩ (x)| ≥ |Δ ∪ Δg | ≥ 3. This forces x = 1. We know that NG (H) = H by Lemma 2.1, applied to the subgroup NG (H), and our hypothesis that G is not a Frobenius group. But we have seen that NG (H) is the stabilizer of Δ, so the assertion follows because |Δ| = 2. For the last statement, we see that |Ω| > 4 because H has odd order and by Hypothesis 1.1. Then the statement is just Remark 1.1 in [22]. 2 Theorem 5.6. Suppose that G is not a Frobenius group. Let α, β ∈ Ω be distinct and such that H := Gα ∩ Gβ = 1. If |H| is odd, then one of the following holds: (1) H or NG (H) has a normal complement in G or (2) G has two normal subgroups N and M such that N < M < G and G/M ∼ = H/(H ∩ M ). Moreover N ∩ NG (H) = 1 and N is nilpotent, the group M := M/N is simple and (M , M /H ∩ M ) satisfies Hypothesis 1.1. Proof. Lemma 5.5 yields that H is a T -subgroup in the sense of [23]. So Theorem 5 in this paper is applicable and it leads to the following possibilities: – H or NG (H) have a normal complement in G or – G has two normal subgroups N and M such that N < M < G and G/M ∼ = H/(H ∩ M ), moreover N ∩ NG (H) = 1, and M/N is a simple group that has exactly one class of involutions. In the second case (H ∩M )N/N is a t.i. subgroup of M/N that has index 2 in its normalizer. In particular, in the factor group M := M/N , the pair (M , M /H ∩ M ) with M acting by right multiplication satisfies

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.27 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

27

Hypothesis 1.1. Finally, the subgroup N is nilpotent by Lemma 2(a) of [23]. So we arrive at exactly the two cases listed. 2 Proof of Theorem 1.4. Suppose that Hypothesis 1.1 holds, but that the pair (G, Ω) is such that Theorem 1.4 does not hold and that |G| + |Ω| is minimal with this property. Let α ∈ Ω. Step 1: Suppose that Gα is metacyclic of odd order and let β ∈ Ω be such that H := Gα ∩ Gβ = 1. Then H does not have a normal complement in G. Proof. Assume otherwise and let K denote such a normal complement. In particular K ∩ H = 1. Let p ∈ π(K) and assume that p ∈ π(H). Then p is odd and Lemma 2.11(a) forces a Sylow p-subgroup of G into Gα and Gβ , hence into H. This is a contradiction and hence H and K have coprime orders. Let x ∈ H be such that x has prime order r. Then a subgroup of index 2 of CK (x) is contained in Gα , by Lemma 2.8. But Gα has odd order, so CK (x) ≤ Gα and similarly CK (x) ≤ Gβ . This means that CK (x) ≤ K ∩ H = 1 and hence that K is nilpotent. In particular G is solvable and this is Case (3) in our theorem. Hence we have a contradiction. 2 Step 2: Suppose that Gα is metacyclic of odd order and let β ∈ Ω be such that H := Gα ∩ Gβ = 1. Then NG (H) does not have a normal complement in G. Proof. First we apply Lemma 5.6 to see that |NG (H) : H| = 2. Assume that K is a normal complement for NG (H) in G. In particular K ∩ NG (H) = 1 and therefore K ∩ H = 1. Thus we argue as for Step 1 to see that H and K have coprime orders and that K is nilpotent. Since NG (H) has twice odd order, it is solvable and therefore G is solvable. This is Case (3) of Theorem 1.4 and hence we have a contradiction. 2 Step 3: Gα is not metacyclic of odd order. Proof. Assume otherwise and let β ∈ Ω be such that H := Gα ∩ Gβ = 1. Steps 1 and 2 yield that Theorem 5.6(2) holds, so we use all the notation from there. In particular we know that (M , M /H ∩ M ) satisfies Hypothesis 1.1 and that M is simple. Hence Theorem 1.2 leads to the possibilities listed in Case (4) of Theorem 1.4, which is a contradiction. 2 Step 4: F ∗ (G) is simple. Proof. Assume otherwise. Then we refer to Theorem 2.23. Step 3 and the fact that we are looking at a counter example to Theorem 1.4 only leaves Case (4) or (5) of Theorem 2.23. We begin with Case (4). If G has dihedral Sylow 2-subgroups, then we apply the Gorenstein–Walter Theorem [13]. Since Theorem 1.4(6) does not hold, the only possibility is that G/O(G) ∼ = A7 . But then Lemmas 5.4 and 3.5 give a contradiction. Next assume that G has semidihedral Sylow 2-subgroups and that Gα has twice odd order, and let S ∈ Syl2 (G). Without loss |Sα | = 2, and then Sα is generated by a noncentral involution t in S. We note that this implies that Gα has a normal 2-complement Kα . Suppose that g ∈ G is such that tg is the central involution in S. Then tg fixes αg and Lemma 2.8 forces a subgroup of S of index at most two into Gαg , hence a 2-group of size at least 8. This is impossible. It follows that t is not conjugate to the central involution in S, in particular Thompson’s Transfer Lemma is applicable and yields that t ∈ / O2 (G). Let M be a normal subgroup of G of index 2 that does not contain t. Then Kα ≤ M , in fact Kα = Mα . We note that M is not a Frobenius group because otherwise Theorem 1.4(1) holds. Also, since |Gα : Mα | = 2 = |G : M |, we see that M acts transitively on Ω. If M acts regularly, then Case (5) of Theorem 1.4 holds, which is a contradiction. It follows that (M, Ω) satisfies Hypothesis 1.1, with point stabilizers of odd order.

JID:JPAA 28

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.28 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

Since M is not a Frobenius group, we may choose β ∈ Ω to be distinct from α and such that U := Mα ∩ Mβ = 1. Then Theorem 5.6 gives us two possibilities, and we begin with (1). So suppose that U or NM (U ) has a normal complement K in M . Then as in Steps 1 and 2 we see that U and K have coprime orders, that K is nilpotent and that M is solvable. But this means that G satisfies Case (5) in Theorem 1.4, which is a contradiction. Theorem 5.6(2) also leads to Case (5) in Theorem 1.4, therefore it is impossible. This last contradiction comes from the assumption that G has semidihedral Sylow 2-subgroups, so we finished this case. Now we assume that Theorem 2.23(5) holds and we take the notation from there. We know by Lemma 5.3 that M/N , together with a suitable set, satisfies Hypothesis 1.1 and so the simplicity of M/N yields that Theorem 1.2 is applicable. Comparing the simple groups appearing there with those from Theorem 2.23(5), we see that Case (6) of Theorem 1.4 holds, which is a contradiction. 2 If F ∗ (G) = G, then G is simple by Step 4 and hence Theorem 1.2 gives a contradiction. Thus G is a nonsimple, almost simple group and we apply Theorem 1.3. But the cases there are all contained in Theorem 1.4(6), so we have our final contradiction. 2 In light of future applications, we state a more detailed version of our main result. Theorem 5.7. Suppose that (G, Ω) satisfies Hypothesis 1.1. Then one of the following holds: (1) G has a subgroup of index at most 2 that is a Frobenius group. This includes the case where G ∼ = S4 and the point stabilizer has order 2, 3, 4 or 6. (2) |Z(G)| = 2 and G/Z(G) is a Frobenius group. (3) G is solvable and there is a normal subgroup K  G, some h ∈ G and 1 = x ∈ CK (h) such that either K/O2 (K) is nilpotent or the following are true: (a) O2,2 (K) is nilpotent, (b) [x, O(K)] = 1, and (c) with the notation K := K/O2,2 (K), we either have that K = ¯ x or, if r = 2n + 1 is a Fermat prime, then Z(K) = ¯ x and K is an extraspecial 2-group of minus type; i.e. a central product of Q8 and (n − 1) copies of D8 . (4) The point stabilizers are metacyclic of odd order and G has normal subgroups N , M such that N < M < G, N is nilpotent, M/N is simple and isomorphic to PSL2 (q), to Sz(q) or to PSL3 (4), and G/M is metacyclic of odd order. This includes the following two special cases: (a) G ∼ = PSL3 (4) and G acts on Ω as on the set of cosets of a subgroup of order 5. (b) G is almost simple, there is a 2-power q ≥ 8 such that F ∗ (G) ∼ = PSL2 (q) and there exists an element g ∈ G\E such that g induces a field automorphism on E of odd order and CE (g) ∼ = PSL2 (2). ∗ Moreover, in the action of F (G) on Ω, the point stabilizers are cyclic of order q + 1. The point stabilizers in G grow by a factor o(g) and in particular they are metacyclic of odd order, but not cyclic. (5) The point stabilizers have twice odd order and G has a subgroup M of index 2 such that either (3) or (4) holds for M or M acts regularly on Ω. If G has semidihedral Sylow 2-subgroups and M is regular, then the point stabilizers have order 2 and moreover O(G) ≤ M and O(G) is abelian. In this case either G has a subgroup of index 2 that is a Frobenius group or M/O(G) ∼ = SL2 (3). (6) G has a normal subgroup N of odd order and if G := G/N , then O2 (G) is either a dihedral or semidihedral 2-group or there exists a prime power q such that it is isomorphic to Sz(q) or to a subgroup of P Γ L2 (q) that contains PSL2 (q). This includes the following special cases: (a) G is a dihedral or semidihedral 2-group with point stabilizers of order 2. (b) G ∼ = A5 and the point stabilizer has order 2, 3, 5, 6, 10 or 12.

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.29 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

29

(c) G ∼ = S5 and the point stabilizer has order 4, 6, 12 or 20. (d) G ∼ = PSL2 (7) acting on Ω as on the set of cosets of a subgroup isomorphic to A4 . ∼ (e) G = PSL2 (q) or PGL2 (q). Let d be the greatest common divisor of 2 and q − 1 and let r denote the characteristic of the field. Then the point stabilizer of the action of PSL2 (q) is the normalizer of a q−1 Sylow r-subgroup of G or it is cyclic of order q+1 d or d . The point stabilizers in PGL2 (q) grow by a factor 2 and in particular they are not cyclic. (f) G ∼ = PGL∗2 (q) (which is PSL2 (q) extended by a diagonal-field automorphism), where q is a power of an odd prime r. Let S ∈ Syl2 (G) and let C and Q denote the maximal subgroups of S that are cyclic and quaternion, respectively. Moreover let T denote a torus of order (q − 1)/2. Then the point stabilizers are normalizers of a Sylow r-subgroup in G or they are conjugate to T C or to T Q. (g) G ∼ = Sz(q) acting 2-transitively on Ω in its natural action or as on the set of cosets of a subgroup of order q − 1. (h) If N = 1 and O2 (G) ∼ = PSL2 (q), then the point stabilizers of the action on Ω are cyclic of order q + 1. Proof. We go through the cases in Theorem 1.4. We have seen that S4 provides examples and since S4 contains a subgroup of index 2 that is a Frobenius group, it is a special case of (1). The exact possibilities are given in Lemma 2.2. In Case (3) we take the notation from there. We recall that this case arises if a point stabilizer has odd order, using Theorem 5.6. Let K denote a normal complement to a nontrivial two point stabilizer H (or to NG (H)) in G. Then the action of K on Λ := G/H is regular, and also (|K|, |H|) = 1 and 2 = |fixΛ (h)| = |CK (h)| for all h ∈ H. Moreover K is solvable, so we may apply Theorem B of [7] to K. This result implies the statements (a)–(c). For the details in (4)(a) we refer to Lemma 3.13. For (4)(b) we suppose that G is almost simple and that there is a 2-power q ≥ 8 such that F ∗ (G) ∼ = PSL2 (q), with an element g ∈ G\E such that g induces a field automorphism of odd prime order on E. Then CE (g) ∼ = PSL2 (2) by Lemma 1.3, the point stabilizers ∗ in F (G) are known by Lemma 3.11, and a Frattini argument gives that the order of the point stabilizers in G grows by a factor o(g). In (5) we look at the special case described there. We let S ∈ Syl2 (G), α ∈ Ω and we suppose that S is semidihedral and that Gα = Sα has order 2, moreover we recall that M acts regularly in this case. Let t ∈ S be such that Sα = t. Then |CM (t)| = 2 and hence Lemma 2.7 is applicable. Let N be a normal subgroup as in the lemma. If N = 1, then the lemma tells us that M/N ∼ = A4 and in particular S has only order 8. But S is semidihedral and hence has order at least 16. Thus N = 1. If |M : N | = 2, then the fact that t inverts N , and hence a Sylow 2-subgroup of N , forces N ∩ S to be cyclic. In particular N has a normal 2-complement, and O(G) = O(N ) ≤ M , moreover O(G) is abelian as stated. Here we see that N · CM (t) is a Frobenius group that has index 2 in G. In the last case we suppose that |M : N | = 2. Then Lemma 2.7 gives that N = Z(M ) and M/N ∼ = A4 . ∼ This is only possible if S ∩ M = Q8 and hence |S| = 16, moreover O(G) = O(N ) is abelian again and our statement follows. In (6) we listed some special cases. The details for (b) and (c) are given in Lemmas 3.1 and 3.2. The details for (d) are exactly as in Theorem 3.20(1)(d). In Case (e) we refer to Theorem 3.20 again and Lemma 3.22. Case (g) is described in Theorem 3.20(3). In Case (h) we let α ∈ Ω and p ∈ π(N ). Now Op (N )Gα is a Frobenius group with Frobenius complement Gα , by Lemma 2.16, which excludes the normalizers of Sylow 2-subgroups in Case (e) from consideration. As cyclic groups of order (q − 1) have fixed points on every nontrivial G-module of odd characteristic, these possibilities are also excluded from consideration. This gives the statement in (h). 2

JID:JPAA

AID:5002 /FLA

30

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.30 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

Concluding remarks. The group S3 in its natural action is an example of what can happen in Case (1) of Theorem 1.4, while Lemma 2.5 gives a series of examples for Cases (1) and (2). We also note that Lemmas 2.6 and 2.3 give rise to infinite series of examples for Cases (1) and (3), respectively. Let q be a power of an odd prime and L := PSL2 (q), moreover let H ≤ L be cyclic of order q+1 2 and let V be an irreducible L-module such that CV (H) = 0. For all n ∈ N, if we let G := V n  L with diagonal action of L and if we consider the action of G on the set of cosets of H in G, then this gives rise to an infinite series of examples for Cases (4) and (6) in Theorem 1.4. We also see that in Case (4), the point stabilizers are cyclic or Frobenius groups (as can be observed in the almost simple groups that appear). Lemma 2.7 gives rise to examples for the special case of Theorem 1.4(5) as explained in Theorem 5.7 above. We would like to mention that the special case of a regular normal subgroup of index 2 already appears in [22], as does Case (1) of Theorem 1.4. We note that the permutation rank of PSL2 (q) acting on the cosets of a cyclic subgroup of order (q + 1) is q and acting on subgroups of order (q − 1) is q + 4. The permutation rank of Sz(q) acting on the cosets of a cyclic subgroup of order q − 1 is q 3 + q 2 + 2q + 4. Perhaps it is this last example which suggests why it may be difficult to analyze the simple groups satisfying our Hypothesis 1.1 without resorting to the Classification of the Finite Simple Groups. Acknowledgements The second author wishes to thank the University of Birmingham for the hospitality during numerous visits and the Deutsche Forschungsgemeinschaft for financial support of this project (the grant number WA 3089/3-1). The first author wishes to thank the Martin-Luther-Universität Halle–Wittenberg for its hospitality during his visits. Both authors wish to thank the referee for helpful comments and Chris Parker for bringing the work of Pretzel and Schleiermacher to their attention, which has lead to a considerable strengthening of the main theorem. References [1] J.L. Alperin, R. Brauer, D. Gorenstein, Finite groups with quasi-dihedral and wreathed Sylow 2-subgroups, Trans. Am. Math. Soc. 151 (1970) 1–261. [2] M. Aschbacher, On Bol loops of exponent 2, J. Algebra 288 (2005) 99–136. [3] M. Aschbacher, G. Seitz, Involutions in Chevalley groups over fields of even order, Nagoya Math. J. 63 (1976) 1–91; Correction: Nagoya Math. J. 72 (1978) 135–136. [4] H. Bender, Transitive Gruppen, in denen jede Involution genau einen Punkt festläßt, J. Algebra 17 (1971) 527–554. [5] H. Bender, Endliche Fastkörper und Zassenhausgruppen, in: Group Theory, Algebra, and Number Theory, de Gruyter, Berlin, 1996, pp. 97–143. [6] R. Brauer, M. Suzuki, On finite groups of even order whose 2-Sylow subgroup is a quaternion group, Proc. Natl. Acad. Sci. USA 45 (1959) 175–179. [7] M. Collins, B. Rickman, Finite groups admitting an automorphism with two fixed points, J. Algebra 49 (1977) 547–563. [8] J. Conway, R. Curtis, S. Norton, R. Parker, R. Wilson, Atlas of Finite Groups, Oxford University Press, 2003. [9] H.M. Farkas, I. Kra, Riemann Surfaces, 2nd ed., Springer, 1992. [10] W. Feit, J.G. Thompson, Solvability of groups of odd order, Pac. J. Math. 13 (1963) 775–1029. [11] H. Fukushima, Finite groups admitting an automorphism of prime order fixing an abelian 2-group, J. Algebra 89 (1) (1984) 1–23. [12] D. Gorenstein, Finite groups the centralizers of whose involutions have normal 2-complements, Can. J. Math. 21 (1969) 335–357. [13] D. Gorenstein, J.H. Walter, The characterization of finite groups with dihedral Sylow 2-subgroups, J. Algebra 2 (1965) 85–151. [14] D. Gorenstein, R. Lyons, R. Solomon, The Classification of the Finite Simple Groups, Number 2, Math. Surv. Monogr., vol. 40.2, American Mathematical Society, Providence, RI, 1996. [15] D. Gorenstein, R. Lyons, R. Solomon, The Classification of the Finite Simple Groups, Number 3, Math. Surv. Monogr., vol. 40.3, American Mathematical Society, Providence, RI, 1998. [16] R. Guralnick, K. Magaard, On the minimal degree of a permutation group, J. Algebra 207 (1) (1998) 127–145. [17] B. Huppert, Endliche Gruppen I, Grundlehren Math. Wiss., Band 134, Springer, 1967.

JID:JPAA

AID:5002 /FLA

[m3L; v 1.134; Prn:26/05/2014; 14:39] P.31 (1-31)

K. Magaard, R. Waldecker / Journal of Pure and Applied Algebra ••• (••••) •••–•••

31

[18] P. Kleidman, The maximal subgroups of the Chevalley groups G2 (q) with q odd, the Ree groups 2 G2 (q), and their automorphism groups, J. Algebra 117 (1) (1988) 30–71. [19] H. Kurzweil, B. Stellmacher, The Theory of Finite Groups, Springer, 2004. [20] M. Liebeck, J. Saxl, G. Seitz, Subgroups of maximal rank in finite exceptional groups of Lie type, Proc. Lond. Math. Soc. (3) 65 (1992) 297–325. [21] K. Magaard, H. Völklein, On Weierstrass points of Hurwitz curves, J. Algebra 300 (2) (2006) 647–654. [22] O. Pretzel, A. Schleiermacher, On permutation groups in which non-trivial elements have p fixed points or none, Proc. Lond. Math. Soc. (3) 30 (4) (1975) 471–495. [23] O. Pretzel, A. Schleiermacher, On permutation groups in which nontrivial elements fix two points or none, J. Algebra 44 (1977) 283–289. [24] M. Schönert, et al., GAP – Groups, Algorithms, and Programming – version 3, release 4, patchlevel 4, Lehrstuhl D für Mathematik, RWTH Aachen, Germany, 1997. [25] M. Suzuki, On a class of doubly transitive groups, Ann. Math. (2) 75 (1962) 105–145. [26] H. Zassenhaus, Kennzeichnung endlicher linearer Gruppen als Permutationsgruppen, Abh. Math. Semin. Univ. Hamb. 11 (1) (1935) 17–40.