The O’Nan–Scott Theorem

Robert A. Wilson

QMUL, November 2005

1 Introduction

The O’Nan–Scott theorem gives us a classification of the maximal subgroups of the alternating and symmetric groups. It does not tell us exactly what the maximal subgroups are—that is too much to ask, rather like asking what are all the prime numbers. It does however provide a first step towards writing down the list of maximal subgroups of A_{n} or S_{n} for any particular reasonable value of n.

Theorem 1 If H is any proper subgroup of S_{n} other than A_{n}, then H is a subgroup of one or more of the following subgroups:

1. An intransitive group S_{k} × S_{m}, where n = k + m;

2. An imprimitive group S_{k} ≀ S_{m}, where n = km;

3. A primitive wreath product, S_{k} ≀ S_{m}, where n = k^{m};

4. An affine group AGL_{d}(p) ≅ p^{d}:GL_{d}(p), where n = p^{d};

5. A group of shape T^{m}.(Out (T) × S_{m}), where T is a non-abelian simple group, acting on the cosets of the ‘diagonal’ subgroup Aut (T)×S_{m}, where n = |T |^{m−1};

6. An almost simple group acting on the cosets of a maximal subgroup.

Note that the theorem does not assert that all these subgroups are maximal in S_{n}, or in A_{n}. This is a rather subtle question, particularly as far as subgroups of A_{n} are concerned. The last category of subgroups also requires us to know all the finite simple groups, or at least those with a maximal subgroup of index n. In practice, this means that we can only ever hope to get a recursive description of the maximal subgroups of A_{n} and S_{n}.

2 Preliminaries

In this section we collect a number of general facts about groups which will be useful in the proof of the O’Nan–Scott theorem. Some you may already know, others may be unfamiliar. They are all important!

Lemma 1 Every normal subgroup N of a primitive group H is transitive, Proof. Otherwise the orbits of N form a block system for H. ⊓ ⊔

Lemma 2 Any two distinct minimal normal subgroups N_{1} and N_{2} of any group H commute.

Proof. We have [N_{1}, N_{2}] ≤ N_{1}∩N_{2}⊴H so by minimality [N_{1}, N_{2}] = N_{1}∩N_{2} = 1. ⊓ ⊔

A subgroup K of a group N is called characteristic if it is fixed by all automorphisms of N. The following is obvious:

Lemma 3 If K is characteristic in N and N is normal in H then K is normal in H.

A group K ≠ 1 is called characteristically simple if K has no proper nontrivial characteristic subgroups.

Lemma 4 If K is characteristically simple then it is a direct product of isomorphic simple groups.

Proof. If T is any minimal normal subgroup of K, then so is T^{α} for any α ∈ Aut K. So by Lemma 2 either T^{α} = T or T ∩ T^{α} = 1. In the latter case T T^{α} = T × T^{α} is a direct product. Since K is characteristically simple, it is generated by all the T^{α}. By induction we obtain that K is a direct product of a finite number of such T^{α}. But then any normal subgroup of T is normal in K, so by minimality of T, T is simple. ⊓ ⊔

Corollary 1 Every minimal normal subgroup N of a finite group H is a direct product of isomorphic simple groups (not necessarily non-abelian).

Proof. By minimality, N is characteristically simple. ⊓ ⊔

A group N is called regular on Ω if for each pair of points a and b in Ω, there is exactly one element of N mapping a to b. In particular N is transitive and |N| = |Ω|, and every non-identity element of N is fixed-point-free.

Lemma 5 If H is primitive, and N is a non-trivial normal subgroup of H, then either C_{H}(N) is trivial, or C_{H}(N) is regular and |C_{H}(N)| = |Ω|.

Proof. By Lemma 1, if C_{H}(N) ≠ 1 then both N and C_{H}(N) are transitive. Moreover, if 1 ≠ x ∈ C_{H}(N) has any fixed points, then the set of fixed points of x is preserved by N. This contradiction implies that every element of C_{H}(N) is fixed-point-free. This means that C_{H}(N) is regular. ⊓ ⊔

This has a number of important consequences.

Corollary 2 If H is primitive, and N_{1} and N_{2} are normal subgroups of H, and [N_{1}, N_{2}] = 1, then N_{2} = C_{H}(N_{1}) and vice versa. In particular, H contains at most two minimal normal subgroups, and if it has an abelian normal subgroup then it has only one minimal normal subgroup.

Proof. By Lemma 1, N_{1} is transitive, and by Lemma 5, C_{H}(N_{2}) is regular. But N_{1} ⊆ C_{H}(N_{2}), and therefore N_{1} and C_{H}(N_{2}) have the same order and are equal. ⊓ ⊔

Corollary 3 With the same notation, N_{1} ≅ N_{2}.

Proof. Fix a point x ∈ Ω, and let K be the stabilizer of x in the group N_{1}N_{2}. Then K ∩ N_{1} = K ∩ N_{2} = 1 as N_{1} and N_{2} are regular. Therefore KN_{1} = KN_{2} = N_{1}N_{2}, and by the isomorphism theorem

K ≅ K/K ∩ N_{1} ≅ KN_{1}/N_{1} = N_{2}N_{1}/N_{1} ≅ N_{2}

and similarly K ≅ N_{1}. ⊓ ⊔

Lemma 6 Suppose that H is primitive and N is a minimal normal subgroup of H. Let K be the stabilizer in H of a point. Then KN = H.

Proof. K does not contain N, for if it did then N would act trivially on Ω, whence N = 1. Moreover, K is maximal in H, since H is primitive, and therefore KN = H for otherwise K < KN < H contradicting maximality. ⊓ ⊔

A subgroup X of H is called K-invariant if K ≤ N_{H}(X). In fact,

Lemma 7 Suppose that H is primitive and N is a minimal normal subgroup of H. Let K be the stabilizer in H of a point. Then K ∩ N is maximal among K-invariant subgroups of N.

Proof. If K ∩ N < X < N and K ≤ N_{H}(X) then KX is a subgroup of H and K < KX < H, contradicting maximality of K. ⊓ ⊔

3 The proof of the O’Nan–Scott theorem

Reduction to the case N unique and non-abelian. Let H be a subgroup of S_{n} not containing A_{n}. Then H is either intransitive (giving case 1 of the theorem), or transitive imprimitive (giving case 2 of the theorem), or primitive. If it is primitive, let N be a minimal normal subgroup of H.

If N is abelian, it is an elementary abelian p-group, acting regularly, and N = C_{H}(N). Therefore H is affine (case 4 of the theorem).

Otherwise, all minimal normal subgroups of H are non-abelian. If there is more than one minimal normal subgroup, say N_{1} and N_{2}, then N_{1} ≅ N_{2} and both N_{1} and N_{2} act regularly on Ω.

Thus N_{1} and N_{2} act in the same way on the n points, so there is an element x of S_{n} conjugating N_{1} to N_{2}. Moreover, by Corollary 2, N_{2} = C_{S_{n}}(N_{1}). Therefore x conjugates N_{2} to N_{1}, and ⟨H, x⟩ has a unique minimal normal subgroup N = N_{1} × N_{2}. So this case reduces to the case when there is a unique minimal normal subgroup.

The case N unique and non-abelian. From now on, we can assume that H has a unique minimal normal subgroup, N, which is non-abelian. If N is simple, we are in case 6 of the theorem. Otherwise, N is non-abelian, nonsimple, say N = T_{1} ×⋯×T_{m} with T_{m} ≅ T simple and m > 1, and H permutes the T_{i} transitively by conjugation.

For each i, let K_{i} be the image of K ∩ N under the natural projection from N to T_{i}. We divide into two cases: either K_{i} ≠ T_{i} for some i (and therefore for all i) or K_{i} = T_{i} for all i.

Case 1. In the former case, K normalizes K_{1} × ⋯ × K_{m}, so that, by maximality, we have K ∩ N = K_{1} × ⋯ × K_{m}, and K permutes the K_{i} transitively. Let k be the index of K_{i} in T_{i}. Then H is evidently contained in the group S_{k} ≀ S_{m} acting in the product action (case 3 of the theorem).

Case 2. There remains the case where K_{i} = T_{i} for all i. Let Ω_{1} be a minimal subset of {1, …, m} such that K ∩ N contains an element whose support is {T_{i} : i ∈ Ω_{1}}. Then the elements with this support still map onto T_{i}, i ∈ Ω_{1}, since they map onto a whole conjugacy class and T_{i} is simple. Now if Ω_{2} is another such set, intersecting Ω_{1} non-trivially, we have elements x and y in K ∩ N such that [x, y] has support {T_{i} : i ∈ Ω_{1} ∩ Ω_{2}}. Minimality of Ω_{1} implies that it is a block in a block system invariant under K, and therefore under H = KN.

The blocks cannot have size 1, for then K contains N, a contradiction. If the block system is non-trivial, with l blocks of size k, say, and l > 1, then n = |T |^{(k−1)l} and H lies inside S_{r} ≀ S_{l}, in its product action, where r = |T |^{k−1}. This is case 3 of the theorem again.

Otherwise, the block system is trivial, so K ∩ N is a diagonal copy of T inside T_{1} × ⋯ × T_{m}, and we can choose our notation such that it consists of the elements (g, g, …, g) for all g ∈ T. Also n = |T |^{m−1}, and the n points can be identified with the n conjugates of K ∩ N by elements of N. The largest subgroup of S_{n} preserving this setup is as in case 5 of the theorem. This concludes the proof of the O’Nan–Scott theorem.