Supercuspidal Gelfand pairs

Journal of Number Theory 100 (2003) 251–269

http://www.elsevier.com/locate/jnt

Supercuspidal Gelfand pairs Jeffrey Hakim Department of Mathematics and Statistics, American University, 4400 Massachusetts Ave. NW, Washington, DC 20016, USA Received 20 September 2001 Communicated by D. Goss

Abstract If G is a totally disconnected group and H is a closed subgroup then, according to the Gelfand–Kazhdan Lemma, if the double coset space H\G=H is preserved by an antiautomorphism of G of order two then ðG; HÞ must be a Gelfand pair in the sense that HomH ðp; 1Þ has dimension at most one for each irreducible, admissible representation p of G: Under certain rather general restrictions, we show that if the symmetry property holds only for almost all double cosets, then ðG; HÞ is a supercuspidal Gelfand pair in the sense that dim HomH ðp; 1Þ for all irreducible, supercuspidal representations p of G: There exist examples of supercuspidal Gelfand pairs which are not Gelfand pairs. r 2003 Elsevier Science (USA). All rights reserved. Keywords: Supercuspidal representations; Gelfand pairs; Symmetric spaces

1. Introduction If G is a totally, disconnected, unimodular group and H is a closed subgroup, then the Gelfand/Kazhdan Lemma, recalled below, gives a necessary condition for ðG; HÞ to be a (generalized) Gelfand pair in the sense that dim HomH ðp; 1Þp1 for every irreducible, admissible representation p of G: This result has proved to be one of the most fundamental tools in the harmonic analysis and representation theory associated to the homogeneous spaces H\G: In practice, however, it can be difficult to apply because it requires a potentially complicated study of the family of all bi-Hinvariant distributions on G:

E-mail address: [email protected]. 0022-314X/03/$ - see front matter r 2003 Elsevier Science (USA). All rights reserved. doi:10.1016/S0022-314X(02)00131-2

252

J. Hakim / Journal of Number Theory 100 (2003) 251–269

The object of this paper is to provide, for symmetric spaces over a nonarchimedean local field, a simpler criterion (stated in Section 8) whose statement does not involve distributions. It uses instead a symmetry hypothesis which must hold for almost all double cosets in H\G=H: The fact that we may ignore a measure zero collection of double cosets is a major virtue of our approach, since, in applications, this can represent a substantial reduction in effort. (This is illustrated with examples in Section 9.) Our criterion applies only to irreducible supercuspidal representations; however, this limitation seems appropriate since in the most important examples one expects dim HomH ðp; 1Þp1 to hold for irreducible, supercuspidal representations, but not for arbitrary irreducible, admissible representations. Indeed, it should not be surprising that the theory of harmonic analysis on symmetric spaces over local fields has special features which are exhibited only by supercuspidal representations and thus are not direct analogues of corresponding results for real groups. The search for special techniques to study supercuspidal representations associated to symmetric spaces is a basic motivation for this work as well as recent work conducted jointly by the author and Murnaghan (see [7–9]). The proof of the main theorem is a reworking of the proof of the Gelfand/ Kazhdan Lemma which uses the Schur orthogonality relations together with certain facts about bi-H-invariant distributions from the work of Rader and Rallis [13], namely, an analogue of the Weyl integration formula for G=H and an analogue of Harish-Chandra’s result that the character of an admissible representation of G is locally constant on the regular set. In re-examining the proof of the Gelfand/Kazhdan Lemma, we first observe that it involves a study of the class of all bi-H-invariant distributions on G when, in fact, * the only relevant distributions are certain generalized matrix coefficients /pðf Þl; lS; * defined below, where l and l are nonzero H-invariant linear forms on the spaces of p and its contragredient p: * When p is an irreducible, supercuspidal representation of G this observation allows us to apply (a slight extension of) the Schur orthogonality relations as follows. Suppose Z is the center of G and o is the central character of p: If the test function f ACcN ðGÞ is chosen so that Z f ðgzÞoðzÞ dz ¼ /v; pðgÞ* * vS Z

for some vectors vAV and v*AV˜ and some Haar measure on Z; then we have the identity * ¼ dðpÞ1 /l; v*S/v; lS; * /pðf Þl; lS where dðpÞ is the formal degree of p: Next, appealing to the results in [13] mentioned above, we obtain the basic formula X 1 Z 1 * dðpÞ /l; v*S/v; lS ¼ jDy ðtÞj1=2 YðtÞFTj ðtÞ dt; > w 0 TAT T Zy \T

J. Hakim / Journal of Number Theory 100 (2003) 251–269

253

* where Y is a locally constant function which represents the distribution /pðf Þl; lS on a certain set of elements in G which are ‘‘regular relative to H:’’ Roughly speaking, the other objects on the right-hand side are as follows: T is a certain family of ‘‘Cartan subsets’’ T of G=H; the numbers wT are constants, Dy is a certain Weyl discriminant, and FTj is a suitable orbital integral of the function jðgÞ ¼ YðgÞ/v; pðgÞ* * vS: Under the hypotheses of the main theorem, the right-hand side of the above equation has a certain symmetry which implies that there exists an intertwining operator I such that we obtain the relation * ¼ /l; IðvÞS/I 1 ð*vÞ; lS: * /l; v*S/v; lS Our theorem follows almost immediately from this identity, in much the same way that the Gelfand/Kazhdan Lemma follows from an analogous identity.

2. Generalized vectors and matrix coefficients In this section, we fix some notations and recall some basic notions which are discussed, for example, in [2]. Let G be a locally compact totally disconnected, unimodular group with center Z: Then G has a system of neighborhoods of the identity K1 *K2 *? consisting of open, compact subgroups. Suppose p is an irreducible, admissible representation of G on a complex vector space V with central ˜ be the contragredient and let /v; v*S denote the usual quasicharacter w: Let ðp; * VÞ ˜ pairing on V  V: ˜ CÞ by identifying vAV with the linear form We may embed V in V˜ n ¼ HomðV; n ˜ ˜ v*//v; v*S: If lAV and v*AV it therefore makes sense to use the notation /l; v*S for lð*vÞ: When gAG and lAV˜ n ; we define pðgÞlAV˜ n by /pðgÞl; v*S ¼ /l; pðg * 1 Þ*vS: We are implicitly abusing notation here by letting p denote both the given representation of G on V and its extension to a representation on the space V˜ n of ‘‘generalized vectors.’’ Recall that V is the space of smooth vectors in V˜ n : In other words, a generalized vector lAV˜ n lies in V precisely when it is fixed by an open compact subgroup of G: Fix a Haar measure on G and use it to define Z f ðgÞpðgÞ dg pðf Þ ¼ G

when f lies in the space CcN ðGÞ of smooth, compactly supported functions on G: Note that if l is a generalized vector then pðf ÞlAV : We apply similar notational practices to the extension of p* to a representation on V n ¼ HomðV ; CÞ: Note that we have extended the pairing /; S from V  V˜ to the n * * is not union of V˜ n  V˜ with V  V n : However, if lAV˜ n and lAV then /l; lS

J. Hakim / Journal of Number Theory 100 (2003) 251–269

254

defined unless either l or l* is a smooth vector. In particular, one cannot simply * define a generalized matrix coefficient by the formula /pðgÞl; lS: On the other hand, such a generalized matrix coefficient does exist as a distribution * Yp;l;l* ðf Þ ¼ /pðf Þl; lS: Recall that a distribution on G is a linear map CcN ðGÞ-C: Lemma 1. Suppose p1 and p2 are irreducible, admissible representations of G: If there exists a nonzero distribution Y which is simultaneously a generalized matrix coefficient for both p1 and p2 then p1 Cp2 : Proof. If Y is a matrix coefficient, in the usual sense, then our assertion is wellknown. The general result is deduced from this special case as follows. Choose f0 ACcN ðGÞ such that Yðf0 Þa0: Then f0 must be K-fixed for some compact open subgroup K of G: Let eK be the characteristic function of K multiplied by the appropriate constant so that eK * eK ¼ eK : When gAG; we normalize the characteristic function of KgK to obtain a function eKgK such that pi ðeK Þpi ðgÞpi ðeK Þ ¼ pi ðeKgK Þ; for i ¼ 1; 2: We can choose linear forms l1 ; l* 1 ; l2 and l* 2 such that Yðf Þ ¼ /p1 ðf Þl1 ; l* 1 S ¼ /p2 ðf Þl2 ; l* 2 S; for all f ACcN ðGÞ: Now *K *K define vectors lK * i ðeK Þl* i ; with i ¼ 1; 2: Then /p1 ðgÞlK i ¼ pi ðeK Þli and li ¼ p 1 ; l1 S ¼ K *K YðeKgK Þ ¼ /p2 ðgÞl2 ; l2 S; for all g: Since we have shown that p1 and p2 share a common nonzero matrix coefficient, our claim follows. & n * Choose generalized vectors lAV˜ n and lAV : We associate to l a distribution f /pðf Þl on G with values in V or, in other words, an element of HomðCcN ðGÞ; V Þ ˜ Tensoring these distributions and, similarly, l* gives a distribution with values in V: together gives a bilinear form

˜ Bp;l;l* : CcN ðGÞ#CcN ðGÞ-V #V-C which is given explicitly by * Bp;l;l* ðf2 ; f1 Þ ¼ /pðf2 Þl; pðf * 1 ÞlS: Note that Bp;l;l* ðf2 ; f1 Þ ¼ Yp;l;l* ðfˇ1 * f2 Þ; ˇ ¼ f ðg1 Þ: where, in general, fðgÞ Define the left and right radicals of Bp;l;l* by LRadðBp;l;l* Þ ¼ ff2 ACcN ðGÞ: Bp;l;l* ðf2 ; f1 Þ ¼ 0; for all f1 ACcN ðGÞg;

J. Hakim / Journal of Number Theory 100 (2003) 251–269

255

RRadðBp;l;l* Þ ¼ ff1 ACcN ðGÞ: Bp;l;l* ðf2 ; f1 Þ ¼ 0; for all f2 ACcN ðGÞg: It is easily verified that these definitions are equivalent to ˜ lg; LRadðBp;l;l* Þ ¼ ff2 ACcN ðGÞ: pð * fˇ2 ÞVCker * RRadðBp;l;l* Þ ¼ ff1 ACcN ðGÞ: pðfˇ1 ÞV Cker lg: On the other hand, we also have ˜ f2 ALRadðB * Þ whenever pð ker l ¼ f*vAV: * fˇ2 Þ*v ¼ v*g; p;l;l ker l* ¼ fvAV : f1 ARRadðBp;l;l* Þ whenever pðfˇ1 Þv ¼ vg: Thus not only is left radical of Bp;l;l* determined by the kernel of l; but, conversely, ker l is determined by LRadðBp;l;l* Þ: We also note that the kernel of l both determines and is determined by the span Cl of l: Thus, in this sense, the three objects LRadðBp;l;l* Þ; ker l and Cl are essentially interchangeable. A similar statement may be made regarding the right radical. We record these equivalences schematically as LRadðBp;l;l* Þ2ker l2Cl; * * RRadðBp;l;l* Þ2ker l2C l:

3. Spherical matrix coefficients If H1 and H2 are closed subgroups of G and w1 and w2 are quasicharacters of ZH1 and ZH2 ; respectively, then we say that a distribution Y on G is ðw1 ; w2 Þ-spherical if it satisfies Yðh1 f h2 Þ ¼ w1 ðh1 Þ1 w2 ðh2 Þ1 Yðf Þ for all h1 AZH1 and h2 AZH2 ; where on G; the distribution Y satisfies

h1 h2

f ðgÞ ¼ f ðh1 gh2 Þ: As a generalized function

Yðh1 gh2 Þ ¼ w1 ðh1 Þw2 ðh2 ÞYðgÞ for h1 AZH1 and h2 AZH2 : * ¼ w1 ðhÞ/v; lS * for all Suppose l* is a linear form on V such that /pðhÞv; lS * hAZH1 : If l is nonzero, this is equivalent to assuming that this relation holds for all

J. Hakim / Journal of Number Theory 100 (2003) 251–269

256

hAH1 and, in addition, w1 jZ ¼ o: The space of such linear forms is denoted HomZH1 ðp; w1 Þ: Similarly, one defines HomZH2 ðp; * w1 2 Þ: Distributions of the form 1 * Yp;l;l* ; where lAHomZH1 ðp; w1 Þ and lAHomZH2 ðp; * w2 Þ are nonzero, will be referred to as ðw1 ; w2 Þ-spherical matrix coefficients of p: These are the most basic examples of ðw1 ; w2 Þ-spherical distributions and they will be of central importance in the remainder of this paper. We will say that p is ðw1 ; w2 Þ-distinguished if the spaces HomZH1 ðp; w1 Þ and HomZH2 ðp; * w1 2 Þ are both nonzero, and thus o ¼ w1 jZ ¼ w2 jZ: * Given lAHomZH1 ðp; w1 Þ and lAHomZH2 ðp; * w1 2 Þ; we have the identities * pðf * h1 Þl* ¼ w1 ðh1 Þpðf * Þl; pðf h2 Þl ¼ w2 ðh2 Þ1 pðf Þl; where f ACcN ðGÞ; f h ðgÞ ¼ f ðghÞ; h1 AZH1 and h2 AZH2 ; and the convolution identity h1

h1

ðfˇ1 * f2 Þh2 ¼ ðf1 1 Þ$ * f2h2

for f1 ; f2 ACcN ðGÞ:

4. The Gelfand/Kazhdan Lemma In this section, we consider the situation in which we are given an automorphism y of G of order two. We are especially interested generalized matrix coefficients Y which are invariant under the anti-automorphism gy ¼ yðgÞ1 in the sense that Yðf Þ ¼ Yðf y Þ for all f ACcN ðGÞ; where f y ðgÞ ¼ f ðgy Þ: If such a distribution Y exists and Y ¼ Yp;l;l* then * ¼ /l; p* y ðf ÞlS * ¼ Y y * ðf Þ Yp;l;l* ðf Þ ¼ Yðf y Þ ¼ /pðf y Þl; lS p* ;l;l for all f ACcN ðGÞ: Here, we are using the notation py for the representation defined by py ðgÞ ¼ pðgy Þ on the same representation space as p: The previous identity shows that p and p* y share a common generalized matrix coefficient. Hence, from Lemma 1, we deduce that pCp* y : More precisely, we have: Lemma 2. Suppose ðp; V Þ is an irreducible, admissible, representation of G: If there exists a nonzero generalized matrix coefficient Y of p such that Yðf Þ ¼ Yðf y Þ for all f ACcN ðGÞ then pCp* y : If Y ¼ Yp;l;l* then Tðpðf ÞlÞ ¼ p* y ðf Þl* defines a nonzero intertwining operator from p to p* y :

J. Hakim / Journal of Number Theory 100 (2003) 251–269

257

Proof. The fact that pCp* y follows from the remarks above. To complete the proof, we must show that T; as specified above, gives a well-defined intertwining operator. First, we observe that V ¼ pðCcN ðGÞÞl: Next, we note that if f ; f 0 ACcN ðGÞ then the condition pðf Þl ¼ pðf 0 Þl is equivalent to the condition f  f 0 A LRadðBÞ which, in turn, is equivalent to the condition f y  f 0y A RRadðBÞ: Hence, pðf Þl ¼ pðf 0 Þl is * It follows that Tðpðf ÞlÞ ¼ pðf equivalent to pðf * fˇ0s Þl: * y Þl* determines a well* y Þl* ¼ pð ˜ It is now elementary to see that T must in fact be a defined map from V to V: nonzero intertwining operator from p to p* y : & We now state an elementary modification of the usual form of the Gelfand/ Kazhdan Lemma. We refer to [4], as well as Proposition 4.2 [5] and the discussion in the introduction to [11]. Lemma 3. Suppose ðp; V Þ is an irreducible, admissible, ðw1 ; w2 Þ-distinguished representation of G and y is an automorphism of order two of G such that Yðf Þ ¼ Yðf y Þ whenever f ACcN ðGÞ and Y is a ðw1 ; w2 Þ-spherical matrix coefficient of p: Then y and dim HomZH1 ðp; w1 Þ ¼ dim HomZH2 ðp; pCp * w1 * 2 Þ ¼ 1: Proof. Choose nonzero linear forms l* A HomZH1 ðp; w1 Þ and l A HomZH2 ðp; * w1 2 Þ: Let Y ¼ Yp;l;l* and B ¼ Bp;l;l* : Since Y is ðw1 ; w2 Þ-spherical, it must be fixed by g/gy ; according to our hypothesis. Thus Bðf2 ; f1 Þ ¼ Yðfˇ1 * f2 Þ ¼ Yðf2y * f1y Þ ¼ Bðf1y ; f2y Þ: It follows that the map f /f y exchanges the left and right radicals of B: Therefore, the left radical determines the right radical and vice versa. Consequently, according to the remarks in Section 2, each of the spaces Cl and Cl* determines the other. But since the choices of l and l* were made independently of each other, it must be the case that, up to scalars, these choices are unique. In other words, HomZH1 ðp; w1 Þ ¼ Cl* and HomZH2 ðp; * w1 2 Þ ¼ Cl: & Note that Lemma 3 implies that, if the stated hypotheses are satisfied, then, up to scalars, there is a unique ðw1 ; w2 Þ-spherical matrix coefficient. However, if H1 and H2 are chosen to be sufficiently small then there will be many ðw1 ; w2 Þ-spherical matrix coefficients and therefore there cannot exist a suitable choice of s: On the other hand, if H1 and H2 are too large then the spaces HomZH1 ðp; w1 Þ and HomZH2 ðp; * w1 2 Þ will be zero and thus there cannot exist any ðw1 ; w2 Þ-distinguished representations. Therefore, the hypotheses of Lemma 3 force H1 and H2 to be neither too small nor too large. Since the antiautomorphism g/gy maps ðw1 ; w2 Þ-spherical matrix coefficients to y y ðw2 ; w1 Þ-spherical matrix coefficients, one usually applies the Gelfand/Kazhdan Lemma when H2 ¼ yðH1 Þ and w2 ¼ wy 1 : One of the most well-known examples is the

258

J. Hakim / Journal of Number Theory 100 (2003) 251–269

example of Whittaker models, where G ¼ GLðn; F Þ; where F is a nonarchimedean field, H1 is the subgroup of G consisting of upper triangular unipotent matrices, H2 ¼ t H1 ; yðgÞ ¼ t g1 and w1 and w2 are nondegenerate characters. This example makes it clear that it is not necessarily the case that the double coset space H1 \G=H2 contains a double coset which is open in G: We also remark that it is sometimes possible to shrink or enlarge H1 without changing HomZH1 ðp; w1 Þ: Such examples make it clear that it will not always be the case that yðH1 Þ ¼ H2 : Most statements of the Gelfand/Kazhdan Lemma use the hypothesis that every biH-invariant distribution on G is invariant under g/gy : However, general properties which apply to all bi-H-invariant distributions on G can be difficult to obtain. What is especially disappointing is that the properties one might expect, based on Harish-Chandra’s theory of admissible distributions on G; or the properties one might desire for applications rarely hold in the generality of all symmetric spaces H\G over a p-adic field F : The extent to which Harish-Chandra’s theory fails to cleanly extend to symmetric spaces H\G is exhibited in great detail in the work of Rader and Rallis [13]. Some of the issues involved are illustrated in the following example. Consider the   double coset O ¼ H 01 11 H in Example 2 from Section 8 below. One could try to define a bi-H-invariant distribution TO on G such that TO ðf Þ represents the average value of f on the double coset O and it would be natural to call TO ðf Þ the orbital integral of f on O: Such a distribution would then be an example of a bi-H-invariant distribution on G which would presumably not be s-invariant. It would also seem natural to conjecture, based on Harish-Chandra’s theory on G; that orbital integrals are ‘‘dense’’ in the family of all bi-H-invariant distributions on G in the sense that if T is a bi-H-invariant distribution on G; the value of Tðf Þ depends on fTO ðf Þg; as O varies over all double cosets O: Unfortunately, Rader and Rallis have shown that, in general, there exist double cosets O which do not admit an invariant measure or, in other words, TO does not exist and, in addition, the collection of those orbital integrals which do exist is not generally dense in the family of all bi-H-invariant distributions on G: 5. Generalized Schur orthogonality Choose a basis of neighborhoods of the identity K1 *K2 *? in G consisting of open, compact subgroups. For each such subgroup Kn ; let en be the characteristic function of Kn in G multiplied by the appropriate positive constant so that en * en ¼ en : Fix a quasicharacter o of Z: For integration on Z and G=Z; we fix a Haar measure on Z and use the corresponding quotient measure on G=Z: In fact, we choose the measure on Z so that the Schur orthogonality relation reads as follows: Z /v; pðgÞ* ˜ dg ¼ dðpÞ1 /w; v*S/v; wS; ˜ * vS/pðgÞw; wS G=Z

where v; wAV ; v*; wA ˜ V˜ and dðpÞ is the formal degree of p:

J. Hakim / Journal of Number Theory 100 (2003) 251–269

259

The map which sends f ACcN ðGÞ to the function Z f ðgzÞoðzÞ dz f 0 ðgÞ ¼ Z

has image CcN ðG; o1 Þ; the space of smooth functions x on G which transform according to xðzgÞ ¼ oðzÞ1 xðgÞ; for zAZ and gAG; and which have compact support modulo Z: Note that ðen * f * en Þ0 ¼ en * f 0 * en ; where the convolutions are taken on G; not G=Z: We define CcN ðG; oÞ similarly. In this section, it is convenient to view the generalized matrix coefficients Yp;l;l* of p as linear functionals on * in C N ðG; o1 Þ: We therefore abuse notations by writing Y * ðf 0 Þ or /pðf 0 Þl; lS c

p;l;l

* place of Yp;l;l* ðf Þ ¼ /pðf Þl; lS: The following elementary result is a more refined version of a similar result in [7]. Lemma 4. Assume ðp; V Þ is an irreducible, supercuspidal representation of G with n * ˜ Fix vAV ; v*AV; ˜ lAV˜ n ; lAV contragredient ðp; and let x be the matrix coefficient * VÞ: of p* defined by xðgÞ ¼ /v; pðgÞ* * vS: If n is chosen to be large enough so that both v and v* are Kn -fixed, and thus x is bi-Kn -invariant, then * * ¼ /pðxÞln ; l* n S ¼ dðpÞ1 /l; v*S/v; lS; /pðxÞl; lS * where ln AV and l* n AV˜ are defined by ln ¼ pðen Þl and l* n ¼ pðe * n Þl: Proof. Since x is bi-Kn -invariant, it follows that * ¼ /pðen en Þl; lS * ¼ /pðxÞln ; l* n S: /pðxÞl; lS * The Schur orthogonality relations now imply that * /pðxÞln ; l* n S ¼ dðpÞ1 /ln ; v*S/v; l* n S ¼ dðpÞ1 /l; v*S/v; lS; since Yn ðgÞ ¼ /pðgÞln ; l* n S is a matrix coefficient (in the usual sense).

&

6. The Rader/Rallis/Weyl integration formula Let F be a nonarchimedean field of characteristic zero with algebraic closure F and let G ¼ GalðF=F Þ: Fix a connected, reductive group G defined over F : (We usually use boldface letters for varieties defined over F and the corresponding nonboldface letters for the F -rational points. In addition, we tend not to distinguish between G and GðFÞ; for example, xAG means xAGðFÞ:) Assume we are given an automorphism y of G of order two which is defined over F and let Gy be the group of fixed points of y: Note that we have an isomorphism m : Gy \G-G> y of

260

J. Hakim / Journal of Number Theory 100 (2003) 251–269

affine varieties given by mðgÞ ¼ g1 yðgÞ; where G> y is the Zariski identity component 1 of fgAG : yðgÞ ¼ g g (see [14]). Assume we have fixed a subgroup H of Gy which is defined over F and which contains the identity component Gy 1 of Gy : Let S ¼ H\G: Following [13], we refer to either the triple ðG; H; yÞ or the space S ¼ H\G as a symmetric space over F : An element gAG is said to be y-regular if mðgÞ is a semisimple regular element of G: The set of y-regular elements in G is denoted G0 : A torus in G is said to be y-split if 0 > it is contained in G> y : If gAG then the centralizer of mðgÞ in Gy is a maximal y-split torus. If gAG then pg ðtÞ ¼ detðt þ 1  AdðmðgÞÞÞ is a polynomial in t: We denote by Dy ðgÞ the coefficient of the term in pg ðtÞ of lowest degree which does not vanish identically for all g: The set G 0 is precisely the set of those gAG for which Dy ðgÞ is nonzero. A Cartan subset of G is the set T of F -rational points of a variety T ¼ xTr ; where Tr is a maximal y-split torus in G which is defined over F and xAHTr -G: In this case, a measure on T obtained from a Haar measure on Tr via the map a/xa will be referred to as an invariant measure on T: We remark that if T is given then Tr may be recovered, without knowing x; by observing that Tr is the centralizer of mðTÞ in G> y : The definition of ‘‘Cartan subset’’ has a left/right symmetry. To be more specific, with the notations of the previous paragraph, we let Tc ¼ xTr x1 : Then Tc is a maximal y-split torus which is defined over F and T ¼ Tc x; where xATc H-G: If in G> mðgÞ ¼ gyðgÞ1 ¼ mðg1 Þ; then Tc is the centralizer of mðTÞ * * y : We stress that since T is not necessarily a group it is not necessarily the case that mðTÞ ¼ mðTÞ: * Two Cartan subsets T1 and T2 of G are said to be equivalent if HT1 H ¼ HT2 H: According to Theorem 3.4(1) of [13], the y-regular set G0 is a disjoint union of the sets HT 0 H; where T 0 ¼ T-G 0 and T ranges over a set of representatives for the equivalence classes of Cartan subsets of G: Using Galois cohomology, it is also shown in [13] that there are only finitely many equivalence classes of Cartan subsets of G: If x is a suitably nice function on G and T is a Cartan subset of G then we may define the (unnormalized) orbital integral of x on T 0 by Z Z FY ðtÞ ¼ xðh1 th2 Þ dh1 dh2 x ZH ðTr Þ\H ZH \H Z Z ¼ xðh1 th2 Þ dh2 dh1 ; H=ZH ðTc Þ

ZH \H

where ZH ¼ Z-H and if V is a subgroup of G then ZH ðVÞ denotes the centralizer of V in H: We now state a symmetric space version of the Weyl integration formula based on Theorem 3.4 [13]. In the statement, we are summing over a set T of representatives > for the equivalence classes of Cartan subsets of G: We also let Z> y ¼ Z-Gy : We will not state here the definition of the positive constants wT ; since it is extremely technical and irrelevant for our purposes. We refer the reader to [13] for more details.

J. Hakim / Journal of Number Theory 100 (2003) 251–269

261

Lemma 5. For suitably normalized invariant measures, the integral formula Z

jðgÞ dg ¼ G=Z

X 1 Z jDy ðtÞj1=2 FTj ðtÞ dt w 0 =Z > T T TAT y

holds for all jACcN ðG=ZÞ: It should be mentioned that the statements of the Weyl integration formula in [13] are for functions in CcN ðGÞ: To obtain the above statement, one does not simply replace G by G=Z in the formulas of Rader and Rallis. Instead, on the left-hand side of the formula, one may realize j as an average of the Z-translates of some fACcN ðGÞ: On the right-hand side, one uses the fact that Z is nearly the product of ZH with Zy> in the following sense. Suppose zAZ: Then z2 ¼ ðzyðzÞÞðzyðzÞ1 Þ; where zyðzÞAZH and zyðzÞ1 AZy> : Therefore, Z 2 CZH Zy> CZ: Since Z=Z 2 is compact, the same is true of Z=ðZH Zy> Þ: Therefore, the average over Z on the left-hand side can be broken down into an average over ZH ; Zy> and Z=ðZH Zy> Þ: In this way, making the necessary adjustments to the invariant measures, one obtains the formula in Lemma 5.

7. Smoothness of bi-H-invariant distributions The following result appears in [13] in the case w1 ¼ w2 ¼ 1: Rader and Rallis’s proof has its origins in [10], where Howe establishes the smoothness of the character of an irreducible, admissible representation of G on the (usual) regular set. It also draws on [6] which treats an especially well-behaved class of symmetric spaces. The main ingredients in our proof are taken from [13], though the structure of our argument follows [6,10]. Lemma 6. Suppose ðG; H; yÞ is a symmetric space over F : If p is an irreducible, * admissible representation of G and lAHom * w1 ZH ðp; w1 Þ; lAHomZH ðp; 2 Þ then there 0 exists a smooth function Fp;l;l* on the y-regular set G such that Yp;l;l* ðf Þ ¼

Z G

f ðgÞFp;l;l* ðgÞ dg

for all f ACcN ðG0 Þ: Fix an open, compact subgroup K of G: After replacing K by K-yðKÞ; we may assume that K is y-stable. We may also assume that K is small enough so that both w1 and w2 are trivial on K: In general, if J is a compact group we let Jˆ denote a set of representatives for the collection of all irreducible, unitary representations of J:

J. Hakim / Journal of Number Theory 100 (2003) 251–269

262

ˆ let Pd denote the projection When dAK; I * dk Pd ðmÞ ¼ degðdÞ tr dðkÞpðkÞm K

H of V onto its d-isotypic component V ðdÞ: Here, K designates integration with respect to the Haar measure on K of total mass one. Note that V ðdÞ is also the d* such ˜ dÞ isotypic component of V : There is a corresponding projection Pd* : V n -Vð n n for all mAV˜ and mAV : that /Pd ðmÞ; mS * ¼ /m; Pd* ðmÞS; * * * ˆ Given dAK; and ðp; l; lÞ as in the hypothesis of the lemma, we define a (uniformly) smooth function Yd by ˜n

* ¼ /pðgÞl; P * ðlÞS: * Yd ðgÞ ¼ /Pd ðpðgÞlÞ; lS d We observe that Yd is left Km -invariant, where m ¼ condðdÞ is the conductor of d; that is, m is the smallest positive integer such that Km Cker d: Let NK ðgÞ ¼ supfm: Yd ðkgÞa0; for some kAK and d with condðdÞ ¼ mg and GK0 ¼ fgAG: NK ðgÞoNg: Thus NK is locally constant on GK0 and, moreover, GK0 is an open subset of G: Suppose gAGK0 : Choose f ACcN ðGÞ which has support in Kg: Then X Yðf Þ ¼ Yd ðf Þ: dAKˆ condðdÞpNK ðgÞ

If we define YðxÞ ¼

X

Yd ðxÞ

dAKˆ condðdÞpNK ðgÞ

R when xAKg; then we have Yðf Þ ¼ f ðxÞYðxÞ dx: In particular, Y is represented by a smooth function on GK0 : Therefore, Lemma 6 follows immediately from the following result: Lemma 7. Given gAG 0 ; there exists a choice of K so that NK ðgÞoN: Consequently, Y is represented by a smooth function on Kg: Proof. We will sketch a proof which, in essence, is in [13]. Our main purpose is to point out where the characters w1 and w2 enter into the proof and to provide the unfamiliar reader with a hint of the main ideas in the rather complicated argument. Replacing G by G=Z; we may as well assume G is semisimple. Let g and h denote the Lie algebras of G and H; respectively. Fix gAG0 and dAKˆ such that Yd ðgÞa0 and

J. Hakim / Journal of Number Theory 100 (2003) 251–269

263

let m ¼ condðdÞ: To establish our claim, it suffices to show that m is bounded by a number which depends on g; but not d: The next step is to use the Kirillov ‘‘orbit method’’ to give a geometric reformulation of the problem. Let t be the centralizer of ggy in s and let m be the centralizer of t in h: Let kðx; yÞ ¼ trðad x ad yÞ be the Killing form on g: According to a result of Vust, m þ t is a Levi subalgebra of g and if q denotes the orthogonal complement of m þ t with respect to k then the y-regularity of g implies that 1  Adðggy Þ defines a linear automorphism of q: Fix a lattice L in g which is stable under y and Ad K and assume L is small enough so that exponentiation maps L homeomorphically onto an open subgroup K0 of L: Next, we fix a prime element $ of F and define lattices Li ¼ $ i L and, when iX1; we define open, compact subgroups of G by Ki ¼ exp Li : If n ¼ Iðm þ 1Þ=2m then Kn =Km is an abelian group. Given a character w of Kn =Km ; we may define Yw by analogy with Yd : We may choose w such that Yw ðgÞa0: Now fix a character c of F : Whenever S is a subset of g we take S> ¼ fxAg: cðkðx; yÞÞ ¼ 1; for all yASg: If q is the order of the residue field of F ; we may define a norm on g by jjxjj ¼ minfqi : xA$ i L> g: Associated to w is a unique coset cw ¼ x þ L> such that wðexp yÞ ¼ cðkð$ n x; yÞÞ for all yALn : The elements of this coset have norm qmn and, since qðm1Þ=2 pqmn ; proving that m is bounded is equivalent to showing that jjzjj is bounded for any element zAcw : It is elementary to verify the relations Yw ðh1 gh2 Þ ¼ w1 ðh1 Þw2 ðh2 ÞYw ðgÞ Yw ðh1 gh2 Þ ¼ wðh1 Þwðgh2 g1 ÞYw ðgÞ

for all h1 AKn -H and h2 AH;

for all h1 AKn -H and h2 Ag1 Kn g-H:

We now assume that K is small enough so that w1 is trivial on K-H and w2 is trivial on g1 Kg-H: It then follows that we may choose elements zAcw -h> and z0 Acw -gh> g1 : Since z  z0 AL> ; we know that jjz  z0 jj is bounded. If zq is the projection of z onto q then one may bound jjzq jj as follows. First, we use the identity ð1  Adðggy ÞÞzq ¼ ð1  Adðggy ÞÞz ¼ AdðgÞðAdðg1 Þðz  z0 Þ þ yðAdðg1 Þðz  z0 ÞÞÞ

J. Hakim / Journal of Number Theory 100 (2003) 251–269

264

to obtain a bound for jjð1  Adðggy ÞÞzq jj: Next, since ð1  Adðggy ÞÞjq is invertible, there is a constant C such that jjð1  Adðggy ÞÞ1 xjjpCjjxjj; for all xAq; and, in particular, we have jjzq jjpCjjð1  Adðggy ÞÞzq jj: Therefore, it suffices to show that there exists a constant C 0 such that jjzjjpC 0 jjzq jj: This may be done exactly as in Lemma 5.5 of [13]. The key fact needed is that cw contains a nilpotent element and this is shown in Lemma 5.7 of [13]. & It would be desirable to generalize Lemma 6 to cover other examples in which we are given two closed subgroups H1 and H2 of G with H2 ¼ yðH1 Þ and linear forms t 1 * lAHom * w1 ZH1 ðp; w1 Þ; lAHomZH2 ðp; 2 Þ: For example, when G ¼ GL2 ; yðgÞ ¼ g ; * H1 is the group of upper triangular unipotent matrices and l and l are Whittaker functionals, then Baruch [1] has shown that the associated ‘‘Bessel distributions’’ are locally integrable and smooth on an open, dense set in G: (Actually, the distributions   studied in [1] are translates of the distributions Yp;l;l* by the matrix 01 10 :) If one tries to generalize the above argument, one can still show that zAcw -h> and 1 > 1 0 z Acw -gh2 g : However, the identity used above to bound jjzq jj breaks down. Returning to our primary objective, we now give a simple result about orbital integrals which we will need in the proof of our main result:

Lemma 8. If jACcN ðG=ZÞ and T is a Cartan subset of G then FTj is a smooth function on the set T 0 of regular elements in T: Proof. Fix j; T and an open compact set O in T 0 : Let A be the image of HOH in G=Z and let B denote the image of O in ZH\G=H: It suffices to show that FTj is smooth on B; since O was chosen arbitrarily. Since the restriction of FTj to B only depends on the restriction of j to A; we may as well assume that j has support in A: Our assertion now follows easily from Harish-Chandra’s submersion principle, as stated on p. 113 of [13], applied to the natural submersion m : A-B: Indeed, for suitably chosen measures the map m * : CcN ðAÞ-CcN ðBÞ of [13] coincides with j/FTj :

&

8. Supercuspidal representations We now turn to the main result of the paper. In order to use the theory of Rader and Rallis, we need to assume we are given a symmetric space ðG; H; yÞ over F : Since this theory has only been fully developed for bi-H-invariant distributions, rather than more general ðw1 ; w2 Þ-spherical distributions defined with respect to subgroups

J. Hakim / Journal of Number Theory 100 (2003) 251–269

265

H1 and H2 ; we are limited to treating those representations p of G which are Hdistinguished in the sense that HomH ðp; 1Þ and HomH ðp; * 1Þ are nonzero. Theorem. Let ðG; H; yÞ be a symmetric space over local nonarchimedean field of characteristic zero. Assume y0 is an automorphism of G of order two such that y0 ðHÞ ¼ 0 H and ZHty H ¼ ZHtH for almost every y-regular double coset ZHtH: Then if p is an irreducible, supercuspidal, H-distinguished representation of G then p* must be 0 equivalent to py and the spaces HomH ðp; 1Þ and HomH ðp; * 1Þ must have dimension one. 0

Recall that gy denotes y0 ðg1 Þ: Before the proof of this theorem, we observe that 0 the requirement that ZHty H ¼ ZHtH for ‘‘almost every’’ y-regular double coset ZHtH may be interpreted in more than one way. In the proof, we need to use the 0 assumption that for each TAT the set of all tAT 0 for which ZHty HaZHtH has measure zero image in Zy> \T: On the other hand, if there is some T for which the latter set does not have measure zero, then the union of the problematic double cosets ZHtH will have nonzero measure in either G or G=Z: Therefore, it suffices to 0 assume that the set of all gAG for which ZHgy HaZHgH has measure zero in G or G=Z: The latter formulations are simpler to use in applications. Proof. Let ðp; V Þ be an irreducible, supercuspidal representation of G with * ˜ Assume lAHom contragredient ðp; * VÞ: * 1Þ are nonzero. H ðp; 1Þ and lAHomH ðp; ˜ Fix vectors vAV and v*AV and define xðgÞ ¼ /v; pðgÞ* * vS: For each positive integer n; let Yn ðgÞ ¼ /pðgÞln ; l* n S and take jn ðgÞ ¼ Yn ðgÞxðgÞ and observe that jn ACcN ðG=ZÞ: Recall from Lemma 4 that if n is large enough so that v and v* are Kn -fixed, we have Z * ¼ /pðxÞln ; l* n S ¼ dðpÞ1 /l; v*S/v; lS: * jn ðgÞ dg ¼ /pðxÞl; lS G=Z

Applying Lemma 5, this becomes X 1 Z jDy ðtÞj1=2 FTjn ðtÞ dt; > w 0 TAT T T =Zy

* ¼ dðpÞ /l; v*S/v; lS 1

where FTjn ðtÞ

¼

Z ZH ðTr Þ\H

Z

Yn ðh1 th2 Þxðh1 th2 Þ dh1 dh2 :

ZH \H

Choose compact subsets O1 CZH \H and O2;T CZH ðTr Þ\H and OT CðZ-Gy> Þ\T; such that if tAT 0 ; h1 ; h2 AH and xðh1 th2 Þa0 then tAOT ; h1 AO1 and h2 AO2;T : For simplicity, let us abbreviate the function Fp;l;l* which represents the distribution Y on G 0 by setting YðgÞ ¼ Fp;l;l* ðgÞ when gAG 0 and YðgÞ ¼ 0 otherwise. Since Y is smooth

266

J. Hakim / Journal of Number Theory 100 (2003) 251–269

on G 0 ; we may choose n to be large enough so that Yðk1 h1 th2 k2 Þ ¼ YðtÞ for all TAT; tAOT -T 0 ; h1 AO1 ; h2 AO2;T and k1 ; k2 AKn : For such n; we have Yn ðh1 th2 Þ ¼ Yðh1 th2 Þ ¼ YðtÞ ¼ Yn ðtÞ for all TAT; tAOT -T 0 ; h1 AO1 and h2 AO2;T : This yields the identity FTjn ðtÞ ¼ YðtÞFTx ðtÞ; which holds when TAT; tAOT -T 0 and hence X 1 Z * ¼ dðpÞ1 /l; v*S/v; lS jDy ðtÞj1=2 YðtÞFTx ðtÞ dt: > w 0 T Zy \T TAT 0

Now let s be the antiautomorphism of G defined by sðgÞ ¼ gy : Then s must fix 0 every Haar measure on G=Z: Thus, if jsn is defined by jsn ðgÞ ¼ jn ðsðgÞÞ ¼ jn ðgy Þ; we must have Z Z jn ðgÞ dg ¼ jsn ðgÞ dg: G=Z

G=Z

0

We now use the assumption that ZHty H ¼ ZHtH for almost all y-regular double cosets ZHtH to deduce that, given TAT and tAT 0 ; there will almost always exist h1 ; h2 AZH such that sðtÞ ¼ h1 th2 and thus YðsðtÞÞ ¼ Yðh1 th2 Þ ¼ YðtÞ: Computing, as before, we obtain Z X 1 Z s j ðgÞ dg ¼ jDy ðtÞj1=2 YðtÞFTxs ðtÞ dt: > w 0 G=Z TAT T Zy \T 0

Note that xs is a matrix coefficient of py which is not orthogonal to the matrix 0 coefficient Yn of p: Thus p must be equivalent to p* y ; according to the Schur orthogonality relations. We may therefore choose an intertwining operator I : V -V˜ 0 such that IðpðgÞwÞ ¼ pðg * y ÞIðwÞ; for all gAG and wAW : Next, we observe that 0 0 and hence js ðgÞ ¼ /pðgy Þv; v*S ¼ /I 1 ð*vÞ; Iðpðgy ÞvÞS ¼ /I 1 ð*vÞ; pðgÞIðvÞS * Z * ¼ js ðgÞ dg: dðpÞ1 /l; IðvÞS/I 1 ð*vÞ; lS G=Z

Therefore, we have shown that * * ¼ /l; IðvÞS/I 1 ð*vÞ; lS: /l; v*S/v; lS It follows from this identity that v lies in the kernel of l* if and only if IðvÞ lies in the kernel of l: Therefore, the kernel of l* determines the kernel of l and, similarly, the

J. Hakim / Journal of Number Theory 100 (2003) 251–269

267

* But to know the kernel of a nonzero linear kernel of l determines the kernel of l: form is equivalent to knowing the one-dimensional space spanned by the linear form. Therefore, the spaces Cl* and Cl determine each other. Since l and l* were chosen independently, it must be the case that Cl* ¼ HomH ðp; 1Þ and Cl ¼ HomH ðp; * 1Þ: The theorem follows. &

9. Examples This section provides three examples which indicate both the applications and the limitations of our techniques, in their current form. The first example exhibits a situation in which our theorem applies, however, it does not offer any significant advantage over the Gelfand–Kazhdan Lemma. Next, we encounter an example where our methods apply and the Gelfand–Kazhdan Lemma does not. In the third example, neither our methods nor the Gelfand–Kazhdan Lemma apply. In the latter case, we point out where the failure occurs. 0

Example 1. The most ideal examples are those for which Hgy H ¼ HgH; for all double cosets in H\G=H: The following is an example of this type. Let E be a quadratic extension of a field F which is a finite extension of the field Qp of p-adic numbers, for some prime p: Let G ¼ GLðn; EÞ and H ¼ GLðn; F Þ; for some positive integer n; and let yðgÞ ¼ g; % where g% is the matrix obtained by applying the nontrivial Galois automorphism of E=F to the matrix entries of g: This a symmetric space over F : It is elementary to show that H g% 1 H ¼ HgH; for all gAG (see [3]). Thus we can apply our theorem with y0 ¼ y to conclude that if p is an irreducible, supercuspidal representation of G and HomH ðp; 1Þ and HomH ðp; * 1Þ are nonzero then these spaces must have dimension one and p must be equivalent to the representation g/pð * gÞ % acting on the space of p: This is essentially the same as the supercuspidal case of Propositions 11 and 12 in [3]. Another possibility is that the union of the double cosets HgH which are not fixed 0 by g/gy is nonempty, but has measure zero. In fact, even in Example 1, it is immediate that the double coset symmetry property holds almost everywhere. An additional argument, which is not too difficult, is needed to show that the symmetry holds everywhere. The following example is considered in [11]. By conducting a detailed study of the bi-H-invariant distributions on G; Jacquet and Rallis are able to show that dim HomH ðp; 1Þp1 for every irreducible admissible representation p of G: Our methods provide a simple alternate method for proving the supercuspidal case of their result. Example 2. Let F be a finite extension of Qp ; for some p; and suppose n is a positive   even integer. Let G ¼ GLðn; F Þ and write elements of G as block matrices ca db ;

J. Hakim / Journal of Number Theory 100 (2003) 251–269

268

where the blocks  lie in glðn=2; F Þ: Let H be the subgroup of G consisting of matrices  0 of the form x0 y0 ; with x; yAGLðn=2; F Þ: Equivalently, this is centralizer of 10 1 or 1 0  the fixed points of the automorphism y ¼ Ad 0 1 : The matrix identity a þ bd 1 c 0

0 ca1 b þ d

!

a c

b d

!1

a 0

0 d

! ¼

a b c d

!

is almost always valid. So the double cosets which satisfy Hg1 H ¼ HgH have full measure. However, for all v; w; x; yAGLðn=2; F Þ; we have v

0

0

w

!

0

1

1

1

!

x

0

0

y

! ¼

0

vy

wx

wy

! a

1

1

1

0

! ¼

0

1

1

1

!1

and thus there are always double cosets which are not fixed by g/g1 : In our theorem, we may take y0 ¼ 1 to conclude that if HomH ðp; 1Þ and HomH ðp; * 1Þ are nonzero then they both have dimension one and p is self-contragredient. Example 3. As in Example 1, let E be a quadratic extension of a field F which is a finite extension of the field Qp of p-adic numbers, for some odd prime p: Let G ¼ GLð3; EÞ and yðgÞ ¼ t g% 1 : Thus H ¼ Uð3; E=F Þ and H\G is a symmetric space over F : Based on [8], for example, one might expect that our results apply with y0 ¼ y: However, the set of double cosets HgH; where g is such that 0

b1 B mðgÞ ¼ @ 0

0 b2

0

0

1 0 C 0A b3

with b1 ANE=F ðE  Þ and b2 ; b3 AF   NE=F ðE  Þ; has nonzero measure in H\G=H and none of these cosets is fixed by s: Thus our theorem does not apply. In [8], more elaborate techniques are used to prove dim HomH ðp; 1Þp1 for all irreducible, tame supercuspidal representations p in this case. Let G be the F -group obtained from GL3 by restriction of scalars from E to F : Then y0 is an automorphism of G which is 0 defined over F : It is easy to show that Hgy H ¼ HgH; for all gAG: We can define an equivalence relation on H\G=H by declaring that Hg1 HBHg2 H if Hg1 H ¼ Hg2 H: For historical reasons, relating to the theory of stable trace formulas, it seems natural to call this equivalence relation ‘‘stable equivalence’’. Any attempt to generalize our methods to account for examples such as the present one will likely involve a careful study of this phenomenon of stable equivalence. A related notion of stability over the residue field of F is a crucial ingredient in [12], which gives an alternate approach to [8] for (not necessarily tame) supercuspidal representations in the special case in which E=F is unramified.

J. Hakim / Journal of Number Theory 100 (2003) 251–269

269

Acknowledgments The author thanks Fiona Murnaghan for patiently reading various drafts of this paper and making valuable suggestions.

References [1] E.M. Baruch, On Bessel distributions for GL2 over a p-adic field, J. Number Theory 67 (1997) 190–202. [2] I.N. Bernshtein, A.V. Zelevinskii, Representations of the group GLðn; F Þ where F is a nonarchimedean local field, Uspekhi Mat. Nauk 31 (3) (1976) 5–70. [3] Y. Flicker, On distinguished representations, J. Reine Angew. Math. 418 (1991) 139–172. [4] I. Gelfand, D. Kazhdan, Representations of GLðn; KÞ where K is a local field, Lie Groups and Their Representations, Wiley, New York, 1975, pp. 95–118. [5] B. Gross, Some applications of Gelfand pairs to number theory, Bull. Amer. Math. Soc. 24 (2) (1991) 277–301. [6] J. Hakim, Admissible distributions on p-adic symmetric spaces, J. Reine Angew. Math. 455 (1994) 1–19. [7] J. Hakim, F. Murnaghan, Globalization of distinguished supercuspidal representations of GLðnÞ; Canad. Bull. Math. 45 (2) (2002) 220–230. [8] J. Hakim, F. Murnaghan, Tame supercuspidal representations of GLðnÞ distinguished by a unitary subgroup, Comput. Math. 133 (2) (2002) 199–244. [9] J. Hakim, F. Murnaghan, Two types of distinguished supercuspidal representations, Internat. Math. Res. Not. 35 (2002) 1857–1889. [10] R. Howe, Some qualitative results on the representation theory of GLn over a p-adic field, Pac. J. Math. 73 (1977) 479–538. [11] H. Jacquet, S. Rallis, Uniqueness of linear periods, Comput. Math. 102 (1996) 65–123. [12] D. Prasad, On a conjecture of Jacquet about distinguished representations of GLðnÞ; Duke Math. J. 109 (1) (2001) 67–78. [13] C. Rader, S. Rallis, Spherical characters on p-adic symmetric spaces, Amer. J. Math. 118 (1) (1996) 91–178. [14] R.W. Richardson, Orbits, invariants, and representations associated to involutions of reductive groups, J. London Math. Soc. 42 (2) (1990) 409–429.