- Email: [email protected]
Oscillating tableaux and nonintersecting lattice paths
ELSEVIER
Journal of Statistical Planning and Inference 54 11996) 75 85
journal of statistical planning and inference
Oscillating tableaux and nonintersecting lattice paths C. K r a t t e n t h a l e r * lnstitut fiir Mathematik der Universitiit Wien, Strudlhofgasse 4, A-1090 Wien, Austria
Received 15 February 1994; revised January 1995
Abstract
We give alternative proofs of determinantal formulas for certain up-down tableaux and down-up tableaux generating functions, which were first given in another paper by the author. These new proofs are based on interpreting up-down and down-up tableaux as certain families of nonintersecting lattice paths with three types of steps.
AMS classification: primary 05El0; secondary 05A15; 20C33; 22E45 Keywords: Oscillating tableaux; Nonintersecting lattice paths
1. Introduction and definitions
We provide proofs by nonintersecting lattice paths for two formulas in Krattenthaler, (1993, 1996) (see (1.1) and (1.2) below) about the generating functions for certain oscillating tableaux. However, before we are able to reproduce the formulas we have to recall some definitions. A partition is a sequence 2 = (21,)~2, --- ) of nonnegative integers which are in decreasing order, i.e. 21 ~>22 >7 -.. , where only finitely many entries differ from zero. Zeros in the "tail" of a partition are frequently omitted. For instance, (6, 4, 4, 1,0, 0 . . . . ) = (6, 4, 4,1,0) = (6, 4, 4, 1). Partitions can be viewed pictorially as left-j ustified arrays of cells where the number of cells in the ith row of the array is given by 21. This array of cells is called Ferrers diagram of the partition ,i. For example, the Ferrers diagram of (6, 4, 4, 1) is displayed in Fig. 1. The partition conjugate to a partition ,i is the partition 2' = (2'1, ,i~ . . . . ), where 21 is the number of cells in the ith column of )~. Equivalently, the Ferrers diagram of ,i' is the Ferrers diagram of ). reflected in its main diagonal. F o r example, the partition conjugate to (6,4,4, 1~ is (4, 3,3, 3, 1, 1). On the set of partitions a partial order ~_ is defined by: # _~ ,i if and
*E-mail: [email protected]. 0378-3758/96/$15.00 ~ 1996 Elsevier Science B.V. All rights reserved Pll S0378-3758(96)001 58-1
C. Krattenthaler /Journal of Statistical Planning and Inference 54 (1996) 75-85
76
only if Pl ~21 for all i. Equivalently,/~ ___2 if and only if the Ferrers diagram of p is contained in the Ferrers diagram of 2. A (generalized) up-down tableau from p to 2 (Gessel, 1993; Roby, 1991; Sagan, 1989; Sundaram, 1986) is a sequence T = (to, ,1 . . . . . r 2 , - 1, ,2,) of partitions (Ferrers diagrams), for some n, such that /t=T0
~,1
--~32----- "-" --~ ~ 2 , - 1
--~ ~ 2 n = ~ - ,
and ,~_ a and *i differ by a horizontal strip for each i = 1, 2 . . . . . 2n. A horizontal strip (cf. M a c d o n a l d , 1979), is an arrangement of cells with at most one cell in each column. The n u m b e r 2n is called the length of the u p - d o w n tableau T. Similarly, a (generalized) down-up tableau from I~ to 2 is a sequence T = (To, ,1, .--, , 2 , - 1, *2,) of partitions, for some n, such that ] ~ = T O ~ - ~ - T 1 ~-~-'2 ~
"'" ~--- *2n
1 ~--- *2n = ~ " ,
and ,~ a and ~ differ by a horizontal strip for each i = 1, 2 . . . . . 2n. Again, the n u m b e r 2n is called the length of the d o w n - u p tableau T. Examples for an u p - d o w n tableau and a d o w n - u p tableau from (2, 1, 1) to (2, 2, 1, 1) are given in Fig. 2. Both have length 4. We say that an u p - d o w n tableau (respectively, d o w n - u p tableau) T = (~o, ,1 . . . . , *2, 1, *2,) has at most r columns (respectively, rows) if each *i, i = 0, 1, . . . , 2n, has at most r columns (rows). So for example, the u p - d o w n tableau in Fig. 2 has at m o s t 5 (6, 7 .... ) columns and the d o w n - u p tableau in Fig. 2 has at most 4 (5, 6 .... ) rows. The weight w(T) of an u p - d o w n or d o w n - u p tableau T is defined by
w(r)
:=
, 211
.
.
.
Thus, the weight of the u p - d o w n tableau in Fig. 2 is given by Xl2 x ° x 4 x 4 = x 2 x2s, while the weight of the d o w n - u p tableau in Fig. 2 is given by x 2 Xl3 x2~ x 2 = x s x2a . Let us write x for the sequence of variables Xl, x2, ...
55Z]
Fig. 1.
up-down tableau
down-up tableau
Fig. 2.
C. Krattenthaler /Journal o f Statistical Planning and Inference 54 (1996) 75 85
77
Now we are able to state the formulas of Krattenthaler (1993, 1996). Let udl,~)~x(x) denote the generating function Y',r w ( T ) summed over all up-down tableaux T from /1 to 2 with at most r columns. Then for any partitions :~ = (at, :~r 1, .-., :q) and /~ = (fir,//r 1. . . . . ill) we have uu~,~,(x)=
% +~ t~, , + k ( x ) e k ( x ) - - ~e~,+.~+j~,+,+k(x)ek(x) l~s,t~r
,
k
(1.1) where era(x) denotes the elementary symmetric function of order m in the variables x:t, x2 .... (cf. Macdonald, 1979, p. 12). Note that we definitely want the entries of a n d / / t o be indexed in reverse order. Likewise, let du~')~x (x) denote the generating function y~r w ( T ) summed over all down-up tableaux T from # to 2 with at most r rows. Then for partitions :~ and [~ as above we have
ut,~(x)=
h,,+~ /,, _ , + k ( x ) h k ( x ) - l~s,t~r
+,,
,
k
(12) where hm(x) denotes the complete homogeneous symmetric function of order m in the variables xl, x2 .... (cf. Macdonald, 1979 p. 14). The significance of the formulas (1.1) and (1.2) is elaborated in Krattenthaler (1996). In particular, from these formulas enumeration results for standard oscillating tableaux with a bounded number of rows can be derived. These standard oscillating tableaux play an important role in Berele's decomposition formula (Berele, 1986) for powers of defining representations of the complex symplectic groups. Also, if :~or fl are rectangular shapes the determinant in (1.1) basically is an irreducible symplectic character. This led to a new combinatorial description of irreducible symplectic characters in terms of oscillating tableaux (Krattenthaler, 1996). The formulas (1.1) and (1.2) were proved in Krattenthaler (1996) by interpreting the right-hand side determinants of (1.1) and (1.2) as generating functions for certain families of two-rowed arrays and then setting up a bijection between these families of two-rowed arrays and up-down and down-up tableaux. These bijections were based on Robinson-Schentsted-type algorithms. In the next section we shall prove (1.1) and (1.2) by using nonintersecting lattice paths and variations of the so-called reflection principle (cf. e.g. Comtet, 1974, p. 22). The method of nonintersecting lattice paths, due to Gessel and Viennot (1985, 1989) (cf. also Stembridge, Section 1, 1990), has been successflllly utilized in numerous papers. (Brenti, 1993, 1995; Bressoud and Wei, 1992; Choi and Gouyou-Beauchamps, 1993; Conca, 1994, 1995; Conca and Herzog, 1994; DesainteCatherine and Viennot, 1986; Doran, 1993; Fulmek and Krattenthaler, 1995; Gessel and Krattenthaler, 1995; Greene, 1992; Herzog and Trung, 1992; Krattenthaler, 1993, 1996; Krattenthaler and Mohanty, 1992; Kulkarni; 1993a, b; Modak, 1992; Okada 1990, 1991; Remmel and Whitney, 1983; 1984; Sagan, 1992; Stembridge, 1990, 19!)5;
78
C. Krattenthaler / Journal o f Statistical Planning and Inference 54 (1996) 75-85
Sulanke, 1990; Ueno, 1991). So this paper might be considered as a further contribution to this list of papers (which very likely is incomplete). Combinations of the nonintersecting lattice path method and variants of the reflection principle earlier appeared in Fulmek and Krattenthaler (1995) and Krattenthaler (1993, 1995, 1996).
2. Nonintersecting lattice paths To begin with, we introduce some lattice path notation. Given a set S of steps we denote the set of all lattice paths from A to E with steps from the set S by ~ ( A --* E; S). We frequently shall write P: A ~ E to indicate that the lattice path P runs from A to E. A set of paths is said to be nonintersectin9 if no two paths of this set have a point in common. Otherwise, it is called intersecting. If to each step s e S a weight w(s) is assigned, the weight w (P) of a path P is defined to be the product of the weights of all its steps. The weight w(P) of a family P = (P1, ---, Pr) of paths is defined to be the product 1-I~'=1 w(Pi) of the weights of all the paths in the family. Given any weight function w defined on a set ~ ' , by the generating function G F ( ~ ' ; w) we mean
Yx ~ w(x). Proof of(1.1): Here we consider lattice paths in the plane integer lattice with steps from the set Sx = {(i,j) ~ (i,j + 1), (i, 2j - 2) ---,(i + 1, 2j - 1), (i, 2j - 1) --* (i - 1, 2j), where i, j run through the integers}. In words, the set of steps consists of vertical steps north, forward diagonal steps northeast from even height to odd height, and backward diagonal steps northwest from odd height to even height. In the sequel, when we say "lattice path" we always mean a lattice path with steps from the set $1. First, we set up a one-to-one correspondence between up-down tableaux from fl' to ~' with at most r columns and families P = ( P ~, P 2 , ... , Pr) of nonintersecting lattice paths where Pi runs from Ai = (fli + i, 0) to Ei = (~i + i, oo ) and never touches the y-axis, i = 1, 2 . . . . ,r. Let T = (Zo, Zl . . . . ,z2,) be an up-down tableau from fl' to e' with at most r columns. The bijection is defined in a way such that the ith path encodes the growth and shrink of the (r - i + 1)st column of T. To be more precise, starting with Ai = (fli + i, 0) for j = 1, 2, ... , n do the following procedure: If 272j_ 1 has one cell more in the (r - i + 1)st column than ~2j-2 then add a forward diagonal step to P~, otherwise a vertical step. If rzj has one cell less in the (r - i + 1)st column than Zzj- 1 then add a backward diagonal step, otherwise a vertical step. Finally, an infinite number of vertical steps is added to each path to connect to the end points (~i + i, oo ). Thus, given r = 5, the up-down tableau in Fig. 2 corresponds to the set of nonintersecting lattice paths displayed in Fig. 3. Note that in this example we have c~ = (4, 2, 0, 0, 0) and fl = (3,1, 0, 0, 0), i.e. ~ 1 = c ~ 2 = ~ 3 = 0 , ~ 4 = 2 , c~5=4,
/~
= / ~ 2 = / ~ 3 = 0 , / L = 1,/~5 = 3.
Clearly this operation sets up the claimed bijection. It becomes weight-preserving if we define the weight w(v) of a vertical step v to be 1, the weight w ( f ) of a forward diagonal step f:(i, 2 j - 2 ) ~ (i + 1, 2 j - 1) as well as the weight w(b) of a backward diagonal step b : (i, 2j - 1) ~ (i - 1, 2j) to be xj. Recall that this weight function w on
C. Krattenthaler / Journal of Statistical Planning and Inference 54 (1996) 75 85
E1 E2 E3
E4
79
E5
T
\
\ /F
e: P~/
/ A4
A1 A2 A3
J P~
As
Fig. 3.
the step set $1 induces a weight w on families of lattice paths as was explained at the beginning of this section. Next we give a c o m b i n a t o r i a l description of the determinant in (1.1). Because of G F ( ~ ( ( a , 0) --* (b, oQ); $1); w) = ~ eb-o+k(x)ek(x), k
we m a y write the determinant in (1.1) as det ( G F ( ~ ( A , ~ E,;Sa); w) - G F ( ~ ( 9 1 A , ~ E~; S~); w)),
(1.3)
where 91 is the reflection in the y-axis. By expanding the determinant, (1.3) can be written in the form E
sgnalL[qi
eel, .E{1,-1}'
i-1
[I i,.i--
1-[ G F ( ~ ( A , ( , ) ~ E i ; S 1 ) ; w ) i,tl,= 1
GF(~(91A~,(i)--+ Ei;S1);w), 1
or
( - 1) ll"ll sgn aw(P),
(I.4)
where Iqr/1] = ~2~= 1 qi. Of course, 9t o means the identity map. By (1.4) we have written the d e t e r m i n a n t in (1.1) as a generating function for families P = ( P 1 , P 2 . . . . , P,) of lattice paths, P~: 91" A ~(~)- . Ei, for some a e ~ r and q ~ {0, 1 }~. Let us call a the permutation associated to P and t/the reflection indicator associated to 1). In the sum (1.4) m a n y terms will cancel. We shall construct a sign-reversing (with respect to ( - 1)'q"I, sgn a) and weight-preserving (with respect to w) involution on the set of all families P = (P1, P2 . . . . . Pr) of lattice paths that were described above, which are either interesecting or contain a path that touches the y-axis. Suppose that this had been done. Then in the sum (1.4) only the terms which come from nonintersecting families of lattice paths that do not touch the y-axis will survive. In particular, none of
80
C. Krattenthaler/Journal of Statistical Planning and Inference 54 (1996) 75-85
the starting points of such a family P could be located to the left of the y-axis. Hence, the reflection indicator associated to P must equal (1, 1 . . . . ,1). Besides, because P is nonintersecting the location of the starting and end points of the paths of P implies that the permutation associated to P must be the identity permutation. But these families P of lattice paths exactly correspond to the up-down tableaux under consideration, as was shown above. This would establish (1.1). Now we define the involution, which we denote by O1. Let P = (P1, P2, ..., Pr) be a family of lattice paths, Pi: 9t"' A~i) --+ Ei, for some a e ~r and ~/~ {0, 1}', which is either intersecting or contains a path that touches the y-axis. First consider all meeting points of a path of P and the y-axis. Let M be the highest of these meeting points. Among the paths which go through M choose Pt with I being minimal. Note that the y-coordinate of M must be even. Now, to P[s initial portion up to M we apply the following modified reflection in the y-axis (see Fig. 4). Beginning from the starting point of PI up to M consider the steps of Pt in groups of two. There are four different cases. Replace two vertical steps by two vertical steps, a vertical step and a backward diagonal step by a forward diagonal step and a vertical step, a forward diagonal step and a vertical step by a vertical step and a backward diagonal step, a forward diagonal step and a backward diagonal step by a forward diagonal step and a backward diagonal step. What is obtained by this procedure is joined with the terminal portion of Pr beginning from M. In Fig. 4 the circled points indicate the grouping of the steps into groups of two that was described above. Let the image of Px under application of this modified reflection be denoted by P}. We then obtain the new family 01(P) = (P1 . . . . . P}, ... ,P~). Since the new starting point of P} is the reflection of the starting point of PI in the y-axis, the permutation associated to O1 (P) in this case is still ~r while the reflection indicator associated to O~(P) is given by If none of the paths of P meet the y-axis consider all meeting points between any two paths of P. Choose the highest meeting points, and among those the left-most.
El •pi
EI
•
•M~r•
iii Fig.4.
C. Krattenthaler /Journal of Statistical Planning and Inference 54 (1996) 75 85
81
Denote this meeting point by M. N o w we apply the usual Gessel Viennot interchanging procedure. To be more precise, a m o n g the pairs of paths which have M in common, choose (Pt, Par), with (I, J) being minimal with respect to the lexicographic order of pairs of integers. Next we decompose P, into Pp) and p]2), where -tPIl) is the portion of Pi going from 9t"' ~4~(t~ to M, and where p~2) is the portion of PI between M and Et. Analogously, Par is decomposed by M into p(al) and p(2).1• Then by concatenation form the new paths PI, == p(j1) p~2) and P's = k,~(1) I p~2). In other terms, the initial portions of Pt and Par up 1o their last meeting point are interchanged, the terminal portions are preserved. Thus, we obtain the new family O1 (P) = (P~ . . . . . P} . . . . . P ) , ... , P~). Obviously, since the lth and Jth starting point have been interchanged, the permutation associated to O1 ( P ) i s a (I J) ((I.1) is the transposition that interchanges I and J), and the reflection indicator associated to O~ (P) is (~/1. . . . . ns . . . . . nt, " ' " , ~ r ) " It is easy to see that in both cases, by application of O1 to O1 (P), one arrives at P again. Consequently, O~ is an involution. Besides, by construction it is weightpreserving with respect to w and sign-reversing with respect to ( - 1 ) I,~lsgn ~r. This completes the proof. Proof of(1.2): Here we consider lattice paths in the plane integer lattice with steps from the set S 2 = { ( i , j ) - + ( i , j + 1), (i, 2 j - 1 ) - - + ( i - 1 , 2 j - 1), (i, 2j)--+(i+ 1,2./), where i, j run through the integers }. In words, the set of steps consists of vertical steps north, backward horizontal steps (west) at odd height, and forward horizontal steps (east) at even height. In the sequel, when we say "lattice path" we always mean a lattice path with steps from the set $2. First, we set up a one-to-one correspondence between down-up tableaux.from fi to with at most r rows and.families P = ( P 1 , P2 . . . . . P~) of nonintersecting lattice paths
where Pi runs from Ai = (fli ~- i, O) to Ei = (9~i q- i, oC ) and never touches the y-a.x'is, i = 1, 2 . . . . , r. Let T = (z0, z~, ... , z2,) be a down-up tableau from fi to c~with at most r rows. The bijection is defined in a way such that the ith path encodes the growth and shrink of the (r - i + 1)st row of T. To be more precise, starting with A~ = ([4~ + i., 0) fi~r j = 1,2 . . . . . n do the following procedure: If r2j ~ has m cells less in the (r-i+ 1)st row than z2s 2 then add a vertical step followed by m backward horizontal steps to P~. If z2s has m cells more in the (r - i + 1)st row than z2j- ~ then add a vertical step followed by m forward horizontal steps to P~. Finally, an infinite number of vertical steps is added to each path to connect to the end points (~ + i, ~). Thus, given r = 4, the down-up tableau in Fig. 2 corresponds to the set of nonintersecting lattice paths displayed in Fig. 5. Note that in this example, we have cq = 1, 0', 2 = l , (~3 = 2 , ~4- = 2, f l l = 0 , f12 = 1, f13 = 1, f14 = 2. Clearly this operation sets up the claimed bijection. It becomes weight-preserving if we define the weight w(v) of a vertical step v to be 1, the weight w ( f ) of a backward horizontal step f:(i, 2 j - 1)--+ ( i - 1 , 2 . ] - 1) as well as the weight w(b) of a forward horizontal step b : (i, 2j) ~ (i + 1, 2j) to be xj. Recall that this weight function w on the step set $2 induces a weight w on families of lattice paths as was explained at the beginning of this section.
C. Krattenthaler /Journal of Statistical Planning and Inference 54 (1996) 75-85
82
E1 E~ T
L
E3 E4 T 1
A~ A3
A4
Fig. 5.
Next we give a combinatorial description of the determinant in (1.2). Because of
GF(,_@((a, O) ~ (b, ~ ) ; $2); w) = ~ hb-.+k(X)hk(X), k
we may write the determinant in (1.2) as det (GF (~(A,
--+
E s ; S 2 ) ; w) - G F ( ~ ( g i A , --+ Es; $ 2 ) ; w ) ) ,
(1.5)
where !R is again the reflection in the y-axis. By expanding the determinant, (1.5) can be written in the form sgnafiq, a~, qe{1,-
i=l
I-[ GF(~(A~,)-+Ei;S2);w) i,qi=l
1}"
I~
GF(~(!RA~(o--+ Ei;S2);w),
l,th = - 1 or
Y,
( - 1) It"it sgn aw(P),
(1.6)
a e ~ , , r / e {0, 1}" P= (P~, ... , P,), Pi: ~q'A¢lo ~ Ei
where again IIr/J[ = ~ = 1 Yli' By (1.6) we have written also the determinant in (1.2) as a generating function for families P = (P1, P2, --., Pr) of lattice paths, Pi : 9~" A , , ) ---,El, for some cre ~, and e {0, 1}r. The only difference to (1.4) is that the set of steps now is $2 instead of $1. Again, let us call a the permutation associated to P and r/ the reflection indicator associated to P. Also here, we shall construct a sign-reversing (with respect to ( - 1 ) It'llsgna) and weight-preserving (with respect to w) involution on the set of all families P = (P1, P2 . . . . , Pr) of lattice paths that were described above, which are either intersecting or contain a path that touches the y-axis. Once this is done, similar arguments as those in the proof of (1.1) show that (1.2) would be established.
C, Krattenthaler /Journal of Statistical Planning and Inference 54 (1996) 75 -85
EI •
•
•
El
T~-,.Oal
•
M o,-..o,,-
•
-
',~3
M •
-
-
.
I.
- " " e
•
.
P - - - ~ . , - e . . . . e . . . . e
. . . . . .
: ....
e,
..
-o.
• • .o.,
o..
- .o,
rP~,
- • .o-
•
-,
.o
.
1
• .o
ml_.rA-
/.
.
.
.
.
Fig. 6.
Now we define the involution, which we denote by 0 2 . Let P = (P1, P2 . . . . . Pr) be a family of lattice paths, Pi:Ot"'A,,)-~Ei, for some a ~ ~ and r/~ {0, 1}", which is either intersecting or contains a path that touches the y-axis. As before, first consider all meeting points of any path P with the y-axis. Let M be the highest of these meeting points. Among the paths which go through M choose PI with I being minimal. Again, note that the y-coordinate of M must be even. Now, to Pt's initial portion up to M we apply the following modified reflection in the y-axis (see Fig. 6). Beginning from the starting point of Px up to M consider the portions of PI which lie between lhe horizontal lines y = 2j - 2 and y = 2j, portions of Pl on the line y = 2j - 2 excluded while portions on y = 2j included. If in this portion there is a vertical step followed by n backward horizontal steps and then a vertical step followed by m forward horizontal steps (note that the vertical steps are obligatory) replace it by a vertical step followed by m backward horizontal steps and a vertical step followed by n forward horizontal steps. What is obtained by this procedure is joined with the terminal portion of Pl beginning from M. In Fig. 6 the circled points indicate where the paths are cut into portions according to the above description. If none of the paths of P meet the y-axis consider all meeting points between any two paths of P. Choose the highest meeting points, and among those the left-most meeting point, and then apply the Gessel-Viennot interchanging procedure, as was described before in the proof of (1.1). Again it is easy to see that in both cases, by application of O2 to 02 (P), one arrives at P again. Consequently, O2 is a sign-reversing (with respect to ( - 1 ) I"l sgn c~) and weight-reversing (with respect to w) involution. This completes the proof,
References Berele, A. (1986). A Schensted-type correspondence for the symplectic group. J. Combin. Theory Ser. A 43, 320-328. Brenti, F. (1993). Determinants of super-Schur functions, lattice paths, and dotted plane partitions. Adv. Math. 98, 27-64.
84
C. Krattenthaler /Journal of Statistical Planning and Inference 54 (1996) 75-85
Brenti, F. (1995). Combinatorics and total positivity. J. Combin. Theory Ser. A 71, 175-218. Bressoud, D.M. and S.-Y. Wei (1992). Determinental formulae for complete symmetric functions. J. Combin. Theory Ser. A 60, 277 286. Choi, S.H. and D. Gouyou-Beauchamps (1993). Enumbration de tableaux de Young semi-standard. Theoret. Computer Science 117, 137-151. Comtet L. (1974). Advanced Combinatorics (D. Reidel, Dordrecht). Conca, A. (1994). Gr6bner bases of ideals of minors of a symmetric matrix. J. Algebra 166, 406 421. Conca, A. (1995). Ladder determinantal rings. J. Pure Appl. Algebra 98, 119-134. Conca, A. and J. Herzog (1994). On the Hilbert function of determinantal rings and their canonical module. Proc. Amer. Math. Soc. 122, 677-681. Desainte-Catherine, M. and G. Viennot (1986). Enumeration of certain Young tableaux with bounded height, In: G. Labelle, P. Leroux, Eds. Combinatoire enumbrative. Springer Berlin, 58-67. Doran, W.F. (1993). A connection between alternating sign matrices and totally symmetric self-complementary plane partitions. J. Combin. Theory Ser. A 64, 265-288. Fulmek, M. and C. Krattenthaler (1995). Lattice path proofs for determinant formulas for symplectic and orthogonal characters. J. Combin. Theory Ser. A (to appear). Gessel, I.M. (1993). Counting paths in Young's lattice. J. Statist. Plann. Inference 34, 125-134. Gessel, I.M. and C. Krattenthaler (1995). Cylindric partitions. Trans. Amer. Math. Soc. (to appear). Gessel, I.M. and G. Viennot (1985). Binomial determinants, paths, and hook length formulae. Adv. Math. 58, 300-321. Gessel, I.M. and G. Viennot (1989). Determinants, paths, and plane partitions, preprint. Greene, C. (1992). Proof of a conjecture on immanents of the Jacobi-Trudi matrix. Linear A19ebra Appl. 171, 65-79. Herzog, J and N.V. Trung (1992). Gr6bner bases and multiplicity of determinantal and Pfaffian ideals. Adv. Math. 96, 1 37. Krattenthaler, C. (1993). Non-crossing two-rowed arrays and summations for Schur functions. In: A. Barlotti, M. Delest, R. Pinzani, Eds., Proc. 5th Conf. on Formal Power Series and Algebraic Combinatorics, Florence, D.S.I., Universitfi di Firenze, 301-314. Krattenthaler, C. (1996). Non-crossing two-rowed arrays (in preparation). Krattenthaler, C. (1995). The major counting of nonintersecting lattice paths and generating functions for tableaux. Mere. Amer. Math. Soc. 115, no. 552, Providence, R.I. Krattenthaler, C. and S.G. Mohanty (1995). Counting tableaux with row and column bounds. Discrete Math. 139, 273 286. Kulkarni, D.M. (1993a). Counting of path-like objects and coefficients of Hilbert polynomial of a determinantal ideal, preprint. Kulkarni, D.M. (1993b). Counting of paths and coefficients of Hilbert polynomial of a determinantal ideal. preprint. Macdonald, I.G. (1979). Symmetric Functions and Hall Polynomials. Oxford University Press, New York/London. M odak, M.R. (1992). Combinatorial meaning of the coefficients of a Hilbert polynomial. Proc. Indian Acad. Sci. (Math. Sci) 102, 93-123. Okada, S. (1990) Partially strict shifted plane partitions. J. Combin. Theory Ser. A 53, 143 156. Okada, S. (1991). Lattice path method and characters of classical groups, preprint. Remmel, J.B. and R. Whitney (1983). A bijective proof of the hook formula for the number of column-strict tableaux with bounded entries, Eur. J. Combin. 4, 45 63. Remmel, J.B. and R. Whitney (1984). A bijective proof of the generating function for the number of reverse plane partitions via lattice paths. Linear Multilinear Algebra 16, 75-91. Roby, T.W. (1991). Applications and extensions of Fomin's generalization of the Robinson Schensted correspondence to differential posers. Ph.D. Thesis, MIT, Cambridge, Ma. Sagan, B.E. (1989) .The ubiquitous Young tableau In: D. Stanton, ed. Invariant theory and tableaux. The IMA Volumes in Math. And Its Appl., Vol. 19, Springer, New York, 262-298. Sagan, B.E. (1992). Log concave sequences of symmetric functions and analogs of the Jacobi Trudi determinants. Trans. Amer. Math. Soc. 329, 795-811.
(L Krattenthaler / Journal of Statistical Planning and Inference 54 (1996) 75 85
~':5
Stembridge, J.R. (1990). Nonintersecting paths, pfaffians and plane partitions. Adv. Math. 83, 96 131. Stembridge, J.R. (1995). The enumeration of totally symmetric plane partitions. Adv. in Math. 111, 227 245. Sulanke, R.A. (1990). A determinant for q-counting lattice paths. Discrete Math. 81, 91 96. Sundaram, S. (1986). On the combinatorics of representations of Sp(2n, Ct. Ph.D. Thesis, Massachusetts Institute of Technology. Ueno, K. (19911. Lattice path proof of the ribbon determinant formula for Sehur functions. Naqova Math. I. 124, 55-59.
Journal of Statistical Planning and Inference 54 11996) 75 85
journal of statistical planning and inference
Oscillating tableaux and nonintersecting lattice paths C. K r a t t e n t h a l e r * lnstitut fiir Mathematik der Universitiit Wien, Strudlhofgasse 4, A-1090 Wien, Austria
Received 15 February 1994; revised January 1995
Abstract
We give alternative proofs of determinantal formulas for certain up-down tableaux and down-up tableaux generating functions, which were first given in another paper by the author. These new proofs are based on interpreting up-down and down-up tableaux as certain families of nonintersecting lattice paths with three types of steps.
AMS classification: primary 05El0; secondary 05A15; 20C33; 22E45 Keywords: Oscillating tableaux; Nonintersecting lattice paths
1. Introduction and definitions
We provide proofs by nonintersecting lattice paths for two formulas in Krattenthaler, (1993, 1996) (see (1.1) and (1.2) below) about the generating functions for certain oscillating tableaux. However, before we are able to reproduce the formulas we have to recall some definitions. A partition is a sequence 2 = (21,)~2, --- ) of nonnegative integers which are in decreasing order, i.e. 21 ~>22 >7 -.. , where only finitely many entries differ from zero. Zeros in the "tail" of a partition are frequently omitted. For instance, (6, 4, 4, 1,0, 0 . . . . ) = (6, 4, 4,1,0) = (6, 4, 4, 1). Partitions can be viewed pictorially as left-j ustified arrays of cells where the number of cells in the ith row of the array is given by 21. This array of cells is called Ferrers diagram of the partition ,i. For example, the Ferrers diagram of (6, 4, 4, 1) is displayed in Fig. 1. The partition conjugate to a partition ,i is the partition 2' = (2'1, ,i~ . . . . ), where 21 is the number of cells in the ith column of )~. Equivalently, the Ferrers diagram of ,i' is the Ferrers diagram of ). reflected in its main diagonal. F o r example, the partition conjugate to (6,4,4, 1~ is (4, 3,3, 3, 1, 1). On the set of partitions a partial order ~_ is defined by: # _~ ,i if and
*E-mail: [email protected]. 0378-3758/96/$15.00 ~ 1996 Elsevier Science B.V. All rights reserved Pll S0378-3758(96)001 58-1
C. Krattenthaler /Journal of Statistical Planning and Inference 54 (1996) 75-85
76
only if Pl ~21 for all i. Equivalently,/~ ___2 if and only if the Ferrers diagram of p is contained in the Ferrers diagram of 2. A (generalized) up-down tableau from p to 2 (Gessel, 1993; Roby, 1991; Sagan, 1989; Sundaram, 1986) is a sequence T = (to, ,1 . . . . . r 2 , - 1, ,2,) of partitions (Ferrers diagrams), for some n, such that /t=T0
~,1
--~32----- "-" --~ ~ 2 , - 1
--~ ~ 2 n = ~ - ,
and ,~_ a and *i differ by a horizontal strip for each i = 1, 2 . . . . . 2n. A horizontal strip (cf. M a c d o n a l d , 1979), is an arrangement of cells with at most one cell in each column. The n u m b e r 2n is called the length of the u p - d o w n tableau T. Similarly, a (generalized) down-up tableau from I~ to 2 is a sequence T = (To, ,1, .--, , 2 , - 1, *2,) of partitions, for some n, such that ] ~ = T O ~ - ~ - T 1 ~-~-'2 ~
"'" ~--- *2n
1 ~--- *2n = ~ " ,
and ,~ a and ~ differ by a horizontal strip for each i = 1, 2 . . . . . 2n. Again, the n u m b e r 2n is called the length of the d o w n - u p tableau T. Examples for an u p - d o w n tableau and a d o w n - u p tableau from (2, 1, 1) to (2, 2, 1, 1) are given in Fig. 2. Both have length 4. We say that an u p - d o w n tableau (respectively, d o w n - u p tableau) T = (~o, ,1 . . . . , *2, 1, *2,) has at most r columns (respectively, rows) if each *i, i = 0, 1, . . . , 2n, has at most r columns (rows). So for example, the u p - d o w n tableau in Fig. 2 has at m o s t 5 (6, 7 .... ) columns and the d o w n - u p tableau in Fig. 2 has at most 4 (5, 6 .... ) rows. The weight w(T) of an u p - d o w n or d o w n - u p tableau T is defined by
w(r)
:=
, 211
.
.
.
Thus, the weight of the u p - d o w n tableau in Fig. 2 is given by Xl2 x ° x 4 x 4 = x 2 x2s, while the weight of the d o w n - u p tableau in Fig. 2 is given by x 2 Xl3 x2~ x 2 = x s x2a . Let us write x for the sequence of variables Xl, x2, ...
55Z]
Fig. 1.
up-down tableau
down-up tableau
Fig. 2.
C. Krattenthaler /Journal o f Statistical Planning and Inference 54 (1996) 75 85
77
Now we are able to state the formulas of Krattenthaler (1993, 1996). Let udl,~)~x(x) denote the generating function Y',r w ( T ) summed over all up-down tableaux T from /1 to 2 with at most r columns. Then for any partitions :~ = (at, :~r 1, .-., :q) and /~ = (fir,//r 1. . . . . ill) we have uu~,~,(x)=
% +~ t~, , + k ( x ) e k ( x ) - - ~e~,+.~+j~,+,+k(x)ek(x) l~s,t~r
,
k
(1.1) where era(x) denotes the elementary symmetric function of order m in the variables x:t, x2 .... (cf. Macdonald, 1979, p. 12). Note that we definitely want the entries of a n d / / t o be indexed in reverse order. Likewise, let du~')~x (x) denote the generating function y~r w ( T ) summed over all down-up tableaux T from # to 2 with at most r rows. Then for partitions :~ and [~ as above we have
ut,~(x)=
h,,+~ /,, _ , + k ( x ) h k ( x ) - l~s,t~r
+,,
,
k
(12) where hm(x) denotes the complete homogeneous symmetric function of order m in the variables xl, x2 .... (cf. Macdonald, 1979 p. 14). The significance of the formulas (1.1) and (1.2) is elaborated in Krattenthaler (1996). In particular, from these formulas enumeration results for standard oscillating tableaux with a bounded number of rows can be derived. These standard oscillating tableaux play an important role in Berele's decomposition formula (Berele, 1986) for powers of defining representations of the complex symplectic groups. Also, if :~or fl are rectangular shapes the determinant in (1.1) basically is an irreducible symplectic character. This led to a new combinatorial description of irreducible symplectic characters in terms of oscillating tableaux (Krattenthaler, 1996). The formulas (1.1) and (1.2) were proved in Krattenthaler (1996) by interpreting the right-hand side determinants of (1.1) and (1.2) as generating functions for certain families of two-rowed arrays and then setting up a bijection between these families of two-rowed arrays and up-down and down-up tableaux. These bijections were based on Robinson-Schentsted-type algorithms. In the next section we shall prove (1.1) and (1.2) by using nonintersecting lattice paths and variations of the so-called reflection principle (cf. e.g. Comtet, 1974, p. 22). The method of nonintersecting lattice paths, due to Gessel and Viennot (1985, 1989) (cf. also Stembridge, Section 1, 1990), has been successflllly utilized in numerous papers. (Brenti, 1993, 1995; Bressoud and Wei, 1992; Choi and Gouyou-Beauchamps, 1993; Conca, 1994, 1995; Conca and Herzog, 1994; DesainteCatherine and Viennot, 1986; Doran, 1993; Fulmek and Krattenthaler, 1995; Gessel and Krattenthaler, 1995; Greene, 1992; Herzog and Trung, 1992; Krattenthaler, 1993, 1996; Krattenthaler and Mohanty, 1992; Kulkarni; 1993a, b; Modak, 1992; Okada 1990, 1991; Remmel and Whitney, 1983; 1984; Sagan, 1992; Stembridge, 1990, 19!)5;
78
C. Krattenthaler / Journal o f Statistical Planning and Inference 54 (1996) 75-85
Sulanke, 1990; Ueno, 1991). So this paper might be considered as a further contribution to this list of papers (which very likely is incomplete). Combinations of the nonintersecting lattice path method and variants of the reflection principle earlier appeared in Fulmek and Krattenthaler (1995) and Krattenthaler (1993, 1995, 1996).
2. Nonintersecting lattice paths To begin with, we introduce some lattice path notation. Given a set S of steps we denote the set of all lattice paths from A to E with steps from the set S by ~ ( A --* E; S). We frequently shall write P: A ~ E to indicate that the lattice path P runs from A to E. A set of paths is said to be nonintersectin9 if no two paths of this set have a point in common. Otherwise, it is called intersecting. If to each step s e S a weight w(s) is assigned, the weight w (P) of a path P is defined to be the product of the weights of all its steps. The weight w(P) of a family P = (P1, ---, Pr) of paths is defined to be the product 1-I~'=1 w(Pi) of the weights of all the paths in the family. Given any weight function w defined on a set ~ ' , by the generating function G F ( ~ ' ; w) we mean
Yx ~ w(x). Proof of(1.1): Here we consider lattice paths in the plane integer lattice with steps from the set Sx = {(i,j) ~ (i,j + 1), (i, 2j - 2) ---,(i + 1, 2j - 1), (i, 2j - 1) --* (i - 1, 2j), where i, j run through the integers}. In words, the set of steps consists of vertical steps north, forward diagonal steps northeast from even height to odd height, and backward diagonal steps northwest from odd height to even height. In the sequel, when we say "lattice path" we always mean a lattice path with steps from the set $1. First, we set up a one-to-one correspondence between up-down tableaux from fl' to ~' with at most r columns and families P = ( P ~, P 2 , ... , Pr) of nonintersecting lattice paths where Pi runs from Ai = (fli + i, 0) to Ei = (~i + i, oo ) and never touches the y-axis, i = 1, 2 . . . . ,r. Let T = (Zo, Zl . . . . ,z2,) be an up-down tableau from fl' to e' with at most r columns. The bijection is defined in a way such that the ith path encodes the growth and shrink of the (r - i + 1)st column of T. To be more precise, starting with Ai = (fli + i, 0) for j = 1, 2, ... , n do the following procedure: If 272j_ 1 has one cell more in the (r - i + 1)st column than ~2j-2 then add a forward diagonal step to P~, otherwise a vertical step. If rzj has one cell less in the (r - i + 1)st column than Zzj- 1 then add a backward diagonal step, otherwise a vertical step. Finally, an infinite number of vertical steps is added to each path to connect to the end points (~i + i, oo ). Thus, given r = 5, the up-down tableau in Fig. 2 corresponds to the set of nonintersecting lattice paths displayed in Fig. 3. Note that in this example we have c~ = (4, 2, 0, 0, 0) and fl = (3,1, 0, 0, 0), i.e. ~ 1 = c ~ 2 = ~ 3 = 0 , ~ 4 = 2 , c~5=4,
/~
= / ~ 2 = / ~ 3 = 0 , / L = 1,/~5 = 3.
Clearly this operation sets up the claimed bijection. It becomes weight-preserving if we define the weight w(v) of a vertical step v to be 1, the weight w ( f ) of a forward diagonal step f:(i, 2 j - 2 ) ~ (i + 1, 2 j - 1) as well as the weight w(b) of a backward diagonal step b : (i, 2j - 1) ~ (i - 1, 2j) to be xj. Recall that this weight function w on
C. Krattenthaler / Journal of Statistical Planning and Inference 54 (1996) 75 85
E1 E2 E3
E4
79
E5
T
\
\ /F
e: P~/
/ A4
A1 A2 A3
J P~
As
Fig. 3.
the step set $1 induces a weight w on families of lattice paths as was explained at the beginning of this section. Next we give a c o m b i n a t o r i a l description of the determinant in (1.1). Because of G F ( ~ ( ( a , 0) --* (b, oQ); $1); w) = ~ eb-o+k(x)ek(x), k
we m a y write the determinant in (1.1) as det ( G F ( ~ ( A , ~ E,;Sa); w) - G F ( ~ ( 9 1 A , ~ E~; S~); w)),
(1.3)
where 91 is the reflection in the y-axis. By expanding the determinant, (1.3) can be written in the form E
sgnalL[qi
eel, .E{1,-1}'
i-1
[I i,.i--
1-[ G F ( ~ ( A , ( , ) ~ E i ; S 1 ) ; w ) i,tl,= 1
GF(~(91A~,(i)--+ Ei;S1);w), 1
or
( - 1) ll"ll sgn aw(P),
(I.4)
where Iqr/1] = ~2~= 1 qi. Of course, 9t o means the identity map. By (1.4) we have written the d e t e r m i n a n t in (1.1) as a generating function for families P = ( P 1 , P 2 . . . . , P,) of lattice paths, P~: 91" A ~(~)- . Ei, for some a e ~ r and q ~ {0, 1 }~. Let us call a the permutation associated to P and t/the reflection indicator associated to 1). In the sum (1.4) m a n y terms will cancel. We shall construct a sign-reversing (with respect to ( - 1)'q"I, sgn a) and weight-preserving (with respect to w) involution on the set of all families P = (P1, P2 . . . . . Pr) of lattice paths that were described above, which are either interesecting or contain a path that touches the y-axis. Suppose that this had been done. Then in the sum (1.4) only the terms which come from nonintersecting families of lattice paths that do not touch the y-axis will survive. In particular, none of
80
C. Krattenthaler/Journal of Statistical Planning and Inference 54 (1996) 75-85
the starting points of such a family P could be located to the left of the y-axis. Hence, the reflection indicator associated to P must equal (1, 1 . . . . ,1). Besides, because P is nonintersecting the location of the starting and end points of the paths of P implies that the permutation associated to P must be the identity permutation. But these families P of lattice paths exactly correspond to the up-down tableaux under consideration, as was shown above. This would establish (1.1). Now we define the involution, which we denote by O1. Let P = (P1, P2, ..., Pr) be a family of lattice paths, Pi: 9t"' A~i) --+ Ei, for some a e ~r and ~/~ {0, 1}', which is either intersecting or contains a path that touches the y-axis. First consider all meeting points of a path of P and the y-axis. Let M be the highest of these meeting points. Among the paths which go through M choose Pt with I being minimal. Note that the y-coordinate of M must be even. Now, to P[s initial portion up to M we apply the following modified reflection in the y-axis (see Fig. 4). Beginning from the starting point of PI up to M consider the steps of Pt in groups of two. There are four different cases. Replace two vertical steps by two vertical steps, a vertical step and a backward diagonal step by a forward diagonal step and a vertical step, a forward diagonal step and a vertical step by a vertical step and a backward diagonal step, a forward diagonal step and a backward diagonal step by a forward diagonal step and a backward diagonal step. What is obtained by this procedure is joined with the terminal portion of Pr beginning from M. In Fig. 4 the circled points indicate the grouping of the steps into groups of two that was described above. Let the image of Px under application of this modified reflection be denoted by P}. We then obtain the new family 01(P) = (P1 . . . . . P}, ... ,P~). Since the new starting point of P} is the reflection of the starting point of PI in the y-axis, the permutation associated to O1 (P) in this case is still ~r while the reflection indicator associated to O~(P) is given by If none of the paths of P meet the y-axis consider all meeting points between any two paths of P. Choose the highest meeting points, and among those the left-most.
El •pi
EI
•
•M~r•
iii Fig.4.
C. Krattenthaler /Journal of Statistical Planning and Inference 54 (1996) 75 85
81
Denote this meeting point by M. N o w we apply the usual Gessel Viennot interchanging procedure. To be more precise, a m o n g the pairs of paths which have M in common, choose (Pt, Par), with (I, J) being minimal with respect to the lexicographic order of pairs of integers. Next we decompose P, into Pp) and p]2), where -tPIl) is the portion of Pi going from 9t"' ~4~(t~ to M, and where p~2) is the portion of PI between M and Et. Analogously, Par is decomposed by M into p(al) and p(2).1• Then by concatenation form the new paths PI, == p(j1) p~2) and P's = k,~(1) I p~2). In other terms, the initial portions of Pt and Par up 1o their last meeting point are interchanged, the terminal portions are preserved. Thus, we obtain the new family O1 (P) = (P~ . . . . . P} . . . . . P ) , ... , P~). Obviously, since the lth and Jth starting point have been interchanged, the permutation associated to O1 ( P ) i s a (I J) ((I.1) is the transposition that interchanges I and J), and the reflection indicator associated to O~ (P) is (~/1. . . . . ns . . . . . nt, " ' " , ~ r ) " It is easy to see that in both cases, by application of O1 to O1 (P), one arrives at P again. Consequently, O~ is an involution. Besides, by construction it is weightpreserving with respect to w and sign-reversing with respect to ( - 1 ) I,~lsgn ~r. This completes the proof. Proof of(1.2): Here we consider lattice paths in the plane integer lattice with steps from the set S 2 = { ( i , j ) - + ( i , j + 1), (i, 2 j - 1 ) - - + ( i - 1 , 2 j - 1), (i, 2j)--+(i+ 1,2./), where i, j run through the integers }. In words, the set of steps consists of vertical steps north, backward horizontal steps (west) at odd height, and forward horizontal steps (east) at even height. In the sequel, when we say "lattice path" we always mean a lattice path with steps from the set $2. First, we set up a one-to-one correspondence between down-up tableaux.from fi to with at most r rows and.families P = ( P 1 , P2 . . . . . P~) of nonintersecting lattice paths
where Pi runs from Ai = (fli ~- i, O) to Ei = (9~i q- i, oC ) and never touches the y-a.x'is, i = 1, 2 . . . . , r. Let T = (z0, z~, ... , z2,) be a down-up tableau from fi to c~with at most r rows. The bijection is defined in a way such that the ith path encodes the growth and shrink of the (r - i + 1)st row of T. To be more precise, starting with A~ = ([4~ + i., 0) fi~r j = 1,2 . . . . . n do the following procedure: If r2j ~ has m cells less in the (r-i+ 1)st row than z2s 2 then add a vertical step followed by m backward horizontal steps to P~. If z2s has m cells more in the (r - i + 1)st row than z2j- ~ then add a vertical step followed by m forward horizontal steps to P~. Finally, an infinite number of vertical steps is added to each path to connect to the end points (~ + i, ~). Thus, given r = 4, the down-up tableau in Fig. 2 corresponds to the set of nonintersecting lattice paths displayed in Fig. 5. Note that in this example, we have cq = 1, 0', 2 = l , (~3 = 2 , ~4- = 2, f l l = 0 , f12 = 1, f13 = 1, f14 = 2. Clearly this operation sets up the claimed bijection. It becomes weight-preserving if we define the weight w(v) of a vertical step v to be 1, the weight w ( f ) of a backward horizontal step f:(i, 2 j - 1)--+ ( i - 1 , 2 . ] - 1) as well as the weight w(b) of a forward horizontal step b : (i, 2j) ~ (i + 1, 2j) to be xj. Recall that this weight function w on the step set $2 induces a weight w on families of lattice paths as was explained at the beginning of this section.
C. Krattenthaler /Journal of Statistical Planning and Inference 54 (1996) 75-85
82
E1 E~ T
L
E3 E4 T 1
A~ A3
A4
Fig. 5.
Next we give a combinatorial description of the determinant in (1.2). Because of
GF(,_@((a, O) ~ (b, ~ ) ; $2); w) = ~ hb-.+k(X)hk(X), k
we may write the determinant in (1.2) as det (GF (~(A,
--+
E s ; S 2 ) ; w) - G F ( ~ ( g i A , --+ Es; $ 2 ) ; w ) ) ,
(1.5)
where !R is again the reflection in the y-axis. By expanding the determinant, (1.5) can be written in the form sgnafiq, a~, qe{1,-
i=l
I-[ GF(~(A~,)-+Ei;S2);w) i,qi=l
1}"
I~
GF(~(!RA~(o--+ Ei;S2);w),
l,th = - 1 or
Y,
( - 1) It"it sgn aw(P),
(1.6)
a e ~ , , r / e {0, 1}" P= (P~, ... , P,), Pi: ~q'A¢lo ~ Ei
where again IIr/J[ = ~ = 1 Yli' By (1.6) we have written also the determinant in (1.2) as a generating function for families P = (P1, P2, --., Pr) of lattice paths, Pi : 9~" A , , ) ---,El, for some cre ~, and e {0, 1}r. The only difference to (1.4) is that the set of steps now is $2 instead of $1. Again, let us call a the permutation associated to P and r/ the reflection indicator associated to P. Also here, we shall construct a sign-reversing (with respect to ( - 1 ) It'llsgna) and weight-preserving (with respect to w) involution on the set of all families P = (P1, P2 . . . . , Pr) of lattice paths that were described above, which are either intersecting or contain a path that touches the y-axis. Once this is done, similar arguments as those in the proof of (1.1) show that (1.2) would be established.
C, Krattenthaler /Journal of Statistical Planning and Inference 54 (1996) 75 -85
EI •
•
•
El
T~-,.Oal
•
M o,-..o,,-
•
-
',~3
M •
-
-
.
I.
- " " e
•
.
P - - - ~ . , - e . . . . e . . . . e
. . . . . .
: ....
e,
..
-o.
• • .o.,
o..
- .o,
rP~,
- • .o-
•
-,
.o
.
1
• .o
ml_.rA-
/.
.
.
.
.
Fig. 6.
Now we define the involution, which we denote by 0 2 . Let P = (P1, P2 . . . . . Pr) be a family of lattice paths, Pi:Ot"'A,,)-~Ei, for some a ~ ~ and r/~ {0, 1}", which is either intersecting or contains a path that touches the y-axis. As before, first consider all meeting points of any path P with the y-axis. Let M be the highest of these meeting points. Among the paths which go through M choose PI with I being minimal. Again, note that the y-coordinate of M must be even. Now, to Pt's initial portion up to M we apply the following modified reflection in the y-axis (see Fig. 6). Beginning from the starting point of Px up to M consider the portions of PI which lie between lhe horizontal lines y = 2j - 2 and y = 2j, portions of Pl on the line y = 2j - 2 excluded while portions on y = 2j included. If in this portion there is a vertical step followed by n backward horizontal steps and then a vertical step followed by m forward horizontal steps (note that the vertical steps are obligatory) replace it by a vertical step followed by m backward horizontal steps and a vertical step followed by n forward horizontal steps. What is obtained by this procedure is joined with the terminal portion of Pl beginning from M. In Fig. 6 the circled points indicate where the paths are cut into portions according to the above description. If none of the paths of P meet the y-axis consider all meeting points between any two paths of P. Choose the highest meeting points, and among those the left-most meeting point, and then apply the Gessel-Viennot interchanging procedure, as was described before in the proof of (1.1). Again it is easy to see that in both cases, by application of O2 to 02 (P), one arrives at P again. Consequently, O2 is a sign-reversing (with respect to ( - 1 ) I"l sgn c~) and weight-reversing (with respect to w) involution. This completes the proof,
References Berele, A. (1986). A Schensted-type correspondence for the symplectic group. J. Combin. Theory Ser. A 43, 320-328. Brenti, F. (1993). Determinants of super-Schur functions, lattice paths, and dotted plane partitions. Adv. Math. 98, 27-64.
84
C. Krattenthaler /Journal of Statistical Planning and Inference 54 (1996) 75-85
Brenti, F. (1995). Combinatorics and total positivity. J. Combin. Theory Ser. A 71, 175-218. Bressoud, D.M. and S.-Y. Wei (1992). Determinental formulae for complete symmetric functions. J. Combin. Theory Ser. A 60, 277 286. Choi, S.H. and D. Gouyou-Beauchamps (1993). Enumbration de tableaux de Young semi-standard. Theoret. Computer Science 117, 137-151. Comtet L. (1974). Advanced Combinatorics (D. Reidel, Dordrecht). Conca, A. (1994). Gr6bner bases of ideals of minors of a symmetric matrix. J. Algebra 166, 406 421. Conca, A. (1995). Ladder determinantal rings. J. Pure Appl. Algebra 98, 119-134. Conca, A. and J. Herzog (1994). On the Hilbert function of determinantal rings and their canonical module. Proc. Amer. Math. Soc. 122, 677-681. Desainte-Catherine, M. and G. Viennot (1986). Enumeration of certain Young tableaux with bounded height, In: G. Labelle, P. Leroux, Eds. Combinatoire enumbrative. Springer Berlin, 58-67. Doran, W.F. (1993). A connection between alternating sign matrices and totally symmetric self-complementary plane partitions. J. Combin. Theory Ser. A 64, 265-288. Fulmek, M. and C. Krattenthaler (1995). Lattice path proofs for determinant formulas for symplectic and orthogonal characters. J. Combin. Theory Ser. A (to appear). Gessel, I.M. (1993). Counting paths in Young's lattice. J. Statist. Plann. Inference 34, 125-134. Gessel, I.M. and C. Krattenthaler (1995). Cylindric partitions. Trans. Amer. Math. Soc. (to appear). Gessel, I.M. and G. Viennot (1985). Binomial determinants, paths, and hook length formulae. Adv. Math. 58, 300-321. Gessel, I.M. and G. Viennot (1989). Determinants, paths, and plane partitions, preprint. Greene, C. (1992). Proof of a conjecture on immanents of the Jacobi-Trudi matrix. Linear A19ebra Appl. 171, 65-79. Herzog, J and N.V. Trung (1992). Gr6bner bases and multiplicity of determinantal and Pfaffian ideals. Adv. Math. 96, 1 37. Krattenthaler, C. (1993). Non-crossing two-rowed arrays and summations for Schur functions. In: A. Barlotti, M. Delest, R. Pinzani, Eds., Proc. 5th Conf. on Formal Power Series and Algebraic Combinatorics, Florence, D.S.I., Universitfi di Firenze, 301-314. Krattenthaler, C. (1996). Non-crossing two-rowed arrays (in preparation). Krattenthaler, C. (1995). The major counting of nonintersecting lattice paths and generating functions for tableaux. Mere. Amer. Math. Soc. 115, no. 552, Providence, R.I. Krattenthaler, C. and S.G. Mohanty (1995). Counting tableaux with row and column bounds. Discrete Math. 139, 273 286. Kulkarni, D.M. (1993a). Counting of path-like objects and coefficients of Hilbert polynomial of a determinantal ideal, preprint. Kulkarni, D.M. (1993b). Counting of paths and coefficients of Hilbert polynomial of a determinantal ideal. preprint. Macdonald, I.G. (1979). Symmetric Functions and Hall Polynomials. Oxford University Press, New York/London. M odak, M.R. (1992). Combinatorial meaning of the coefficients of a Hilbert polynomial. Proc. Indian Acad. Sci. (Math. Sci) 102, 93-123. Okada, S. (1990) Partially strict shifted plane partitions. J. Combin. Theory Ser. A 53, 143 156. Okada, S. (1991). Lattice path method and characters of classical groups, preprint. Remmel, J.B. and R. Whitney (1983). A bijective proof of the hook formula for the number of column-strict tableaux with bounded entries, Eur. J. Combin. 4, 45 63. Remmel, J.B. and R. Whitney (1984). A bijective proof of the generating function for the number of reverse plane partitions via lattice paths. Linear Multilinear Algebra 16, 75-91. Roby, T.W. (1991). Applications and extensions of Fomin's generalization of the Robinson Schensted correspondence to differential posers. Ph.D. Thesis, MIT, Cambridge, Ma. Sagan, B.E. (1989) .The ubiquitous Young tableau In: D. Stanton, ed. Invariant theory and tableaux. The IMA Volumes in Math. And Its Appl., Vol. 19, Springer, New York, 262-298. Sagan, B.E. (1992). Log concave sequences of symmetric functions and analogs of the Jacobi Trudi determinants. Trans. Amer. Math. Soc. 329, 795-811.
(L Krattenthaler / Journal of Statistical Planning and Inference 54 (1996) 75 85
~':5
Stembridge, J.R. (1990). Nonintersecting paths, pfaffians and plane partitions. Adv. Math. 83, 96 131. Stembridge, J.R. (1995). The enumeration of totally symmetric plane partitions. Adv. in Math. 111, 227 245. Sulanke, R.A. (1990). A determinant for q-counting lattice paths. Discrete Math. 81, 91 96. Sundaram, S. (1986). On the combinatorics of representations of Sp(2n, Ct. Ph.D. Thesis, Massachusetts Institute of Technology. Ueno, K. (19911. Lattice path proof of the ribbon determinant formula for Sehur functions. Naqova Math. I. 124, 55-59.