Algorithms for processing partial match queries using word fragments

Since m
328

V.S. ALAGAR

cost of step 3 is (N-it 1)t and that of step 4 is o[(t - b)@ - i)]. Thus the total cost of constructing all dependence trees is q[(n-i+l)t+(t-b)(p-i)]+b@-1)

mine the expected value of X,, assuming the elements Cd, . . . d,) are independent and uniform. Although the assumption of independence and uniformity are not justified, the result gives some insight to the value of m; moreover in practice the true lower bound will be much higher than what is stated in the theorem.

=m[N-$m-l)t(r-l)(p-m-l)]tb@-1). Substituting for (t-b) from (2) we approximate and simplify this expression to get o(mN). The algorithm will need slight modifications when the whole matrix can not be held in memory. Given the equiprobable nature of fragments the value of m will increase with b, the amount of overlap. Unless the joint distribution of fragments are known, the quantity b cannot be predicted. However we can estimate m in a probabilistic sense. Towards this we introduce the concept of absorbing fragments. A fragment di in DP = (dr,. . . , dp) will be called an absorbing fragment with respect to a set of records if for all k, i
d2

d,

R, R2

1 1

0 0

I R3

0

1

1 1 1

d2 is the only absorbing fragment. Note that there are two dependence trees, one with root d, and the other with root d2. It is clear from the definition and the examples that an absorbing fragment is a candidate for the root of a DTC. However the root of a DTC need not be an absorbing fragment, since a maximal chain containing the root and a leaf node of a DTC need not necessarily contain all fragments imposed by the above definition. Thus we assert that if there are X, absorbing fragments among Q,, these must have occurred among the roots of the dependence trees constructed by the algorithm DTC. This implies X, < m. The bounds on m are given by the following theorem. Theorem

I

The lower and upper bounds of the expected value of m are log p and 2dp respectively. Proof: Since m 2 X,,, the lower bound of the expected value of m is the expected value of X,,. We will deter-

Let G,(y) = $Pr(X,

= n)y”

(3)

be the generating function of the probability distribution function of X,,. If we pass from the state D, to Dp+,, then with probability I/p t 1 a new absorbing fragment is added and with probability p/p t I no new absorbing fragment is added. Thus we write G,+,(Y) = G,(Y)(P + Y)/(P + 1) =Ilp/fi(s+y) s=o = l/P$,

(- I)p+rs(P, r)y’

where S(p, r) are Stirling numbers of the second kind. This gives P(X, = n) = S(p, n)/p!.

(4)

From (4) the expected value of X, turns out to be Fl/p, which is equal to log p t y, where y is Euler’s constant. For the upper bound consider all disjoint chains included in the dependence trees. If W, denotes this number, we must have m < W,. Assuming that chains have different fragments, the quantity W, turns out to be the length of a largest increasing sequence in a permutation of p elements. No exact formula is known for this quantity, however simulation studies reported in [20] reveal that 2dp is an excellent approximation for the average of W,. Hence the theorem is proved. Towards efficient query processing we need the dictionary D in which each di, i = 1,. . . , m keeps an index to its DTC and di, j = m t 1,. . . , p keeps an index to its inverted list. In addition to this we associate with each di, i= I,... m a bit string of length i - I which indicates the dependence trees in which di occurs. If i,, . . . , is are s fragments generated from the terms of a user query, the pertinent records may be obtained as described below: ALGORITHM DTCL (1) [Initialize] set X,c( } and X2+ }. (2) [Search D] for j: = I to s do this step. Search D for ib If ii = d,, I I r~ m put r in X,; otherwise put r in X2.

Note that both X, and X2 cannot be empty. (3) [X, is not empty] If X, is empty execute step 4. Let r be the minimum of the integers in X,. Let Y{jldi < d,}. This information is available in the bit string associated with d,. If 1 t 1Y( > s -IX,/ execute step 4 else do the following: read in the dependence trees d, and di, jE Y. The each

pertinent of

these

record

numbers

dependent

will

trees

belong that

to chains

contain

all

in the

329

Algorithms for processing partial m atch queries using word fragments

specified fragments. Let L denote this set of record numbers. Execute step 5. (4) [X, is empty] For each r in X2 read the inverted list corresponding to d,. Form L, the intersection of these lists. (5) [Read pertinent records] For each record number in 15, read the record and decode. Storage and cost analysis We need extra storage to store the m dependence trees, their inverted lists and the inverted lists corresponding to the p-m fragments. The size of the inverted lists is t(p - m) t N, since the inverted lists of the dependence trees are mutually disjoint. Thus the amount of additional storage demanded in this method over the pure inverted list design is (N - tm t storage for the trees). This additional storage is the trade off one has to pay for the gain in processing speed. Since the expected length of I_. would be the same as in Method 1, the cost of steps 3 and 4 must be compared to (A t pt)s. Intuitively it is clear that when the framework for query processing is as enunciated in Section 3, the fields that are most likely and more often to be specified in queries will not only contribute more fragments to D than other fields would contribute, but also that these are more likely to be selected as the roots of dependency trees. In other words, X, will not be empty for most of the queries. When this happens step 3 or step 4 is executed, whereas when X1 is empty only step 4 is executed. Let L, and U, denote the lower and upper bounds of the average retrieval cost A,. Consider the fragment generation from query terms as random; this is justified because the queries themselves are random. Let Q = m/p. The probability that exactly j out of s fragments will be the roots of the dependence trees is qj =

0

Case I l s - j, j > 1. Thus we have

Since !jqi = sa, we have Us = s(l - a) < s Case 2 2d(m)-l
ls-j,

l~j
andm/jtl
j,
Hence we have il-1

U,=sqot

C(s-j)qit

~(mij+~)qiti~,(s-j)q~

I

il

Now consider

us-s=

us-s(qot

. . . tq,)=

C -jqit~(m/jtl-s)

I

; a’(1 - .)s-i. 4i + j$,

Lower bound L, : With probability qo, the s fragments belong to p -m fragments that are not roots of dependence trees. For j > I, the j fragments that match the roots of dependence trees can be such that step 3 is executed just once. Hence we have

(7)

-

il

hi.

Using the result (7), we have Us < s. Case 3 Itm/2GsSp

The roots j, and j2 are real and distinct but j, is negative. So the conditions ( ) become

and L, < s. Upper bound II,: Let XI consist of integers i,, . . . , ii. Assume that these integers are uniform and independent in the range 1,..., m. The expected value of their minimum is close to m/jtl. For j>l, if m/jtl
m/jtl>s-j

j2
m/jtl
l
and the rest of the analysis is similar to case 2. Since the upper bound remains smaller than s in all cases, we establish A, < s. An exact expression for A, is difficult to obtain even when the conditional distributions are known; however it can be computed. Although our analysis assumed independence, experiments conducted with a small data set of 50 records confirm our analysis. To sum up, the main advantages of this method are: (1)

330

V. S. ALAGAR

The reduction in the size of the inverted lists; (2) The gain in processing speed; and (3) The cost function remaining directly proportional to the size of the result. Next we shall show how the average retrieval cost per query can be minimized by a linear arrangement of records with redundancy. 4.3 Ghosh [ 161first introduced the property of consecutive retrieval for a given query set and file. A set of queries is said to have the CR (consecutive retrieval) property with respect to a record set if the records can be placed in a linear storage medium (such as tracks of a disk) so that the records pertinent to every query can be placed in consecutive storage locations without redundancy. Easwaran[l7] has given necessary and sufficient conditions for a query set to have the CR property with respect to a file. In general, a query set and a file may not have the CR property. Similar to the spirit of the CR property, we introduce the consecutive retrieval on fragment (CROF) property in the following way: Let

R(D) = IS,,

. $1

where S, = R(di). F = U{Si} = {RI,. . . , R,}. Thus F is the file consisting of the union of the sets Si, where Si is the non-empty inverted list of the fragment di. Since the records belonging to F - 5’; are not related to di they should not be retrieved when di alone is given in a query. We call the elements of F - S, as foreign to di. We say the set of fragments D has CROF property if the following condition holds: There exists a one-to-one function that maps the reocrds in F into the set {I, . N} such that for each d, ED, the corresponding Si can be mapped onto an interval 1, which contains the images of all the elements in Si but not the images of those in F- $ If D has the CROF property then all relevant records pertinent to any partial chain can be obtained in minimum time. It is sufficient to know the first and last records pertinent to a partial chain in order to retrieve the whole set of relevant records. Since the fragments occur equiprobably on partial chains, we can expect to achieve minimum overall retrieval time, if the CROF property holds for D.

Example 2 Assume D = {d,, d2, d3, d4} and there are 6 records in F = {RI,. . , Rh}. Let the inverted lists be given by Si = R(di), where S, =

{R*, Rx, Rs}

S, = {R,, R5, RJ

Example 3 Let F=(R,,...,

Rs). Assume d,, d2, d, and d4 are fragments. The record-fragment incidence matrix is

R, R2 R3 M=R., Rs Rg R, Rs

d,

ds

d,

d.,

0 0 0

I

0

0 0

0 1

I 0 1

0 1 0 I

0 I I 0 0 0 0

0

1

I I

I 0 0

0 0

The matrix M can be transformed the rows to M,,

by a permutation

S2 = {RI, R2, Wr

S, = fR2,

R,.

’ d,

I-4.

Define the .mapping g : F + { 1,2, . . (6)

g(R,) = I, g(R,) = 3, g(R,) = 4, g(k) = 2, g(Rs) = 5, g(Rn) = 6. Now g(S,) = ]3,51, gW = [l, 31, g&I = 14,619AS.+)=

r&41.

It is straightforward to verify that g is a one-to-one function and maps the sets Si so that the CROF property holds for the fragments with respect to the given records. Assume that the records (RI,. , R6) are stored in the locations (I, 4, 2, 3, 5, 6). Consider any partial chain composed of (d,, . . . , d4); say c = (* d2 * d4). The records that are pertinent to this chain are R2 and R., (the intersection of S, and S4) and these are in consecutive storage locations. One can verify that for every partial chain the pertinent records can be found in consecutive locations of storage. If the CROF property holds for D, then there exists a mappingg:F+{l,..., N} for which Si = R(di) will have an interval [Ii, Ui] as the image under g. The image of a partial chain c, = (d,,, . . . , d,) is [I,, u,] where I, = max{/,i} and u, = min{u,}. If b > u, there is no record satisfying the query from which the partial chain was generated. In terms of the record fragment incidence matrix we say that the CROF property holds if and only if there exists a permutation of the rows of the matrix such that in each column the l’s occur in consecutive rows. Thus once the fragments D which occur with equiprobability in a file F are selected by the algorithm[l], we have to examine the existence of the CROF property for D. However if the CROF property does not exist for D with respect to the file, we need to examine the possibility of allowing redundant storage of records so as to achieve minimum possible retrieval time for any partial chain of length s.

dz

d,

d4

0 0

0 0

R, Rq

0 I

1 I

&

1 1 0 0 0 0

:, 0 0 0 0

M,=Rs R, R, R2 R,

0

0

I I I

o 0 I

0 0

1 1

of

Algorithms for processing partial match queries using word fragments

The CROF property exists and one organization of the records is RI R4 R6 R, R3 R2 Rs. The CROF property does not hold for the example 1. However if the records RS, R2 and R7 are placed twice as in Rt RS RZ RT Rg R6 Ri RJ Rs Rq Rz, all queries that specify one or more fragments will have pertinent records in consecutive storage locations, In this organization the redundancy factor is 318 and is much smaller than the redundancy in a total invested file organization. For this example 318 is the minimum possible redundancy factor. This type of organization aill be called consecutive retrieval on fragments with redundancy (CROFR) organization. Four questions arise naturally. These are (I) Does CROF property exist? (2) How to obtain an organization having CROFR property when CROF property does not exist? (3) What is the cost of such an investigation? and (4) What is the redundancy factor? In [17] necessary and sufficient conditions are given for the existence of CR property with respect to a query set. In general the investigation of the existence is a difficult problem. in [I91 methods are given to find query inverted file organization with redundancy, Thus the answer to the first three questions are similar to the fmdings in [17,19]. We shall answer the fourth question by deriving an upper bound for the redundancy factor. Estimation of redundancy There are p -m fragments all of which occur among the m DTCs constructed by the Algorithm DTC. All chains that occur among these fragments occur as partial chains in one or more DTCs. Let w denote this number. Assuming that these chains are equally likely to be absorbed among the DTCs, each one of these chains occur among m/w DTCs on the average. The number of records pertinent to each one of these chains is t(p m)/w, since on the average there are t records pertinent to each fragment. In order to force a linear ~rangement of records such that records pertinent to all fragments are found in adjacent storage locations, we must associate the records of these w chains with at most [m (m/w)] DTCs. Hence the redundancy can not exceed [m -(m/w)] (t(p - m)/w). It remains to estimate w. This is equal to the length of the longest ascending sequence in a permutation of p - m elements. The approximation given in [20] for the average of this value seems quite good even for moderately small values. Hence we take w = 21,/(p - m) and obtain an upper bound for the number of redundant records and is given by

(8) From Theorem (1) we have m 52~‘~. Hence we have

This shows that the derived upper bound is strictly smaller than the redundancy factor in a total inverted

331

organization. The following two examples give some insight into the extent of reduction achievable in a CROFR organization.

Consider 8 records from which 5 fragments are extracted. The record-incidence matrix M and the dependence trees are shown below: /

dl

dz

d,

d,

dS

An organi~tion of the records with redundancy factor 112 is: RJ R5 Ri R6 Rs Rz Rs Rs R7 Rs R, R4. There exists no organization with redundancy factor smaller than l/2 and the upper bound (8) gives 5.28/S. The redundancy factor in a total inverted organization is 1l/4. Example 5 We consider all binary attributed records (excluding the string of O’seach having four attributes (bits). There are 15 records. We consider 9 fragments, each being pertinent to four records. These fragments are: dr = (Ol4, dz = (IO--), d3 = (I I--), dd = (--OIL ds = (--IO), d6 = G--11), d? =(-Ol-1, ds = (-10-J and d,= (-II-). The records are numbered as R, = 0001, Rz = 0010,. . . R,5 = I I I I. The dependence trees and a linear organization are shown in Diagram 4. A linear organization with redundancy is Rts R, R3 RI1 R,c Rs % R, Rn Rs R, f-h R5 Ra Rn R,, RT Rta Rts Ra RZRN R3 RII. The redundancy factor is 9115and the upper bound gives the value 14115,The redundancy factor of a total inverted organization is 4.

332

V. S. ALAGAR

o(mN), log p I m I 2dp. The cost of setting up CROFR organization is high, since this is an inherently difficult one. However methods suggested in [19] and the nice structure of the dependence trees should be used in an implementation of method 3. Typically the design cost is incurred only once. The drastic reduction in the operational cost of the system will adequately compensate and in fact pay dividends for the initial cost incurred. Acknowledgement-The author thanks the referees for their valuable comments which enabled the revision of the paper to the present form. 1”9

(151

Diagram 4.

5. CONCLUDING REMARKS

The feasibility of designing an information retrieval system that uses word fragments as basic language elements in a query has been investigated. The discussion has been restricted to queries that are partially specified. Any preprocessing that might be done to transform query terms to fragments and any further processing of records that are necessitated after the records have been brought to main memory are common to all the methods discussed in this paper; yet these may differ in detail and their complexity. The three methods are compared and critically evaluated in terms of the retrieval cost which is the on-line measure, i.e. the number of disk accesses to answer a single query. In the first method that uses the tranditional inverted lists for the fragments, the cost as a function of s (the number of specified fragments) increases with s and thus is inversely proportional to the result. This implies that one has to pay more to get less. This situation is remedied in the second method and to a great extent eliminated in the third method. Upper and lower bounds for the average cost in method 2 show that the dependence trees and their associated inverted lists decrease the retrieval cost considerably. Method 3 refines the second method to achieve a linear ordering of records with nearly minimum redundancy and force minimum overal retrieval time. Thus in terms of retrieval, the costs of the three methods form a strictly monotonic decreasing sequence. Since the dependence trees together with their inverted lists are kept, method 2 seems to require more storage than method I. However our analysis for method 3 shows that the maximum redundancy in CROFR organization is strictly smaller than that of a total inverted organization. Hence method 3 requires least space and has least retrieval cost among the methods discussed. Finally one can not but compare the design costs. It is clear that method I has least design cost. It has been shown that the cost of setting up the dependence trees is

REFERENCES

E. J. Schuegraf and H. S. Heaps: Selection of equifrequent word fragments for information retrieval. Inform. Star. Retr. 9, 697-71 I (1973). PI V. R. Walker: Compaction of names by X-grams. Proc. Am. Sot. Inform. Sci. 6, 129-135 (l%9). f31D. S. Colombo and J. E. Rush: Use of word fragments in computer based retrieval systems. J. C/rem. Lbcum. 3, 4750 (1%9). [41E. J. Schuegraf and H. S. Heaps: Query processing in a retrospective document retrieval system that uses word fragments as language elements. Inform. Proc. Management 12, 283-292 (1976). (51R. L. Rivest: Partial-match retrieval algorithms. SIAM _f. Comput. 5, 19-50 (1976). 161W. A. Burkhard: Partial match retrieval. BIT 16, 13-31 (1976). f71D. E. Knuth: The Art of Computer Programming: Vol. 3 Sorting and Searching. Addison-Wesley, Reading, Mass. (1974). PI E. Wong and T. C. Chiang: Canonical structure in attribute based file organization. CACM 14, 309-319 (1971). f91C. T. Abraham, S. P. Ghosh and D. K. Ray-Chaudri: File organization schemes based on finite geometries. Inform. Co&. 12, 143-163 (1%8). [toI R. A. Gustafson: A randomized combinatorial file structure for stroage and retrieval systems. Ph.D. Thesis. University of South Carolina (1%9). D. Hsiao and F. Harary: A formal system for information retrieval from files. CACM 13, 218-222 (1970). J. L. Bentley and W. A. Burkhard: Heuristics for partial match retrieval database desinn. IPL 132-135 (1976). V. S. Alagar and C. Soochai: Partial match retrieval for non-uniform query distributions. Proc. NCC’79, New York, pp. 775-780. (141A. V. Aho and 1. D. Ullman: Optimal partial match retrieval when fields are independently specified. ACM Trans. Database Systems 4(2), 168-179 (1979). WI C. J. Van Rijsbergen: A theoretical basis for the use of co-occurrence data in information retrieval. J. Docum. 2, 109-l I9 (1977). [I61 S. P. Ghosh: File Organization: The consecutive retrieval property. CACM IS, 802-808 (1972). [I71 K. P. Easwaran: Faithful representation of a family of sets by sets of intervals. SIAM J. Comput. 4, 56-67 (1975). WI S. P. Ghosh: Database Organization for Data Management. Academic Press, New York (1977). 1191S. P. Ghosh: The consecutive storage of relevant records with redundancy. CACM IS, 464-471 (1975). PO1R. M. Baer and P. Brock: Natural sorting over permutation spaces. Math. Comput. 22,385-410 (1968).