Unifying concurrency control and recovery of transactions

infomtation SystemsVol. 19,

Pergamon

Elsevier Science

030~4379(94)EOOOCD

No. 1, pp. 101-115,

1994

Ltd. Printedin GreatBritain 0306-4379194$6.00 + 0.00

UNIFYING CONCURRENCY CONTROL AND RECOVERY OF TRANSACTIONS GUSTAVO ALONSOI, RADEK VINGRALEK~, DIVYAKANT AGRAWALI, YURI BREITBART~ AMR EL ABBADI~ , HANS-J.

SCHEK~ and GERHARD WEIKUM~

‘Department of Computer Science, University of California, Santa Barbara, CA 93106 2Department of Computer Science, University of Kentucky, Lexington, KY 40506 3Department of Computer Science, ETH Zurich, CH-8092 Zurich, Switzerland (Received

18 June

1993; in final revised form

1 November

1993)

Abstract - Transaction management in shared databases is generally viewed as a combination of two problems, concurrency control and recovery, which have been considered as orthogonal problems. Consequently, the correctness criteria derived for these problems are incomparabie. Recently a unified theory of concurrency control and recovery has been introduced that is based on commutativity and

performs transaction recovery by submitting inverse operations for operations of aborted transactions. In this paper we provide a constructive correctness criterion that leads to the design of unified protocols that guarantee atomicity and serializability.

1. INT~ODUCTrON The classical theory of transaction management treats issues of serializability and atomicity independently [7]. Based on that theory, concurrency control protocols that ensure serializability are supplemented by recovery protocols that ensure transaction atomicity. Since concurrency control and recovery have been considered orthogon~ly, solutions have been devised independently. The result is that the different criteria are incomparable except for the most restrictive criterion in each component. In particular, strict two-phase locking and log-based recovery have become the de facto standard in commercial systems [7, 161. Recently, a unified theory for concurrency control and recovery has been developed based on the notion of an expanded history [26]. An expanded history is used to represent the transaction’s recovery actions explicitly. As a result, the unified theory can be used to determine whether a given execution is correct both from a concurrency control as well as recovery points of view. A hierarchy of correctness classes is defined and the authors introduce a particular class, called prefix-reducibility (PRED). PRED is argued to be the most permissive class of executions that are intuitively correct accounting for both concurrency and recovery in the system. The authors state that an important research topic is to further explore the class of such histories in order to identify a m~imum subclass that can be recognized by a dynamic scheduler with acceptable efficiency. We concentrate here on this problem. As in [26], we model log-based recovery instead of deferred update methods, since virtually all database systems use logging for efficiency reasons. We start by modifying the original definition of expanded histories to exclude certain incorrect executions. The original definition of PRED was stated in a recursive manner, which hinders its use for the development of dynamic schedulers. Hence we develop a new definition, Se~u~~zab~Z~t~ with Ordered Te~~~ut~~~ (SOT), which is procedural and is shown to be equivalent to PRED, Our characterization leads to extensions of traditional serialization graph testing [4, 121, two-phase locking protocols [13], and timestamp ordering protocols, such that the extended protocols ensure both atomicity and serializability. This paper combines the results of [l] and [27] in which closely related results have been developed independently and in parallel. The rest of the paper is organized as follows. In the next section we review the database system model. In Section 3 we introduce a class of histories that are ~e~~~izab~e with ordered termination (SOT), and prove that this class of histories coincides with prefix reducible histories. In Section 4 serialization graph testing protocols implementing SOT are discussed. Section 5 presents several locking protocols that unify concurrency control and recovery. Section 6 briefly IS

19:1-H

101

discusses extensions to timestamp-based protocols. Section 7 concludes the paper with an outlook on possible generalizations of the presented work. 2. MODEL DESCRH’TION A database is a collection of data objects. Actions that allow access to a database are called upread and write (respectively denoted by r and w). Users interact with the database by issuing transitions. A transaction, Ti, is a partial order, , where T&, . . . ,Ti, are from the transition set T. This operation, called group abort, indicates that an abort should be executed for each transaction from Tdl, . . . , Z”b. However, the execution of these aborts is conducted concurrently. Operation afTi,, . . . , Z’i,) is interpreted as an abort of all active transactions in H when a system crash occurs. Transaction Ti is said to be committed (aborted) in H if W contains ci (ui or a(. . . ,T;, . . .)) operation(s). Transaction Ti is active in H if it is neither committed nor aborted in Hi. In the sequel we consider a(Ti) and ai as identical operations. A complete history H over a set of transitions 7 is a history in which each transaction in 7 is either committed or aborted. A projection of H on a set of tr~sactions T is a history that contains only operations of tr~sactions from 7. A coquetted projection of history H is a history C(H) that contains only operations of committed tr~sactions in H. We say that tr~s~tions Ti and Tj are in direct WT (ww, VW>concoct in history H if and only if history if contains operation w<]z] (wi [z], ~6[x]) fo11owed by operation rj [z] (wj [z], wj [x]), and Ti does not abort before r-j/z] (wj[ x 1,w.3 [x ]) is executed. If Ti and Tj are in direct wr, or ww, or TW conflict we say that Tg and Tj are in direct conflict. Two histories are confect-e~~~va~ent if they are defined on the same set of transactions, have the same operations and the same set of pairs of conflicting operations of committed transactions. History H is ~~~~~ct setiakzuble if its committed projection is convict-equivalent to a serial history, A se~al~zat~o~ graph for history H is a directed graph with nodes corresponding to transactions that are committed in H and with a set of edges such that Ti --+ Tj if Ti directly conflicts with Tj in H. History H is seri~izable if and only if its serialization graph is acyclic 171. In order to consider the notions of seri~izability and recovery uniformly, we have to ensure that a scheduler not only generates serializable schedules but also guarantees that no transaction reads data from aborted transactions. Namely, following I?], we say that transaction 7”’ reads-z-&m transaction Ti if and only if Ti and Tj are in direct WT conflict and there is no operation wk [x] in H that follows wi[z] and precedes operation T~(z] in H such that TI, and Tj are in direct WT conflict. A history is called ~cove~able if for every two transactions T;, Tj such that Tj reeds-x-from Ta and Tj commits, then Ta commits before Tj [17]. Recoverability, however, does not guarantee freedom from cascading aborts. Cascading aborts occur when an abort of one transaction requires an abort of another transaction. For example, when a transaction reads from an uncommitted transaction that later aborts. As we wil1 see later in the unified model, cascading aborts could also occur when a transaction overwrites data written by another transaction that later aborts. Finally, if the database uses in-place updating, it is convenient to implement the abort mechanism by restoring the before-images of all aborted writes [28, 18, 71. In order to use before-images to eliminate the er~t~ons. We consider two operations on data items from the database:

Unifying Concurrency

Control and Recovery of ‘Ikausactions

103

effects of aborted transactions, read and write operations on a data object CCmust be executed only on committed values. Executions with this property are called strict. In order to handle aborted transactions explicitly in a history we replace a transaction abort statement with a sequence of transaction undo operations to eliminate the partial effects of an aborted transaction. Each operation of transaction T; that changes the database state, i.e., a write operation in our model, is replaced in the history with its inverse operation denoted by ~7’. On the other hand, the transaction’s operations that do not &ect the database state, in our case read operations, can be safely discarded from the history. Thus, if a scheduler produces a serializable history of transaction operations for committed transactions and undo operations for aborted transitions by discarding read operations of aborted transitions, then issues of seri~izability and recovery are treated by such a scheduler in a uniform way. To guarantee. correct recovery it is assumed that active transactions will abort. These ideas proposed in [26] lead to the following definition of an e~nd~ history: Definition 1 (Expanded History) Let H = (A, <) be a history, where A is the set of operations in H and < is a partial order over those operations. Its expansion, fi-, or expanded history g, is a tuple (A, <) where: 1. /i is a set of actions which is derived from A in the following way: (a) For each transaction

Ti E H, if oi E 2”; and oi is not an abort operation,

then oi E &

(b) Active tr~s~tions are treated as aborted transitions, by adding a group abort a(T;, . . . Ti,) at the end of H, where Ti, . . . Ti, are all active transactions in H. fc) For each aborted transaction Tj E H and for every write operation wj[Z] E Tj, there exists an inverse write WY’[x] E a, which is used to undo the effects of the corresponding write operation. An abort operation oj E: H is changed to cj E a. Operation a(Ti, . . . T,,) is replaced with a sequence of cil,. , . , cit. 2. The partial order, ?, is determined as follows: (a) For every two operations,

oi and oj, if

oi < oj

in H then oi
(b) If a(T,, 2. . . , Ti,) E H, then every two conflicting undo operations of transactions from are in B in a reverse order of the two corresponding write operatheset {Ti,,..*,Tik) tions in H. (c) All undo operations of every transaction Ti that does not commit in H follow the transaction’s original read-write operations and must precede oi in ii. (d) Whenever o, < o(Ti,, . ..qTiJ < o, and some undooperation 07~ (j E {ir,...,ik}) conflicts with o, (o,), then it must be true that o;‘
GUSTAVO ALONSO et al.

104

which an abort operation is substituted by an inverted list of the corresponding undo operations for the aborted transaction (note that active transactions are treated as aborted transactions). Thus, in an expanded history every transaction commits. A transaction that committed in an original history does not have any undo operations in the expanded history, while transactions that are active or aborted in an original history have undo operations for its write operations in the expanded history. While an actual log manager will undo operations following a reversed order for both aborted and active transactions, the definition of expanded histories we provide is more general. As point 2.b states, the reversed order is only necessary for co~~~c~~~gundo operations; undo operations that do not conflict could be performed in any order. Furthermore, crash recovery mechanisms usually undo active transactions by using the reverse global ordering of the updates of those transactions. Points 2.d and 2.e specify that undo operations replace the abort operation in the same order as in the original history. To illustrate, consider history Hi: ‘wi[z] w2[2] w~[y] wr [jj], Since Nr contains only active transactions Z’r and Tz, we first expand Iii with a(Tl, Tz). The set of operations of expanded history tii then consists of (wr[~], ws[z], ws[y], wl[y], wJ’[x], wr’[y], wgl[z], w;‘[y], cl, ~2). Then, using rules 2.a, 2.b, and 2.c the expanded history Hi is obtained as follows:

Note that given a history N, there exists a unique expanded history I?. The expanded history Z? is obtained from H by substituting every abort operation with undo operations performed in a reverse order of forward operations for an aborted transaction (or transactions, in the case of group abort) followed by the transaction’s commit operations. We say that history H is reducible (RED) [26] if its expanded history fi can be transformed into a serial history by applying the following three rules: 1. Commutativity such that oi
Rule:


If two operations oi and oj do not conflict in fi, and there is no ok the ordering oa
2. Undo Rule: If wi fz] and wi’ [z] are in & and w&r] Z:w,’ [zr] and there is no oj such that w~[~c]?:~j-?w~~[~], then wi[z] and wi”[z] can be omitted from the expanded history. 3. Null rule: from _ii.

If T+ does not commit in H then all operations ri[z] can be removed for all z

The notion of reducing one history into another based on commutativity of operations is also useful in the context of semantics based concurrency control theory [3]. Unfortunately, the class RED is not prefix-closed and hence cannot be used for on line scheduling of transactions. Consider, for instance, the history wi[z]w~[z]c~ci which is RDD, but the prefix wr[s]wz[~]ez is not. This is resolved by further restricting this class of histories. In particular, a history Ii is prefi reducible (PRED) if every prefix of H is reducible 1261. Note that this definition of expanded histories is more restrictive than that in [26]. With the original definition, the expansion of the history

could be

According to the original definition, the processing of ai may be postponed until the end of the history. However, because of the conflict between WI(X) and ws(z), transaction Tl can be aborted correctly only if the operations of transaction TZ axe undone first, This consideration shows that there is actually no point in postponing the abort of Ti. The only “gain” that is achieved by postponing the abort of Tr is that Tz is able to invoke further operations (namely, Q(Z) in the example) before it will eventually be forced to abort. Obviously, this gain of concurrency

Unifying Concurrency

Control and Recovery of Transactions

105

for aborted and, thus, wasted work is undesirable from a practical point of view, and could be eliminated from the model without losing any “interesting” schedules and, therefore, the generality of the original definition is not required. To summarize, the major difference between the original definition of expanded histories and our definition here is that in the case of the original definition the order in which scheduler performs undo operations is not important while in our case, it is of the utmost importance. The class of prefix reducible histories is a proper subclass of serializable and recoverable histories [26]. For example, the reader can easily check that history H: wi [z] w2 [z] ai us is recoverable but not prefix reducible. The main reason for an existence of such histories is that wr and ww conflicts that are equal from the prefix reducible histories viewpoint are not equal from the recoverability point of view. In fact, recoverable histories are not concerned with ww conflicts at all. Every strict [7] history is also prefix reducible [26]. History HZ: wi[z] w2[2] ci ~2, on the other hand, is prefix reducible but not strict. Finally, there are histories that avoid cascading aborts but are not prefix reducible and vice versa. For example, history Hs: wr [z] ~[z] w~[z] TQ[Z]cl c2 cs is prefix reducible but has cascading aborts and history H4: wi[z] wz[zJ c2 cl avoids cascading aborts but is not prefix reducible. Although the class PRED histories seems to be the weakest class of executions that can be allowed using the traditional recovery mechanisms, it has several undesirable properties. In particular, PRED histories may result in cascading aborts, which significantly reduce overall performance. ~aditionally, cascading aborts occur when a transactions are allowed to read uncommitted data. However, in the unified theory, PRED executions may cause another type of cascading aborts that arise due the underlying recovery mech~ism. In particulars consider the following execution:

The abort of Ti forces the abort of Ts, a similar phenomenon to that of cascading aborts but in this case for write-write conflicts. We conclude this section by noting that Rastogi, Korth and Silberschatz [24] introduce a very general transaction model and prove a result similar to the one in [26]. Namely, that strict histories in their model allow a uniform treatment of both seri~izability and atomicity in a failure prone transaction environment. The model considered in [24] is more general than the model we use. We consider here a traditional transaction model with only read/write operations. However, our model can be extended along the lines proposed in [26] to include an arbitrary set of operations with some operations having inverse operations and some operations not having any effect on the final state of the database. Thus, most of the results presented here are applicable to transactions models that are based on semantic notion of recovery [15, 29, 19, 8, 11, 30, 21, 201. 3.

CHARACTERIZATION

OF PREFIX REDUCIBLE HISTORIES

We start this section by describing in more detail the standard recovery mechanism [lg, 7, 161. In this model, a log is maintained on stable storage for recovery purposes. Note that virtually all commercial systems employ a log-based recovery method [16], whereas deferred update methods have not been adopted in practice for performance reasons. The log is composed of a set of log records, each corresponding to a write operation. The log is used for two main purposes. First, when a transaction aborts, it is used to retrieve the tr~saction’s log records for undoing the updates of the transaction. Second, after recovery from failures, the log is traversed by a special restart routine that redoes missing updates of all committed transactions and undoes the updates of all aborted or active transactions. A common implementation technique for transaction undo and redo is the restoration of before-images and after-images of the database objects; however, undo and redo may also be based on more general operations at the storage level. We now consider the various restrictions that are imposed by log-based recovery on the order in which transactions may terminate. In particular, we are interested in the restrictions, if any, imposed on the commitment or abortion of transactions when they execute conflicting operations, i.e., WT, ww, and TW conflicts. These restrictions are as follows. To guarantee recoverable histories, if wi [x] < ~2[x] then cl < cs. Transaction undo involves the invocation of undo steps that reinstall

GUSTAVO ALONSO et al.

106

the state of database objects as of the time before the update that is undone; the order of the undo steps is the reverse of the order of the corresponding forward operations. Thus, if lul [z] < w2[z] then if both transactions abort, a2 < al or a(Tr, 2’2); if 2’1 aborts then Tz cannot commit; and if TI commits and Tz aborts, then there is no restriction on the termination order. Finally, the fact that commit decisions must be made on-line without knowing the future outcome of active transactions forces that if wr[z] < wz[z] then cl < cz; otherwise, if Tz committed first, then a subsequent abort of Tl could not be realized properly by means of log-based undo steps. These restrictions are formalized in the following definition. Definition 2 (Serializability with Ordered Termination (SOT)) A history H is SOT if it is recoverable, serializable, and for every pair of conflicting operations Wi[z] and wj[z] in H such that wi[z] precedes w, [z] and Ti and Tj are in direct conflict, the following conditions hold: 1. if Tj commits 2. If Ti aborts

then it commits

after Ti commits

and

then either it aborts after Tj aborts or H contains

a group abort a(. . . Ti . . . Tj . , .).

Note that in the definition of SOT, serializability is with respect to the committed projection of the history, while the constraints on the termination of transactions are in terms of both committed as well as aborted transactions in the history. The notion of SOT histories is related to the criterion of strong recoverability [6,9], also known as commitment ordering [23], which have been introduced in the context of multidatabase systems. Strong recoverability imposes a constraint on the ordering of commit operations (similar to but even stronger than the first condition of SOT); however, it disregards the ordering of aborts. This ordering of aborts is exactly one of the most important issues in the current paper. We now show that SOT is equivalent to PRED. Theorem

1 The class of SOT histories is equivalent to the class of PRED histories.

To prove the theorem

we need the following

auxiliary

lemma.

Lemma 1 Let H be an SOT histoy. Then all operations of non-committed transactions can be completely eliminated from I? by using the commutativity, undo, and null rules. Proof. The claim is trivially satisfied for read operations since all read operations of non-committed transactions can be dropped from any Z?. Therefore, the lemma needs to be proven only for write operations of non-committed transactions in H. Let transaction Ti be uncommitted in H that contains wi[z]. Assume that there are n operations performed on z and preceding w;* [CC]in fi. Using induction on n we show that wi[z] can be eliminated from some modification of H. If n = 0, then direct application of undo rule eliminates Wi[z] from B. Before proving the induction step we observe that if wi[z] and wj[z] are two operations that are not in direct conflict with Wi[z] preceding wj[z] in H then in the expansion fi of H, operation UT’ [x] precedes wj [z] in fi. Assume n > 0, and the claim holds for all k < n. Assume oj[z] is the
Unifying Concurrency

3.

Oj

=

If wi[x] and wj[z] are in direct

W?‘.

follows that fi using eliminate

Control and Recovery of Transactions

Wi[x]
Operation

conflict

H, then

in

wj[z] (together

107

since w;‘[z]

with w~‘[x])

can


wi’[z],

be eliminated

it

from

the inductive assumption. Then the inductive assumption can be reused again to w~[x]. If, on the other hand, transaction T; aborts before operation wj[z] in H,

then in fi the following holds: wi’[z]
our assumption

that

Now we continue with the proof of the theorem. Let H be an SOT history. Assume to the contrary that H is not prefix reducible. Then there exists a prefix of H, HpTef that is not reducible. Prom the SOT definition, it follows that Hpref is also SOT history. By Lemma 3.2, fip,,f can be modified by a finite number of applications of the commutativity, undo, and null rules into history a&f containing only operations of committed transactions in Hpref. Prom Lemma 3.2 we obtain

that

fi&

contains

operations

of transactions

that

are committed

in H,,,f

and Hpref

by a finite number of applications of the is serializable. Therefore, &$e, can be transformed commutativity rule into a serial history. Therefore Hpref is reducible. Contradiction! Thus, H is prefix reducible. Assume that H is a prefix reducible history. We prove now that H is also SOT history. Assume this is not the case, i.e., there is a history H such that H E PRED but H # SOT. If H is not serializable, then H cannot be PRED by definition, hence a contradiction. Assume H is serializable and it contains a conflict of the form wi[z]wj [z]. If H is not an SOT history, then either 1. Tj commits 2. Ti aborts If Tj commits

and either Ti does not commit and either

Tj does not abort

and T; does not commit,

or Ti commits

or Tj aborts

after

Tjor

after Ti.

then in history

H wi[x]
in the expanded history fi wi[z]
obtain prefix in the of the Cl

SOT forms the basis of a family of scheduling protocols that guarantee both transaction atomicity and serializability in a uniform way. The developed protocols are more powerful than strict two-phase locking, the standard protocol in database systems, in that they allow more concurrent histories. 4. SERIALIZATION

GRAPH

TESTING

PROTOCOLS

The main objective of this section is to exhibit a serialization graph testing protocol that guarantees prefix reducible histories. Based on results of the previous section we, therefore, need to provide a protocol that generates SOT histories. Let H be a history. We expand the notion of serialization graph SG(H) of history H by considering direct conflicts between any two transactions (not only committed ones). The edges of the graph are annotated with the type of conflict between two transactions (rw, wr, or ww).

GUSTAVO ALONSO et al.

108

Let C C {rzu, WT, WV}. Then the projection of the serialization graph SG(H) of some history H, SGc(H), is corresponding only to conflicts in C. In terms of serialization graphs we can restate a history property to be SOT as follows. Consider C = {wr, ww} and a projection of a serialization graph on C. Then history H is SOT if and only if SG(C(H)) is acyclic, and for every edge < Ti, Tj > in the serialization graph projection on C if Tj commits, then Ti commits before Tj does and if Ti aborts, then either Ti and Tj abort concurrently or Ti aborts after Tj does so. 4.1. Pessimistic

Serialization

Graph Testing

From the above constraints we can derive a serialization graph testing algorithm that generates SOT histories. Let PRECEDE(Tj) be the set of transactions Ti such that Ti -+wrlww Tj, where T; -+wT/wwT3 means that there is an edge from Ti to Tj in the serialization graph that corresponds to a WT or ww Tk such that Tj +YT/WW . . . +wrlww conflict. Let FOLLOW(Ti, . . . ,T,) be the set of transactions Tk or Tk = Tj, where 1 <= j <= n. The algorithm works as follows: 1. When an operation oi that is neither abort nor commit is submitted, add a node for Ti to the serialization graph, if it is not already there. Then add all corresponding ww, WT, and rw edges. If a cycle appears, oi is submitted. 2. When an operation ci is submitted, check whether ci for execution. After ci is executed, all commit waiting to commit are scheduled for execution. predecessor and successor lists. If, in addition, Ti removed from there. If Ti’s immediate predecessor on a queue of transactions waiting to commit.

PRECEDE(T;) is empty. If it is, submit operations of all successors of Ti that are Then, Ti is removed from all immediate is a source in the SG(H) graph, it is also list is not empty, transaction Ti is placed

then if FOLLOW(T1,. . . , T,) is empty, the 3. When an operation a(Tl, . . . , T,) is submitted, . . , T,)) is submitted for processing of a(Tl , . . . , T,) is done. Otherwise, a(FOLLOW(T1,. execution. After that, all aborted transactions are removed from a queue of transactions waiting to commit, from the serialization graph, and from all immediate predecessor and successor lists. Note that it is possible that an abort operation is issued when the transaction has already been aborted as a result of cascading aborts. In this case, the abort operation is acknowledged immediately without further action. This implies that histories such as wr[z]wz[z]uiuz are not generated by this protocol. Instead, when this sequence of submitted operations occur, the history that actually gets executed is WI [z]wz [s]a(Tl, Tz). We assume that the scheduler has some mechanism for ensuring that no two scheduled conflicting operations are processed at the same time. An example is the handshake used in (71. This is important to guarantee no interference of undo operations scheduled in rule 3 above with forward operations being scheduled after the abort. Note that some schedulers do not require this mechanism by themselves (e.g. 2PL) and we will indicate under which circumstances this requirement can be relaxed when discussing specific protocols. The correctness of this algorithm can be established by showing that this algorithm generates only SOT histories. Indeed, by rule 1, each history generated by the algorithm is serializable. To show that the history satisfies other requirements of SOT histories the following arguments are used: l

In case of a wi[z]rj[z] direct conflict, ci < cj is ensured by rule 2. This is because Tj cannot commit until its PRECEDE set is empty. By definition, Ti E PRECEDE(Tj), thus for PRECEDE(Tj) to be empty, Ti must have committed or aborted (in the latter case Tj will also get aborted). Hence, this algorithm generates recoverable histories.

Unifying Concurrency

l

Control and Recovery of Transactions

109

In case of wi[z]wj[z] direct conflict either ci < cj (by rule 2, similar to the WT conflict) or ci < aj (once Ti commits it does not impose any restrictions on Tj) or aj < ci (rule 3 does not depend on the PRECEDE set) or aj < ai (the abortion of Tj does not affect Ti at all, thus Ti can be committed or aborted once Tj aborts). No other combination is possible since rule 2 does not allow Tj to commit until Ti terminates, and rule 3 forces the abortion of Tj before Ti can be aborted.

The functionality

of the protocol will be elucidated by the following examples:

Example 1 Let us assume that so far the following history has been executed w1 [s]wz[z]wZ[y] and operation wi[y] is submitted. The scheduler tests acyclicity of SG(H) (viz. figure 4.1) and finds a cycle. Therefore wi[y] is rejected and by rule 1 of the protocol a(Tl) is submitted instead. To execute a(Tl) the scheduler finds FOLLOW = {Tl,Tz} and by rule 3 submits a(Tl,Tz) for execution.

Fig. 1.

Example 2 Let us assume that so far w~[z]T~[x] has been executed and cs is submitted. The scheduler finds that PRECEDE(T2) = {TI}. Th ere fore, TZ is placed on the waiting queue of transactions waiting to commit by rule 2 of the protocol. After that cl is submitted. Since PRECEDE(Tl) = 0, cl gets immediately submitted for execution (rule 2). Once cl is executed, transaction T2 is removed from the waiting queue and cs gets submitted for execution (rule 2).

Tloh

T2 Fig. 2.

Protocols along these lines have been proposed in [25, 14,221. These protocols allow non-strict histories and thus have to postpone commits and deal with cascading aborts under certain conditions. However, none of this previous work has recognized the importance of ordered termination. The bookkeeping of the protocol in [14] ignores write-write conflicts and thus cannot ensure correct (log-based) recovery. The protocol suggested in [25], on the other hand, is overly restrictive in that read-write conflicts lead to the postponing of commits and to cascading aborts unnecessarily. Finally, the protocol in [22] appears to be most closely related to the algorithm above; however, the recovery in this approach is based on object versions rather than undo logging. In addition, and most importantly, all three proposals lack a formal correctness model for the interdependencies of concurrency. 4.2. Optimistic Serialization Graph Testing Compared to non-optimistic protocols, optimistic protocols process transactions’ termination operations in a slightly more complex way. To reduce the overhead of concurrency control, conflict cycle detection (or a sufficient certification test) is performed lazily (i.e., less frequently) in that it is initiated only upon processing a transaction’s commit request. Therefore, at the time of submission of a termination operation, the serialization graph may contain cycles (whereas all non-optimistic protocols guarantee acyclic serialization graph at all times). Consequently, we must add one more test to the algorithm to be suitable even for optimistic scheduling. Similarly to its pessimistic counterpart, the optimistic algorithm is formulated in three steps:

110

GIJSTAVO ALONSO et al.

1. Operations different from commit and abort are immediately ing edges have been added to the graph.

scheduled,

after the correspond-

2. When operation ci is submitted, the graph is tested for cycles. If there are no cycles, ci gets processed in the same way as in the pessimistic algorithm. Otherwise, c; is rejected and ai is submitted. 3. When rithm.

operation

a(Tl , . . . , T,) is submitted,

The difference with the pessimistic trated by the following example:

version

the algorithm

of serialization

proceeds

graph

testing

as the pessimistic

protocol

algo-

will be illus-

Example 3 Let us assume that history wi[z]w~[z]wz[y]wi[y] has been executed so far (note the difference with example 1 when the last operation wi[y] gets rejected by the scheduler). The commit operation of TI, cl, is submitted. The scheduler tests the serialization graph SG(H) for acyclicity. Since a cycle is found (the serialization graph is the same as that in the figure 4.1), cl is rejected and a(Tl) is submitted instead by rule 1 of the protocol. As in example 1 the set for execution by rule 3. FOLLOW(T1) = {Tl,Tz} is found and a(Tl,T 2) is submitted As in the previous section, we assume that there exists some mechanism ensuring that no two scheduled conflicting operations are performed at the same time. The proof of this algorithm is similar to the proof of its pessimistic counterpart. 5. LOCKING

PROTOCOLS

Although a serialization graph testing approach (pessimistic or optimistic) recognizes, in general, all SOT histories, it may incur unacceptable overhead. Many widely used concurrency control protocols use locking as a basic primitive for synchronization. Traditionally, there are two types of relationships between locks: shared and non-shared. For example, read locks can be shared but a write lock cannot be shared with any other lock. Two phase locking [13] is the most widely accepted concurrency control protocol. In what follows, we provide locking protocols that expand standard 2PL, both in its optimistic and pessimistic versions, as well as an extension of 2PL with ordered shared locks [2]. 5.1. Pessimistic

Two Phase Locking

In order to guarantee that the basic 2PL protocol generates histories that are from SOT, we must extend the protocol with additional rules that constrain the ordering of commits and aborts of all transactions. In order to do so, we assume that sets PRECEDE(Ti) and FOLLOW(Ti) are maintained for each transaction Ti. This can be easily done by requiring the 2PL scheduler to maintain these list based on the lock table information. We also make a standard assumption that with each data item x there exist read and write locks that are granted to transactions before performing the corresponding operation on data item x. We skip the details of the protocol, since it is a combination of the 2PL protocol for read/write operations with rules 2 and 3 of the pessimistic serialization graph testing protocol. Not every history that is SOT can be generated by the standard two-phase locking protocol (2PL). The 2PL protocol requires that once a transaction releases a lock, it cannot acquire any more locks. Consider, for instance, the following history: w1[4w2[4w3[?+1

[?/]cSclcZ

This history is SOT, but it cannot be generated by a 2PL scheduler, since TI releases the write lock on z before acquiring a write lock on y. On the other hand, there are 2PL histories that are not SOT. Consider, for instance, the following history: ‘wl [+J2

[&a2

Unifying Concurrency

Control and Recovery of Transactions

111

If operation ci is placed on a waiting queue of transactions waiting to commit, then we claim that such action cannot generate any additional deadlocks. Indeed, if we assume that there is a deadlock in the system, then there is a waiting cycle Ti + T2 ---) . . . ---) Tk + TI where each transaction is either waiting for a lock or to execute a commit operation. At least one of the transactions in the cycle is waiting to commit, otherwise the cycle is not caused by the SOT rules. Assume that Tl waits for Tz to commit. Then the serialization graph contains an edge TZ + TI. In such case, T2 cannot wait for any lock. Thus, since the 2PL scheduler must generate serializable histories, the waiting cycle implies a cycle in the serialization graph, which cannot happen. The protocol described above may, however, generate several aborts as a result of one abort operation. This phenomenon is known as cascading aborts [7]. Using the strict 2PL protocol [7] would guarantee an absence of cascading aborts. However, the proposed non-strict 2PL protocol offers higher concurrency. (while handling cascading aborts in the correct way). We can also show that if the strict 2PL protocol is used, we do not need any additional mechanism to ensure that no two scheduled conflicting operations are performed at the same time. Use of locks in any 2PL guarantees that there cannot be any interference between two forward operations. If the locks are not released after submission of abort, but rather after performing all undo operations related to the abort of a given transaction, it is easy to see that locks also guarantee no interference between undo and forward operations and two undo operations.

5.2.

Optimistic

Two Phase

Locking

Typically, optimistic 2PL schedulers [7] do not maintain the serialization graph SG(H), but rather Read and Write sets of each transaction since these structures are necessary for certification. Unfortunately, Read and Write sets do not convey any information about the order of operations. However, we can show that instead of the serialization graph we can maintain much simpler structures from which all the necessary information can be recovered. With every data item x we maintain a conflict list, cl(z), associated with it. Entries of the conflict list are of the form < transacticmid, access >. Whenever transaction Ti performs operation o~[z] the entry < i,o > is inserted at the tail of cl(x) ( c I( x ) is initially empty). Whenever Ti terminates, all entries associated with Ti are removed from all cl(x)‘s. Notice that the order of entries in cl(x) corresponds to the order of operations being performed on x. To simplify the exposition, we will say that Tj C-conflict follows (precedes) Ti if there exists some list cl(x) such that there is some entry < j, qj > following (preceding) entry < i,pi > in cl(x), where pi[z] is an operation of Ti, qj[x] is an operation of Tj and < p,q >E C, where C C {ww, wr, TW}. Clearly, Tj C-conflict precedes Ti iff Tj and Ti are in direct C-conflict and Tj C-conflict succeeds Ti iff T, and Tj are in direct C-conflict. Therefore conflict lists can be viewed as a faithful encoding of serialization graphs. Obviously, at any moment we could construct the serialization graph out of the conflict list, but as it will turn out it is not necessary. The formulation of the protocol follows: 1. All read and write operations are maintained.

are scheduled

as they are received.

In addition,

all lists cl(x)

2. If c, is received, then all cl(x) are checked. If none of them contains some access that is in conflict with some of Ti’s accesses (i.e. at least one of them is write-access) then if there are no transactions that {ww, wr}-conflict precede Ti, then Ti commits. If, in addition, there are no transactions that {rw}-conflict precede Ti then all entries associated with it are removed from all conflict lists cl(z). If there are transactions that {ww, wr}-conflict precede Ti, then Ti is put on the waiting list. If Ti has at least one access that conflicts with some other access, then ai is submitted. 3. Ifa(Ti,... , Tn) is received, let 7 be the set of transactions that {ww, wr}-conflict follow Ti. Then schedule undo operations to all write operations of transactions in 7. Finally, remove all entries associated with any of the transaction in 7 from all conflict lists cl(z).

GUSTAYO ALONSO et al.

112

Correctness of the ~gorithm follows from the correctness of the serialization graph ~gorithms and the considerations carried out prior to the protocol’s formulation. Like the pessimistic version of the 2PL, the optimistic 2PL protocols cannot generate all prefix reducible histories.

Recently in [Z], a new locking primitive was introduced that allows a new relationship, referred to as orderer s~uT~~g. Ordered sharing can be used with two-phase locking to eliminate the blocking between read and write operations. For example, in order to eliminate read-write blocking, a transaction Tj can be granted a write lock on an object even if a transaction T’ holds a read lock on the same object. We say that there is an ordered shured reiut~o~sh~pfrom Ti’s read lock to Tj’s write lock. The advantage of using ordered sharing is that it eliminates blocking between read and write operations, but can be used to restrict the execution of commit or abort operations. Serializability with ordered termination can be enforced through ordered sharing locking by observing the following rules: must obtain a read (write) lock on an object before executing a read (write) operation on that object.

1. L;oclcRule+ A transaction

2. Two Phase Rule. A transition

may not acquire any locks once it has released a single lock,

3. Lock ~~qu~~~t~o~Rule. If Tj acquires a lock with an ordered shared relationship with respect to a lock held by another transaction Ti, the corresponding operation of Tj must be executed after that of Tb. 4. Lock Re~~nq~~shRule. If tr~saction Tj acquired a lock with an ordered shared relationship with respect to a lock held by transaction Ti and Ti has not released any of its locks, then Tj cannot release any of its locks. 5. Dellaged commitment land ubo~~o~. If Tj reads from Ti then Tj cau commit only if Ti has committed, and if Tj overwrites a value written by Ti thea Tj does not commit before T; terminates and Ti does not abort before Tj terminates. The ordered sharing of locks is necessary because SOT histories do not impose any restriction on the order in which the operations are executed, as long as the overall history is serializable. The rules of the two-phase locking with ordered sharing guarantee the serializability of the committed projection of the history. The delayed commitment and abortion rule ensures that the expanded history is serializable. This protocol accepts a strict subset of SOT histories. However, we note that the SOT histories that are not accepted by this protocol are not ordered preserving serializable histories [2]f . In order to avoid the the problem of cascading aborts, both for read and write operations, a restricted form of ordered sharing can be used where non-shared locks are used from writers to both readers and writers and locks are released only when transactions terminate, Hence, a non-shared relationship from write locks to both read and write locks is used in this protocol. For write after read conflicts, locks with ordered sharing are used and transactions must adhere to the ordered sharing rule. Since write locks are held until commit* tr~sactions cannot read or write uncommitted data. Hence all executions resulting from this protocol avoid cascading aborts. Note that this restriction, i.e., non-shared locks and releasing write locks at commit, results in strict executions, however, it still &lows more concurrency than strict 2PL since read locks can be shared, i.e., read-write conflicts may occur before transactions terminate. [If explores techniques to eliminate this restriction and still avoid cascading aborts. tOrder-preserving history [lo, 51.

serializable histories maintain the order of non-interleaved transactions

in the equivalent serial

Unifying Concurrency

Control and Recovery of Transactions

6. TI~ESTA~P

~~E~N~

113

PROTOCOLS

In this section we discuss how timestamp ordering protocols can be easily augmented with additional mechanisms to generate SOT histories. We make a standard assumption that each transaction has a unique timestamp value ts(T;) associated with it.

As in the pessimistic 2PL protocol, we assume that in order to guarantee the proper ordering among commit and abort operations, the two sets P~~C~~E(Ti~ and HOLLOW are maintained with each transaction Ti. We skip the details of the protocol as they are a combination of basic TO rules for read and write operations and rules 2 and 3 of the pessimistic serialization graph testing. We can again show that not every SOT history can be generated by TO protocol. One such example is history H: H : w? klw We assume that am

< us

I+2

Cl

in H.

The same data structures as by optimistic 2PL schedulers are maintained also by optimistic TO schedulers. The only difference is in performing the certification test. Rather than testing conflicting accesses in all conflict lists cl(z), we test whether there does not exist some transaction T’ {WY,TW,ww}-conflict following Ti such that ts(Ti) > ts(Tj). Besides this difference, the formulation of optimistic TO protocols is the same as that of the optimistic 2PL protocol. Similarly to the pessimistic case, optimistic TO cannot generate all possible SOT histories. 7. CONCLUSIONS

Concurrency control and recovery theory is often presented as a theoretic~ly sound approach to the problem of transaction management. However, this theory has been built upon particular assumptions that constrain the correct executions. One of these assumptions is that the recovery and concurrency control mechanisms are implemented as independent modules (the lock manager and the log manager). Recently, a unified theory for concurrency control and recovery has been proposed in [26]. However, this work takes a strictly theoretical viewpoint, and no protocols are proposed for recognizing these correctness classes. In this paper, we have extended this theory by providing efficient dynamic schedulers to recognize classes of histories based on the unified theory. The model presented in this paper is restrictive in several ways. First, the model relies on a particular implementation of the undo operation based on logs. We are working on a more general model of undo operations that will allow more concurrency. In particular, we have applied the unified theory to integrate the traditions lock and log m~agers used in commercial databases [l]. We also intend to generalize the proposed theory to other recovery mechanisms such as deferred updates. The other restriction is that the model is limited to read and write operations. However, both the model and the developed protocols can be generalized to transactions with semantically rich operations. Aborts of such transactions involve compensating operations, and our model is being extended to deal with this kind of semantic recovery. Ultimately, semantically rich operations should be allowed to invoke further operations; to deal with the resulting operation trees, we aim to combine the framework of this paper with the models of multilevel transactions and open nested transactions 111, 5, 301. We are currently working on such extensions of the model and protocols. acknowledgements and 1~-91~79~~

-

This material is based in part upon work supparted

and by grant from Hewlett-Puked

was on one year sabbatical

Corporation.

by NSF grants IRI-9221947,

~~-9~12902

This work has been performed while Y. Breitbart

and R. Vingralek was visiting for 8 months the database research group at ETH, Zurich.

114

GIJSTAVOALONSO et al.

REFERENCES [I]

G. Alonso, D. Agrawal, and A. El Abbadi. Technical Report, Department of Computer (1993).

A unified implementation of concurrency control and recovery. Science, University of California at Santa Barbara, TRCS93-19

[2]

D. Agrawal and A. El Abbadi. Locks with Constrained on Principles of D&abase Systems, pp. 85-93 (1990).

[3]

D. Agrawal, A El Abbadi, and A. K. Singh. Consistency and Orderability: Semantics-Based Criteria for Databases. ACM l?ansactiona on Data Bases, 18(3):460-486 (1993).

[4]

D. Z. Badal. Correctness of Concurrency Control and Implications ceedings of COMPSAC Conference, pp. 588-593, (1979).

[5]

C. Beeri, P. A. Bernstein, and N. Goodman. of the ACM, 36(2):230-269 (1989).

[6]

Y. Breitbart, tinsactions

[7]

P. A. Bernstein, V. Hadzilacos, Addison-Wesley, 1987.

[8]

B.R. Badrinath and K. Ramamrithan. Semantic-based nansacliona on Database Systems, 17(l) (1992).

[9]

Y. Breitbart, A. Silberschatz. Strong recoverability in multidatabase systems. Proceedings of the 2nd IEEE Inlemational Workshop on Research Issues on Data Engineering: Transaction and Query Processing (RIDETQP), Tempe, Arizona (1992).

[lo]

P. A. Bernstein, D. W. Shipman, and W. S. Wong. Formal Aspects of Serializability Control. IEEE Transactions on Software Engineering, 5(5):203-216 (1979).

[ll]

C. Beeri, H.J. Schek, and G. Weikum. Multi-level transaction management, theoretical In Proceedings of International Conference on Extending Database Technology (EDBT)

Sharing. In Proceedings

in Distributed

Databases.

A Model for Concurrency in Nested Transactions

D. Georgakopoulos, M. Rusinkiewicz, A. Silberschatz. on Software Engineering, 1’7(9) (1991). Concurrency

and N. Goodman.

The Concurrency [12] M. A. Casanova. Computer Science. Springer-Verlag,

of Ihe Ninth ACM

Control Problem Berlin (1981).

control

and recovery

control:

[17] V. Hadzilacos.

Transaction

Processing:

Concepts

Principles of Transaction-Oriented

[19] E. Levy, H.F. Korth, and A. Silberschatz. of Distributed Computing (1991).

IEEE

ayalema.

in database

ACM

in Database Concurrency art or practical need? (1988). Notes

in

and Predicate Locks in

processing

Journal

of the ACM, 35(l),

Database Recovery.

A theory of relaxed atomicity.

[20] N. Lynch, M. Merritt, W. Weihl, and A. Fekete. (1993). [21] E. Moss, N. Griffeth, and M. Graham. Abstraction ference on Management of Data (1986).

Atomic

in a distributed

database.

ACM

and Techniques. Morgan Kaufman (1993).

A Theory of Reliability in Database Systems.

[18] T. Hkder and A. Reuter. 15(4), 287-317 (1983).

Systems. Journal

Volume 116 of Lecture

Systems.

Pro-

Reliable scheduling of of database transactions for unreliable of Database Systems (1984).

[15] H. Garcia-Molina. Using semantic knowledge for transaction Transactions on Database Systems, 8(2), 1983. [16] J. Gray and A. Reuter.

In IEEE

Beyond commutativity.

[13] K. P. Eswaran, J. N. Gray, R. A. Lorie, and I. L. Traiger. The Notions of Consistency a Database System. Communications of the ACM, 19(11):624-633 (1976). [14] M.H. Graham, N. Griffeth, and B. Smith-Thomas. systems. In PTOC. ACM Symposium on Principles

Correctness

On rigorous transaction scheduling.

concurrency

of Database

Symposium

Z%ansactions.

ACM

In Proceedings

121-145 (1988). Computing

of ACM on Principles

Morgan Kaufmann,

in recovery management.

Surveys,

In PTOC. ACM

San Mateo, CA SIGMOD

[22] E. Nett, R. Kroeger, and J. Kaiser. Providing recoverability in a transaction oriented distributed system. In PTOC. 6th IEEE International Conference on Distributed Computing Systems (1986).

Con-

operating

[23] Y. Raz. The principle of commitment ordering, or guaranteeing serializability in a heterogeneous, multi resource manager environment using atomic commitment. Proceedings of the 18th International Conference on Very Large Data Bases (VLDB) (1992). [24] R. Rastogi, H.F. Korth, and A. Silberschatz. Strict histories in object-based database Report MITL-TR-35-92, Matsushita Information Technology Laboratory (1992). [25] A. Silberschatz. A case for non-two-phase Soflware Engineering,9(4) (1983).

locking protocols

that ensure atomicity.

systems.

Technical

Transactions

on

[26] H. J. Schek, G. Weikum, and H. Ye. Towards a Unified Theory of Concurrency Control and Recovery. Proceedings of the ACM Symposium on Principles of Database Systems, pp. 300-311 (1993).

In

IEEE

[27] It. Vingralek, Y. Breitbart, H.-J. Schek, G. Weikum. Concurrency Control Protocols Guaranteeing and Serializability. Technical Report 199, Department of Computer Science, ETH Zurich (1993).

Atomicity

Unifying Concurrency

[28] J. S. M. Verhofstad. (1978).

Recovery

Techniques

Control and Recovery of ‘Bansactions

for Database

f29] W.E. Weihl. The impact of recovery on concurrency on Principles of Database Systems (1989).

Systems.

ACM

control. data types.

[30] G. Weikum. Principles and realization strategies of multilevel transaction Database Systems, 18(l) (1991).

Computing P~ceedi~gs management.

115

Surveys,

10(2):X7-196

of the ACM Symposium A CM Transactions

eta