On the assessment of learning in competence based knowledge space theory

Journal of Mathematical Psychology (

)

–

Contents lists available at ScienceDirect

Journal of Mathematical Psychology journal homepage: www.elsevier.com/locate/jmp

On the assessment of learning in competence based knowledge space theory Luca Stefanutti *, Debora de Chiusole University of Padua, Italy

highlights • • • •

In CbKST the student’s skills are inferred from her responses to a set of items. There is no one-to-one correspondence between competence and performance states. There could be no way for establishing whether learning has occurred or not. We establish conditions for resolving this impasse.

article

info

Article history: Received 4 March 2016 Received in revised form 10 August 2017 Available online xxxx Keywords: Competence based-knowledge space theory Knowledge assessment Learning assessment Well-graded competence space Outer fringe Exclusive skill function

a b s t r a c t The objective of an assessment in competence based-knowledge space theory (Cb-KST) is to infer the skills of an individual from her responses to a subset of problems. A major issue in this approach is the lack of a one-to-one correspondence between the competence states and performance states. The assessment is possible, but it cannot go beyond an approximation. The problem becomes even more serious if Cb-KST is used for the assessment of learning, since changes caused at the competence level may not be represented by changes at the performance level. The consequence is that there could be no way for establishing whether learning has occurred or not. This impasse can be resolved for the class of conjunctive skill functions by pretending that the competence space induced by the skill function is well-graded. Under this condition an individual can make tangible progresses along the performance structure by learning one skill at a time, until full mastery is eventually reached. If the competence structure is a space satisfying certain compatibility conditions and the skill function satisfies a special property, called exclusiveness, then well-gradedness can be assured. A test of exclusiveness of a conjunctive skill function is described and exemplified. © 2017 Elsevier Inc. All rights reserved.

1. Introduction The theory of knowledge spaces (KST, Doignon & Falmagne, 1985, 1999; Falmagne & Doignon, 2011; Falmagne, Koppen, Villano, Doignon, & Johanessen, 1990) provides a valuable mathematical framework for the development of computerized web-based systems for the assessment of knowledge and learning. The very basic and central notion of the whole theory is that of a knowledge state, which is operationally defined as the set K of all those problems an individual is capable of solving in a finite domain of knowledge Q . As a single individual is characterized by a knowledge state, a whole population of individuals is represented by a knowledge structure, which is a pair (Q , K) where K is a collection of knowledge states containing at least Q and the empty set. author. * Corresponding E-mail address: [email protected] (L. Stefanutti).

A special type of knowledge structure is called knowledge space, and plays a central role in KST because of the closure under union property which is assumed on its collection of states. A knowledge structure K is closed under union when, given any subfamily F ⊆ ⋃ K, the union F is in K. This assumption inspired the KST’s authors since it has a reasonable empirical interpretation: if two students having two different knowledge states are involved in an extensive interaction, it is plausible that, at some point, their knowledge arise from the union of their initial knowledge states. Of course, this situation may not happen, but the knowledge structure should cover this case. As far as learning is concerned, it is conceivable that a student whose knowledge state is K , after learning some new material will end up to a new knowledge state K ′ that, if no forgetting occurs, is a strict superset of K . If the collection of all possible knowledge states is correctly represented by K, then learning can be described as a chain K = K1 ⊂ K2 ⊂ · · · ⊂ Kn = K ′ of knowledge states in K. Behind this description of learning there is the simple idea that,

http://dx.doi.org/10.1016/j.jmp.2017.08.003 0022-2496/© 2017 Elsevier Inc. All rights reserved.

Please cite this article in press as: Stefanutti, L., de Chiusole, D., On the assessment of learning in competence based knowledge space theory. Journal of Mathematical Psychology (2017), http://dx.doi.org/10.1016/j.jmp.2017.08.003.

2

L. Stefanutti, D. de Chiusole / Journal of Mathematical Psychology (

in moving from the empty set of items to the total mastery Q , a student must traverse a number of states of intermediate mastery. For a student in knowledge state K ∈ K, the ‘‘smallest possible learning step’’ would consist of learning exactly one new item, among those in Q \ K . Under a pedagogical perspective, this ‘‘smallest step’’ plays an important role, since it represents the easiest way a student has for making some tangible progress in learning. The notion of outer fringe (Doignon & Falmagne, 1999) formalizes these ideas. For an arbitrary knowledge state K ∈ K, it is defined as K ◦ = {q ∈ Q \ K : K ∪ {q} ∈ K}. Informally, the outer fringe of K is regarded as ‘‘what the student is ready to learn from her knowledge state’’. It should be noted that the outer fringe of a knowledge state K could even be empty, meaning that there is no way for a student in knowledge state K to make progresses by learning exactly one new item. There is a special class of knowledge structures in which every knowledge state has a nonempty outer fringe. They are known as the well-graded knowledge spaces or, in the more recent literature on KST (Falmagne, Albert, Doble, Eppstein, & Hu, 2013; Falmagne & Doignon, 2011) learning spaces. A learning space is a ∪-closed subfamily of 2Q in which the additional condition holds true that, for every nonempty knowledge state K ∈ K there is some item q ∈ K such that K \ {q} ∈ K. ‘‘Being capable of solving a problem’’, the very basic statement at the core of the definition of a knowledge state, is a very pragmatical one. Early developments of KST did not pay much attention to ‘‘how problems are solved by individuals’’, or to which skills, abilities and knowledge are involved in the solution of the problems belonging to the knowledge state of an individual. Indeed, rather than problems, what people learn are the skills and the knowledge necessary for solving problems. Skills that have been learned from the solution of one problem can then be transferred by individuals to the solution of other problems. A number of theoretical extensions have been worked out by various authors (Doignon, 1994; Düntsch & Gediga, 1995; Falmagne et al., 1990; Gediga & Düntsch, 2002; Heller, Stefanutti, Anselmi, & Robusto, 2015; Heller, Ünlü, & Albert, 2013; Korossy, 1993, 1997, 1999) to incorporate the cognitive level of the skills into the theory. Basic KST equipped with these extensions is known as competence based-knowledge space theory (Cb-KST, Heller, Augustin, Hockemeyer, Stefanutti, & Albert, 2013; Heller, Ünlü et al., 2013) and is divided into two parallel and interdependent levels: the performance level and the competence level. The knowledge domain Q and the knowledge state K ⊆ Q of an individual are located at the performance level. At the competence level a set S of skills, required for solving the problems in Q , is assumed and the individual knowledge is represented by a subset C ⊆ S, named the competence state. The two levels are connected by two mappings, known as the skill function and the problem function (Düntsch & Gediga, 1995). By assigning skills to problems, the skill function goes from the performance level to the competence level. The problem function instead goes in the opposite direction: given any competence state C , it specifies the subset K ⊆ Q of problems that can be solved by C . Cb-KST has been used as the reference theory in the development of existing computer systems for skill assessment and learning at the various school grades and for high-level education. To give some examples, we mention the APeLS system (Hockemeyer, Conlan, Wade, & Albert, 2003), for learning in Newtonian mechanics, the iClass system for self-regulated personalized learning (Heller, Augustin et al., 2013), and the two educational games ELEKTRA (http://www.elektra-project.org/) and 80Days (http:// www.eightydays.eu/) developed within projects funded by the

)

–

European Commission. Furthermore, the Knowlab prototype (http: //www.knowlab.org/) has been developed at the University of Padua and it is currently applied for the assessment and learning of statistics at the University level. Some probabilistic models together with empirical applications were also developed in Cb-KST (Anselmi, Robusto, & Stefanutti, 2012, 2013; Anselmi, Stefanutti, de Chiusole, & Robusto, 2017; de Chiusole, Anselmi, Stefanutti, & Robusto, 2013; de Chiusole & Stefanutti, 2013; Lukas & Albert, 1993; Robusto, Stefanutti, & Anselmi, 2010; Stefanutti, Anselmi, & Robusto, 2011). A parallel framework in which cognitive assessment was studied in depth, is the theory of the cognitive diagnostic models (CDM; Bolt, 2007; de la Torre, 2009; DiBello & Stout, 2007; Junker & Sijtsma, 2001; Tatsuoka, 2002, 2009) in which the main interest is at the competence level. There are close connections between CDM and Cb-KST, as recently pointed out by Heller et al. (2015). The objective of an assessment in Cb-KST is to infer the competence state of an individual from her responses (coded as ‘‘correct’’ or ‘‘wrong’’) to some suitable subset of problems in Q . Essentially, the assessment involves two separate tasks: (1) infer the knowledge state K from the responses to the problems and (2) derive the competence state C from K . In task (1) inference is usually probabilistic, due to the fact that the answer to a problem could be the result of a careless error or a lucky guess (Falmagne & Doignon, 1988a, b). Task (2) instead is deterministic and it is based on the problem function. Since this function is a mapping from competence states to knowledge states, if it is a bijection, its inverse can be applied for inferring the competence state corresponding to the knowledge state assessed in step (1). However, as pointed out by Heller et al. (2015), the problem function might fail to be a bijection and therefore an inverse function could not exist. This fact poses an unescapable problem to knowledge and learning assessment under the framework of Cb-KST: individual assessment cannot go beyond an approximation, in which only a portion of the full set S of skills can be classified as ‘‘mastered’’ or ‘‘not mastered’’, whereas the status of the remaining skills is unknown. The problem becomes even more stringent when repeated assessments are carried out in different occasions on the same individual with the objective of monitoring learning. This is a typical task, for instance, of a computer-based tutoring system, which constantly switches between assessment sessions and teaching and training sessions. Due to the lack of a one-to-one correspondence between the competence states and the knowledge states, a change in the competence state might not be reflected by a corresponding change in the knowledge state. In the worst situation, changes at the competence level could even produce no change at all at the performance level. This could be a serious problem for any tutoring system developed under the Cb-KST framework. The theoretical work presented in this article develops upon the notion of an effective skill. Once learned by an individual in competence state C , an effective skill produces a corresponding change in her knowledge state. If such a special skill exists and can be pointed out for every possible competence state, then an individual can always make observable progresses by gradually learning one skill at a time, until full mastery is eventually attained. For the class of the conjunctive skill functions it is shown that the existence of (at least) one effective skill per competence state can be assured if the collection of the so-called minimal competence states is a well-graded space. A consequence of these requirements is that they assure the union-closure of the competence structure, and the intersection-closure of the knowledge structure. If, at a first look, this situation seems incoherent with the original KST theory, in which the ∪-closure of the knowledge structures was recommended, the nice result is that this property moves from the performance level to the competence level, which is the focus in Cb-KST.

Please cite this article in press as: Stefanutti, L., de Chiusole, D., On the assessment of learning in competence based knowledge space theory. Journal of Mathematical Psychology (2017), http://dx.doi.org/10.1016/j.jmp.2017.08.003.

L. Stefanutti, D. de Chiusole / Journal of Mathematical Psychology (

If the collection of all competence states is a space satisfying certain compatibility conditions, this property corresponds to a property of the skill function, named exclusiveness. Besides these theoretical developments, an algorithm is described that can be used for testing the exclusiveness of a conjunctive skill function. 2. Competence based KST: a brief overview Let Q be a finite set of items, and denote with S a finite set of skills that are relevant for solving the items in Q . From a purely deterministic perspective the general idea behind Cb-KST is to characterize each of the items in Q with one or more subsets of skills in S, each of which is minimally sufficient for solving it. This idea is captured by the notion of a skill function (Düntsch & Gediga, S 1995), a triple (Q , S , µ) where µ : Q → 22 is a function assigning a collection of subsets of S to each of the items in Q . The skill function satisfies the following three basic properties for every item q ∈ Q : (1) µ(q) ̸ = ∅; (2) M ̸ = ∅ for all M ∈ µ(q); (3) the subsets in µ(q) are pairwise incomparable. Skill functions are a special class of the skill multimaps introduced by Doignon (1994), which only need satisfy conditions (1) and (2) above. Doignon (1994) and Heller, Ünlü et al. (2013) examine two special classes of skill functions, respectively named conjunctive and disjunctive. A skill function is conjunctive if it assigns exactly one nonempty subset of skills to each of the items, that is for every item q it has the form µ(q) = {M }, with ∅ ⊂ M ⊆ S. It is disjunctive if each of the subsets of skills assigned to an item is a singleton, that is, for any q ∈ Q , it has the form µ(q) = {{s} : s ∈ M }, with ∅ ⊂ M ⊆ S. It should be noted that in general, arbitrary skill functions might belong to neither of these two classes. However, especially conjunctive skill functions play a central role in this article. The skill function links the competence level to the performance level. The problem function does the opposite: it is an order preserving function p : 2S → 2Q such that p(∅) = ∅ and p(S) = Q . Düntsch and Gediga (1995) have shown that, for finite Q and S, there is a bijection between the family of all skill functions µ and that of all problem functions p. The two levels provide different ways of characterizing a student’s knowledge. When the skill function µ is established, the performance state K ⊆ Q is completely specified by the underlying competence state C ⊆ S, by an application of the problem function p defined by: p(C ) = {q ∈ Q : M ⊆ C for some M ∈ µ(q)},

(1)

S

for every competence state C ∈ 2 . Heller, Ünlü et al. (2013) show that, for the conjunctive skill functions, the problem function p preserves intersections (i.e., p(C ∩ C ′ ) = p(C ) ∩ p(C ′ ) for all C , C ′ ∈ 2S ) whereas, for the disjunctive skill functions it preserves unions (p(C ∪ C ′ ) = p(C ) ∪ p(C ′ ) for all C , C ′ ∈ 2S ). If p is the problem function corresponding to skill function µ, and C ∈ 2S is a competence state, then K = p(C ) is the performance state delineated by C via the skill function µ. The collection Kp = {p(C ) : C ∈ 2S }

)

–

3

Accordingly, given any competence state C ∈ C ,

[C ] := {C ′ ∈ 2S : C ′ ∼p C } is the equivalence class of competence state C , induced by p. The class [C ] collects all subsets of S delineating the same performance state as C . Moreover, if the skill function is conjunctive then the equivalence class of C ∈ 2S is closed under⋂intersection and thus it contains its greatest lower bound, that is [C ] ∈ [C ]. It should be realized that the greatest lower bound of [C ] is, indeed, the collection of all skills that are minimally sufficient for delineating the performance state K = p(C ). In the sequel we will refer to this subset as the floor of C , and denote it by

⌊C ⌋ =

⋂ [C ].

(2)

If the skill function is conjunctive then every competence state C ∈ 2S has a floor and, as observed by Heller et al. (2015), Proposition 11, the collection Cp := ⌊C ⌋ : C ∈ 2S

{

}

of all floors is closed under union and thus it forms a competence space which, henceforth, will be named the space of floors (or floor space) induced by the problem function p. The nice result is that, while performance states and competence states are not necessarily in a one-to-one correspondence, for every performance state K there is exactly one floor delineating K . More precisely, by Proposition 10 in Heller et al. (2015), the performance structure Kp is isomorphic to Cp . Due to this one-to-one correspondence, a performance state K ∈ Kp conveys unambiguous information on the minimum subset of skills that a student masters, that is the floor corresponding to K . We close this section by mentioning that, although the whole article is confined to conjunctive skill functions, it might be possible to extend it to disjunctive skill functions. In fact, there is a well-known duality between the two types of skill functions. 3. Generalization to competence structures So far it has been assumed that every subset of S is a competence state, so that the collection of competence states is the full power set 2S . In many situations this is not the case, and the collection of competence states is only a strict subset C of the full power set 2S . Logical or pedagogical dependencies among the skills may provide reasons for excluding a number of subsets of S from the collection of the competence states. Theoretical implications of considering a strict subset C of 2S as the collection of competence states were investigated by Korossy (1997) in his competence/performance framework. Although the details of this approach are not considered here, the notion of a competence structure is used in this article to refer to any collection C ⊆ 2S of competence states containing at least the empty set and the full set of skills. The term competence space, instead, is used to refer to competence structures that are closed under union. The definitions of an equivalence class of states and that of a floor can be readily adapted to the more general case C ⊆ 2S .

of all performance states delineated by the competence states in 2S forms the performance structure delineated by 2S via µ. If µ is conjunctive, then Kp is closed under intersection. If µ is disjunctive, then Kp is closed under union. As Heller et al. (2015) have shown, the problem function p need not be one-to-one. In Cb-KST this fact constitutes a serious problem when the competence state of a student has to be inferred from her performance state. The problem function p induces an equivalence relation ∼p on the competence states C ∈ 2S . For any two states C , C ′ ∈ 2S , let

the quotient structure ⋂ (set of equivalence classes) of C . Moreover, indicate with ⌊C ⌋C = [C ]C the C -floor of C , so that

C ∼p C ′ if and only if p(C ) = p(C ′ ).

Cp = {⌊C ⌋C : C ∈ C }

Definition 1. Let (S , C ) be a competence structure, and p : 2S → 2Q be a problem function. Given any competence state C ∈ C , let [C ]C = [C ] ∩ C be the equivalence class of C restricted to C and denote with C /p = {[C ]C : C ∈ C }

Please cite this article in press as: Stefanutti, L., de Chiusole, D., On the assessment of learning in competence based knowledge space theory. Journal of Mathematical Psychology (2017), http://dx.doi.org/10.1016/j.jmp.2017.08.003.

4

L. Stefanutti, D. de Chiusole / Journal of Mathematical Psychology (

is the collection of all the C -floors. A competence structure C is called floor-inclusive, with respect to p , if Cp ⊆ C . Proposition 1. The function φ : C /p → Cp such that φ ([C ]C ) = ⋂ [C ]C for every [C ]C ∈ C /p is a bijection. Proof. The function φ is onto Cp by construction. To see that it is also injective, take any two distinct classes [C⋂ ]C , [C ′ ]C ∈ C /p. ′ Since⋂[C ] and [C ] are closed under⋂ intersection, ⋂ ([C ] ∩ C ) ∈ [C ] and ([C ′ ] ∩ C ) ∈ [C ′ ]. Therefore [C ]C = [C ′ ]C would imply ⋂ [C ]C ∈ [C ′ ], which is not possible because [C ] ̸= [C ′ ] implies [C ] ∩ [C ′ ] = ∅. □ If it is true that in the special case C = 2S every equivalence class [C ] contains a unique least element, this condition could no longer be valid if C is any competence structure. Example 1. With Q = {1, 2, 3} and S = {s, t , u} let µ be such that

µ(1) = {{s}},

µ(2) = {{s, t , u}},

µ(3) = {{t , u}},

and consider the two competence spaces C1 = {∅, {t }, {u}, {s, t }, {t , u}, S },

–

follows that p(⌊C1 ⌋C \ {t }) = p(⌊C1 ⌋C ): a contradiction, because ⌊C1 ⌋C is a floor. Hence ⌊C1 ⌋C ∪ ⌊C2 ⌋C must be a floor. Proposition 10 in Heller et al. (2015) holds for the case C = 2S . In the general case C ⊆ 2S , the 1–1 correspondence between Cp and Kp is preserved (for C , C ′ ∈ Cp , C ̸ = C ′ implies [C ]C ̸ = [C ′ ]C which implies p(C ) ̸ = p(C ′ )), however the two structures need not be isomorphic anymore (in the sense of an order-isomorphism of Cp onto Kp , where the order relation is ⊆), as shown by the following example. Example 2. With Q = {1, 2, 3} and S = {a, b, c , d}, let µ be such that µ(1) = {{a}}, µ(2) = {{a, b}}, and µ(3) = {S }. For the competence space C = {∅, {a, b}, {a, c }, {a, b, c }, S }

the performance structure turns out to be Kp = {∅, {1}, {1, 2}, Q }, and the corresponding floor space is Cp = {∅, {a, b}, {a, c }, S }. Here we see that p({a, c }) = {1} ⊆ {1, 2} = p({a, b}), however {a, c } ̸⊆ {a, b}. 4. Fringes of a competence state From a learning perspective it is important to correctly identify all those skills that are immediately accessible from the competence state of a student. The collection of all these skills forms the outer fringe C ◦ of the competence state C ∈ C :

and C2 = {∅, {t }, {u}, {s, t }, {s, u}, {t , u}, S }.

Both C1 and C2 delineate the performance structure Kp = {∅, {1}, {3}, Q }. However, while all the equivalence classes of C1 contain a minimum, this is not true with C2 . For instance, the equivalence class of all competence states delineating performance state {1} is

{{s, t }, {s, u}}, in which there is no unique minimal element or, stated another way, the minimum {s} lies outside the class. If an equivalence class [C ]C ∈ C /p lacks a minimum, then it provides ambiguous information on the set of minimally sufficient skills solving the items in p(C ). When this happens, the collection Cp of the floors is no longer a subset of C . Proposition 2. Let C be a competence structure on the set S, p : 2S → 2Q be the problem function corresponding to a conjunctive skill function (Q , S , µ), and Cp be the collection of the floors. Then C is floorinclusive if and only if every equivalence class [C ]C ∈ C /p contains a unique minimal element.

⋂

)

⋂

Proof. For C ∈ C , ⋂ [C ]C ∈ [C ]C iff [C ]C ∈ [C ] ∩ C . Thus ⋂ [C ]C ∈ [C ]C implies ⋂ [C ]C ∈ C . Moreover [⋂ C ]C ⊆ [C ] and the ∩ ⋂-closure of [C ] entail [C ]C ∈ [C ]. Therefore [C ]C ∈ C implies [C ]C ∈ [C ]C . □ It should be observed that in the general case C ⊆ 2S , the collection Cp need not be closed under union. However, Cp is a space whenever C is a floor-inclusive space. Proposition 3. If C is a floor-inclusive competence space then Cp is a space. Proof. Obviously, if ⌊C1 ⌋C , ⌊C2 ⌋C ∈ Cp then ⌊C1 ⌋C ∪ ⌊C2 ⌋C ∈ C . Suppose ⌊C1 ⌋C ∪ ⌊C2 ⌋C is not a floor. Then, without loss of generality, there is t ∈ ⌊C1 ⌋C such that p(⌊C1 ⌋C ∪ ⌊C2 ⌋C \ {t }) = p(⌊C1 ⌋C ∪ ⌊C2 ⌋C ). But then, since p(⌊C1 ⌋C ) ⊆ p(⌊C1 ⌋C ∪ ⌊C2 ⌋C ) it

C ◦ = {s ∈ S \ C : C ∪ {s} ∈ C }. If a skill s is in the outer fringe of a competence state C then that skill can be learned from C without the need of learning any other skill. If every competence state different from S has a nonempty outer fringe, then the skills in S can be learned one at the time, gradually moving towards larger and larger competence states, until the full set S of skills is mastered. Throughout this section it is assumed that the competence structure C is any subset of the power set 2S and that the skill function (Q , S , µ) is conjunctive. 4.1. Effective fringes and observable changes Since skills cannot be directly observed, the only way for establishing if a skill has been learned is to look at changes in the performance state p(C ). However, sometimes no changes are produced even when a skill is actually learned. Example 3. With Q = {a, b, c , d}, and S = {s, t , u}, consider the conjunctive skill function

µ(a) = {{s}}, µ(b) = {{s, t }}, µ(c) = {{s, u}}, µ(d) = {{t , u}} and, with C = 2S , suppose that a student is in competence state C = ∅. The outer fringe of this state is C ◦ = {s, t , u}. By learning skill s this student will move from the empty competence state to {s}, and her performance state will change from p(∅) = ∅ to p({s}) = {a}. In this case the change occurred at the competence level is reflected by a change at the performance level. Suppose now that, instead of learning s, the student learns t. Then her competence state changes from ∅ to {t }, but there are no corresponding changes in the performance state, which remains empty. We could say that skill s is ‘‘effective’’ in transforming the performance state of this student, whereas skill t is not. Speaking generally, what Example 3 points out is that the outer fringe of a competence state C can be partitioned into two subsets: one containing all those skills that are effective in producing some

Please cite this article in press as: Stefanutti, L., de Chiusole, D., On the assessment of learning in competence based knowledge space theory. Journal of Mathematical Psychology (2017), http://dx.doi.org/10.1016/j.jmp.2017.08.003.

L. Stefanutti, D. de Chiusole / Journal of Mathematical Psychology (

‘‘observable’’1 change in the performance state delineated by C , and one containing those that are not. Following these observations, we call effective (outer) fringe of a competence state C the collection: C + = {s ∈ C ◦ : p(C ) ⊂ p(C ∪ {s})} of all skills that are effective in producing an observable change. 4.2. Collective fringes Suppose a tutoring system has to select the next skill to be learned by a student in competence state C . In ideal conditions (i.e. assuming that no careless errors or lucky guesses can occur) what the system knows is the performance state K = p(C ), and the aim is to select a skill that, once learned by the student, will transform the performance state K of this student to a larger one. For this, a skill has to be selected among those belonging to the effective fringe C + of the current competence state. But for identifying C + , it is necessary to know what exactly C is, and the point is that we are only given with a rough estimation of C , represented by the class [C ]C . Thus, a first question is whether the effective fringe of C can be at least partially recovered from this rough estimation. Definition 2. Given any nonempty subfamily F ⊆ C of competence states, the collective outer fringe of F is the intersection F◦ =

⋂

{C ◦ : C ∈ F }

of the outer fringes of the states in F . Two rather immediate properties of the collective outer fringe are provided by the following statement. Proposition 4. Given any nonempty subfamily F ⊆ C , the following properties hold true for the collective outer fringe F ◦ : (i) F ◦ ⊆ S \ F, (ii) C ∪ {s} ∈ C \ F for all C ∈ F and all s ∈ F ◦ .

⋃

Proof. Property (i) immediately follows from the observation that ◦ C ◦ ⊆ S \ C for all C ∈ F . Property (ii): the fact that s ∈ ⋃F implies C ∪ {s} ∈ C for all C ∈ C is obvious. Moreover s ̸ ∈ F implies C ∪ { s} ̸ ∈ F . □ By properties (i) and (ii) in Proposition 4 we know that, if the competence state of an individual is one of those in F , and this individual learns one of the skills in F ◦ then the new state will necessarily lie outside F . Thus, if the competence state of a student is contained in equivalence class [C ]C ∈ C /p, and this student learns any one of the skills in [C ]◦C , then this change is reflected by a corresponding change at the performance level. That is, if [C ]◦C is nonempty, then it contains effective skills. Proposition 5. Given a competence state C ∈ C , the collective outer fringe [C ]◦C is the intersection of all the effective fringes of the competence states in the class:

[C ]◦C =

⋂

{D+ : D ∈ [C ]C }.

Proof. By Proposition 4, s ∈ [C ]◦C implies s ∈ D+ for all D ∈ [C ]C . On the other hand, if s ∈ D+ for all D ∈ [C ]C then s ∈ D◦ for all D ∈ [C ]C and hence s ∈ [C ]◦C . □ Coming back to our tutoring system, by Proposition 5, it now knows that every skill belonging to the collective fringe of any class is effective for any state of that class and, in case it will be learned, it will cause an observable change in the student’s performance state. 1 In the present analysis we are considering an ideal situation in which lucky guesses and careless errors are excluded, so that the observable pattern of correct and wrong responses of the student exactly reflects her performance state.

)

–

5

4.3. Floor fringes Given the bijection, established by Proposition 1, between the quotient structure C /p and the collection Cp of the floors, each equivalence class in C /p is uniquely represented by a floor in Cp . If the competence structure C is a floor-inclusive space, then a characterization of the collective outer fringes can be obtained by only looking at the effective fringes of the floors, henceforth named floor fringes. Proposition 6. If C is a floor-inclusive space then the collective outer fringe [C ]◦C of the class of C equals the effective fringe ⌊C ⌋+ C of the floor, that is the equality [C ]◦C = ⌊C ⌋+ C holds true for every C ∈ C . Proof. Since ⌊C ⌋C is in [C ]C for every C ∈ C , the relation [C ]◦C ⊆ ⌊C ⌋+ C immediately follows from Proposition 5. Thus we only have ◦ to show that, if C is a space then ⌊C ⌋+ C ⊆ [C ]C for all C ∈ C . + Let s ∈ ⌊C ⌋C and D ∈ [C ]C . From ⌊C ⌋C ∪ {s} ∈ C , ⌊C ⌋C ⊆ D, and the ∪-closure of C it follows that D ∪ {s} = (⌊C ⌋C ∪ {s}) ∪ D is in C . For the monotonicity of p, ⌊C ⌋C ∪ {s} ⊆ D ∪ {s} implies p(⌊C ⌋C ∪ {s}) ⊆ p(D ∪ {s}). Hence we have: p(D) = p(⌊C ⌋C ) ⊂ p(⌊C ⌋C ∪ {s}) ⊆ p(D ∪ {s}). In particular, p(D) ⊂ p(D ∪ {s}) entails s ∈ D+ . Since this result holds for any D ∈ [C ]C , for Proposition 5, we conclude that s ∈ [C ]◦C . □ If C is a floor-inclusive competence structure, then it is not necessary to consider the whole structure for obtaining the floor fringes. Only Cp is actually needed. Proposition 7. If C is a floor-inclusive competence structure then the effective fringe of the floor ⌊C ⌋C of a competence state C ∈ C is the outer fringe of ⌊C ⌋C in the collection Cp of all the floors, that is the equality ◦ ⌊C ⌋+ C = ⌊C ⌋Cp := {s ∈ S \ ⌊C ⌋C : ⌊C ⌋C ∪ {s} ∈ Cp }.

holds true for all C ∈ C . Proof. Since ⌊C ⌋C ∈ Cp ⊆ C , it suffices to show that for every s ∈ ⌊C ⌋+ C , ⌊C ⌋C ∪ {s} is itself a floor. Proceeding by contradiction, suppose ⌊C ⌋C ∪ {s} is not a floor. Then there is t ∈ ⌊C ⌋C ∪ {s}, with t ̸ = s, such that p(⌊C ⌋C ∪ {s} \ {t }) = p(⌊C ⌋C ∪ {s}). This implies that t ̸ ∈ T ∈ µ(q) for all q ∈ p(⌊C ⌋C ∪ {s}). But s ∈ ⌊C ⌋+ C entails p(⌊C ⌋C ) ⊂ p(⌊C ⌋C ∪ {s}). Thus t ̸ ∈ T ∈ µ(q) for all q ∈ p(⌊C ⌋C ) and, hence p(⌊C ⌋C ) = p(⌊C ⌋C \ {t }), which is a contradiction, because ⌊C ⌋C is minimum in its class. □ Corollary 1. If the competence structure C is floor-inclusive, then the collective outer fringe [C ]◦C is a subset of the floor fringe ⌊C ⌋◦Cp . Moreover, if C happens to be a space then the two fringes are identical. Proof. Since, by Proposition 5, the collective outer fringe [C ]◦C is a subset of the effective fringe of every state in [C ]C , it is also a ◦ ◦ subset of the effective fringe ⌊C ⌋+ C of the floor. Hence [C ]C ⊆ ⌊C ⌋Cp immediately follows from Proposition 7. The result for the case that C is a space is now straightforward. □ When the competence structure C is a floor-inclusive space, using the floor fringe in place of the collective outer fringe can be advantageous. The main reason is that the collection Cp of the floors is a subset of the whole space C and, in some cases, it could contain much less states than C . In terms of computer memory, the computation of the floor fringe only requires storage of Cp , whereas the computation of the collective fringe cannot avoid the storage of the whole space C . A special case is represented by the quasi-ordinal (i.e., ∩-closed) competence spaces.

Please cite this article in press as: Stefanutti, L., de Chiusole, D., On the assessment of learning in competence based knowledge space theory. Journal of Mathematical Psychology (2017), http://dx.doi.org/10.1016/j.jmp.2017.08.003.

6

L. Stefanutti, D. de Chiusole / Journal of Mathematical Psychology (

)

–

Corollary 2. If C is a quasi-ordinal competence space on S and p : 2S → 2Q is the problem function corresponding to a conjunctive skill function then [C ]◦C = ⌊C ⌋◦Cp . Proof. Since C is ∩-closed and p⋂ preserves closure under intersec⋂ tion, for any C ∈ C we have p( [C ]C ) = ⋂ {p(D) : D ∈ [C ]C }. But p(D) = p(C ) for all D ∈ [C ]C , hence [C ]C ∈ [C ]C . Thus the classes in C /p contain their minima. Since C is also ∪-closed, the result follows from Proposition 6. □ 5. Well-gradedness and effective fringes Two observations arise at this point: (1) in the first place, since

[C ]◦C is only a subset of C + , the collective fringe could provide the tutor with an incomplete list of all the effective skills; (2) in the worst case the collective fringe could be empty, leaving the tutor with no choice at all, as the following example shows. Example 4. Consider again the conjunctive skill function (Q , S , µ) given in Example 1. The corresponding problem function applied to C = 2S gives: p(∅) = ∅,

p({s}) = {a},

p({t }) = ∅,

p({u}) = ∅,

p({s, t }) = {a},

p({s, u}) = {a},

p({t , u}) = {c },

p(S) = Q .

The Hasse diagrams of the competence structure 2S and the delineated performance structure are displayed in Fig. 1. Suppose that the tutor observes the performance state p(C ) = {a}. This state is delineated by three different competence states, namely [C ] = {{s}, {s, t }, {s, u}}. The collective outer fringe is [C ]◦ = {s}◦ ∩ {s, t }◦ ∩ {s, u}◦ = {t , u} ∩ {u} ∩ {t } = ∅. Depending on which is the actual competence state C of the student, the effective fringe of C could be:

• C + = ∅, if C = {s}, • C + = {u} if C = {s, t }, or • C + = {t } if C = {s, u}. Now, suppose the tutor wants to teach a skill and chooses t as the next skill to be taught (by symmetry the following arguments also apply if skill u is taught). Everything works fine if (i) the competence state from which t is learned is {s, u}, and (ii) the skill is successfully learned. In this case in fact the new competence state is S, and the delineated performance state is Q . Thus, in this particular case, there is an observable change in the performance state, and the tutor knows that skill t has been learned. Consider now the case in which the student’s competence state is C = {s, t }. Obviously, there is no need to learn skill t from this competence state. Therefore, once skill t has been (uselessly) taught, the student’ performance state remains {a}. The problem is that the same situation arises if the student is in competence state {s, u} and skill t is not learned: also in this case the student remains in performance state {a}. The fact is that the tutoring system has no way for distinguishing the case C = {s, t } and skill t (previously) learned, from the case C = {s, u} and skill t not learned. It is true that it could insist in teaching skills t and u many times. However we will have no idea of what is exactly going on with the student’s learning, until some tangible change occurs. Needless to say, all of this would happen to the detriment of the student. The situation is even worse if the competence state is C = {s}. In this case in fact there is no way at all to produce an observable change in the performance state by learning exactly one skill. This example makes it clear that having nonempty collective outer fringes is a desirable property in an efficient tutoring system. Even better would be a situation in which all the effective skills

Fig. 1. Hasse diagrams of the competence structure C = 2S (left diagram) and the performance structure Kp (right diagram) delineated by C via the skill function µ of Example 4. Gray circles represent equivalence classes in the competence structure.

are contained in the collective fringe of the class of the student’s competence state. By Proposition 5 the collective fringe [C ]◦C of a competence state C provides at least partial information about the effective fringe C + . The next propositions establish conditions under which this information is total. We recall that in a conjunctive skill function µ, there is exactly one subset of minimally sufficient skills for every item, that is µ(q) = {T }, ∅ ⊂ T ⊆ S for every q ∈ Q . Lemma 1. If C is floor-inclusive and Cp is a well-graded space, then, for every C ∈ C and every s ∈ ⌊C ⌋C there is q ∈ p(⌊C ⌋C ) such that s ∈ T ∈ µ(q). Proof. We proceed by contradiction. For s ∈ ⌊C ⌋C , suppose that s ̸ ∈ T ∈ µ(q) for all q ∈ p(⌊C ⌋C ). Obviously ⌊C ⌋C \ {s} cannot be in Cp , for otherwise we would have p(⌊C ⌋C ) = p(⌊C ⌋C \{s}) and hence ⌊C ⌋C would not be minimal. However, since Cp is well-graded, there is t ∈ ⌊C ⌋C such that ⌊C ⌋C \ {t } ∈ Cp , with s ∈ ⌊C ⌋C \ {t } and s ̸ ∈ T ∈ µ(q) for all q ∈ p(⌊C ⌋C \ {t }). By induction, this leads to the conclusion that either {s} ∈ Cp (a contradiction, because {s} cannot be minimal), or Cp is not well-graded (also a contradiction). Therefore there must be q ∈ p(⌊C ⌋C ) with µ(q) = {T } and s ∈ T. □ Lemma 2. If C is floor-inclusive and Cp is a well-graded space then s ∈ C + implies ⌊C ⌋C ⊂ ⌊C ∪ {s}⌋C for all C ∈ C . Proof. By contradiction: Suppose that ⌊C ⌋C ̸ ⊂ ⌊C ∪ {s}⌋C . Since s ∈ C + , it holds that ⌊C ⌋C ̸ = ⌊C ∪ {s}⌋C , thus consider any t ∈ ⌊C ⌋C \ ⌊C ∪ {s}⌋C . By Lemma 1 there must be q ∈ p(⌊C ⌋C ) with µ(q) = {T } and t ∈ T . As a consequence, t ̸∈ ⌊C ∪ {s}⌋C implies q ̸ ∈ p(⌊C ∪ {s}⌋C ) and hence p(⌊C ⌋C ) ̸ ⊂ p(⌊C ∪ {s}⌋C ), which is a contradiction because from s ∈ C + it follows that p(⌊C ⌋C ) = p(C ) ⊂ p(C ∪ {s}) = p(⌊C ∪ {s}⌋C ). □ Proposition 8. If C is floor-inclusive and Cp is a well-graded space then C + ⊆ ⌊C ⌋+ C for all C ∈ C . If, moreover, C is a space then C + = ⌊C ⌋+ C for all C ∈ C . Proof. Let C be a floor-inclusive competence structure and, for any C ∈ C , let s ∈ C + . Then, by Lemma 2, ⌊C ⌋C ⊂ ⌊C ∪ {s}⌋C . Since Cp is a well-graded space, there is t ∈ ⌊C ⌋+ C such that ⌊C ⌋C ∪ {t } ∈ Cp

Please cite this article in press as: Stefanutti, L., de Chiusole, D., On the assessment of learning in competence based knowledge space theory. Journal of Mathematical Psychology (2017), http://dx.doi.org/10.1016/j.jmp.2017.08.003.

L. Stefanutti, D. de Chiusole / Journal of Mathematical Psychology (

Fig. 2. Hasse diagrams of the floor space Cp induced by the skill function in Example 5 (diagram on the left), and the corresponding performance structure Kp (diagram on the right) obtained by an application of the problem function p to Cp .

with ⌊C ⌋C ⊂ ⌊C ⌋C ∪ {t } ⊆ ⌊C ∪ {s}⌋C . The first strict inclusion implies t ̸ ∈ C , for if t ∈ C then p(C ∪ {t }) = p(C ) which contradicts + t ∈ ⌊C ⌋+ C ⊆ C . On the other hand, the second inclusion relation C ∪ {t } ⊆ ⌊C ∪ {s}⌋C implies t ∈ ⌊C ∪ {s}⌋C ⊆ C ∪ {s}. It follows that t = s and hence s ∈ ⌊C ⌋+ C . If C is a space, then by Proposition 6, ◦ + ⌊C ⌋+ C = [C ]C ⊆ C and hence the result is straightforward. □ Recalling that the well-gradedness of Cp assures that all competence states have a nonempty floor fringe, by Proposition 8 this also implies that, whenever C is a space, every competence state has a nonempty effective fringe. The importance of this result is in the fact that a student can make observable progresses by learning one skill at a time, if C is a space and Cp is well-graded. We conclude this section by showing that, although the wellgradedness of the space of floors is a sufficient condition for having nonempty effective fringes, it is not necessary. Example 5. Let Q = {1, 2, 3, 4, 5} be a set of items, S = {a, b, c , d} be a set of skills, and µ be a conjunctive skill function such that:

µ(1) = {{a}}, µ(4) = {{b, c , d}},

µ(2) = {{b}}, µ(5) = {{a, c , d}}.

µ(3) = {{a, c }},

For C = 2S , this skill function delineates the knowledge structure Kp depicted in Fig. 2, right diagram, and induces the floor space Cp displayed in the same figure, left diagram. This competence space is not well-graded because there is no skill s ∈ S such that {b, c , d} \ {s} is a competence state. Nonetheless, the reader can easily check that every competence state has a nonempty floor fringe (implying that the effective fringes are also nonempty). However, since Cp is not well-graded, the floor fringes of some competence states only partially reconstruct the corresponding effective fringes. It is the case, for instance, of competence state {b, c }, whose floor fringe is {a}, whereas the corresponding effective fringe is {a, d}.

)

–

7

(say, e.g. 20) then C would contain more than a million competence states, among which to search the minimal elements. If after applying the procedure it is discovered that Cp is not well-graded (and thus some of the outer fringes are empty), some corrections should be applied and the whole testing procedure should be repeated again and again. In this section we look for properties of the conjunctive skill function µ that make the floor space Cp well-graded. If C is also a space, which is assumed all along this section, then an answer can be provided. Since the analysis is restricted to skill functions that are conjunctive, the notation will be simplified. If µ is conjunctive, then µ(q) has always the form of a singleton. Therefore, using the notation introduced by Falmagne et al. (1990), in the sequel the mapping τ : Q → 2S , satisfying the equality µ(q) = {τ (q)} for all items q ∈ Q , will be used as an alternative representation of a conjunctive skill function. It is clear that the two representations of a conjunctive skill function are equivalent. Example 6. The conjunctive skill function (Q , S , µ) of Example 5 can be rewritten as (Q , S , τ ), where:

τ (1) = {a}, τ (4) = {b, c , d},

τ (2) = {b}, τ (5) = {a, c , d}.

τ (3) = {a, c },

It is easily seen that, with this simplified notation, the problem function p corresponding to a conjunctive skill function (Q , S , τ ) takes on the form p(C ) = {q ∈ Q : τ (q) ⊆ C }.

(3)

Other two concepts, not yet considered, are necessary. A competence space on a finite set S admits a basis. The basis of a space C is the minimal family B ⊆ C whose closure under union is C (Doignon & Falmagne, 1999). Given any skill s ∈ S, a competence state C ∈ C is an atom at s if it is a minimal subset in C containing s. Then, the basis B of the space C is the collection of all the atoms (Theorem 1.26 in Doignon & Falmagne, 1999). Some of the items in Q have a special role in determining the induced floor space Cp and its basis. They also have a role in determining whether Cp is well-graded or not. We shall start by providing a definition for such special items. Definition 3. In a conjunctive skill function (Q , S , τ ), an item q ∈ Q is atomic for a skill s ∈ S if: (1) s ∈ τ (q), and (2) s ∈ τ (q′ ) implies τ (q′ ) ̸ ⊂ τ (q) for all q′ ∈ Q . Define moreover the mapping α : S → 2Q such that, for any s ∈ S,

α (s) := {q ∈ Q : q is atomic for s}, and let A=

⋃ {α (s) : s ∈ S }

be the collection of all the atomic items. 6. Exclusive skill functions In Section 5 it has been shown that a student can make observable progresses along the performance structure, by learning one skill at a time, if C is a floor-inclusive space and the family of floors Cp is a well-graded space (Proposition 8). A direct test of the well-gradedness of Cp could be carried out in practice by a straightforward procedure that: (i) derives Cp from C and (ii) checks, for every nonempty ⌊C ⌋C ∈ Cp if there is s ∈ ⌊C ⌋C such that ⌊C ⌋C \ {s} is in Cp . However, this procedure would turn out to be rather inefficient, since step (i) requires to form all the equivalence classes [C ] of states in the competence structure C and to select the minimal element in each of them, whereas step (ii) requires to iterate through the states in Cp . If for instance C is the whole power set on S, and the number of skills is not small

A useful consequence of Definition 3 is the following one. Lemma 3. Given a competence structure (S , C ) and a conjunctive skill function (Q , S , τ ) the following equality holds true for all competence states C ∈ C :

⋃ ⋃ {τ (q) : q ∈ p(C )} = {τ (q) : q ∈ p(C ) ∩ A}, where p : 2S → 2Q is the problem function corresponding to (Q , S , τ ). Proof. The left hand term of the equality can be decomposed as

⋃ ⋃ {τ (q) : q ∈ p(C )} = {τ (q) : q ∈ p(C ) ∩ A} ⋃ ∪ {τ (q) : q ∈ p(C ) \ A}.

Please cite this article in press as: Stefanutti, L., de Chiusole, D., On the assessment of learning in competence based knowledge space theory. Journal of Mathematical Psychology (2017), http://dx.doi.org/10.1016/j.jmp.2017.08.003.

8

L. Stefanutti, D. de Chiusole / Journal of Mathematical Psychology (

)

–

If q ∈ p(C ) \ A (i.e., q is not atomic) then for every skill s ∈ τ (q) there is an atomic item q′ ∈ A such that s ∈ τ (q′ ) and τ (q′ ) ⊂ τ (q). That is, there is q′ ∈ p(C ) ∩ A ⋃ such that s ∈ τ (q′ ). Hence there is F ⊆ p(C ) ∩ A such that τ (q) = {τ (q′ ) : q′ ∈ F }. □ The following two are the key definitions in this section. Definition 4. A conjunctive skill function (Q , S , τ ) is exclusive if the implication s ̸ = t H⇒ α (s) ∩ α (t) = ∅. holds true for all skills s, t ∈ S. That is, s and t do not have any common atomic items.

Fig. 3. The floor space Cp′ induced by the skill function τ ′ of Example 6 is well graded. Though isomorphic to Cp′ , the corresponding performance structure K′p is not a well graded performance space.

The term exclusive was borrowed from Koppen (1998) who used it, in a different but related context, with surmise mappings.

Proposition 10. Let (S , C ) be a space compatible with the conjunctive skill function (Q , S , τ ). Then the family of floors Cp is well-graded if and only if τ is exclusive.

Example 7. An application of Definition 3 to the skill function in Example 6 yields:

Proof. Immediately follows from Definition 4, Proposition 9 and Theorem 4.5 in Koppen (1998). □

α (a) = {1},

α (b) = {2},

α (c) = {3, 4},

α (d) = {4, 5}.

Since α (c) ∩ α (d) = {4} ̸ = ∅, we conclude that this skill function is not exclusive. In particular, item 4 is atomic for both skills c and d. Definition 5. A space (S , C ) is said to be compatible with the conjunctive skill function (Q , S , τ ) if for every q ∈ Q and every s ∈ τ (q) there is an atom B ∈ C at s such that B ⊆ τ (q). Since in a competence space C each atom for a skill s can be regarded as a background or set of prerequisites for s (Doignon & Falmagne, 1999), the compatibility condition in Definition 5 just says that, whenever an item requires a skill, it should require at least one of the backgrounds for that skill. Obviously, when C = 2S this condition is always satisfied. Lemma 4. Let C and C ′ be two spaces on S having, respectively, bases B and B′ . Then C ⊆ C ′ if and only if for every B ∈ B and every s ∈ B there is an atom B′ ∈ B′ at s such that B′ ⊆ B. ′ Proof. ⋃ Sufficiency: For every B ∈ B there is F ⊆ B⋃such that B= F . For every C ∈ C there is G ⊆ B such that C ⋃= G . Hence for every C ∈ C there is F ⊆ B′ such that C = F . Necessity: Suppose that for certain B ∈ B and s ∈ B there is no atom B′ ⋃ ∈ B′ ′ ′ of s such that B ⊆ B. Then there is no F ⊆ B such that B = F, hence B ̸ ∈ C ′ . □

When C is a space and its atoms are compatible with τ , only atomic items have a role in determining the family of floors Cp . Proposition 9. Let (S , C ) be a space and (Q , S , τ ) be a conjunctive skill function. Then Cp has basis Bp = {τ (q) : q ∈ A} if and only C is compatible with τ . Proof. Let M be the space having basis Bp . Sufficiency: Let C be compatible with τ . Then, from Definition 5 and Lemma 4 it follows that M ⊆ C . If M ∈ M then obviously M = ⌊M ⌋ ∈ Cp and hence M ⊆ Cp . On the ⋃other hand, if C ∈ C \ M then there is M ∈ M such that M = {τ (q) : q ∈ p(C ) ∩ A}. That is, there is M ∈ C with M = ⌊M ⌋ and p(M) = p(C ). It follows that C ̸ ∈ Cp . Thus, indeed Cp = M. Necessity: if C is not compatible with τ then M ̸ ⊆ C and (being Cp ⊆ C ) it follows that Cp ̸ = M. □ We notice in passing that a competence space (S , C ) compatible with the conjunctive skill function (Q , S , τ ) is also floor-inclusive. Therefore, the competence spaces identified by Definition 5 form a subfamily of the floor-inclusive competence structures.

Example 8. It was shown in Example 7 that the skill function of Example 6 is not exclusive. This skill function can be easily converted into an exclusive one by adding a new item that requires either of the two skills c or d, but not both. For instance, define a new skill function (Q ′ , S , τ ′ ) such that Q ′ = Q ∪ {6}, τ ′ (q) = τ (q) for all q ∈ Q , and τ ′ (6) = {b, c }. With this skill function we have:

α (a) = {1},

α (b) = {2},

α (c) = {3, 6},

α (d) = {4, 5}.

Since there are no two skills sharing the same atomic items, skill function τ ′ is exclusive and the corresponding floor space Cp′ (see Fig. 3) is well graded. It should be observed that, though isomorphic to Cp′ , the performance structure Kp′ delineated by τ ′ is not a well-graded knowledge space. To begin with, it is not closed under union (e.g. {1, 2}, {2, 6} ∈ Kp but {1, 2, 6} ̸ ∈ Kp ). In fact, we recall from Section 2 that the performance structure delineated by a conjunctive skill function is closed under intersection, but need not being closed under union. We also observe that some of the performance states have empty outer fringes. It is the case of performance state {1, 2}, for instance. This is quite interesting because there is no single item that a student can learn from this performance state, however there is a single skill. The minimal competence state corresponding to {1, 2} is {a, b}. The floor fringe of this state is {c }. By learning skill c the student will move to competence state {a, b, c } and her performance state will be {1, 2, 3, 6}. On the whole, this student will learn two new items by learning exactly one new skill. To summarize, provided that C is a competence space compatible with τ , exclusiveness of a conjunctive skill function (Q , S , τ ) can be tested by applying the following three steps: (1) for every skill s ∈ S construct the set α (s) by checking, for each item q ∈ Q , conditions (1) and (2) of Definition 3. The elements of α (s) are those items in Q satisfying both conditions; (2) for each pair s, t ∈ S of skills check the exclusiveness condition in Definition 4; (3) if the exclusiveness condition holds true for all pairs of skills then the skill function (Q , S , τ ) is exclusive, and the corresponding space of floors Cp is well-graded. Some remarks on the efficiency of this procedure, compared to the ‘‘direct test’’ described at the beginning of this section, are in order. Steps from 1 to 3 amount to testing conditions with respect to the skill function (Q , S , τ ). Hence, unlike the direct test, the procedure is not affected by the size of the competence space C .

Please cite this article in press as: Stefanutti, L., de Chiusole, D., On the assessment of learning in competence based knowledge space theory. Journal of Mathematical Psychology (2017), http://dx.doi.org/10.1016/j.jmp.2017.08.003.

L. Stefanutti, D. de Chiusole / Journal of Mathematical Psychology (

)

–

9

Table 1 Conjunctive skill function (Q1 , S , τ1 ) for the knowledge assessment of the ‘‘Addition and subtraction of fractions’’ domain. Problems q ∈ Q1 are listed in columns 1 and 3 whereas the subsets τ1 (q) ⊆ S of skills necessary for solving them are listed in columns 2 and 4. The skill definitions are: (a) Sum or subtract fractions having the same denominator; (b) Divide the least common multiple (LCM) of the denominators of two fractions by the denominators themselves; (c) Find the LCM of two or more integer numbers, when it equals one of them; (d) Find the LCM of two or more integer numbers, when it equals their product; (e) Multiply the LCM of two fractions by the corresponding numerators; (f ) Find the LCM of two or more integer numbers, when it is a strict multiple of each of them; (g) Reduce fraction to minimal terms (simplify); (h) Find a common divisor of two integer numbers.

τ1 (q)

Problem q

{a, b, c , e, f }

4 7

+

9 10

+ +

Problem q

(1

+

2 9

7 10 8 13

6

8 12 2 9 2 7 5 9

+

3 4

{a, g , h}

5 8

−

3 16

=

5 2

{a, b, d, e}

3 8

+

5 12

=

4 15

+

3 5

+

6 9

)

−

7 36

=

τ1 (q) {a, b, e, g , h}

□ 28 □−□ 16

{b, e}

□×3+□×5 24

{b}

{a, b, e, f , g , h}

2 7

+

5 6

{a, b, e, f }

2 3

+

3 14

{a, b, c , e, g , h}

LCM(6, 8) = □

+

1 6

{a, b, e, f }

=

5×□+7×□ 35

{e}

=

□ 9

{a, b, e, g , h}

=

□ □

{f }

Of course, a pre-condition for the application of the procedure is that C is compatible with τ . However, for testing this condition, only the basis of C is needed. The procedure turns out to be particularly efficient in the special case C = 2S , because the pre-condition is always true. In fact it can be easily verified that 2S is always compatible with τ (the atoms of 2S are the singletons {s}, s ∈ S). But this is also the case in which C contains the largest number of competence states. Finally, if C is not a space, then the direct test remains the only possible route. 7. A real world example The theoretical results exposed in the previous sections can be used, in practical applications, for the development and improvement of conjunctive skill functions. In this section a real world example is illustrated, in which the knowledge domain is ‘‘Addition and subtraction of fractions’’. For the topic at hand, the conjunctive skill function (Q1 , S , τ1 ) displayed in Table 1 was developed, and the competence structure was hypothesized to be the full power set on the skills (C = 2S ). The skill function comprises a collection S of 8 skills and a set Q1 of 13 open answer problems. The conjunctive skill function displayed in Table 1 could provide a rather simplistic representation of the fraction subtraction subject matter and, in this sense, it could be rather artificial. In particular, some of the problems could admit more than a single solution way. Nonetheless, the example is only aimed at illustrating the application of an algorithm to a concrete set of problems, for which a hypothetical conjunctive skill function is available. Suppose that τ1 is used by an intelligent tutoring system for assessing students’ learning in the ‘‘Addition and subtraction of fractions’’ domain. The tutor has to perform the following tasks: (i) assess the performance state K ∈ Kp of the student (e.g. by using the assessment procedure developed by Falmagne and Doignon (1988a)); (ii) apply the formula (2) for obtaining the floor ⌊C ⌋ of the competence state C ∈ 2S of the student; (iii) select one of the effective skills s contained in the floor fringe ⌊C ⌋◦Cp ; (iv) provide the student with some content for learning s. It is clear that tasks (iii) and (iv) can be successfully applied only if the floor fringe ⌊C ⌋◦Cp is nonempty. In Section 6, it has been shown that this condition holds true if the skill function is exclusive. An algorithm that implements the three steps described at the end of Section 6 was developed in MATLAB for testing the exclusiveness of a skill function. The output of the algorithm is a twocolumn table. For each row of the table, the first column displays

Fig. 4. Bipartite graph G1 representing the binary relation between the set A˜1 of atomic items and the set S˜ of skills for the skill function (Q1 , S , τ1 ). Black circles represent items whereas white circles represent skills.

an atomic item and the second one shows the set of skills sharing that item. The exclusiveness condition is violated whenever this set is not a singleton. The algorithm was applied to the skill function τ1 obtaining the following output: It is clear that the hypothesized skill function is not exclusive for two problems and four skills: Problem 2 is atomic for three skills, whereas item 3 is atomic for two skills. By Proposition 10, this implies that for some competence states the floor fringe could empty. In particular, for the example at hand, competence state {a, b, c , d, e, f } has an empty floor fringe. For every student in this state, tasks (iii) and (iv) described above cannot be applied, leading to an impasse of the tutoring system. The question arises at this point how to turn τ1 into an exclusive skill function. Different approaches could be followed in this direction, but exploring the details of any of them is just beyond the scope of this illustration. Nonetheless it is worth mentioning one of the possible ways. It consists of constructing a new conjunctive skill function (Q2 , S , τ2 ) with Q1 ⊆ Q2 , τ1 (q) = τ2 (q) for all q ∈ Q1 and τ2 exclusive. The bipartite graph G1 = (A˜ 1 ∪ S˜ , E) depicted in Fig. 4 represents the binary relation between the atomic items in A˜ 1 = {2, 3} and the skills in S˜ = {a, d, g , h}. In the graph, two vertices v1 , v2 ∈ A˜ 1 ∪ S˜ are connected by an edge in E if v1 is an atomic item q ∈ A˜ 1 (black circles in the figure) and v2 is a skill s ∈ S˜ such that s ∈ τ (q) (white circles in the figure). In graph theory (see, e.g., Gibbons, 1985) a matching is defined as a subset M ⊆ E of edges such that no two edges in M share the same vertex (e.g., the set {(q, s), (q′ , s)} of edges is not a matching because s is shared by the two edges). A matching M is maximal if there is no edge (q, s) ∈ E \ M such that M ∪ {(q, s)} is a matching. A maximum matching is a maximal matching containing the largest possible number of edges. Finally, a matching M covers the set S˜ if every skill in S˜ is matched to some item by M. It is obvious that for G1 it is not possible to find a matching that ˜ simply because there are more skills than atomic items. covers S, Indeed, in G1 any maximum matching M covers at most two of ˜ However, if for each of the skills that are not the four skills in S. matched by M a new item is supplemented that matches that skill (i.e., which is atomic for it, and nonatomic for any other skill), then a maximum matching that covers the whole set S˜ can be obtained. The result of the procedure outlined above need not be unique, simply because there could be many different ways of obtaining a maximum matching for the same bipartite graph. For instance G1 allows 5 different ways of producing a maximum matching. One of them is M0 = {(2, h), (3, d)}. If this solution is retained, then two new items could be supplemented such that one is atomic for skill g and the other one is atomic for skill a. In particular an item requiring g (‘‘reduce a fraction’’) but not h (‘‘find a common divisor’’) would be needed. Such an item cannot be produced,

Please cite this article in press as: Stefanutti, L., de Chiusole, D., On the assessment of learning in competence based knowledge space theory. Journal of Mathematical Psychology (2017), http://dx.doi.org/10.1016/j.jmp.2017.08.003.

10

L. Stefanutti, D. de Chiusole / Journal of Mathematical Psychology (

Fig. 5. The two steps for restoring the exclusiveness condition of skill map τ1 . At each step an extra item is added (items 14 and 15, going from left to right) until each skill is matched by exactly one atomic item. Gray circles represent the extra items. Dashed gray lines indicate removed edges.

because every problem requiring to reduce a fraction, also requires to find some common divisor between the numerator and the denominator (Heller, Anselmi, Stefanutti, & Robusto, 2017). Among the other four maximum matchings, there is the following one: M1 = {(2, g), (3, d)}. With this solution, two new items (call them items 14 and 15) could be supplemented such that item 14 is atomic for skill h (‘‘find a common divisor’’) and item 15 is atomic for skill a (‘‘adding/subtracting fractions with same denominator’’). This is easily done by taking τ (q) for each of the two pairs (q, s) in M1 and providing a new item q′ with τ (q′ ) = τ (q) \{s}. In the example at hand, the result is that each of the two items will require exactly one skill, say: skill h for item 14 and skill a for item 15. Fig. 5 shows how the bipartite graph G1 is modified by the introduction of the two new items. At each step an extra item is added (items 14 and 15, going from left to right) until each skill is connected to only one atomic item. At this point one has to provide the content of the two new items in a way which is consistent with the set of skills assigned to each of them. An example for item 14, which only requires skill h, could be for instance: ‘‘Find a common divisor of 36 and 90’’, whereas that for item 15, which only needs skill a, could be 11 7

+

8

=

□

.

7 □ The new skill function (Q2 , S , τ2 ) has the same 8 skills as τ1 , whereas the new set of items Q2 contains 2 more problems. The algorithm testing the exclusiveness of a skill function applied to τ2 , did not find any violations. This means that the floor fringe of all the competence states C ∈ C2 is nonempty. This new competence space can be used by a tutoring system for the assessment and learning of students’ knowledge in the domain at issue. 8. Final remarks As stated in the introduction, the objective of an assessment in Cb-KST is to infer the competence state of an individual from her responses to a subset of problems. A major issue in this approach to knowledge assessment is due to the lack of a one-to-one correspondence between the competence states and the performance states, in general. The assessment is still possible, but it cannot go beyond an approximation, which is represented by the class of all competence states delineating the same performance state. The problem becomes even more critical if the Cb-KST machinery is used for the assessment of learning, a typical task of computerized tutoring systems. In fact, as explained in Section 4, changes caused by learning at the competence level, could not be reflected by corresponding changes at the performance level.

)

–

The consequence is that a tutoring system might have no way of establishing whether learning has occurred or not. In the present article we have shown that this impasse can be resolved for a special class of skill functions, named conjunctive, by pretending that the floor space Cp induced by the skill function is a well-graded space. Under this condition an individual can make tangible progresses along the performance structure by learning one skill at a time, until full mastery is eventually reached. In the special case in which the competence structure is compatible with the skill function, well-gradedness of the competence space induced by the skill function can be assured if this last satisfies a special property, called exclusiveness. A test of exclusiveness of a conjunctive skill function has been described in Section 6, and it is rather easy to apply. The question remains open, what to do if the skill function is not exclusive. A possible way of tackling this question is suggested in Example 6 and the real world application described in Section 7 indicates a pathway along this direction. However it is not exhaustive and further possibilities could be explored. For instance, such developments could lead to computerized procedures that assist human experts while they are developing skill maps for specific applications. The present study is restricted to the class of conjunctive skill functions, and the question is to establish which of the theoretical results obtained in this article remain valid outside this class. As stated in Section 2, conjunctive skill functions admit only a single necessary and sufficient set of skills for every item. In practical applications this condition could be too restrictive. A number of questions remain still open concerning general (nonconjunctive) skill functions. Examples could be easily found of nonconjunctive skill functions in which the equivalence classes [C ] of the competence states are such that there is no unique subset ⌊C ⌋ of skills delineating p(C ). This calls into question the one-to-one correspondence again. Further studies are needed for extending the results in this article to general, nonconjunctive skill functions. Acknowledgments The research developed in this article was carried out under the research project ‘‘Learning how students learn. Mathematical modeling of learning processes in intelligent tutoring system navigation’’, funded by the University of Padua, Italy (CPDR152105). Furthermore we would like to thank Jean-Paul Doignon and two anonymous reviewers for their helpful suggestions and comments. References Anselmi, P., Robusto, E., & Stefanutti, L. (2012). Uncovering the best skill multimap by constraining the error probabilities of the gain-loss model. Psychometrika, 77(4), 763–781. Anselmi, P., Robusto, E., & Stefanutti, L. (2013). A procedure for identifying the best skill multimap in the gain-loss model. Electronic Notes in Discrete Mathematics, 42, 9–16. Anselmi, P., Stefanutti, L., de Chiusole, D., & Robusto, E. (2017). The assessment of knowledge and learning in competence spaces: The Gain–Loss Model for dependent skill. The British Journal of Mathematical and Statistical Psychology. http://dx.doi.org/10.1111/bmsp.12095. Advance online publication. Bolt, D. (2007). The present and future of IRT-based cognitive diagnostic models (ICDMs) and related methods. Journal of Educational Measurement, 44(4), 377–383. de Chiusole, D., Anselmi, P., Stefanutti, L., & Robusto, E. (2013). The gain–loss Model: bias and variance of the parameter estimates. Electronic Notes in Discrete Mathematics, 42, 33–40. de Chiusole, D., & Stefanutti, L. (2013). Modeling skill dependence in probabilistic competence structures. Electronic Notes in Discrete Mathematics, 42, 41–48. de la Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115–130. DiBello, L. V., & Stout, W. (2007). Guest editors’ introduction and overwiew: IRTbased cognitive diagnostic models and related methods. Journal of Educational Measurement, 44(4), 285–291.

Please cite this article in press as: Stefanutti, L., de Chiusole, D., On the assessment of learning in competence based knowledge space theory. Journal of Mathematical Psychology (2017), http://dx.doi.org/10.1016/j.jmp.2017.08.003.

L. Stefanutti, D. de Chiusole / Journal of Mathematical Psychology ( Doignon, J.-P. (1994). Knowledge spaces and skill assignments. In G. Fischer, & D. Laming (Eds.), Contributions to mathematical psychology, psychometrics and methodology (pp. 111–121). New York: Springer-Verlag. Doignon, J.-P., & Falmagne, J.-C. (1985). Spaces for the assessment of knowledge. International Journal of Man-Machine Studies, 23, 175–196. Doignon, J.-P., & Falmagne, J.-C. (1999). Knowledge spaces. Berlin, Heidelberg, and New York: Springer-Verlag. Düntsch, I., & Gediga, G. (1995). Skills and knowledge structures. The British Journal of Mathematical and Statistical Psychology, 48, 9–27. Falmagne, J.-C., Albert, D., Doble, C., Eppstein, D., & Hu, X. (2013). Knowledge spaces: Applications in education. New York: Springer-Verlag. Falmagne, J.-C., & Doignon, J.-P. (1988a). A class of stochastic procedures for the assessment of knowledge. The British Journal of Mathematical and Statistical Psychology, 41, 1–23. Falmagne, J.-C., & Doignon, J.-P. (1988b). A Markovian procedure for assessing the states of a system. Journal of Mathematical Psychology, 32, 232–258. Falmagne, J.-C., & Doignon, J.-P. (2011). Learning spaces: Interdisciplinary applied mathematics. Berlin, Heidelberg: Springer. Falmagne, J.-Cl., Koppen, M., Villano, M., Doignon, J.-P., & Johanessen, L. . (1990). Introduction to knowledge spaces: how to build, test and search them. Psychological Review, 97, 204–224. Gediga, G., & Düntsch, I. (2002). Skill set analysis in knowledge structures. The British Journal of Mathematical and Statistical Psychology, 55, 361–384. Gibbons, A. (1985). Algorithmic graph theory. Cambridge University Press. Heller, J., Anselmi, P., Stefanutti, L., & Robusto, E. (2017). A necessary and sufficient condition for unique skill assessment. Journal of Mathematical Psychology, 79, 23–28. Heller, J., Augustin, T., Hockemeyer, C., Stefanutti, L., & Albert, D. (2013). Recent developments in competence-based knowledge space theory. In J.-C. Falmagne, D. Albert, C. Doble, D. Eppstein, & X. Hu (Eds.), Knowledge spaces: Applications in education (pp. 243–286). New York: Springer-Verlag. Heller, J., Stefanutti, L., Anselmi, P., & Robusto, E. (2015). On the link between cognitive diagnostic models and knowledge space theory. Psychometrika, 80(4), 995–1019.

)

–

11

Heller, J.,Ünlü, A., & Albert, D. (2013). Skills, competencies and knowledge structures. In J.-C. Falmagne, D. Albert, C. Doble, D. Eppstein, & X. Hu (Eds.), Knowledge spaces: Applications in education (pp. 229–242). New York: Springer-Verlag. Hockemeyer, C., Conlan, O., Wade, V. P., & Albert, D. (2003). Applying competence prerequisite structures for eLearning and skill management. Journal of Universal Computer Science, 9(12), 1428–1436. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272. Koppen, M. (1998). On alternative representations for knowledge spaces. Mathematical Social Sciences, 36(2), 127–143. Korossy, K. (1993). Modellierung von wissen als kompetenz und performance. eine erweiterung der wissensstruktur-theorie von doignon und falmagne, Modelling knowledge as competence and performance. An extension of the theory of knowledge structures by Doignon and Falmagne. Unpublished doctoral dissertation, University of Heidelberg, Heidelberg. Korossy, K. (1997). Extending the theory of knowledge spaces: A competenceperformance approach. Zeitschrift Für Psychologie, 205, 53–82. Korossy, K. (1999). Modeling Knowledge as Competence and Performance. In D. Albert, & J. Lukas (Eds.), Knowledge spaces: Theories, empirical research, applications (pp. 103–132). Mahwah, NJ: Lawrence Erlbaum Associates. Lukas, J., & Albert, D. (1993). Knowledge Assessment Based on Skill Assignment and Psychological Task Analysis. In G. Strube, & K. Wender (Eds.), The cognitive psychology of knowledge (pp. 139–160). Amsterdam: North-Holland. Robusto, E., Stefanutti, L., & Anselmi, P. (2010). The gain-loss model: A probabilistic skill multimap model for assessing learning processes. Journal of Educational Measurement, 47(3), 373–394. Stefanutti, L., Anselmi, P., & Robusto, E. (2011). Assessing learning processes with the gain-loss model. Behavior Research Methods, 43(1), 66–76. Tatsuoka, C. (2002). Data-analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society Series C (Applied Statistics), 51, 337– 350. Tatsuoka, K. K. (2009). Cognitive assessment: An introduction to the rule space method. Routledge.

Please cite this article in press as: Stefanutti, L., de Chiusole, D., On the assessment of learning in competence based knowledge space theory. Journal of Mathematical Psychology (2017), http://dx.doi.org/10.1016/j.jmp.2017.08.003.