- Email: [email protected]
Discussion of IT assurance competencies
International Journal of Accounting Information Systems 5 (2004) 275 – 279
Discussion
Discussion of IT assurance competencies Brian Leader Institute of Chartered Accountants of Ontario, 69 Floor St. E. Toronto, Ontario, Canada M4W 1B3 Received 1 January 2004; accepted 1 January 2004
1. Introduction Depending on the audience I am talking to, I would call myself either a pracademic or acapractic because I work on both sides as the V-P Learning for the ICAO. I know when we talk about competencies and education that there are some limitations. I know that practitioners would like their entry-level employees to know everything they need to know and have a lot of chargeable hours. They are looking for 22-year-olds with the equivalent of 10 years of experience in an industry environment and who can do what 50-year-olds can do. But it is important to remember that when those students go into practice, for a lot of them, it is the first time they are out working, the first time they have moved away from home. When employers meet them, they frequently remark, ‘‘Wow, are they young!’’ So, I do not think that employers are ever actually going to get away from needing to train them. I will comment on the three papers on IT competencies in the order they were presented.
2. Comments on Greenstein and McKee The paper of Greenstein and McKee is relevant and timely. It addresses views of academics and practitioners to identify differences and potential gaps in preparation. Greenstein and McKee’s sample size and response rates appear sufficient to provide valid and reliable data. It is useful to identify and list important information technology (IT) topics. However, the list of 36 may be too many. Perhaps, a shorter list could be created, taking into account the competency of students entering university (each generation will be more prepared for the demands of IT). I found the following findings very interesting.
There is a relatively low level of knowledge for e-commerce and advanced technologies and audit automation constructs by both educators and practitioners. E-mail address: [email protected] (B. Leader).
1467-0895/$ - see front matter D 2004 Elsevier Inc. All rights reserved. doi:10.1016/j.accinf.2004.01.014
276
B. Leader / Int. J. Account. Inf. Syst. 5 (2004) 275–279
There is a relatively high level of knowledge for office automation and accounting firm office automation constructs. Educators with more teaching experience, but lower reported IT knowledge levels, tend to teach auditing. There is a potential ‘‘learning gap’’ between educators and practitioners for 5 of the 36 technologies examined. I think we need to look at the reasons for these gaps and at ways to try and help rectify them. The most important is the potential learning gap. Asking whether competencies are best developed in the academic environment or in practice may not be meaningful. Probably, it is not one or the other. It is both. You need to get to a certain level in one and then build on that in the other. Of course, there may be a few competencies where you would say you know that it is better in one or the other environment. In those cases, developments should be focused there. Areas highlighted by Greenstein and McKee where more attention needs to be paid include:
Assigning courses based on knowledge level rather than seniority. I think that is an important thing. Training awareness programs for educators and practitioners will help deal with those issues. Coverage of the ‘relevant’ IT in the university curriculum. That is important as well because the list of topics seems too big to deal with. How can you do justice to so many topics in the amount of time you have? Especially when that is not the only thing you are trying to do. In addition, while it is useful to look at topics, you really need to consider the breadth or depth of knowledge in those topics. How much do they need? Do they just need to comprehend what it is about, do they actually have to be able to do something with it, or do they have to be able to go out and perform it on a job, which means that they also have to understand when they need to use that knowledge. Our favorite saying in the last few years from the competency perspective is that ‘‘it’s not just knowing, it’s doing’’. When talking to students, I say that knowledge just is not enough on its own. It is great that you have the knowledge, but if all you can do is put it down on a test to get a few marks, that is not going to help you in the business world. You are going to forget it by the time you go out to work and you do not know how to identify where to use it. That message needs to be communicated in the educational environment and to be reinforced on the job. There needs to be an appropriate mix of theory and practice and of breadth and depth, and the focus should be on developing competencies not just on knowledge transfer. I think it is important to recognize that education and experience compliment and supplement each other. Thus, the survey is a good start, but is not sufficient. We need to have more sessions like this symposium with representatives from both sides, discussing the issues. We need practitioners input. Graduates find employment in small, medium and large firms designated to train students. About half are employed in large firms, the other
B. Leader / Int. J. Account. Inf. Syst. 5 (2004) 275–279
277
half in small-to-medium sized firms. Different sized firms have different needs and different views. It tends to be more difficult to get the small-to-medium firms involved but we need to recognize that their input is very important. We need to develop a consensus on what and where competencies are going to be developed. Maybe an advisory group with representatives from practice and academe can start discussing these issues. Specific items need to be identified, and an action plan needs to be developed to actually make something happen. An issue for Greenstein and McKee to consider is to use the age data for their respondents to correlate the differences in age and experience with the level of IT knowledge. How does that change when you are younger versus when you are older? What about when you are in mid career? I have heard a lot of people say that some things are better left to the next generation. IT keeps changing and changing. Sometimes, people get tired of keeping up to date. I believe that younger people tend to have more general knowledge across a wider range of IT topics. I have three kids that pick up programs very quickly. They just fool around with them and learn how to use them. They can all use word processing and spread sheet software and can produce impressive PowerPoint presentations. They are very used to the technology and are not afraid of it at all. They will learn a variety of software packages, but they will never get into a lot of depth, just enough to make it usable. In the middle years, I think you tend to specialize more. There are too many things to know. Then, when you are older the fatigue factor sets in. What about Professional Development (PD)? Who do you design that for? What are their needs? Again, you are looking at the different groups and what their needs are and how to design programs to meet their needs. In summary, you have to keep communication channels open like we do in these symposia. That is what we are trying to do here.
3. Comments on Wilkinson Doug reminds us why IT is important. It is just the way of the world. The 1996 vision report from the CICA said that CAs need more finance and more IT competencies. We are slowly, but surely, working on both of those. As you have noticed, some things take quite a while to do well. One of the most important things about the CA Candidates’ Competency Map (the Map) is that it gives us the ability to work together to address important questions about our candidates’ capabilities and related course requirements. As an aside, it is interesting that before the Sarbanes – Oxley Act (SOX) came along, we wanted to concentrate on the consulting aspects of IT competencies. But in the aftermath of SOX, the competencies have been recast in assurance terms. In Ontario, we have 19 universities that deliver our course requirements. This coursebased competency driven structure can stifle the flexibility and innovation that is needed. Under a competancy-based approach more compact program modules can be put together in a number of different ways than the traditional university course. We are trying to put less emphasis on the content or the inputs and more emphasis on the outputs or the competencies. That is not to say that knowledge is not important. It is very important. You really cannot do anything if you do not have the foundational knowledge. But you have to have a good balance of both knowledge and skills.
278
B. Leader / Int. J. Account. Inf. Syst. 5 (2004) 275–279
A key ongoing initative in Ontario is working with the academics as they compare their programs to competency requirements set out in the Map. However, competencies are written in words. Everybody has a little different view on what those words mean. A very important part of the process is the feedback to make the Map better each year. Each year, the competency descriptions will be fine tuned with the goal of ensuring that most readers get the same message. That is a very difficult process. There is some skepticism regarding competency-based education—whether it really represents a huge change. There is certainly a lot of PR and fanfare around it. But, there is not a lot of rigorous research. Hence, it would be useful to look at countries that have put competency-based approaches into their education programs, e.g., the UK or Australia. What has happened there? What has really changed as a result? Is it getting better? Has the education process improved? Whether you agree with the competency based approach or not, I am sure that you would agree that it at least gives us a different way to look at things. When you look at inputs, you are standing at one side of the table, and when you go around to the other side and look at the outputs side, you look at it in a little different way. While I don’t think that this different perspective will result in everything changing, I do believe that it will help us make improvements in what we are doing and how we do it. As noted earlier, the 2003 Uniform Evaluation (UFE), which was just written in September, was our first fully competency-referenced evaluation. The questions are all case questions. They are very much oriented to doing rather than knowing. We allowed the candidates to take in their CICA handbook (containing accounting and auditing standards) and their Tax Act and, next year, hopefully, they will be taking in their laptops with CD-ROM databases. They will be able to research the information they need. The assessment process is very costly. Instead of point scoring, it involves reading over a particular indicator of competency. There will be primary indicators of competence and secondary indicators, representing important issues for the particular case. If you are the assessor, you are reading it to assess whether the answer is reasonable from an entry-level CA and to determine whether this person is not competent, reaching competency, competent or highly competent. To deal with the obvious increase in subjectivity, there has to be more controls over how the assessment is done. For example, this year, every paper is being marked twice independently. If there is a discrepancy, there is an arbitration by a senior assessor to make sure that there are not any wrong assessments. Within the university and professional programs, we can start to concentrate more on the formative assessment versus the summative. The formative assessment helps candidates improve. For example, in our School of Accountancy in Ontario, we give the students a report on a lot of detail where their weaknesses and what they need to do to improve to get ready for the UFE. This is much better to simply reporting, ‘‘Sorry your attempt was unsuccessful.’’ We are working together with the universities to make these changes, and it is going to take some time. Probably, this is leading to an accreditation process. To get away from the course by course assessment, we will look at a whole program in order to accredit it. This will provide a lot more flexibility in design as long as it meets the Competency Map requirements. The results for candidates that come through those
B. Leader / Int. J. Account. Inf. Syst. 5 (2004) 275–279
279
programs and write the UFE will help us assess how well the programs are preparing their students, as will feedback from employers of their graduates. As for the practical experience aspect, all the offices designated to train students will also be affected by the competency approach. Many practice offices will say that they already do competency assessment in their performance reviews. But now, the Competency Map provides them with the specific details, and they need to have a way of verifying/documenting that they are exposing their candidates to enough of the competencies and a sufficient variety of them appropriate for an entry-level CA over the 30month experience requirement.
4. Comments on Lord Allan’s paper represents another great initiative. Based on your presentation, I see you must have a lot of leadership skills and patience. You are working with all of those people across the world to get something done. It is a great initiative, but it is taking too long. You probably know that. In IT, things change so quickly that by the time you get through defining the curriculum, let alone implementing part of it, which is going to take another few years, you will need to change it again. Hence, you need to somehow find a way to speed up the process. Maybe you need a smaller group of people to design the model and send it out for consensus building with the right representation to speed it up. Again you are addressing the topic, ‘‘What do we really want them to be able to do?’’ Hence, building on your progress to date, I suggest you look specifically at how much depth of coverage you need on individual topics? Which ones are more important than others? Is it depth or breadth in terms of comprehends, detects or performs in a competency? Adding that into the model can help you decide how much time you need to spend on particular items in terms of course time.
Discussion
Discussion of IT assurance competencies Brian Leader Institute of Chartered Accountants of Ontario, 69 Floor St. E. Toronto, Ontario, Canada M4W 1B3 Received 1 January 2004; accepted 1 January 2004
1. Introduction Depending on the audience I am talking to, I would call myself either a pracademic or acapractic because I work on both sides as the V-P Learning for the ICAO. I know when we talk about competencies and education that there are some limitations. I know that practitioners would like their entry-level employees to know everything they need to know and have a lot of chargeable hours. They are looking for 22-year-olds with the equivalent of 10 years of experience in an industry environment and who can do what 50-year-olds can do. But it is important to remember that when those students go into practice, for a lot of them, it is the first time they are out working, the first time they have moved away from home. When employers meet them, they frequently remark, ‘‘Wow, are they young!’’ So, I do not think that employers are ever actually going to get away from needing to train them. I will comment on the three papers on IT competencies in the order they were presented.
2. Comments on Greenstein and McKee The paper of Greenstein and McKee is relevant and timely. It addresses views of academics and practitioners to identify differences and potential gaps in preparation. Greenstein and McKee’s sample size and response rates appear sufficient to provide valid and reliable data. It is useful to identify and list important information technology (IT) topics. However, the list of 36 may be too many. Perhaps, a shorter list could be created, taking into account the competency of students entering university (each generation will be more prepared for the demands of IT). I found the following findings very interesting.
There is a relatively low level of knowledge for e-commerce and advanced technologies and audit automation constructs by both educators and practitioners. E-mail address: [email protected] (B. Leader).
1467-0895/$ - see front matter D 2004 Elsevier Inc. All rights reserved. doi:10.1016/j.accinf.2004.01.014
276
B. Leader / Int. J. Account. Inf. Syst. 5 (2004) 275–279
There is a relatively high level of knowledge for office automation and accounting firm office automation constructs. Educators with more teaching experience, but lower reported IT knowledge levels, tend to teach auditing. There is a potential ‘‘learning gap’’ between educators and practitioners for 5 of the 36 technologies examined. I think we need to look at the reasons for these gaps and at ways to try and help rectify them. The most important is the potential learning gap. Asking whether competencies are best developed in the academic environment or in practice may not be meaningful. Probably, it is not one or the other. It is both. You need to get to a certain level in one and then build on that in the other. Of course, there may be a few competencies where you would say you know that it is better in one or the other environment. In those cases, developments should be focused there. Areas highlighted by Greenstein and McKee where more attention needs to be paid include:
Assigning courses based on knowledge level rather than seniority. I think that is an important thing. Training awareness programs for educators and practitioners will help deal with those issues. Coverage of the ‘relevant’ IT in the university curriculum. That is important as well because the list of topics seems too big to deal with. How can you do justice to so many topics in the amount of time you have? Especially when that is not the only thing you are trying to do. In addition, while it is useful to look at topics, you really need to consider the breadth or depth of knowledge in those topics. How much do they need? Do they just need to comprehend what it is about, do they actually have to be able to do something with it, or do they have to be able to go out and perform it on a job, which means that they also have to understand when they need to use that knowledge. Our favorite saying in the last few years from the competency perspective is that ‘‘it’s not just knowing, it’s doing’’. When talking to students, I say that knowledge just is not enough on its own. It is great that you have the knowledge, but if all you can do is put it down on a test to get a few marks, that is not going to help you in the business world. You are going to forget it by the time you go out to work and you do not know how to identify where to use it. That message needs to be communicated in the educational environment and to be reinforced on the job. There needs to be an appropriate mix of theory and practice and of breadth and depth, and the focus should be on developing competencies not just on knowledge transfer. I think it is important to recognize that education and experience compliment and supplement each other. Thus, the survey is a good start, but is not sufficient. We need to have more sessions like this symposium with representatives from both sides, discussing the issues. We need practitioners input. Graduates find employment in small, medium and large firms designated to train students. About half are employed in large firms, the other
B. Leader / Int. J. Account. Inf. Syst. 5 (2004) 275–279
277
half in small-to-medium sized firms. Different sized firms have different needs and different views. It tends to be more difficult to get the small-to-medium firms involved but we need to recognize that their input is very important. We need to develop a consensus on what and where competencies are going to be developed. Maybe an advisory group with representatives from practice and academe can start discussing these issues. Specific items need to be identified, and an action plan needs to be developed to actually make something happen. An issue for Greenstein and McKee to consider is to use the age data for their respondents to correlate the differences in age and experience with the level of IT knowledge. How does that change when you are younger versus when you are older? What about when you are in mid career? I have heard a lot of people say that some things are better left to the next generation. IT keeps changing and changing. Sometimes, people get tired of keeping up to date. I believe that younger people tend to have more general knowledge across a wider range of IT topics. I have three kids that pick up programs very quickly. They just fool around with them and learn how to use them. They can all use word processing and spread sheet software and can produce impressive PowerPoint presentations. They are very used to the technology and are not afraid of it at all. They will learn a variety of software packages, but they will never get into a lot of depth, just enough to make it usable. In the middle years, I think you tend to specialize more. There are too many things to know. Then, when you are older the fatigue factor sets in. What about Professional Development (PD)? Who do you design that for? What are their needs? Again, you are looking at the different groups and what their needs are and how to design programs to meet their needs. In summary, you have to keep communication channels open like we do in these symposia. That is what we are trying to do here.
3. Comments on Wilkinson Doug reminds us why IT is important. It is just the way of the world. The 1996 vision report from the CICA said that CAs need more finance and more IT competencies. We are slowly, but surely, working on both of those. As you have noticed, some things take quite a while to do well. One of the most important things about the CA Candidates’ Competency Map (the Map) is that it gives us the ability to work together to address important questions about our candidates’ capabilities and related course requirements. As an aside, it is interesting that before the Sarbanes – Oxley Act (SOX) came along, we wanted to concentrate on the consulting aspects of IT competencies. But in the aftermath of SOX, the competencies have been recast in assurance terms. In Ontario, we have 19 universities that deliver our course requirements. This coursebased competency driven structure can stifle the flexibility and innovation that is needed. Under a competancy-based approach more compact program modules can be put together in a number of different ways than the traditional university course. We are trying to put less emphasis on the content or the inputs and more emphasis on the outputs or the competencies. That is not to say that knowledge is not important. It is very important. You really cannot do anything if you do not have the foundational knowledge. But you have to have a good balance of both knowledge and skills.
278
B. Leader / Int. J. Account. Inf. Syst. 5 (2004) 275–279
A key ongoing initative in Ontario is working with the academics as they compare their programs to competency requirements set out in the Map. However, competencies are written in words. Everybody has a little different view on what those words mean. A very important part of the process is the feedback to make the Map better each year. Each year, the competency descriptions will be fine tuned with the goal of ensuring that most readers get the same message. That is a very difficult process. There is some skepticism regarding competency-based education—whether it really represents a huge change. There is certainly a lot of PR and fanfare around it. But, there is not a lot of rigorous research. Hence, it would be useful to look at countries that have put competency-based approaches into their education programs, e.g., the UK or Australia. What has happened there? What has really changed as a result? Is it getting better? Has the education process improved? Whether you agree with the competency based approach or not, I am sure that you would agree that it at least gives us a different way to look at things. When you look at inputs, you are standing at one side of the table, and when you go around to the other side and look at the outputs side, you look at it in a little different way. While I don’t think that this different perspective will result in everything changing, I do believe that it will help us make improvements in what we are doing and how we do it. As noted earlier, the 2003 Uniform Evaluation (UFE), which was just written in September, was our first fully competency-referenced evaluation. The questions are all case questions. They are very much oriented to doing rather than knowing. We allowed the candidates to take in their CICA handbook (containing accounting and auditing standards) and their Tax Act and, next year, hopefully, they will be taking in their laptops with CD-ROM databases. They will be able to research the information they need. The assessment process is very costly. Instead of point scoring, it involves reading over a particular indicator of competency. There will be primary indicators of competence and secondary indicators, representing important issues for the particular case. If you are the assessor, you are reading it to assess whether the answer is reasonable from an entry-level CA and to determine whether this person is not competent, reaching competency, competent or highly competent. To deal with the obvious increase in subjectivity, there has to be more controls over how the assessment is done. For example, this year, every paper is being marked twice independently. If there is a discrepancy, there is an arbitration by a senior assessor to make sure that there are not any wrong assessments. Within the university and professional programs, we can start to concentrate more on the formative assessment versus the summative. The formative assessment helps candidates improve. For example, in our School of Accountancy in Ontario, we give the students a report on a lot of detail where their weaknesses and what they need to do to improve to get ready for the UFE. This is much better to simply reporting, ‘‘Sorry your attempt was unsuccessful.’’ We are working together with the universities to make these changes, and it is going to take some time. Probably, this is leading to an accreditation process. To get away from the course by course assessment, we will look at a whole program in order to accredit it. This will provide a lot more flexibility in design as long as it meets the Competency Map requirements. The results for candidates that come through those
B. Leader / Int. J. Account. Inf. Syst. 5 (2004) 275–279
279
programs and write the UFE will help us assess how well the programs are preparing their students, as will feedback from employers of their graduates. As for the practical experience aspect, all the offices designated to train students will also be affected by the competency approach. Many practice offices will say that they already do competency assessment in their performance reviews. But now, the Competency Map provides them with the specific details, and they need to have a way of verifying/documenting that they are exposing their candidates to enough of the competencies and a sufficient variety of them appropriate for an entry-level CA over the 30month experience requirement.
4. Comments on Lord Allan’s paper represents another great initiative. Based on your presentation, I see you must have a lot of leadership skills and patience. You are working with all of those people across the world to get something done. It is a great initiative, but it is taking too long. You probably know that. In IT, things change so quickly that by the time you get through defining the curriculum, let alone implementing part of it, which is going to take another few years, you will need to change it again. Hence, you need to somehow find a way to speed up the process. Maybe you need a smaller group of people to design the model and send it out for consensus building with the right representation to speed it up. Again you are addressing the topic, ‘‘What do we really want them to be able to do?’’ Hence, building on your progress to date, I suggest you look specifically at how much depth of coverage you need on individual topics? Which ones are more important than others? Is it depth or breadth in terms of comprehends, detects or performs in a competency? Adding that into the model can help you decide how much time you need to spend on particular items in terms of course time.