- Email: [email protected]
Measuring audit performance: Are the instruments fine enough?
Computer Audit Journal, 3 (1995)20-28
Measuring Audit Performance: Are The Instruments Fine Enough? Frank E. Bowles & Michael J.A. Parkinson l?O. Bm 225, Dir&mu ACT 2602, Carrbcrra, Atrstralia.
Nature of the audit service philosophy We should distinguish between measures which tell us how well we are doing the job from those which tell us how well we are performing. To use a rather crass analogy - it is pointless to be the fastest gun in the west if you are only firing blanks! During this paper we will present a conceptual model to place current measurement approaches, their uses and their pitfalls. It is our contention that the measures of performance are the most important to the survival, or growth, of audit and/or review functions. It may sound like motherhood to say that the people who pay for the audit service are more interested in what comes out of the process than the process itself. From the viewpoint of internal audit managers it is essential to monitor the process and to make appropriate corrections to ensure credibility. However, the paying client expects this as a matter of course.
B. Where WCwish lo be (an improved position). Management wishes to be in, or to move quickly to, a position in which they feel comfortable. In Government this may be a form of administrative nirvana. However as individuals are unique so there are multiple positions of comfort. For example, one manager may be happy knowing that no-one is stealing the petty cash, another may be comfortable having the world’s best system in place. C. Where we will be at the conclusion of the audit (that is wit/r wrcertainty removed). We are unable to move to our desired goal with confidence without first being certain about the location of our starting point. For example if we set a course for Melbourne believing we were leaving from Hobart we would get a great surprise if we were actually departing from Brisbane. If the audit process does no other thing it must provide management with reliable information about the current situation. D. Tlrc &on C fo B.
~vlliclz managment
must take to move from
The process of inquiry To explore the nature of the internal audit service we need to explore a model for the process of inquiry. Wf make reference to the work of Tony Golsby-Smith in the development of the ABCD thinking model. The
ABCD
Model
We undertake an external audit because the shareholders are not in a position to know if the financial statements fairly present the transactions of the entity. Some internal audits are conducted because management wants to know if the external auditors will find errors in those statements. Then again some internal audits are undertaken because management wants to know if remote mnnagers a-e really doing their job properly. 20
We previously referred to plotting our course for Melbourne. In doing so we must take into account the fact that we will be flying over treacherous terrain, that we must comply with Airway Navigation regulations, must pay the cost of consumables etc., have trained our crew and established alternative landing sites to cover the possibility of poor weather conditions during flight or on our arrival. It is not the auditor’s responsibility to undertake this preparation but their assistance would certainly be appreciated.
Commissioned
due to uncertainty
We do the work because management wants a knowledge outcome. As described previously, various managers will have varying ideas about the desired positioning of their system or business. The very 096%2593/95/$7.00
0 1995, Elsevier
Science
Ltd
Computer Audit Journal
nature of the type of question will determine the type of audit work which should be undertaken. Unfortunately some managers don’t have a ‘schmick’ about where it is they are heading. “The customer is king but the king is blind”. This saying has been borrowed from the Japanese and is used in our context to demonstrate the need for auditors to be proactive in establishing the nature of the management uncertainties. For too long auditors have accepted briefs of the nature of “Tell me if that system works”. The nature of the uncertainty is the agent which focuses the inquiry and the nature of the report at the end of the process. If we define at the start of the process the information needs of the management we are better able to use the most appropriate technique, with the best qualified staff, in the shortest possible time frame. Further we know immediately the nature of the message which must be conveyed in the audit report. The process of defining the information needs of the customer involves defining a number of focusing questions which the review is to answer. These questions should be very specific in their requirements, so that there is no doubt about whether or not they have been answered. So the question is not “tell me whether the system works” but rather something along the lines of: “Tell me: 1. Is the launch cause the ATM
of the new system software network to fail? and if so,
2. How long will it take to restore the old system”
normal
going
service
to
on
What is the output of the audit process? Business systems exist to serve a defined purpose. A system is a collection of components in an nrrangement which convertS raw material into some desired product. For the product to be desirable it must be relevant to business needs. Similarly any review of a business system must be cognisant of the needs of the business. Thus the audit must be focused on the business need being addressed by the function under review (that is Point B in our conceptual model).
Not all news can be good credible.
news, but all news r?lust be
We should also be aware that the news worthiness of audit findings is directly related by management’s knowledge. How often have you been told that the auditors did not tell me anything new? Or to borrow an expression denigrating consultants - “They borrowed my watch so that they could tell me time”. Answers? Audit credibility can best be achieved by giving timely answers to relevant questions asked, and owned, by management. We can illustrate the point by referring to instructions to a student preparing for a history examination: Rrrle No. 1: Amwer the question set 6)) the examiner. W71en you are asked a6out the battle of Waterloo, you are not being nsked to describe the easiest marching route _fiom Paris to Moscow! This does not, to my mind, mean that you should ignore an issue solely because you were not asked about it. Ifyou are asked the cost ofbuilding two new classrooms at a school it is perhaps an idea to mention the fact that the school has just been closed. It is relevant, so mention it, but remember to answer the set question. There are also times when management might not be asking the right questions. This is an area of extreme difficulty for the auditor. There is however one worse situation - when the management does not know what it wants to know! We must walk the line between spending our resources on unapproved activities or allowing our ‘blind kings’to walk into a pit.Thankfully, enlightened organizations allow their auditor some scope for independent activity but it remains necessary to educate our managers into looking at the right things - to prompt them into asking the right questions. None of this is risk fi-ee for the auditor. It is increasingly common amongst the managers of failing organizations to call for the blood of the auditor. This can happen even if the auditor has raised the right questions and has been told not to look at them. Very often, managers are not interested in these things until it is too late. It is the nature of managers to offer little assistance with development and to be highly critical of the outcome.
Assurance? Suggestions? We often refer to the job of the internal auditor as being to ensure that senior management can sleep easily at night. This does not mean providing them with a sedative. False assurance is worse than no assurance. It might be that the auditor must, on occasion, wake management up - shock them into action.
We add value when we provide options whereby management can move to their preferred position B from position C. To us it is obvious that you cannot measure the quality of the delivered product in isolation. You cannot add value if you do not know the bnsr fi-om which the addition will be built. We will
21
F. E. Bowles & M. J.A. Parkinson Performance: Are The instruments
explore this concept should be measured?’
further
in describing
Measuring Audit Fine Enough?
‘what
However it is management’s responsibility to choose to implement (or not to implement) and to action the recommendation. All of the above? YES! All ofthe above adds up to the delivery of useable knort,lfdge to management, Or “We have the fastest gun, we shoot live ammunition and we have an excellent aim.”
What is measured currently? Completion
of the approved work program
The Australian National Audit Oflice (ANAO) in its best practice guide referred to the completion of the approved internal audit work plan as the primary criteria for internal audit performance. Ifour program is really good at the time of approval and it remains relevant (that is it asks the questions of relevance; managers of the organization have signed-off on it; the questions asked have long-term relevance; and if no momentous issues emerge) then this might be a good measure for (internal) audit. Mostly, however, it does not satisfy our defined criteria. We contend that completion of the audit work program is an external audit measure and is not a significant indicator to most audit committees. By reference to our conceptual model we can see why this is so. The focusing question for the external audit is:
sfntcmerzts yrovided_for nttcstatiofi?
number of hours but use a senior a new graduate. This measure may tell us how budget but does not necessarily of the audit work undertaken. retirement joke. A supervisor at “Well John, how are you remainder of your last afternoon I have to go back and complete the last quarter”.
auditor
rather
than
well we fudged the relate to the quality We could refer to a a farewell lunch asks going to spend the with us?” John: “Oh, my charge sheets for
Time recording systems are really only good for providing base data for future planning - but only if we are to answer a similar focusing question in a similar area. Even when you are charging the client, it is common to book more time to a job than the client will be billed for. At least we hope that’s how it works! Number
of recommendations
accepted
Here we have the age old quandary - “Does quantity move us towards quality. 7” It is probably fair to say that one significant recommendation may add more value to management than the rest of the audit program. We have all seen recommendations of a motherhood nature which do not progress the business. Many of you may have seen recommendations which run something like this - “You should continue to.....“, “This should be reviewed next year.....“. If recommendations are in this form they may add to numbers but do not add to value. Their acceptance does not show either good work or good performance. We have seen customers who agree to almost anything. Whether this is a case of if we agree they will go away or some other motivation their presence in the equation devalues the measure for everybody else. Number
of recommendations
implemented
To answer this question the auditor must form an opinion as to each of the material components of the statements. For example, debtors, creditors and assets. For each component the auditor must do sufficient work to answer questions as to existence, valuation, authorization etc. To do this the auditor sets up a program which must be completed in order to meet auditing standards and indemnity insurance requirements.
If they agreed to implement changes but did not actually do anything, is it the fault of the recommendation, the auditor, or management? Remember the action to correct findings lies in position D. Thus whether or not recommendations are implemented or not is reflection on the management and not on the auditor.
This framework does not fit internal audit work activity. Internal audits cover multiple areas, with differing foci, and no specific relationship to each other - except perhaps in some loose linkage to the corporate plan.
What if they disagree at first but do it anyway? Or what if they do something different from that suggested in the recommendation but which achieves the same result. It is our contention that looking for implementation percentages is more a measure of the nature of the recommendation than the quality of the audit work which preceded it.
Performance
against budget
Performance against budget is a measure of how we went against our budget estimate for the specific audit. We can be well over budget if we take the same
22
$ Savings If we can accurately to a recommendation,
quantifjr the savings attributable who deserves the credit for its
Computer Audit Journal
achievement? Dollar savings can lead us into arguments with the managers about who brought forward the idea and when. Good recommendations are a matter of team work and synergy. The good ideas which come from an audit very often develop from ideas that the client has been exploring before the review started. If the client sees a better way because of a question asked by the auditor who should get the credit? Remember the recommendations represent the action D in our conceptual model. Management owns action D. Is there evidence to support the theory that quantifying savings preserves jobs? The only research which we have seen is that those audit units which use dollar savings as performance measures suffer loss of staff in a similar proportion to those which use qualitative assessments as their justification. Client
satisfaction
The Institute of Internal Auditors Inc. recommends surveying customers to identifjr their level of satisfaction with the audit they received. Even this is a difficult area. It requires tremendous strength of character to provide a good reference to someone who has just been critical of you. So who do you ask the opinion ofthat is which ofthe multiple clients or customers do we ask? Should we be asking: Did the answers to the questions help to clarify the business issues at hand? At least we would be starting from a position which was clarified at the outset of the assignment. Liaison
with the customer
Broadly described as “Did the auditor get on with the customer and the staf? We expect our staff to be courteous, tactful and professional in the performance of their duties. More often than not we are aware before our customers are that we have a problem child. Whilst receiving customer feedback may be used to show improvement it will not tell us something we did not already know. Quality
know how late reports are lodged after the field work is performed? An independent reviewer will look at the working papers and determine the time frames independently so asking the client does not necessarily add anything to our knowledge. Percentage
of staff who
are qualified
There is a basic premise that the work will be better if performed by staff with tertiary qualifications, or well advanced in obtaining them. In my experience we have used tertiary qualifications as a hall mark of an individual’s motivation to study. Ask yourself the question - “This person has a CISA or CIA qualification but does this mean they can write more than a single sentence?” I have seen many excellent auditors who know how to ask the right questions, are great with clients, and write good reports but are not professionally qualified. I do not feel it appropriate to downgrade their contribution merely because they do not have a piece of paper with a seal on it.
What should be measured? Defining
quality
in a TQM
context
We would like to recap on some important concepts fi-om TQM to bring performance measurement into a unified concept. We believe that using this conceptual base helps to clear away as much of the foggy thinking which surrounds debates on audit performance. By the customer The customer defines the quality required ofa service. In service provision the recipient of the service is the customer, that is the person paying for the service. The implication is that products, or services, which do not meet the customers’ requirement will not be highly regarded and will unlikely be on a repurchase list.
of the Recommendations
Did the recommendations help the manager to improve their understanding of the business or the effectiveness of the business? This is a subjective judgement on the part ofthe customer. What quality would you assign to a recommendation to abolish your audit function. What is the appropriate starting point fi-om which the customer makes the judgementl Timeliness
of reporting
Was the knowledge provided in a suitable time fi-amr! This is certainly I-elevant to the client but do you really think that the Director of the Audit Unit does not
Using
the purpose
to which
the service
will be
Put In defining quality the customer considers the use to which the product will be put. The salesperson will have to ask questions of you to determine the usage to which you will put your refrigerator in order to identitji the type you need. You do not need the E T)~pe, 400 litre with power sunroof and racing stripes if you are only looking for a bar fridge in which you can store beer for the football game which you and t\vo mates are going to watch.
23
F. E. Bowles & M.J.A. Parkinson - Measuring Audit Performance: Are The Instruments Fine Enough? However even if we know you only need a bar fridge, will it be appropriate if you and your mates are wine drinkers and want to store your p5t&, smoked salmon and other related delicacies as you ‘pig out’ for the night? The customer will have a particular basket ofresources which can be given in exchange for the product. If J only have so much money I cannot obtain a higher priced product. If I have a limited budget and I want to buy a quality German car, I can buy Volkswagen. The same principle applies to our refrigerator purchase and, believe it or not, to our audit services. Looking
at service
delivery
The customer will look at service delivery in assessing the product. We may have the best product to meet the customer’s requirement but they will purchase from us if they consider that they have been demeaned or ill treated. The way in which you provide the service will impact in their assessment of the quality of your products. Looking
at performance
after the sale is made
An implicit criteria by the customer is that the product will have a reasonable life expectancy after purchase. They may very well expect that the good should not survive the warranty period but for a reasonable period thereafter. They can often be prepared to pay more for this certainty. Expected service quality during the warranty period may form a crucial factor in the sale. Have you ever been annoyed by the repair person who failed to turn up to an appointment after you had taken the morning off work to be there! There are subtle lessons in these observations providers of audit services. Kipling’s
six honest
serving
for the
men
The audit approach must contain the following elements in order to allow the customer to participate in a manner which mirrors the above criteria. The use of Kipling is a reflection of one of his writing:
The wit)! of the exercise defines the relevant question to be answered. It is derived through a process of consultation and clarification with management. Jnstead of accepting a brief of the nature of - “Does the system work?” - we should be asking for details of the parts of the system that are causing concern. That is “Are you concerned with access to the concerned with delays in the system. ‘*‘; “Are you
24
receipt of reports?“; or “Do you have any statement about this system which starts with the word BUT?” This tells us the nature of the message to be canvassed in the audit report. Earlier we referred to firing live ammunition with a good aim. In this context the live ammunition is credible answers and the good aim is the answering of the questions which are of value to management. Once we know the message to convey in the audit report it is that much simpler to write. Who
The customer is buying the individuals and not the reputation of the service provider you are generally doing so for some purpose other than answering your question. Either you are doing it to form some credibility with other parties or you are desirous of accessing their professional indemnity insurance if the thing goes wrong. If you do not believe this proposition picture yourself as a senator on a committee hearing. Which would you give more weight to - an internal report by a seconded of%cer or a report from a ‘BIG SIX’ firm. This does not mean that the information is any less reliable but there is a perception that it is. In the ideal world the customer looks with the experience and training greatest assurance that the answer question will be credible. Therefore them who we propose to use on reasons for doing so.
for an individual which gives the to the focusing we should tell the job and the
If we tell the customer that certain individuals will be involved, for specific periods, then we must negotiate the contract if those individuals will not subsequently be available. Audit services are little contracts which require little proposals to support them. If the customer is buying your service on the basis of the skills of the person doing the job then they deserve to have what they thought they were paying for. Wlien The customer must make arrangements to facilitate the audit process and therefore deserves to know when the audit visit may occur. It is only in the most extreme of circumstances that we are unable to telegraph our punches. If the customer has end ofmonth processing cycles it is not unreasonable for us to wait an extra week before undertaking the visit. Similarly ifkey staff are elsewhere it is probably in our interests to wait for their return. WJiat Although the customer is not interested in the details of the audit process they are interested in knowing the nature of the work which will be undertaken. They are interested in knowing if we will be interviewing staff, snr-veying customers, extracting information fi-0111 the computer system etc. With this knowledge
Computer Audit Journal
they are able to brief their staff and smooth for us.
Remember that the auditor has responsibility only for the journey from A to C. Management has responsibility for all other areas. That is they have responsibility for how the system came to where it currently is, they have the vision for where it should be and they owned the uncertainties which we used in focusing the assignment. Most importantly they own the process whereby they move from knowledge towards their vision.
the process
Where In larger organizations the function may be performed in more than one geographical location. The choice of locations impacts on the timing of the exercise and the related audit expenses. It is ofien valuable to visit those sites considered by central management to be models and those considered to be problems. Finding problems in the model sites can be a great eye-opener for management.
How The exercise will gain credibility if the customer believes that the audit method will encompass the areas they perceive to be of significance. Remember we should bejudged on the credibility ofour answers. We should be able to show the managers that we will not be firing blanks. To use another analogy - that pillars on the bridge are actually under the roadway. Relevance
and credibility
There is no gain in using the latest, most esoteric, technique if that leads us to answer the wrong question(s). In the earlier part of this paper we referred to the examination requirement to talk about the Battle of Waterloo and not the marching route of Paris to Moscow. lfwe do not focus our audit approach to the question to be answered then we run a very great risk ofmissing the point. In other words the audit is to provide the answer not to do our process. We should never subordinate end to means. The customer will not believe our answer if they perceive that we have been unprofessional or shoddy. It is imperative that we use quality systems, that we abide by auditing standards, properly document OUI work and monitor progress for timeliness and quality. That is all part of the package. This is much the same as the client’s expectation that consultants wear dark suits and lace up shoes.
The implications The theory
of measurement
of measurement
TQM specialists tell us that you cannot manage what you cannot measure. The implication is that you must adjust your pertbrmance in accordance with the measurements you have taken. Relevant and timely measurements enable us to fine tune our process to ensure our effectiveness, efficiency or economy. It is therefore essential that we are measuring the ri$t things. If we return to our ABCD model we c-an chart where responsibilities are placed.
We can also see where the various performance measures fit within a conceptual framework. By comparing the points on the model we can see where the performance measures fit. For the auditor all of the measures in regard to the audit process are internal within the journey from A to C. They are important to the audit manager but not to the client. For the client the performance measure is as follows: 1 set a question(s) for the auditors. They informed me of the WHAT, HOW, WHO, WHEN, WHERE and HOW MUCH ? was required to answer my question. 1 now have the answer. Firstly, do 1 believe the answer. Secondly did they do it in the time frame, with the stated personnel, using the stated method,in the stated places and at the quoted cost. If the answer is yes to all of these points then it must be true that the expected quality (or value) was delivered. The audit would add value if the auditor was able to provide suggestions to the manager as to how to use the knowledge (point C) and move through D to the desired benefits at point B.
Examples from external audit A common target for criticism from bureaucrats is the reporting offindings by the Australian National Audit Office (ANAO). 1 emphasize that 1 am not targeting any individuals within that offlce but rather quoting fi-om past esperience. Guardian
of the ‘gotcha’
If findings are the measure of the auditor we may wl-eak havoc on the auditor/customer relationship. If you think about public expectations you would anticipate that the external auditor for government must find examples of waste mismanagement or else be considered to be a poor performer. Hence there is consistent pressure on external auditors to find faults \vhich are sufficiently news worthy to be brought forward in Parliament. In the past this has led to: musing internal acknowledgement; l
audit
quoting as findings things all-eady spent six months ;ludit; ;111ci
findings
without
which management had correcting prior to the
25
F. E. Bowles & M. J.A. Parkinson Performance: Are The Instruments
Measuring Audit Fine Enough?
@reporting on minor things in the view that if you find enough smoke then there must be fire somewhere In general such forms ofreporting did nothing to stop the recurrence of the same issues at a subsequent audit (which in terms of the external audit measurement criteria is very handy) and generally caused the recipients to be heartily annoyed with the process. Implications
for reporting
Reports are written for the reader. If we ask our customers about their knowledge requirements we may find that they are not interested in much of the guff we currently put into them. In my experience readers of audit reports do not want to know about the audit’s scope boundaries and method. They generally only want to know - “What is the answer?” and “What do I have to do about it?“. They do not want to search through a long torturous report to find those answers.
If the resources are insufficient to answer the question we must come back and redefine the question. That is management cannot have a Mercedes for the price of a Volkswagen. If this means they ask someone else to answer the question so be it - let them loose their shirt trying to answer the original question at such a low price. Each component of the agreement should be signed off. Either it is in the Audit Plan approved by the Audit Committee or it is a separate sign off by the person paying the bill. Follow through to the ‘how’ Never allow our audit methods to stand in the way of defining how we can obtain the knowledge required by the customer. Don’t move to the detailed planning of your audit approach until the question has been settled. An audit is not like a condom - one size does not fit all! For example would you use the same approach if the question for a general ledger review moved from:
Our model for measurement Specie
We believe that we should measure performance against the initial agreement between the auditor and the client as expressed in the audit plan. Unless we have agreed to a change in the focus due to circumstances which arose during the audit,the measurement criteria should be the initial agreement. The plan should embody the customer’s expectation. There must be a direct linkage between the client’s expectation, as expressed in an assignment plan, the audit field plan and the audit report. In our view it must be driven by the focusing question which enables audit management to pass down understanding with the plan rather than confusion. Allowing
to:
the client expectation
for the complications
Unfortunately there is generally more than one management level who have an interest in the assignment. To use a simple analogy - the line manager may have specific concerns which are different fr-om those ofthe Board. If the Board pays our salary we have to ensure that their concerns are also reflected in the assignment pl 311. Agree the ‘why’ The paramount function of audit management is to determine the knowledge requirements of the customers. Through the negotiation process we come to a con1111o11 position as to the knowledge which is required and the resources which can be applied.
Some case studies DAS If we look in depth at the distribution of time in an audit we sometimes get rude shocks. Consider how long it takes in the various stages of the audit process. How are the proportions allocated between planning, fieldwork and reporting. When we did the exercise in DAS we found that we were spending more than twice the expected time in reporting and half the expected time in planning. When we asked our customers about our effectiveness we received an even ruder shock.Specifically we asked our clients about our performance and the manner in which they used our reports. We found that very little happened on our reports because we did not convey information which was valuable to management. This of course comes after you have shed blood in preparing a report and then moving it through the various levels of audit management. Our improvement project dramatically overcame the absence of impact of our reporting and better harnessed our audit resources.
KPMG The
26
principles
which
we have described
apply even
Computer Audit Journal
more so in the cold world ofcommercial relationships. The approach described has been adopted as the national approach to internal audit by KPMG. We have done so because of the acceptance of the approach by our clients and the ef?iciencies it provides in the audit process. It is also seen by our clients to make us extremely accountable for our performance. Several of our internal audit clients have picked up the approach for their use in other aspects oftheir internal audit work. They have done so because of the credibility which focused audit work has generated for them. Further it enables better coordination across multiple geographical locations.
Developing performance the system development
measures for process
Perhaps the most difficult aspect of IT audit work is the development of performance and quality indicators for our involvement in the development of computer applications. In the following paragraphs we will outline the nature of the problem and then in the workshop we will develop a measurement framework. The problem The IS Auditor, however, often has a different role. They are not so much reviewing as building and the influence they may exert is often dependent upon structural issues as upon the skill of the auditor concerned. The auditor becomes deeply involved in the business processes and the accountability structure of the organization. Perhaps this is as it should be since, without accountability there is little value in measuring performance. Why waste your time measuring something which does not matter? Let’s spend a few minutes examining the accountability structure within which we are operating. Or, perhaps we should describe it as the nonaccountability structure, for there are more factors which mitigate against accountability in computer systems than assist it.
that” rather than the more honest “we that type of service”. How willing we inanimate object rather than accept inadequacy or take positive steps to tomer’s needs. Why do we do this?
do not provide are to blame an some form of meet the cus-
Firstly, because there are too many people involved to be able to isolate accountability. Information systems are generally the products of the cooperative work of many people: even of whole corporations. So who is accountable? The president, the IS manager, the project leader, the auditor who signed off on the system, the testing team? All of the above? The same problem is not confined to computer systems. Who is accountable for a loss of cash from a cash tin when the advance is almost never checked, the custodian has never been advised of their duties, the office manager failed to lock the safe, and the district supervisor had not visited the location in four years? A lot of people share the blame (for each of them might have prevented the fault) but none of them is likely to feel accountable. Secondly, we get used to ‘bugs’ in computer software. For ‘bug’ of course you should read ‘fault’it is an axiom that complex systems contain faults. It seems faults in computer software are more common and more widely accepted than in other complex technologies. We do not make any effort to differentiate the faults which are natural hazards when we approach something new fi-om those faults which are wellknown and ought to have been avoided. When it was discovered that the Challenger Space Shuttle was lost through the failure of O-rings which were known to have severe limitations, the management ofNASA was regarded as having been reckless. We seem not to take the same approach with computer system failures. Where do auditors come into this? Firstly we must recognize that as members of an information systems design team we are individually accountable for the performance of the system. We must take real care in specif+ng and checking the system controls. When things go wrong it will be the controls which keep system behnviour within reasonable bounds. Further we must encourage the IS professionals to use the best available methods ofdesign and construction. We may have to accept that there are things which will go wrong in spite of the programmer’s or designer’s best efforts, be we can do without the errors which arise fi-om poor practice.
In the world of computer sofnvare development (as in many other parts of the world) we seem to have lost sight of the concept of accountability. Accountability is that process by which an individual says “that failure was (partly) my fault”. It does not necessarily mean that the individual is liable to compensate for losses (although that possibility remains), it is rather the healthy process by which society determines what has gone wrong and takes steps to prevent a recurrence. We are, today, far too ready to blame the computeland force the injured party to sort themselves out.
Audit responsibilities
How many of our businesses respond to customers with statements like “the computer won’t let me do
l
Now, what of the implications for measuring audit pelformancel Perhaps we can look at the auditor’s responsibilities to various interested parties and see if they provide any means of measurement. l
to promote
to the employer a reasonable
profit
27
F. E. Bowles & M. J.A. Parkinson Performance: Are The Instruments
-to
l
advocate environment
Responsibility -active
l
communication
of requirements
to system
user
promote provision of quality software appropriate to user needs within reasonable budgetary constraints
-to
l
a suitable economic and statutory for quality software to system sponsor
Responsibility -to
ensure
any automation
is prudently
-to
ensure that the system working environment
-to
represent development
-to
promote timely and appropriate the use of the system
Responsibility the systenl -to
Measuring Audit Fine Enough?
design protection economic
the interests team
to the
training
in
by the operation
of
which provide reasonable physical, emotional and
-to
pronlote honest statenaents limitations of systenr capabilities
-to
pronlote openness software correctness
about
about
development
The relationship between audit input and the quality of systetms is extremely complex, but there is little doubt that the auditor’s performance is of relevance. It r-night be that we end up with indicators of bad performance rather than indicators of good: the question of accountability arises. A systerm which proves to have basic controls missing is an indicator ofpoor audit performance.
Why are auditors involved?
a better
of the end-user
to those affected
controls against harrm
promotes
done
professionals the discipline of the process will have a far greater effect.
the
the limitations
of
It is part of the auditor credo that retrofitting of controls is considerably more expensive than building therm into a systenl at design state. Thus, the auditor is involved so that the system may be well controlled from the start. So, to xmeasure this do we get another auditor to review the controls of the new system and use this result as an indicator of the first auditor’s performance? Sounds incestuous to me. Yet, a system which warranted audit involvement at the start is likely to be reviewed on more than one subsequent occasion, so why should these reviews not be regarded as an audit perforn~ance assessment. Auditors, as the guardians of process should also be ensuring that proper developnlent processes (including consultation and training) take place. Thus the absence of these (or dissatisfaction with their level) might also be a (negative) indicator of audit perforimance.
How is the auditor involved? Let us work through
the rmodel for rneasurenlent: The auditor is involved as a teanr member. Contribution is generally through the team rather than by n~emorandum or representation to authority. So do we ask the design teanl to vote on the auditor’s performance? I am not so certain that a popularity poll provides definitive information. Yet as professionals we should permit ourselves to be judges by other professionals. The IS people with whom we have been working should be dispassionate enough to be able to assess our technical cornpetence,our knowledge ofthe business problem and our recognition of practical reality.
Client expectation The
client
expects
a system
l
solves the business
problem
l
is fundamentally
l
does not fail catastrophically;
l
which
which: which
it addresses;
reliable;
was acquired
and
and operates
at reasonable
cost.
Can we rate the auditor according to how well these expectations are met? As earlier discussed, we cannot simply rate the auditor according to the final performance of the system and the quality of the system Frank E. Bowles is the Director for Internal Audit and Computel Audit Services nt KI’MG Rat Mnnvick in Cnnbcwn, Auztl-nlin.
28
Reference I. Golsby-Smith
T. and Bowles F., Focusing Rean Improvement Project, Conference qft/rc A~rstrdnsiarr Evdrrntior~ Sodety, July 1992.
ports for Quality:
Michael J.A. Parkinson is Manager of Group Audit at CSRO, Cnnbcrrn,Attsmlia 2nd regional Vice-President ofthe Information Systcnls Audit 2nd Control Association (ISACA). This paper was 6I-stprcccntsd at EDPAC ‘9-l - the 1904 conference of Region r; of ISACA.
Measuring Audit Performance: Are The Instruments Fine Enough? Frank E. Bowles & Michael J.A. Parkinson l?O. Bm 225, Dir&mu ACT 2602, Carrbcrra, Atrstralia.
Nature of the audit service philosophy We should distinguish between measures which tell us how well we are doing the job from those which tell us how well we are performing. To use a rather crass analogy - it is pointless to be the fastest gun in the west if you are only firing blanks! During this paper we will present a conceptual model to place current measurement approaches, their uses and their pitfalls. It is our contention that the measures of performance are the most important to the survival, or growth, of audit and/or review functions. It may sound like motherhood to say that the people who pay for the audit service are more interested in what comes out of the process than the process itself. From the viewpoint of internal audit managers it is essential to monitor the process and to make appropriate corrections to ensure credibility. However, the paying client expects this as a matter of course.
B. Where WCwish lo be (an improved position). Management wishes to be in, or to move quickly to, a position in which they feel comfortable. In Government this may be a form of administrative nirvana. However as individuals are unique so there are multiple positions of comfort. For example, one manager may be happy knowing that no-one is stealing the petty cash, another may be comfortable having the world’s best system in place. C. Where we will be at the conclusion of the audit (that is wit/r wrcertainty removed). We are unable to move to our desired goal with confidence without first being certain about the location of our starting point. For example if we set a course for Melbourne believing we were leaving from Hobart we would get a great surprise if we were actually departing from Brisbane. If the audit process does no other thing it must provide management with reliable information about the current situation. D. Tlrc &on C fo B.
~vlliclz managment
must take to move from
The process of inquiry To explore the nature of the internal audit service we need to explore a model for the process of inquiry. Wf make reference to the work of Tony Golsby-Smith in the development of the ABCD thinking model. The
ABCD
Model
We undertake an external audit because the shareholders are not in a position to know if the financial statements fairly present the transactions of the entity. Some internal audits are conducted because management wants to know if the external auditors will find errors in those statements. Then again some internal audits are undertaken because management wants to know if remote mnnagers a-e really doing their job properly. 20
We previously referred to plotting our course for Melbourne. In doing so we must take into account the fact that we will be flying over treacherous terrain, that we must comply with Airway Navigation regulations, must pay the cost of consumables etc., have trained our crew and established alternative landing sites to cover the possibility of poor weather conditions during flight or on our arrival. It is not the auditor’s responsibility to undertake this preparation but their assistance would certainly be appreciated.
Commissioned
due to uncertainty
We do the work because management wants a knowledge outcome. As described previously, various managers will have varying ideas about the desired positioning of their system or business. The very 096%2593/95/$7.00
0 1995, Elsevier
Science
Ltd
Computer Audit Journal
nature of the type of question will determine the type of audit work which should be undertaken. Unfortunately some managers don’t have a ‘schmick’ about where it is they are heading. “The customer is king but the king is blind”. This saying has been borrowed from the Japanese and is used in our context to demonstrate the need for auditors to be proactive in establishing the nature of the management uncertainties. For too long auditors have accepted briefs of the nature of “Tell me if that system works”. The nature of the uncertainty is the agent which focuses the inquiry and the nature of the report at the end of the process. If we define at the start of the process the information needs of the management we are better able to use the most appropriate technique, with the best qualified staff, in the shortest possible time frame. Further we know immediately the nature of the message which must be conveyed in the audit report. The process of defining the information needs of the customer involves defining a number of focusing questions which the review is to answer. These questions should be very specific in their requirements, so that there is no doubt about whether or not they have been answered. So the question is not “tell me whether the system works” but rather something along the lines of: “Tell me: 1. Is the launch cause the ATM
of the new system software network to fail? and if so,
2. How long will it take to restore the old system”
normal
going
service
to
on
What is the output of the audit process? Business systems exist to serve a defined purpose. A system is a collection of components in an nrrangement which convertS raw material into some desired product. For the product to be desirable it must be relevant to business needs. Similarly any review of a business system must be cognisant of the needs of the business. Thus the audit must be focused on the business need being addressed by the function under review (that is Point B in our conceptual model).
Not all news can be good credible.
news, but all news r?lust be
We should also be aware that the news worthiness of audit findings is directly related by management’s knowledge. How often have you been told that the auditors did not tell me anything new? Or to borrow an expression denigrating consultants - “They borrowed my watch so that they could tell me time”. Answers? Audit credibility can best be achieved by giving timely answers to relevant questions asked, and owned, by management. We can illustrate the point by referring to instructions to a student preparing for a history examination: Rrrle No. 1: Amwer the question set 6)) the examiner. W71en you are asked a6out the battle of Waterloo, you are not being nsked to describe the easiest marching route _fiom Paris to Moscow! This does not, to my mind, mean that you should ignore an issue solely because you were not asked about it. Ifyou are asked the cost ofbuilding two new classrooms at a school it is perhaps an idea to mention the fact that the school has just been closed. It is relevant, so mention it, but remember to answer the set question. There are also times when management might not be asking the right questions. This is an area of extreme difficulty for the auditor. There is however one worse situation - when the management does not know what it wants to know! We must walk the line between spending our resources on unapproved activities or allowing our ‘blind kings’to walk into a pit.Thankfully, enlightened organizations allow their auditor some scope for independent activity but it remains necessary to educate our managers into looking at the right things - to prompt them into asking the right questions. None of this is risk fi-ee for the auditor. It is increasingly common amongst the managers of failing organizations to call for the blood of the auditor. This can happen even if the auditor has raised the right questions and has been told not to look at them. Very often, managers are not interested in these things until it is too late. It is the nature of managers to offer little assistance with development and to be highly critical of the outcome.
Assurance? Suggestions? We often refer to the job of the internal auditor as being to ensure that senior management can sleep easily at night. This does not mean providing them with a sedative. False assurance is worse than no assurance. It might be that the auditor must, on occasion, wake management up - shock them into action.
We add value when we provide options whereby management can move to their preferred position B from position C. To us it is obvious that you cannot measure the quality of the delivered product in isolation. You cannot add value if you do not know the bnsr fi-om which the addition will be built. We will
21
F. E. Bowles & M. J.A. Parkinson Performance: Are The instruments
explore this concept should be measured?’
further
in describing
Measuring Audit Fine Enough?
‘what
However it is management’s responsibility to choose to implement (or not to implement) and to action the recommendation. All of the above? YES! All ofthe above adds up to the delivery of useable knort,lfdge to management, Or “We have the fastest gun, we shoot live ammunition and we have an excellent aim.”
What is measured currently? Completion
of the approved work program
The Australian National Audit Oflice (ANAO) in its best practice guide referred to the completion of the approved internal audit work plan as the primary criteria for internal audit performance. Ifour program is really good at the time of approval and it remains relevant (that is it asks the questions of relevance; managers of the organization have signed-off on it; the questions asked have long-term relevance; and if no momentous issues emerge) then this might be a good measure for (internal) audit. Mostly, however, it does not satisfy our defined criteria. We contend that completion of the audit work program is an external audit measure and is not a significant indicator to most audit committees. By reference to our conceptual model we can see why this is so. The focusing question for the external audit is:
sfntcmerzts yrovided_for nttcstatiofi?
number of hours but use a senior a new graduate. This measure may tell us how budget but does not necessarily of the audit work undertaken. retirement joke. A supervisor at “Well John, how are you remainder of your last afternoon I have to go back and complete the last quarter”.
auditor
rather
than
well we fudged the relate to the quality We could refer to a a farewell lunch asks going to spend the with us?” John: “Oh, my charge sheets for
Time recording systems are really only good for providing base data for future planning - but only if we are to answer a similar focusing question in a similar area. Even when you are charging the client, it is common to book more time to a job than the client will be billed for. At least we hope that’s how it works! Number
of recommendations
accepted
Here we have the age old quandary - “Does quantity move us towards quality. 7” It is probably fair to say that one significant recommendation may add more value to management than the rest of the audit program. We have all seen recommendations of a motherhood nature which do not progress the business. Many of you may have seen recommendations which run something like this - “You should continue to.....“, “This should be reviewed next year.....“. If recommendations are in this form they may add to numbers but do not add to value. Their acceptance does not show either good work or good performance. We have seen customers who agree to almost anything. Whether this is a case of if we agree they will go away or some other motivation their presence in the equation devalues the measure for everybody else. Number
of recommendations
implemented
To answer this question the auditor must form an opinion as to each of the material components of the statements. For example, debtors, creditors and assets. For each component the auditor must do sufficient work to answer questions as to existence, valuation, authorization etc. To do this the auditor sets up a program which must be completed in order to meet auditing standards and indemnity insurance requirements.
If they agreed to implement changes but did not actually do anything, is it the fault of the recommendation, the auditor, or management? Remember the action to correct findings lies in position D. Thus whether or not recommendations are implemented or not is reflection on the management and not on the auditor.
This framework does not fit internal audit work activity. Internal audits cover multiple areas, with differing foci, and no specific relationship to each other - except perhaps in some loose linkage to the corporate plan.
What if they disagree at first but do it anyway? Or what if they do something different from that suggested in the recommendation but which achieves the same result. It is our contention that looking for implementation percentages is more a measure of the nature of the recommendation than the quality of the audit work which preceded it.
Performance
against budget
Performance against budget is a measure of how we went against our budget estimate for the specific audit. We can be well over budget if we take the same
22
$ Savings If we can accurately to a recommendation,
quantifjr the savings attributable who deserves the credit for its
Computer Audit Journal
achievement? Dollar savings can lead us into arguments with the managers about who brought forward the idea and when. Good recommendations are a matter of team work and synergy. The good ideas which come from an audit very often develop from ideas that the client has been exploring before the review started. If the client sees a better way because of a question asked by the auditor who should get the credit? Remember the recommendations represent the action D in our conceptual model. Management owns action D. Is there evidence to support the theory that quantifying savings preserves jobs? The only research which we have seen is that those audit units which use dollar savings as performance measures suffer loss of staff in a similar proportion to those which use qualitative assessments as their justification. Client
satisfaction
The Institute of Internal Auditors Inc. recommends surveying customers to identifjr their level of satisfaction with the audit they received. Even this is a difficult area. It requires tremendous strength of character to provide a good reference to someone who has just been critical of you. So who do you ask the opinion ofthat is which ofthe multiple clients or customers do we ask? Should we be asking: Did the answers to the questions help to clarify the business issues at hand? At least we would be starting from a position which was clarified at the outset of the assignment. Liaison
with the customer
Broadly described as “Did the auditor get on with the customer and the staf? We expect our staff to be courteous, tactful and professional in the performance of their duties. More often than not we are aware before our customers are that we have a problem child. Whilst receiving customer feedback may be used to show improvement it will not tell us something we did not already know. Quality
know how late reports are lodged after the field work is performed? An independent reviewer will look at the working papers and determine the time frames independently so asking the client does not necessarily add anything to our knowledge. Percentage
of staff who
are qualified
There is a basic premise that the work will be better if performed by staff with tertiary qualifications, or well advanced in obtaining them. In my experience we have used tertiary qualifications as a hall mark of an individual’s motivation to study. Ask yourself the question - “This person has a CISA or CIA qualification but does this mean they can write more than a single sentence?” I have seen many excellent auditors who know how to ask the right questions, are great with clients, and write good reports but are not professionally qualified. I do not feel it appropriate to downgrade their contribution merely because they do not have a piece of paper with a seal on it.
What should be measured? Defining
quality
in a TQM
context
We would like to recap on some important concepts fi-om TQM to bring performance measurement into a unified concept. We believe that using this conceptual base helps to clear away as much of the foggy thinking which surrounds debates on audit performance. By the customer The customer defines the quality required ofa service. In service provision the recipient of the service is the customer, that is the person paying for the service. The implication is that products, or services, which do not meet the customers’ requirement will not be highly regarded and will unlikely be on a repurchase list.
of the Recommendations
Did the recommendations help the manager to improve their understanding of the business or the effectiveness of the business? This is a subjective judgement on the part ofthe customer. What quality would you assign to a recommendation to abolish your audit function. What is the appropriate starting point fi-om which the customer makes the judgementl Timeliness
of reporting
Was the knowledge provided in a suitable time fi-amr! This is certainly I-elevant to the client but do you really think that the Director of the Audit Unit does not
Using
the purpose
to which
the service
will be
Put In defining quality the customer considers the use to which the product will be put. The salesperson will have to ask questions of you to determine the usage to which you will put your refrigerator in order to identitji the type you need. You do not need the E T)~pe, 400 litre with power sunroof and racing stripes if you are only looking for a bar fridge in which you can store beer for the football game which you and t\vo mates are going to watch.
23
F. E. Bowles & M.J.A. Parkinson - Measuring Audit Performance: Are The Instruments Fine Enough? However even if we know you only need a bar fridge, will it be appropriate if you and your mates are wine drinkers and want to store your p5t&, smoked salmon and other related delicacies as you ‘pig out’ for the night? The customer will have a particular basket ofresources which can be given in exchange for the product. If J only have so much money I cannot obtain a higher priced product. If I have a limited budget and I want to buy a quality German car, I can buy Volkswagen. The same principle applies to our refrigerator purchase and, believe it or not, to our audit services. Looking
at service
delivery
The customer will look at service delivery in assessing the product. We may have the best product to meet the customer’s requirement but they will purchase from us if they consider that they have been demeaned or ill treated. The way in which you provide the service will impact in their assessment of the quality of your products. Looking
at performance
after the sale is made
An implicit criteria by the customer is that the product will have a reasonable life expectancy after purchase. They may very well expect that the good should not survive the warranty period but for a reasonable period thereafter. They can often be prepared to pay more for this certainty. Expected service quality during the warranty period may form a crucial factor in the sale. Have you ever been annoyed by the repair person who failed to turn up to an appointment after you had taken the morning off work to be there! There are subtle lessons in these observations providers of audit services. Kipling’s
six honest
serving
for the
men
The audit approach must contain the following elements in order to allow the customer to participate in a manner which mirrors the above criteria. The use of Kipling is a reflection of one of his writing:
The wit)! of the exercise defines the relevant question to be answered. It is derived through a process of consultation and clarification with management. Jnstead of accepting a brief of the nature of - “Does the system work?” - we should be asking for details of the parts of the system that are causing concern. That is “Are you concerned with access to the concerned with delays in the system. ‘*‘; “Are you
24
receipt of reports?“; or “Do you have any statement about this system which starts with the word BUT?” This tells us the nature of the message to be canvassed in the audit report. Earlier we referred to firing live ammunition with a good aim. In this context the live ammunition is credible answers and the good aim is the answering of the questions which are of value to management. Once we know the message to convey in the audit report it is that much simpler to write. Who
The customer is buying the individuals and not the reputation of the service provider you are generally doing so for some purpose other than answering your question. Either you are doing it to form some credibility with other parties or you are desirous of accessing their professional indemnity insurance if the thing goes wrong. If you do not believe this proposition picture yourself as a senator on a committee hearing. Which would you give more weight to - an internal report by a seconded of%cer or a report from a ‘BIG SIX’ firm. This does not mean that the information is any less reliable but there is a perception that it is. In the ideal world the customer looks with the experience and training greatest assurance that the answer question will be credible. Therefore them who we propose to use on reasons for doing so.
for an individual which gives the to the focusing we should tell the job and the
If we tell the customer that certain individuals will be involved, for specific periods, then we must negotiate the contract if those individuals will not subsequently be available. Audit services are little contracts which require little proposals to support them. If the customer is buying your service on the basis of the skills of the person doing the job then they deserve to have what they thought they were paying for. Wlien The customer must make arrangements to facilitate the audit process and therefore deserves to know when the audit visit may occur. It is only in the most extreme of circumstances that we are unable to telegraph our punches. If the customer has end ofmonth processing cycles it is not unreasonable for us to wait an extra week before undertaking the visit. Similarly ifkey staff are elsewhere it is probably in our interests to wait for their return. WJiat Although the customer is not interested in the details of the audit process they are interested in knowing the nature of the work which will be undertaken. They are interested in knowing if we will be interviewing staff, snr-veying customers, extracting information fi-0111 the computer system etc. With this knowledge
Computer Audit Journal
they are able to brief their staff and smooth for us.
Remember that the auditor has responsibility only for the journey from A to C. Management has responsibility for all other areas. That is they have responsibility for how the system came to where it currently is, they have the vision for where it should be and they owned the uncertainties which we used in focusing the assignment. Most importantly they own the process whereby they move from knowledge towards their vision.
the process
Where In larger organizations the function may be performed in more than one geographical location. The choice of locations impacts on the timing of the exercise and the related audit expenses. It is ofien valuable to visit those sites considered by central management to be models and those considered to be problems. Finding problems in the model sites can be a great eye-opener for management.
How The exercise will gain credibility if the customer believes that the audit method will encompass the areas they perceive to be of significance. Remember we should bejudged on the credibility ofour answers. We should be able to show the managers that we will not be firing blanks. To use another analogy - that pillars on the bridge are actually under the roadway. Relevance
and credibility
There is no gain in using the latest, most esoteric, technique if that leads us to answer the wrong question(s). In the earlier part of this paper we referred to the examination requirement to talk about the Battle of Waterloo and not the marching route of Paris to Moscow. lfwe do not focus our audit approach to the question to be answered then we run a very great risk ofmissing the point. In other words the audit is to provide the answer not to do our process. We should never subordinate end to means. The customer will not believe our answer if they perceive that we have been unprofessional or shoddy. It is imperative that we use quality systems, that we abide by auditing standards, properly document OUI work and monitor progress for timeliness and quality. That is all part of the package. This is much the same as the client’s expectation that consultants wear dark suits and lace up shoes.
The implications The theory
of measurement
of measurement
TQM specialists tell us that you cannot manage what you cannot measure. The implication is that you must adjust your pertbrmance in accordance with the measurements you have taken. Relevant and timely measurements enable us to fine tune our process to ensure our effectiveness, efficiency or economy. It is therefore essential that we are measuring the ri$t things. If we return to our ABCD model we c-an chart where responsibilities are placed.
We can also see where the various performance measures fit within a conceptual framework. By comparing the points on the model we can see where the performance measures fit. For the auditor all of the measures in regard to the audit process are internal within the journey from A to C. They are important to the audit manager but not to the client. For the client the performance measure is as follows: 1 set a question(s) for the auditors. They informed me of the WHAT, HOW, WHO, WHEN, WHERE and HOW MUCH ? was required to answer my question. 1 now have the answer. Firstly, do 1 believe the answer. Secondly did they do it in the time frame, with the stated personnel, using the stated method,in the stated places and at the quoted cost. If the answer is yes to all of these points then it must be true that the expected quality (or value) was delivered. The audit would add value if the auditor was able to provide suggestions to the manager as to how to use the knowledge (point C) and move through D to the desired benefits at point B.
Examples from external audit A common target for criticism from bureaucrats is the reporting offindings by the Australian National Audit Office (ANAO). 1 emphasize that 1 am not targeting any individuals within that offlce but rather quoting fi-om past esperience. Guardian
of the ‘gotcha’
If findings are the measure of the auditor we may wl-eak havoc on the auditor/customer relationship. If you think about public expectations you would anticipate that the external auditor for government must find examples of waste mismanagement or else be considered to be a poor performer. Hence there is consistent pressure on external auditors to find faults \vhich are sufficiently news worthy to be brought forward in Parliament. In the past this has led to: musing internal acknowledgement; l
audit
quoting as findings things all-eady spent six months ;ludit; ;111ci
findings
without
which management had correcting prior to the
25
F. E. Bowles & M. J.A. Parkinson Performance: Are The Instruments
Measuring Audit Fine Enough?
@reporting on minor things in the view that if you find enough smoke then there must be fire somewhere In general such forms ofreporting did nothing to stop the recurrence of the same issues at a subsequent audit (which in terms of the external audit measurement criteria is very handy) and generally caused the recipients to be heartily annoyed with the process. Implications
for reporting
Reports are written for the reader. If we ask our customers about their knowledge requirements we may find that they are not interested in much of the guff we currently put into them. In my experience readers of audit reports do not want to know about the audit’s scope boundaries and method. They generally only want to know - “What is the answer?” and “What do I have to do about it?“. They do not want to search through a long torturous report to find those answers.
If the resources are insufficient to answer the question we must come back and redefine the question. That is management cannot have a Mercedes for the price of a Volkswagen. If this means they ask someone else to answer the question so be it - let them loose their shirt trying to answer the original question at such a low price. Each component of the agreement should be signed off. Either it is in the Audit Plan approved by the Audit Committee or it is a separate sign off by the person paying the bill. Follow through to the ‘how’ Never allow our audit methods to stand in the way of defining how we can obtain the knowledge required by the customer. Don’t move to the detailed planning of your audit approach until the question has been settled. An audit is not like a condom - one size does not fit all! For example would you use the same approach if the question for a general ledger review moved from:
Our model for measurement Specie
We believe that we should measure performance against the initial agreement between the auditor and the client as expressed in the audit plan. Unless we have agreed to a change in the focus due to circumstances which arose during the audit,the measurement criteria should be the initial agreement. The plan should embody the customer’s expectation. There must be a direct linkage between the client’s expectation, as expressed in an assignment plan, the audit field plan and the audit report. In our view it must be driven by the focusing question which enables audit management to pass down understanding with the plan rather than confusion. Allowing
to:
the client expectation
for the complications
Unfortunately there is generally more than one management level who have an interest in the assignment. To use a simple analogy - the line manager may have specific concerns which are different fr-om those ofthe Board. If the Board pays our salary we have to ensure that their concerns are also reflected in the assignment pl 311. Agree the ‘why’ The paramount function of audit management is to determine the knowledge requirements of the customers. Through the negotiation process we come to a con1111o11 position as to the knowledge which is required and the resources which can be applied.
Some case studies DAS If we look in depth at the distribution of time in an audit we sometimes get rude shocks. Consider how long it takes in the various stages of the audit process. How are the proportions allocated between planning, fieldwork and reporting. When we did the exercise in DAS we found that we were spending more than twice the expected time in reporting and half the expected time in planning. When we asked our customers about our effectiveness we received an even ruder shock.Specifically we asked our clients about our performance and the manner in which they used our reports. We found that very little happened on our reports because we did not convey information which was valuable to management. This of course comes after you have shed blood in preparing a report and then moving it through the various levels of audit management. Our improvement project dramatically overcame the absence of impact of our reporting and better harnessed our audit resources.
KPMG The
26
principles
which
we have described
apply even
Computer Audit Journal
more so in the cold world ofcommercial relationships. The approach described has been adopted as the national approach to internal audit by KPMG. We have done so because of the acceptance of the approach by our clients and the ef?iciencies it provides in the audit process. It is also seen by our clients to make us extremely accountable for our performance. Several of our internal audit clients have picked up the approach for their use in other aspects oftheir internal audit work. They have done so because of the credibility which focused audit work has generated for them. Further it enables better coordination across multiple geographical locations.
Developing performance the system development
measures for process
Perhaps the most difficult aspect of IT audit work is the development of performance and quality indicators for our involvement in the development of computer applications. In the following paragraphs we will outline the nature of the problem and then in the workshop we will develop a measurement framework. The problem The IS Auditor, however, often has a different role. They are not so much reviewing as building and the influence they may exert is often dependent upon structural issues as upon the skill of the auditor concerned. The auditor becomes deeply involved in the business processes and the accountability structure of the organization. Perhaps this is as it should be since, without accountability there is little value in measuring performance. Why waste your time measuring something which does not matter? Let’s spend a few minutes examining the accountability structure within which we are operating. Or, perhaps we should describe it as the nonaccountability structure, for there are more factors which mitigate against accountability in computer systems than assist it.
that” rather than the more honest “we that type of service”. How willing we inanimate object rather than accept inadequacy or take positive steps to tomer’s needs. Why do we do this?
do not provide are to blame an some form of meet the cus-
Firstly, because there are too many people involved to be able to isolate accountability. Information systems are generally the products of the cooperative work of many people: even of whole corporations. So who is accountable? The president, the IS manager, the project leader, the auditor who signed off on the system, the testing team? All of the above? The same problem is not confined to computer systems. Who is accountable for a loss of cash from a cash tin when the advance is almost never checked, the custodian has never been advised of their duties, the office manager failed to lock the safe, and the district supervisor had not visited the location in four years? A lot of people share the blame (for each of them might have prevented the fault) but none of them is likely to feel accountable. Secondly, we get used to ‘bugs’ in computer software. For ‘bug’ of course you should read ‘fault’it is an axiom that complex systems contain faults. It seems faults in computer software are more common and more widely accepted than in other complex technologies. We do not make any effort to differentiate the faults which are natural hazards when we approach something new fi-om those faults which are wellknown and ought to have been avoided. When it was discovered that the Challenger Space Shuttle was lost through the failure of O-rings which were known to have severe limitations, the management ofNASA was regarded as having been reckless. We seem not to take the same approach with computer system failures. Where do auditors come into this? Firstly we must recognize that as members of an information systems design team we are individually accountable for the performance of the system. We must take real care in specif+ng and checking the system controls. When things go wrong it will be the controls which keep system behnviour within reasonable bounds. Further we must encourage the IS professionals to use the best available methods ofdesign and construction. We may have to accept that there are things which will go wrong in spite of the programmer’s or designer’s best efforts, be we can do without the errors which arise fi-om poor practice.
In the world of computer sofnvare development (as in many other parts of the world) we seem to have lost sight of the concept of accountability. Accountability is that process by which an individual says “that failure was (partly) my fault”. It does not necessarily mean that the individual is liable to compensate for losses (although that possibility remains), it is rather the healthy process by which society determines what has gone wrong and takes steps to prevent a recurrence. We are, today, far too ready to blame the computeland force the injured party to sort themselves out.
Audit responsibilities
How many of our businesses respond to customers with statements like “the computer won’t let me do
l
Now, what of the implications for measuring audit pelformancel Perhaps we can look at the auditor’s responsibilities to various interested parties and see if they provide any means of measurement. l
to promote
to the employer a reasonable
profit
27
F. E. Bowles & M. J.A. Parkinson Performance: Are The Instruments
-to
l
advocate environment
Responsibility -active
l
communication
of requirements
to system
user
promote provision of quality software appropriate to user needs within reasonable budgetary constraints
-to
l
a suitable economic and statutory for quality software to system sponsor
Responsibility -to
ensure
any automation
is prudently
-to
ensure that the system working environment
-to
represent development
-to
promote timely and appropriate the use of the system
Responsibility the systenl -to
Measuring Audit Fine Enough?
design protection economic
the interests team
to the
training
in
by the operation
of
which provide reasonable physical, emotional and
-to
pronlote honest statenaents limitations of systenr capabilities
-to
pronlote openness software correctness
about
about
development
The relationship between audit input and the quality of systetms is extremely complex, but there is little doubt that the auditor’s performance is of relevance. It r-night be that we end up with indicators of bad performance rather than indicators of good: the question of accountability arises. A systerm which proves to have basic controls missing is an indicator ofpoor audit performance.
Why are auditors involved?
a better
of the end-user
to those affected
controls against harrm
promotes
done
professionals the discipline of the process will have a far greater effect.
the
the limitations
of
It is part of the auditor credo that retrofitting of controls is considerably more expensive than building therm into a systenl at design state. Thus, the auditor is involved so that the system may be well controlled from the start. So, to xmeasure this do we get another auditor to review the controls of the new system and use this result as an indicator of the first auditor’s performance? Sounds incestuous to me. Yet, a system which warranted audit involvement at the start is likely to be reviewed on more than one subsequent occasion, so why should these reviews not be regarded as an audit perforn~ance assessment. Auditors, as the guardians of process should also be ensuring that proper developnlent processes (including consultation and training) take place. Thus the absence of these (or dissatisfaction with their level) might also be a (negative) indicator of audit perforimance.
How is the auditor involved? Let us work through
the rmodel for rneasurenlent: The auditor is involved as a teanr member. Contribution is generally through the team rather than by n~emorandum or representation to authority. So do we ask the design teanl to vote on the auditor’s performance? I am not so certain that a popularity poll provides definitive information. Yet as professionals we should permit ourselves to be judges by other professionals. The IS people with whom we have been working should be dispassionate enough to be able to assess our technical cornpetence,our knowledge ofthe business problem and our recognition of practical reality.
Client expectation The
client
expects
a system
l
solves the business
problem
l
is fundamentally
l
does not fail catastrophically;
l
which
which: which
it addresses;
reliable;
was acquired
and
and operates
at reasonable
cost.
Can we rate the auditor according to how well these expectations are met? As earlier discussed, we cannot simply rate the auditor according to the final performance of the system and the quality of the system Frank E. Bowles is the Director for Internal Audit and Computel Audit Services nt KI’MG Rat Mnnvick in Cnnbcwn, Auztl-nlin.
28
Reference I. Golsby-Smith
T. and Bowles F., Focusing Rean Improvement Project, Conference qft/rc A~rstrdnsiarr Evdrrntior~ Sodety, July 1992.
ports for Quality:
Michael J.A. Parkinson is Manager of Group Audit at CSRO, Cnnbcrrn,Attsmlia 2nd regional Vice-President ofthe Information Systcnls Audit 2nd Control Association (ISACA). This paper was 6I-stprcccntsd at EDPAC ‘9-l - the 1904 conference of Region r; of ISACA.