Learning After each Assignment Is Limited

As mentioned before, when I work with audit functions I will often review their audit methodology to offer advice about how it can be amended to better drive added value and reduce waste. In around 50% of the cases I see, there is very little written in the audit methodology about learning reviews after each assignment.

When I talk to the CAEs concerned they explain that, although a learning review may not be explicitly required, they nonetheless expect their audit managers to consider the key learning points from each assignment and debrief their staff accordingly. However, even where this is the case, some CAEs accept that they have limited insight into the detail of what happens, and several CAEs I have spoken to have confessed that they suspect this is an area that is not done particularly well as a result of workload pressures.

Audit Methodologies and Software Have a Range of Shortcomings

As readers will appreciate, lean ways of working are concerned with defining value adding and waste free processes as well as being concerned about the Gemba of what is happening in practice. In an audit context often an audit methodology (if it exists) may be intended to help auditors know what they should do during an assignment. However, it is not unusual to find auditors have a range of concerns about the audit methodology they are expected follow:

1. A sense that the methodology is principally about completing a series of key documents or meeting the needs of audit software requirements.

   Reading the audit methodology becomes a lengthy “how to” guide in relation to which forms to complete and how to complete these properly, or how to use the audit software (and which drop down boxes to use).

2. A sense that the methodology has a limited appreciation for the reality of the difficulties and dilemmas that may be encountered when delivering an audit assignment.

   It is almost as if by not discussing the challenges in assignment delivery they will not happen! In addition, many audit methodologies underplay the practical (Gemba) difficulties of navigating between quality, time, and the completion of the audit file alongside managing relationships with management.

Note that these concerns can arise whether or not there is an extensive audit methodology or a very brief methodology. In any event, there is a significant risk that an audit methodology is neither an enabler to the delivery of added value, nor a tool to drive productivity in the audit process, but instead it is inadvertently a barrier to both.

Metrics Are Not Collected At All, or Are Collected to Little Practical Effect

I see a wide variety of practices in relation to measuring audit performance. When I speak to auditors at workshops or training events, a number are frustrated with the amount of time they spend recording how they have spent their time, or are concerned about the time they spend explaining detailed variances, with no obvious benefit that they can see. I know of a number of instances where a metric to deliver an audit report within a set timescale has auditors rushing out reports, only then to result in negative feedback for failing to properly engage management on the remediation plan.

In addition, auditors can sometimes feel pressured into doing other dysfunctional things, for example, delaying the assignment closing meeting, or calling it something else (“a pre-close meeting”) so that they can meet a target they have been set in relation to the time between the close of the meeting and the issue of the final report. Phil Gerrard (CAE, Rolls-Royce) shares the following perspective on dysfunctional aspects around time recording:

“I’ve seen teams completing timesheets with very little coming out of it. Worse still I suspect some of the data was flawed, with people estimating how they were using their time. Moreover, it wasn’t being used as an active management tool.

So think very carefully about what you collect.”

I also see CAEs encountering difficulties with plan delivery towards the end of the audit year who admit their audit function metrics have not really helped them anticipate these issues, or drive better delivery performance during the year. What value do metrics have if they do not help the audit team to manage the key things that would be of value to key stakeholders and the external customer?!

Chris Baker (Technical Manager, IIA UK) provides a more general observation about typical audit metrics:

“Many audit functions should move away from the traditional things that we see, like number of audits completed, time lapses to issue audit reports and all of the things we are familiar with; and develop a broader set of metrics. Because I think there’s a much wider way of understanding how effective internal audit is.”

Quality Assessments Are Done Reluctantly and Not Seen as a Tool for Driving Added Value

I remember, after becoming CAE for AstraZeneca, requesting an External Quality Assessment (EQA) for the audit function after six months in my role. I wanted to get input on whether we were on the right track with the changes we had started to work on. The process was very helpful and I will never forget a Big Four partner saying to me:

“It’s great to do an EQA for a CAE, it makes it much more collaborative. The majority of EQAs we do are at the request of the CFO or Audit Committee, and this can be a sign that they are concerned with what is going on in audit. Indeed it’s not unusual to see the results of our review lead to a change in the leadership of the audit function.”

To this day, based on my experience as an EQA assessor for the IIA UK, there appear to be a number of CAEs who still have serious doubts about the need or benefit of having an EQA review, perhaps based on problems in the past. I have some sympathy with this view, since it is easy to be reviewed by someone with limited real world experience managing an audit function, who takes a rather idealistic approach to what needs to be changed. However, with some of my colleagues on the IIA UK EQA panel, we have also wondered whether there is something in the culture of audit that enjoys auditing others, but is less comfortable when it is being scrutinized itself?

Post Assignment Learning is Essential

The starting point for driving a mindset of value add and efficiency within the audit function is to recognize the importance of the “Voice of the Customer” and this means taking regular soundings from the sponsors of audit assignments, as well as other key stakeholders. At AstraZeneca, gathering feedback from key internal customers (assignment sponsors, senior managers and the audit committee) became a central foundation of our new way of working.

In addition auditors were asked to do a “lessons learned” review after each assignment. Things that had gone well were just as important as things that could be done better, since this is a way of building good practices and not just seeing the review process as negative. Post assignment reviews could result in: affirmation of the audit methodology (e.g. “It was great that we clarified the criteria for assessment early on, because that seemed to help later”), individual coaching points for auditors (and pin-pointing further training needs, if there was a recurring need for improvement), or even lead to clarifications or improvements in the audit methodology.

Following the lean mindset of delivering value to customers, it is good practice to supplement feedback from the auditee after each assignment with feedback from time to time from key stakeholders, such as assignment sponsors, senior management and the board.

Actions for internal audit to consider:

• Regard post assignment reviews as a fundamental engine for driving continuous improvement in the audit function;
• Do these reviews on a timely basis, focusing on what went well as well as what could be improved;
• Use learning reviews to drive auditor coaching as well as enhancements to the audit methodology or the way it is communicated;
• Ensure that feedback from assignment sponsors, senior managers and the board is also obtained, recognizing their role as key stakeholders – and not just auditees.

Revisit the Audit Methodology and Ensure it is Guiding Good Practice

At AstraZeneca one of the key changes we made after our lean review of audit was to overhaul the audit assignment methodology. The goal was to make it much easier to understand what good practice looks like, and to allow for flexibility, within a framework of having a small number of key mandatory requirements and to concentrate on value and driving productivity. The new audit methodology was much easier to follow, which made it easier to refer to on a regular basis, more straightforward to train new audit staff and guest auditors, and much easier to keep up to date.

Since then I have worked on the upgrading of a number of audit methodologies with CAEs to better focus on efficiency and value add. Here are the observations of Stephen Foster (CAE, Cargotec) about the benefits of such an upgraded approach:

“First of all I wanted to provide a consistent approach which is the same across the audit team. I think it’s important that an audit function has a solid base in relation to its key assignment process, the business would expect that.

The second reason is to get everybody on message in terms of the way we want the business to be looked at in order to add value during assignments.

Finally if we are trying to educate the business about the value of a structure and discipline in its processes, we need this sort of approach built into the audit way of working as well. It can help us say: ‘You should be looking at processes in this way – and this is how we ourselves do it.’

Updating our manual along lean principles has helped me, as the CAE, clarify how we can best add value.

As a consequence we have found ourselves, as a team, talking more and more about how we approach assignments and what value we are getting out of doing certain things. How do we justify that something is worth doing?

It has acted as a catalyst to actually put value-adding principles into practice.”

Actions for Internal Audit to consider:

• As an audit function, discuss the strengths and weaknesses of the current audit methodology (is it a useful tool, regularly referred to?);
• Does the methodology address the Gemba of real assignment challenges and guide auditors to navigate their way through challenges of delivery to time, completion of tasks and the need to add value?

Revisit Metrics for Their Usefulness and Focus on Value and Productivity

Whilst some traditional audit metrics have their place, lean ways of working encourage audit functions to focus on metrics that really drive added value and efficiency. This can mean enhancing some metrics significantly and dropping others. John Earley (Partner, Smart Chain International) summarizes the essence of what it means to have metrics that are lean:

“If you can’t make a direct line back to measuring customer value then you’ve got a question whether that metric is actually effective, or whether you are just filling in a box for the sake of it.”

At audit within AstraZeneca we dropped a number of metrics after our lean review, but improved our metrics around assignment delivery. One CAE explains their approach to managing assignment delivery:

“I know exactly where the audits are at any one time. As soon as they pass a deadline date, I will get an amber flag and expect the reasons to be chased, if slippage increases we step up our efforts to identify the reasons for this and the actions we need to take to progress the assignment without adversely impacting quality.”

Phil Gerrard (CAE, Rolls-Royce) outlines his approach to assignment metrics:

“I think managing budgeted times on audit, by key stage, can give you lots of intelligence around where your bottlenecks are. Look at these metrics regularly and look at the variation between what you planned for each assignment and what has happened and then try to understand the reasons for differences. Why did we get that wrong? What should we do differently? How can we be smarter? Or, oh, that was clever, what did we do there to deliver the assignment ahead of expectations”?

Good audit metrics should also address questions of adding value, Karen Dignan (CAE, Group Head Office, OMG) offers her reflections:

“We have a questionnaire after each assignment which is very short, just five questions. The main questions are narrative, rather than scoring. However, there’s an overall value rating that we ask the audit client to score and what’s more we ask ‘If you had to pay for the assignment, how much would you have paid for it?’

We also have a basket of other measures. Some of these are around how quickly we get our reports out, but we set different targets between local and group-wide audits, since the latter are typically more complex to land. Also we aim for 85% achievement of various targets – we try to recognize that there will be exceptions to simple time based targets.”

Greg Coleman (CAE, ITG) offers his reflections:

“The metrics that are used to measure internal audit functions often focus on areas such as ‘How quickly do you issue reports, how much money are you spending, how many audits are you issuing in a year?’ But really, they can be superficial. It doesn’t really speak to whether you are adding any value. Because anybody can issue 50 unimportant audits, if that’s your target.

In terms of adding value, it’s more about what audits you do and also being flexible regarding bringing in new audits if required, and helping out on projects, change initiatives etc.

I think adding value is as much about the feedback after assignments and how the organization reacts to you overall. If management come to you with requests and engage with you on a regular basis, I think that is a good start to ensuring you are a value adding audit function.”

Actions for Internal Audit to consider:

• As an audit function discuss whether current metrics focus adequately on value add and productivity;
• Is the time taken collecting metrics justified in terms of the benefit they are providing?
• Engage key stakeholders in a discussion about how metrics are helping to serve their interests.

Regard External Quality Assessments (EQAs) as a Performance Improvement Tool

My experience of working on audit External Quality Assessments (EQAs) is that if care is taken in engaging the right team to carry out the EQA, a good deal of added value can be obtained. I recommend that CAEs take a proactive approach to initiating EQAs since this should allow them to use the EQA to validate progress made on areas already identified as requiring improvement as well as highlighting key new areas for improvement. As much as possible, I recommend any EQA should always go beyond questions about compliance with IIA standards.

I start all of the EQA reviews I work on with the maxim: “No audit function is perfect” and with a view that the purpose of the EQA should not be to pinpoint all of the minor flaws in the operation of audit, but rather to focus attention on the most important opportunities (if they exist) to make a step change in relation to adding value or productivity.

I also start my work on EQAs with the view that those CAEs who request a quality review are showing a good degree of maturity since – sadly – many audit functions do not use this process to “step up their game”. I am certain that increasingly the best audit functions will be the ones that carry out quality reviews on a regular basis; after all, if we believe that internal audits can act as a catalyst for improvement in organizations, the same should be true for the audit functions, provided that the EQA itself is done from a progressive, value adding frame of mind.

If an audit function has not had an EQA for a while, I recommend carrying out a “pre-EQA” review comprising some self-assessment alongside some independent input, but not amounting to a full EQA; this helps to put the audit function on the right track before having a full EQA review.

I believe a good EQA should also be used to name issues about board and management engagement with audit, so that the EQA can help start important conversations about the risk and control culture of the organization and the appropriate role and focus of audit.

Actions for Internal Audit to consider:

• Take a proactive approach to EQAs;
• Take care to select the right EQA team that goes beyond compliance to look at value add and productivity;
• Act as a role model so that key stakeholders can see the value of independent assurance in practice;
• If it has been a while since the last EQA, consider a pre-EQA review to help gear up the audit function for a full EQA later without the cost and effort of a full EQA immediately.