Thursday, February 3, 2011

What is he difference between an Appraisal and an Audit?

What is the difference between an appraisal (ala CMMI) and an audit (ala TS9100)?  The answer to that has several parts, so stay with me :)
Part I: Intent
The intent of the authors of these documents (the CMMI-DEV model and the TS Audit Specification) is different.  CMMI is not a spec, it is a collection of "lessons learned" or "best practices" that are suggested by the authors, who were a collection of companies, consultants, and research scientists.  TS is a strict collection of work products and procedure requirements.  TS was intended as an audit method to ensure that organizations were "doing what they say they are doing" and that they comply, at the deliverable level, to the audit requirements.  The CMMI was intended as a tool for helping companies "see" how a great company behaves, and to use it as a way to incrementally make themselves better.   It assumes that companies get to decide what is best for them.  In a way, CMMI is one-level of abstraction higher than TS in it's requirement to adhere to certain behaviors, with CMMI being "more trusting" of the TS of your professionals.


Part 2: Collaboration
The SEI specifically states that since CMMI is something that is collaborative in nature (with the participants being management, practitioners, and the Lead Appraiser) it does not have an "audit" component, but is appraised for further opportunities for improvement by a joint team led by the Lead appraiser, whose job it is to ensure that the appraisal method is followed with full integrity.  To support Part I (above), it is the Lead Appraiser/Appraisal Team's job to simply identify strengths and weaknesses, not to "pass" an organization.  OEMs and the DOD have mis-applied the intent so that the line between "audit" and "appraisal" has been blurred.  But that doesn't mean they are the same thing.


Part 3: Behaviors vs. Documents
The TS audit seeks documents that 1) say what you are going to do and 2) show that you did them.  The CMMI seeks evidence that people understand the behaviors that are desired by management, and that they are using those behaviors, and continuously improving them.  Along the way they collaborate with their lead appraiser for advice and assistance in interpreting the model itself.  The documents are merely one form, among several, of demonstrating what behaviors have been used and what the results are.


Part 4: Interpretation of Evidence (this is probably the most important difference)
For a TS audit, documentary evidence is paramount.  Do the documents exist?  An Auditor is following a checklist . For a CMMI appraisal, the documents are merely one consequence of process, and it is vital that the Lead Appraiser and Appraisal Team have a great depth of understanding about how the organization works, what their business needs are, and what they are trying to accomplish.  They synthesize this to try to determine if what is being done is useful and meets the needs of the organization, and their findings will reflect that.  It's not pass or fail, but strengths and weaknesses.  The fact that OEMs and the DOD are trying to force a Pass/Fail mentality does not mean that it should be treated that way.

If you follow a process and a plan, and you incrementally make the improvements you need to make, then the appraisal will come when you are ready.  This is one reason why it's hard to "jam" it into a short schedule as you might a TS audit.  You need to convince the appraisal team (and the certified lead appraiser) that it truly is useful and an infrastructure exists for continuous improvement . . . . and that they are all being used by everyone!


Contact Broadsword