Standards, Audits, and Certifications: Which One is Right?

Tuesday, January 10, 2012

Jon Long


So Many Security Standards, Audits, and Certifications. Which One is Right?!

Last week I had a great conversation with a prominent member of the information security community (@HackSec), and he raised the following concern: 

Many are confused about when to use ISO 27001 certificationPCI certification, SOC 1 (aka SSAE16), SOC 2 & 3, NIST, and CSA STAR to give customers comfort about their security. 

I explained my position that I think organizations should use SOC2 as an umbrella for all of them especially because many service organizations are required by their customers to comply with multiple standards and produce multiple reports. 

To this, he expressed his frustration that if the information security community cannot decide which one to standardize on, how can customers be expected to know what to do?

He said: "We'll stay in this nexus of confusion where every attestation is required, and yet customers still do not have confidence in our security and demand the right to audit." He went on to explain that the Trust Services Principles and Criteria (TSPC), that a SOC 2 is based on, is not detailed enough, and that CSA STAR and NIST are much more thorough standards for ensuring security.  

In our continuing dialogue, I explained the idea that they can all get along under the SOC2 umbrella. Here's why: Accountants understand auditing, security professionals know security, and the international standards organization is just that... international.  

Each has something that the other does not, and if you bring it all together, you have one heck of a team.  Let me explain a little more about each:

image Accountants understand auditing:  Accountants can trace the history of auditing back to ancient Egypt, but in its more modern form of independent CPA accounting back to 1593.  

They were auditing and being held responsible for their opinions (through lawsuits) long before information security was invented.  

By this I am trying to say that there is an audit process that is missing from the security assessment space.

PCI and ISO 27001 assessments provide a point in time assurance which is no assurance at all, and CSA STAR is a self-assessment at this point.  While I agree that the TSPC is a weak standard, at least with SOC2 you get a period of time assurance by taking advantage of the audit process that the attestation standard requires.  

Then all you have to do is add the other standards into the covered areas and enhance the audit procedures to ensure that the controls are not only "in place", but that they "have been in place" for the period of time that is covered by the report.   image

Security professionals know security:  CSA STAR, NIST, and ISO27001 are great standards, and security professionals are the only ones who can test them. 

Accountants know that they cannot test security which is probably why the TSPC are so vague.  Security professionals have the right security standards, but they do not understand what assurance is, or how it is achieved. 

The very fact that entire industries throw around words like "certification" and "compliant" demonstrate this.  Accountants understand that when you use words like this, the entity providing the attestation is opened up to huge liability. 

Accountants are very careful to design and perform their tests to mitigate this risk, and use terms like "reasonable assurance", "in all material respects", and "in our opinion" to ensure that organizations that rely on their opinion know exactly what they can rely on.   image

ISO is international:  Professionals in other countries resist "American" standards because it makes them feel less sovereign.  They hate the arrogance of it.  

Right now there are no International Standards on Attestation Engagements (ISAEs) for security because ISO 27001 dominates in that space, and international accountants are not so brazen that they think they can get into the security space like the AICPA is.  

So for now SOC2 is all there is for period of time attestations, and it is embraced by Canada because they invented TSPC.  That is a start, and if we add ISO 27001 under a SOC2 umbrella, we're golden with the international community.  They would get period of time coverage, and strong security controls.

The SOC2 attestation standard is flexible enough to incorporate "additional subject matter", and all of the previously mentioned standards can be covered in the auditor's opinion as long as accountants use competent "technical specialists" to test the controls. 

This has led some to argue that SOC2 is the "Silver Bullet" that satisfies all compliance and reporting requirements.  However, even if accountants use competent security professionals, there is still a problem; they cannot issue the reports that customers want such as ISO 27001 certifications, PCI Reports on Compliance (ROCs), or CSA STAR attestations because they are controlled by governing bodies that CPA firms are not registered with. 

Most of the large CPA firms will never associate with those organizations because they lack influence in them, and consequently cannot control the risk it exposes them to.   There is a way that service organizations can avoid the dilemma of having to undergo multiple audits to satisfy their customer's demands for multiple reports. 

The way it works is that a CPA firm partners with an ISO certifying organization, security firm proficient in CSA STAR, or QSA (in the case of a PCI report) to jointly conduct the testing.  Because there is significant overlap in the standards, service organizations can take advantage of the testing efficiencies that result.

At the completion of the engagement, the organization will receive multiple reports from a single attestation engagement.  This approach takes advantage of the best of all worlds: great audit process, the best security standards, and risk assurance for their client that is meaningful.

Cross-posted from The Risk Assurance Guy

var sc_project=7558660; var sc_invisible=1; var sc_security="7ecbc61c"; free hit counters

Cloud Security General HIPAA PCI DSS
Post Rating I Like this!
Hedge Hog For good reason, SOC 2 has minimal acceptance in the US and the outlook is not bright. The practical application of the examination is a nightmare. When given the choice, companies would be better off selecting ISO 27001 certification. Unlike the convoluted requirements of SOC 2, ISO 27001 is clear cut and more holistic, has enjoyed a nearly 20 year track record stretching back to BS 17799, results in a certification, and has international recognition. Additionally, a Google search of news releases including the two terms will demonstrate a far more impressive list of companies opting for ISO 27001 certification.
Jon Long Thank you so much for the comment. Perhaps I could ask you to read the article again? ISO 27001 is a point in time assurance which is no assurance at all. ISO 27001 can be tested under SOC2 audit process to provide period of time assurance.
Hedge Hog That's incorrect. ISO 27001 certification is valid for three years into the future assuming successful completion of requisite annual surveillance audits. In a lot of people's minds, that is superior to an SOC 2 report which may opine on a point in time (Type 1) or a period of time (Type 2), both of which are always in the past, thus making the SOC 2 report "dated" before it is even issued.

I gather that you don't work for a CPA firm and presumably have little or no hands on experience performing SOC 2 examinations. Those of us that have had to perform SOC 2 examinations already knows that it is definitely NOT the one audit to unite them all. Although your claim is theoretically true that portions of the ISO 27001/2 standard could be included in an SOC 2 examination, the result would be so sloppy and convoluted. It is so impractical that it is hard to understand why any company would attempt to do so.

All of the audits you speak of have specific purposes with intended audiences. A company that needs/wants SOC 2 and ISO 27001 certification (or coverage of 27002 topics) would be well advised to have them performed separately and not be the "test case" for the theoretical.
Jon Long HedgeHog, thank you for you second comment. In order to prove your assertion that ISO 27001 is not a point in time assurance, and in fact gives ongoing or real-time assurance, which would be amazing if it were true, I would respectfully ask for a link to an independent third party 27001 auditor's opinion letter that provides such an assurance.

As to period of time assurance being "dated", it is true that it provides reasonable assurance that controls placed in operation were effective over the period of time that is covered by the report which can only be historical in nature. What this accomplishes for organizations is a cultural adoption of their security controls leading to an increased likelihood that those controls will remain effective in the future. Hedge Hog, if you are a CPA, you should understand this.
Hedge Hog Simply go to the BSI cert directory ( start typing in major corporations' names. You should find hundreds of examples of accredited ISO registrar certifications of on-going conformance with the ISO 27001 requirements. But even if you were right, the very same flaw exists in SOC 2. There is nothing on-going or real-time about SOC 2. The fact that an SOC 2 report may opine over a period of time does not change that.

Your second statement is misleading as well. First of all, the claim that an SOC 2 examination "accomplishes for organizations is a cultural adoption of their security controls leading to an increased likelihood that those controls will remain effective in the future" is not a trait unique to SOC 2. It's equally true of ISO 27001, SOC 1, SOC 3, PCI DSS, FedRAMP, etc. Secondly, that's not the point of SOC 2; but ironically, it is a pretty close description of ISO 27001 ISMS certification. Every SOC 2 opinion letter firmly contradicts your point when it advises the user of the SOC 2 report to refrain from making any assumptions about future periods. Every opinion letter warns that "...the projection to the future of any evaluation of the fairness of the presentation of the description or conclusions about the suitability of the design or operating effectiveness of the controls to meet the applicable trust services criteria is subject to the risks that the system may change or that controls at a service organization may become inadequate or fail."

So as I said, ISO certification provides assurance into the future that an effective system of controls is in place. SOC 2 does not. The distinction is slight as it has to do with the design of the audit format.

At the end of the day, your article makes the error of assuming that CPA firms don't have in-house info sec / IT audit capabilities. This is patently false. It ruins your conclusion b/c you are suggesting that CPA firms team with a third parties to perform SOC 2 examinations. Before doing so, a CPA firm would have to conclude that it lacks the requisite "capabilities and confidence" to perform an SOC 2 examination. If a firm is not qualified to perform the work itself, it also lacks the ability to oversee and QA the work of the third party. Professional standards obligate such a firm to refrain from performing engagements they are not qualified to perform. You have basically written an article that recommends CPA firms engage in malpractice, the result of which could be very troublesome for any CPA firm or the poor clients that might get caught up in such a mess.

Ref. SOC 2 Guide Par. 2.03
Jon Long Hedge Hog, thank you very much for your comments. Hearing the kinds of objections that I will be faced with as I go forward with promoting this solution helps me to refine my message.

The link you gave us only shows the certification status of the listed companies as either active or inactive. I am looking for the actual report that is provided by the independent third party auditor that states their responsibility, and that the assurance they are providing the company extends to the future per your assertion.

As to SOC2 not being any better, this is where we disagree. Please let me explain. No third party, including ISO certifying organizations, can provide assurance that controls will remain effective after they leave, because they do not have control over their client's environment. The client can stray from the established control design at any time without the auditor even knowing about it.

The difference between the ISO and the SOC audit approach, however, is that SOC engagements incorporate statistical sampling in the testing procedures to increase the likelihood of catching companies when they did not follow the program while the auditor was away. ISO is focused on validating whether “…IT systems are configured in accordance with the organization’s information security policies, standards and guidelines.” SOC engagements validate whether IT systems have remained configured per the standards over the period of time selected.

A practical and easy to illustrate example of this would be the password length parameter of any given platform or application. ISO validates that the length is configured to be 8 characters long. SOC validates that the length has been configured to be 8 characters long during the period of time selected. The SOC approach is superior to the ISO approach because, under the ISO approach, companies can simply revert to more secure settings right before their auditor performs the point in time testing, and pass the audit. The SOC approach is much more likely to expose when companies do this through analysis of logs, file integrity monitoring, and the use of other continuous audit tools and techniques.

This is why I referenced the cultural adoption benefit of the SOC approach. When companies figure out that they actually have to live by their controls, instead of just cleaning up their environment before the auditors get there, there is a forced culture change. I understand that CPA firms cannot warrant that the control will remain effective going forward, hence the language in the report, but users understand that they are much more likely to, given the rigorous nature of period of time testing.

Finally, perhaps I was not clear enough about the purpose of partnering with QSAs and ISO certifying organizations. It is to take advantage of their expertise in testing the more comprehensive security controls that are included in the ISO and PCI-DSS standard. I reject the assertion that a CPA firm is unable to supervise the work of technical specialists unless they are competent to perform the work themselves. If that were the case, there would be no need for technical specialists in the first place. While it is sensational to accuse me of writing an article that encourages professional malpractice, the facts do not support your assertion.

Please forgive my lengthy reply. By the way, all of our comments are ending up being longer than the original article! This is good, and I’m happy that it has prompted this dialogue.
Hedge Hog Ref. SOC 2 Guide Par. 2.03 is clear regarding engagement acceptance requirements. Your opinion to the contrary is erroneous. SAS 73 deals with the use of specialist. Per the first paragraph of the SAS, "a specialist is a person (or firm) possessing special skill or knowledge in a particular field other than accounting or auditing." Obviously, any QSA, ISO Registrar, or info sec compliance company (etc.) is specialized in auditing controls and can not be considered a "specialist" for the purpose you describe. A CPA firm that accepts an SOC 2 engagement is required to have the requisite expertise, or pass on the engagement. To do otherwise would be a violation of professional standards.

Anyone that wishes to independently confirm engagement acceptance requirements should e-mail the AICPA's audit and attestation task force at

SAS 73 -
Jon Long Hedge Hog, thank you once again for your comment. That is what is great about the AICPA's shift to the attestation standards. We are in the world of attestation standards now, not audit standards which is what SAS 73 is. See SSAE no. 14 and AT-101 (

Adequate Knowledge of Subject Matter
.21 The second general standard is—The practitioner must have adequate knowledge of the subject matter. [As amended, effective when the subject matter or assertion is as of or for a period ending on or after December 15, 2006, by Statement on Standards for Attestation Engagements No. 14.]
.22 A practitioner may obtain adequate knowledge of the subject matter through formal or continuing education, including self-study, or through practical experience. However, this standard does not necessarily require a practitioner to personally acquire all of the necessary knowledge in the subject matter to be qualified to express a conclusion. This knowledge requirement may be met, in part, through the use of one or more specialists on a particular attest engagement if the practitioner has sufficient knowledge of the subject matter (a) to communicate to the specialist the objectives of the work and (b) to evaluate the specialist's work to determine if the objectives were achieved.
Hedge Hog To imply that the SAS's are not relevant to SSAE is not correct. I recommend that you consult with a CPA regarding the current hierarchy of authoritative standards.

But even if you were correct, your position is not practical. A company would be ill-advised to select a CPA firm that lacks the in-house capability to perform a SOC 2 when so many qualified CPA firms are readily available. Down sides of doing so include experiencing a lack of integration and methodology from the providers, paying increased fees resulting from duplicate overhead, and interacting with personnel that are generally inexperienced as demonstrated by the need to outsource (i.e., blind leading the blind), etc., etc.

The best method for a company starting at square one would be to type "SOC 2" into Google and select from licensed CPA firms that have the requisite in-house SOC 2 expertise. AVOID ANY CPA FIRM THAT HAS TO RELY ON THIRD PARTIES TO SUPPLEMENT THEIR EXPERTISE (OR LACK THEREOF)!
Jon Long Hedge Hog, thank you for your input.
nikita reva John and Hedge raised a very interesting topic, SOC vs. ISO-27001 and the other alphabet soup. I have the ISO-27001 LA cert but can honestly say that the ISO standards are mind-numbing, require serious interpretation of applicability and are honestly outdated. In my opinion, SOC is an improvement over SAS-70 as it introduces the trust principles, however it’s not a silver bullet. At the end of the day, building trust in your providers is not a matter of SOC or ISO stamps of approval. In my opinion, these assurance documents suggest a higher level of maturity. As you can imagine, Cloud Computing will continue to lead the push for assurance for the next decade... Organizations are demanding more transparency and clarity around liability from their providers. How will they achieve this? This will keep the lawyers busy for some time. From a practical standpoint, I think the answer depends on what matters to them. Any wise organization should use their enterprise risk mgmt processes to assess the provider against controls that matter to them.
Jon Long More on the fact that ISO 27001 is only a "point-in-time" assurance:
April Sage Great discussion and clarification Jon.

I thought I would share our perspectives as a company who has gone through many of these audits. For reference, Online Tech is a data center hosting company, which is the basis for the perspective below.

First, I see the concern about the point-in-time audits vs audited over time. From our experience, going through the SAS 70 Type II, SSAE 16 Type II, and SOC 2 audits certainly did contribute to our company culture integrating and embracing the changes over time, but we're fortunate to have a strongly process and compliance oriented management and operations teams, so perhaps we take security and compliance issues to heart in ways that other companies who are scrambling to meet a point-in-time audit are not. So yes, I believe that the "over time" audits help encourage and ingrain the security and compliance mind set in day-to-day operations.

That said, you can't really say that the PCI audit is meaningless, because of all the audits, PCI is one of the more sophisticated, rigorous, and prescriptive in terms of what technologies must be in place. The penetration and vulnerability testing really contributed to the PCI audit being the one that helps our CEO sleep best at night. Let me add that in this case especially, it was completely appropriate to combine the talents of both our auditors (UHY Advisors) AND PCI security experts (High Bit Security). Much as I respect and appreciate our auditors, I wouldn't give their vulnerability and penetration testing much credibility if they tried to do it themselves. The mark of a professional is knowing what you don't know. To me, it makes perfect sense to combine experts from both the security and compliance fields.

For SOX compliance, I would have to agree that SOC 2 is significant improvement over SAS 70 and SSAE 16 for service organizations. It's of much greater importance to our clients to know that our policies around security, privacy, availability, processing integrity, and confidentiality (the focus of SOC 2) vs the financial controls of our accounting department (the focus of SSAE 16). In addition, because SOC 2 has a pre-defined, consistent set of criteria with which to compare service organizations, it's a whole different animal than SAS 70 and SSAE 16 where companies get to choose the standards they are audited against. I mean really, does an "A" on a test mean much if you got to choose your own final exam questions? Of course not. You need a standard set of questions as common ground in order to make any kind of intelligent comparison. For those who are stuck having to rely on SAS 70 and SSAE 16 reports if service organizations haven't invested in a SOC 2, I really hope they read and compare the fine print of the audited controls. Some companies define a dozen or less controls. Some companies choose 100. Some say that "we lock the doors at night" is an appropriate security control for an audit. But where sensitive PHI, CHD, or other data is concerned, that just won't cut it.

Despite the improvement of the SOC 2 over SAS 70 and SSAE 16, in our experience, it still has a far distance to go to meet the security rigor of PCI.

Wish I had enough experience with ISO to share perspectives there, but, not yet! :-)

Jon Long Well said April.
Jon Long Hedge Hog, thank you for linking to Infosec Island from Twitter. As to your comment, it appears that Douglas Barbin from your firm agrees with me. His comment is here:
Jon Long I would appreciate it if everyone would please "Like" my post if you agree.
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.