The Urban Legend of Multipass Hard Disk Overwrite

Sunday, August 28, 2011

Brian Smithson

7ca9cf570bb97d22b119f3a70d335ede

The Urban Legend of Multipass Hard Disk Overwrite and DoD 5220-22-M

Multipass disk overwrite and  the “DoD 5220-22-M standard 3-pass wipe” are, at best, urban legends. At worst, they are a waste of time and electricity.

Blame Gutmann...

In 1996, Peter Gutmann presented a paper [GUT96] at a USENIX Security Symposium in which he claimed that overwritten data could be recovered using magnetic force microscopy (MFM) and scanning tunneling microscopy (STM) techniques.

This seminal paper alerted many people to the possibility that data which had been overwritten on an HDD could be recovered using such techniques.

Lacking other research in this area, and despite a lack of corroboration, many of those people adopted Gutmann’s conclusions and recommendations and have ever since believed that multiple overwrites are required to effectively render remnant data irretrievable.

Gutmann’s ultimate recommendation was that no fewer than 35 (!) overwrite passes should be performed to ensure that the original data cannot be retrieved.

However, in the context of current HDD technology, there are several problems with Gutmann’s work:

  • Gutmann focused on two disk technologies — modified frequency modulation and run-length-limited encoding — that rely on detection of a narrow range of analog signal values and have not been used for HDDs in the last 10-15 years. Modern HDDs use various kinds of partial-response maximum-likelihood (PRML) sequence detection that uses statistical techniques to determine the maximum likelihood value associated with multiple signal detections [WRIG08].
  • Further, areal density (density of data per square unit of area, the product of bit-per-inch linear density and track-per-inch track density) has increase by at least three orders of magnitude [SOBE04] [WIKI08] since the publication the Gutmann paper. To achieve such densities, head positioning actuators have become significantly more accurate and repeatable.
  • Moreover, Gutmann’s work paper was theoretical, and I am not aware of any practical validation that data could be recovered using the techniques he described.

Gutmann’s work has resulted in the formation of an urban legend: that the US government requires a 3-pass overwrite and specifies it in DoD 5220-22-M.

What about those often-cited US Government standards?

There are many HDD overwrite standards from which to choose [BLAN08]. Among those that are often cited in both procurement and product specifications are DoD 5220.22-M and NSA 130-1. Less often cited, but more current, is NIST SP 800-88.

DoD 5220-22-M

DoD 5220-22-M is the National Industrial Security Program Operating Manual (NISPOM), which a broad manual of procedures and requirements for government contractors handling classified information.

The 1997 version of this document [DOD_97] specified that rigid magnetic disks should be sanitized by writing some character, its complement, and then a random character. However, this “algorithm” was removed from subsequent issues of the NISPOM.

Indeed, the entire table of clearing and sanitization methods is no longer present in the current issue of NISPOM [DOD_06].

NSA 130-1

NSA 130-1 may well have specified a clearing or sanitization procedure by writing a random character, another random character, and then a known value. However, I am not able to find a copy of NSA Manual 130-1 or 130-2 (perhaps they were classified documents).

However, the current issue of the NSA/CSS Storage Device Declassification Manual [NSA_07] (Manual 9-12, which supersedes Manual 130-2) does not specify any overwriting methods for HDDs, and instead requires degaussing or physical destruction.

It is not clear to me if the DoD and NSA no longer recommend overwrite methods because they are ineffective or because their effectiveness as a single technique is uncertain when applied to a variety of HDD technologies.

NIST Special Publication 800-88

The National Institute of Standards and Technology has a special publication “Guidelines for Media Sanitization” that allows HDD clearing by overwriting media “using agency-approved and validated overwriting technologies/methods/tools”.

For purging, it specifies the Secure Erase [UCSD10] function (for ATA-based devices), degaussing, destruction, or the rather vague “purge media by using agency-approved and validated purge technologies/tools”.

The original issue of SP 800-88 [NIST06-1] claimed that “Encryption is not a generally accepted means of sanitization. The increasing power of computers decreases the time needed to crack cipher text and therefore the inability to recover the encrypted data can not be assured”, but that text was removed from SP 800-88 Revision 1 which was issued one month later.

Most interestingly, SP 800-88 states that “NSA has researched that one overwrite is good enough to sanitize most drives”. Unfortunately, the NSA’s research does not appear to have been published for public consumption.

Current Research

Fortunately, several security researchers presented a paper [WRIG08] at the Fourth International Conference on Information Systems Security (ICISS 2008) that declares the “great wiping controversy” about how many passes of overwriting with various data values to be settled: their research demonstrates that a single overwrite using an arbitrary data value will render the original data irretrievable even if MFM and STM techniques are employed.

The researchers found that the probability of recovering a single bit from a previously used HDD was only slightly better than a coin toss, and that the probability of recovering more bits decreases exponentially so that it quickly becomes close to zero.

Therefore, a single pass overwrite with any arbitrary value (randomly chosen or not) is sufficient to render the original HDD data effectively irretrievable.

References

Cross-posted from Grot
Possibly Related Articles:
48013
General
Hardware
data destruction Hardware Secure Erase Degaussing HDD Hard Drives DoD 5220-22-M
Post Rating I Like this!
B32b392ce3a707f05f4838c48c67d9cf
Christopher Hudel Very well written! I think too often we tend to "hold on" to urban legends and requirements like these tend to stay in policy and process documentation without question.
1314710424
8b5e0b54dfecaa052afa016cd32b9837
Craig S Wright I am happy that our research is starting to make it into more mainstream sources. Later this year we will have a paper ready for publication detailing the recovery of not only wiped data, but drilled, crushed, heated, magnetized and more. The economics and time constraints will also be discussed.
1314773662
B8b580348b4e717042d0e394ee072001
security curmudgeon Did you read Gutmann's paper in the last ten years? He has added a note at the top:

This paper is now more than fifteen years old, and discusses disk storage
technology that was current 15-20 years ago. For an update on the current
situation with data deletion see the [Link: "epilogue"].

He also has two epilogues, one that deals with WRIG08:

So while it fairly convincingly demonstrates that applying the wrong
technique to the wrong technology doesn't work, it unfortunately doesn't
expand the body of knowledge of secure data deletion much.
1314851283
8b5e0b54dfecaa052afa016cd32b9837
Craig S Wright Gutmann never tested anything. He made a theory and proposed it as fact. I saw his reply and he has just sought to confuse the issue. In the testing, older and newer drives had been used.

His idea of using an oscilloscope was never valid even using floppy disks.

The idea behind Gutmann's paper was fundamentally flawed and based on a misunderstanding of physics.

Others such as the NSA and NIST have validated the findings we made in WRIG08, but then being .gov we can see these are conspiracy theory?
1314852288
7ca9cf570bb97d22b119f3a70d335ede
Brian Smithson Craig, thank you for the comments. I can't wait to see your new paper!
1314989600
70e177868d7bc383ce3ea10b6f976ada
Andrew Baker Gutmann does have some updated info, as found here: http://www.cs.auckland.ac.nz/~pgut001/pubs/secure_del.html#Epilogue
1315135280
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.