Data remanence is the residual representation of data that has been in some way nominally erased or removed. This residue may be due to data being left intact by a nominal delete operation, or through physical properties of the storage medium. Data remanence may make inadvertent disclosure of sensitive information possible, should the storage media be released into an uncontrolled environment (e.g., thrown in the trash, or given to a third-party).
Causes
Many operating systems, file managers, and other software provide a facility where a file is not immediately deleted when the user requests that action. Instead, the file is moved to a holding area, to allow the user to easily revert a mistake.
Even when an explicit deleted file retention facility is not provided or when the user does not use it, most computers do not actually remove the contents of a file when it is deleted, simply because this requires less work and is therefore usually faster. Instead, they simply remove the file's entry from the file system directory. The contents of the file -- the actual data -- remain on the storage medium. The data will remain there until the operating system reuses the space for new data. In some systems, enough filesystem metadata is also left behind to enable easy undeletion by commonly available utility software. Even when undelete has become impossible, the data, until it has been overwritten, can be read by software that reads disk sectors directly.
Likewise, reformatting, repartitioning or reimaging a system is not always guaranteed to write to every area of the disk, though all will cause the disk to appear empty or, in the case of reimaging, empty except for the files present in the image, to most software.
Finally, even when the storage medium is overwritten, physical properties of the medium may make it possible to recover the previous contents using laboratory techniques.
Overwriting
A common method used to counter data remanence is to overwrite the storage medium with new data. This is often called a wiping or shredding a file or disk. Because such methods can often be implemented in software alone, and may be able to selectively target only part of a medium, it is a popular, low-cost option for some applications. Overwriting is generally an acceptable method of clearing, as long as the media is writable and not damaged.
The simplest overwrite technique writes the same data everywhere -- often just a pattern of all zeros. At a minimum, this will prevent the data from being retrieved simply by reading from the medium again using standard system functions.
To counter more advanced data recovery techniques, specific overwrite patterns are often prescribed. These may be generic patterns intended to eradicate any trace signatures. For example, writing repeated, alternating patterns of ones and zeros may be more effective than zeros alone. Combinations of patterns are frequently specified.
One challenge with an overwrite is that some areas of the disk may be inaccessible, due to media degradation or other errors. Software overwrite may also be problematic in high-security environments which require stronger controls on data commingling than can be provided by the software in use. The use of advanced storage technologies may also make file-based overwrite ineffective.
Feasibility of recovering overwritten data
Peter Gutmann investigated data recovery from nominally overwritten media in the mid-1990s. He suggested magnetic force microscopy may be able to recover such data, and developed specific patterns, for specific drive technologies, designed to counter such. These patterns have come to be known as the Gutmann method.
Daniel Feenberg, an economist at the private National Bureau of Economic Research, claims that the chances of overwritten data being recovered from a modern hard drive amount to "urban legend".
As of Nov 2007, the DoD considers overwriting acceptable for clearing magnetic media, but not as a sanitization method. Only degaussing or physical destruction is acceptable for the latter.
On the other hand, according to the 2006 NIST Special Publication 800-88 (p. 7): "Studies have shown that most of today’s media can be effectively cleared by one overwrite" and "for ATA disk drives manufactured after 2001 (over 15 GB) the terms clearing and purging have converged."