Bit Rot: A Threat To Digital Data?

[Leesbro63]

What exactly is BIT ROT? And is it a threat for those of us who are entirely "paperless"? I was chatting with an archivist at the local history center and she was explaining that going digital isn't a perfect solution for them. Because of bit rot. And it got me thinking...

[Zagor]

http://en.wikipedia.org/wiki/Bit_rot

The key is to always have multiple back-ups in different locations.

[prudent]

Bit rot refers to how digital records can "decay" over time. Magnetic media (floppies, hard drives), flash media and even optical media (CDs/DVDs) cannot store data indefinitely. Film holds up better over long periods of time but requires proper storage. This would be a real concern to an archivist. Not only is bit rot something to worry about, but there is also file format obsolescence and hardware/application obsolescence to deal with. We create digital records and assume that software to read JPG and PDF files is always going to be around, and we will always have a CD drive that can read them.

At a conference a few years ago, one of the presenters mentioned that it was less than 20 years after the moon landing when NASA discovered a lot of the telemetry data they collected was unusable - the data on the magnetic tapes was decaying, plus they no longer had the computers and the software to interpret the data.

Is it a concern? Probably not for the average person. If for some reason CD drives became obsolete in your lifetime, you'd know in time to transfer your data to something else. For an archivist, they have to think in much longer spans of time.

[statsguy]

Not only is bit rot something to worry about, but there is also file format obsolescence and hardware/application obsolescence to deal with.

Tell me about it, we backed up onto 3.5" disks that we can no longer access.

[bertilak]

Not only is bit rot something to worry about, but there is also file format obsolescence and hardware/application obsolescence to deal with.

Tell me about it, we backed up onto 3.5" disks that we can no longer access.

How about us poor souls that tried to use Zip and Jazz drives? (Curse you, Iomega!)

Luckily the Zip drives and media were so bad it was almost immediately obvious that they could not be relied on for more than occasionally moving data between non-networked systems.

Lesson: Never use proprietary formats! (I.e. if you have any DOC files you want to be able to look at in 10-20 years you'd better print them off or convert them to PDF.

I still have a few old documents in DeScribe format.

[Leesbro63]

So is BITROT a worry for me or not? I'm not worried about media obsolescence...I'll keep up as new technologies develop.

[Epsilon Delta]

So is BITROT a worry for me or not? I'm not worried about media obsolescence...I'll keep up as new technologies develop.

Yes, because new technology is a large part of the BITROT problem. The price of retaining data is eternal vigilance.

[JuanZ]

In another forum (for SPSS, a statistical package) people were complaining that the forum archives from 1991 to 1996 are lost. It's not just a question of physical backups, but of whole institutional infrastructures being wiped out, not accesible online anymore.

[patrick]

In my opinion, the sky isn't falling. Assuming that technological civilization doesn't collapse I don't expect major problems in the future. Today, I have little difficulty viewing word processing documents or bitmap images I made in 1992, or even running programs I wrote in 1992. And the computer I used then was neither a PC nor a Mac!

3.5" floppy disk drives stopped being standard equipment on PCs sometime around 10 years ago (give or take, depending on how common something must be to count as "standard") but you can still buy new ones today for $20 that connect to the USB port of any modern PC. 5.25" floppies stopped being the standard 20 years ago, but you can still buy used equipment to read them quite cheap on eBay. So even if DVD/Blu-ray drives are no longer included in PCs soon, I would expect similar results. With all the millions of people that have huge CD/DVD/Blu-ray collections, do you really think it would soon happen that there would not only be no market to make new ones addons like the new 3.5" floppy drives are today, but also that all the old ones that have already been made would vanish?

As to file format, consider what it would take for a widely used format like Microsoft Word documents to become unreadable. Microsoft going bankrupt wouldn't be enough. You'd also have to assume that no one buys up the bankrupt assets and continues making Word. And you'd also have to assume OpenOffice would somehow decide to shut down their efforts or remove the ability to open Word documents from OpenOffice. And you'd have to assume computers would change enough that the old versions of Word and OpenOffice that could open Word documents were unusable. And you'd also have to assume that no one would make an emulator for the older computers that would run on the newer computers.

Media will of course wear out eventually, but even that seems a bit overblown. The big advantage of digital is of course that you can always make an exact copy, allowing the data to outlast the media, unlike analog which will degrade when copied. But even then I have been able to succesfully read hard disks over 20 years old, and sometimes even 20 year old floppy disks. I don't have any 20 year old CD-Rs but have read them after more than 10 years. Purely for curiosity's sake, of course -- I had copied them to newer disks much sooner and wouldn't recommend relying on media lasting the long -- better safe than sorry.

[dumbmoney]

Media will of course wear out eventually, but even that seems a bit overblown. The big advantage of digital is of course that you can always make an exact copy, allowing the data to outlast the media, unlike analog which will degrade when copied. But even then I have been able to succesfully read hard disks over 20 years old, and sometimes even 20 year old floppy disks. I don't have any 20 year old CD-Rs but have read them after more than 10 years. Purely for curiosity's sake, of course -- I had copied them to newer disks much sooner and wouldn't recommend relying on media lasting the long -- better safe than sorry.

My oldest CD-Rs are from 1994 and they are still readable. But as you say, you can't rely on media longevity. Regular copying is key.

[Mudpuppy]

Just as an example of how bitrot can creep in, OpenOffice will not open WordPerfect files, not due to technical reasons, but because the license for the WordPerfect importer is incompatible with the license for OpenOffice (meanwhile, it is compatible with the license for LibreOffice). Often times, it is policy and license conflicts that contribute more towards file format bitrot than technical issues.

[pjstack]

Um,... There's always paper (gasp!)

[pheleven]

Most of these answers are not about bitrot - they're about media obsolescence. Both are problems, but very different ones.

Media obsolescence is a medium to long term problem and one you can likely easily mitigate against if you're talking about storing relatively small quantities of data - as you likely are for personal data.

Bitrot is a (potential) problem today, compounding tomorrow. Bitrot occurs in a static copy from media going bad (hard drives, cd/dvds, tapes, etc having mechanical failure - often intermittent and undetected - or wearing out), it can occur during a copy for a large number of reasons such as your RAM 'flipping a bit' (solar flare, power surge - less common bitrot), and it can occur from a software error (your OS or a program interacting with the files). Bitrot is what causes corruption, it's fairly common. Most of the time many backups are the solution, but if you copy an already corrupted copy, you've just got two bad copies.

There are very excellent ways to combat bitrot, but they are not all that common (yet?) - especially for the average consumer. The best way is to keep hashes [http://en.wikipedia.org/wiki/Hash_function] of all of your data - this is not convenient to do yourself, but your filesystem can do it for you [http://en.wikipedia.org/wiki/ZFS] - this protects from the mechanical/wear issues and potentially software issues. You can also choose hardware that is more resilient to bitrot [http://en.wikipedia.org/wiki/ECC_memory], though it is more common in servers and mid to high end workstations - this protects you from (some) random hardware issues and solar flares/random chance. And finally backups and snapshots are the only way to protect against software issues.

In the end backups are enough for most people, most of the time. At work we rely on a combination of all of the above to combat bitrot.

PS - You can totally keep your own hashes of your data if you want; md5 and sha1 are common hash algorithms across all common operating systems.

To others suggesting digital copies are 'exact', you're right... probably 99.999% of the time - I move around and store a lot of files and that's not anywhere near good enough to trust it's 'exact' without proof.

[Valuethinker]

Um,... There's always paper (gasp!)

I don't know how bad a problem it is now, but every looked at a 50 year old paperback? (there are many in my parents' basement). Crumbling.

Most paper in the 20th century was high acid. It won't last a century. Whereas documents from the 1500s, 1600s, 1700s are still around, most paper documents from the 20th centurey will be dust in 100 years.

http://en.wikipedia.org/wiki/Acid-free_paper

Paper made from wood-based pulp that has not had its lignin removed turns yellow, becomes brittle, and deteriorates over time.[2] When exposed to light and/or heat, the molecules in the acidic paper will break down even faster.[3] It was only in the 1930s that the effects of wood-pulp paper became known, when William Barrow (a librarian) published a report about the deterioration of acidic paper in the libraries.[4]

A related problem has been the destruction of paper documents for space considerations. Nicholson Baker wrote a book about it. The British Library was going to destroy its collection of 19th century American newspapers-- I think he managed to get it bought by an American archive. But most libraries and librarians are looking to go digital and cut storage space-- storing paper documents is expensive.

After Watergate, I suspect government departments everywhere in the world destroy a lot more than they used to.

The destruction of almost the entire 20th century archives of Iraq during the early days after the fall of Baghdad was a huge tragedy for historians. Doubtless it is, or will be, repeated in Libya, Syria etc.

[Valuethinker]

The canonical example is the Landsat photographs from a NASA satellite in the early 1970s. First comprehensive survey of the Earth.

When 20 years later they wanted to look at them again to see how human habitation was changing, there was no longer any way of reading the data. They had to rebuild the machine from scratch to do it.

Partly in response to this, with the atomic bomb, as the US no longer tests atomic weapons, they videoed interviews with the designers and builders, people who had worked on the bomb in the 40s, 50s, 60s. There is a lot of 'art' to building a bomb, the science is apparently fairly trivial but the knack is not. They didn't want to lose that.

[Valuethinker]

We should also be aware that any kind of high atmospheric level atomic detonation, or a specifically designed Electro Magnetic Pulse (EMP) weapon, could wipe out much of the data in a wide area below it.

So if there was some sort of 'high altitude war' that led to say 10-20 atomic detonations in the stratosphere, a *lot* of data would be corrupted, forever. Depends then really whether you have backups on the far side of the world.

[pheleven]

We should also be aware that any kind of high atmospheric level atomic detonation, or a specifically designed Electro Magnetic Pulse (EMP) weapon, could wipe out much of the data in a wide area below it.

So if there was some sort of 'high altitude war' that led to say 10-20 atomic detonations in the stratosphere, a *lot* of data would be corrupted, forever....

Better get one of these to hold your (properly hashed) backups... http://en.wikipedia.org/wiki/Faraday_cage

[kitteh]

Um,... There's always paper (gasp!)

I don't know how bad a problem it is now, but every looked at a 50 year old paperback? (there are many in my parents' basement). Crumbling.

Most paper in the 20th century was high acid. It won't last a century. Whereas documents from the 1500s, 1600s, 1700s are still around, most paper documents from the 20th centurey will be dust in 100 years.

As you note, acid free paper is the solution to this, and is widely used in legal documents, for example. While current paperbacks do not last long, hardcover books I have that are fifty years old or older are in fine shape. Meanwhile books from centuries past have decaying leather covers.

[kitteh]

My oldest CD-Rs are from 1994 and they are still readable. But as you say, you can't rely on media longevity. Regular copying is key.

I have data from the 1960s. Try copying a large set of data that started out on mag tapes. This is a solution that is guaranteed to fail eventually, one slip up and you're cooked.

Not the mention the cardboard boxes of family videos on beta and vhs tapes taking up space in my closet because I don't want to devote weeks to copying them to another format and the cost of doing this commercially is very large.

[The Wizard]

We used to use 9-track magnetic tapes on PDP-11 and VAX minicomputers back in the 1980s. Each reel was about a foot in diameter, held 3000 feet of tape, and a WHOPPING 44 megabytes of data at 6250 bpi, IIRC.
But so what? Essentially ALL of the data we ever recorded was of only temporary value. After a year at most, all that data was obsolete in our engineering lab.
Now something like Vital Statistics records are a different story...

[Valuethinker]

Um,... There's always paper (gasp!)

I don't know how bad a problem it is now, but every looked at a 50 year old paperback? (there are many in my parents' basement). Crumbling.

Most paper in the 20th century was high acid. It won't last a century. Whereas documents from the 1500s, 1600s, 1700s are still around, most paper documents from the 20th centurey will be dust in 100 years.

As you note, acid free paper is the solution to this, and is widely used in legal documents, for example. While current paperbacks do not last long, hardcover books I have that are fifty years old or older are in fine shape. Meanwhile books from centuries past have decaying leather covers.

That's the solution, but we have millions of books that are not on low acid paper. We didn't realize the problem and its scale until relatively recently.

A leather bound book that has lasted hundreds of years is a better bet than losing something after less than 100. Even hardcovers the pages yellow.

Another threat is simply that storage costs. A lot of paper books and records are being thrown out by archives and libraries. *hopefully* it is being digitized and *hopefully* that will remain secure.

[Valuethinker]

We should also be aware that any kind of high atmospheric level atomic detonation, or a specifically designed Electro Magnetic Pulse (EMP) weapon, could wipe out much of the data in a wide area below it.

So if there was some sort of 'high altitude war' that led to say 10-20 atomic detonations in the stratosphere, a *lot* of data would be corrupted, forever....

Better get one of these to hold your (properly hashed) backups... http://en.wikipedia.org/wiki/Faraday_cage

Will that block ElectroMagnetic Pulse (EMP)?

US bombers (B52s) continued to have vaccuum tubes for some components (which they sourced from the last manufacturer: the USSR! ) down to the end of the Cold War because vaccuum tubes have inherent resistance against EMP.

[nisiprius]

Data points from my personal experience. Executive summary: yes, analog is more durable than digital.

ANALOG:

a) Photos. I have many, many old family photographs on paper in albums, stored under ordinary household conditions in the ordinary photo albums of the day. Some are a century old. Many are fifty years old.

EVERY black-and-white photo is "readable" and enjoyable. That includes 1950s Polaroid photos that didn't have perfect Print-Coater applications, and some I processed at home when I was ten years old and did a lousy job of rinsing.

MOST of the color photos show enough fading to be clearly noticeable, but nevertheless "read" as "a faded color image" and carry the information and emotional content. Rather to my disappointment, fading is very clearly noticeable on Kodachrome slides kept in metal boxes in the dark, so don't give me any of that "it depends on the color process" guff. Kodak knew exactly what they were doing when they put the disclaimer about the images being "dyes, and like other dyes may fade" warning on the box. All color photos are a LOT LESS durable than silver-based black and white. It's relatively few, but I have prints from the 1940s and 1950s that have turned nothing but shades of magenta; a scanner cannot restore color even with a "restore color" setting, but a perfectly satisfactory black-and-white image is recoverable.

b) Books. Let me state this as simply as possible. I must have well over a hundred hardbound books printed before 1940, and every single one of them is in sound, readable condition, including a few from the mid-1800s. Some hardbounds printed with wartime paper restrictions are yellowed and the pages are dangerously brittle, and paperbacks that are more than about 35 years old are readable one last time--but each page detaches as it's turned.

DIGITAL:

a) Around 1998-2000 I undertook a project to "preserve" all my LPs by copying them to analog CDs. I bought good-quality name-brand CD-R's like Verbatim. I prepared neat labels for each using a labeling kit specifically made for the purpose, with a little press and labels specifically intended for putting on CDs. A few years later I read an article that said that labels that are specifically intended for use on CDs are not safe for CDs, and I panicked (I had since discarded the original LPs) and I bought a bunch of Mitsui Gold $1 each CD-Rs and made backup copies (which I labeled only with a CD marking pen). No problems in making the backups. I put the backups away. Around 2005 or so I decided to rip all my audio CDs into iTunes, and keep the CDs but rely mostly on hard drives for my music storage. Since the backups were put away, and also because I was curious to assess the state of the labeled CDs. A good 10% of the labeled CDs were unreadable, just 5-7 years after being made. I was able to read the backups. I have not had the courage to see how the backups are doing now.

b1) I lost some of my financial history on 400K Mac diskettes, not because the diskettes became unreadable, or because I didn't have a diskette drive--but because Apple dropped the capability of reading them from one OS revision to another, without warning. Sure, have been recoverable with sufficient effort.

b2) I lost some of my financial history in the form of Multiplan spreadsheet files, through pure software rot. (NOT bit rot). Microsoft Excel "always" had the ability to open Multiplan files, through three or four successive major revisions, and then it was dropped without any conspicuous warning. Again, sure, recoverable if I wanted to make the effort.

b3) I lost some of my AppleWorks documents, because although "Pages can open AppleWorks documents"--it can't. I haven't traced the exact history, but it is my belief that the same version of Pages running under OS X 10.8 will not open the same AppleWorks documents that it was able to open under OS X 10.7, that is this was actually caused an OS change, not an application change.

Those who are inclined to blame the victim will point out that all of these are surmountable with enough time and energy. But really, think about it. In order to avoid this kind of software rot, it is necessary to have in place a continuous evaluation program that takes a continuous inventory of every digital document you have, and continuously upconvert them to new formats. A task rendered difficult by the fact that many pieces of software will upconvert individual documents when you open them, but do not have any batch conversion capability.

I'm not even talking about my graduate research records on 7-track NRZI 1/2" tape...

[patrick]

Those who are inclined to blame the victim will point out that all of these are surmountable with enough time and energy. But really, think about it. In order to avoid this kind of software rot, it is necessary to have in place a continuous evaluation program that takes a continuous inventory of every digital document you have, and continuously upconvert them to new formats. A task rendered difficult by the fact that many pieces of software will upconvert individual documents when you open them, but do not have any batch conversion capability.

That's sound pretty hard to do, but I haven't done that and I still haven't lost access to any of my old documents from formats becoming obsolete (I can read my documents going back to 1991 -- I had only gotten my first computer shortly before that and my very oldest files are gone because I didn't keep any of the disks). Continuously converting is not only troublesome but also actually cause data loss -- the conversion may not preserve everything. It's much easier to just keep a copy of the old document viewing program, and copy it along with the documents every few years to a new hard disk / DVD / flash drive / whatever the fad is. At most you'd only need an inventory of every format used (so you could make sure to have at least one viewer for each). Copying a bunch of data is not that hard to do in a batch. Also, choosing the most widely used formats over niche format greatly reduces the risk of them becoming obsolete to being with.

I'm not even talking about my graduate research records on 7-track NRZI 1/2" tape...

Analog videotapes from the same era would not be much easier to view. At least the data on the digital tape would be fine if it had been copied.

[Phineas J. Whoopee]

See for example:
http://researchdata.wisc.edu/manage-your-data/data-formats/

Beyond interfering with the maintenance of our own personal data, bit rot will become an increasingly serious problem for the preservation of human culture.

In the past archival records were stored on paper. Paper deteriorates, but some kinds are worse than others. Our technology for physical preservation continues to develop, as does our technology for recovering information from seemingly irretrievable physical damage.

In performing historical research from times more than a couple of hundred years ago, a major problem is nobody wrote records about anything the upper classes, the only literate ones, weren't interested in. Thus we have detailed battle information from, say, the 100 Years War, but little to nothing about how the countries carried out the logistics to keep their armies fed.

Hundreds of years from now the complaint is likely to be they have nothing unless a continuous chain of custodians chose to copy information to new formats. The problem won't be lack of contemporaneous record keeping, but lack of continuous record preservation.

PJW