Wednesday, July 20, 2011

Blog Moveage

I've set up a new blog here: http://blog.daviddufty.com/. I'll mainly be using it for blogging about my book, How to build an android.

I've also got a science/philosophy blog here: www.empiricist.com .It was fun blogging here but I haven't done it for a couple of years now. My focus is on writing larger, more substantive pieces. Specifically, books. I may one day take up blogging again on issues about science and philosophy, and if so, it may be back here at time-etc.com.

But right now I have a lot of time commitments, and blogging is an indulgence I can't afford, especially for something as whimsical and unfocused as this site. Anyway, head over to my new blog at David F Dufty, and if you want to read more in-depth analysis by me, well... you'll just have to buy my book!

Tuesday, May 17, 2011

Lost in Transit


Coming soon to Australia and New Zealand:
"LOST IN TRANSIT: The Strange Story of the Philip K Dick Android" by David Dufty.
You can already pre-order online if you are so inclined.

LOST IN TRANSIT is the strange but true story of the Philip K Dick android that was built in 2005 by Hanson Robotics and the University of Memphis and was exhibited at Chicago NextFest. It was lost on a flight to San Francisco in December 2005 and has never been seen since.

By the way, I might be setting up a proper standalone blog soon, a non-blogspot one with my name and everything. I've set up a twitter account but that will no doubt not be the end of the story.

Monday, April 21, 2008

we are alone

From Discovery News:

Given the amount of time it has taken for human beings to evolve on Earth and the fact that the planet will no longer be habitable in a billion years or so when the sun brightens, Andrew Watson, with the United Kingdom's University of East Anglia in Norwich, says we are probably alone.

Earthlings overcame horrendous odds -- Watson pegs it at less than 0.01 percent over 4 billion years -- to achieve life. The harsh reality is that we don't have much time left.


This is basically modern support for the so-called "rare earth" theory, that says that intelligent life is an improbable abberation. If it's true, then SETI is a waste of time.

The problem is establishing the probability of an event after the event has actually occurred. While this is a knotty problem for statisticians, human brains are wired to perform such calculations quickly (but wrongly).
"We're here." goes the rule-of-thumb reasoning. "So how unlikely could it be for us to have evolved? Not very!"
However the fallacy of the reasoning can be seen if you look at the point of view of someone who has won the lottery. "It happened to me, so how unlikely can it be?"
Pretty unlikely.
The same might be true for the existence of intelligent life. We're cosmic lottery winners.

Saturday, March 29, 2008

multiple sclerosis

My father has multiple sclerosis (MS), and it has been slowly degenerating over the last couple of years. He's in hospital right now because it got to the point that he wasn't sufficiently mobile to stay at home. What happens from here is anybody's guess.
My mother has started a blog about multiple sclerosis called Multiple Sclerosis Carer (www.mscarer.com). It's a combination of insights into MS, dealing with someone with MS, and her own life.
There is no cure for this disease. Experts tell us that a cure is not far away, so we remain hopeful.

By the way, I have been neglectful of this blog of late. There are several reasons for this. First, I don't have internet at at home at my current residence. That really blows a hole in daily blogging. Second, my energies have been going into other things such as academic articles and a book.

I may resurrect this soon, particularly as I am in the late stages of finishing my largest writing project ever. I will also be soon getting internet on at home again!

Saturday, January 19, 2008

The edge annual question: the condensed version

Every year, The Edge (www.edge.org) asks intellectual luminaries one question. The answers are usually a mixed bag. Some are throwaways, some are boring, some are entertaining, some are insightful, and some are just mindblowing. The 2008 question is “What have you changed your mind about?”

The Edge site has one-line headings for many of the responses, but there is no summary of all the responses. I wrote one, in part to have a record for myself, and in part to keep track of what I had read and what I hadn’t read. I’ve published it here in case anyone else finds it useful.

I’ve resisted the urge to editorialize the answers in all but a few cases.

MARTIN SELIGMAN, a one-time supporter of SETI, is now certain that we are alone. There are no aliens, Martin now believes, here or anywhere else.

JOSEPH LEDOUX changed his view of memory. It turns out that memories are used exactly once. On subsequent occasions we recall the ‘memory,’ we’re actually recalling the previous time that we remembered.

KARL SABBAGH realized that experts are no better than non-experts. Sabbagh suggests that we shouldn’t respect the opinions of others any more than our own. (so should we ignore this advice?)

DOUGLAS RUSHKOFF is bitter and disillusioned about the internet not ushering in a new era of anarchic freedom and ideals.

PIET HUT has figured out that explanations and metaphors can distort or even strangle the subtleties of a complex concept. When you try to explain something, you may be in fact communicating something entirely different.

HOWARD GARDNER looks back on his hero Piaget, and realizes Piaget was wrong about a great many things.

DONALD HOFFMAN – we don’t perceive what is “true” or “out there”, we perceive what is “useful”. (in the sense - I think- that cockroaches don’t know much, but survive okay).

MICHAEL SHERMER no longer believes in the blank slate.

JAMES O'DONNELL, thinks the Romans were over-rated.

COLIN TUDGE sees the limits of science. Colin’s insights are insightful, but the specific targets he chooses to demonstrate science’s failings are patsies: namely behaviorism and genetically modified organisms. Behaviorism is a much-flogged dead horse, and the GMO scare may turn out to be a false alarm.

IRENE PEPPERBERG thinks falsification is over-rated. Irene despairs about mediocre science: “Many journal submissions lack any discussion of alternative competing hypotheses: Researchers don't seem to realize that collecting data that are consistent with their original hypothesis doesn't mean that it is unconditionally true.” (Do you truly think scientists are that ignorant, Irene?)

MARCELO GLEISER rejects the quasi-religious search for unification in physics.

FREEMAN DYSON does not believe that the atomic bombs dropped on Hiroshima and Nagasaki made any contribution to ending World War Two. It was effectively already over.

ED REGIS doesn’t believe forecasters, predictors and prognosticators. Some systems are computationally irreducible, says Ed. If that’s right, that means there are some places that science just can’t go. But should we give up trying?

DAVID BRIN sees America sliding into “bona fide future shock.”

RUDY RUCKER does not draw a distinction between humans and machines any more.

CHARLES SEIFE realized that science is not a cornerstone of democracy. Maybe not even a building block.

DAVID BODANIS finds the bible much more interesting now that he’s an adult.

HAIM HARARI thinks that definitions get in the way.

TIMOTHY TAYLOR says that all this relativism business has gone too far.

LEON LEDERMAN thinks scientists should be more politically active, so they can solve global warming, nuclear arms etc.

DAN SPERBER thinks evolutionary psychology rocks!

THOMAS METZINGER became an ethical skeptic.

MARC D. HAUSER ponders the limits of Darwin in explaining anything and everything.

KEITH DEVLIN rejects the Platonic notion of mathematics. Instead, says Keith, mathematics is a synthesis of objective reality and human perception… and imagination.

DAVID G. MYERS – has also lost his taste for the blank slate. He also thinks ECG is good, that some people are born gay, “personality is unrelated to birth order”, and “economic growth has not improved our morale”. On this last one, I hope he’s reading the list because this one gets busted later, Dave.

DANIEL EVERETT thinks Chomsky was wrong (about grammar)

DAVID DALRYMPLE is excited about distributed computing.

MAX TEGMARK sees “consciousness” as a separate field of inquiry from physics.

Defying childhood teachings, ROBERT SAPOLSKY now believes that the brain grows new neurons throughout life.

TOR NØRRETRANDERS points out that the mind is software, and the body is software too. Nice.

HELEN FISHER discovered that the seven year itch is a myth, but the four year itch is real and maybe even a biological imperative.

STEVE NADIS says that scientists are less objective than he previously thought.

PAUL STEINHARDT believes the inflationary theory of cosmology is wrong on many levels.

RODNEY A. BROOKS no longer believes in “computation" as the defining concept in understanding either humans or non-humans.

ROBERT TRIVERS gives a rambling, yet passionate and rich, account of his investigations of self-deception. One of his discoveries is that psychology is not crap after all. There's hope for you yet, Bob.

LAURENCE C. SMITH has changed his mind about global warming. He now thinks it’s even more urgent than he previously thought (he previously thought it was pretty urgent).

LEE M. SILVER is fatalistic about the possibility of a science-literate educated elite. Irrationality prevails, it seems, even in educated people. A rational humanist populace that accepts even basic scientific facts will never appear.

GARY MARCUS thinks Chomsky was wrong.

LEE SMOLIN believes in time as a fundamental and important property of reality, in contrast to his quantum buddies, who see it as just an emergent property of the wavefunction of the universe. He also believes that physical laws evolve.

A. GARRETT LISI cannot answer the question, because nobody can really change their mind. Even when our mind seemingly changes, he says, the new knowledge does not replace the old, it is actually just stacked on top of the old knowledge. (Hmm, that's not what my Intro to Psychology textbook said...)

JOHN BAEZ is disillusioned with quantum physics, in particular the search for quantum gravity. He blames Lee Smolin (another contributor), among others for igniting the “quantum-gravity debates” that eventually drove him away.

KEN FORD is saddened to realise that scientists are “only human”: there are just as many cheats, crooks, and liars among the ranks of scientists as the rest of the population.

JEFFREY EPSTEIN cannot answer the question, because the word “you” in “what have you changed your mind about” falsely assumes some kind of unitary identity separate from the rest of the world.

GARY KLEIN says that changing your mind often means changing the way you perceive the situation. In fact, this change can be so profound that the expression might be better said as exchanging your old mind for an entirely new one:

ALAN KRUEGER gets the award for the briefest entry: a single sentence. He cryptically says: “I used to think the labor market was very competitive, but now I think it is better characterized by monopsony, at least in the short run.” Being as I am in the labor market right now, I have no idea why he would say that. Perhaps next year we will get the second sentence in Alan’s train of thought.

SETH LLOYD has shifted from being a technophobe to a tentative technophile.

JOHN MCCARTHY has found that people use facts to support their pre-existing beliefs, rather than to challenge them. (I guess this is what psychologists call the confirmatory bias).

ERNST PÖPPEL is no longer a fan of Wittgenstein. In fact, Wittgenstein was a "straightjacket” from which he had to escape.

SCOTT SAMPSON now accepts that an asteroid killed the dinosoars (apparently this is still being debated in the paleontological research community)

PETER SCHWARTZ is a convert to nuclear power, in part to offset the dangers of global warming.

STEPHEN KOSSLYN thinks that independent levels of analysis (such as cognition and brain function) are ineffective.

MARCEL KINSBOURNE is a mirror neuron skeptic, but does see them as an important indicator to the interconnectedness between perception and memory.

KEVIN KELLY sees a new communitarian movement online, powered by Wikipedia.

MARTI HEARST had a conversion to “grandmother cells” when she learned about Wernicke’s aphasia. Yet she’s also a fan of co-occurrence, one of the blindest, not-knowingest approaches there is.

ALAN KAY learned at age 10 that vacuums don’t actually “suck.”

From her views as a 60’s feminist, DIANE F. HALPERN now takes the biological contribution to individual difference, especially sex differences, very seriously.

STEPHEN H. SCHNEIDER is concerned about global
warming
, while in the 70’s he was concerned about global cooling.

XENI JARDIN now sees that online communities require active management.

CARLO ROVELLI was bored with quantum mechanics, until he realized that at a fundamental level, it is self-contradictory. Now he is re-energised as he works to save quantum mechanics from itself.

ROGER C. SCHANK once thought that AI was just around the corner. Now he thinks that AI will never arrive, at least not in the way he originally imagined.

JOHN HORGAN is no longer a mind-body mysterian. In fact, he thinks the brain will be completely decoded.

SHERRY TURKLE is against human-robot mixed marriage.

DANIEL GILBERT says that being able to change your mind isn’t as great as you think it is. And he’s got the data to prove it. (his studies show that irrevocable decisions cause more happiness than revocable ones).

STEWART BRAND says “good old stuff sucks.” The idea of an old house or antique furniture might seem quaint, but the truth is that a new house with new furniture will look nicer and be more comfortable.

OLIVER MORTON says, Let’s give up on human spaceflight. Our current efforts have been disappointing, there is no clear benefit, and maybe in a hundred years or so, it will be easier and more successful. “Leaving a technology fallow for a few decades and coming back with new people, tools and mindsets is not such a bad idea.”

JUDITH RICH HARRIS says that generalisation, a mainstay of Psych 101, is – in her polite words- “the exception, not the rule.” In fact, babies have a bias against generalizing.

GEORGE CHURCH worries about taking scientific claims “on faith”.

TERRENCE SEJNOWSKI says neurons can do a lot more than he ever imagined.

JONATHAN HAIDT once had contempt for fraternities and sports. He now sees them as important and healthy cultural phenomena.

PATRICK BATESON had dinner with a creatonist and shifted from agnosticism to atheism.

ALAN ALDA has changed his mind twice about the existence of God.

STEVEN PINKER is swayed by new evidence that humans are still evolving, and may even be evolving rapidly.

PAUL DAVIES now thinks that the laws of the universe are changeable.

GEORGE B. DYSON views the Russian occupation of American territories to have been better than it is usually portrayed.

JUAN ENRIQUEZ recognises that applied science is the key ingredient to national survival in today’s world.

REBECCA GOLDSTEIN is no longer a believer in falsifiability.

EDUARDO PUNSET is impressed at new scientific insights that show how the brain links past and future.

JOHN ALLEN PAULOS, due to a severe case of cerebral stenosis, is unable to answer the question.

LEO CHALUPA’s enthusiasm for brain plasticity has been tempered by the evidence against it.

SCOTT ATRAN has found that group membership – “fictive kinship”- is a more powerful driver than religion or politics, even for terrorists.

MARCO IACOBONI- thought science would slowly eradicate irrational thinking in the public, but this is far from inevitable.

RICHARD WRANGHAM figured out that humans evolved to eat neither raw meat nor raw vegetables, but cooked food.

SEAN CARROLL no longer enjoys being an anti-establishment rebel. Leave it to the youth!

LINDA STONE has discovered the importance of breathing.

STANISLAS DEHEANE is excited about Friston’s law, neuroscience’s answer to superstring theory. Its tenuous relationship to data does not bother him at all.

MARY CATHERINE BATESON doesn’t have an answer because she thinks we don’t change our mind about facts, we simply “rearrange” them.

WILLIAM CALVIN used to think global warming was a serious problem. Now he thinks it’s a really serious problem.

CAROLYN PORCO fears we are entering a time not unlike the Dark Ages, “when the practitioners of science were discredited, imprisoned, and even murdered.” She may be confusing scientists with witches.

BRIAN GOODWIN has converted to a new religion, which he calls “pan-sentience.

LISA RANDALL finds the ‘solar neutrino puzzle’ much more interesting these days.

NICHOLAS CARR fears the internet and its sinister consolidation of power.

AUBREY de GREY thinks scientists should be less curious. He also sees scientists and technologists as less similar than he once believed.

HELENA CRONIN accepts the “larger variance” argument for why there are more male Nobel laureates.

DANIEL C. DENNETT realized that he wasn’t reductionist enough.

NICHOLAS A. CHRISTAKIS thinks culture can influence human evolution.

RUPERT SHELDRAKE is skeptical about skeptics. Quote from Rupert: “The more militant the skeptic, the stronger the belief.” (in the opposite point of view).

CHRIS ANDERSON has shifted from climate skeptic to carbon zealot.

FRANK WILCZEK replaced religion with science.

TIM O'REILLY thought online social networks would never take off.

JAMES GEARY thinks neuroeconomics rocks. (neuroeconomics is just a fancy word for ‘the psychology of economics’)

DANIEL GOLEMAN found two Tibetan monks with some really weird brain patterns.

ANDRIAN KREYE’s intuitions about the ascendancy of secularism were overturned by scientific evidence about the thousands of new religions that bloomed in the twentieth century. Andrean has a newfound respect for the persistence and cultural power of religious faith.

DAVID BUSS still doesn’t understand women.

YOSSI VARDI is more skeptical of scientific modeling these days.

SAM HARRIS does not trust mother nature: “There may be current threats to civilization that we cannot even perceive, much less resolve, at our current level of intelligence.” The moral:
we should be prepared to try anything to keep on surviving and prospering, including tampering with the genome.

ROBERT SHAPIRO has learned the hard way that good ideas can be killed by simply being ignored. Or as Oscar Wilde said, “the only thing worse than being talked about is not being talked about.”

HANS ULRICH OBRIST thinks that clinging to the past by clinging to buildings and objects may be futile. This leads him to wonder about the value of museums.

BRIAN ENO got sucked in by Maoism, until he underwent “a long process of re-evaluation.” He continues “I had to accept that I was susceptible to propaganda, and that propaganda comes from all sides — not just the one I happen to dislike.” A humble and powerful admission.

SEIRIAN SUMNER has downgraded the importance of kin selection theory in social evolution.

PAUL EWALD doesn’t trust expert opinions, especially when they talk about topics that they are not actually experts on. Well, there goes the Edge annual survey.

NICHOLAS HUMPHREY has salvaged the “Cartesian theater of consciousness” from the intellectual junkyard and has given it a new polish.

ADAM BLY thinks that technology is important to science.

PZ MYERS can’t answer the question because “there is a substantive context in which we do not change our minds.”

Widespread ignorance about basic statistics makes GERD GIGERENZER pessimistic about the advent of a new age of health literacy.

The advent of quantum information has helped ANTON ZEILINGER not be ashamed of being a “useless” quantum physicist any more.

ESTHER DYSON isn’t worried about online privacy any more.

MARTIN REES says we should be planning for the post-human era.

JANNA LEVIN thinks the universe is finite.

JARON LANIER underestimated the healing power of virtual reality.

DIMITAR SASSELOV could not conceive of a planet having an orbital period as short as 4.2 days.

A book called “What is really science” opened FRANCESCO DE PRETIS to the idea that science has a strong social component.

ROGER HIGHFIELD is disillusioned with science. Contrary to popular myth, facts rarely topple theories, even when they should. More often, scientists squeeze theories into facts.

DANIEL ENGBER says ethics committees, by their very existence, let scientists off the hook from thinking about the ethics of their own research. They encourage an attitude of thinking that ethics means ‘getting it past the ethics committee.’

AUSTIN DACEY sees the concept of “I” as an illusion, but values and reasons are real, biologically driven phenomena.

SIMON BARON-COHEN is an equality skeptic.

DAVID SLOAN WILSON is a latecomer to complexity.

J. CRAIG VENTER is very worried about global warming and overpopulation.

NEIL GERSHENFELD wonders about the true relationship between computing and physics.

ALISON GOPNIK sees the ubiquity of imagination as a clue to fundamental questions about what it means to be human. Specifically, our talent
is not for perceiving reality, but creating and molding reality.

JORDAN POLLACK sees email (non) privacy as a threat that the public does not yet understand.

PAUL SAFFO anticipates the day when “prediction engines” are better forecasters than humans, thus putting him out of a job.

CHRIS DIBONA found that programmers are more productive the less they are supervised.

BEATRICE GOLOMB says the field of medicine chases after fads and ineffective remedies (such as HRT) because of widespread ignorance among doctors about research methods, statistics, and inferential reasoning.

STEPHON ALEXANDER thinks the problems with the theory of cosmic inflation will be solved by somehow incorporating quantum non-locality.

GEORGE JOHNSON got bored with modern physics, so he amuses himself by replicating great experiments of the past.

GEOFFREY MILLER says that psychologists who want to study fear should talk to lion tamers.

STEVE CONNOR thinks that global warming will destroy the planet. Steve says ominously, “the four horsemen of the apocalypse will ride again.”

BARRY SMITH sees neuroscience as the key to understanding consciousness. The normally functioning mind is a marvel that astounds.

JESSE BERING: “I Have No Destiny (and Neither Do You)” This is the one that stayed with me the most.

ROGER BINGHAM has lost his faith in the Church of Evolutionary
Psychology
.

RICHARD DAWKINS admits that Zahavi’s theory of prestige is correct, after publicly trashing it years ago in a book. However Dawkins neutralizes the appearance of humility when he complains: “Although I was wrong in my scepticism, and I have now changed my mind, I was still right to have been sceptical in the first place!” Note to Richard: you weren’t “wrong in your skepticism” as you put it, you were wrong. The verbal contortions are unseemly.

GREGORY BENFORD wonders if we can have a “law of laws,” a theory of how the laws of physics might have come about.

LERA BORODITSKY has learned through studies that language can shape low-level perception.

JAMSHED BHARUCHA once thought that one of the goals of education was to “Settle eventually on a framework or set of frameworks that organize what you know and believe and that guide your life as an individual and a leader.” Jamshed now sees this as clutching for certainty, which is seductive, but best avoided.

DENIS DUTTON has realized that human sexual selection allows intentionality into evolution “through a side door.” Says Dennis, “Though it is directed toward other human beings, it is as purposive as the domestication of those wolf descendents that became familiar household pets.” Dutton’s thoughts are a must read for devotees of Evolutionary Psychology, especially advocates of Caveman Psychology.

CLAY SHIRKY says that science and religion are incompatible.

KAI KRAUSE says that software is ephemeral. Its lifespan is so fleeting that it should be referred to as “performance art”.

LINDA S. GOTTFREDSON is intrigued by the cumulative power of small but consistent effects, in for example, human evolution.

RANDOLPH M. NESSE says that “Universities may be best show in town for truth pursuers, but most stifle innovation and constructive engagement of real controversies, not just sometimes, but most of the time, systematically.” And we need to understand “how hard it is for even the smartest experts to offer objective conclusions”.

BART KOSKO prefers the sample median over the sample mean, because means are not robust (especially with Cauchy distributions).

Computer users are far more adaptable and willing to embrace the new than DAVID GELERNTER once gave them credit for.

W.DANIEL HILLIS says that hot water freezes faster than cold water.

NASSIM TALEB has lost faith in the concept of “probability”.

DANIEL KAHNEMAN learned from scientific studies that money does buy happiness.
And probably love, too.

Friday, August 31, 2007

Data decay: even computers forget

In 1971, a young student at the University of Illinois typed the Declaration of Independence and saved it on a Xerox mainframe computer. The student was Michael Hart, and the electronic document was the beginning of Project Gutenberg.

Project Gutenberg’s aim is to create, store, and distribute electronic copies of books that are open domain: that is, works that are out of copyright, or for some other reason do not have copyright attached. From Project Gutenberg or any one of its many affiliate sites, you can download Shakespeare, the Bible, Franz Kafka, or the works of Plato. Works that might not have survived the ravages of time in real libraries can live on in Hart's virtual library.

The project takes its name from the Gutenberg press, developed by Johannes Gutenberg around 1450. Gutenberg's printing machine used moveable type, an innovation that allowed the mass production of printed material, such as bibles, pamphlets, and books.

Thanks to digital media, knowledge is more accessible than ever. With just a few fragments, I was able to find a nursery rhyme from my childhood. The fragment was:
“Hush little baby, don’t say a word;
Momma’s gonna buy you a mockingbird”

I found regional variants and learned that the version I knew was a rare version of the song.
Nothing is forgotten now. No piece of information is out of reach.
At least, that’s how it seems.

But is digital storage really eternal? Will generations a thousand years from now be reading Kafka online, or finding Mockingbird with a Google search (if Google even exists)?

Physical storage ages and decays

Digital storage, on CD or DVD or thumb drive or hard drive, is subject to the same force of nature as old-fashioned storage such as paper: aging, decay, and disintegration. Data is stored on hard drives magnetically: Tiny alignments of magnetic material on the surface of the drive can be switched one way or another, making long strips of magnetic material that are used to store zeros and ones. These zeros and ones, these bits of “binary data,” are the building blocks of all digital storage.
It works like this: in eight bit storage, eight zeros means “Zero”. Seven zeros and a one means “one.” Six zeros, a one, then a zero, means “two.” Etc. You can build it up from there to sophisticated programs, renditions of the Mona Lisa, an0d copies of Franz Kafka’s Metamorphosis.
The problem with magnetic storage is that the earth itself has a huge magnetic field (this is why compasses can find the north pole: they align themselves to the Earth’s magnetic field). Over time, that weak, but ever-present field gently tugs at the magnetic strips, coaxing them all into aligning in the same direction. It always wins in the end, erasing all data. This is why it is always risky passing laptops through x-ray machines at airports. The x-ray machines need to be specially adjusted for such devices. Even then, some degradation occurs.
It is also is why old video cassette tapes and audio music tapes are unreadable: they also rely on magnetic storage. The North Pole and the South Pole wipe them all clean.

Time also erases optical media

But you don’t have to keep all your information on such an unreliable medium. More and more information is now stored with optical media: the CD ROM and the DVD. Optical media uses lasers to read markings in a disc, and are immune to the ravages of the Earth and other magnetic fields. Unfortunately, they don’t last forever either.
Optical media are subject to bleeding ink and degradation due to temperature and decay. Jerome Hartke of Media Science conducted an experiment on the longevity of CDs. Instead of just waiting around for time to erase all, he sped up the aging process by keeping the CDs in storage for 200 hours at a temperature of 85 degrees Celsius (185 degrees Fahrenheit) and 85 percent humidity.
Hartke found considerable damage to the CDs, but it varied from disc to disc. By one measure, half of all discs with recorded information had at least some defect. Unused discs fared worse: over time, the ability of many of these discs to be modified to accept new data was lost.
Hartke noted that this kind of deterioration was not the only threat to CD information, and possibly not even the most important. A major cause of CD deterioration, he said, was “overconfidence in the robust construction and error correction of CD-R media.”
People think that CDs are more reliable, and more durable, than they actually are. And that belief in the toughness and durability of CDs is a bigger risk to CD health than any of the ravages of time.
Why do people have such faith in the CD? In part, it is because the machines that read them have such excellent error-correction technology, protecting us from seeing their imperfections. All discs generate thousands of read errors, due to defects, noise, and high data densities. These errors are caught and corrected using a standard called the Cross-Interleave Reed Solomon Code (CIRC), creating the illusion - for us - of pure, error-free storage. But over time errors accumulate, and cracks appear. The errors eventually reach a critical mass that is no longer readable. This is the fate of every CD.
There are now longer-lived storage products such as the Plasmon UDO (ultra density optical) storage, that lasts for about 50 years. That’s long by digital standards, but is a twinkle in the eye historically. The Rosetta Stone, ancient writings carved into clay, survived several thousand years.

Hardware obsolescence

Even if information survives for centuries, will it be any use?
The fast rate of change of technology has created a whole new problem: obsolete devices. When hardware standards change, the storage items that were used with them become obsolete. When CD music became available, hundreds of millions of vinyl records became obsolete. This didn’t happen overnight: people still had record players to play them, and even now there is a niche market for vinyl playing equipment. But over time, more and more vinyl record players break, or are thrown out. And if the niche market for the players disappears, finding the equipment will become more difficult.
Elvis and the Beatles have successfully made the transition to CD and again to MPEG and iPOD, but what about the Goddards? Jigsaw? Chicken Shack? The Ozark Mountain Daredevils? How much music was ultimately lost forever in the transition?

Vinyl records are a “contact medium:” the needle has to actually come into contact with the record. The needle follows a groove that spirals inward, and tiny bumps in the groove make the needle vibrate at different frequencies. Every time the needle follows that track, the needle gets worn down a little, and so does the record.
A vinyl record can’t be played an infinite number of times. There are recordings all over the world that have not been converted to digital format. Eventually, it will be too late.
At least vinyl records can be played, even without a specialized vinyl record player: it is possible to rig a makeshift player with a pin or regular needle (as the playing needle) and a roll of paper (as the amplifier). By holding the needle to the record and spinning it, a semblance of the original recording can be heard. This is not the case for floppy disks.
In the early days of computing, data was almost universally stored on floppy disks (although some computers, such as my ZX81, used music cassettes). You inserted them into the floppy disk drive, wrote your files to the floppy disk, and took it out. I had piles of them next to my computer.
At first, they were large and were actually floppy: they were soft and flexible. Over time as technology improved, they became smaller and smaller. While 8 inch floppy disks were the standard in 1970, by 1980 smaller ones were available, only 5 ¼ inches in diameter, and by 1990 the standard size was 3 ½ inches across.
Each of these improvements in floppy disk technology meant that new floppy disk reading machines, “disk drives” needed to be built and distributed, and data that was on old formats needed to be transferred to new formats. During changeover periods, computers would often have two drives, one for a larger format and one for a smaller format. This lulled people into complacency about data stored on larger floppy formats. Eventually, they found themselves in a world where the older format was no longer supported, and retrieving the data was, if not impossible, problematic. Today, computers are sold with no floppy drives at all.

Software obsolescence

When it was first released, Visicalc was revolutionary. Designed by was the original spreadsheet. The idea was not patented, and before long, Lotus, Microsoft, and others created their own spreadsheets. Visicorp, the company that made Visicalc, no longer exists, so getting new copies or and update or product support for your existing copy is impossible. Luckily, the fame and widespread use of Visicalc means that modern computers can run a “Visicalc emulator,” a program that simulates Visicalc so perfectly that old files can still be used.
To get a glimpse of the effort needed to access information in old formats, take the case of Terry, who wanted to retrieve 800 poems that were originally written on an Olympia Carrera word processor. The word processor itself- a dedicated machine that only ran word processing software – is long gone. All that remained were the floppy disks, that were written in a format unique to Olympia Carrera. Terry needed to do the following to get the poems:
1) install a floppy drive to read the correct size floppys,
2) install a program called 22 disk;
3) install a program called ANADISK;
4) build a custom diskette specification using ANADISK.
The job was successful, but it shows how much trouble you can be in when the technology moves on.
Formats for other programs, especially custom-designed programs, may be impossible to decode. This is due to the complexity of digital files, which may need arbitrarily sophisticated algorithms to read.

A ticking time bomb

If the specifications for any file format are lost, then all information stored in that format will also be lost. Decrypting a pdf file, for instance, would be impossible without the specifications. Such specs typically run from hundreds to thousands of pages.
The problem of information being lost in old data formats has alarmed the British National Archives. The chief executive of the National Archives, Natalie Ceeney, describes the situation as a “ticking time bomb.”
They have over 580 terabytes of data in unsupported file formats, and have
teamed up with Microsoft to create a range of systems to retrieve the information using emulators. Even when the emulators are built, transferring that amount of data to a new format is a formidable challenge, and will require large scale automated tasks.
The irony of the National Archives teaming up with Microsoft is that companies with proprietary software (such as Microsoft) have made the problem worse. The specifications for proprietary software and file formats are not freely available, because they are the commercial property of the company that owns it. This hinders the ability of others to make compatible applications.
Microsoft often gets the blame for this situation, partly because file formats for old versions of Microsoft software are not always supported by new versions. However IBM, Novell and other companies have done the same thing.
Product support cycles (the time that a company is willing to help you keep the software working) are usually between five and ten years. After that, you’re on your own. In many cases, the software itself disappears, and all that is left are the files. Files that were once considered important enough to save or archive, now with no way of being retrieved.

Drowning in information

But what if all that information was lost anyway? It seems like we are swimming –sometimes drowning- in information today. Digital information may not be more durable, but it is easy to distribute and easy to copy. It is the new revolution.
Perhaps the Gutenberg Project, the British National Archive, archive.org , and institutions like them are the key to immortal knowledge.
That may be the case, but the vast amount of information around us blinds us to the fact that the information is still transient. It is as if we were given a giant newspaper rack to store all our daily newspapers. Rather than throwing them out, we could keep them all in a room at the back of our house. However, newspapers still decay at the same rate.

Still, it is true that it is now easier to keep information alive. Sure, the information has to stay on the run, moving from storage to storage and home to home. Deletion and permanent erasure stalks it, one step behind, waiting for a slip-up. But with diligence, it is possible to keep the data alive. Storage is replaceable and cheap.

Information half-life

Once, people wrote letters to each other. Now they write emails. Let’s say that letters have some kind of mortality function with a half-life of six months. It’s not unreasonable. Most letters are disposed of within a month or two of arrival. A few are treasured for decades. Emails, too, have a mortality function, but their half-life is a lot longer. Excluding spam, I would guess their half-life would be around two years, considering losses due to changing or deleting email accounts, changing computers, forgetting passwords, accidental erasure and clearing of old files. That means that, even though the data will disappear eventually, on average it lives longer.
Information lifetime doesn’t tell the whole story. Part of the increase in information lifetime is the storage of items that would have previously been instantly destroyed, or at least disposed of in the near future. Communication between co-workers and friends was once almost entirely verbal, and therefore transient. Now, as we transfer daily communication to digital media, we leave a trail with more and more of our daily interactions. Our idle morning chit-chat threatens to outlive us.
Before computers, a paperback novel would have a brief moment of glory. Thousands of copies sit in displays in thousands of bookshops, stacked one on the other. Thousands of hands, thousands of readers, thousands of bedroom shelves. One by one, the copies would meet their ultimate fate: they would be broken, or burned or thrown into the trash. They would be shredded by playing children, or dropped in a puddle. Some would be stored in boxes in attics and sheds, eventually being warped by the elements, and rendered unreadable.
Only the few survived: the outstanding works that entered a second printing, and a third. The books that had instant fame, or a cult following. For them, immortality beckoned, but for most, they are gone. The vast majority of books published in the nineteenth century, for example, have completely disappeared.
Project Gutenberg can now grant immortality not just to the chosen few, but to unlimited numbers of works. But is this a good thing? Can we really be the custodians of an ever-increasing store of mediocre information? Perhaps there is merit in letting things die- a sort of information natural selection. Perhaps our wisdom on wikipedia or Yahoo Answers, or in tens of millions of blogs, journals, and newspapers, is best forgotten for the most part.
So we have a curious situation: On the one hand, information is constantly decaying, and is threatened by changes in technology and media storage standards. On the other hand, the world is experiencing an explosion in the total amount of stored information, much of which is of questionable value. From this, two conclusions can be drawn:
1) we need to focus on what information we want to store. We need to decide how important each data stream is, and how long it needs to be stored.
2) For the information that we decide to keep, we need to be diligent about keeping it.

When I say “we”, I am referring to us as individuals, organizations, and as a society. For instance, as individuals, how many emails do we wish to keep? At your current rate, how much email data will you have in twenty years’ time, and how will you manage it? This is a question that is better asked sooner than later.
Plato’s work has survived for two and a half millennia. It is likely to survive just as long again, but if it is buried under a mountain of mediocrity, nobody might bother to read it.



Further reading for the interested:
Media Science: longevity of data
Problem of media obsolescence
Techworld take on the storage issue
A history of Visicalc by its creator
Webmasterworld discussion: digital data a ticking time bomb (the thread that got me thinking about and researching the issue, and eventually led to this article)