The end of history will come not when nothing more ever changes, but when nobody can work out what has actually happened. We may be closer to that point than we like to think.

In the world of organisations, historically, the creation of records was a by-product of actually doing the work. It wasn’t hard to remember to do the filing, because the file – and before that the ledger – was the unit of activity. Sorting out the files of a few years before to weed out the ephemeral was a bit more of a chore, but no more than that. And while there might be newer and shinier filing cabinets, and while metal was replaced by plastic at the ends of the treasury tags, fundamentally nothing had changed for decades. Historians of today are still in that world since they operate with the time lag of the thirty year rule, and it will be a few years yet before the reduction to twenty years makes much difference to that.

James Lapping has recently argued that in each of the last eight centuries there has been a standard way of keeping records, but that the explosion of systems and technologies around the beginning of this century means that those standards have vanished:

In the 20th century there was competition between companies supplying organisations with paper, filing stationery and filing cabinets. But none of those companies were trying to disrupt the way individuals communicated or stored information. No company was trying to come up with an alternative to the hard copy document, or an alternative to the practice of gathering hard copy documents together into files. They could not have been able to disrupt these practices even if they had wanted to.

In the 21st century there has been a constant stream of new formats and applications disrupting the way people record and communicate information. Documents are still important. But successive waves of new formats (e-mail, instant message, blogs, discussion boards, wikis, status updates) have come in alongside the document.

I doubt that there has ever been a time when people created records for the sake of it beyond any immediate purpose they may have had for them (though I say that with the insight of a practising bureaucrat, rather than with the knowledge of a historian). If that’s right, it’s probably a safe bet that our successors will be no more motivated to do so than we are. So if progressing the work continues to diverge from creating records of what has been done, the raw material of history may be thinner in future than it has been for centuries (and history here means medium term institutional memory as much as it does the work of historians). That problem will not be solved by exhortations to do better filing: it will be solved, if at all, by tools which support what people are trying to do in the short term while quietly adding what may be needed for the longer term – which is easier said than done, as James Lapping notes:

If you can’t predict what format or what applications people are going to use within your organisation then your only option is to have a system that will manage records produced in any format, in any application. […] However there are a great many unanswered questions about the model. Even if the functionality is there, no-one knows how organisations could make it work. There are no case studies out there that I have heard of, and no guidance notes from any national archive or professional society.

On the face of it, all that is a strange state of affairs. In the wider online world, the worry is that we will remember too much, not too little, with ever greater concern being expressed. As the Wall Street Journal recently reported an interview with Eric Schmidt, the CEO of Google:

“I don’t believe society understands what happens when everything is available, knowable and recorded by everyone all the time,” he says. He predicts, apparently seriously, that every young person one day will be entitled automatically to change his or her name on reaching adulthood in order to disown youthful hijinks stored on their friends’ social media sites.

Even more pertinently, the New York Times ran a long essay a few weeks ago called The Web Means the End of Forgetting:

All around the world, political leaders, scholars and citizens are searching for responses to the challenge of preserving control of our identities in a digital world that never forgets. Are the most promising solutions going to be technological? Legislative? Judicial? Ethical? A result of shifting social norms and cultural expectations? Or some mix of the above? Alex Türk, the French data-protection commissioner, has called for a “constitutional right to oblivion” that would allow citizens to maintain a greater degree of anonymity online and in public places. In Argentina, the writers Alejandro Tortolini and Enrique Quagliano have started a campaign to “reinvent forgetting on the Internet,” exploring a range of political and technological ways of making data disappear.

Of course, that may be plain wrong, as Scott Rosenberg has trenchantly argued in a post on the NYT article

There is an entropic quality to everything that is shared online. Data gets lost; servers die; databases are corrupted; formats fall into disuse; storage media deteriorate; backups fail.The Web is now old enough for us to know just how badly links rot over time. Much of the material from the early days of the Web is already gone. Facebook and Twitter actually make it nearly impossible for you to find older material, even stuff that you’ve contributed yourself. The more dynamic the Web gets and the more stuff we move into “the cloud,” the less confident we can be that information that once was public will remain available to the public.

There are plenty of versions of this dilemma around once we start looking for them. Take the apparently simple case of how storing personal photographs differs between the twentieth century and the twenty first. Deane Barker reflected recently on being asked to select some photographs – why choose, he asks, when it is so easy to keep all of them. He concludes with the interesting thought:

So, image media has officially been reversed. The digital version used to be an annoying stepping stone to a physical version. Now, the physical version is almost a weird afterthought.

Almost simultaneously, and as far as I know with no connection, John Naughton was reporting and endorsing precisely the opposite argument:

Parr thinks that we should also print our pictures, and I suspect he’s right. “We are in danger”, he says,

of having a whole generation – and this will continue into the future – that has no family albums, because people just leave them on their computer, and then suddenly they will be deleted. You have to print them and put them in an album or a box, otherwise they could be lost. And write captions. You might think you are going to remember what is happening in a picture, but you probably won’t in 10 years’ time.

So where does all that leave our debt to posterity when, notoriously, posterity has done nothing for us?

The approach for the last ten years or so has been, in effect, to create an electronic version of the old paper filing system. The Modernising Government white paper stated in 1999 that:

It is our aim that by 2004 all newly created public records will be electronically stored and retrieved.

Some departments spent a lot of time and money on failing to reach that objective, others did little or nothing and also failed to meet it. What those systems demonstrated, in my very limited experience, was essentially that users are very intolerant of friction, of doing anything at all which got in the way of their immediate task. Confronted with systems which imposed the least degree of friction, they simply worked round the obstacle. I claim no expertise in this area, so it may be that the tools have now improved so dramatically that the friction has completely vanished, but whatever the degree of improvement, that still feels like a model locked in to an old way of defining the problem.

The old model assumes that archive capacity is a constrained resource. But what if it isn’t?

There are two aspects to that question: what we have the capacity to store and what we have the capability to find afterwards. Historically the two were closely linked: storage was physical and had to be highly organised in order to stand any chance of finding anything later. But on the face of it, both those constraints are now broken. We can store pretty much what we like, and even at eye-watering enterprise storage prices, I strongly suspect that it is frequently more expensive to go through the process of deciding not to store something than it is just to let it be stored (though it is even more certain that those costs fall on completely different budgets, that the second is visible and the first is not, and that as a result the trade off is not optimised).

As Scott Rosenberg put it in a piece I quoted from five years ago,

My in box is not a desk that must be cleared. It is a river from which I can always easily fish whatever needs my attention. Why try to push the river? Computer storage is cheaper than my time; archiving is easier than deleting.

Finding things again remains a challenge: Rosenberg’s argument about entropy and Lapping’s about the need to manage not just current formats but obsolete ones and those yet to be invented are both powerful ones. Even there though, the quality of search tools and the availability of the computing power needed to make them effective strongly supports the shift from the old approach to the new. It doesn’t matter how big the haystack is, if a search for ‘needle’ always returns the needle you are looking for.

History will, of course, look after itself. It always has. But the future history of our time will be different from our histories of past times, and that will not be because we have an eye to the future, but because we are always relentlessly focused on the present.

Header image by Doug Downen, licensed under creative commons


  1. Fascinating piece, and a reminder of my days watching the introduction (and bypassing) of electronic filing in Dept of Health and the Treasury.

    I think the real challenge for historians will not be the finding of specific information (I can, for example, find out that CLG spent £800 with Berkhamsted Couriers last year) but how different discussions and policy documents link to each other, and what the process for decision making was. This is a separate issue from ‘sofa government’ – though related to it – but the last few hectic days before something is agreed will be a lot more of a jigsaw puzzle than they were.

    1. I think that’s absolutely right, and true not just for last few hectic days. Policies and decisions don’t, on the whole, burst into life fully formed, they have histories and ancestry which may go back months or years, and it is indeed the links and influences between them which can be the key to understanding why certain things are done, or done in the way they are.

      But finding the specific information is not much easier either. One of the purposes of this blog is to help me find things I half remember thinking five years ago. I have no equivalent tool at work for finding my thoughts, let alone anybody else’s. That’s an important reason why so much energy is devoted to the reinvention of wheels.

  2. The point about “friction” reminds me that on the topic of “friction as a product not a by-product” I have discovered a truly remarkable hypothesis which this comment box is too small to contain.

  3. I have been warning friends and acquaintances for years that they will lose their digital images. They know too little about how not to lose them.
    I became concerned about how public records would not get as far the archives around the time I was part of a small team to implement the first Local Area Network in English Heritage. Someone made a decision to get rid of the last computer in the office that could read the 5 1/4 inch floppy disks on which I had stored the region’s planning data. The printouts were too big to fit in files. I think that they probably threw out the disks that I had kept in the hope that we could find a machine to transfer them.
    Our first LAN server was affected badly by power cuts for months. We had to do back ups overnight. There was always a possibility of losing mail and documents. E-mails were not archived, and yet we discussed a lot informally via e-mail.
    I don’t suppose it was the only public organisation to have these issues.
    As a historian, I worry that I will never be able to look back into the documents of the 1990s and 2000s and find out why decisions were taken, get an idea of characters and relationships between people that I could from the older, paper-based archives.

  4. I think this discussion highlights for me why I always felt uncomfortable working in a public sector setting where record-keeping was so paramount. I have always been more interested in doing things and making things happen than in recording the minutiae of it. And, I know that is a big weakness. I was probably the person who Janet was chasing for files to archive when I probably (a) hadn’t put anything in them; (b) didn’t know where they were, or (c) was desperate to hold on to them to remind me of what I was supposed to be doing. This is a key reason why I became so interested in social media as a means, among other functions, of capturing in real-time the outcomes of what people are doing. I hold out hope it will be the salvation of us all. Until we find ways of recording outcome automatically without taking people away from the activities they love, then there will always be problems like this. ;)

Comments are closed.