Government Archives and Official Histories

Lord Stevenson of Balmacara Excerpts

Wednesday 10th July 2013 Lords Chamber

[Lord Stevenson of Balmacara]

My Lords, I declare an interest as a board member of the Brown Archive Trust, a Scottish registered charity that owns the personal papers of Gordon Brown MP, which are in the process of being deposited with the Churchill Archives Centre in Cambridge. I congratulate the noble Lord, Lord Rodgers of Quarry Bank, on securing this debate and on his persistence in sponsoring earlier debates on similar themes. He does us all a great service.

This has been a very high-quality debate, which has possibly fallen into three topics. First, there was official histories: the balance of opinion seems clear that they are a very important part of the overall architecture of the responsibilities of governance and accountability. I hope the Minister will be able to give us some thoughts on that when he comes to reply. I certainly find them fascinating. They are important, and we ignore them at our peril. Secondly, there was the question picked up initially by the noble Lord, Lord Rodgers, and graphically explained by my noble friend Lord Prescott: are the papers in good order? I want to come back to that later on. Thirdly, there was the interesting point from the noble Lord, Lord Butler, about historical advisers. I will also be interested to hear the response of the Minister to that.

I will focus on e-mails. I have done some work on the Brown papers—the papers relating to the Administration headed by Gordon Brown. I thank the staff in the Cabinet Office and Treasury for their expertise and support when I had to access the files for various issues in the past year or so. I have had some sense of what my noble friend Lord Prescott was saying but by and large my experience has been good and I have been able to find the papers I needed reasonably in the time. I am grateful to those who supported us in that.

However, Mr Brown’s Administration was the first that was almost entirely digital. Papers, minutes, notes and messages were all exchanged electronically, and the key evidence of meetings and events—which I notice is one of the main foci of the National Archives’ work—were in electronic form, as were the diaries. Of course, there are some traditional papers in the manila files which characterise the way Whitehall keeps its data on Administrations, but they are mostly simply print-outs from the electronic system. I know to my cost, because I spent many hours looking at them, that the paper files contain some very substantial gaps. My main concern is that the e-mails that support and exemplify how policy was decided are not generally incorporated into the paper files. Indeed, the e-mail files are kept separate and no work seems to have been done on them since the end of Mr Brown’s Administration.

Can the Minister explain what the government policy is here? I am assured—and have some evidence to back this up—that e-mails are being kept and that technology is being looked at so that they are progressively kept alive. However, keeping records is not the same as keeping records permanently. If you keep records permanently, it means that somebody has assessed the records and found them to have enduring long-term value, selected them, made them safe and secure and can find them when they are required. Keeping records indefinitely means we cannot find a basis to set a retention rule on them.

Although staff in No. 10 were encouraged to file material, we need a lot more than that. The current standard seems to be that e-mail accounts get removed after the Government change or a member of staff leaves. Surely that should happen only after a pre-exit process in which an archivist and an employee go through the e-mail account together and decide how to deal with the e-mails or in a post-exit process where an archivist looks for e-mails that need to be kept and ensures that they are catalogued and tagged for future access. That approach would at least recognise that, in the real world, people cannot be relied on comprehensively and routinely to deal with their individual e-mails by filing or deleting as they go along. E-mail communications are exchanged with such frequency that backlogs quickly scale up to a size that makes patient sifting and sorting virtually impossible.

We also need to recognise that the electronic way of working is intrinsically different from earlier, paper-based systems and our archiving needs to reflect that. E-mails typically deal with several different topics in one chain. How are they to be broken up and filed across those various aspects? Even if an individual never used their work e-mail account for non-work correspondence, their account is still likely to contain personal information of a sensitive nature exchanged with colleagues. That needs to be addressed. E-mails within a ministerial context are often political in nature: issues that perhaps should not reach the permanent archive but should be made available to those parties involved. Also, within e-mails it is hard to establish electronic document management systems that work. Access to e-mail archives is problematic because the information contained in the totality of the archive—virtually accessible if you go straight into an e-mail archive—is so sensitive that the National Archives might well have difficulty in imposing a rule that does not exclude very large amounts of information. That point was made earlier.

Of course, this is a very general area. I am sure that the issues that I have touched on here work in commercial companies as well as in government. A quote from an eminent historian of American higher education, Winton Solberg, is worth recalling at this stage:

“historical research will be absolutely impossible in the future unless”,

archivists find,

“a way to save e-mail”.

We need an approach to e-mail that results in staff leaving behind an e-mail account that their colleagues and successors can routinely access and use, without unduly harming either the account holder or people mentioned in their correspondence. We also need defensible access rules and, importantly, retention rules. I suspect it is beyond the ability of a single organisation to develop such an approach because it involves changes to available tools in the technology, to the way we think of an e-mail account and to how we ask colleagues to treat e-mails. I look forward to hearing from the Minister about what progress has been made in this crucial area.

[Lord Ahmad of Wimbledon]

My Lords, I thank my noble friend Lord Rodgers of Quarry Bank for initiating this debate. I am sure the House is fully aware of his longstanding interest in these matters and the great experience he brings to bear in debates such as these, as demonstrated by his contribution today. I also thank all noble Lords for the valuable contributions they made which raised several important questions. I hope that I am able to address most, if not all, of the issues. I will write on any questions left outstanding.

I also pay tribute to my noble friend for his work within the privy counsellors’ group, the “three wise men” as they are often referred to in this particular area. If debates such as this are about prompting interest, as a Minister in the Government, this was a new area for me. It has certainly prompted my interest, and I am looking forward to my visit to the National Archives in the next couple of weeks or so.

Turning first to official histories, my noble friend Lord Rodgers and the noble Lord, Lord Bew, referred to the reviews commissioned by the then Cabinet Secretary, the noble Lord, Lord O’Donnell, and conducted by Sir Joe Pilling and Mr Bill Hamilton. They recommended that the official history programme should be continued under the auspices of the Cabinet Office under the name “the public history programme”. They proposed substantial changes to raise the profile and relevance of the programme, including an increase in the involvement of sponsor departments and outside bodies, a revamping of the publishing arrangements and an enhancement to its governance procedures.

Several noble Lords referred to the fact that, given the current economic constraints, the Government do not plan to implement the proposed changes at the current time, and I will return to this. However, we are moving forward with the completion of the existing programme, which will conclude with the publication of The Official History of the Joint Intelligence Committee: Volume 2 in 2016. Work on this volume will, we hope, be completed by the end of 2015, after which point the recommendations will be revisited.

My noble friend Lord Rodgers referred to expense. Without incurring disproportionate expense, it is not possible to determine the overall cost of the current series of official histories. However, for the last year for which published costs are available, 2006-07, the net cost was £176,000. This cost includes fees and expenses of historians and research assistants and costs associated with publication, but excludes staff costs of Cabinet Office administrative support and accommodation-related overheads. Noble Lords will understand that until the future shape of any programme has been determined it will not be possible to estimate the likely future costs. I reiterate the words of my noble friend Lord McNally when he previously answered a debate on this subject:

“As for the official history programme, a good deal of work is already in progress, and I hope that we can review future work in happier economic circumstances. I emphasise again my enthusiasm for the programme of official histories. It would be a tragedy if we were to allow them to wither on the vine”.—[Official Report, 17/1/12; col. 547.]

I share his sentiments.

The noble Lord, Lord Hennessy, in his excellent contribution to today’s debate, referred with his usual aplomb to the histories of MI5 and the Secret Intelligence Service written, respectively, by Professor Christopher Andrew and Professor Keith Jeffery. I should clarify for the House that these were authorised histories, more akin to departmental histories, and were not commissioned under the official history programme. The noble Lord also suggested that an authorised or official history of GCHQ would be a valuable addition to those recent intelligence histories. In fact, nearly all of GCHQ’s records of the period roughly corresponding to that covered by Professor Jeffery’s history of SIS have already been released at the National Archives. I agree with the noble Lord that it is therefore open to any historian—indeed, we have historians in the Chamber—to write their own history of GCHQ. I look forward to such books being written.

Turning to the arrangements for preserving government archives, we have grounds to be optimistic given the progress made in a number of areas since 2008. First, on the responsibility for public record keeping, in line with the Public Records Act 1958, government departments are responsible for their records up to the point that they are transferred to the care of the National Archives. The National Archives provides departments with guidance and supervision, but decisions on which records to select for permanent preservation remain the departments’ own.

On guidance, in June 2009 the Cabinet Office and the National Archives revised the guidance on the management of private office papers. November 2010 saw the revision of the Civil Service Code, which now emphasises the importance of keeping accurate official records and handling information as openly as possible within the legal framework. In December 2010, the Cabinet Manual was issued, and this includes a section on official information and maintaining official records for departments. A question was raised by my noble friend Lord Rodgers about reminding Permanent Secretaries about their accountability for record keeping in their departments. It is from the Cabinet Manual that Permanent Secretaries should draw their guidance.

The noble Lord, Lord Prescott, raised several issues in his contribution, which I am sure we all found entertaining. To save on the high cost of file storage in central London, certain records have been outsourced to secure locations outside London. Regrettably, I am informed that mislabelling of the box containing the diaries of the noble Lord, Lord Prescott, led a more extensive search being required. I am sorry for any delay that that caused. However, I am sure that all noble Lords are delighted to learn that he has now perceived a positive response, and I am sure we are all looking forward to the publication of the noble Lord’s diaries; I am sure that they will make an entertaining read for us all.

[Lord Stevenson of Balmacara]

I am sure that my noble friend Lord Prescott can speak for himself, but I think his point was that there are points, particularly in today’s world, where it is vital for people to be able to respond quickly and precisely to allegations made, for whatever reason, in the press. I accept the Minister’s general point, but I do not think he responded to my noble friend’s point. Can he give us some assurance about how quickly these things can be dealt with in future?

[Lord Ahmad of Wimbledon]

First, I have apologised for the delay. It has been recognised that that should not have happened. Of course measures have been taken to ensure that the archive records should be labelled properly. I give an assurance that that has been done.