Lord Rodgers of Quarry Bank
To ask Her Majesty’s Government what is their latest assessment since February 2008 of the arrangements for preserving Government archives and preparing official histories.
Lord Rodgers of Quarry Bank
My Lords, as noble Lords will have recognised, the title, or description, of this debate is a reprise. In a short debate on 5 February 2008, I expressed my concern about two separate but related issues. One was the way in which ministerial and other papers are kept when they leave departments but are not destroyed or sent to the National Archives at Kew. The other was the need to continue the long-standing series of official histories and how best to choose, organise and publish the books.
As I explained in my debate in 2008, my interest in the first issue arose from the fate of government papers when I was seeking documents from my time as Secretary of State for Transport. After six months of fruitless exchanges between me and the department, I finally abandoned hope of anything substantial. The departmental records officer where the papers were stored in Hastings apologised for—in his words—“an unsatisfactory situation”. He recognised quite openly how this had arisen. The essential discipline—these are my words—had been lacking to ensure that they were properly catalogued, remained in the correct files or changed their names appropriately and to ensure that they were returned when they had been borrowed.
In the end, I wrote to the then Cabinet Secretary on the general question of government records and their condition. I said that some departments took great care of their archives but others did not, especially when departments were chopped and changed. I understood that his predecessors had reminded Permanent Secretaries that they were obliged to keep accurate records and to keep them in good condition. I received a helpful reply. Eventually, he said that within a year or so he would again remind Permanent Secretaries of their duty in this respect.
Since the previous debate, most, if not all, Permanent Secretaries have retired or gone elsewhere. Some departments have adopted new names and policy areas have been changed. So where are the papers now and has today’s Cabinet Secretary reminded new Permanent Secretaries of their responsibilities in keeping records? Or is it now the head of the Civil Service, Sir Bob Kerslake, who keeps Permanent Secretaries up to the mark? Put simply, can the Minister reassure me that government papers as I have described them are now in good order?
I turn to official histories. On the previous occasion, five years ago, I mentioned Problems of Social Policy by RM Titmuss, published in 1950—a seminal study of poverty and deprivation in wartime—as the first official history I had read. Another I mentioned was SOE in France by MRD Foot, which I was required to read out of ministerial interest at that time. I then referred to Churchill’s Man of Mystery: Desmond Morton and the World of Intelligence, an unusual choice even among the eclectic selection of titles in the current series. It was this book about Desmond Morton that focused my mind on some wider policy questions about the official history programme: the timetable, the shape of the programme, the publisher, the contracts with authors and the marketing of the books. The Desmond Morton book seemed to be a one-off, as it had a well designed jacket and told a fascinating story. It should have sold well, been serialised in newspapers and been considered for a television programme. I asked how many copies had been sold—at £49, about twice the usual price—and whether the book had been reviewed in magazines and journals. In replying to the debate, the Minister was full of good will but no figures or substantial response.
However, after the debate, things began to move. Before the end of that year the Cabinet Office said that it was commissioning “a fundamental review” and that Sir Joe Pilling, a retired civil servant, would undertake it. He took evidence quickly and widely, and his report was completed by April 2009. A second, associated report was written by Bill Hamilton, a literary agent, about the publishing arrangements. These were internal classified reports but, two years later, when the noble Lord, Lord Hennessy, intervened with the Cabinet Secretary, it was decided to make them publicly available on the Cabinet Office website. I welcomed this step although I was bothered that some key paragraphs in the Hamilton report had been removed as they were “commercially sensitive”.
The principal terms of reference for the Pilling report were to review the official history programme and consider whether it should continue. Sir Joe Pilling’s recommendation was strongly positive. He said that,
“the overwhelming weight of evidence supported the continuation of the programme”.
He then went on to make detailed suggestions to ensure that the programme was,
“better, stronger and more useful”.
I should say that in 1997, the Prime Minister nominated three Privy Counsellors—the noble Lord, Lord Healey, the noble and learned Lord, Lord Howe of Aberavon, and me—to be asked whether they were content with the subject and the author for each proposed book in advance of the Prime Minister’s formal approval. This led to my interest in the whole official histories programme and how it was put together and published. In this respect, I have much exceeded my proper and limited role.
However, I have kept in touch with the noble and learned Lord, Lord Howe, from time to time. At the beginning of my inquiries he shared my view that the programme appeared to fall “not short of chaos”. Later, he wrote directly to the Cabinet Secretary expressing his concern. Then, in a letter to me on 25 August 2010, copied to the noble and learned Lord, Lord Howe, and the noble Lord, Lord Healey, Sir Gus O’Donnell—now the noble Lord, Lord O’Donnell—said that, since the current programme of official histories was coming to an end:
“Given the current challenging economic climate, I am sure that there is likely to be a hiatus in commissioning new titles”.
That is where things stand. There is now a hiatus and a gap in the official histories. The last volume of the existing programme was commissioned five years ago, so the break in the sequence is already lengthening.
I am grateful to members of the official histories team at the Cabinet Office for their helpful responses to my persistent inquiries over a long period. However, I do not know who decided that there should be a hiatus. Was there a ministerial decision, and, if so, by whom and when? If a new programme either marks time or goes ahead, what are the financial consequences? Is stopping official histories because of the “challenging economic climate” really justified? As I said earlier, given the deletions in the Hamilton report, there are no relevant figures and costs, but they must be peanuts against public expenditure.
I think the latest book published in the existing programme is the second volume of The Official History of Britain and the European Community by Sir Stephen Wall. It is outstanding and wholly relevant to the possibility of a new referendum in the next Parliament. I remain disturbed about the publishing and marketing arrangements for official histories, given that this book is priced at £70—a ridiculous figure.
History never stops, and the Secretary of State for Education is very anxious that Britain’s own story is recorded and studied. To stop official histories is short-sighted. Although the Minister cannot announce today a reversal of this unhappy situation, I hope that the Government will make an early decision to resume the publishing programme.
Lord Prescott
My Lords, I congratulate the noble Lord, Lord Rodgers, on raising the matter of government archives and records. I was a bit unsure whether my experience, which I wish to relate to the House, fitted into this, but in his first few words he made clear the kind of chaos that is associated with records and archives. That has been exactly my experience in the past two years, although it does not go back as far as that of the noble Lord, Lord Rodgers.
I need to set the background as to why I needed access to the records. The background came from the dirty tricks department in the Department for Communities and Local Government led by the Secretary of State, Mr Pickles, who decided to go back to my record of expenditures in the department in 2004-06. There was clearly a political reason. He spelt out all the expenditures that were done with government procurement cards in my name. That meant, of course, that the information was given to the press. There were PQs here and in the Commons. There were stories of me running around everywhere, eating in the best restaurants and so on. They were just not true but they were put out, and they were politically inspired. My concern was how to get access to the information and challenge it.
In 2011, I wrote to the then Cabinet Secretary, Sir Gus O’Donnell—now the noble Lord, Lord O’Donnell—saying that I was very concerned about the allegations because they were not true. He wrote two letters to me, which I have here, and I spoke to him about these. In his first letter, dated 18 November, he said that there had been some fraudulent expenditures, that the person had been caught and disciplined and that I should have been told before this information was released—that is supposed to be the normal courtesy, although it normally does not apply to me. Five days later, I received another letter from him contradicting his earlier letter. This one had his name stamped on it but was without a signature. I ended up with two letters from the Cabinet Secretary. The essential difference between the two letters was that both the information about finding the criminal, the person responsible, and the comment about my not being told had been deleted from the second letter. Clearly, I got a little concerned about who signed what where. I then asked Sir Gus O’Donnell, who said that he had signed only one letter and knew nothing about the second one. He was still in that early stage before his retirement. I clearly wanted to know, so I asked the department, but it would not give me the information.
I then applied to the Information Commissioner under the Freedom of Information Act. He looked at the matter and said that he backed the Secretary of State. I went to another appeal but again he backed the Secretary of State. What was he backing? He was backing the view, under Section 36 of the Freedom of Information Act, that if it is in the public interest not to tell you, he can deny you access to that information. The Cabinet Secretary and the Information Commissioner made it clear that they knew who the person was who gave the instruction. Since they will not give me the information, I suspect that we are back to the old SPADs—Mr Shapps was the Minister at the time, and Mr Pickles was Secretary of State. Somebody directed the Cabinet Office over the weekend to change the letter. They did not say to Sir Gus O’Donnell, “We’re changing your letter. Is that all right?”. It was a political intervention to take out two important bits of information and therefore certainly relevant to what I was concerned about.
The Information Commissioner confirmed that the information was available but said that it was not in the public interest to tell me because it was a high-profile public case. I am not sure that is a sufficient answer. Why is it a high-profile case? It is because they released the information to the press about my expenditures. That is what has made it high-profile. That then becomes the justification for the Information Commissioner not to tell me why it is not in the public interest. Telling you what is happening could have a “chilling effect” on civil servants. We are talking about political intervention here, not civil servants. One civil servant does not do this. This was a direct intervention on essential information by somebody in the Department for Communities and Local Government and with the Cabinet Office over the weekend.
My concern, therefore, is how do I get that information? I cannot get it through the Information Commission, who confirms the decision by the Secretary of State, Mr Pickles. What can you do about that? I know that it is difficult when you are up against the Information Commissioner who takes that view, but I am still concerned about this political intervention. There are many expenditures in that department and they are all listed. According to the department, I must have eaten myself around the ruddy world, and in all the best restaurants, of course. Therefore, I wonder whether I can get access to my diaries. One expenditure of more than £1,000 was spent in a hotel on Christmas Eve. Your Lordships will not be surprised to learn that I was not in London having a do; I was at home. I need the diary to be able to prove that these allegations against me are just wrong.
I then went to the Cabinet Office to ask for the information, but remember that the Information Commissioner says that I must appeal if I want to go to the final body for appeal, and I have to do it within 28 days. So I wrote to the Cabinet Office and I rang them there. I said that I wanted the information from my diaries. They told me that they were very difficult to find. In the end, I got a letter this morning telling me that they had found them—this is weeks later. The trouble is that the appeal has gone. Is that the normal service that one can expect? I know that if you are a privy counsellor you are supposed to get some privilege. I am not one now, but I was one then, so I should still have access to the information on the fast track—I expect the slow track from now on. However, the circumstances are such that I could not get the diaries. I needed the diaries to show where I was on what dates and perhaps throw doubt on all the expenditures involved. To that extent, I am frustrated. I did not think that I could bring the matter up until I heard the noble Lord, Lord Rodgers, talk about delay in access to information, which Ministers have passed. If the Government can get it—and they have, in my case—and so can the Ministers, why not I, the man whom they are attacking? That is what I call political dirty tricks. I hope that they will read this speech. I just felt there was frustration and there was a chance in the debate of the noble Lord, Lord Rodgers, to put my point of view before the eminent people who will follow and who have direct experience of what happens in the Cabinet Office.
Lord Hennessy of Nympsfield
My Lords, I am left breathless, but I will try. I thank the noble Lord, Lord Rodgers of Quarry Bank, for giving us the opportunity to debate this important theme. I declare an interest as president of the Friends of The National Archives and as a teacher of contemporary British history at Queen Mary, University of London.
I will concentrate on a special oeuvre within the genre of official histories: those dealing with the secret services. Two very fine ones have appeared since the noble Lord, Lord Rodgers, last enabled your Lordships’ House to discuss official histories: first, The Defence of the Realm: the Authorized History of MI5, spanning the years 1909 to 2009, by Professor Christopher Andrew, which was published in 2009; secondly, the following year saw the publication of Professor Keith Jeffery’s MI6: The History of the Secret Intelligence Service 1909 to 1949.
Both books brim with their own special mixtures of analysis, swash and buckle, operations and organisation. Both studies kindle that fascination to which we Brits are so susceptible when reading about the King’s or the Queen’s most secret servants in either fictional or non-fictional form. It is here in that strange, twilit terrain between the facts of Professor Andrew and Professor Jeffery and the imaginations of the spy novelists that one finds the real utility of intelligence history. Spying and counterspying are activities that uniquely lend themselves to fantasy and conspiracy theory. The meticulous, careful reconstructions and assessments filtered through the minds and pens of Chris Andrew and Keith Jeffery are the best antidote we have to what one astute critic called the “snobbery with violence” practised by Commander Bond on both page and screen.
Part of the special utility of secret service official histories and historians is that they possess another virtue, a Heineken lager quality, for they can reach those parts of the secret state that others cannot reach because of the stratospheric classification levels of many of the documents on which they draw and the care needed to avoid blowing both human and technological techniques, the sources and methods of the craft that remain of enduring value.
The paper product of our secret agencies does not flow that easily into the public domain on the tide of the 30-year—soon to be 20-year—rule. But the appearance of the official histories of the Security Service and the Secret Intelligence Service has provided a scholarly bonus —a Keynesian multiplier effect—which I had not anticipated. Once the volumes had been published, a proportion of the retained files on which Professors Andrew and Jeffery had drawn could be prepared for declassification at the National Archives, and indeed were.
At Kew last May the Foreign Office, the Cabinet Office and the National Archives mounted what can only be called a cornucopian release, drawing on two of the most secret collections the British state has ever created. The FCO batch covered the Foreign Office’s dealings with the SIS—the province of what the FCO euphemistically calls the Permanent Under-Secretary’s Department—covering the years 1903 to 1951 and shedding much new light on how the secret state coped with successive threats from the Kaiser, Hitler and Stalin. The second tranche flowed from what I call the Cabinet Secretaries’ “too hot to handle” cupboard, formally known as the Cabinet Secretary’s Miscellaneous Papers. This batch covered the years 1936 to 1951 and contained material which, in my judgment, is of continuing value to today’s guardians of national security in the secret agencies and in Whitehall.
I particularly have in mind the report prepared for Mr Attlee and a small group of Ministers in 1951 on The Secret Intelligence and Security Services. Written by that great technician of state, Sir Norman Brook, the Cabinet Secretary to four Prime Ministers, it painted for Ministers a vivid and unsparing portrait of how the secret world had fared and was faring against the toughest target British intelligence had ever faced: Stalin’s Russia.
I hope that the current heads of service and the Cabinet Secretary will read the Brook report. Not only is it an exemplar of concision and penetration and the jargon-free language at which Whitehall excelled before departments sought the assistance of management consultants, it is a model for how such a review might be commissioned today; for example, if the Prime Minister wished to review the workings of the secret world as a whole with now more than three years’ experience available of his National Security Council as the taskmaster and pacemaker of the agencies.
I profoundly hope that the current austerity will not dam the flow of official histories for the foreseeable future, not least those dealing with the secret world. It would be hugely beneficial, for example, if Cheltenham could authorise an official historian to start work on a volume designed for the general reader, embracing as much as can safely be divulged about the Government Communications Headquarters and its predecessor institutions to complete the secret agency set, as it were, and make it three.
I like to think that in this House we have a keen sense of history across all our Benches; in fact, I know we do. I know, for example, that the noble Lord, Lord Cormack, would have added his wisdom to our deliberations this evening had he not been involved with our other business today. I will finish by giving my thanks to all those across the departments and agencies who provide us with such a rich paper trail—although regrettably not to the noble Lord, Lord Prescott—and those who care for the documentary product permanently at the National Archives. We are truly in their debt.
Lord Butler of Brockwell
My Lords, I, too, am grateful to the noble Lord, Lord Rodgers, for initiating this debate—at least I was until I heard the speech of the noble Lord, Lord Prescott. I am tempted to say that things must have gone downhill since my day but actually I think I can solve his problem for him. Unless things have very much changed, he does not have to go to the Information Commissioner to get the papers either about his diary or about his restaurant bills. Certainly it always was the case—I believe it still is—that any former Minister can consult the papers which he himself dealt with. My advice to the noble Lord, Lord Prescott, if he finds himself being traduced again, is to go directly to the department and ask to see them and not to bother with the Information Commissioner.
Lord Butler of Brockwell
I think the noble Lord should have gone to his own department.
I very much endorse what the noble Lords, Lord Hennessy and Lord Rodgers, said about the value of both the official histories and the National Archives, and the importance of good record-keeping in government, not just for the benefit of academic historians but because of their relevance to current decision-making. When the Government make decisions when they are not fully aware of the history of the subjects they are discussing, they are like a driver who goes out into the traffic without having taken the trouble to check in his rear mirror before taking action on the road. In that respect, both the files and the official histories are very valuable.
However, files and official histories are not enough. We need something else to exploit the lessons of history for policy decisions. If, when a crisis arises, there is no official history on the subject—certainly, if there is, it ought to be consulted; but it will be a monumental work that does not cover all areas of government policy—it is too late to go back to the archives. We need to ask what else is needed to exploit the lessons of history when they are necessary for informing policy decisions.
Of course, it would be impossibly expensive to employ enough historians in government to cover the vast span of each department’s responsibilities. But it is not too expensive for each department to have a historical adviser who would not necessarily be able to give advice on all major issues but who would have sufficient tendrils into the academic world to know where such advice could be obtained. It is not sufficient for this advice simply to be fed into the department. A historical adviser must be present at the table when major decisions are taken.
Of course, there were many in the Foreign Office and State Department who were fully aware of the complexities of both Iraq and Afghanistan when policy decisions were being taken on those countries. The question is whether they had the opportunity to make themselves fully heard by the decision-makers. In this respect, I admired the practice of Margaret Thatcher as Prime Minister, whose first step when taking a major decision was to hold a seminar of experts. The seminar she held when facing the unification of Germany has become well known. It did not necessarily overcome her prejudices but it was certainly a counterweight to them. It said that under the pressure of economies, all departments, with the notable exception of the Foreign Office, had disbanded their historical sections. If that is so, it is a grave disadvantage to the operation of government.
Will the Minister tell me, either in replying to the debate tonight or by writing to me, whether it is true that all departments except the Foreign Office no longer have historical sections? If that is untrue, which departments still have them? I also invite his comment on whether, even if departments cannot afford historical sections, they should at least have a historical adviser. Without such, government decisions will be taken blindly.
Lord Liddle
My Lords, I, too, welcome the debate initiated tonight by the noble Lord, Lord Rodgers. I want to make one simple point very strongly in favour of official history and about wanting to make official history more contemporary, not less. I believe that history is vital to people when facing difficult decisions. In the years that I spent as an adviser in government, one thing that struck me a great deal was the lack of institutional memory in government departments. Even in a long-serving Government such as the Labour Government from 1997 to 2010, Ministers changed jobs frequently and, except in one or two cases, it was very unusual to have Ministers who had a long period of office in one department.
Apart from the ministerial merry-go-round, there were frequent changes in the Civil Service. For instance, one of the most striking things in the book by my noble friend Lord Adonis on education is that, in the case of the academy programme, eight different people were in charge of it in the Department for Education in nine years when he was the relevant person at No. 10 and a junior Minister in the department. There is far too much changing around and as a result there is a lack of institutional memory. I remember when my noble friend Lord Mandelson came back from Brussels and went to BERR, I think it was called, and started to think about industrial policy, there was very little available that one could turn to that analysed the strengths and weaknesses, from the perspective of government records, of government policies in the past.
More history and more contemporary history would be good for us. The most recent official history that I have read is Sir Stephen Wall’s excellent book on Britain and the European Community from 1963 to 1975. It made me reassess what was probably a rather too jaundiced view of Harold Wilson. Stephen has given us the benefit of the minute books of the Cabinet Secretaries, so you know what each Minister said in Cabinet meetings, and you come to admire Wilson’s skill in handling questions such as the Common Market at Cabinet. The lessons for what we are currently going through—the renegotiation that the current Prime Minister proposes—come out of that book extremely strongly.
Taking the Europe example again, the reason why I would like some more contemporary examination of the records is that many of the issues that will be raised if there is a renegotiation in the next Parliament were raised during the European convention in the period from 2000 to 2003. Many of these issues about competences, repatriation of powers and the legitimacy and accountability of European institutions were thoroughly gone through then, yet I suspect that there has been no proper examination of the lessons of that experience by officials internally and certainly none by historians externally. So let us have more official history, and let us make it more contemporary.
Lord Bew
My Lords, I, too, thank the noble Lord, Lord Rodgers of Quarry Bank, for securing this debate and, indeed, for opening up this whole question with his debate five years ago. I must declare an interest, as secretary to the All-Party Group on Archives and History and, like my noble friend Lord Hennessy, as a working professor of history in Queen’s University Belfast. One of the effects of the important debate introduced by the noble Lord, Lord Rodgers, five years ago was the production of the Pilling report. The noble Lord referred to it tonight. There is a striking sentence by Sir Joe Pilling at the heart of his report endorsing the project of official histories. He said that he had come to see the work of publishing official histories as,
“the gold standard of accountability to the country from those who have been privileged to hold senior office”.
It is for that reason that Sir Joe Pilling advocates essentially an improvement and an increase in the production of official histories.
I recall that five years ago I referred to the history of MI6—which my noble friend Lord Hennessy mentioned tonight—by my colleague Keith Jeffery. It was then basically something that existed on Keith’s desk, but now it shows how these things can sell. One of the many points that are so important about that book is that it has sold hundreds of thousands of copies across the world. It may be because the Chinese copy actually has a gun smoking on the cover, but none the less it is an indication that no definite economic death follows the production of official histories. It is important to understand that the work of producing official histories is of great significance.
However, there is a darker side to Sir Joe Pilling’s report. If you read between the lines, he was aware as a member of the Dacre committee with Sir David Cannadine that it was likely that we would move from a 30-year rule. Actually, the Dacre committee recommended moving to 15 years, but in practice it is now a 20-year rule. Sir Joe is also clearly aware that this will become an argument for the state to say that it does not need these official histories any longer. He clearly tacitly acknowledges that that argument is just around the corner. It is the argument that is related to the hiatus that has been announced in the context of austerity. I can completely understand it, there is a forceful logic to it, but there is another hiatus, if I might put it like this. It would be forcefully true when the 20-year rule comes into effect and will be much truer in five, seven or eight years from now insofar as it has truth today. We still have a pressing need in public areas for a certain type of work in an official history programme.
Let me explain what I mean and take the case of the history of the Northern Ireland Office. If you want to look at the volume of official publications on Northern Ireland on the shelf of our Library, you will find a shelf of official publications including the report on Bloody Sunday and the Finucane report. There are hundreds of thousands of pages, mostly dealing with matters of that sort, occasions when the state has been seen to behave not very well. I have no objection to that; I was the historical adviser to the Bloody Sunday tribunal report. However, it is somewhat ridiculous that there is no account of the work of those officials on the British side who struggled to bring about a peace process. It is astonishing that we are silent on the more creative, positive, though no doubt deeply flawed aspects of the work of our state officials while we are so loud in announcing some of the rather bad things that went on. That seems an astonishing way to proceed.
The same point can be made about how aspects of Foreign and Commonwealth Office history in the 1940s and 1950s have been dealt with in recent times. We now seem to think that the way to educate the public about what happened in the past is to apologise or to have a large and expensive inquiry about something very bad. That may well be necessary, but this is the case for a proper, official history programme. With the hiatus that still now exists, particularly in the Northern Ireland Office, the many arguments there have been about legacy in Northern Ireland, the sense there that the past has not been properly dealt with and the small sums involved, there is a case for the Government to reconsider their approach of austerity.
There is a further point made in Sir Joe Pilling’s report that is worth drawing attention to. He says that the internet now allows you to publish a lengthy set of footnotes and a lengthier text—the full scholarly history—but a 200-page shorter version as the book. That seems another way for government to avoid unnecessary expense. In other words, I am arguing that this can be done quite cheaply. I would like to see the Government reconsider the force of the Pilling report of 2009.
Lord Thomas of Swynnerton
Forgive me for speaking in the gap. I agree very strongly with the arguments of my noble friend Lord Butler for a historical adviser to many government departments. I had the privilege of accompanying the noble Lord, Lord Carrington, on a visit he made in 1981 to South America. It was not at all clear what I was doing, but I accompanied him as a historian of the Spanish world. When we got to Brazil, an official of the Brazilian Foreign Minister asked me what my mission was. I said, “I am Lord Carrington’s historical adviser”. He had not appointed me; I appointed myself. The Brazilian official said, “What a good idea —we must have one too”.
Lord Stevenson of Balmacara
My Lords, I declare an interest as a board member of the Brown Archive Trust, a Scottish registered charity that owns the personal papers of Gordon Brown MP, which are in the process of being deposited with the Churchill Archives Centre in Cambridge. I congratulate the noble Lord, Lord Rodgers of Quarry Bank, on securing this debate and on his persistence in sponsoring earlier debates on similar themes. He does us all a great service.
This has been a very high-quality debate, which has possibly fallen into three topics. First, there was official histories: the balance of opinion seems clear that they are a very important part of the overall architecture of the responsibilities of governance and accountability. I hope the Minister will be able to give us some thoughts on that when he comes to reply. I certainly find them fascinating. They are important, and we ignore them at our peril. Secondly, there was the question picked up initially by the noble Lord, Lord Rodgers, and graphically explained by my noble friend Lord Prescott: are the papers in good order? I want to come back to that later on. Thirdly, there was the interesting point from the noble Lord, Lord Butler, about historical advisers. I will also be interested to hear the response of the Minister to that.
I will focus on e-mails. I have done some work on the Brown papers—the papers relating to the Administration headed by Gordon Brown. I thank the staff in the Cabinet Office and Treasury for their expertise and support when I had to access the files for various issues in the past year or so. I have had some sense of what my noble friend Lord Prescott was saying but by and large my experience has been good and I have been able to find the papers I needed reasonably in the time. I am grateful to those who supported us in that.
However, Mr Brown’s Administration was the first that was almost entirely digital. Papers, minutes, notes and messages were all exchanged electronically, and the key evidence of meetings and events—which I notice is one of the main foci of the National Archives’ work—were in electronic form, as were the diaries. Of course, there are some traditional papers in the manila files which characterise the way Whitehall keeps its data on Administrations, but they are mostly simply print-outs from the electronic system. I know to my cost, because I spent many hours looking at them, that the paper files contain some very substantial gaps. My main concern is that the e-mails that support and exemplify how policy was decided are not generally incorporated into the paper files. Indeed, the e-mail files are kept separate and no work seems to have been done on them since the end of Mr Brown’s Administration.
Can the Minister explain what the government policy is here? I am assured—and have some evidence to back this up—that e-mails are being kept and that technology is being looked at so that they are progressively kept alive. However, keeping records is not the same as keeping records permanently. If you keep records permanently, it means that somebody has assessed the records and found them to have enduring long-term value, selected them, made them safe and secure and can find them when they are required. Keeping records indefinitely means we cannot find a basis to set a retention rule on them.
Although staff in No. 10 were encouraged to file material, we need a lot more than that. The current standard seems to be that e-mail accounts get removed after the Government change or a member of staff leaves. Surely that should happen only after a pre-exit process in which an archivist and an employee go through the e-mail account together and decide how to deal with the e-mails or in a post-exit process where an archivist looks for e-mails that need to be kept and ensures that they are catalogued and tagged for future access. That approach would at least recognise that, in the real world, people cannot be relied on comprehensively and routinely to deal with their individual e-mails by filing or deleting as they go along. E-mail communications are exchanged with such frequency that backlogs quickly scale up to a size that makes patient sifting and sorting virtually impossible.
We also need to recognise that the electronic way of working is intrinsically different from earlier, paper-based systems and our archiving needs to reflect that. E-mails typically deal with several different topics in one chain. How are they to be broken up and filed across those various aspects? Even if an individual never used their work e-mail account for non-work correspondence, their account is still likely to contain personal information of a sensitive nature exchanged with colleagues. That needs to be addressed. E-mails within a ministerial context are often political in nature: issues that perhaps should not reach the permanent archive but should be made available to those parties involved. Also, within e-mails it is hard to establish electronic document management systems that work. Access to e-mail archives is problematic because the information contained in the totality of the archive—virtually accessible if you go straight into an e-mail archive—is so sensitive that the National Archives might well have difficulty in imposing a rule that does not exclude very large amounts of information. That point was made earlier.
Of course, this is a very general area. I am sure that the issues that I have touched on here work in commercial companies as well as in government. A quote from an eminent historian of American higher education, Winton Solberg, is worth recalling at this stage:
“historical research will be absolutely impossible in the future unless”,
archivists find,
“a way to save e-mail”.
We need an approach to e-mail that results in staff leaving behind an e-mail account that their colleagues and successors can routinely access and use, without unduly harming either the account holder or people mentioned in their correspondence. We also need defensible access rules and, importantly, retention rules. I suspect it is beyond the ability of a single organisation to develop such an approach because it involves changes to available tools in the technology, to the way we think of an e-mail account and to how we ask colleagues to treat e-mails. I look forward to hearing from the Minister about what progress has been made in this crucial area.
Lord Ahmad of Wimbledon
My Lords, I thank my noble friend Lord Rodgers of Quarry Bank for initiating this debate. I am sure the House is fully aware of his longstanding interest in these matters and the great experience he brings to bear in debates such as these, as demonstrated by his contribution today. I also thank all noble Lords for the valuable contributions they made which raised several important questions. I hope that I am able to address most, if not all, of the issues. I will write on any questions left outstanding.
I also pay tribute to my noble friend for his work within the privy counsellors’ group, the “three wise men” as they are often referred to in this particular area. If debates such as this are about prompting interest, as a Minister in the Government, this was a new area for me. It has certainly prompted my interest, and I am looking forward to my visit to the National Archives in the next couple of weeks or so.
Turning first to official histories, my noble friend Lord Rodgers and the noble Lord, Lord Bew, referred to the reviews commissioned by the then Cabinet Secretary, the noble Lord, Lord O’Donnell, and conducted by Sir Joe Pilling and Mr Bill Hamilton. They recommended that the official history programme should be continued under the auspices of the Cabinet Office under the name “the public history programme”. They proposed substantial changes to raise the profile and relevance of the programme, including an increase in the involvement of sponsor departments and outside bodies, a revamping of the publishing arrangements and an enhancement to its governance procedures.
Several noble Lords referred to the fact that, given the current economic constraints, the Government do not plan to implement the proposed changes at the current time, and I will return to this. However, we are moving forward with the completion of the existing programme, which will conclude with the publication of The Official History of the Joint Intelligence Committee: Volume 2 in 2016. Work on this volume will, we hope, be completed by the end of 2015, after which point the recommendations will be revisited.
My noble friend Lord Rodgers referred to expense. Without incurring disproportionate expense, it is not possible to determine the overall cost of the current series of official histories. However, for the last year for which published costs are available, 2006-07, the net cost was £176,000. This cost includes fees and expenses of historians and research assistants and costs associated with publication, but excludes staff costs of Cabinet Office administrative support and accommodation-related overheads. Noble Lords will understand that until the future shape of any programme has been determined it will not be possible to estimate the likely future costs. I reiterate the words of my noble friend Lord McNally when he previously answered a debate on this subject:
“As for the official history programme, a good deal of work is already in progress, and I hope that we can review future work in happier economic circumstances. I emphasise again my enthusiasm for the programme of official histories. It would be a tragedy if we were to allow them to wither on the vine”.—[Official Report, 17/1/12; col. 547.]
I share his sentiments.
The noble Lord, Lord Hennessy, in his excellent contribution to today’s debate, referred with his usual aplomb to the histories of MI5 and the Secret Intelligence Service written, respectively, by Professor Christopher Andrew and Professor Keith Jeffery. I should clarify for the House that these were authorised histories, more akin to departmental histories, and were not commissioned under the official history programme. The noble Lord also suggested that an authorised or official history of GCHQ would be a valuable addition to those recent intelligence histories. In fact, nearly all of GCHQ’s records of the period roughly corresponding to that covered by Professor Jeffery’s history of SIS have already been released at the National Archives. I agree with the noble Lord that it is therefore open to any historian—indeed, we have historians in the Chamber—to write their own history of GCHQ. I look forward to such books being written.
Turning to the arrangements for preserving government archives, we have grounds to be optimistic given the progress made in a number of areas since 2008. First, on the responsibility for public record keeping, in line with the Public Records Act 1958, government departments are responsible for their records up to the point that they are transferred to the care of the National Archives. The National Archives provides departments with guidance and supervision, but decisions on which records to select for permanent preservation remain the departments’ own.
On guidance, in June 2009 the Cabinet Office and the National Archives revised the guidance on the management of private office papers. November 2010 saw the revision of the Civil Service Code, which now emphasises the importance of keeping accurate official records and handling information as openly as possible within the legal framework. In December 2010, the Cabinet Manual was issued, and this includes a section on official information and maintaining official records for departments. A question was raised by my noble friend Lord Rodgers about reminding Permanent Secretaries about their accountability for record keeping in their departments. It is from the Cabinet Manual that Permanent Secretaries should draw their guidance.
The noble Lord, Lord Prescott, raised several issues in his contribution, which I am sure we all found entertaining. To save on the high cost of file storage in central London, certain records have been outsourced to secure locations outside London. Regrettably, I am informed that mislabelling of the box containing the diaries of the noble Lord, Lord Prescott, led a more extensive search being required. I am sorry for any delay that that caused. However, I am sure that all noble Lords are delighted to learn that he has now perceived a positive response, and I am sure we are all looking forward to the publication of the noble Lord’s diaries; I am sure that they will make an entertaining read for us all.
Lord Stevenson of Balmacara
I am sure that my noble friend Lord Prescott can speak for himself, but I think his point was that there are points, particularly in today’s world, where it is vital for people to be able to respond quickly and precisely to allegations made, for whatever reason, in the press. I accept the Minister’s general point, but I do not think he responded to my noble friend’s point. Can he give us some assurance about how quickly these things can be dealt with in future?
Lord Ahmad of Wimbledon
First, I have apologised for the delay. It has been recognised that that should not have happened. Of course measures have been taken to ensure that the archive records should be labelled properly. I give an assurance that that has been done.
Lord Ahmad of Wimbledon
I am sure that the noble Lord will agree with me that labelling matters; it is good to know whose diaries are where.
In response to another point raised by the noble Lord, Lord Prescott, the noble Lord, Lord Butler, is absolutely correct that former Ministers can see their papers within their former departments. I assure the House that this is also outlined in the Cabinet Manual, which is available online. The noble Lord, Lord Butler, also talked about the coverage of historical advisers and sections across Whitehall. The FCO still has a historical section, the head of which is Patrick Salmon. I will write to the noble Lord on the coverage of historical advisers across Whitehall in general and, of course, place a copy of that letter in the Library.
On other initiatives, the National Archives’ information management assessment programme began in 2008. To date, most of the departments of state and several key agencies have been assessed and the remainder will be assessed during 2013-14. The National Archives is also about to begin a series of ongoing reassessments. The published reports of these assessments highlight good practice and make targeted, pragmatic recommendations for improvement. The National Archives works with each department to develop an action plan to address any risks and issues identified in the report.
The noble Lord, Lord Bew, referred to the 20-year rule. As noble Lords will be aware, from 1 January this year central government began its 10-year transition from the previous 30-year rule to the new 20-year rule. To smooth this transition, the National Archives has, with the active participation of departments across government, comprehensively revised its guidance and processes for the selection and transfer of records. The National Archives has been tasked to collect and publish regular reports on departments’ progress in reviewing and selecting records for permanent preservation during the transition period. The most recent report, with returns from 84 departments and agencies, was published on the National Archives’ website on 1 July. This level of transparency around government’s records management is, I suggest, unprecedented. With these reports and the transition itself, we have come a long way from the days, prior to the Freedom of Information Act, when our best hopes for transparent government lay with such excellent initiatives as the one led by the noble Lord, Lord Waldegrave.
The noble Lord, Lord Stevenson, rightly raised digital records. Much work has been carried out to clarify and address the challenges presented by the shift from paper to digital records in the business of government. The National Archives now has greater confidence that the much discussed black hole in our history wrought by obsolete digital formats is unlikely to materialise on the scale that had once been feared. However, it is important that in the National Archives programme new technology is fully embraced. Digital continuity is also now taken much more seriously across government than it was five years ago. The programme of training instigated by the noble Lord, Lord O’Donnell, during his time as Cabinet Secretary has certainly aided a better understanding of our digital records and improved usability and accessibility. Of course, there remains the challenge of reviewing large volumes of digital records for sensitivity ahead of their potential release under the 20-year rule, and the National Archives is working with other expert bodies to develop solutions.
I am pressed for time but, in conclusion, we all recognise that there remains much to do to ensure that government records in all forms survive for future researchers and historians; indeed, that was expressed by all noble Lords today. However, I hope that noble Lords will take from today’s debate some reassurance that these issue continue to be explored and addressed and, more importantly, that much more of the Government’s work in this area is open to scrutiny by Members of this House than ever before.
The noble Lord, Lord Butler, said in his contribution that Governments need to look at history. He referred to the late Lady Thatcher and her policy of a panel of experts; I think that we can learn a great deal. The National Archives represents our history. I suppose, as a Minister of the current Government, that it is apt to finish with a quote from Sir Winston Churchill, who said:
“Study history, study history. In history lies all the secrets of statecraft”.