Regulation of Political Opinion Polling Bill [HL]

Lord Foulkes of Cumnock Excerpts

Friday 19th June 2015 Lords Chamber

Moved by

[Lord Foulkes of Cumnock]

That the Bill be now read a second time.

[Lord Foulkes of Cumnock (Lab)]

My Lords, many Members of the House will recall that I introduced this Bill in the last Session. It only got to First Reading, but when I introduced it in this Session it got a much more animated welcome, for reasons that I will go into.

First, I will go back a few years to when I was a student—well, that is many, many years. I did some work on polling then, and I learnt some of the techniques of polling, such as random sampling and the importance of finding the people who were chosen for the random sample and going back to them until you actually get their views; you do not take any substitutes. I learnt about quota sampling and the importance of getting class, sex and age in the right numbers and the right groupings to represent a cross-section of society: the make-up of the whole population.

I also learnt about the inexorable margin of error, however good your polling is. Most important of all, I was told about what makes a good poll and what does not. A poll should be run in a scientific, politically neutral manner, with no influence from those who have commissioned and indeed paid for it. That means, to take just one example, avoiding leading questions and ensuring that the wording of questions is fair and unbiased.

For some time now I have been concerned at the direction in which polling in this country has been moving. Rigour and accuracy seem to be subordinated to the demands of speed in getting the poll out and keeping the cost down: instant polling and cheaper polling. The media expect polls to be completed in an extraordinarily short time, often to be ready for publication the day after the sample has been taken. That means that there has been a preference for a predominance of internet and telephone polling, often using predetermined panels which stay the same throughout the whole of the polling period.

My concern about the dangers of this corner cutting were reinforced when I attended a seminar chaired by my noble friend Lord Lipsey, who I am glad to see is going to speak in the debate. Polling experts John Curtice and Peter Kellner were in attendance and they confirmed, in answer to a question I put, the paramountcy of speed over accuracy because of the incessant demands of the media. That is the background which gave rise to my strong determination to introduce this Bill.

What reinforced for me the point that accurate polling is an important issue for the future of our democracy was the one rogue YouGov poll held on 7 September 2014 that seemed to indicate for the first time in the referendum that the Yes campaign was ahead, by 51% to 49%. This caused a widespread panic among politicians in the Better Together campaign. It resulted in a vow set out on the front page of the Daily Record to go for even greater devolution of powers to Scotland and led to the creation of the Smith commission. In light of the actual result of the referendum, it is clear that the fears of a Yes victory were unfounded and that the nationalists had directly benefited from just one highly inaccurate poll. It is not right that the real issues of democratic politics should have been so materially affected by a statistical prediction that turned out to be so wrong. Indeed, the course of history was changed by that one inaccurate poll.

The general election gave further evidence of the direct and highly undesirable impact of polling on politics and on events. The constant polling—and it was constant—and the constant media coverage which resulted from that made it seem beyond doubt to the media and to all of us that the contest for the general election was going to be neck and neck. Some polls suggested that Labour would be the largest party. All of that polling shaped the nature of the debate. Members of this House will recall that throughout the election, the main topics of debate were not important policy issues such as defence, foreign affairs and the health service, it was the consequences of a Labour minority Government, with the SNP set to hold the balance of power.

Noble Lords will remember the posters of Ed Miliband in Alex Salmond’s top pocket and of Nicola Sturgeon pulling the strings. All those resulted from the polls, which predicted that the election was going to be neck and neck. Consequently, major policy issues were absent from the campaign and the result of the election could well have been different if we had focused on those major policy issues. Inaccurate polls, as they turned out to be, again appear to have changed the course of history.

As a result of those manifest polling errors, I have found that when I talk to—I was going to say “comrades”—colleagues and others, there is now far greater support for the creation of some kind of regulatory body overseeing political opinion polling than ever before. That is why I have reintroduced this Bill in this Session. The political will is there to ensure that similarly damaging mistakes are not made in the future and that our democratic process is not undermined.

Even the British Polling Council realised that something went wrong and admitted it. It has set up an inquiry into why the polls in the run-up to the election were so consistently inaccurate. But the British Polling Council is a self-regulating body and will be so in carrying out the inquiry. It proposes merely a one-off investigation. Understandably, the council claims credit when the polls are correct but it needs to accept some blame now that its methods have been shown to be ineffective. To ensure that a more rigorous and accurate system is introduced, we need an independent and permanent regulator.

Contrary to what has been said in some media headlines and comment, the Bill does not legislate to ban polls but would allow the proposed regulatory authority to impose limits on their publication if it thought fit. That is already the case in Spain, France, India and Italy where, if it is thought that that could be helpful, the decision could be taken. The Bill would replace the self-regulation of the British Polling Council with an independent body which would have responsibility for issuing regulation and guidance on things such as sampling methods, the wording of questions and arrangements concerning publication, including how close to election day polls could be published.

I should answer some of the criticism made by Professor Ron Johnston of Bristol University in a letter to some noble Lords—interestingly, he did not send a copy to me—and by the Political Studies Association on Twitter. I wish that both had approached me directly. They have expressed concern that the Bill might infringe academic freedom to undertake polling on political attitudes and behaviour for the purposes of independent research. That certainly is not my intention. Clause 1(8) makes clear that the authority’s regulatory powers will be restricted specifically to polling concerning voting intentions in, first, local authority elections; secondly, in parliamentary elections, including the Welsh, Northern Irish and Scottish Parliaments; and, thirdly, referenda. Academic research on other political behaviour will therefore not be affected. If there is any worry that wording changes might be needed, I would be happy to consider any amendment necessary.

It is also important that all those with an interest in polling are represented on the board of the authority. The Bill proposes that we should have representatives nominated by the British Polling Council, which would represent the industry, as well as representatives from all the political parties and the media. Indeed, I am open to other suggestions as well. Transparency is also important, and this Bill provides that the authority would publish its rules within six months of its establishment and consider amendments at least annually.

I welcome the fact that the noble Lord, Lord Cooper of Windrush, will speak in the debate—in the absence, sadly, of the super-pollster Lord Ashcroft from our midst. Are we not less spectacular and exciting in his absence? Incidentally, he is not really a pollster. He contracted organisations to carry out his polling during the election. He decided where it would be, the questions and the publication—he is a multimillionaire, of course, so he can pay for it—but he would not tell us which organisations carried out his polling.

So we have the noble Lord, Lord Cooper, from Populus, who will no doubt give us a view from the industry. I will be interested to hear what he has to say. I feel that the industry has been unduly defensive about my Bill. Given that the BBC is regulated, that we have regulators such as Ofcom, and that other organisations and industries are regulated, it surely makes sense that the multimillion-pound political opinion polling industry is brought into line with the others. I look forward to the noble Lord’s comments.

In conclusion, polling has grown exponentially in recent years.

[Lord Blencathra (Con)]

I am very grateful to the noble Lord. I do not want to be mischievous—well, not too mischievous—but has he seen the report about the Glasgow pensioner who got odds of 7:1 on an outright Conservative victory, apparently put down £30,000 and cleared £240,000? Has the noble Lord considered recruiting him to head up his polling organisation?

[Lord Foulkes of Cumnock]

That is a very interesting suggestion. As usual, the noble Lord is not being mischievous; he is being very helpful. It would be good to have an independent chair of the polling authority. I am not exactly sure whether it should be that pensioner. We must find out who he is and whether other predictions and suggestions he has made have been successful. We certainly should take that on board.

[Lord Hamilton of Epsom (Con)]

Can I raise a more serious question: is there any point to polling at all? You ask people what they are going to do at the next election. They say, “The next election is not for weeks or months. I haven’t even made up my mind”, so their views are not even very relevant.

[Lord Foulkes of Cumnock]

That is a very interesting question and much wider than what I am suggesting. If the noble Lord is suggesting that the regulatory body I am setting up should have a wider remit, that is certainly something the House can look at. I would not be averse to looking at it.

As I was saying, in the general election of 2015 we saw almost daily polls for a while—it was astonishing. However, almost all of them turned out to be wrong. The media moguls, who are very rich and own most of our newspapers, commission most of those polls. They publicise them and they become a very powerful election tool. As the noble Lord implied, this has moved beyond a method of independent measurement of voting intention to having real and increasing influence over the result, with potentially serious consequences for our democracy. Polls now play a major part in deciding the future of our country. It is therefore essential that they be carried out in a rigorous and unbiased manner. That is what the minimal and independent oversight that I am putting forward in the Bill sets out to achieve. It is with that aim in mind that I beg to move.

[Lord Cooper of Windrush (Con)]

My Lords, as the noble Lord, Lord Foulkes of Cumnock, noted, following the recent retirement from your Lordships’ House of my noble friend Lord Ashcroft, I think that I am the only Member of this House who is a pollster by trade. Therefore, I declare my obvious interest and draw the attention of the House to my entry in the register. I am the co-founder of a research company. For well over a decade I have earned my living by conducting research. A very small part of that research is polling, a very small part of which is political polling. The Market Research Society says that less than 1% of all the market research conducted in this country is polling and a tiny fraction of that is political polling. The noble Lord, Lord Foulkes, exaggerates when he describes it as a multi-million pound political polling industry.

I put firmly on the record that I, of course, accept that the recent general election was a serious failure for those of us who produce opinion polls that try to capture accurately the proportions in which people support the different political parties. The polls did not get it all wrong; they were pretty accurate in describing and predicting the scale of the landslide that took place in Scotland. They got the vote share for the Liberal Democrats and UKIP about right, but they got wrong the single most important thing—the proportions of Labour and Conservative votes in England. That was a serious error.

Voting intention polls are meant to scatter either side of a mean—in other words, when you look back after the election, there should be about as many polls a bit over what each party got as there are a bit under. However, that did not happen with the recent general election. All the final polls overstated the support for the Labour Party and understated the support for the Conservatives. In fact, if we look back further, there were more than 1,000 polls in the second half of the last Parliament, and, far from scattering either side of a mean, only one of those polls put the Conservative share of the vote higher than the 38% that they eventually got.

We know that at some point between 2010, when the polls were pretty accurate, and 2015, when they were wrong, tried and tested methods suddenly failed to capture accurately a snapshot of how voters were going to vote, and in what proportions they were going to support the Labour and Conservative Parties. One of the flaws of the regulatory body proposed by the noble Lord, Lord Foulkes, is that, had it existed during the last Parliament, it would have had no point of reference and no way of detecting during that five-year period that suddenly, poll methods which had been accurate had become inaccurate. It is only after they err at an election that we can see that clearly, and at that point, obviously, they must be addressed.

There was a serious failure and the polling industry takes it seriously. Before breakfast on the morning after the election, the polling organisations had all agreed without reservation that a full and open inquiry had to be held. That inquiry was established within 24 hours under the joint auspices of the British Polling Council and the Market Research Society and under the independent chairmanship of Professor Patrick Sturgis, a highly respected academic and director of the ESRC’s National Centre for Research Methods. The inquiry holds its first public evidence-gathering session this afternoon.

Since the general election, I have encountered an amazingly large number of people who are very keen to tell me that they knew all along that the polls were wrong, and they had always foreseen a Conservative victory. I struggle to recall many people who said that before the fact, but I note for the record that the Minister, my noble friend Lord Bridges of Headley, is one of the few who certainly did. As noted in yesterday’s Question for Short Debate, he not only placed a bet a year ago that the Conservatives would win a majority but even correctly predicted the exact size of that majority. As the noble Baroness, Lady Hayter, said, perhaps we should all have saved ourselves the trouble and just polled my noble friend Lord Bridges. It would have been quicker, more accurate and cheaper than polling the 4,000 others whom we polled in our final pre-election poll.

However, it is important to remember that we have been here before, and more important still to remember the lessons of that history. One of the reasons why almost everybody assumed the polls were right was that they had been right for the previous four general elections. However, those four consecutive successes for the polls came after another humiliating failure. As many noble Lords will remember, in 1992 the polls were also wrong—in fact, they were even more wrong in 1992 than they were in 2015. After that failure there was a full inquiry, conducted publicly and transparently, just as there is now. Its conclusions led to a series of changes in the way that voting polls were conducted. These changes by and large fixed the problem. As I noted, at the next four elections, the polling organisations that used those post-1992 methods got the result right. They scattered either side of party vote shares and remained within their margin of error. That is about as accurate as we can expect polls to be, as the noble Lord, Lord Foulkes, noted earlier. Most if not all the changes that were adopted by the industry to fix the failure in 1992, which resulted in the polls being more accurate subsequently, would have been less likely if the noble Lord’s Bill had applied and the regulatory body he proposes had existed.

The noble Lord has expressed a nostalgic attachment to face-to-face polling, as if that is the immutable gold standard of doing a poll, but the switch away from face-to-face polls was one of the central recommendations and conclusions of the 1992 inquiry. The inquiry concluded that for many different reasons it had become too difficult to get a representative sample of the whole population—of all different types of voters—by doing a face-to-face poll, so one of the recommendations was to switch to random digit-dial telephone polling. It was very controversial at the time. Many people opposed it and felt that switching from face-to-face to telephone at a time when only about 90% of households had a fixed-line telephone was a dubious step to take. After the inquiry, some pollsters switched to the phone method; some did not. Learning from the inquiry, some adopted new measures of weighting polls; others did not. At the next election in 1997, the pollsters which had made those switches were accurate and the ones that had not got the result wrong again.

The lesson from the last time the polls were wrong is that we need to define the problem openly, frankly and fully, and then innovate to solve it. The flaw at the heart of the Bill, in my opinion, is that it would obstruct this process, not help it. The noble Lord’s Bill would give a new regulatory authority responsibility for, “specifying approved sampling methods”. As I say, it is highly likely that in 1992 such a body would have judged telephone polling to be too risky and would have probably stopped the change that made the single biggest step towards fixing the problem and restoring accuracy to the polls. The noble Lord also wants “the wording of questions” to be governed by the new authority that the Bill would create. That seems to be verging on the Orwellian—the idea that we have a state-established body that will decide what you can and cannot ask, and in what terms, seems extraordinary to me.

[Lord Foulkes of Cumnock]

Just on that specific point, does the noble Lord not agree that in the Scottish referendum the wording was vital? There were discussions between both Governments about it, and wider discussions about it, but it was an absolutely crucial issue.

[Lord Cooper of Windrush]

I entirely accept, of course, that the wording of questions matters. Everybody who works in opinion research and everyone who does polls knows perfectly well that the way you ask the question can make a big difference to the answer you get. But my point is that on any issue of consequence, substance or controversy, where there will be impassioned views on both sides, there is no universally accepted neutral way of expressing a poll question. That goes to a central error at the heart of the Bill, which is the idea that these things are or ever could be subject to clear-cut right or wrong answers. There is no consensus—or anything close to one—among research organisations about either the best way to conduct a poll or the right way to phrase the questions that you may ask. That is why the British Polling Council was created: to ensure transparency and disclosure, rather than attempting to define the undefinable or police a supposedly objective “correct” approach, because no such approach exists. In my opinion, the state-backed regulator proposed by the noble Lord would stifle or kill the experimentation and innovation that have worked for the industry in the past, and which it still needs.

The third and final power proposed for the new regulatory authority is the power to ban the publication of polls before elections if it so chose. I think this would be an extraordinarily illiberal step, an affront to freedom of expression and one which certainly the courts of France and other places have judged to be a prima facie breach of Article 10 of the European Convention on Human Rights. There are two further very clear practical reasons why it is a terrible idea.

First, it would create an asymmetry of information. The Bill posits restrictions on the publication of polls. It would not and could not stop polls being conducted, certainly by political parties, which would continue to conduct private polls and would no doubt continue to talk to journalists about what those polls say. It would not stop polls being conducted by, for example, hedge funds and investment banks, and quite likely by media clients as well, as the noble Lord says. All it would do is stop those polls being openly reported. Pollsters would get to sell what would have become privileged information to private clients—information which, without the provisions of the Bill, they would have had to read in the newspaper like everybody else. The Bill would turn a world where everyone has the same information into one in which the powerful would know what was going on but the voters would not.

I was struck by the way that the noble Lord spoke about the supposed influence of the polls in the recent election. Most of his remarks were actually about the way that the polls were reported and the dominance in the media coverage of the picture that they were telling which, as we now know and clearly accept, turned out to be wrong. However, unless he is also proposing to circumscribe the freedom of the press to comment on what they think is going on in public opinion and how they read the state of the parties, I do not see how his Bill would in any case achieve the objectives that he set out. I assume that he would not propose to circumscribe the freedom of the press as well.

The second fundamental flaw with the Bill is that, in the internet era, it should be pretty obvious that banning the publication of polls is totally unenforceable. That fact was explicitly at the heart of the judgment of the French courts in 1998 to scrap the law that France had to ban polls in the seven days before elections because they judged even then, nearly 20 years ago, that the viability of media blackouts was fatally undermined by the emergence of the internet. There would be nothing we could do to stop foreign polling organisations conducting online surveys and publishing them online. There would be nothing we could do to stop anyone else publishing on websites with domains beyond UK jurisdiction the results of these polls, so any attempt to ban the publication of polls is simply unenforceable and futile.

I submit that the provisions in the Bill could not and would not have done anything to alter the fact that the polls were wrong, or to stop them being wrong. In fact, it would have made it much more difficult for the polling industry to respond responsibly to those problems. That is why anyone who cares about opinion polls and their accuracy, and their important role in a vibrant, free democracy, should oppose the Bill.

--- Later in debate ---

[Lord Foulkes of Cumnock]

My Lords, this has been a very interesting, and at times quite exciting, debate. I loved the Orwell/Clouseau allusions.

Perhaps I may deal with the Minister’s comments first. My point about Space Invaders had nothing to do with aliens or ET; it concerned the machines that young people get addicted to. In the 1980s, it was Space Invaders; now, there are other kinds of machines and activities which young people get addicted to and which create problems. One of my colleagues in another place raised this matter very recently.

The Minister mentioned the Scottish referendum and challenged me, because not just one poll but three polls were wrong. That reinforces my argument. All three of them were wrong and that makes it even worse. I am sure that the Minister’s reply was written before he heard my speech. I did not argue that I was introducing the Bill because Labour lost the election; quite the reverse. That was not part of my argument. I will come back to the Minister shortly but I think that in the end, he gave at least a little bit of an indication that he was willing to discuss this issue further.

I welcomed the response from my noble friend Lady Hayter on the Front Bench. She made a particularly important point about recall ballots. It would be quite outrageous if opinion polls were used in the way that they have been against an MP who was facing a recall ballot. That is a very important point and I was grateful that my noble friend said that she welcomed the debate on this issue.

I was particularly grateful for my noble friend Lord Lipsey’s welcome. He and I usually agree on most things. We agreed on most of this today, and I look forward to further discussions with him, in Committee and outside, so that we can consider this whole issue further.

I think that the noble Lord, Lord Cooper, with no disrespect to the Minister, made the most effective criticism of my Bill. I respect that, because he knows exactly what he is talking about. This issue needs further discussion. He said that the pollsters got it wrong in 1992. I remind him that they also got it wrong in 1970. He may be too young to remember that far back, but I remember it very well; I was a candidate in that election. They thought that they had got it right in 1992, but they got it wrong again. It seems that they get it wrong every 22 years.

The noble Lord referred to tried and tested methods. However, it seems that the methods that were tried and tested are being abandoned, and that is my concern. He also raised a question about publication and suggested that it may be the media that need regulation. I would not disagree with that. That is being dealt with elsewhere in another context, but it should be looked at in relation to this issue as well.

I hope that we will look at this. I said in my introductory speech, and I reinforce and underline it now, that I am willing to look at amendments. I just want this to be looked at. If there is a better way—if my noble friend Lord Lipsey or even the noble Lord, Lord Cooper, can come up with a better way—I am willing to look at it and to consider a substantive amendment.

I must say that I respect the Minister—his speech was fascinating and enjoyable, especially his Orwell and Clouseau allusions—but way back in the 1980s I proposed something. I was strongly, sometimes violently, opposed by the industry, the Minister pooh-poohed me and the other place threw out my proposal. It said that it was ridiculous. Why was I suggesting it? Why was I imposing this limitation on people’s freedom? I was proposing at that time to ban smoking in public places. Now it is the accepted norm. Everyone agrees with it.

[Lord Purvis of Tweed (LD)]

I am most grateful to the noble Lord—perhaps I may call him my noble friend—for letting me intervene. I was rather struck when listening to the Minister’s contribution because I read an article from 2012 in the Daily Telegraph by a certain George Bridges—I am assuming that it is the same George Bridges as the Minister—in which he said:

“But politicians who are guided by polls are chasing will-o’-the-wisp in a forlorn search for popularity. They are not selling baked beans, but something more complex: vision, belief and leadership. And the more politicians change to reflect every passing fad, the less the public believes what they say, and will-o’-the-wisp flits away”.

I congratulate the noble Lord on his championing of the ban on smoking in public places at the time, which shows a bit of the vision and leadership that was not entirely deflected by the polls. I thank him for airing this issue in the Chamber today.

[Lord Foulkes of Cumnock]

I am really grateful to my noble friend, but I would have been even more grateful if he had given me that quotation earlier. I could have used it in my speech because it is a devastating one in relation to the Minister.

He has reminded me of something else that I was going to say to the noble Lord, Lord Cooper. He said that political polling makes up less than 1% of the income of polling and market research organisations such as Populus. However, it is a key and prestigious part. Their reputations depend on getting it accurate. The soap manufacturers and chocolate producers look carefully at how accurate these things are, so although it is only small, it is an important part.

There has been substantial criticism from the other side and support on this side, but this is something we should look at further. I shall conclude by asking the House to give the Bill a Second Reading.

Bill read a second time and committed to a Committee of the Whole House.