Higher Education and Research Bill

Lord Watson of Invergowrie Excerpts

Committee: 5th sitting (Hansard - continued): House of Lords
Monday 23rd January 2017 Lords Chamber

[Lord Lucas (Con)]

My Lords, to pick up on the recently finished speech of the noble Baroness, Lady Wolf, I thoroughly agree with the three main points she made. First, producing a mixed indicator, as the Government propose, would not be useful to students or others looking at the quality of a university or a course. It would be like composing a meal out of mincemeat, cornflakes and cleaning fluid. Each of those things is useful in its own right, but mix them together and they have no function. Keep them separate, as the noble Baroness advocated, and you get some very useful data on which students can judge in their own terms the quality of a university.

Secondly, let these things be criterion-referenced. We have a real problem at the moment in GCSE—we are saying that every child should get English and Maths, but we are making that impossible, because we make these exams harder as students do better. About 30% are required to fail in order to meet the requirements of Ofqual. We have to be careful about this when we are looking at a bronze, silver or gold indicator. If we do not make these indicators criterion-referenced, we are saying that, whatever happens— however well our universities do—we will always call 20% of them bronze. In other words, we will put them into an international students’ “avoid at all costs” category. That seems a really harmful thing to do. If these criteria mean anything —if there is a meaning to any of the elements going into the TEF—we should be able to say, “We want you to hit 60%.” Why not? Why do the criteria have to be relative? They do not mean anything as relative criteria. They must have absolute meanings and they must be absolute targets.

Thirdly, this really adds up. The noble Lord, Lord Liddle, made it clear that gold, silver and bronze indicators—this big step change between the three grades —are not suited to a collection of imprecise measures. You do not know whether an institution that you have placed towards the bottom of silver is actually bronze or, worse, whether something in bronze is actually in the middle of silver. It is not that exact. You have to do what the Government do elsewhere in education statistics—for example, in value added on schools—which is, yes, to publish a value, but publish a margin of error too. That way, people get to learn that you might be saying: “This is actually 957 on your scale of 1,000, but the error margin is somewhere between 900 and 1,010.” You get used to the imprecision, to understand that this is not precise, so you can put a proper value on the information you are being given.

[Lord Watson of Invergowrie (Lab)]

My Lords, I am speaking to the proposal, in the name of my noble friend Lord Stevenson, that Clause 25 should not stand part of the Bill.

That clause refers to the Office for Students taking over HEFCE’s current administrative responsibilities to deliver the TEF on behalf of the Secretary of State. I say in passing how disappointed I am that so many in your Lordships’ House, whom I thought would come to hear this debate on TEF metrics, have now departed. Perhaps that was not the reason they were here after all. Those of us who are ploughing through the Bill until all hours of the night realise that this is an important topic. The fact that we have had so many speakers on it is a clear reflection of that.

As the Minister will be aware, there is widespread concern across the sector at the use of proxy metrics, including statistics on graduate earnings, in an exercise that was supposed to be about teaching quality. On the face of it, there is some logic to the metrics. It is difficult to imagine an excellent course, the teaching, support and assessment for which the students think are rubbish, and that a large proportion of the students do not complete; or that hardly anyone who completes it manages to find employment or get a place on a postgraduate course.

Where metrics are used, they have to be much more securely evidence-based than those suggested. Last week in Committee, our Amendments 196 and 198 would have obliged the Office for Students to assess the evidence that any proposed metric for assessing teaching quality is actually correlated to teaching quality, and ensured that, prior to making that assessment, the OfS consulted those who know first-hand what is needed to measure teaching quality: academic staff and students. The Minister did not comment on that point, so it remains one on which I should like to hear his opinion. The importance of ensuring the statistics used are reliable and evidence-based cannot be overstated. They must earn and retain the confidence of the higher education sector—and that involves academics, students and administrators.

In her Amendment 201, the noble Baroness, Lady Wolf, seeks to ensure the quality of the statistics used by the OfS, and this should be a basic requirement. I support my noble friend Lord Lipsey in questioning the validity and value of the National Student Survey. The survey merely asks students about their perceptions of teaching at their institution. By definition, these perceptions are subjective and cannot involve comparing institutions. I heard what the noble Lord, Lord Willetts, said, when he suggested that similar institutions could be compared in terms of their ethnic make-up and students’ economic background. That kind of benchmarking sounds improbable at best because, even if suitable comparators could be found, the question is, how would the outcome be weighted?

It sounds as though gold, silver and bronze categories would be created before the metrics had even been measured. As I said, that sounds improbable to me, and I agree with the noble Baroness, Lady Wolf, that benchmarking is surely not the answer. Linking institutions’ reputations to student satisfaction is likely to encourage academics to mark more generously and, perhaps, even avoid designing difficult, more challenging courses.

With academics increasingly held accountable for students’ learning outcomes, students’ sense of responsibility for their own learning—something I thought was a core aspect of higher education—will surely diminish. We are now entering an era where students dissatisfied with their grades can sue their universities. Improbable as that sounds, only last week the High Court ruled that Oxford University had a case to answer, in response to a former student who alleged that what he termed “boring” and “appallingly bad” teaching cost him a first-class degree and the opportunity of higher earnings.

This may be the shape of things to come. Last year, nearly 2,000 complaints were made by students to the higher education Ombudsman, often concerning contested degree results. Nearly a quarter were upheld, which led to universities being ordered to pay almost £500,000 in compensation. Does anyone seriously believe that the introduction of the TEF metrics will lead to a reduction in such complaints?

Metrics used to form university rankings are likely to reveal more about the history and prestige of those institutions than the quality of teaching that students experience there. The Office for National Statistics report, on the basis of which the TEF is being taken forward, made it clear that they were told which metrics to evaluate, leading to the conclusion that these metrics were selected simply because the data were available to produce them. It is widely acknowledged that students’ experience in their first year is key in shaping what they gain from their time at university, yet the focus of the proposed metrics, of course, is mainly on students’ experiences in their final year and after graduation.

The ONS report was clear that the differences between institutions’ scores on the metrics tend to be narrow and not significant. So the majority of the judgment about who is designated gold, silver or bronze will actually be based on the additional evidence provided by institutions. In other words, an exercise that is supposedly metrics-driven will in fact be decided largely by the TEF panels, which is, by any other description, peer review.

Although the Minister spoke last week about how the TEF would develop to measure performance at departmental level, the ONS report suggested that the data underpinning the metrics would not be robust enough to support a future subject-level TEF. Perhaps the Minister can clarify why he believes that this will not be the case—the quality of courses in a single university tend to be as variable as the quality of courses between institutions. As I said in Committee last week, this would also mean that students’ fees were not directly related to the quality of the course they were studying. A student at a university rated gold or silver would be asked to pay an enhanced tuition fee, even if their course at that university was actually below standard—a fact that was disguised in the institution’s overall rating.

Learning gain—or value added—has been suggested as an alternative, perhaps better, measure of teaching quality and is being explored in other countries. At a basic level, this measure looks at the relationship between the qualifications and skills level a student has when starting their degree programme, compared to when they finish—in other words, a proper, reliable means of assessing what someone has gained from their course of study.

The BIS Select Committee report on the TEF metrics published last year recommended that priority should be given to the establishment of potentially viable metrics relating to learning gain. I hope the Minister will have something positive to say on that today, or, failing that, on Report. We do not believe that the metrics as currently proposed are fit for purpose; more importantly, nor do many of those within the sector who will be directly involved with the TEF. That should be a matter of some concern for the Minister, for his colleague the Minister for Universities and Science, and indeed for the Government as a whole.

[Viscount Younger of Leckie (Con)]

My Lords, when we last met, and as the noble Lord, Lord Lipsey, said, we had a useful and wide-ranging debate on the TEF, and I value a further debate on this important subject.

The Conservative manifesto committed that we will,

“introduce a framework to recognise universities offering the highest teaching quality”.

During last Wednesday’s debate, I was pleased that, as the noble Lord, Lord Watson, noted, all noble Lords who spoke were in favour of improving teaching quality and of having a teaching excellence framework in some form.

Before discussing the specific issues raised today, I should like to clear up what appear to be some misapprehensions about how the TEF will operate. Before doing so, I should say that I will write to the noble Baroness, Lady Wolf, who raised a number of detailed points. I think it is best if I address those specific points in another letter. I should reassure noble Lords that I have just signed a letter relating to our previous day in Committee, and that should arrive on their doorsteps shortly.

It is important that when we discuss the TEF we do so in the context of the framework that has been set out, in detail, by the Government. To be clear, this framework has been designed over the past year and a half with the sector, through two consultations, and using the input of experts such as HESA and the ONS.

First, the TEF is not only—not even primarily—about the NSS, as I think the noble Lord, Lord Lipsey, acknowledges. The NSS is just one of three principal sources of metrics data being used, and we have explicitly said that the NSS metrics are the least important.

Secondly, the TEF is about much more than metrics. Providers submit additional evidence alongside their metrics, and this evidence will be given significant weight by the panel. The work of the panel will be driven as much by judgment as by metrics, ensuring that the distinctive character of institutions, as well as the diversity of missions and approaches across the sector, are recognised in the ratings awarded. Furthermore, final decisions on TEF ratings will be taken by a peer review panel, not by Ministers or civil servants.

We also consider it vital that judgments are based on a combination of core metrics, with additional and qualitative evidence, wedded together by expert peer judgment. It is for providers to determine what and how to teach, and excellent teaching can take many forms. However, great-quality teaching, defined broadly, increases the likelihood of good outcomes. In our consultation, over 70% of those who responded welcomed our approach to contextualising the data and provider submission.

I reassure noble Lords that we are not naive about the use of metrics. Chris Husbands, the TEF chair, has noted that the approach that the TEF takes is realistic about the difficulty of assessing teaching quality. He said:

“It does not pretend to be a direct audit of the quality of teaching. Instead, it uses a range of evidence to construct a framework within which to make an assessment—looking at a range of data on teaching quality, learning environments and student outcomes”.


Turning to Amendments 187, 197 and 190, that is why the development of the TEF, including metrics, is a phased process of development. Our consultation on the metrics included a table of the potential unintended consequences and our proposed mitigations. We will continue to collaborate and work with the sector to make further improvements, learning lessons from the initial, trial year. The aim is to instil and gain the confidence of the sector, and I believe we have made a very positive start. As Dame Julia Goodfellow, president of Universities UK, said:

“The government’s response to the Teaching Excellence Framework consultation demonstrates that it has consulted and listened to the university sector”.


I am concerned that some of the amendments in this group add a level of process which could reduce the incentive to make further changes to the scheme or the metrics by requiring that they are laid before Parliament as they change. This reduction in flexibility is not required by other schemes supported by many noble Lords, such as the research excellence framework.

I now turn to amendments to prohibit the use of the National Student Survey. We are listening carefully to concerns on the NSS, but we cannot ignore the only credible, widely used metric that captures students’ views. We are not using the general satisfaction ratings in the TEF; rather, we are using specific questions related to teaching quality. My noble friend Lord Willetts highlighted that point. However, as the noble Lord, Lord Lipsey, acknowledges, we recognise the limitations of the NSS and have taken steps to mitigate these, including directing TEF assessors not to overweight the three NSS-based metrics and making them aware that NSS scores can be inversely correlated with stretch and rigour. Looking at three years-worth of data will mitigate concerns about the effects on small providers. It will also help to address the concerns of the noble Lord, Lord Lipsey, about spikes created by a non-response. The need for care when interpreting results for small providers has been drawn to the attention of the TEF assessors. However, overall the panel will be encouraged in its assessment to reward and recognise quality wherever it finds it, without being bound by guideline distributions of gold, silver and bronze.

--- Later in debate ---

With that, I ask the noble Lord to withdraw his Amendment 187.

[Lord Watson of Invergowrie]

My Lords, the Minister said there are no quotas, but unless my memory fails me, when we discussed the TEF, he said he thought that gold and bronze would have roughly 20% each and the rest would be what he termed “in the middle”. I understand that they are not firm quotas, but it seems that the Government have a fairly clear idea of what they expect the outcome to be.

[Viscount Younger of Leckie]

I will have to check Hansard, but I believe I was speaking about the current system and how it is working now. I should stress that there is no quota and it could well be that these percentages are different when operated under the TEF. There is no particular expectation. I believe I was answering the question about how it might be likely to be very different.