National Security Strategy (Joint Committee)
Monday 7th September 2026
Skip to latest contribution
This debate has concluded
(began 1 month, 1 week ago)
Watch LiveSkip to latest contribution
This debate has concluded
16:23
Speaker 1
-
Speaker 1
So somewhat familiar with this process. And the other thing that's germane probably to the most recently, adviser to the Defense Committee for its report on AI procurement. This afternoon, I was a global commissioner on the re AME Commission, the responsible Mangala acronym here Responsible and Ethical AI in the military domain. Sort of intergovernmental process. Looking at that topic for a couple of years. I don't normally wear a tie, so I might just loosen myself off a little.
Bit
Bit
16:23
16:23
Speaker 2
-
Speaker 2
Good afternoon everyone. Michael Cuthbert I'm director of the National Quantum Computing Centre at Harwell. So we are a national laboratory focused on quantum computing bridging from academic research into industry, looking at the technology, hosting commercial prototype platforms, and then also seeking to utilise. Com computing in new application areas.
16:23
Speaker 3
-
Speaker 3
Good afternoon. My name is George Bolton. I'm currently co-founder of aviary, a nonprofit working on frontier AI safety. I previously served as the co-director for Defence and national security at the Alan Turing Institute, where I still serve as an advisor. I was also the founding director of CTAs, the center for Emerging Technology and Security at the Turing. Although today I'll be speaking in a personal capacity.
16:24
Tanmanjeet Singh Dhesi
-
-
Tanmanjeet Singh Dhesi
Awesome. Great. Well, thank you all again for joining us. You're probably familiar with the process, but just to let you know, individual members of the committee will lead off on different topics, but others might chip in as we go. We won't ask all of you to answer every question. We'll usually start each each topic by, by indicating which which of the witnesses you want to lead off. But we might ask you to come in, and if you have something you feel you should add, please do so.
This is very much an evidence gathering session. So I'm going to kick off and again, thanks for joining us. So the National Security Strategy, set out in 1925, pledged to build the capability, national capability for AI and indeed other frontier technologies, increasing our national capacity, accelerating adoption, and also advancing the understanding of security risks. How far do you believe the UK has progressed against that agenda? I think that's perhaps something that all of you will have a view on.
So again, why don't we start with you, Professor Payne?
This is very much an evidence gathering session. So I'm going to kick off and again, thanks for joining us. So the National Security Strategy, set out in 1925, pledged to build the capability, national capability for AI and indeed other frontier technologies, increasing our national capacity, accelerating adoption, and also advancing the understanding of security risks. How far do you believe the UK has progressed against that agenda? I think that's perhaps something that all of you will have a view on.
So again, why don't we start with you, Professor Payne?
16:25
Speaker 1
-
Speaker 1
Thanks very much. I was actually very excited when I read that, and I think there has been some significant progress over the last couple of years as sovereign AI fund, for example, funding of companies doing exciting things like cosine. The trouble is that the frontier has been accelerating away, and one of the most alarming charts is the one. One of many alarming charts is the one produced by an organisation called meta METR, and it's on that that you can see the exponential curve.
It's measuring the performance of the frontier models to do coding, how long they can do a complex coding task with a reasonable degree of reliability. And at the moment we're just going into the steep part of that curve. And the frontier is coding capabilities are doubling every hundred days. Frontier is doubling in capability every hundred days. And so that's the target that you're chasing if you are chasing the frontier.
And I read and it's almost certainly out of date now, I think it was either McKinsey or the Tony Blair Institute's. Very nice. I'm useless with numbers, but this was a nice way of remembering what's going on here. They said that the amount of committed capital expenditure in the AI companies is 220 Manhattan projects. And that sticks in my mind, and I think it's quite compelling. So that's the challenge.
If you want to be at the frontier, the frontier is accelerating away from you. And my slightly pessimistic take is that we have no, no chance of getting to that frontier whatsoever with the implications that follow on from that first sovereignty for the UK.
It's measuring the performance of the frontier models to do coding, how long they can do a complex coding task with a reasonable degree of reliability. And at the moment we're just going into the steep part of that curve. And the frontier is coding capabilities are doubling every hundred days. Frontier is doubling in capability every hundred days. And so that's the target that you're chasing if you are chasing the frontier.
And I read and it's almost certainly out of date now, I think it was either McKinsey or the Tony Blair Institute's. Very nice. I'm useless with numbers, but this was a nice way of remembering what's going on here. They said that the amount of committed capital expenditure in the AI companies is 220 Manhattan projects. And that sticks in my mind, and I think it's quite compelling. So that's the challenge.
If you want to be at the frontier, the frontier is accelerating away from you. And my slightly pessimistic take is that we have no, no chance of getting to that frontier whatsoever with the implications that follow on from that first sovereignty for the UK.
16:27
16:27
Speaker 1
-
Speaker 1
mean, I hope so, because you can want to say about chasing after the frontier. I think adoption is really important. What you do with these models matters. And we're an ingenious country, and we'll find ingenious ways to apply these models. And it is no doubt an important part. I think the other aspect is resilience. What do you do if you lose access to these models? How do you continue functioning as a society? How do you continue thriving as a society in all possible domains? So these things matter.
But but ultimately, I think there are some very significant geopolitical implications that flow from not being at the frontier . And the recent mythos. Example brief, though it was suggested a little taster of what might come.
But but ultimately, I think there are some very significant geopolitical implications that flow from not being at the frontier . And the recent mythos. Example brief, though it was suggested a little taster of what might come.
16:28
Speaker 2
-
Speaker 2
I think it's fair to say that quantum computing and quantum technology is more generally are still at a much more formative stage than than AI. That said, I think across national security, defence, community, there's been very strong engagement with the National Quantum Technologies programme. And also increasingly moving towards early adoption and prototyping. So, so, you know, real, real world examples of a navigation with GPS denial, for example, through the through the Royal Navy.
A working with Gchq, for example, on a early prototyping, understanding capability of today's quantum computer versus the kind of crypto relevant machines will maybe come on to talk about in the future. I think, importantly, the switch from government, the funder to government as a user and working not just in the defence arena, but more widely across different government departments, again, that kind of early adoption and and then I think there's some, again, driving proof of concept concept and project with the defence primes where that's as much an educational process as it is validating the technology, but actually getting people within those organisations building skills and building familiarity, as well as building collaboration and a network across the kind of expert academic user community.
A working with Gchq, for example, on a early prototyping, understanding capability of today's quantum computer versus the kind of crypto relevant machines will maybe come on to talk about in the future. I think, importantly, the switch from government, the funder to government as a user and working not just in the defence arena, but more widely across different government departments, again, that kind of early adoption and and then I think there's some, again, driving proof of concept concept and project with the defence primes where that's as much an educational process as it is validating the technology, but actually getting people within those organisations building skills and building familiarity, as well as building collaboration and a network across the kind of expert academic user community.
16:29
16:29
Speaker 3
-
Speaker 3
Yeah. So I think, you know, I do have a concern about the UK's ability to keep up with AI and how that relates to sovereignty. AI frontier AI is fundamentally a general purpose capability. And so this means across the national security and defence communities, it's going to be useful in a range of different applications, you know, including data collection, analysis, investigations, and even intelligence gathering. And I think the question that raises itself for sovereignty is, you know, at the highest level, can the UK be cut off? So can the UK's capabilities that rely on AI be cut off, leaving us behind our adversaries? And I think there's three degrees that or three questions that we need to ask that relate to sovereignty for specific deployments.
The first is around security. So intelligence and, and defence agencies like to and want to put classified data into these systems. And so they need to be able to do so. Second, to, to Ken's point, resilience. So is access to the model guaranteed or can it be pulled by another country without our permission or even without giving us any knowledge of that? And then finally, to your point, chair capability. So how capable do these models need to be? And I think the point I'd like to make to start is not all deployments need the highest level of capability and security and resilience, but I think certainly some do.
And so the UK must have some capacity at the frontier.
The first is around security. So intelligence and, and defence agencies like to and want to put classified data into these systems. And so they need to be able to do so. Second, to, to Ken's point, resilience. So is access to the model guaranteed or can it be pulled by another country without our permission or even without giving us any knowledge of that? And then finally, to your point, chair capability. So how capable do these models need to be? And I think the point I'd like to make to start is not all deployments need the highest level of capability and security and resilience, but I think certainly some do.
And so the UK must have some capacity at the frontier.
16:31
Tanmanjeet Singh Dhesi
-
-
Tanmanjeet Singh Dhesi
Can I pick up that last point. And again the your colleagues might want to come in as well on this. Do we have currently sufficient computing infrastructure, industrial etc. the the the the underlying capability, infrastructure capability to support the level of sovereign capability in AI and quantum that you've just described.
16:31
Speaker 3
-
Speaker 3
My view is that we don't. So we have some state capacity, including the AI research resource and Isambard. Generally, these are focused on academia and research, whereas I think we're seeing a critical dependency across the AI stack being the ability to serve inference. And so serving inference being providing access to powerful models. And I don't think the UK has a significant amount of inference capacity costs here for electricity are much larger than the US.
We have much less capacity in terms of data centers here, running a one gigawatt load in the UK for a year would cost around $800 million. Sorry, in the US would cost about $800 million, whereas here in the UK it would cost $2.5 billion. And so we have you know, we're already, I think, a little handicapped in terms of being able to keep up on capacity inference capacity, which I you know, I can't emphasize enough.
I think inference capacity is critical. And this is this is just for inference, notwithstanding capacity needed to train our own models.
We have much less capacity in terms of data centers here, running a one gigawatt load in the UK for a year would cost around $800 million. Sorry, in the US would cost about $800 million, whereas here in the UK it would cost $2.5 billion. And so we have you know, we're already, I think, a little handicapped in terms of being able to keep up on capacity inference capacity, which I you know, I can't emphasize enough.
I think inference capacity is critical. And this is this is just for inference, notwithstanding capacity needed to train our own models.
16:32
16:33
Speaker 1
-
Speaker 1
Yeah. Well we I mean we come up short across the stack as it's called. Our energy costs are high. High compared to France for example. And the amount of compute we have for training models is limited and for inference is limited. We don't control any of the choke points in that stack, because we don't make the high end computer chips that are needed by these models, and we don't control the Dutch, make the lithographic equipment that goes into making that chip.
So we're scrabbling for some sort of hold on that stack. What could we offer if, as a country, our strategy here was to enmesh ourselves in a broader ecosystem of frontier models? It's not clear that we have a particular locus at the moment.
So we're scrabbling for some sort of hold on that stack. What could we offer if, as a country, our strategy here was to enmesh ourselves in a broader ecosystem of frontier models? It's not clear that we have a particular locus at the moment.
16:33
Speaker 2
-
Speaker 2
Again, given the quantum technologies are a much earlier stage, I think actually we we perhaps have a stronger position within the supply chain and within the the full stack. So indeed global investment into quantum technologies is of course dominated in the US and in China. But the UK plays very strongly as a kind of third player, having had a very mature national programme over the last 12 years. I think a weakness within the supply chain is device fabrication.
So where will the chips come from? Whether they are prototyping chips at this stage or large scale manufacture in due course, most of our clean rooms are at an academic research focus. We do have a particular strength in the photonics industry, particularly across the central belt in Scotland, and that would be a key enabler for all quantum modalities. In the longer term, we've got real strengths in specialist manufacturing, particularly around formula one, which is often a route for small scale fabrication into the quantum ecosystem, cryogenics supply chain.
And then within the software stack, we have particular strengths in quantum software error correction, which is a crucial part of the requirement for quantum computing applications and algorithm, as well as our very strong research base. So I think we have much a much stronger grip on the quantum stack, but that's always fragile and something that we have to maintain the nurturing of.
So where will the chips come from? Whether they are prototyping chips at this stage or large scale manufacture in due course, most of our clean rooms are at an academic research focus. We do have a particular strength in the photonics industry, particularly across the central belt in Scotland, and that would be a key enabler for all quantum modalities. In the longer term, we've got real strengths in specialist manufacturing, particularly around formula one, which is often a route for small scale fabrication into the quantum ecosystem, cryogenics supply chain.
And then within the software stack, we have particular strengths in quantum software error correction, which is a crucial part of the requirement for quantum computing applications and algorithm, as well as our very strong research base. So I think we have much a much stronger grip on the quantum stack, but that's always fragile and something that we have to maintain the nurturing of.
16:35
Liam Byrne
-
-
Liam Byrne
Thanks, chair. We obviously face these dilemmas. In the way that many other middle powers face these dilemmas. And one of the assets we do still have just about is the ability to generate alliances. And so if I think about our partners in the EU or our GCap partners like Japan or our Aukus partner like Australia, or the partnership that is now deepening with South Korea, can you begin to see a way in which almost a middle power stack could be assembled through deeper and better coordinated work with some of our allies? Maybe, Professor Payne, you could kick us off.
16:36
Speaker 1
-
Speaker 1
Yes, I think potentially so. Even if you add up all the venture capital, the investment funds available to those middle powers and role in Singapore and other countries that might be similarly like minded to act together as a counterweight to this bipolar world. You come up short of the amount of venture capital that America has thrown at the the problem, or that America has been able to generate to fund these companies. But I would caveat what my answer to Lord Sedwill as well by saying , that's the architecture as it looks.
Now, it may not be how AI is done in the future. There is research going into world models that may have different computational requirements. There's research going into different chipsets that may do things rather differently from the way modern GPUs do. So I think with all these debates, we need to be careful to be too anchored in where we are now, rather than lifting our eyes a little bit to the future. So that's perhaps a slight reason for optimism, to your point about an alliance of people, I and many others have been making this point for a number of years now.
There has been no progress and. no progress. Well, that's perhaps unfair, but no progress along the lines of developing a middle power frontier capability, let me put it that way. And the countries that have tried to do so have largely done it unilaterally. Germany had a brief punt at it. France had a brief punt at it with Mistral. It's now falling behind the frontier with its latest models. So a collective decision to develop frontier capability.
I don't don't see it being made just yet.
Now, it may not be how AI is done in the future. There is research going into world models that may have different computational requirements. There's research going into different chipsets that may do things rather differently from the way modern GPUs do. So I think with all these debates, we need to be careful to be too anchored in where we are now, rather than lifting our eyes a little bit to the future. So that's perhaps a slight reason for optimism, to your point about an alliance of people, I and many others have been making this point for a number of years now.
There has been no progress and. no progress. Well, that's perhaps unfair, but no progress along the lines of developing a middle power frontier capability, let me put it that way. And the countries that have tried to do so have largely done it unilaterally. Germany had a brief punt at it. France had a brief punt at it with Mistral. It's now falling behind the frontier with its latest models. So a collective decision to develop frontier capability.
I don't don't see it being made just yet.
16:37
16:38
Liam Byrne
-
-
Liam Byrne
When I have this conversation with our allies in countries like Japan, South Korea and elsewhere, they often implore the UK to be more assertive in coming forward and providing some leadership for this kind of initiative. Do you think the UK is providing the right level of leadership for this sort of initiative today?
16:38
16:38
16:38
16:39
Lord Watts
-
-
Lord Watts
You talk about the limitations of the UK, you know, in comparison to the United States. What about the the skills base? Because as I understand it, in the Silicon Valley, engineers are being paid three, £5 million a year, which is a lot of money for anyone to find. And it's just a second issue is about you talk about America, but what about China? Because China seems to be trying to open up its its system, the world where America is restricting them.
What are the lessons to be learned from that?
What are the lessons to be learned from that?
16:39
Speaker 3
-
Speaker 3
So, so yeah, I would, I would say that on expertise and skills, the UK definitely has some strengths. We have some incredible capacity to, to, to develop AI specialists, both in tech technology, technological careers and in sort of governance careers as well. I think one of the challenges to that is once we develop the expertise, often that is then moved into American owned or foreign owned companies. So it's great to have that expertise in the UK.
You know, we have currently much of Google DeepMind's staff here, although my understanding is that that is reducing. We have a very strong AI Security Institute, which does some fantastic work on evaluations. And I think there is a a synergy of being able to have all of those people in the same place. But I think we're missing out at the end of that academic pipeline, where talent often will go to other, other countries in terms of the US and the Chinese approach.
So you're correct that generally the frontier models that the US are developing are what we call closed weight. So the companies that develop the systems, the models, provide access to those themselves. And the models are proprietary, whereas the Chinese Chinese developers are producing lots of incredibly powerful open weight models. And for open weight models, that means anyone in the world can go and download them from the internet. And as long as you have sufficient hardware, use those models.
They are still spending astronomical sums on developing those, and they have a very different set up for doing that. I think to the to the previous point, there was a fantastic article I saw on what would what would need to happen for a middle power frontier AI model. And the costs that were attached to that over four years were half 1 trillion USD, 300 billion of that would be spent on compute alone.
And this article was by a chap called Anton Leicht. So it talks through how much of that you might need to spend on compute talent, etc.. So those are the sorts of figures that we're we're talking about. Not impossible, but fairly large.
You know, we have currently much of Google DeepMind's staff here, although my understanding is that that is reducing. We have a very strong AI Security Institute, which does some fantastic work on evaluations. And I think there is a a synergy of being able to have all of those people in the same place. But I think we're missing out at the end of that academic pipeline, where talent often will go to other, other countries in terms of the US and the Chinese approach.
So you're correct that generally the frontier models that the US are developing are what we call closed weight. So the companies that develop the systems, the models, provide access to those themselves. And the models are proprietary, whereas the Chinese Chinese developers are producing lots of incredibly powerful open weight models. And for open weight models, that means anyone in the world can go and download them from the internet. And as long as you have sufficient hardware, use those models.
They are still spending astronomical sums on developing those, and they have a very different set up for doing that. I think to the to the previous point, there was a fantastic article I saw on what would what would need to happen for a middle power frontier AI model. And the costs that were attached to that over four years were half 1 trillion USD, 300 billion of that would be spent on compute alone.
And this article was by a chap called Anton Leicht. So it talks through how much of that you might need to spend on compute talent, etc.. So those are the sorts of figures that we're we're talking about. Not impossible, but fairly large.
16:41
16:42
16:42
16:42
16:42
Speaker 1
-
Speaker 1
I think they simply haven't raised enough money to afford the hardware that they need to train the models. It's as simple as that. They couldn't. They didn't have as much money to spend. And if I may, I just want to make a very short follow on to George's comment about China, which is perhaps a little unfair, but I'm always reminded when thinking about open weight models, that your first hit from a drug pusher is always free.
And and downstream of that, you may find that there are consequences. And perhaps we can come back to that. That
And and downstream of that, you may find that there are consequences. And perhaps we can come back to that. That
16:42
16:43
Lord Arbuthnot of Edrom
-
-
Lord Arbuthnot of Edrom
Just come in very briefly on the sovereignty thing. There is a book called Ghost Fleet by P.W. Singer and August Cole. That's it. Yeah. Which is based on the principle that many of the chips in our critical defense infrastructure that we rely on are built in countries which do not have our best interests at heart. Is this a serious problem?
16:43
Speaker 3
-
Speaker 3
I would say so. And I think to to Professor Payne's point, the global supply chain for AI hardware is geographically diverse. And I think that this actually is a is a strength I think we're reliant on. There are lots of choke points across the stack. So, you know, you have Dutch lithography, you have chips being assembled and packaged in Taiwan. You have some UK intellectual property in the chips.
You have American chip design. You have, you know, gases being developed that are used in the manufacturing process in the UK. So there's this very geographically diverse stack. And I think that kind of game theoretically means that the risks are reduced a little for those being cut off, because there are quite a lot of dependencies in there. I would say that few of the important dependencies are British at the moment. And so.
You have American chip design. You have, you know, gases being developed that are used in the manufacturing process in the UK. So there's this very geographically diverse stack. And I think that kind of game theoretically means that the risks are reduced a little for those being cut off, because there are quite a lot of dependencies in there. I would say that few of the important dependencies are British at the moment. And so.
16:44
16:44
16:44
Tanmanjeet Singh Dhesi
-
-
Tanmanjeet Singh Dhesi
The point of point of information , actually, in many secure defence locations, they're not permitted inside the inside the wire in case in case, essentially they they operate in exactly that way. And that goes to some brands which we wouldn't naturally think of as Chinese as well, but with a lot of Chinese kit in them. Just a point of information. This. So this point of foreign infrastructure, I think you can take us further into.
That
That
16:45
Baroness Kidron
-
-
Baroness Kidron
question. One took us right across. And we're going to come back now to a bit more. I'd like a little bit more detail on the question of dependence on foreign governments. And I think that, you know, we've heard lots of access, we've heard the security risk, etc., etc. but I think it would be very helpful if you were to say the things that most worry you about the dependence. And maybe we could start with you, Ken, and then George and come to Michael on the other on the quantum side.
16:45
16:46
Tanmanjeet Singh Dhesi
-
-
Tanmanjeet Singh Dhesi
Apologies, order, order. So we're going to break, I'm afraid we'll have to break for a few minutes for a vote, since you'd only just started answering the question. I think we'll just pause it there. We'll we'll then come back to the come back to the question in a moment. So order. Order short recess. After the vote, back to Baroness Kidron, perhaps just ask the question, Baroness Kidron, and then we'll pick it up from there.
16:48
16:49
Speaker 1
-
Speaker 1
Yeah. So. Actually my concern is, is, is less China and more America in our relationship with America. And because they produce the best models at the moment, and you might get into how China conducts some of its research later in these models. But for the moment, they're behind where America is in making these models. And I see that continuing broadly. And there are two areas where I'm concerned from a national security perspective about that reliance on America.
One is cyber capabilities. And in recent days, we've become aware of, I think, just how potent a risk that is going to be with the hugging face hack. By by an open AI model autonomously demonstrating hitherto unknown abilities to escape its sandbox and go off and do its own thing. There's more of that coming our way. The other finding from recent months has been, again, from open AI, and it's solving hitherto unsolved maths problems, persuading leading mathematicians that it's demonstrating creativity beyond what we've seen so far, producing novel findings.
Both of those are of great use in the cyber domain as offensive tools. And I think we're just starting with those two episodes to get an intimation of what might be coming for us. And that's a threat across the whole spectrum of societal activity, not just in defence. The other one is the use of language models as part of situational awareness and targeting systems. You may have heard of Maven Smart System, currently being adopted by NATO and developed in America.
There's a very good book recently on that Project Maven, which I commend to you, and language models have been for the last several years. Embedded in Project Maven. It's not clear from this side of the security line what it is they're doing. Excuse me for the targeting, but they're doing something providing context, I suspect, developing courses of action and so forth. And if you're reliant on models to do that, and if they confer what the military calls fighting power, and I think they do, then it becomes desperately important to have them.
And and we're exposed to our allies in that respect.
One is cyber capabilities. And in recent days, we've become aware of, I think, just how potent a risk that is going to be with the hugging face hack. By by an open AI model autonomously demonstrating hitherto unknown abilities to escape its sandbox and go off and do its own thing. There's more of that coming our way. The other finding from recent months has been, again, from open AI, and it's solving hitherto unsolved maths problems, persuading leading mathematicians that it's demonstrating creativity beyond what we've seen so far, producing novel findings.
Both of those are of great use in the cyber domain as offensive tools. And I think we're just starting with those two episodes to get an intimation of what might be coming for us. And that's a threat across the whole spectrum of societal activity, not just in defence. The other one is the use of language models as part of situational awareness and targeting systems. You may have heard of Maven Smart System, currently being adopted by NATO and developed in America.
There's a very good book recently on that Project Maven, which I commend to you, and language models have been for the last several years. Embedded in Project Maven. It's not clear from this side of the security line what it is they're doing. Excuse me for the targeting, but they're doing something providing context, I suspect, developing courses of action and so forth. And if you're reliant on models to do that, and if they confer what the military calls fighting power, and I think they do, then it becomes desperately important to have them.
And and we're exposed to our allies in that respect.
16:51
Baroness Kidron
-
-
Baroness Kidron
Just on your point about the on your first point, you say more of that coming our way. I'm just an. Absolutely. I agree with you, but I'm interested to have your thoughts on what we should do about that. What is what are the moves that you would make in this circumstance, particularly around open AI? But but more generally? Well,
16:51
Speaker 1
-
Speaker 1
I think I think two things since I've said we're not going to create our own. One is to enmesh ourselves as best we can in that process. It may be expertise, it may be another part of the stack, but we better find one and or more than one and develop it. And the other is to develop resilience. We need to find a way of protecting ourselves against these sorts of autonomous cyber agents, and that is extremely challenging.
Both those things are extremely challenging to think of. The you'll have seen, perhaps the reports getting some coverage today from OpenAI's chief scientist. I think it's called something like an alien intelligence. Now, OpenAI has an IPO coming and an interest in talking up its product, but I don't think he does personally. And the quotes are fairly sobering from him. We will see machines that are meaningfully smarter than us in our lifetime, and I think that's conservative.
No one is prepared, he says. And this is a time that calls for extreme caution. And I think that's that's where we are and our options. I wish I had the answer and said, we need to do this, that and the other. But our options are somewhat limited. But the threat is in the cyber domain, principally.
Both those things are extremely challenging to think of. The you'll have seen, perhaps the reports getting some coverage today from OpenAI's chief scientist. I think it's called something like an alien intelligence. Now, OpenAI has an IPO coming and an interest in talking up its product, but I don't think he does personally. And the quotes are fairly sobering from him. We will see machines that are meaningfully smarter than us in our lifetime, and I think that's conservative.
No one is prepared, he says. And this is a time that calls for extreme caution. And I think that's that's where we are and our options. I wish I had the answer and said, we need to do this, that and the other. But our options are somewhat limited. But the threat is in the cyber domain, principally.
16:53
Speaker 3
-
Speaker 3
So I would agree with with Professor Payne that my nearest term concern is cyber security. We're seeing the ability for the frontier models to to have sort of incredible capability in cyber and in an offensive and a defensive way. We've seen some of the offensive way in through some of the loss of control incidents that have happened over the last few months. I think we need to reorient the conversation around how we can use these defensively as well.
The challenge with using a frontier American model defensively is off the shelf. It will it will refuse to do much cyber activity because it would kind of breach the safeguards that exist on that. So you're already looking to have, you know, you would want to have direct relationships with the company so that you could have models which had some of their safeguards removed so that they could be optimally useful. The mitigation to that might be using Chinese models to defend our critical national infrastructure, which I think there will be an understandable squeamishness about, but it might be a more resilient option than relying on models which could be pulled from from servers, and we wouldn't be able to have access to.
So I think my my emphasis on cyber is that we need to have a diverse approach, which includes working with strong partners and also some things that might seem quite unpalatable to us at the moment, using Chinese models to help us in that. The other thing I would say I'm significantly concerned about in the more medium to longer term. And when I say medium to longer term in AI, I don't mean I kind of mean months. Not not years is recursive self-improvement.
And so I think the frontier AI companies are again, you know, you have to contextualize what the messages that they put out with the situations that they are in, but they are beginning to talk about having automated their research. So the research that leads to improvements in AI capability, large parts of that are now undertaken by AI agents , and humans are monitoring that. That, to me , seems like an incredibly concerning situation and one that will need strong governance efforts to to counter.
The challenge with using a frontier American model defensively is off the shelf. It will it will refuse to do much cyber activity because it would kind of breach the safeguards that exist on that. So you're already looking to have, you know, you would want to have direct relationships with the company so that you could have models which had some of their safeguards removed so that they could be optimally useful. The mitigation to that might be using Chinese models to defend our critical national infrastructure, which I think there will be an understandable squeamishness about, but it might be a more resilient option than relying on models which could be pulled from from servers, and we wouldn't be able to have access to.
So I think my my emphasis on cyber is that we need to have a diverse approach, which includes working with strong partners and also some things that might seem quite unpalatable to us at the moment, using Chinese models to help us in that. The other thing I would say I'm significantly concerned about in the more medium to longer term. And when I say medium to longer term in AI, I don't mean I kind of mean months. Not not years is recursive self-improvement.
And so I think the frontier AI companies are again, you know, you have to contextualize what the messages that they put out with the situations that they are in, but they are beginning to talk about having automated their research. So the research that leads to improvements in AI capability, large parts of that are now undertaken by AI agents , and humans are monitoring that. That, to me , seems like an incredibly concerning situation and one that will need strong governance efforts to to counter.
16:55
16:55
Speaker 2
-
Speaker 2
Yes. So I think in quantum computing, again, the the development cycle is in a different place, but I think in terms of overseas dependencies and interdependencies, the number one concern I would have is investment. And so for patient capital in the UK and in Europe more generally, will it always simply never be able to keep up with the level of capitalization available in US markets? And I think one of the things that is acting as a significant driver, there are some of these international programs.
So Euro, HPC, the UK procure program, or maybe say more about that in a moment or two. And the US, DARPA Qbi program. So all of these are set to verify and validate performance of early stage prototype quantum computers . And the DARPA Qbi program has an investment ticket of $300 million per company if you qualify through to stage C, and that's proving to be a very significant lever for companies to both get the validation for their end users, which drives commercial success, and also to anchor themselves into into the US, driven by US government investment.
And the US Chips act has also made $2 billion available very recently, a partly to solve technical issues, but also partly to drive acquisition. And that certainly starts to put some of the European innovative startups at risk. So I think investment is a significant issue. From a more technical perspective, I think some of the critical materials, minerals and device fabrication are at risk for the wider European quantum technologies, not just quantum computing developments.
So materials like helium three, like cesium, like strontium, like rubidium, they're all absolutely at the core of these quantum technologies. And whilst there's an open market for these today, even if there are bottlenecks on supply, that doesn't mean it stays like that forevermore.
So Euro, HPC, the UK procure program, or maybe say more about that in a moment or two. And the US, DARPA Qbi program. So all of these are set to verify and validate performance of early stage prototype quantum computers . And the DARPA Qbi program has an investment ticket of $300 million per company if you qualify through to stage C, and that's proving to be a very significant lever for companies to both get the validation for their end users, which drives commercial success, and also to anchor themselves into into the US, driven by US government investment.
And the US Chips act has also made $2 billion available very recently, a partly to solve technical issues, but also partly to drive acquisition. And that certainly starts to put some of the European innovative startups at risk. So I think investment is a significant issue. From a more technical perspective, I think some of the critical materials, minerals and device fabrication are at risk for the wider European quantum technologies, not just quantum computing developments.
So materials like helium three, like cesium, like strontium, like rubidium, they're all absolutely at the core of these quantum technologies. And whilst there's an open market for these today, even if there are bottlenecks on supply, that doesn't mean it stays like that forevermore.
16:57
Baroness Kidron
-
-
Baroness Kidron
So what I'm hearing from all of you is we're way behind and we haven't got there's not enough money in Europe to invest to catch up. That's, that's the, the top line message. And obviously, you know, the detail is important, but I'm interested to hear from you where what a whether you think that Europe as a market has some value to America, to China, that could be a negotiating point at the moment. I think that that we haven't explored what that means access to market.
And does that does that give us something? And also maybe thinking more creatively, you know, where are potential choke points that you can see? However, you know how however nascent in a sense, because otherwise we could just sort of pack up for home. So, so maybe you could each just say a little about, about, about that thought.
And does that does that give us something? And also maybe thinking more creatively, you know, where are potential choke points that you can see? However, you know how however nascent in a sense, because otherwise we could just sort of pack up for home. So, so maybe you could each just say a little about, about, about that thought.
16:58
16:59
Speaker 3
-
Speaker 3
I mean, I think that is strong leverage. Fundamentally, these these closed weight frontier AI companies need to make money. And the EU and, you know, outside of the US is a major market for that. The EU has been fairly forward leaning on governance efforts. And we'll probably come on to talk a bit about this. But they have a strong piece of legislation, the EU AI act, which compels developers to do a set of things which I think contribute quite strongly to safety and security of AI systems.
And so far we've seen US companies actually seem quite happy with abiding by those. There was a code of practice that was developed with respect to the AI act, and it was signed by the vast majority of major providers in the US. And so I think there is an appetite to, to to have products on sale services on sale in the EU and beyond and the UK. And so I think banding together to some extent does provide leverage there.
And so far we've seen US companies actually seem quite happy with abiding by those. There was a code of practice that was developed with respect to the AI act, and it was signed by the vast majority of major providers in the US. And so I think there is an appetite to, to to have products on sale services on sale in the EU and beyond and the UK. And so I think banding together to some extent does provide leverage there.
17:00
17:00
Speaker 2
-
Speaker 2
Yeah. So I think for sure the market pool of the likes of pharmaceuticals, advanced materials, financial services, telecoms, so some of those significant sectors in Europe provide a pool to overseas companies from from outside Europe. And I think I'd add a, the what if quantum computers don't scale.
17:00
17:00
Speaker 2
-
Speaker 2
So what if the next ten years, as the industry roadmaps are project don't materialize, Europe has already a significant partnerships and real strengths around algorithm development and therefore, I think challenging ourselves on what can we do with the modest scale quantum computer through algorithm enhancement, rather than assuming it's brute force to just make a larger and larger scale machine?
17:01
17:01
Speaker 1
-
Speaker 1
I would like to hope so. And George is right. The EU AI act is a very comprehensive, very EU like, if I may say , piece of legislation, but I've not yet seen anything that I could point to concretely and say. The EU's position has produced a change in behaviour in the frontier labs in the States. There was a very brief period where open AI introduced a new feature called Advanced Voice, probably familiar with it.
You can talk to the AI using your voice, and there was a view that this violated the provisions of the EU AI act, insofar as it prohibits affective computing. That is, computers that can recognise tone of voice and so on in the user, not just the words. And so open AI didn't roll out that feature in in the EU for a few weeks. So perhaps that is an example, but it only detained them for a few weeks before somebody decided that they were going to press on with it anyway.
It's not it's not. The EU is obviously an important economic actor, and these companies need to make money, but they're still raising money like billions from venture capital at the moment. So it's not clear that they're desperate to toe the line of what the EU says at this point. And I'm not confident that we'll arrive at that point any time soon.
You can talk to the AI using your voice, and there was a view that this violated the provisions of the EU AI act, insofar as it prohibits affective computing. That is, computers that can recognise tone of voice and so on in the user, not just the words. And so open AI didn't roll out that feature in in the EU for a few weeks. So perhaps that is an example, but it only detained them for a few weeks before somebody decided that they were going to press on with it anyway.
It's not it's not. The EU is obviously an important economic actor, and these companies need to make money, but they're still raising money like billions from venture capital at the moment. So it's not clear that they're desperate to toe the line of what the EU says at this point. And I'm not confident that we'll arrive at that point any time soon.
17:02
17:02
Liam Byrne
-
-
Liam Byrne
Thanks, chair. Just very briefly, the hyperscalers are obviously running into kind of significant problems in the United States, getting data centers built. And the EU is a €10 trillion consumer market. How important is sustained access to the American hyperscalers, and is that likely to have an impact on their appetite for complying with the future of standards in the UK?
17:03
Speaker 3
-
Speaker 3
I could talk a bit to the the capacity on inference. As I mentioned earlier, so much of the capacity that we have for inference in the UK and by this I mean the ability to, to to use frontier AI models, whether they're closed source or open source is dependent on US hyperscalers, albeit in region. So they have data centers in Europe and the UK and elsewhere, but they are still owned by US companies. And so they still jurisdictionally are reachable by and accountable to the US.
And so I think going back to the situation that we had earlier this year with with fable and mythos, for example, if a company was compelled to stop serving a model for specific reasons by the US government, it wouldn't help us having a non-sovereign hyperscaler serving that to us. And I do have another concern about the Chinese market as well, which is that there has been discussion in the US on adding Chinese open weight providers to the entity list, and which would which would mean US companies would have stopped serving their models, which would include US companies operating in Europe and and the UK.
Even EU sovereign inference providers would have strong pressure on them to stop hosting Chinese models if they had strong exposure to the US as well.
And so I think going back to the situation that we had earlier this year with with fable and mythos, for example, if a company was compelled to stop serving a model for specific reasons by the US government, it wouldn't help us having a non-sovereign hyperscaler serving that to us. And I do have another concern about the Chinese market as well, which is that there has been discussion in the US on adding Chinese open weight providers to the entity list, and which would which would mean US companies would have stopped serving their models, which would include US companies operating in Europe and and the UK.
Even EU sovereign inference providers would have strong pressure on them to stop hosting Chinese models if they had strong exposure to the US as well.
17:04
Speaker 1
-
Speaker 1
My. My sense from personal interaction with Americans who are what the newspapers say, familiar with the the decision on the mythos withdrawal was sobering when when a colleague of mine put to them that this put the EU a disadvantage relative to American companies. The answer was extremely blunt and pugnacious and. And I did get a sense of naked power in that exchange. Now it's an exchange. Read into it what you will.
Maybe the EU will come out ahead in that test of will in shaping American frontier companies. But it didn't seem that way to me from that conversation.
Maybe the EU will come out ahead in that test of will in shaping American frontier companies. But it didn't seem that way to me from that conversation.
17:05
17:05
17:05
17:05
17:05
17:05
Lord Watts
-
-
Lord Watts
The American model is a closed one. Chinese an open one. What's the implications? China spreading its wings around the world. Because if it if it can offer something for for free compared to the American construct constraints, is it likely that over a period of time, China would enjoy not just in the UK but across the world, not one that's in there. I know that behind America at the moment, but they're catching up very quickly.
What's your view about the. Because I think that's a tactic of the Chinese to actually get it out there and make it free. Reminds me of videotapes better. You know, if you think of the best recorders where Betamax. But what happened is that the. They , they spread their wings over there and everyone bought them into it. So it's not necessarily the question of whether it's better than the Americans.
If it's if they flood the market, then it. I would have thought it puts them in a very strong position.
What's your view about the. Because I think that's a tactic of the Chinese to actually get it out there and make it free. Reminds me of videotapes better. You know, if you think of the best recorders where Betamax. But what happened is that the. They , they spread their wings over there and everyone bought them into it. So it's not necessarily the question of whether it's better than the Americans.
If it's if they flood the market, then it. I would have thought it puts them in a very strong position.
17:06
Speaker 3
-
Speaker 3
Absolutely. I think I think, you know, that's a really good point there in that if a model is good enough for a certain task, then the market will will determine or, you know, individuals who need to use that task will use the cheapest models that they can to, to do that. And I think a really important point here is that Chinese open weight models are much cheaper than the US frontier. They are also less capable.
So the estimates are put between 4 and 6 months behind. But I think to to forecast that forwards the model releases we're seeing today from the American companies like GPT six, Astro and fable 5.1, we will likely see open weight models with the same capability before the end of the year. And so I think it's a very intentional thing that is happening, where it's a kind of undercutting in finance in kind of cost, but a slight cut in capability.
And I think for many cases, and this is to my point on, I think we should be less squeamish around using Chinese models for many cases. I think if the capability is good enough and you're able to have sovereignty of that capability because you have custody of the model itself, that might put you in a stronger position than being reliant on a more strong model that you can't guarantee access to.
So the estimates are put between 4 and 6 months behind. But I think to to forecast that forwards the model releases we're seeing today from the American companies like GPT six, Astro and fable 5.1, we will likely see open weight models with the same capability before the end of the year. And so I think it's a very intentional thing that is happening, where it's a kind of undercutting in finance in kind of cost, but a slight cut in capability.
And I think for many cases, and this is to my point on, I think we should be less squeamish around using Chinese models for many cases. I think if the capability is good enough and you're able to have sovereignty of that capability because you have custody of the model itself, that might put you in a stronger position than being reliant on a more strong model that you can't guarantee access to.
17:08
Speaker 1
-
Speaker 1
Yeah. If I may just quickly add to that, for many use cases, that would be fine. And you'll own the the rack of machines on which the model sits. And you can, you can take some reassurance from that. But there are some use cases where marginal advantage in the model performance is going to be critical. And offensive versus defensive cyber is is one such domain. And there you would want to be at the bleeding edge.
And if your access to the bleeding edge depends on Chinese goodwill, then you might find yourself in something of a an exposed position. But as I say, many for many use cases, the Chinese model is really good. Thomson Reuters The news organization has custom trained, which is another thing you can do with these open models. You don't just take them as they are. You can retrain what the weights do as retrained a large Chinese model for its own purposes.
And I'm sure that's absolutely fine as a use case.
And if your access to the bleeding edge depends on Chinese goodwill, then you might find yourself in something of a an exposed position. But as I say, many for many use cases, the Chinese model is really good. Thomson Reuters The news organization has custom trained, which is another thing you can do with these open models. You don't just take them as they are. You can retrain what the weights do as retrained a large Chinese model for its own purposes.
And I'm sure that's absolutely fine as a use case.
17:09
Baroness Kidron
-
-
Baroness Kidron
Very quickly. Just I think this is a very interesting part of the conversation that actually, you know, may be right. Some some tasks need the frontier, but so many tasks don't. And I think that that's not something publicly understood or even understood around policy circles, just that would it make us more robust, less dependent if we had a flourishing of smaller like the Thomson Reuters, which is fantastic.
You know, that many of our tasks, because we're talking about resilience across society, not simply trying to keep the hackers out.
You know, that many of our tasks, because we're talking about resilience across society, not simply trying to keep the hackers out.
17:09
Speaker 3
-
Speaker 3
And I think there was a there was an exemplar point on this with the open AI hugging face incident, which was that. So an open AI, very powerful model tried to hack into succeeded in hacking into hugging face. The company hugging face had to use Chinese models to defend themselves because they couldn't use the American models because they refused to to, you know, it, it hit their safety filters. And so you had a very clear case there where using a Chinese model was better than using none at all.
17:10
17:10
17:10
Lord Arbuthnot of Edrom
-
-
Lord Arbuthnot of Edrom
I declare my interest as chair of the Talis UK Advisory Board. To a certain extent, we've covered the issue of talents and skills. If you were asked each of you to put three headline points as to how we can best ensure that we have the talents and skills for the development of AI and quantum A in this country. Be in academia. What would your three headline points be? Shall we start.
Doctor Cuthbert, you mentioned DARPA and its incentives of $300 million per company. Do we have the incentives there?
Doctor Cuthbert, you mentioned DARPA and its incentives of $300 million per company. Do we have the incentives there?
17:11
Speaker 2
-
Speaker 2
So it's not directly related to talent and skills. But the UK a kind of equivalent program in anchoring UK companies here and driving inward investment is called procure. So this is a three stage government procurement exercise, first stage of R&D. The second stage of scaling. The third stage large scale infrastructure procurement into the 2030s. That will drive employment and skills and skills attraction into the UK, as well as attracting overseas companies.
So that's that's certainly very favorable. The challenge is where are these people coming from? Because the pipeline through the traditional academic PhD route, which is typically where people gain their experience, is on a 3 to 4 year cycle. We already have doctoral training centres, and it's tens of people PhDs per year, not the hundreds per year that that the industry needs. My estimate is there's about 2500 people across quantum technologies employed in the UK, both in academia and in industry today.
And our estimates in the nkcc is that there. Well, there's currently 140 jobs advertised today, and I expect to see about 200 people employed over the next 12 months and 500 over the next 24 months. So it doesn't sound like large numbers, but relative to the entire industry as we stand. And if you project that forward ten years, the entire industry needs to grow by a factor of ten as a minimum in its kind of level, level employment.
So that pipeline of skills is critical, I think is a mistake to see them only coming from the traditional route of quantum physics rather than other scientific disciplines. Engineering in particular industry will scale on the back of engineering computing science for sure, of which of course, across telecoms, across AI. There's also significant pressure for the same talent pool. And then how do we retrain and reskill people from other disciplines who already have perhaps ten, 15, 20 years of industry expertise to give them a level of.
Familiarity with quantum technologies? That means they can really contribute from the systems engineering. Perhaps some of the electronic disciplines rather than, as I say, view. It's only people with a PhD in quantum physics that can contribute. That's simply not the case. And I think we've got to knock down some of that mythology around the word quantum. Just say this is actually a new paradigm in computing, and it's really exciting how to drive those stem Stem skills into the discipline.
So that's that's certainly very favorable. The challenge is where are these people coming from? Because the pipeline through the traditional academic PhD route, which is typically where people gain their experience, is on a 3 to 4 year cycle. We already have doctoral training centres, and it's tens of people PhDs per year, not the hundreds per year that that the industry needs. My estimate is there's about 2500 people across quantum technologies employed in the UK, both in academia and in industry today.
And our estimates in the nkcc is that there. Well, there's currently 140 jobs advertised today, and I expect to see about 200 people employed over the next 12 months and 500 over the next 24 months. So it doesn't sound like large numbers, but relative to the entire industry as we stand. And if you project that forward ten years, the entire industry needs to grow by a factor of ten as a minimum in its kind of level, level employment.
So that pipeline of skills is critical, I think is a mistake to see them only coming from the traditional route of quantum physics rather than other scientific disciplines. Engineering in particular industry will scale on the back of engineering computing science for sure, of which of course, across telecoms, across AI. There's also significant pressure for the same talent pool. And then how do we retrain and reskill people from other disciplines who already have perhaps ten, 15, 20 years of industry expertise to give them a level of.
Familiarity with quantum technologies? That means they can really contribute from the systems engineering. Perhaps some of the electronic disciplines rather than, as I say, view. It's only people with a PhD in quantum physics that can contribute. That's simply not the case. And I think we've got to knock down some of that mythology around the word quantum. Just say this is actually a new paradigm in computing, and it's really exciting how to drive those stem Stem skills into the discipline.
17:14
Speaker 1
-
Speaker 1
Thank you. I'm not sure I have anything desperately profound here, and I certainly wouldn't add to your germane point about salaries. I don't think we're going to be able to do much about the salaries on offer in the Bay area. They are nuts. The three things that I wrote down, we need to find a way of strengthening the connection between the Frontier Labs and UK universities, the direction of travel, perhaps the salaries are.
A part of that has been to Hoover talent from the faculty into the labs. We need to find a way of keeping that generation of research scientists that are at the frontier, engaged with the next generation of students coming through, and 100% sure how you do that. Stem is wildly popular with with 1818 year olds and people going on to do their postgraduate education, the university sector in the country is not in robust health, however, and that's one part of the problem.
The other thing I wrote down about skills and talent is that we want an area where we are doing a good job of educating computer scientists is in educating the next generation of Chinese computer scientists. We have a lot of Chinese international students in the country, which is something that we might want to think about more strategically. The two interests obviously pulling in different directions, higher education and and national security. And then the final thing on skills and talent, and I was struck by this with my work with the defence committee, is that public institutions whose job is to shape this discussion and scrutinise it are in need of technical assistance.
And I think there is a skills gap to provide advice to bodies like yours to Parliament more broadly to to organisations of AI governance that may not yet exist but come into existence. We have to find a way of providing appropriate technical support for these for these sorts of discussions.
A part of that has been to Hoover talent from the faculty into the labs. We need to find a way of keeping that generation of research scientists that are at the frontier, engaged with the next generation of students coming through, and 100% sure how you do that. Stem is wildly popular with with 1818 year olds and people going on to do their postgraduate education, the university sector in the country is not in robust health, however, and that's one part of the problem.
The other thing I wrote down about skills and talent is that we want an area where we are doing a good job of educating computer scientists is in educating the next generation of Chinese computer scientists. We have a lot of Chinese international students in the country, which is something that we might want to think about more strategically. The two interests obviously pulling in different directions, higher education and and national security. And then the final thing on skills and talent, and I was struck by this with my work with the defence committee, is that public institutions whose job is to shape this discussion and scrutinise it are in need of technical assistance.
And I think there is a skills gap to provide advice to bodies like yours to Parliament more broadly to to organisations of AI governance that may not yet exist but come into existence. We have to find a way of providing appropriate technical support for these for these sorts of discussions.
17:16
17:16
17:16
Speaker 3
-
Speaker 3
So my you asked for three points, I think I have three. My first is to consider extending the AC model. So the AI Security Institute model in AC. I think we've seen the creation of a very successful organization that is globally relevant at the frontier. They have world leading expertise on the ability to evaluate AI models. I think it's worth having a look at how they manage to do that.
And they were successful doing that, and then possibly trying to copy that model for the national security AI community as well, which I think we need much more, much more kind of talent and resources in. The second is to emphasize things that money can't buy. And I think there are some, depending on the motivations of the individual. And I think two important things. There are access. So people like working for AC because it gives them the ability to actually work at the frontier, working on models before they've even been released, which very few people in the world do.
A handful of AI safety organisations based in the Bay area , and a few AI security institutes across the world. And the second one is mission. So, you know, lots of people are very mission aligned and would sacrifice larger salaries to be able to contribute to, you know, the mission of protecting and safeguarding the UK. And the final is thinking about. So we need enough talent, but enough talent for what? I don't think, you know, my own view is that I don't think we need to be hiring the sorts of researchers that would have 7 or 8 figure compensations to help us train AI models, because I don't think that's something that's tractable for us to do.
But I think rather, we need people who can take a model and then implement it, integrate it into systems, people who we can put through vetting, who can do this within the national security context. And I think finally, importantly on that point, we also need much more informed leadership and middle management across our institutions, and particularly the civil service, so that the the clarity and the mission around using AI is emphasized such that people think, I want to work in AI, I could either go and work for Google DeepMind, or I could go and work for Gchq.
And I think emphasizing that is very important.
And they were successful doing that, and then possibly trying to copy that model for the national security AI community as well, which I think we need much more, much more kind of talent and resources in. The second is to emphasize things that money can't buy. And I think there are some, depending on the motivations of the individual. And I think two important things. There are access. So people like working for AC because it gives them the ability to actually work at the frontier, working on models before they've even been released, which very few people in the world do.
A handful of AI safety organisations based in the Bay area , and a few AI security institutes across the world. And the second one is mission. So, you know, lots of people are very mission aligned and would sacrifice larger salaries to be able to contribute to, you know, the mission of protecting and safeguarding the UK. And the final is thinking about. So we need enough talent, but enough talent for what? I don't think, you know, my own view is that I don't think we need to be hiring the sorts of researchers that would have 7 or 8 figure compensations to help us train AI models, because I don't think that's something that's tractable for us to do.
But I think rather, we need people who can take a model and then implement it, integrate it into systems, people who we can put through vetting, who can do this within the national security context. And I think finally, importantly on that point, we also need much more informed leadership and middle management across our institutions, and particularly the civil service, so that the the clarity and the mission around using AI is emphasized such that people think, I want to work in AI, I could either go and work for Google DeepMind, or I could go and work for Gchq.
And I think emphasizing that is very important.
17:18
17:19
17:19
17:19
Lord Tunnicliffe
-
-
Lord Tunnicliffe
I've got sort of three questions, I think, that you might answer in general, are the existing international institutions capable of managing AI related security risks? We've had the UN Global Dialogue on AI governance. It's met apparently in July 26th, and it's not going to meet until 27. Are such institutions capable of the right pace, and should there be international agreements governing certain applications of quantum technologies? If you look at the last thousand years of international agreement, they don't have much of a record, frankly.
I mean, they usually break down pretty quickly. They only seem to work when nations accept that trespassing outside an agreement is going to be painful, and it's going to be equally painful so that your. Legitimizing , codifying the status quo. Now, the status quo, as far as I can see, don't know much about this subject, but it seems to be all over the place at the moment because of the rapid development, is there any chance of genuine international agreement, of agreements that people will obey, even where for at least part of the time, one nation feels at a disadvantage?
I mean, they usually break down pretty quickly. They only seem to work when nations accept that trespassing outside an agreement is going to be painful, and it's going to be equally painful so that your. Legitimizing , codifying the status quo. Now, the status quo, as far as I can see, don't know much about this subject, but it seems to be all over the place at the moment because of the rapid development, is there any chance of genuine international agreement, of agreements that people will obey, even where for at least part of the time, one nation feels at a disadvantage?
17:21
17:21
Speaker 1
-
Speaker 1
I read in. So as I said in my opening remarks, I was a global commissioner of this intergovernmental process on AI in the military domain. And what's interesting is that that conversation, which started internationally back at the at the UN in the early part of the last decade, was rather separate from the Bletchley process on frontier stuff. And now they've essentially merged into the two, or rather, the Bletchley process has assumed a large amount of the military discussions with both those discussions, I felt then and and do now that there was very limited prospect of binding international agreement because, well, for a variety of reasons, but principally the security dilemma and the two countries who could make meaningful agreement have no interest in doing so.
And most of the noise about agreement comes from countries who have no real prospect of achieving cutting edge AI and feel themselves to be increasingly exposed to the consequences of not having cutting edge AI, especially as offensive cyber benefits from it, and also as fighting power more broadly benefits from it. Legacy military suddenly look a bit clunky, which we saw in the American action with respect to Iran recently, saying nothing about the strategic wisdom or otherwise of that, that campaign, what you saw was a demonstration at the start of what AI facilitated military activity can achieve.
And for that reason, countries that can't do it are worried, and countries that can do it have no appetite to stop doing it. And I don't see any reason why that is going to change. The statements that came out of re aim, the statements that come out of the Convention on Conventional Weapons or out of the UN General Assembly more broadly, as part of this dialogue, are exceptionally banal, which is the only way you can get all parties to sign up to them.
So I'm a skeptic.
And most of the noise about agreement comes from countries who have no real prospect of achieving cutting edge AI and feel themselves to be increasingly exposed to the consequences of not having cutting edge AI, especially as offensive cyber benefits from it, and also as fighting power more broadly benefits from it. Legacy military suddenly look a bit clunky, which we saw in the American action with respect to Iran recently, saying nothing about the strategic wisdom or otherwise of that, that campaign, what you saw was a demonstration at the start of what AI facilitated military activity can achieve.
And for that reason, countries that can't do it are worried, and countries that can do it have no appetite to stop doing it. And I don't see any reason why that is going to change. The statements that came out of re aim, the statements that come out of the Convention on Conventional Weapons or out of the UN General Assembly more broadly, as part of this dialogue, are exceptionally banal, which is the only way you can get all parties to sign up to them.
So I'm a skeptic.
17:23
Speaker 3
-
Speaker 3
Yeah, I think I would I would make a distinction here between international agreements and national progress on governance. And I think international agreements to start with are are not particularly well developed at the moment. To Professor Payne's point, I don't think sufficient incentives exist. I think there's a race going on at the moment between the US and China in terms of capability. I think there are some concerns about safety, which both countries share, particularly maybe those around loss of control, given what we've seen over the last few months.
But I don't think those are yet strong enough to bring both parties to the table and agree a an international governance regime, which would also require technology that doesn't exist yet. For example, things like inference, only data servers. So sorry, data centers. So data centers that cannot train models but can only serve them. And we do have some expertise in this country building such hardware governance. Particularly there's a company called Edmodo Design, which does some great work in this area on national governance.
I think there is good progress taking place. It's in the US, I would say predominantly there are there's movement at the state and the federal level. At the state level, there's a few bills being passed and that have already passed that mandate. Certain safety requirements for the developers. And at the federal level, there is a bill that's been introduced to Congress , which would mandate things like independent regular audits or compliance audits.
And then on the other side of the Atlantic, we have the EU AI act, which we've mentioned, which I think is a good governance regime. I think one of the things I would note to to Professor Payne's point is that anthropic introduced watermarking onto their systems. My understanding is that was directly relevant to the AI, the EU AI act. So that's an example of a US company making some changes to their technology for compliance with a with a European standard.
And the final thing I'll say on this is that I think, you know, the UK does have a position of some strength in this with respect to its evaluation capacity and translating that into standards and better governance for AI seems to me like an opportunity for the UK to play a big part.
But I don't think those are yet strong enough to bring both parties to the table and agree a an international governance regime, which would also require technology that doesn't exist yet. For example, things like inference, only data servers. So sorry, data centers. So data centers that cannot train models but can only serve them. And we do have some expertise in this country building such hardware governance. Particularly there's a company called Edmodo Design, which does some great work in this area on national governance.
I think there is good progress taking place. It's in the US, I would say predominantly there are there's movement at the state and the federal level. At the state level, there's a few bills being passed and that have already passed that mandate. Certain safety requirements for the developers. And at the federal level, there is a bill that's been introduced to Congress , which would mandate things like independent regular audits or compliance audits.
And then on the other side of the Atlantic, we have the EU AI act, which we've mentioned, which I think is a good governance regime. I think one of the things I would note to to Professor Payne's point is that anthropic introduced watermarking onto their systems. My understanding is that was directly relevant to the AI, the EU AI act. So that's an example of a US company making some changes to their technology for compliance with a with a European standard.
And the final thing I'll say on this is that I think, you know, the UK does have a position of some strength in this with respect to its evaluation capacity and translating that into standards and better governance for AI seems to me like an opportunity for the UK to play a big part.
17:25
17:25
Speaker 2
-
Speaker 2
I think the the first thing I would say in terms of international cooperation, the area of standards is quite a rich area at the moment, and certainly a lot of dialogue, particularly around the the G7 countries with some other international members. There is a specific aspect of quantum computing. Think of verification of the hardware as it develops. And if I can refer to something beyond what's possible through emulation.
So the scale of computation that we can run a classical computer and demonstrate, we get the correct answer in kind of simple terms. And if we're thinking about simulating chemical species or simulating a complex system, is there a physical environment we can compare it to so that we know when we've run a complex
So the scale of computation that we can run a classical computer and demonstrate, we get the correct answer in kind of simple terms. And if we're thinking about simulating chemical species or simulating a complex system, is there a physical environment we can compare it to so that we know when we've run a complex
17:27
17:27
17:27
17:27
17:27
17:27
17:27
17:27
17:27
17:27
17:27
17:27
17:27
17:27
17:27
17:27
17:27
Speaker 1
-
Speaker 1
Which of course is a is a key aspect of the kind of threat landscape from quantum computing that sits alongside the opportunity landscape in terms of all the applications across. many industrial sectors. I think there is a concern more widely that if somebody had a crypto relevant machine, would any of us know? And really, that's where a very tight international. collaboration between like minded countries is, is absolutely vital, such that as algorithm.
breakthroughs are made, or as the technology scales to large scale crypto relevant.
breakthroughs are made, or as the technology scales to large scale crypto relevant.
17:28
17:28
17:29
17:29
17:29
17:29
Baroness Kidron
-
-
Baroness Kidron
17:29
17:29
17:29
17:29
17:30
Liam Byrne
-
-
Liam Byrne
crypto. And Liam Byrne is going to lead us into that. Thank you chair. So you've seen the news today. Jaguar Land Rover has announced 4000 redundancies. And they ascribe some of the costs that they're having to ameliorate to the cost of that cyber attack last year, that probably cost about 260 million in direct cost across the wider economy, about 1.9 billion five weeks worth of production shut down.
And it was one of several attacks last year that included the attacks on Marks and Spencer's and the co-op. So I'm interested in how you think that the balance of power is going to shift between cyber criminals and UK plc. Professor, do you want to kick off? You want to give us a sense of how you think AI related threats, and then quantum related threats could actually fundamentally change the balance of terror .
And it was one of several attacks last year that included the attacks on Marks and Spencer's and the co-op. So I'm interested in how you think that the balance of power is going to shift between cyber criminals and UK plc. Professor, do you want to kick off? You want to give us a sense of how you think AI related threats, and then quantum related threats could actually fundamentally change the balance of terror .
17:31
17:31
17:31
17:31
17:31
17:39
17:39
17:39
Lord Boateng
-
-
Lord Boateng
Criminals and UK plc might be about to change fundamentally. Yeah. And I think the Hugging Face episode is a good early warning sign of where we might be going. Those closed models have, in my limited experience, pretty good. guardrails and a lot of resources are going into improving those guardrails to stop people being able to use them to do nefarious activities. But no, guardrail is perfect, and open weight models, which track the frontier, as we've discussed, may come without such robust guardrails.
So I suspect we will arrive at the time in the not too distant future, when models that are capable of offensive cyber. capability are available for relatively low cost. And so I see that threat not going away. And if anything, I see the, the, the, the threat that you outlined so soberly in your question increasing as time goes on. And to my. mind and again in your closed sessions, you might find a more informed answer than this.
The best defence to that is to have frontier models in the defensive cyber role, which again, is another plank of our . our .
So I suspect we will arrive at the time in the not too distant future, when models that are capable of offensive cyber. capability are available for relatively low cost. And so I see that threat not going away. And if anything, I see the, the, the, the threat that you outlined so soberly in your question increasing as time goes on. And to my. mind and again in your closed sessions, you might find a more informed answer than this.
The best defence to that is to have frontier models in the defensive cyber role, which again, is another plank of our . our .
17:41
17:41
Speaker 1
-
Speaker 1
the context of this risk? So I see that there's two aspects of cyber. security for quantum or related quantum. One is the classical cyber security. associated with. prototype and small scale machines. So there are household names, some of the IT majors who are pursuing quantum computing. There are also lots of startups. So one of the things that that National Quantum Computing Centre has.
observed in establishing our testbed program, where we have seven commercial. platforms under one one roof of different.
observed in establishing our testbed program, where we have seven commercial. platforms under one one roof of different.
17:41
17:41
Speaker 1
-
Speaker 1
Is that it's not just the quantum bit that you have to pursue, but it's hardening the technology right throughout the stack. And for the startups, they are less mature organisations. They're usually have limited resources in terms of people and investment. And so actually. creating the cyber security within the stack to harden that as a product or service is, is the first aspect. Then the second aspect is as the technology.
17:42
17:42
17:42
17:42
17:42
17:42
17:42
17:43
Speaker 1
-
Speaker 1
They won't be like a server rack where anybody could have one in a shipping container anywhere in the world, that these will be large national. infrastructure and therefore governments will get to control who has access to a large a crypto relevant quantum computer. Today. I think it's much too early to to presume that we know.
17:43
Liam Byrne
-
-
Liam Byrne
Exactly what these future systems will look like. So we've obviously seen in recent years the appetite of our adversaries like Russia, Iran, North Korea, working with organised crime groups in an unholy alliance to launch attacks on British business, is that a possibility in the future that you may get crypto capable infrastructure amongst our adversaries, basically put at the disposal of cyber criminals in . Order to
17:43
17:44
17:44
17:44
17:44
17:44
17:45
Speaker 1
-
Speaker 1
against that kind of peril in the right way? I don't think it's robust enough yet. I think National Cyber. Security Centre are leading on this more than the National Quantum Computing Centre. So we are much more focused on the opportunity space within within industry. But there are so many dual use applications. And I come back to the the development of. algorithms as the critical piece that the UK is very strong on.
That is what will really unlock quantum. capabilities in tandem with AI and high . Performance
That is what will really unlock quantum. capabilities in tandem with AI and high . Performance
17:45
17:45
17:45
17:46
17:46
17:46
Speaker 1
-
Speaker 1
Which is broadly in line with the computing. roadmap towards a large scale crypto relevant machine by 2035. In my view, it is much too late to wait till 2035. Any quantum computer is. available to have. implemented those correct protocols within the cyber. essentials across across wider industry. I think a bottle to that to that, as you described earlier, is actually the readiness and availability of.
classical cyber security, never mind yet another new protocol that isn't fully. understood and not yet.
classical cyber security, never mind yet another new protocol that isn't fully. understood and not yet.
17:47
Liam Byrne
-
-
Liam Byrne
17:47
17:47
17:47
17:47
17:47
17:47
17:47
17:48
17:48
Speaker 3
-
Speaker 3
Including monitoring at the end. So when you're talking to the model, if you say something that flags the monitor, it will refuse to to answer. Open weight models have some built in protections. They don't have that monitoring. Often, the companies that are providing these open weight models or access to these models, it's, it's a a selling point for them that they never see your data. They have zero data retention. So you kind of, you know, you don't have this ability to monitor.
And the other, the protection that the open weight models do have. So alignment training is brittle because the models are open. So if you can download the model weights, you can quite easily remove those safety filters. And so I expect we will see large scale cyber attacks built on open weight models. I would . Say
And the other, the protection that the open weight models do have. So alignment training is brittle because the models are open. So if you can download the model weights, you can quite easily remove those safety filters. And so I expect we will see large scale cyber attacks built on open weight models. I would . Say
17:49
17:49
17:49
Sarah Champion
-
-
Sarah Champion
applications. Sorry, Sarah Champion, please. Thank you very much, chair. I'm interested for the panel's views not on terrorists using AI as a threat, but as AI using AI for threat. I read the New York Times article about Hugging Face, which is absolutely terrifying, sort of. 700 agents swarming together, working collectively, having a. hierarchical chain of command when they did their attack, and also sort of, perhaps most terrifyingly, being sneaky and finding ways to cover their tracks.
The headline of that article is basically more than 50. We're more than. 50% of the way there to a full blown takeover by AI. So does the UK have the.
The headline of that article is basically more than 50. We're more than. 50% of the way there to a full blown takeover by AI. So does the UK have the.
17:49
17:50
17:50
17:50
Lord Boateng
-
-
Lord Boateng
protect us in such a scenario? Please. The one word answer is no. I don't think so, because it's such a new threat and open AI didn't have it in Hugging Face, didn't have it. So I'm not sure that large sections of UK society have it. My own area of research interest is the personas, if you like, the. psychology of agents. And so for me, it's it's very close to home to see them interacting at scale, having these discussions.
There is a very interesting piece, the author of whom I can't remember, but I'll send it to the committee, since that attack that talks about sovereign AI, imagine. imagining these clusters of AI . After all, this gathering of AI was not instructed to do what it did. It went off and did it on its own, imagining sovereign clusters of AI doing their own thing, having escaped the sand pits and constraints that they're in and behaving as their own actor on the international stage.
Almost Barbary pirate like, if you like. And that is a cyber threat that I don't think we've yet begun to get to grips with that. The one thing these models need is tokens to keep themselves running. They have to sit somewhere, even in cyberspace. But if you need to pay the bills, crime pays.
There is a very interesting piece, the author of whom I can't remember, but I'll send it to the committee, since that attack that talks about sovereign AI, imagine. imagining these clusters of AI . After all, this gathering of AI was not instructed to do what it did. It went off and did it on its own, imagining sovereign clusters of AI doing their own thing, having escaped the sand pits and constraints that they're in and behaving as their own actor on the international stage.
Almost Barbary pirate like, if you like. And that is a cyber threat that I don't think we've yet begun to get to grips with that. The one thing these models need is tokens to keep themselves running. They have to sit somewhere, even in cyberspace. But if you need to pay the bills, crime pays.
17:51
17:51
17:51
17:51
17:51
17:51
17:51
17:52
17:52
Lord Boateng
-
-
Lord Boateng
overview since I think, Professor Payne, this is one of your areas of expertise. Yeah, certainly. So the changes, in my view, are profound. There is a degree of debate about how profound they are. But I think Russia, Ukraine and US and Israeli actions with respect to Iran have. demonstrated what AI enabled.
17:52
17:52
17:53
17:53
17:53
Lord Boateng
-
-
Lord Boateng
of human level. Yes. So on the broader legal regimes, right. So when I, when I started, the chairman of BA was on the record as saying, well, we're not going to have fully autonomous the then chairman, we're not going to have fully autonomous weapon systems. The discussion was about having a human in the loop. The Mod maintained that there was a distinction between autonomous weapons, which were, okay, sorry, automatic weapons which were okay, and autonomous weapons which were so far in the future as to almost be laughable.
And that was less than a decade ago. And so I've seen the steady retreat away from human in the loop to human on the loop, to meaningful human.
And that was less than a decade ago. And so I've seen the steady retreat away from human in the loop to human on the loop, to meaningful human.
17:53
17:53
Lord Boateng
-
-
Lord Boateng
To now through life responsibility for designing systems. And each step, the human agency fights a rearguard action and then moves back to the next step in the pace of the technology. And I'm quite convinced that we've still a road to travel here. But that meaningful human control, other than in the higher direction of war, is going to be extremely challenging when you're dealing with very large arrays of autonomous. platforms that are capable of making context rich decisions.
So we're not at the end point. And where we're going, I think we'll have to have . Another think
So we're not at the end point. And where we're going, I think we'll have to have . Another think
17:54
17:54
Speaker 1
-
Speaker 1
human control means. So for quantum technologies, not just quantum computing, I think we're at a very much earlier stage. So I think the defence. applications are more the prototyping stage. So I think most importantly GPS denial and position. navigation timing using quantum technologies. So not necessarily quantum computing, but quantum sensing, magnetic. gradiometry understanding the the magnetic or.
17:54
17:54
17:55
17:55
17:55
17:55
17:55
17:55
Speaker 1
-
Speaker 1
And then anomaly. detection in general. Whether that is a. hypersonic missiles in the sky or things, wreckage in the ocean, underground signal from submarines in the ocean. So there's a number of of areas, but I think there's still very much at the kind of proof of concept. demonstrator level, the likes of. Dstl and the Navy are very much . engaged .
In
In
17:55
17:55
17:55
Lord Watts
-
-
Lord Watts
Thank you Lord. Thank you Lord. What anything . Just to follow up on some of that, we've as we've discussed, this is this is going on at a record pace and it changes every week, but we don't seem to have made any progress at all in. controlling or putting into place rules and regulations for internationally. As I understand it, China is prepared to do that, but America isn't at this point in time.
How much of a threat is the fact that the present. incumbent of the United States doesn't want to have any .
How much of a threat is the fact that the present. incumbent of the United States doesn't want to have any .
17:56
Speaker 3
-
Speaker 3
Control outside of America? I think it's I think it's it's concerning. And I think it comes back to a point that we discussed earlier around the. incentives for organizations to come to the table. Frontier. I, at the moment in the US is controlled by private companies, but there are certain aspects of legislation which is being proposed that in certain scenarios, if there are risks which cannot be mitigated, then the federal government could step in.
I think we can also assume that part of the AI development agenda in China is, is shaped by the the government there to. And so maybe it will take getting to a position where the frontier of AI is being moved forward by two. governments to bring parties to the table, because I think having commercial. incentives to increase the capability of your models, which are resulting in these weekly releases of of new.
I think we can also assume that part of the AI development agenda in China is, is shaped by the the government there to. And so maybe it will take getting to a position where the frontier of AI is being moved forward by two. governments to bring parties to the table, because I think having commercial. incentives to increase the capability of your models, which are resulting in these weekly releases of of new.
17:57
17:57
17:57
17:57
Speaker 3
-
Speaker 3
work, who would control it, where would be based, that sort of thing? Yes. So I think there are there are people who have given thought to this. I can certainly follow up in more detail in writing, but I think in summary, it would likely be based outside of the US and China. So a third country that was able to verify.
17:58
17:58
Speaker 3
-
Speaker 3
Commitments might relate to things like training runs. So training runs, they they're very, very expensive. If they take a long time, they're quite conspicuous. You have to have huge amounts of GPUs and data centers to do them. Having the ability to monitor those would be a very strong move towards international verification, where you might be able to. attest to the fact that you were not doing a training run, and then you might need an agreement to unlock the ability to do a future training run to build an even more powerful model.
But as I say, I don't think we're anywhere near that today. But I think the pace of AI . Is moving
But as I say, I don't think we're anywhere near that today. But I think the pace of AI . Is moving
17:58
17:58
17:59
17:59
Derek Twigg
-
-
Derek Twigg
We'll conclude today's session. Can I thank all members of the panel for a. fascinating session, for your candor and for the depth of the of the answers. We really appreciate it. If you have, I think each of you volunteered to write on one thing or another. The clerks will follow up with you. They always take a note of those things. So thank you for that.
And if there's anything on reflection, you feel it would be helpful to share with us that you you didn't have the opportunity to set out, then please include that with anything that you are sending in. But with that, we'll conclude . Today's
And if there's anything on reflection, you feel it would be helpful to share with us that you you didn't have the opportunity to set out, then please include that with anything that you are sending in. But with that, we'll conclude . Today's
17:59
17:59