Question to the Home Office:

To ask the Secretary of State for the Home Department, what assessment her Department has made of the accuracy of (a) benchmarking, (b) validation methods and (c) error rates of AI tools used in asylum casework.

There are two AI tools in use in asylum casework currently; Asylum Case Summarisation (ACS) and Asylum Policy Search (APS).

Bench Marking & Validation - Both evaluations used a mixed-methods approach to collect primary data during and after the pilots. All participants in the test and comparison groups were asked to log information for each case undertaken. For the Asylum Case Summarisation pilot, the logging exercise captured data on 334 cases in the test group and 95 cases in the comparison group. For the Asylum Policy Search pilot, the logging exercise captured data on 270 cases in the test group and 214 cases in the comparison group.

The following published research note provides more detail on both pilots and the findings we documented - Evaluation of AI trials in the asylum decision making process - GOV.UK.

Error Rates - Technical specialists reviewed all summaries created by the Asylum Case Summarisation tool for accuracy prior to use in the pilot. A small proportion of summaries produced (9%) were deemed to be inaccurate or had missing information and were therefore removed from the pilot and these cases progressed in the business-as-usual way. Of the summaries that progressed in the pilot, 23% of users reported they were not fully confident in the summary information and would warrant further exploration in a full roll out. The Asylum Case Summarisation Tool has not yet been rolled out operationally.

The methodologies used in the evaluation of our Asylum AI tools were reviewed by and agreed with the Cabinet Office Evaluation Taskforce, and apply to both ACS and APS.