UK police (NDAS): bug inflates accuracy

01 · CONTEXT

What the model tried to predict

The prediction system called Most Serious Violence (MSV) was part of the National Data Analytics Solution (NDAS) project. The model was designed to predict whether a person would commit their first violent crime with a firearm or knife in the following two years. People who had already had contact with the two police forces involved in the development received a risk score, and the higher the score, the more likely the crime.

To develop the system, historical data on 2.4 million people in the West Midlands database and 1.1 million in West Yorkshire's was used. It came from crime and custody records, intelligence reports, and the national police database. This is a case of data pipeline, of machine learning system and for monitoring the quality of what the model delivers.

The system was never used in police operations. The case, therefore, is not about harm caused by a model in production, but about an evaluation number that did not survive checking before it got there.

02 · THE ERROR

A flaw in the training definition

1. Crime and custody records, intelligence reports, and the national police database feed the data ingestion.

2. A coding error in the definition of the training set made the problem statement unworkable. An NDAS spokesperson described it as a data ingestion problem.

3. Before the error was found, the NDAS claimed accuracy of up to 75 percent.

4. With the system rebuilt, accuracy in the West Midlands case fell to a range between 14 and 19 percent.

5. The proposal to take the system forward was unanimously rejected by the ethics committee.

An NDAS briefing, published by the West Midlands police ethics committee, records that a coding error in the definition of the training set made the MSV problem statement unworkable. An NDAS spokesperson told the reporter that the error was a data ingestion problem discovered during development.

03 · THE NUMBERS

What the error did to accuracy

The NDAS acknowledged that the real level of accuracy was significantly lower, and the report summarizes that the original performance had been overestimated. The rebuilt system was more accurate when the original criteria (first crime, weapon type, and weapon use) were removed.

To give weight to a number, you need a reference. The NDAS document to the committee compares the lower accuracy with random choice: among about 400,000 people, the chance of drawing someone who commits a serious violent crime would be 1.9%. Meanwhile, a researcher interviewed for the report points out that rare events are much harder to predict than common events.

04 · GOVERNANCE

The ethics committee as a gate

The West Midlands police ethics committee saw it as positive that the team had evaluated its own work and identified flaws. It also asked for an assessment of whether the available police data represent all the relevant information for this type of model. The proposal to continue with the system was unanimously rejected, and the committee advised that the project, as it stood, be discontinued.

In the document to the committee, the NDAS itself states that being on the high-risk list would not be enough to justify using police resources on an intervention. For another use case of the program (modern slavery), the committee recommended communicating the error in a fixed format: out of a population of N people, how many are expected to be identified, how many of those will have been falsely identified, and how many the model will fail to identify.

05 · DESIGN LESSONS

How the pipeline could defend itself

1. In this example design, between ingestion and training there is a check: the definition of the set (who is included and what the outcome to predict is) is compared with the source records.

2. The evaluation compares the model with a simple baseline, such as picking people at random, so that the number has a reference.

3. The evaluation uses data from after the training period, to resemble real use.

4. The gate releases the model only if the measure holds up and the expected error is written in a way the decision-maker understands.

5. A person reviews each flag and justifies the action, and the monitor compares the prediction with what actually happened.

The general lesson of the case is that the evaluation number is the product of a pipeline, and the pipeline also needs to be tested. Checking the training set definition against the source, comparing with a simple reference, evaluating on later data, and requiring a gate before use are independent layers of defense.