code4thought

SOFTWARE QUALITY

TRUSTWORTHY AI

Metrics for Deploying
Human-in-the-Loop
Classification Models

test test
At code4thought, continuous innovation, scientific rigor, and practical impact drive everything we build. That is why we are proud to highlight a recent academic achievement by our team.
Our colleague, Petros Stavroulakis, in collaboration with Pavlo Burda – ICTI, and Sieuwert van Otterloo – Hogeschool Utrecht, recently published and presented their paper titled “Metrics for Deploying Human-in-the-Loop Classication Models” at the 38th Benelux Conference on Artificial Intelligence (BNAIC 2026) in Maastricht.
Motivation and hypothesis. Machine-learning classifiers in sensitive domains are rarely deployed as fully autonomous decision-makers. Regulation increasingly requires human oversight: GDPR Article 22 restricts fully automated consequential decisions, and Article 14 of the EU AI Act requires human supervision of high-risk systems. Even where no rule applies, as in network intrusion detection, uncertain predictions are routed to human experts. Yet candidate models are still chosen using standard metrics such as accuracy, precision, F1 or ROC-AUC, which assume every prediction is final.
The paper’s central hypothesis is that, when a human reviewer is part of the pipeline, these headline metrics describe a system that is never actually deployed. They can hide the model that makes the best use of a given review budget, or even rank the wrong model first. The guiding question is how to express classifier performance in a way that reflects real human-in-the-loop deployment and that diverse stakeholders (engineers, operations managers, business decision-makers) can discuss together.
Proposed metrics. Building on the selective-classification literature, the authors propose two metrics for an already-trained probabilistic classifier. Each prediction gets a confidence score, its distance from the decision threshold |p̂ − t|. The most confident fraction c of cases (the coverage) is automated. The remaining 1 − c is deferred to a human reviewer, who is assumed in the main analysis to be always correct.
  • Deferral-Aware F1 (DAF1) is the ordinary F1 score computed only on the automated subset at coverage c. It can be reported as a single headline number at a chosen deferral rate. It can also be shown as a curve that starts at full coverage (equal to binary F1) and runs down to a coverage floor. It is the F1 counterpart of selective risk and risk–coverage curves.
  • Cost-Aware Score (CAS): applies a deferral penalty to DAF1: CAS(α, c) = DAF1(c) · (1 − α(1 − c)). Here α ∈ [0, 1] is the cost of a human review relative to the cost of a misclassification. At α = 0 review is free; at α = 1 the score scales with coverage itself. Maximizing CAS gives an explicit optimal operating point for each cost assumption.
The metrics are not meant to replace trained rejection estimators or learning-to-defer methods. They are a tool for evaluating and explaining a conventional classifier against a human review budget.
Experimental setup. The metrics are illustrated with a Security Operations Center (SOC) alert-triage scenario. High-confidence verdicts, such as blocking a connection, run automatically, while uncertain alerts go to analysts. Analyst time is scarce, and alert fatigue is the main operational problem.
The authors use the UNSW-NB15 intrusion-detection dataset: 257,673 connections, 196 features after encoding, and 63.9% attacks in the Kaggle version. The data is split 70/15/15 into training, calibration and test sets. They compare four default-configuration classifiers: logistic regression (LR), random forest (RF), XGBoost and a small multi-layer perceptron (MLP). The RF, XGBoost and MLP scores are Platt-calibrated, and each model’s threshold is tuned on the calibration set to maximize F1.
Results. Under binary F1, RF ranks first (0.961), followed by XGBoost (0.960), MLP (0.952) and LR last (0.931). Deferral changes this picture:
  • DAF1 rises steeply for every model with just 5–10% deferral.
  • For LR, deferring 10% of cases lifts F1 from 0.931 to 0.962. It also removes about half of the false positives (−1,116) and false negatives (−619).
  • At 30% deferral, LR, the simplest and most explainable model, ties RF for the best DAF1 (0.996) and overtakes XGBoost. This reverses the binary-F1 ranking.
The CAS analysis uses α = 0.1, 0.2 and 0.3 (cheap, moderate and expensive review):
  • As review gets more expensive, the optimal coverage rises.
  • RF has the highest CAS at every α. XGBoost needs the fewest deferrals, and LR achieves the largest error reduction (about 91% at 26% deferral when α = 0.1).
  • With α ≤ 0.2, all four models reach an F1 of about 0.985–0.995 at their CAS optimum, also measured on the whole workload with an ideal reviewer.
This suggests that a simple, cheap model combined with human review can be a credible alternative to a heavier, costlier and less explainable detector.
Conclusions and limitations. DAF1 and CAS give stakeholders a shared vocabulary for trade-offs. Engineers see what a model achieves at each coverage level, operations staff see which deferral rates their capacity allows, and business analysts see the price of a review against the cost of an error. The authors acknowledge several limitations:
  • The evaluation uses a single dataset and scenario.
  • F1 weights false positives and false negatives equally.
  • CAS assumes every review costs the same and, in the main analysis, that the reviewer is never wrong. An appendix shows the gains shrink with imperfect reviewers.
  • The single-threshold confidence rule cannot catch predictions that are confident but wrong, and it struggles with strong class imbalance. An appendix sketches a two-threshold alternative.
Future work includes validating the metrics in user studies with real decision-makers, extending them to classifiers without usable confidence scores, and developing a two-sided, class-specific DAF1.
You can read the full paper here.

FURTHER READING