selective-classification

A list of papers that studies out-of-distribution (OOD) detection and misclassification detection (MisD)

Probability-aware evaluation for typed decision models: calibration, selective risk, latency, and reproducible benchmarks.

PyTorch implementation of our ECCV 2022 paper "Rethinking Confidence Calibration for Failure Prediction"

A decision-safety lab for loan approval: trains a baseline classifier, calibrates probabilities (ECE/Brier), sweeps confidence thresholds to build a coverage, quality frontier and outputs a defensible abstention policy (auto-decide vs review). Includes a Streamlit dashboard for report cards, triage UI, and data quality checks.

Longform article reframing abstention (reject option / selective prediction) as product design, not model weakness. Covers coverage as a KPI, calibration as a prerequisite, threshold selection under review capacity and risk, queue/UX design for human-in-the-loop workflows, and anti-patterns that break safety in production.

Decision-safe evaluation + Streamlit dashboard for AI vs Human vs Post-Edited AI text detection. Generates a reliability report card (Accuracy, Macro F1, ECE, Brier), calibration plots, confidence histograms, and a coverage-vs-performance abstention curve. Recommends an operating threshold for human-review routing.

Promises and Pitfalls of Threshold-based Auto-labeling (NeurIPS 2023, Spotlight)

Code for "Budgeted Classification with Rejection: An Evolutionary Method with Multiple Objectives", at IEEE CEC 2022

Robust Selective Classification of Skin Lesions with Asymmetric Costs

Reliable and Efficient Selective Machine Learning

More human oversight can make an AI agent less safe. Headroom is a human-in-the-loop firewall for coding agents that measures when to trust the human: oversight as resource allocation, not just classification.

Confidence-aware calibration framework for blood agglutination image classification (MobileNetV2/EfficientNetV2 + AutoGluon, isotonic calibration, class-conditional review policy). Externally validated on NIH Malaria dataset. Paper + reproducible pipeline.

‘‘I know that I don’t know... and I explain why’’ Robust abstention via counterfactual explanations

Event-level risk-controlled triage for conjunction data message streams

Recorded motor-imagery EEG demo with participant-disjoint evaluation, confidence-based abstention, ONNX inference and reproducible teaching exercises.

Applied NLP decision-support project for CFPB complaint routing, comparing TF-IDF + Linear SVM with DistilBERT through leakage-safe evaluation, selective routing, Human Review, and 2025 retrospective analysis.

Few-shot text classifiers that know when they don't know: selective classification with certified risk (SGR) + episodic prototypical fine-tuning. 12 Portuguese corpora, 7 OOD baselines.

Étude académique : classification histologique, calibration des probabilités et abstention sur PathMNIST. Scikit-learn, PyTorch et Streamlit.

Official implementation of the ICLR paper "Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It"

Risk-aware banking support router for 77 intents using calibrated classification, selective prediction, human escalation policies, and FastAPI.