"Our model has 95% accuracy!" This statement, proudly presented in countless project reviews, is often meaningless — and sometimes dangerously misleading. In a dataset with 95% negative examples, a model that always predicts "negative" achieves 95% accuracy while being completely useless.

Choosing the right evaluation metrics is one of the most important — and most overlooked — decisions in any ML project. The metrics you optimize for fundamentally shape the model you build and the value it delivers.

Why Accuracy Fails

Accuracy measures the percentage of correct predictions across all classes. It fails in three common scenarios:

  • Imbalanced classes: Fraud detection (0.1% fraud), disease diagnosis (rare conditions), anomaly detection. Accuracy is dominated by the majority class.
  • Asymmetric costs: Missing a fraudulent transaction costs far more than flagging a legitimate one. Accuracy treats all errors equally.
  • Multi-class problems: A model that's excellent at common classes but terrible at rare ones can still show high accuracy.

Metrics That Tell the Real Story

Precision and Recall

Precision: Of all the items the model flagged as positive, how many actually were? High precision means few false alarms.

Recall: Of all the items that actually were positive, how many did the model catch? High recall means few missed cases.

The tension between precision and recall reflects a fundamental business trade-off. In fraud detection, you want high recall (catch all fraud) and accept lower precision (some false alarms). In spam filtering, you want high precision (never misclassify legitimate email) and accept lower recall (some spam gets through).

F1 Score and F-beta

The F1 score is the harmonic mean of precision and recall — a balanced metric when both matter equally. When they don't matter equally, use F-beta: set beta > 1 to weight recall more heavily, or beta < 1 to weight precision more heavily.

AUC-ROC

The Area Under the Receiver Operating Characteristic curve measures a model's ability to discriminate between classes across all possible thresholds. An AUC of 0.5 means random guessing; 1.0 means perfect separation. AUC-ROC is threshold-independent, making it useful for comparing models before choosing an operating threshold.

AUC-PR (Precision-Recall Curve)

For highly imbalanced datasets, AUC-PR is more informative than AUC-ROC. It focuses on the model's performance on the positive class, which is usually what you care about.

Business-Aligned Metrics

Technical metrics are necessary but insufficient. Every model should also be evaluated on business-relevant metrics:

  • Cost-weighted error: Assign dollar values to different error types and minimize total cost rather than error count
  • Lift over baseline: How much better is the model compared to the current process (rules, heuristics, or human judgment)?
  • Time to decision: How quickly does the model deliver a prediction compared to the manual alternative?
  • Coverage: What percentage of cases can the model handle versus those requiring human intervention?

Choosing Metrics by Use Case

  • Fraud detection: Recall (catch rate), precision at high recall, cost-weighted F-score
  • Medical diagnosis: Sensitivity (recall for disease), specificity, negative predictive value
  • Recommendation systems: NDCG, MAP, click-through rate, revenue per recommendation
  • Demand forecasting: MAPE, RMSE, bias (systematic over/under-prediction), coverage probability
  • NLP classification: Macro-F1 (treats all classes equally), per-class precision and recall

The Evaluation Process

  1. Start with business objectives. What decision does this model inform? What's the cost of each error type?
  2. Choose primary and secondary metrics. Optimize for one metric; monitor several others for balance.
  3. Set thresholds with stakeholders. "We need at least 90% recall with precision above 60%" — let the business define acceptable trade-offs.
  4. Evaluate on realistic data. Use stratified holdout sets that reflect production conditions, not random splits of clean training data.
  5. Monitor in production. Metrics calculated on test sets are estimates. Continuously verify against real-world performance.

The Bottom Line

The right metrics turn model evaluation from a technical exercise into a business conversation. When you can say "this model catches 92% of defective products while only flagging 3% of good ones for re-inspection," stakeholders understand and trust the system. When you say "95% accuracy," you've communicated nothing useful.