Why "Accuracy" Is the Wrong Metric for Probabilistic Prediction Models

1Key Takeaways

• If you build a model that predicts the probability of an outcome — a race result, a match winner, a medical risk score, anything where the output is meant to be a probability rather than a hard label — "accuracy" is almost always the wrong number to optimize for, and often the wrong number to even report.
• I run ai-race.jp , which produces probabilistic predictions for horse races, and the gap between "the model is accurate" and "the model is useful" shows up constantly in that domain.