Prediction Probability

Prediction probability is the likelihood that a given prediction is correct, expressed as a number between 0 (impossible) and 1 (certain).

Prediction probability is commonly used in machine learning to quantify the confidence of model predictions and to make decisions based on the predicted probabilities.

Prediction probability works by assigning a numerical value to the likelihood of a predicted outcome. This value can be used to set thresholds for decision-making, evaluate risks, and interpret the confidence of model predictions. For instance, in multiclass prediction, the prediction probability can be used as a tie-breaker to assign the most probable class. In addition to making predictions, prediction probabilities can also be used to evaluate the performance of a model. Metrics such as log loss and Brier score measure the accuracy of probabilistic predictions by comparing the predicted probabilities to the actual outcomes.

Analysing the prediction probability associated with a prediction allows setting custom decision thresholds, too. For example, in a binary classification problem, a model classifies an email as spam with a prediction probability of 0.8. This means that the model is 80% confident that the email is spam. instead of using only the predicted classification directly, a user might decide to mark emails as spam only if the probability is above a certain threshold, such as 0.9. This increases the precision of the classification but decreases its recall. In recall-oriented settings where it is more important not to miss an instance of the target class, users may decided that a confidence of 0.2 is already sufficient to raise an alarm or similar.

A discussion how and when to set a custom decision threshold can be found here

Prediction probability is an essential concept in machine learning to build models that can handle uncertainty and make informed decisions based on the likelihood of different outcomes.