The UCB algorithm is related to a principle known as optimism in the face of uncertainty, a statistics-based principle based on the law of large numbers. UCB constructs an optimistic guess, based on the sample mean of the rewards, and on the estimation of the upper confidence bound of the reward. The optimistic guess determines the expected pay-off of each action, also taking into consideration the uncertainty of the actions. Thus, UCB is always able to pick the action with the higher potential reward, by balancing the risk and the reward. Then, the algorithm switches to another one when the optimistic estimate of the current action is lower than the others.

Specifically, UCB keeps track of the average reward of each action with , and the UCB (hence the name) for each action. Then, the algorithm picks the arm which maximizes the following:

(12.1)

In this formula, the role of  is to provide an additional argument to the average reward that accounts for the uncertainty of the action.