Bootstrapping is a resampling method. In statistics, resampling entails the use of many samples, generated from an original sample. In machine learning terms, the sample is our training data. The main idea is to use the original sample as the population (the whole domain of our problem) and the generated sub-samples as samples.

In essence, we are simulating how a statistic would behave if we collected many samples from the original population, as shown in the following diagram: