Which boosting variant is described as 'new trees learn from the remaining errors'?

Which boosting variant is described as 'new trees learn from the remaining errors'?

• A. AdaBoost
• B. Gradient Boosting
• C. Boosting
• D. Random Forests

Answer: (B)

The key idea is sequentially adding models that fix what the current ensemble gets wrong. The variant that does this by training each new tree on the errors remaining after the previous trees is gradient boosting. In gradient boosting, every new tree tries to predict the negative gradient of the loss function with respect to the ensemble’s predictions, which effectively are the current residuals or remaining errors. By fitting to these residuals and combining the trees (often with a learning rate to control progression), the model gradually reduces the overall loss and improves accuracy. This approach differs from AdaBoost, which adjusts the weights of training samples to emphasize misclassified ones rather than fitting residuals; from Random Forests, which build many trees in parallel on bootstrap samples and aggregate them; and from the broader idea of boosting, which is the general concept of creating successive models to correct prior mistakes without specifying the residual-based update.

The key idea is sequentially adding models that fix what the current ensemble gets wrong. The variant that does this by training each new tree on the errors remaining after the previous trees is gradient boosting. In gradient boosting, every new tree tries to predict the negative gradient of the loss function with respect to the ensemble’s predictions, which effectively are the current residuals or remaining errors. By fitting to these residuals and combining the trees (often with a learning rate to control progression), the model gradually reduces the overall loss and improves accuracy.

This approach differs from AdaBoost, which adjusts the weights of training samples to emphasize misclassified ones rather than fitting residuals; from Random Forests, which build many trees in parallel on bootstrap samples and aggregate them; and from the broader idea of boosting, which is the general concept of creating successive models to correct prior mistakes without specifying the residual-based update.