Home

A Random Forest is not simply a collection of trees trained on the same data — it is a carefully engineered diversity machine. Its accuracy advantage over a single tree comes almost entirely from the diversity of its constituent trees:…

The Random Subspace Method (RSM) is a diversity-generating technique that creates an ensemble by training each base learner on a different random subset of features — not a different sample of rows. Where bagging varies the training examples, RSM varies…

Random Forest extends bagging with one additional idea: at every split in every tree, only a random subset of features is considered as candidates. This feature subsampling decorrelates the trees far more than bootstrap sampling alone, driving the inter-tree correlation…

Bagging reduces variance by averaging predictions from many bootstrap-trained models. But variance reduction only improves generalisation if the base learner actually has high variance to begin with. A logistic regression model trained on two different 80% subsets of your data…

A single decision tree trained on your full dataset is highly sensitive to the training data: change a few rows and the tree changes completely. Bagging — Bootstrap Aggregating — fixes this by training many trees on independently sampled versions…

Training a 1,000-round AdaBoost model to zero training error sounds like a recipe for catastrophic overfitting. Classical statistical learning theory — which ties generalisation error to model complexity — would predict a massive generalisation gap. But empirically, AdaBoost trained to…

Boosting is uniquely sensitive to label noise. The same reweighting mechanism that makes AdaBoost powerful on clean data — assigning higher weights to misclassified examples — causes catastrophic weight concentration on mislabelled examples when noise is present. A single incorrectly…

Most classification problems ask “which one class does this example belong to?” Multi-label classification asks “which subset of labels does this example belong to?” — a fundamentally different task. A news article can be tagged sports, finance, and politics simultaneously.…

Binary boosting — AdaBoost, gradient boosting, XGBoost — is well understood, but most real classification tasks involve more than two classes: digit recognition, product categorisation, disease type diagnosis. Extending boosting to K > 2 classes requires a different loss function…