Gradient Boosting Advanced Tutorial: XGBoost, LightGBM, CatBoost (2026)
Three libraries dominate gradient boosting: XGBoost with regularized objectives, LightGBM with GOSS and EFB for speed, and CatBoost with ordered boosting and native categorical handling.
By 2026, gradient boosting remains unbeaten on structured data benchmarks. Each library has converged while maintaining unique strengths.
XGBoost Core Concepts
XGBoost adds L1 and L2 regularization to the objective function. Key hyperparameters are learning_rate, n_estimators, max_depth, and subsample.
The most impactful insight was using early stopping with a validation set instead of guessing the number of trees.
import xgboost as xgb\nmodel = xgb.XGBRegressor(n_estimators=1000, learning_rate=0.01, max_depth=6, subsample=0.8)\nmodel.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)LightGBM for Speed
LightGBM uses GOSS to keep under-trained samples and EFB to combine mutually exclusive features. Leaf-wise tree growth improves accuracy.
I use LightGBM first for datasets over 100,000 rows due to its training speed. The categorical_feature parameter handles categoricals natively.
import lightgbm as lgb\nmodel = lgb.LGBMClassifier(boosting_type="gbdt", num_leaves=31, learning_rate=0.01, n_estimators=1000)\nmodel.fit(X_train, y_train, categorical_feature=["education"])CatBoost for Categoricals
CatBoost uses ordered target encoding to prevent target leakage and ordered boosting to reduce prediction shift.
CatBoost is my go-to for datasets with many high-cardinality categorical features. Default hyperparameters often perform surprisingly well.
from catboost import CatBoostClassifier\nmodel = CatBoostClassifier(iterations=1000, learning_rate=0.01, depth=6, cat_features=["education"])Hyperparameter Optimization
Bayesian optimization with Optuna is the preferred approach. The TPE sampler learns from trial results to focus on promising regions.
Optimal learning rates were consistently 0.005-0.02 across libraries, but optimal depth varied: XGBoost preferred 8, LightGBM preferred 32 leaves, CatBoost preferred 7.
import optuna\ndef objective(trial):\n params = {"lr": trial.suggest_float("lr", 1e-3, 0.1, log=True), "max_depth": trial.suggest_int("max_depth", 4, 12)}\n return cross_val_score(xgb.XGBRegressor(**params), X, y, cv=3).mean()Feature Importance
Each library provides different importance metrics: weight, gain, and cover in XGBoost; split and gain in LightGBM; PredictionValuesChange in CatBoost.
SHAP values provide the most consistent feature attribution through TreeSHAP.
import shap\nexplainer = shap.TreeExplainer(model)\nshap_values = explainer.shap_values(X_test)\nshap.summary_plot(shap_values, X_test)Ensembling Boosting Models
Stacking combines XGBoost, LightGBM, and CatBoost through a meta-model. Each captures different patterns.
A stacking ensemble with 13 base models achieved top-1 percent in a Kaggle competition, outperforming any single model by 2-5 percent RMSE.
from sklearn.ensemble import StackingRegressor\nensemble = StackingRegressor([("xgb", xgb.XGBRegressor()), ("lgb", lgb.LGBMRegressor()), ("cat", CatBoostRegressor(verbose=0))])Frequently Asked Questions
Which library should I use?
Start with XGBoost for maturity. Use LightGBM for speed on large datasets. Use CatBoost for categorical features. Ensemble all three for maximum performance.
How to handle missing values?
XGBoost learns the optimal direction for missing values automatically. LightGBM and CatBoost also handle missing values internally.
Best learning rate?
Typically 0.005 to 0.1. Lower values (0.01) with more trees produce better results. Use early stopping to find the optimal trade-off.
How to prevent overfitting?
Reduce learning rate, increase regularization, reduce max_depth, increase subsample, and use early stopping with a validation set.
Originally published on Ayodhyyya. Last updated June 1, 2026.