XGBoost internals from scratch: the second-order gain, the closed-form leaf weight, and what λ and γ actually do

Plain gradient boosting fits each new tree to the previous model's residuals. XGBoost keeps that skeleton but rebuilds the engine — and the rebuild is what makes it the algorithm that keeps winning tabular competitions. I built a tiny 1-D demo that computes every piece of it for real, and once you see the four moving parts, the "magic" turns out to be a couple of closed-form formulas. Here they are. Every point carries a gradient and a curvature Instead of chasing raw residuals, XGBoost writes…