A faithful, dependency-free reproduction of Gradient Regularized Newton Boosting Trees — the algorithm that turns Newton's second-order signal into a globally convergent boosting iteration. Vanilla Newton diverges. GRN converges.
Loss per iteration, log scale — paper reproduction on Wine Quality with the Charbonnier loss.
experiments/wine_charbonnier.py
λk = λbase + √(M · ||gk||H)
Reproducing & extending
Overview
A reference implementation of Newton boosting you can actually read, with no C extensions, no framework lock-in, and no mystery in the iteration. Every closed-form in the paper has a name, a test, and a diagnostic.
Gradient boosting uses first-order signal. Newton boosting uses the curvature. Faster convergence per iteration, but unstable: unregularized Newton can diverge when the Hessian is non-constant.
GRN replaces the static λ with λk = λbase + √(M · ||gk||H), where M = M0 · √N. The Hessian-Lipschitz constant M0 is analytic for four standard losses.
Proposition 5.1 of the paper guarantees global convergence. The implementation ships with Hilbert-space diagnostics to verify Θk, γk, and Lemma 4.2 on every trained model.
Algorithms
Vanilla Newton boosting (Algorithm 1) and Gradient Regularized Newton boosting (Algorithm 2) share everything — loss, tree builder, diagnostics — except a single line that changes the entire convergence behaviour.
Algorithm 1. Uses a fixed λbase at every iteration. On losses with non-constant Hessians — e.g. Charbonnier or cross-entropy — the iteration can diverge.
for k = 1, 2, …, K:
g_k, h_k = ∇L(F_k), H(F_k)
f_k = NewtonTree(g_k, h_k, λ_base)
F_{k+1} = F_k + η · f_k Algorithm 2. λk scales with the gradient norm (Proposition 5.1). The iteration converges — steadily and provably.
for k = 1, 2, …, K:
g_k, h_k = ∇L(F_k), H(F_k)
λ_k = λ_base + √(M·||g_k||_H)
f_k = NewtonTree(g_k, h_k, λ_k)
F_{k+1} = F_k + η · f_k
w = -Σg / (Σh + λ)
Algorithm 1, line 4 — exact, no gradient descent on leaves.
½[ G²_L/(H_L+λ) + … − G²/(H+λ) ]
Used for greedy split selection; closed form, no approximations.
M0 ∈ {0, ¼, 1}
Analytic Hessian-Lipschitz constants for MSE, BCE/CCE and Charbonnier.
Capabilities
A focused surface area: six core APIs, four losses, three engines. Built for clarity, not for benchmarks on big data.
Exact greedy splits with closed-form leaf weights and exact Newton gain. No gradient descent on leaves, no approximations.
Vector-valued tree leaves share split structure across classes. Block Hessians diag(p) − ppᵀ extracted to per-class diagonals.
Each loss ships with its analytic Hessian-Lipschitz constant M₀ ∈ {0, ¼, 1}, ready to plug into Proposition 5.1.
Exact Newton direction f = -g/(h+λ), cosine angle Θₖ, weak gradient edge γₖ, and Lemma 4.2 — all numerically checked.
A single runtime dependency. Optional extras: scikit-learn for datasets, matplotlib for the experiment plots.
Every closed-form in the paper has a test. Mathematical identities are checked numerically, byte-faithful to the formulas.
Mathematical guarantees
Every closed-form in the paper maps to a test. The library is
byte-faithful to Zozoulenko et al. (2026); see
docs/fidelity.md for the
section-by-section report.
Algorithm 1, line 4. Each leaf's weight is the exact Newton step on its share of the loss.
Used to rank candidate splits greedily, with no approximation.
With M = M₀ · √N and ||gₖ||_H = ||gₖ|| / √N, matching the paper's notation.
Verified numerically per iteration by verify_lemma_4_2.
Hessian-Lipschitz constants, hard-coded per loss — no estimation step.
Extracted to per-class diagonals for the K-class Newton tree builder.
Developer experience
Familiar .fit / .predict surface,
strict type hints, mypy-clean, and four short lines to a paper-faithful
training run.
import numpy as np
from grnbt import VanillaNewtonBoosting, MSELoss
rng = np.random.default_rng(42)
X = rng.standard_normal((500, 8))
y = X[:, 0] + 0.5 * X[:, 1] ** 2 + 0.1 * rng.standard_normal(500)
model = VanillaNewtonBoosting(
loss=MSELoss(),
n_estimators=100,
learning_rate=0.1,
max_depth=4,
)
model.fit(X, y)
preds = model.predict(X)
print(f"final loss: {model.history['loss'][-1]:.6f}")from grnbt import (
GradientRegularizedNewtonBoosting,
CharbonnierLoss,
)
model = GradientRegularizedNewtonBoosting(
loss=CharbonnierLoss(),
n_estimators=100,
learning_rate=1.0,
max_depth=4,
lam_base=1e-3,
)
model.fit(X, y)from grnbt import (
MultiClassNewtonBoosting,
CategoricalCrossEntropyLoss,
)
logits = np.stack([X[:, 0], -X[:, 0] + X[:, 1], X[:, 2]], axis=1)
y = np.argmax(logits, axis=1)
model = MultiClassNewtonBoosting(
loss=CategoricalCrossEntropyLoss(n_classes=3),
n_estimators=100,
learning_rate=0.1,
max_depth=3,
n_classes=3,
)
model.fit(X, y)
probs = model.predict_proba(X) # (n, 3), rows sum to 1from grnbt.diagnostics import (
cosine_angle_theta,
exact_newton_direction,
verify_lemma_4_2,
weak_gradient_edge_gamma,
)
# Replay iteration k to inspect Θₖ and γₖ.
f_exact = exact_newton_direction(g, h, lam_k)
f_weak = tree.predict(X)
theta_k = cosine_angle_theta(g, h, f_exact, f_weak)
gamma_k = weak_gradient_edge_gamma(g, h, lam_k, f_weak)
verify_lemma_4_2(g, h, lam_k, f_exact)Reproducible experiments
Three experiment scripts reproduce Figures 1, 2 and the hyperparameter
ablations from Zozoulenko et al. (2026). Machine-readable artifacts
written to experiments/.
Vanilla Newton diverges. The static-high-λ baseline plateaus biased. GRN's adaptive λₖ converges — steadily and provably.
Record the cosine angle Θₖ and weak gradient edge γₖ per iteration. Verify Lemma 4.2 numerically on a trained model.
Loss × engine × η × depth × λ_base grid, written to a tidy CSV. Aggregate with one line of pandas.
When to use GRNBT
GRNBT is the right tool when clarity matters more than throughput. For big-data production workloads, prefer a system built for it.
Run experiments that reproduce Figures 1 and 2 of the paper. Trace every identity numerically.
Use it in a graduate ML seminar. Every line maps to a closed-form on the projector slide.
Drop in a custom loss with its analytic M₀, swap in your own tree, and run.
API surface
Three engines, four losses, two tree learners, four diagnostics — everything you need to reproduce the paper and explore beyond it.
VanillaNewtonBoosting Static-λ second-order GBDT (Algorithm 1) GradientRegularizedNewtonBoosting Adaptive-λ boosting (Algorithm 2) MultiClassNewtonBoosting K-class softmax boosting MSELoss Mean-squared-error loss (M₀ = 0) CharbonnierLoss Pseudo-Huber-style loss (M₀ = 1) BinaryCrossEntropyLoss BCE loss (M₀ = ¼) CategoricalCrossEntropyLoss Multi-class CCE loss (M₀ = ¼) NewtonTree Scalar-output greedy Newton tree MultiClassNewtonTree Vector-output K-class Newton tree cosine_angle_theta Hilbert-space angle Θₖ exact_newton_direction Closed-form f = -g/(h+λ) verify_lemma_4_2 Lemma 4.2 identity check weak_gradient_edge_gamma γₖ weak gradient edge Quick start
pip install to
paper-faithful training in seconds.
The package is pure NumPy. Optional extras pull in scikit-learn for datasets and matplotlib for plots.
pip install grnbtgit clone https://github.com/sachncs/gradient-regularized-newton-boosting-trees.git
cd gradient-regularized-newton-boosting-trees
pip install -e .pip install -e ".[dev]" # adds pytest, mypy, scikit-learn, matplotlibReference
If you use this implementation in your research, please cite the accompanying preprint. The library is reproduction code only; the paper contents remain the property of the original authors.
@article{zozoulenko2026grnbt,
title = {Gradient Regularized Newton Boosting Trees
with Global Convergence},
author = {Zozoulenko, Nikita and
Falkowski, Daniel and
Cass, Thomas and
Gonon, Lucien},
journal = {Preprint (arXiv id verification pending)},
year = {2026}
}A faithful, dependency-free reference implementation of Newton's second-order boosting — regularized, proven, and ready to inspect.