QQN Optimizer — Quadratic-Quasi-Newton, Illustrated

Illustrated summary

One iteration, step by step

Every iteration of QQN runs the same loop. Four of the steps are strategy slots — pluggable components with a fixed interface. Everything else is glue that never changes.

Phase 1 · Orient
1

Evaluate

Compute f(x) and ∇f(x) at the current iterate.
2
slot · Gradient
Steepest-descent direction

g̃ = ‖q‖·(−∇f)/‖∇f‖ — the gradient direction, rescaled to the length of the oracle step. It sets the path’s tangent at t = 0 (the anchor that guarantees a descent direction always exists) while cancelling ‖∇f‖, so the path never stretches with the gradient’s magnitude.
3
slot · Oracle
Quasi-Newton endpoint

q = −H∇f from the oracle (L-BFGS two-loop recursion by default). It may be aggressive — it does not need to be a descent direction on its own.
4

Build the quadratic path

d(t) = t(1−t)·g̃ + t²·q. A single 1-D search space of states x + d(t), from “barely moved along −∇f” to “full oracle step”. Because the gradient leg carries the oracle’s length, t means the same thing from iteration to iteration.
Phase 2 · Step
5
slot · Search
Walk the path

The line search probes t ∈ [0, 1] directly (Armijo backtracking by default, optionally refined by a cubic Hermite spline fit to every probe) and selects t*.
6
slot · Region
Project every probe

Each candidate is remapped, project_R(x, x + d(t)) — box clip, orthant, trust-region sphere… — so the search navigates the feasible path. Identity when unused.
7

Accept

x ← x + d(t*), only if sufficient decrease holds: f(x_new) ≤ f(x) + c₁⟨∇f, d(t*)⟩.
8

Learn

Oracle ingests the curvature pair (s, y) = (x_new − x, ∇f_new − ∇f) (admitted only if ⟨y, s⟩ > ε); region adapts its state (e.g. trust radius via ρ = ared / pred).
↺
Repeat until ‖∇f‖ ≤ tol or iter ≥ maxiter.

d′(0) = g̃ ∝ −∇f

The directional derivative at the origin is ⟨∇f, g̃⟩ = −‖q‖·‖∇f‖ < 0. No matter how poor the oracle is, the beginning of the path always decreases f. This single fact is what makes global convergence provable.

d(1) = −H∇f

Whatever the oracle proposes is reachable exactly at t = 1. Near a well-conditioned minimum the search lands there, and QQN inherits L-BFGS’s superlinear convergence.

0 < t < 1

A smooth blend, weighting the gradient by t(1−t) and the oracle by t². The right blend is discovered by the search, not hand-tuned — “which direction?” becomes “where on the curve?”.

Scale-invariant by construction

Only the gradient’s direction enters the path: g̃ = ‖q‖·(−∇f)/‖∇f‖, so ‖∇f‖ cancels. Multiply the objective (or the gradient) by any c > 0 and d(t) — hence the entire 1-D search space — is unchanged. The oracle alone sets the scale.

Why not a straight line?

A linear blend (1−t)g̃ + t·q starts at g̃ with full magnitude — you can’t take a tiny step. The quadratic path passes through the origin tangentially, so arbitrarily small, guaranteed-valid steps exist along the same curve.

Interactive illustration

Drag the directions. Slide along the path.

Drag the orange handle to set the gradient direction −∇f and the blue handle to set the oracle direction −H∇f. The background is a toy quadratic landscape consistent with that gradient; the lower plot is what the line search actually sees: φ(t) = f(x + d(t)) − f(x). Note that only the direction of the orange handle moves the curve: the gradient leg is rescaled to the oracle step’s length (the faint dashed orange arrow g̃), so stretching −∇f leaves d(t) exactly where it is.

drag the handles · contours = toy f · the path uses ∇f’s direction only

−∇f (gradient — direction only) q = −H∇f (oracle) path d(t) x + d(t) g̃ = ‖q‖·(−∇f)/‖∇f‖ · leg t(1−t)·g̃ t²·q trust region / projected path

φ(t) along the path · click/drag to set t

φ(t) = f(x + d(t)) − f(x) Armijo threshold c₁⟨∇f, d(t)⟩ (c₁ = 0.1, exaggerated) accepted region t* = argmin φ

path parameter t 0.600

landscape condition number κ 6

trust-region sphere (Region slot)

t: —
t(1−t) · gradient weight: —
t² · oracle weight: —
‖d(t)‖: —
φ(t) = f(x+d(t)) − f(x): —
Armijo test at t: —
t* = argmin φ: —
φ(t*): —

Drag the orange handle radially. The path doesn’t budge. Only ∇f’s direction enters d(t); its magnitude cancels against ‖q‖. Rotating the handle does reshape the curve.

Try “Uphill oracle”. Even when −H∇f points uphill, the start of the path still descends — the green accepted band hugs t → 0. That is the convergence anchor.

Try “Oracle = Newton step”. With κ large, the gradient alone zig-zags across the valley, but the path bends into the minimizer and t* → 1. That is the speed.

Try the trust region. Probes are projected onto the sphere inside the search; the green curve is the feasible path the line search actually walks.

Strategy selector

Four slots. Many providers.

Gradient, Oracle, Search and Region are conceptually orthogonal and independently swappable. Each is a pure interface; providers plug in without touching the solver loop. Pick a slot to see what it does and who can fill it — then build a configuration below and see which classical optimizer you just reproduced.

Build your optimizer

QQN as a configuration space

Classical methods are the points where one or two slots are fixed to a canonical choice.

Oracle Search Region spline refinement

reproduces

What’s different

Five things QQN does that most optimizers don’t

None of these are exotic on their own. Together they turn “an L-BFGS variant” into an optimizer with a provable descent guarantee and a genuinely modular architecture.

🛡️

Non-decreasing search

Every accepted step must satisfy f(x + d(t)) ≤ f(x) + c₁⟨∇f, d(t)⟩. The sequence of objective values is monotone — there are no “escape” steps, no learning-rate warm-ups, no hoping a bad step averages out later. If no acceptable state is found, the iteration fails loudly rather than moving uphill.

📐

Valid tiny steps ⇒ convergence proven

Because d′(0) = g̃ = ‖q‖·(−∇f)/‖∇f‖ is a positive multiple of −∇f, we have ⟨∇f, d′(0)⟩ = −‖q‖·‖∇f‖ < 0: for any ∇f ≠ 0 there is a t > 0 that decreases f. The search always has a valid fallback on the same curve, so global convergence holds regardless of how bad the oracle direction is — the oracle only has to be useful, never safe.

🎯

Deterministic objective function

A line search compares f at different probes; those comparisons are only meaningful if f(x) is a function of x. QQN treats the objective as deterministic — full-batch, or a minibatch held fixed for the whole iteration — instead of a noisy estimate that must be averaged over many small steps.

🔁

2-phase optimization loop

Orientation and stepping are separated. First the path is constructed; then it is searched and the result is fed back. The original MindsEye implementation lives in the opt/orient package for exactly this reason — QQN was born as an orienting strategy.

Phase 1 · Orientgradient + oracle → d(t)

Phase 2 · Stepsearch + region → accept t*, update state

🧩

Strategy-provider pattern

Gradient, Oracle, Search and Region are interfaces (init / direction or project / update). Providers are pure functions; combinators such as Fallback, Blend and Sequential compose them. State is threaded through one immutable QQNState, so the whole thing stays jit/vmap/grad friendly — and many classical optimizers fall out as particular slot choices.

〰️

Bonus: information-reusing search

Every probe along d(t) yields both a value and a directional derivative. The optional spline refinement fits a cubic Hermite spline through all probes and jumps to its stationary points — reusing gradient information a naive line search throws away.

History

From a Java hobby project to a paper and three ports

QQN has been quietly training neural networks for the better part of a decade.

< 2020
Original implementation — MindsEye (Java)

QQN first appeared as an orienting strategy in MindsEye, a Java deep-learning framework. It was used to train style-transfer and texture-synthesis models whose outputs are collected in the DeepArtist gallery — the “deterministic objective, non-decreasing search” philosophy dates from here.

☕ QQN.java (actual implementation) 📦 MindsEye project 🖼️ Image gallery
2025
Formal paper and Rust implementation

The algorithm was written up formally — the quadratic path, its descent guarantee, and a benchmark suite — alongside a from-scratch Rust implementation, qqn-optimizer, used to produce the paper’s results.

📄 Paper (PDF) 🔗 DOI 🦀 qqn-optimizer
2026
Python variants and MNIST benchmarking

Pure-functional ports for the major Python frameworks, generalising the design into the four strategy slots (oracles, regions, spline search) and benchmarking the whole configuration space on MNIST.

The academic paper

Quadratic-Quasi-Newton Optimization: Combining Gradient and Quasi-Newton Directions Through Quadratic Interpolation. Full derivation, convergence analysis and benchmarks.

📄 Open the PDF ⬇️ Download 🔗 DOI 🖥️ Embedded viewer

QQN(
    fun,
    oracle="lbfgs",          # t = 1 endpoint
    line_search="armijo",    # walks t ∈ [0, 1]
    spline=False,            # Hermite refinement
    region=None,             # feasibility projection
    history_size=10, tol=1e-5, maxiter=100,
)

One iteration, step by step

Evaluate

Steepest-descent direction

Quasi-Newton endpoint

Build the quadratic path

Walk the path

Project every probe

Accept

Learn

Drag the directions. Slide along the path.

Four slots. Many providers.

QQN as a configuration space

Five things QQN does that most optimizers don’t

Non-decreasing search

Valid tiny steps ⇒ convergence proven

Deterministic objective function

2-phase optimization loop

Strategy-provider pattern

Bonus: information-reusing search

From a Java hobby project to a paper and three ports

Original implementation — MindsEye (Java)

Formal paper and Rust implementation

Python variants and MNIST benchmarking

The academic paper