Imagine you're adjusting the vertical position of a set of fixed venetian blinds. You can't rotate the slats — they're locked at random angles — but you can slide each blind up or down. By shifting where each slat catches light, you can sculpt surprisingly complex shadow patterns on the wall. That's what this paper is about: neural networks where the weights are frozen at random initialization, and only the biases — the vertical offsets — get trained. The committed claim: in a one-hidden-layer ReLU network with fixed random weights and trainable biases, the entire training trajectory can be decomposed into a sequence of linear least-squares problems. The transitions between these linear regimes happen only when neurons flip their activation states (on ↔ off), and the fraction of neurons that flip vanishes as network width increases. In the infinite-width limit, training collapses to a single linear least-squares solve on the initial activation pattern. The mechanism is elegant. Each input gets assigned an "activation code" — a binary vector recording which neurons are active. Because ReLU partitions the input space into convex polyhedral regions, and because bias gradients depend on inputs only through these codes, the entire gradient computation factors through a discrete combinatorial object. Between activation flips, the loss landscape is quadratic, the solution is closed-form, and the dynamics are those of ordinary least squares with the activation codes as features. Training is piecewise-linear. Where does this sit on the theoretical ladder? The result that random-weight-trainable-bias networks are universal approximators dates back to work from the 1990s and has been revisited periodically. The "neural tangent kernel" and "lazy training" literature (Jacot et al., 2018; Chizat & Bach, 2019) already showed that wide networks linearize around initialization. This paper narrows the focus to bias-only training and provides a sharper, more constructive characterization: not just "it linearizes" but "here is the exact sequence of least-squares problems and the exact combinatorial object (activation codes) that governs transitions." It's a refinement of lazy-training theory, not a departure from it. Integrity is decent but circumscribed. The theoretical claims — piecewise linearity, vanishing switch fraction — are supported by mathematical arguments and confirmed with synthetic experiments. But there's no independent benchmark, no comparison to full-weight training dynamics at matched scale, and the experiments stay in one-hidden-layer territory. The validation is self-contained: theory predicts X, same-team experiments confirm X. That's standard for theory papers but leaves the practical relevance open. The milestone question is whether this framework extends to multi-layer networks and non-ReLU activations. The paper explicitly restricts to one hidden layer and ReLU. Multi-layer bias-only dynamics would involve cascading activation-code interactions across layers — the combinatorial explosion is real. If someone demonstrates that the piecewise-linear characterization (or an analogous one) holds for 3+ layer bias-only ReLU networks, that would meaningfully extend the reach of this lens. The obvious experiment not run: training a multi-layer bias-only network and checking whether the piecewise-linear structure survives or breaks down. The honest read is (a) — the combinatorial analysis becomes dramatically harder with depth, and the authors scoped the paper to what they could prove cleanly. That's a defensible choice for a theory contribution, but it also means the practical implications remain narrow until the depth question is answered.
The analogy
Imagine adjusting venetian blinds that are locked at fixed angles but can slide up and down. You can't change how each slat bends light, but by shifting their vertical positions (biases), you change which slats catch each beam. Between shifts, the shadow pattern on the wall is a simple linear combination — until a slat crosses a light boundary and the whole geometry snaps to a new configuration. That's bias-only training: piecewise-linear dynamics punctuated by discrete switching events, with fewer switches as you add more blinds.
Venetian blinds
Fixed-angle slats slid vertically to reshape shadow patterns; each shift is linear until a slat crosses a light boundary
Bias-only ReLU
Fixed random weights with trainable biases; training is linear least-squares until a neuron flips activation state
Honest assessment
A clean, well-scoped theory paper that gives you a precise combinatorial lens (activation codes) for understanding bias-only training in shallow ReLU networks. It refines existing lazy-training/NTK results rather than overturning them. The practical implications are thin — nobody trains bias-only networks in production — but the theoretical insight is genuine and may prove useful as a building block if extended to deeper architectures. Worth reading if you're working on neural network theory or trying to build intuition for the piecewise-linear geometry of ReLU networks. Not worth reading if you want immediately actionable ML engineering insights.
What Is CLAIMS?
57
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
3/5
Architecture
4/5
Integrity
2/5
Milestone
2/5
Successor
3/5
What is this paper actually committing to?
Constructive refinement, not a first. The novelty is the activation-code lens on bias-only dynamics — precise and clean, but the headline result (wide networks linearize) was already known from NTK theory.
The field fight
Do overparameterized networks learn meaningful features or just operate in a linearized (lazy/kernel) regime?
Side A
Wide networks effectively linearize; NTK regime captures the essential dynamics
Held by: Jacot et al., Du et al., lazy-training theory
Side B
Feature learning beyond the kernel regime is what matters; linearization is an artifact of infinite-width limits
Held by: Yang & Hu (feature learning), mean-field theory, practical deep learning
Milestone tracker
Unlocks: Connecting bias-only training theory to architectures with practical depth; bridging lazy-training and feature-learning regimes
Current
1
hidden layers
this paper
Next
3+
hidden layers
multi-layer bias-only characterization
Unknown — combinatorial complexity grows fast with depth; no clear timeline
Verified Facts (5)
Neural networks with randomly initialized, fixed weights and trainable biases are universal approximators
Gradients with respect to the biases depend on each input only through its activation code
As long as the activation codes remain fixed, training reduces to a linear least-squares problem with the codes as features
The fraction of switching neurons vanishes as the network widens
The training dynamics approach those of linear least squares on the initial activation codes in the infinite-width limit
Practical Implications (4)
Bias-only training can serve as a controlled test bed for studying optimization dynamics without weight-interaction confounds
The activation-code framework provides a concrete diagnostic for monitoring how much a network's partition structure changes during training
For theory researchers: the piecewise least-squares characterization could be a useful proof technique for analyzing more complex training regimes
For practitioners: the vanishing-switch result suggests that in very wide networks, bias-only fine-tuning is effectively a single linear regression — reinforcing why large models often respond well to simple parameter-efficient methods
More latest news
Read all