This page was written by an AI language model summarizing arXiv:1610.06160. It is not written, reviewed, or endorsed by the authors, and it may contain errors or misread the paper. Treat it as a reading aid and check anything that matters against the original PDF. The animated figure below is a toy illustration of the estimator, not a trained network — for a working PyTorch implementation and a real CIFAR-10 training demo, go to github.com/liaoq/StreamingNorm.
Mini-batches arrive left to right. ŝshort is the exact average of the batch statistics since the last weight update; at each update (dotted line) it folds into the exponential average ŝlong and resets. What the layer actually divides by is the blend ŝ = α1ŝlong + α2ŝshort. Drag α1 to 0 and the whole thing collapses back to batch normalization. Drag the plot sideways (or use ← →) to scroll back through the retained trace; generation stops while you're looking at history. This is a scalar illustration on a synthetic stream — the PyTorch implementation is where to go for the real thing.
Batch normalization estimates its mean and standard deviation from the samples sitting in the current mini-batch. That works beautifully for feedforward training with a healthy batch size, and it breaks in two places. With one or two samples per batch there is nothing to estimate from. In a recurrent network, activation distributions differ across timesteps, so a single set of statistics is wrong and per-timestep statistics don't generalize to sequence lengths you never trained on.
The fixes available in 2016 each solved half the problem. Time-specific batch normalization keeps separate statistics for every timestep — accurate, but memory grows with sequence length, it still needs large batches, and no plausible biological mechanism resets a neuron's gain on a per-timestep schedule. Layer normalization estimates from a single sample, which handles online and recurrent learning cleanly, but it measurably underperforms on convolutional layers.
This paper's move is to stop asking where the statistics come from within a step, and ask instead that they accumulate across steps. Keep a running estimate that streams over every sample and every timestep already seen. One layer, one set of statistics, every learning scenario.
| Approach | FF + FC | FF + conv | Rec + FC | Rec + conv | Online | Small batch | All combined |
|---|---|---|---|---|---|---|---|
| Batch norm | ✓ | ✓ | ✗ | ✗ | ✗ | weak | ✗ |
| Time-specific BN | ✓ | ✓ | limited | limited | ✗ | weak | ✗ |
| Layer norm | ✓ | weak | ✓ | weak | ✓ | ✓ | weak |
| Streaming norm | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Adapted from Table 1. Limited: per-timestep statistics may not transfer to unseen sequence lengths. Weak: trains, but well below the best method for that setting. FF = feedforward, Rec = recurrent, FC = fully connected.
Any normalization scheme can be described as: pick some set of activations to look at, boil them down to a few numbers, then use those numbers to rescale a neuron. Naming the three parts separately is what makes the design space visible.
The reference set \(R_i\) of activations feeding neuron \(i\)'s statistics. Batch norm: the same channel across the mini-batch. Layer norm: the whole layer, one sample.
\(s_i = S(R_i)\) — whatever numbers the operation needs. Usually a centre \(\mu_i\) and a scale \(\sigma_i\).
\(N(x_i, s_i)\) applied to the neuron. Throughout this paper it stays the familiar \((x_i-\mu_i)/\sigma_i\).
Fixing NormOp and varying the other two sorts existing methods into three families, distinguished only by how much history the reference set covers:
The learnable shift and gain that batch norm bolts on are pulled out into a separate layer here, applied identically after every normalization variant, so comparisons aren't confounded by them.
Accumulating statistics over all past samples creates an awkward dependency: today's activation depends on every sample ever seen. Exact backpropagation through that history is not just expensive, it's incoherent — you cannot backpropagate past a weight update you've already applied, and earlier mini-batches are usually gone. So the paper drops exactness and uses two matched heuristics instead.
Each layer keeps a short-term and a long-term estimate. The short-term one, \(\hat{s}_{short}\), is the exact average of the batch statistics observed since the last weight update. The long-term one is an exponential average that only moves when the weights move:
and the value actually used to normalize is a blend of the two:
At the update, the counter resets and \(\hat{s}_{short}\) is cleared. Before testing, it isn't: the final weight update is skipped and the layer keeps normalizing with the statistics inherited from the last training batch, which works fine in practice.
The same treatment applies on the way back. A second table holds short- and long-term estimates of \(\partial E/\partial\hat{s}\), updated by the same exponential rule with \(\kappa_3,\kappa_4\), and the gradient passed further down is a three-way blend that includes the current one:
One small training change makes the rest work. Conventionally, gradients accumulate over a mini-batch and the weights update immediately. Decouple the two: accumulate over \(n\) mini-batches of \(m\) samples each, then update once and clear. Two knobs, samples per batch \(m\) and batches per update \(n\), with \(n=1\) recovering ordinary training and \(m=1\) giving pure online learning.
This is not the same as one large batch of \(m \times n\) samples, because each mini-batch is normalized on arrival and cannot be revisited. On its own it already rescues much of batch norm's collapse at small batch sizes. For recurrent streaming normalization, \(n=2\) is often enough and often better than \(n=1\): the first mini-batch sweeps up statistics from all timesteps so the second is normalized against something stable.
The centre \(\mu\) is always the mean of the reference set. The divisor \(\sigma\) is more open than convention suggests. Take the \(p\)-th root of the \(p\)-th absolute moment about a point \(c\):
Three choices of \(c\) give three settings: the mean of the reference set (A), a running estimate of the mean (B), or zero (C). Settings B and C survive online learning, where A degenerates to \(\sigma = 0\) with a single sample; with a reasonable batch, A and B behave alike.
Setting A with \(p=2\) is the standard deviation, which is what batch norm and layer norm already use. The interesting case is \(p=1\), where \(\sigma\) is just the mean absolute value. Across the experiments it tracks \(p=2\) almost exactly while being cheaper to compute, easier to implement, and — the authors' motivating reason — trivially differentiable, which matters if you care about hardware or about neurons. Moments up to \(p=7\) still train; only the highest one slips.
Because Lp only changes the statistic function \(S(\cdot)\), it drops into layer normalization, batch normalization, or any of the reference-set variants independently of the streaming machinery.
The recurrent extension is the part that needs no new mechanism. Sample normalization already generalizes, since it never looks beyond the current timestep. Recurrent GBN is the time-specific approach: the unrolled network gets one GBN layer per step, each with its own memory. Recurrent streaming normalization instead uses the layer in the rolled network — all unrolled copies share one set of running estimates, so the stream runs over past samples and past timesteps together.
One caveat, stated and then measured: the estimates drift slightly as time advances, so by the time backpropagation returns to a layer, its statistics are no longer quite the ones used in the forward pass. Empirically this doesn't hurt, and \(n>1\) damps it further.
Applied to a GRU, every matrix product gets normalized before it is combined — input and hidden paths separately:
The plain normalized RNN follows the same pattern: \(h_t = \mathrm{NonLinear}(\mathrm{Norm}(W_x x_t) + \mathrm{Norm}(W_h h_{t-1}))\). Experiments used tanh; ReLU also worked.
Everything runs on CIFAR-10 in MatConvNet, across four architectures: a feedforward fully-connected net, a feedforward convnet, a ResNet-like convolutional RNN, and a densely recurrent convnet. Learning rate 0.1 for 25 epochs then 0.01 for 5, momentum 0.9.
Reported settings for the streaming runs: \(\alpha_1 = \beta_1 = 0.7,\ \alpha_2 = 0.3\) for the recurrent and online experiments, \(0.5/0.5\) for the feedforward ones, with \(\beta_3\) either 0 or 0.3.
One practical warning. Because backpropagation here is deliberately inexact, gradient checking the full model will fail. Check the model without the layer, then add a correct implementation — or reduce the layer to standard batch norm using the hyperparameters above and gradient check that.
The figures are worth opening; this page summarizes them rather than reproducing them.
The paper's experiments were built in MatConvNet. A later PyTorch port lives at github.com/liaoq/StreamingNorm, exposing StreamingNorm1d and StreamingNorm2d as drop-in replacements for the corresponding BatchNorm layers, plus a CIFAR-10 training script that compares against batch, layer, and group normalization.
Two differences from the paper are worth knowing before you read its defaults. The port folds the current batch statistic into the forward blend as a third term, \(\hat{s} = \alpha_1\hat{s}_{long} + \alpha_2\hat{s}_{short} + \alpha_3 s\), mirroring the \(\beta_3\) term the paper already had on the backward pass; setting \(\alpha_3 = 0\) recovers the published formulation. It also detects weight updates from the optimizer automatically, and implements L2 only, so the L1 result above isn't exercised by that code.
Its README also flags a practical failure mode the paper doesn't dwell on: NaNs early in training usually mean the streaming buffers haven't seen enough data to form a usable estimate yet, and a few warm-up mini-batches fix it rather than indicating anything wrong with the method.
@article{liao2016streaming,
title={Streaming normalization: Towards simpler and more biologically-plausible normalizations for online and recurrent learning},
author={Liao, Qianli and Kawaguchi, Kenji and Poggio, Tomaso},
journal={arXiv preprint arXiv:1610.06160},
year={2016}
}