Brendan J. Frey, Peter Dayan and Geoffrey E. Hinton 1997.
A simple algorithm that discovers efficient perceptual codes.
In M. Jenkin and L. R. Harris (editors),
Computational and Psychophysical Mechanisms of Visual Coding.
Cambridge University Press: New York, NY.
We describe the ``wake-sleep'' algorithm that allows a multilayer,
unsupervised, neural network to build a hierarchy of representations of
sensory input. The network has bottom-up ``recognition'' connections that are
used to convert sensory input into underlying representations. Unlike most
artificial neural networks, it also has top-down ``generative'' connections that
can be used to reconstruct the sensory input from the representations. In the ``wake''
phase of the learning algorithm, the network is driven by the bottom-up
recognition connections and the top-down generative connections are trained to
be better at reconstructing the sensory input from the representation
chosen by the recognition process. In the ``sleep'' phase, the network is driven
top-down by the generative connections to produce a fantasized representation
and a fantasized sensory input. The recognition connections are then trained
to be better at recovering the fantasized representation from the fantasized
sensory input. In both phases, the synaptic learning rule is simple and
local. The combined effect of the two phases is to create representations of
the sensory input that are efficient in the following sense: On average, it
takes more bits to
describe each sensory input vector directly than to first describe the
representation of the sensory input chosen by the recognition process and
then describe the difference between the
sensory input and its reconstruction from the chosen representation.
Compressed postscript,
uncompressed postscript.
Back to Brendan Frey's home page.