Scattering transform#
Some papers involving the scattering transform and similar developments bringing structure to replace learned filters.
About this chapter
The scattering transform builds a convolutional network whose filters are all fixed wavelets: it cascades wavelet transforms with a modulus and averaging, which gives a representation that is invariant to translation. The appeal is that every filter is defined, so the network is easier to interpret and analyze.
The page goes roughly in order: reviews, the papers that introduce the transform, hybrid networks that put learned layers (often 1x1 convolutions) on top of scattering coefficients, related work on group equivariance, wavelets with learned parameters, and the Bölcskei lab’s theory for general frames. The last section holds unrelated nanoscience notes, background on \(L^2\) functions, and invertible models.
some of the researchers involved
edouard oyallan, joan bruna, stephan mallat, Helmut Bölcskei, max welling
Goals#
Why fix the filters at all: every filter is a known wavelet, so the network is easier to interpret and closer to biology. The transform cascades wavelet transforms with a modulus and averages the result, which makes the output invariant to translation.
benefits
all filters are defined
more interpretable
more biophysically plausible
scattering transform - computes a translation invariant repr. by cascading wavelet transforms and modulus pooling operators, which average the amplitude of iterated wavelet coefficients
Review-type#
Understanding deep convolutional networks (mallat 2016)
Mathematics of deep learning (vidal et al. 2017)
Geometric deep learning: going beyond euclidean data (bronstein et al. 2017)
Initial papers#
Mallat’s group invariant scattering paper introduces the transform; Bruna and Mallat then implement it as a CNN with fixed filters.
classification with scattering operators (bruna & mallat 2010)
recursive interferometric repr. (mallat 2010)
group invariant scattering (mallat 2012)
introduces scat transform
Generic deep networks with wavelet scattering (oyallan et al. 2013)
Invariant Scattering Convolution Networks (bruna & mallat 2012)
introduces the scattering transform implemented as a cnn
Deep scattering spectrum (anden & mallat 2013)
Scat_conv#
Hybrid networks: scattering coefficients at the bottom, learned layers on top. A recurring finding is that fixed scattering is good enough for the early layers.
Deep roto-translation scattering for object classification (oyallan & mallat 2014)
use 1x1 conv on top of scattering coefs (only 1 layer)
can capture rounded figures
can further impose robustness to rotation variability (although not full rotation invariance)
Deep learning in the wavelet domain (cotter & kingbury, 2017)- each conv layer is replaced by scattering transform + 1x1 conv
Visualizing and improving scattering networks (cotter et al. 2017)
add deconvnet to visualize
Scattering Networks for Hybrid Representation Learning (oyallon et al. 2018)
using early layers scat is good enough
i-RevNet: Deep Invertible Networks (jacobsen et al. 2018)
Scaling the scattering transform: Deep hybrid networks (oyallon et al. 2017)
use 1x1 convolutions to collapse accross channels
jacobsen_17 “Hierarchical Attribute CNNs”
modularity
cheng_16 “Deep Haar scattering networks”
Deep Network Classification by Scattering and Homotopy Dictionary Learning (zarka et al. 2019) - scat followed by sparse coding then linear
Neuro style#
https://arxiv.org/pdf/1809.10504.pdf
Papers by other groups#
Related ideas from outside Mallat’s group: group equivariant convolutions that share more weights than ordinary ones, encoder-decoder networks with interpretable transformations, and a theory paper on the invariance and stability of convolutional representations.
cohen_16 “Group equivariant convolutional networks”
introduce G-convolutions which share more wieghts than normal conv
worrall_17 “Interpretable transformations with encoder-decoder networks”
look at interpretability
bietti_17 “Invariance and stability of deep convolutional representations”
theory paper
Wavelet style transfer#
Photorealistic Style Transfer via Wavelet Transforms> (yoo et al. 2019)
Adaptive wavelet papers#
A middle ground between fixed and learned filters: wavelets whose parameters are trained with the rest of the network.
Parameterized Wavelets for Convolutional Neural Networks (2020) - a discrete wavelet CNN
An End-to-End Multi-Level Wavelet Convolutional Neural Networks for heart diseases diagnosis (el bouny et al. 2020) - stationary wavelet CNN
Fully Learnable Deep Wavelet Transform for Unsupervised Monitoring of High-Frequency Time Series (michau et al. 2021)
Helmut lab papers#
Theory that generalizes the scattering network: any semi-discrete frame as filters, any Lipschitz-continuous nonlinearity and pooling, possibly different in each layer. Features become more translation-invariant with depth.
Deep Convolutional Neural Networks Based on Semi-Discrete Frames (wiatowski et al. 2015)
allowing for different and, most importantly, general semidiscrete frames (such as, e.g., Gabor frames, wavelets, curvelets, shearlets, ridgelets) in distinct network layers
translation-invariant, and we develop deformation stability results
wiatoski_18 “A mathematical theory of deep convolutional neural networks for feature extraction”
encompasses general convolutional transforms - general semi-discrete frames (including Weyl-Heisenberg filters, curvelets, shearlets, ridgelets, wavelets, and learned filters), general Lipschitz-continuous non-linearities (e.g., rectified linear units, shifted logistic sigmoids, hyperbolic tangents, and modulus functions), and general Lipschitz-continuous pooling operators emulating, e.g., sub-sampling and averaging
all of these elements can be different in different network layers.
translation invariance result of vertical nature in the sense of the features becoming progressively more translation-invariant with increasing network depth
deformation sensitivity bounds that apply to signal classes such as, e.g., band-limited functions, cartoon functions, and Lipschitz functions.
wiatowski_18 “Energy Propagation in Deep Convolutional Neural Networks” (wiatowski et al. 2017)
Nano papers#
Notes on nanoengineering and microscopy papers, separate from the rest of the page. The subsections below are math background (\(L^2\) functions, Hilbert spaces) and invertible models.
yu_06 “A Nanoengineering Approach to Regulate the Lateral Heterogeneity of Self-Assembled Monolayers”
regulate heterogeneity of self-assembled monlayers
used nanografting + self-assembly chemistry
bu_10 nanografting - makes more homogenous morphology
fleming_09 “dendrimers”
scanning tunneling microscopy - provides highest spatial res
combat this for insulators
lin_12_moire
prob moire effect with near-field scanning optical microscopy
chen_12_crystallization
L2 functions#
Definitions the theory papers assume: square-integrable functions, measures, Hilbert spaces and diffeomorphisms.
\(L^2\) function is a function \(f: X \to \mathbb{R}\) that is square integrable: \(\vert f\vert ^2 = \int_X \vert f\vert ^2 d\mu\) with respect to the measure \(\mu\)
\(\vert f\vert\) is its \(L^2\)-norm
measure = nonnegative real function from a delta-ring F such that \(m(\empty) = 0\) and \(m(A) = \sum_n m(A_n)\)
Hilbert space H: a vectors space with an innor product \(<f, g>\) such that the following norm turns H into a complete metric space: \(\vert f\vert = \sqrt{<f, f>}\)
diffeomorphism is an isomorphism of smooth manifolds. It is an invertible function that maps one differentiable manifold to another such that both the function and its inverse are smooth.
Reversible/invertible models#
williams inverting w/ autoregressive models (nash et al. 2018)
iResNet
Key ideas
The scattering transform is a CNN whose filters are all fixed wavelets, combined with a modulus and averaging, and its output is invariant to translation.
Hybrid networks put learned layers, often 1x1 convolutions, on top of scattering coefficients; fixed scattering is good enough for the early layers.
Adaptive wavelets sit between fixed and learned filters by training the wavelet parameters with the rest of the network.
The Bölcskei lab’s theory extends the guarantees to general frames, Lipschitz nonlinearities and pooling, with features that become more translation-invariant with depth.