Scattering transform#

Some papers involving the scattering transform and similar developments bringing structure to replace learned filters.

About this chapter

The scattering transform builds a convolutional network whose filters are all fixed wavelets: it cascades wavelet transforms with a modulus and averaging, which gives a representation that is invariant to translation. The appeal is that every filter is defined, so the network is easier to interpret and analyze.

The page goes roughly in order: reviews, the papers that introduce the transform, hybrid networks that put learned layers (often 1x1 convolutions) on top of scattering coefficients, related work on group equivariance, wavelets with learned parameters, and the Bölcskei lab’s theory for general frames. The last section holds unrelated nanoscience notes, background on \(L^2\) functions, and invertible models.

  • some of the researchers involved

    • edouard oyallan, joan bruna, stephan mallat, Helmut Bölcskei, max welling

Goals#

Why fix the filters at all: every filter is a known wavelet, so the network is easier to interpret and closer to biology. The transform cascades wavelet transforms with a modulus and averages the result, which makes the output invariant to translation.

  • benefits

    • all filters are defined

    • more interpretable

    • more biophysically plausible

  • scattering transform - computes a translation invariant repr. by cascading wavelet transforms and modulus pooling operators, which average the amplitude of iterated wavelet coefficients

Review-type#

Initial papers#

Mallat’s group invariant scattering paper introduces the transform; Bruna and Mallat then implement it as a CNN with fixed filters.

Scat_conv#

Hybrid networks: scattering coefficients at the bottom, learned layers on top. A recurring finding is that fixed scattering is good enough for the early layers.

  • Deep roto-translation scattering for object classification (oyallan & mallat 2014)

    • use 1x1 conv on top of scattering coefs (only 1 layer)

    • can capture rounded figures

    • can further impose robustness to rotation variability (although not full rotation invariance)

    • Deep learning in the wavelet domain (cotter & kingbury, 2017)- each conv layer is replaced by scattering transform + 1x1 conv

  • Visualizing and improving scattering networks (cotter et al. 2017)

    • add deconvnet to visualize

  • Scattering Networks for Hybrid Representation Learning (oyallon et al. 2018)

    • using early layers scat is good enough

  • i-RevNet: Deep Invertible Networks (jacobsen et al. 2018)

  • Scaling the scattering transform: Deep hybrid networks (oyallon et al. 2017)

    • use 1x1 convolutions to collapse accross channels

  • jacobsen_17 “Hierarchical Attribute CNNs”

    • modularity

  • cheng_16 “Deep Haar scattering networks”

  • Deep Network Classification by Scattering and Homotopy Dictionary Learning (zarka et al. 2019) - scat followed by sparse coding then linear

Neuro style#

  • https://arxiv.org/pdf/1809.10504.pdf

Papers by other groups#

Related ideas from outside Mallat’s group: group equivariant convolutions that share more weights than ordinary ones, encoder-decoder networks with interpretable transformations, and a theory paper on the invariance and stability of convolutional representations.

Wavelet style transfer#

Adaptive wavelet papers#

A middle ground between fixed and learned filters: wavelets whose parameters are trained with the rest of the network.

  • Parameterized Wavelets for Convolutional Neural Networks (2020) - a discrete wavelet CNN

  • An End-to-End Multi-Level Wavelet Convolutional Neural Networks for heart diseases diagnosis (el bouny et al. 2020) - stationary wavelet CNN

  • Fully Learnable Deep Wavelet Transform for Unsupervised Monitoring of High-Frequency Time Series (michau et al. 2021)

Helmut lab papers#

Theory that generalizes the scattering network: any semi-discrete frame as filters, any Lipschitz-continuous nonlinearity and pooling, possibly different in each layer. Features become more translation-invariant with depth.

  • Deep Convolutional Neural Networks Based on Semi-Discrete Frames (wiatowski et al. 2015)

    • allowing for different and, most importantly, general semidiscrete frames (such as, e.g., Gabor frames, wavelets, curvelets, shearlets, ridgelets) in distinct network layers

    • translation-invariant, and we develop deformation stability results

  • wiatoski_18 “A mathematical theory of deep convolutional neural networks for feature extraction”

    • encompasses general convolutional transforms - general semi-discrete frames (including Weyl-Heisenberg filters, curvelets, shearlets, ridgelets, wavelets, and learned filters), general Lipschitz-continuous non-linearities (e.g., rectified linear units, shifted logistic sigmoids, hyperbolic tangents, and modulus functions), and general Lipschitz-continuous pooling operators emulating, e.g., sub-sampling and averaging

      • all of these elements can be different in different network layers.

    • translation invariance result of vertical nature in the sense of the features becoming progressively more translation-invariant with increasing network depth

    • deformation sensitivity bounds that apply to signal classes such as, e.g., band-limited functions, cartoon functions, and Lipschitz functions.

  • wiatowski_18 “Energy Propagation in Deep Convolutional Neural Networks” (wiatowski et al. 2017)

Nano papers#

Notes on nanoengineering and microscopy papers, separate from the rest of the page. The subsections below are math background (\(L^2\) functions, Hilbert spaces) and invertible models.

  • yu_06 “A Nanoengineering Approach to Regulate the Lateral Heterogeneity of Self-Assembled Monolayers”

    • regulate heterogeneity of self-assembled monlayers

      • used nanografting + self-assembly chemistry

  • bu_10 nanografting - makes more homogenous morphology

  • fleming_09 “dendrimers”

    • scanning tunneling microscopy - provides highest spatial res

    • combat this for insulators

  • lin_12_moire

    • prob moire effect with near-field scanning optical microscopy

  • chen_12_crystallization

L2 functions#

Definitions the theory papers assume: square-integrable functions, measures, Hilbert spaces and diffeomorphisms.

  • \(L^2\) function is a function \(f: X \to \mathbb{R}\) that is square integrable: \(\vert f\vert ^2 = \int_X \vert f\vert ^2 d\mu\) with respect to the measure \(\mu\)

    • \(\vert f\vert\) is its \(L^2\)-norm

  • measure = nonnegative real function from a delta-ring F such that \(m(\empty) = 0\) and \(m(A) = \sum_n m(A_n)\)

  • Hilbert space H: a vectors space with an innor product \(<f, g>\) such that the following norm turns H into a complete metric space: \(\vert f\vert = \sqrt{<f, f>}\)

  • diffeomorphism is an isomorphism of smooth manifolds. It is an invertible function that maps one differentiable manifold to another such that both the function and its inverse are smooth.

Reversible/invertible models#

Key ideas

  • The scattering transform is a CNN whose filters are all fixed wavelets, combined with a modulus and averaging, and its output is invariant to translation.

  • Hybrid networks put learned layers, often 1x1 convolutions, on top of scattering coefficients; fixed scattering is good enough for the early layers.

  • Adaptive wavelets sit between fixed and learned filters by training the wavelet parameters with the rest of the network.

  • The Bölcskei lab’s theory extends the guarantees to general frames, Lipschitz nonlinearities and pooling, with features that become more translation-invariant with depth.