Disease#

Alzheimer's and Parkinson's, then pathology and cancer grading, with papers on deep learning for whole-slide images.

About this chapter

Two short neurological sections come first. Alzheimer’s starts in the hippocampus and the cholinergic basal forebrain, with amyloid plaques and tau tangles as its markers; Parkinson’s is the loss of dopamine neurons in the substantia nigra (the motor chapter explains why that stops movement).

The rest is about pathology, written from the point of view of building AI tools for it: how a biopsy becomes a stained slide and a written report, how tumors are staged and graded (TNM, Gleason scores for prostate, invasiveness and grade for bladder), how much pathologists disagree with each other, and the papers that train models on very large whole-slide images with only slide-level labels.

  • ECT - shock therapy given to brain when patient is struggling with severe mental illness (~100k patients in the US / year)

Alzheimer’s#

The figure lays out the amyloid hypothesis as a loop: Aβ42 builds up, triggers inflammation, the injury makes more Aβ42, and so on. Most people think plaques and tangles are left behind by dead cells; some think they cause the disease.

  • The amyloid cascade: sporadic causes or APP/PS1/PS2 alleles raise Aβ42, which causes inflammation and injury that produce more Aβ42. The notes beside it ask whether anti-inflammatory or Aβ42-blocking

    Fig. 136 Figure 1. The amyloid cascade: sporadic causes or APP/PS1/PS2 alleles raise Aβ42, which causes inflammation and injury that produce more Aβ42. The notes beside it ask whether anti-inflammatory or Aβ42-blocking treatment comes too late, and list cholinergic drugs for managing symptoms.#

  • overview

    • age-associated - tons of people get it

    • doesn’t kill you, secondary complications like pneumonia will kill you

    • rate is going up

    • very expensive to treat

  • declarative memories are affected by Alzheimer’s

    • these are memories that you know

  • first 2 areas to go in Alzheimer’s

    1. hippocampus

      • patient HM had no hippocampus

      • hippocampus stores 1 day of info

        • offloading occurs during sleep (REM sleep) to prefrontal cortex, temporal lobe, V4

        • dreaming - might see images as you are offloading

    2. basal forebrain - spread synapses all over cortex

      • uses Ach

      • ignition key for entire cortex

  • alzheimer’s characteristics only found in autopsy

    • amyloid plaques

      • maybe A-beta causes it

      • A-beta comes from APP

      • A-beta42 binds to itself

        • prion (starts making more of itself)

        • this cycle could be exacerbated by injury

        • clumps and attracts immune system which kills local important cells

          • this could cause Alzheimer’s

        • rare genetic mutations in A-beta increase probability you get Alzheimer’s

        • anti-inflammation may be too late

        • can take drugs that increase Ach functions - ex. cholinergic agonists, cholinesterase inhibitors

    • tangles

      • tangles made of protein called Tau

    • most people think these are just dead cells resulting from Alzheimer’s but some think they cause it

Parkinson’s#

Parkinson’s is the loss of dopamine neurons in the substantia nigra pars compacta, with symptoms once about 20% are left. Its Lewy bodies are clumps of alpha-synuclein, like the Aβ42 clumps of Alzheimer’s.

  • loss of substantia nigra pars compacta dopaminergic neurons

    • when you get down to 20% what you were born with

    • dopaminergic neurons form melanin = dark color

    • hits to head can give inflammation

  • know what they need to do - don’t have enough dopamine to act

  • treat with L Dopa -> something like dopamine -> take out globus pallidus

  • Lewy bodies are clumps of alpha synuclein - appear at dopaminergic synapses

    • clumps like A-beta42

    • associated with early-onset Parkinson’s (rare) associated with genetic mutations

  • bradykinesia - slowness of movement

  • age can give parksinson’s

  • no evidence that toxins can induce parkinsons

  • PTP/ pesticides can induce Parkinson’s in test animals

  • 1/500 people

Pathology#

The rest of the chapter is about tissue: how slides are made and read, the deep learning papers built on them, and a public dataset.

Basics#

The steps from tissue to diagnosis, and the vocabulary used in the papers below: H&E vs. immunohistochemistry stains, whole-slide images (WSI) and tissue microarrays (TMA).

  • pathologists work with tissue samples either visually or chemically

    • anatomic pathology relies on the microscope whereas clinical pathology does not

  • pathologists convert from tissue image into written report

  • when case is challenging, may require a second opinion (v rare)

  • steps (process takes 9-12 hrs)

    From biopsy to digital slide: the tissue is collected, cut into cassettes, fixed and embedded in paraffin, sectioned a few microns thick, stained, and scanned.

    Fig. 137 Figure 2. From biopsy to digital slide: the tissue is collected, cut into cassettes, fixed and embedded in paraffin, sectioned a few microns thick, stained, and scanned.#

    • tissue is surgically removed

      • more tissue collected is generally better (gives more context)

      • this procedure is called a biopsy

      • much is written down at this step (e.g. race, gender, locations in organ, different tumors in an organ) that can’t be seen in slide alone

    • fixation: keeps the tissue stable (preserves dna also) - basicallly just soak in formalin

    • dissection: remove the relevant part of the tissue

    • tissue processor - removes water in tissue and substitute with wax (parafin) - hardens it and makes it easy to cut into thin strips

    • microtone - cuts very thin slices of the tissue (2-3 microns)

    • staining

      • H & E - hematoxylin and eosin stain - most popular (~80%) - colors the cells in a specific way, bc cells are usually pretty transparent

        • hematoxylin stains nucleic acids blue

        • eosin stains proteins / cytoplasm pink/red

      • immunohistochemistry (IHC) - tries to identify cell lineage: 10-15%

        • identifies targets

        • use antibodies tagged with chromophores to tag tissues

      • gram stain - highlights bacteria

      • giemsa - microorganisms

      • others…for muscle, fungi

    • viewing

      • usually analog - put slide on something that can move / rotate

      • whole-slide image (WSI) - resulting entire slide

        • tissue microarray (TMA) - smaller, fits many samples onto the same slide

      • with paige: put slide through digital scanner (only 5% or so of slides are currently digital)

    • later on, board meets to decide on treatment (based on pathology report)

      • usually some discussion betweeon original imaging (pre-biopsy) and pathologist’s interpretation

    • resection - after initial diagnosis, often entire tumor is removed (resection)

  • how can ai help?

    • can help identify small things in large images

    • can help with conflict resolution

  • after (successful) neoadjuvant chemotherapy, problem becomes more difficult

    • very few remaining cancer cells

    • cancer/non-cancer cells become harder to distinguish (esp. for prostate)

    • tumor bed is patchily filled with cancer cells - need to better clarify presence of cancer

Papers#

The recurring problem is size. A whole-slide image has far too many pixels for one network pass, and labels usually exist only for the whole slide, so most methods classify small patches and then aggregate them, often with multiple-instance learning.

  • Deep Learning Models for Digital Pathology (bentaieb & hamarneh, 2019)

    • note: alternative to histopathology are more expensive / slower (e.g. molecular profiling)

    • to promote consistency and objective inter-observer agreement, most pathologists are trained to follow simple algorithmic decision rules that sufficiently stratify patients into reproducible groups based on tumor type and aggressiveness

    • magnification usually given in microns per pixel

    • WSI files are much larger than other digital images (e.g. for radiology)

    • DNNs can be used for many tasks: beyond just classification, there are subtasks (e.g. count histological primitives, like nuclei) and preprocessing tasks (e.g. stain normalization)

    • challenge: multi-magnification + high dimensions (i.e. millions of pixels)

      • people usually extract smaller patches and train on these

        • this loses larger context

        • one soln: pyramid representation: extract patches at different magnification levels

        • one soln: stacked CNN - train fully-conv net, then remove linear layer, freeze, and train another fully-conv net on the activations (so it now has larger receptive field)

        • one soln: use 2D LSTM to aggregate patch reprs.

      • challenge: annotations only at the entire-slide level, but must figure out how to train individual patches

        • e.g. use aggregation techniques on patches - extract patch-wise features then do smth simple, like random forest

        • e.g. treat as weak labels or do multiple-instance learning

          • could just give slide-level label to all patches then vote

      • can use transfer learning from related domains with more labels

    • challenge: class imbalance

      • can use boosting approach to increase the likelihood of sampling patches that were originally incorrectly classified by the model

    • challenge: need to integrate in other info, such as genomics

    • when predicting histological primitives, often predict pixel-wise probability maps, then look for local maxima

      • can also integrated domain-knowledge features

      • can also have 2 paths, one making bounding-box proposals and another predicting the probability of a class

      • alternatively, can formulate as a regression task, where pixelwise prediction tells distance to nearest centroid of object

      • could also just directly predict the count

    • can also predict survival analysis

  • Clinical-grade computational pathology using weakly supervised deep learning on whole slide images (campanella et al. 2019)

    • use slide-level diagnosis as “weak supervision” for all contained patches

    • 1st step: train patch-level CNNs using MIL

      • if label is 0, then all patches should be 0

      • if label is 1, then only pass gradients to the top-k predicted patches

    • 2nd step: use RNN (or another net) to combine info across S most suspicious tiles

  • Human-interpretable image features derived from densely mapped cancer pathology slides predict diverse molecular phenotypes (diao et al. 21)

  • An artificial intelligence algorithm for prostate cancer diagnosis in whole slide images of core needle biopsies: a blinded clinical validation and deployment study (pantanowitz et al. 2020 - ibex)

    • 549 train, 2501 internal test slides, 1627 external validation

    • predict cancer prob., gleason score 7-10, gleason pattern 5, perneural invasion, cancer percentage

    • algorithm

      • GB classifies background / non-background / blurry using hand-extracted features for each tile

      • each tile gets predicted probability for 18 pre-defined classes (e.g. GP 3)

        • ensemble of 3 CNNs that operate at different magnifications

      • aggregation: 18-probability heatmaps are combined to calculate slide-level scores

        • ex (for predicting cancer): sum the cancer-related channels in the heatmap, apply 2x2 local averaging, then take max

Datasets#

  • ARCH - multiple instance captioning dataset to facilitate dense supervision of CP tasks

Cancer#

Overview#

Background for the two cancers below. Staging (pTNM: tumor size, lymph nodes, distant metastases) says where a cancer is and how far it has spread; prognosis is the chance of recovery.

  • tumor = neoplasm - a mass formation from an uncontrolled growth of cells

    • benign tumor - typically stays confined to the organ where it is present and does not cause functional damage

    • malignant tumor = cancer - comprises organ function and can spread to other organs (metastasis)

  • relation network based aggregator on patches

  • lymphatic system drains fluids (non-blood) from organs into lymph nodes

    • cancer often mestastasize through these

  • staging - describes where cancer is located and where it has spread

    • clinical staging - based on non-tissue things

    • pathological staging - elements of staging pTNM

      • size / depth of tumor “T”

      • number of lymph nodes / how many had cancer “N”

      • number of metastatic foci in non-lymph node organ “M”

      • these are combined to determine the cancer stage (0-4)

  • prognosis - chance of recovery

Treatments#

Chemotherapy kills replicating cells and is given after the tumor is removed (adjuvant) or before (neoadjuvant). Targeted therapies and immunotherapy aim at what makes cancer cells different, such as receptors like PD1 and CTLA4 that some tumors use to switch off immune cells.

  • chemo

    • traditional chemotherapy disrupts cell replication

      • hair loss and gastrointestinal symptoms occur bc these cells also rapidly replicate

    • adjuvant chemotherapy - after cancer is removed, most common

    • neoadjuvant chemo - after biopsy, but before resection (when very hard to remove)

  • targeted therapies

    • ex. address genetic aberration found in cancer cells

    • immunotherapy - enhance body’s immune response to cancer cells (so body will attack these cells on its own)

      • want the antigens on the tumor to be as different as possible (so they will be characterized as foreign)

      • to measure this, can conduct total mutational burden (TMB) or miscrosatellite instability (MSI) test

        • genetic tests - hard to do by looking at glass slide

      • some tumors express receptors (e.g. CTLA4, PD1) that shut off immune cells - some drugs try to block these receptors

Prostate cancer#

The Gleason score adds the patterns (3 to 5) of the two largest cancer regions. The human benchmarks show low agreement between pathologists (kappas mostly between 0.15 and 0.45), which is the baseline a model is compared with.

Interactive 1. Cohen's κ, the agreement measure in these benchmarks, corrects the raw agreement \(p_o\) for the agreement two raters would reach by chance, \(p_e\), given how often each calls each grade: \(\kappa = (p_o - p_e)/(1 - p_e)\). With a 50/50 mix of grades, 85% agreement gives κ = 0.70. Keep 85% and make pattern 4 the call on 90% of slides: chance agreement rises to 82% and κ falls to 0.17, inside the range the Gleason studies report. A low κ can come with high raw agreement when one grade dominates.
  • tests

    • feel with finger

    • antigen test - blood test

    • ultrasound - probe inserted

    • biopsy - needle inserted to take out tissue

  • grading

    • stages (they have subdivisions, e.g. IIA, IIB, IIC)

      • I - early, slow-growing

      • II - small, but risky

      • III - likely to spread

      • IV - has spread beyond the prostate

      • recurrent - has come back after treatment

    • in addition to stages 0-4, prostate cancer is also given Gleason score

      • look at 2 biggest cancer regions and identifies them as a Gleason pattern from 3 (best) to 5 (worst)

      • this results in a sum (e.g. 5+4, 3+4) - note 3+4 is not same as 4+3

  • treatments

    • prostatectomy - remove the prostate

    • radiation therapy - kills specifically cancer cells

    • radiative seed implants - implated into prostate to kill cancer cells

    • cryotherapy - kill prostate cancer cells by freezing them

    • hormone therapy - block hormone which grows prostate cancer cells

    • chemotherapy

  • human benchmarks

  • ai papers

    • Learning Whole-Slide Segmentation from Inexact and Incomplete Labels using Tissue Graphs (anklin et al. 2021)

      • SegGini, a weakly supervised segmentation method using graphs

        • constructs a tissue-graph for WSI (node is tissue region)

        • weakly-supervised segmentation via node classification

      • data

        • UZH dataset - 5 five TMAs with 886 spots (each 3100×3100 pixels) with complete pixel-level annotations and inexact image-level gradess

        • SICAPv2 dataset - 155 WSIs and 18,783 tiles of size 512×512 with complete pixel annotations

Bladder cancer#

Two things are judged: shape (papillary or flat) and grade (low or high). Shape is much easier to call than grade, and once a tumor is high grade the question becomes how far it has invaded.

  • tests

    • urinalysis - look for things like blood in urine

    • urine cytology - use microscope to look for cancer cells in urine

    • urine tests for specific tumor parkers

    • cystoscopy - invasive lens takes image of bladder

    • tests lead to a biopsy

  • grading

    • invasiveness: can be non-invasive, invasive (grows into deeper layers of bladder)

      • superficial = non-muscle invasive - hasn’t grown into main muscle layer of bladder

    • grade: again asigned stages 0 - IV based on TNM

      • low-grade = well-differentiated

      • high-grade (worse) = poorly differentiated, undifferentiated

  • human benchmark

    • The reliability of staging and grading of bladder tumours. Impact of misinformation on the pathologist’s diagnosis (olsen et al. 1993)

      • 4 consultant pathologists

      • 40 biopsy specimens of bladder tumours staging invasion

        • grading using Bergkvist classification

      • kappa < 0.50

  • ai papers

    • Bladder cancer in the time of machine learning: Intelligent tools for diagnosis and management (2021)

      • bladder cancel ranks tenth in worldwide absolute cancer incidence

    • non-pathology

    • cystoscopy - few DNN papers here

    • pathology

      • Urinary Bladder Tumor Grade Diagnosis Using Online Trained Neural Networks (2003)

        • 92 patients with BC

        • 90%, 94.9%, and 97.3%, for Grade I, II, and III respectively

        • builds on Neural network-based segmentation and classification system for automated grading of histologic sections of bladder carcinoma (2002)

      • Deep Learning Predicts Molecular Subtype of Muscle-invasive Bladder Cancer from Conventional Histopathological Slides (woerl et al. 2020) - predict molecular subtype using histopathology images in Cancer Genome Atlas Urothelial Bladder Carcinoma dataset

  • bladder basics

    • muscles in bladder contract and force urine out

      • urethelium - inner layer that is able to stretch (has many layers) - this is where cancer originates

        • in situ - cancer only here

        • invasive - goes into the muscle

          • if it goes into the urine, can easily test (also usually triggers blood in the urine)

      • biopsy usually looks mostly at urethelium and vessels right next to it (will not go all the way to the muscle, as this could puncture the bladder)

        • very targeted (unlike prostate biopsy), slide will come with some tag like “in area with redness” from scopy

          • 4 possibilities

            • big mass - should see cancer

            • inflammation - could be cancer or many other things (e.g. atypia vs carcinoma)

        • get many parts / sites of biopsies

H & E slide

  • shape:

papillary

flat

can also have a combo

pap_blad

flat_blad

  • grade:

low

high

low_grade_blad

high_grade_blad

  • when shape is flat, grade often can’t be determined reliably

    • lots of names for uncertain (e.g. upump - uncertain malignant potential, or atypia)

  • much easier to decide shape than grade

  • once you find high grade, look for invasiveness (and deeper layers are worse)

\(\,\)

Key ideas

  • Alzheimer’s hits the hippocampus and the cholinergic basal forebrain first; amyloid plaques and tau tangles are its markers, and whether they cause it or follow it is debated.

  • Parkinson’s is the loss of substantia nigra dopamine neurons, so patients know what to do but cannot start the movement.

  • A slide goes from biopsy through fixation, paraffin, sectioning and staining (mostly H&E) to a written report, and now sometimes to a scanned whole-slide image.

  • Whole-slide images are too large for one network pass and are usually labeled only per slide, so most methods classify patches and aggregate them, often with multiple-instance learning.

  • Pathologists often disagree on grades (Gleason kappas mostly 0.15 to 0.45), which sets the bar a model is compared with.