thinking about interpretability in science and medicine 🤔
senior researcher at microsoft research (deep learning group); phd from berkeley (with the amazing prof. bin yu)
thinking about interpretability in science and medicine 🤔
senior researcher at microsoft research (deep learning group); phd from berkeley (with the amazing prof. bin yu)
🔎 My research focuses on how we can build trustworthy machine-learning systems by making them interpretable. In my work, interpretability is grounded seriously via close collaboration with domain experts, e.g. medical doctors or cell biologists. These collaborations have given rise to useful methodology, roughly split into two areas: (1) building more effective transparent models and (2) improving the trustworthiness of black-box models. Going forward, I hope to help bridge the gap between transparent models and black-box models to improve real-world healthcare.
🌳 Whenever possible, building transparent models is the most effective route towards ensuring interpretability. Transparent models are interpretable by design, including models such as (concise) decision trees, rule lists, and linear models. My work in this area was largely motivated by the problem of clinical decision-rule development. Clinical decision rules (especially those used in emergency medicine), need to be extremely transparent so they can be readily audited and used by physicians making split-second decisions. To this end, we have developed methodology for enhancing decision trees. For example, replacing the standard CART algorithm with a novel greedy algorithm for tree-sums can substantially improve predictive performance without sacrificing predictive performance. Additionally, hierarchical regularization can improve the predictions of an already fitted model without altering its interpretability. Despite their effectiveness, transparent models such as these often get overlooked in favor of black-box models; to address this issue, we've spent a lot of time curating imodels, an open-source package for fitting state-of-the-art transparent models.
🌀 My second line of my work focuses on interpreting and improving black-box models, such as neural networks, for the cases when a transparent model simply can't predict well enough. Here, I work closely on real-world problems such as analyzing imaging data from cell biology and cosmology. Interpretability in these contexts demands more nuanced information than standard notions of "feature importance" common in the literature. As a result, we have developed methods to characterize and summarize the interactions in a neural network, particularly in transformed domains (such as the Fourier domain), where domain interpretations can be more natural. I'm particularly interested in how we can ensure that these interpretations are useful, either by using them to embed prior knowledge into a model or identify when it can be trusted.
🤝 There is a lot more work to do on bridging the gap between transparent models and black-box models in the real world. One promising avenue is distillation, whereby we can use a black-box model to build a better transparent model. For example, in one work we were able to distill state-of-the-art neural networks in cell-biology and cosmology into transparent wavelet models with <40 parameters. Despite this huge size reduction, these models actually improve prediction performance. By incorporating close domain knowledge into models and the way we approach problems, I believe interpretability can help unlock many benefits of machine-learning for improving healthcare and science.
| year | title | authors | tags | paper | code | misc |
|---|---|---|---|---|---|---|
| '22 | Emb-GAM: an interpretable and efficient predictor using pre-trained language models | singh & gao | 🔎🌀🌳 | arXiv | ||
| '22 | Explaining patterns in data with language models via interpretable autoprompting | singh*, morris*, aneja, rush, & gao | 🔎🌀 | preprint | ||
| '22 | Interpretable deep learning for accurate molecular partner prediction in clathrin-mediated endocytosis | singh*, li*, et al. | 🔎🌀🦠 | in prep | ||
| '22 | Predictability and stability testing to assess clinical decision instrument performance for children after blunt torso trauma | kornblith*, singh* et al. | 🔎🌳💊 | PLOS digital health | ||
| '22 | Fast interpretable greedy-tree sums (FIGS) | tan*, singh*, nasseri, agarwal, & yu | 🔎🌳 | arxiv | ||
| '22 | Hierarchical shrinkage: improving accuracy and interpretability of tree-based methods | agarwal*, tan*, ronen, singh, & yu | 🔎🌳 | icml (spotlight) | ||
| '22 | VeridicalFlow: a python package for building trustworthy data science pipelines with PCS | duncan*, kapoor*, agarwal*, singh*, & yu | 💻🔍 | joss | ||
| '21 | imodels: a python package for fitting interpretable models | singh*, nasseri*, et al. | 💻🔍🌳 | joss | ||
| '21 | Adaptive wavelet distillation from neural networks through interpretations | ha, singh, et al. | 🔍🌀🌳 | neurips | ||
| '21 | Matched sample selection with GANs for mitigating attribute confounding | singh, balakrishnan, & perona | 🌀 | cvpr workshop | ||
| '21 | Revisiting complexity and the bias-variance tradeoff | dwivedi*, singh*, yu & wainwright | 🌀 | topml workshop | ||
| '20 | Curating a COVID-19 data repository and forecasting county-level death counts in the United States | altieri et al. | 🔎🦠 | hdsr | ||
| '20 | Transformation importance with applications to cosmology | singh*, ha*, lanusse, boehm, liu & yu | 🔎🌀🌌 | iclr workshop (spotlight) | ||
| '20 | Interpretations are useful: penalizing explanations to align neural networks with prior knowledge | rieger, singh, murdoch & yu | 🔎🌀 | icml | ||
| '19 | Hierarchical interpretations for neural network predictions | Singh*, Murdoch*, & Yu | 🔍🌀 | ICLR | ||
| '19 | interpretable machine learning: definitions, methods, and applications | Murdoch*, Singh*, et al. | 🔍🌳🌀 | pnas | ||
| '19 | disentangled attribution curves for interpreting random forests and boosted trees | devlin, singh, murdoch & yu | 🔍🌳 | arxiv | ||
| '18 | large scale image segmentation with structured loss based deep learning for connectome reconstruction | Funke*, Tschopp*, et al. | 🧠🌀 | TPAMI | ||
| '18 | linearization of excitatory synaptic integration at no extra cost | Morel, Singh, & Levy | 🧠 | J Comp Neuro | ||
| '17 | a consensus layer V pyramidal neuron can sustain interpulse-interval coding | Singh & Levy | 🧠 | Plos One | ||
| '17 | a constrained, weighted-l1 minimization approach for joint discovery of heterogeneous neural connectivity graphs | Singh, Wang, & Qi | 🧠 | neurips Workshop | , |
Misc resources. There's also some dumb stuff here.
Notes in machine learning / neuroscience.
Fall 22 - Present
deep learning research
Fall 17 - spring 22
phd research in interpretable ml
Summer 21 - Summer 22
developed deep learning models for pathology
Summer 20
developed methods for causal benchmarking of fairness in computer vision models
Summer 19
developed methods for interpreting medical machine-learning models
Summer 17
Worked on unsupervised pretraining of CNNs for semantic segmentation
Fall 16 - Spring 17
Contributed to development of novel weighted-L1, multi-task Gaussian graphical models
Summer 14, 15, 16
Improved CNNs for neural-image segementation + explored dendritic backpropagation
Fall 14 - Fall 16
Simulated computational models of stochastic neurons
I've been lucky to be advised by / collaborate with many amazing people
It has been my pleasure to help advise some incredible undergrads at berkeley