thinking about interpretability in neuroscience and medicine 🤔


senior researcher at microsoft research (deep learning group); phd from berkeley (with prof. bin yu)

Research
Read research overview

🔎 My research focuses on how we can build trustworthy machine-learning systems by making them interpretable. In my work, interpretability is grounded seriously via close collaboration with domain experts, e.g. medical doctors or cell biologists. These collaborations have given rise to useful methodology, roughly split into two areas: (1) building more effective transparent models and (2) improving the trustworthiness of black-box models. Going forward, I hope to help bridge the gap between transparent models and black-box models to improve real-world healthcare.

🌳 Whenever possible, building transparent models is the most effective route towards ensuring interpretability. Transparent models are interpretable by design, including models such as (concise) decision trees, rule lists, and linear models. My work in this area was largely motivated by the problem of clinical decision-rule development. Clinical decision rules (especially those used in emergency medicine), need to be extremely transparent so they can be readily audited and used by physicians making split-second decisions. To this end, we have developed methodology for enhancing decision trees. For example, replacing the standard CART algorithm with a novel greedy algorithm for tree-sums can substantially improve predictive performance without sacrificing predictive performance. Additionally, hierarchical regularization can improve the predictions of an already fitted model without altering its interpretability. Despite their effectiveness, transparent models such as these often get overlooked in favor of black-box models; to address this issue, we've spent a lot of time curating imodels, an open-source package for fitting state-of-the-art transparent models.

🌀 My second line of my work focuses on interpreting and improving black-box models, such as neural networks, for the cases when a transparent model simply can't predict well enough. Here, I work closely on real-world problems such as analyzing imaging data from cell biology and cosmology. Interpretability in these contexts demands more nuanced information than standard notions of "feature importance" common in the literature. As a result, we have developed methods to characterize and summarize the interactions in a neural network, particularly in transformed domains (such as the Fourier domain), where domain interpretations can be more natural. I'm particularly interested in how we can ensure that these interpretations are useful, either by using them to embed prior knowledge into a model or identify when it can be trusted.

🤝 There is a lot more work to do on bridging the gap between transparent models and black-box models in the real world. One promising avenue is distillation, whereby we can use a black-box model to build a better transparent model. For example, in one work we were able to distill state-of-the-art neural networks in cell-biology and cosmology into transparent wavelet models with <40 parameters. Despite this huge size reduction, these models actually improve prediction performance. By incorporating close domain knowledge into models and the way we approach problems, I believe interpretability can help unlock many benefits of machine-learning for improving healthcare and science.

year title authors tags paper code misc
'23 Explaining black box text modules in natural language with language models singh*, hsu*, antonello, jain, huth, yu & gao 🔎🌀 arXiv
'23 Augmenting Interpretable Models with LLMs during Training singh, askari, caruana, & gao 🔎🌀🌳 arXiv
'23 Self-Verification Improves Few-Shot Clinical Information Extraction gero*, singh*, cheng, naumann, galley, gao, & poon 🔎🌀💊 arXiv
'22 Explaining patterns in data with language models via interpretable autoprompting singh*, morris*, aneja, rush, & gao 🔎🌀 arXiv
'22 Interpretable deep learning for accurate molecular partner prediction in clathrin-mediated endocytosis singh*, li*, et al. 🔎🌀🦠 in prep
'22 Predictability and stability testing to assess clinical decision instrument performance for children after blunt torso trauma kornblith*, singh* et al. 🔎🌳💊 PLOS digital health
'22 Fast interpretable greedy-tree sums (FIGS) tan*, singh*, nasseri, agarwal, & yu 🔎🌳 arxiv
'22 Hierarchical shrinkage for trees: improving accuracy and interpretability of tree-based methods agarwal*, tan*, ronen, singh, & yu 🔎🌳 icml (spotlight)
'22 VeridicalFlow: a python package for building trustworthy data science pipelines with PCS duncan*, kapoor*, agarwal*, singh*, & yu 💻🔍 joss
'21 imodels: a python package for fitting interpretable models singh*, nasseri*, et al. 💻🔍🌳 joss
'21 Adaptive wavelet distillation from neural networks through interpretations ha, singh, et al. 🔍🌀🌳 neurips
'21 Matched sample selection with GANs for mitigating attribute confounding singh, balakrishnan, & perona 🌀 cvpr workshop
'21 Revisiting complexity and the bias-variance tradeoff dwivedi*, singh*, yu & wainwright 🌀 jmlr
'20 Curating a COVID-19 data repository and forecasting county-level death counts in the United States altieri et al. 🔎🦠 hdsr
'20 Transformation importance with applications to cosmology singh*, ha*, lanusse, boehm, liu & yu 🔎🌀🌌 iclr workshop (spotlight)
'20 Interpretations are useful: penalizing explanations to align neural networks with prior knowledge rieger, singh, murdoch & yu 🔎🌀 icml
'19 Hierarchical interpretations for neural network predictions Singh*, Murdoch*, & Yu 🔍🌀 ICLR
'19 interpretable machine learning: definitions, methods, and applications Murdoch*, Singh*, et al. 🔍🌳🌀 pnas
'19 disentangled attribution curves for interpreting random forests and boosted trees devlin, singh, murdoch & yu 🔍🌳 arxiv
'18 large scale image segmentation with structured loss based deep learning for connectome reconstruction Funke*, Tschopp*, et al. 🧠🌀 TPAMI
'18 linearization of excitatory synaptic integration at no extra cost Morel, Singh, & Levy 🧠 J Comp Neuro
'17 a consensus layer V pyramidal neuron can sustain interpulse-interval coding Singh & Levy 🧠 Plos One
'17 a constrained, weighted-l1 minimization approach for joint discovery of heterogeneous neural connectivity graphs Singh, Wang, & Qi 🧠 neurips Workshop ,

resources + posts



Misc resources. There's also some dumb stuff here.

Notes in machine learning / neuroscience.

experience


I've been lucky to be advised by / collaborate with many amazing people

It has been my pleasure to help advise some incredible undergrads at berkeley