seeking good explanations with machine learning
senior researcher at microsoft research (deep learning group)
seeking good explanations with machine learning
senior researcher at microsoft research (deep learning group)
🔎 My research focuses on how we can build trustworthy machine-learning systems by making them interpretable. In my work, interpretability is grounded seriously via close collaboration with domain experts, e.g. medical doctors or cell biologists. These collaborations have given rise to useful methodology, roughly split into two areas: (1) building more effective transparent models and (2) improving the trustworthiness of black-box models. Going forward, I hope to help bridge the gap between transparent models and black-box models to improve real-world healthcare.
🌳 Whenever possible, building transparent models is the most effective route towards ensuring interpretability. Transparent models are interpretable by design, including models such as (concise) decision trees, rule lists, and linear models. My work in this area was largely motivated by the problem of clinical decision-rule development. Clinical decision rules (especially those used in emergency medicine), need to be extremely transparent so they can be readily audited and used by physicians making split-second decisions. To this end, we have developed methodology for enhancing decision trees. For example, replacing the standard CART algorithm with a novel greedy algorithm for tree-sums can substantially improve predictive performance without sacrificing predictive performance. Additionally, hierarchical regularization can improve the predictions of an already fitted model without altering its interpretability. Despite their effectiveness, transparent models such as these often get overlooked in favor of black-box models; to address this issue, we've spent a lot of time curating imodels, an open-source package for fitting state-of-the-art transparent models.
🌀 My second line of my work focuses on interpreting and improving black-box models, such as neural networks, for the cases when a transparent model simply can't predict well enough. Here, I work closely on real-world problems such as analyzing imaging data from cell biology and cosmology. Interpretability in these contexts demands more nuanced information than standard notions of "feature importance" common in the literature. As a result, we have developed methods to characterize and summarize the interactions in a neural network, particularly in transformed domains (such as the Fourier domain), where domain interpretations can be more natural. I'm particularly interested in how we can ensure that these interpretations are useful, either by using them to embed prior knowledge into a model or identify when it can be trusted.
🤝 There is a lot more work to do on bridging the gap between transparent models and black-box models in the real world. One promising avenue is distillation, whereby we can use a black-box model to build a better transparent model. For example, in one work we were able to distill state-of-the-art neural networks in cell-biology and cosmology into transparent wavelet models with <40 parameters. Despite this huge size reduction, these models actually improve prediction performance. By incorporating close domain knowledge into models and the way we approach problems, I believe interpretability can help unlock many benefits of machine-learning for improving healthcare and science.
| year | title | authors | tags | paper | code | misc |
|---|---|---|---|---|---|---|
| '23 | Tree Prompting | morris*, singh*, rush, gao, & deng | 🔎🌀🌳 | emnlp | ||
| '23 | Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs | zhang et al. | 🔎🌀 | arxiv | ||
| '23 | Explaining black box text modules in natural language with language models | singh*, hsu*, antonello, jain, huth, yu & gao | 🔎🌀 | arXiv | ||
| '23 | Augmenting Interpretable Models with LLMs during Training | singh, askari, caruana, & gao | 🔎🌀🌳 | nature communications | ||
| '23 | Self-Verification Improves Few-Shot Clinical Information Extraction | gero*, singh*, cheng, naumann, galley, gao, & poon | 🔎🌀💊 | icml workshop (imlh) | ||
| '22 | Explaining patterns in data with language models via interpretable autoprompting | singh*, morris*, aneja, rush, & gao | 🔎🌀 | emnlp workshop (blackboxnlp) | ||
| '22 | Stress testing a clinical decision instrument performance for intra-abdominal injury | kornblith*, singh* et al. | 🔎🌳💊 | PLOS digital health | ||
| '22 | Fast interpretable greedy-tree sums (FIGS) | tan*, singh*, nasseri, agarwal, & yu | 🔎🌳 | arxiv | ||
| '22 | Hierarchical shrinkage for trees | agarwal*, tan*, ronen, singh, & yu | 🔎🌳 | icml (spotlight) | ||
| '22 | VeridicalFlow: a python package for building trustworthy data science pipelines with PCS | duncan*, kapoor*, agarwal*, singh*, & yu | 💻🔍 | joss | ||
| '21 | imodels: a python package for fitting interpretable models | singh*, nasseri*, et al. | 💻🔍🌳 | joss | ||
| '21 | Adaptive wavelet distillation from neural networks through interpretations | ha, singh, et al. | 🔍🌀🌳 | neurips | ||
| '21 | Matched sample selection with GANs for mitigating attribute confounding | singh, balakrishnan, & perona | 🌀 | cvpr workshop | ||
| '21 | Revisiting complexity and the bias-variance tradeoff | dwivedi*, singh*, yu & wainwright | 🌀 | jmlr | ||
| '20 | Curating a COVID-19 data repository and forecasting county-level death counts in the United States | altieri et al. | 🔎🦠 | hdsr | ||
| '20 | Transformation importance with applications to cosmology | singh*, ha*, lanusse, boehm, liu & yu | 🔎🌀🌌 | iclr workshop (spotlight) | ||
| '20 | Interpretations are useful: penalizing explanations to align neural networks with prior knowledge | rieger, singh, murdoch & yu | 🔎🌀 | icml | ||
| '19 | Hierarchical interpretations for neural network predictions | Singh*, Murdoch*, & Yu | 🔍🌀 | ICLR | ||
| '19 | interpretable machine learning: definitions, methods, and applications | Murdoch*, Singh*, et al. | 🔍🌳🌀 | pnas | ||
| '19 | disentangled attribution curves for interpreting random forests and boosted trees | devlin, singh, murdoch & yu | 🔍🌳 | arxiv | ||
| '18 | large scale image segmentation with structured loss based deep learning for connectome reconstruction | Funke*, Tschopp*, et al. | 🧠🌀 | TPAMI | ||
| '18 | linearization of excitatory synaptic integration at no extra cost | Morel, Singh, & Levy | 🧠 | J Comp Neuro | ||
| '17 | a consensus layer V pyramidal neuron can sustain interpulse-interval coding | Singh & Levy | 🧠 | Plos One | ||
| '17 | a constrained, weighted-l1 minimization approach for joint discovery of heterogeneous neural connectivity graphs | Singh, Wang, & Qi | 🧠 | neurips Workshop | , |
Notes in machine learning / neuroscience.
Mini personal projects. There's also some dumb stuff here.
generating explanations with large language models for science & medicine, working in the deep learning group
researched interpretable machine learning, especially for neural networks and in scientific applications, working in the bin yu group
worked on unsupervised semantic segmentation at meta ai, and worked on causal fairness benchmarking in computer vision at aws
worked on graphical models under Yanjun Qi and on biophysical modeling under William Levy -- also spent 3 summers doing research at HHMI Janelia under Srini Turaga
I've been lucky to be advised by / collaborate with many amazing people
It has been my pleasure to help advise some incredible students
MSR PhD interns