embeddings view markdown

Notes on embedding models

See related papers in the đź“Ś llm basics and đź“Ś llm research pages.

Embedding models take in unstructured input (here, often text) and transform it into a vector representation that is usually used for search. Key considerations include: fast inference for embedding a new input, small/quantized embeddings for efficient storage, indexability for fast search with retrieval algorithms.

overviews

embedding model distinctions

  • leading models as of summer 2026: google (gemini embedding), voyage (voyage series, open-source voyage-4-nano), alibaba (qwen embedding, GTE), cohere, openai (relatively outdated), BAAI (BGE), Jina (bought by elastic), NVIDIA (NV-Embed), Microsoft (E5), nomic (nomic-embed)
  • embedding approaches: embedding_interaction_levels

  • 3 levels of interaction

    1. bi-encoder: separately encode query & doc into a single vec

    2. late-interaction encoder: separately encode but keep a vector per token & learn some params on how to compute similarity between them

      • e.g. ColBERT (khattab & zaharia, 2020) uses maxsim to compare, which keeps the vector for each token and then for each token in the query, finds the candidate token it matches best (highest cosine similarity), then averages those best-match scores across all query tokens

      • Your Embedding Model is SMARTer Than You Think (zhang…lee, 2026) - use the maxsim score (along with the original last-token score) to do late interaction without requiring post-training

    3. cross-encoder: encode query and doc together

best practices for training

  • basic training pipeline

    1. standard self-supervised pre-training, e.g. BERT or off-the-shelf model like Qwen

    2. weak unsupervised pre-training, e.g. weakly related text pairs, such as QA pairs from forums like StackExchange and Quora

    3. high-quality contrastive finetuning on curated paired data, e.g. QA from web searches

  • after retrieving results, a reranker model that takes in both the inputs and the query can be used to quickly improve performance
  • key tricks to improve performance
    • training-time
      • instruction/task conditioning: prompt or prefixes like search query, search document, classification, clustering before embedding so model knows how to customize the embedding
      • synthetic data from LLMs: generate tuples of (task, query, positive, hard negative) for contrastive training, especially crafting good hard negatives
      • distillation from more compute-intensive model: e.g. cross-encoder to bi-encoder or post-reranker to before reranker
      • quantization-aware training
      • MoE training
      • sparse representation learning (e.g. UHD-BERT (jang…seo, 2021))
      • joint learning with index
      • prior work: query expansion, term dependency model (e.g. tf-idf), topic model, translation model
      • matryoshka representation learning - apply the contrastive loss simultaneously at dims 256, 512, 1024, 2048, etc so users can truncate to tradeoff acc for storage/speed
      • tailoring to specialized domains (e.g. medical, finance)
      • sharing embedding spaces across different model sizes (like in voyage-4) so different sizes can be used for embedding/querying
      • same backbone across modalities (like in voyage-3.5-multimodal) so that inputs aren’t biased towards any one modality
      • minor tricks: model souping / checkpoint merging on different task mixtures, loss balancing across tasks, curriculum over data quality stages, long-context extensions
    • document embedding-time
      • contextualized chunking - generate chunk embeddings for multiple chunks in one forward pass so they have context (could even aggregate context across chunks)
    • query inference-time
      • query expansion & reweighting
      • hybrid retrieval, combining with BM-25
      • reranking with a cross-encoder for topk results

top-performing models

  • baseline simple models
    • BM25 - uses TF-IDF based on word counts and document frequences
    • Simple but Tough-to-Beat Baseline for Sentence Embeddings (arora, liang & ma, 2017)
    • average word embeddings in a sentence, downweighting words by their frequency
    • to remove the “common background direction”, compute the top pca component from many sentence embeddings then remove that direction
  • early models just trained with bi-encoders, e.g. SBERT=Sentence BERT (reimers & gurevych, 2019), SimCSE (gao, yao & chen, 2021)
    • PromptBERT (jiang…furu wei…zhang, 2022): This sentence: “ [text] ” means [MASK] then use the embedding of the mask token - this prevents how all raw embeddings have similar cosine similarities
      • Scaling Sentence Embeddings with LLMs (jiang, …, zhuang, 2023): This sentence: “ [text] ” means in one word: then use the embedding of the final token
  • next era of models started using contrastive pre-training, e.g. E5 (wang…wei, 2022), GTE (li…zhang, 2023), and BGE (github)
  • next, models initialized using a pre-trained LLM, e.g. E5-mistral-instruct (wang…wei, 2023), SGPT (muennighoff, 2022)
  • multimodal embeddings
  • query inference-time expansions
  • rerankers
    • 3 common versions: pointwise (score each document independently), pairwise (learn “A beats B”), listwise (optimize the ordering of the whole list, targeting metrics like NDCG directly)
    • early models like ms-marco-MiniLM were cross-encoders, people later found that prompting an LLM did well but was expensive, so these were distilled
    • newer models like Rank1 use reasoning for reranking
    • rerank-2.5 (aug 2025) - includes instruction following, e.g. “Prioritize the title and ignore the abstract”
  • contextualized chunking
    • voyage does this via explicitly training a model to output separate vectors for each chunk
    • jina does this via late chunking (embed whole doc, get embeddings for a chunk by mean pooling over its tokens)
    • Contextual retreival (Anthropic blog post, 2024) - prepends an LLM-generated summary to every chunk before embedding
    • more researchy
      • RAPTOR (sarthi…manning, 2024) - build hierarchical index by embedding, clustering, summarizing, and embedding the summaries
      • Contextual Document Embeddings (morris & rush, 2024) - embed docs conditioned on other docs (requires training to do this well)
  • papers with a little trick
    • training-time
      • GritLM (meunninghoff…kiela, 2024) - train a single model that, given different instructions, can produce either generations or embeddings
      • Matryoshka Representation Learning (kusupati…kakade, jain, & farhadi, 2022) - in training given an embedding of full dimensionality M (e.g. 2048), learn N different distance functions for each prefix of the embedding (e.g. l2_norm(embedding[:32]), l2_norm(embedding[:64]), l2_norm(embedding[:128]), etc).
        • Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation (wen…you, 2025) - instead learn sparse mask on top of original embedding
        • AGRAME: Any-Granularity Ranking with Multi-Vector Embeddings (reddy…potdar, 2024) - rank at varying levels of granularity while maintaining encoding at a single (coarser) level
      • EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory (nie, fu, feng & shan, 2026) - maintains a continuously updated latent memory as it sequentially processes inputs, and uses it alongside the raw content to jointly generate embeddings
      • DREAM: Dense Retrieval Embeddings via Autoregressive Modeling (tang & yang, 2026) - instead of contrastive learning, use retriever to replace attention scores for particular heads in next-token prediction task
    • query inference-time
      • EchoEmbeddings: Repetition Improves LM Embeddings (springer, kotha, fried, neubig, & raghunathan, 2024)
        • Feed a prompt such as “Rewrite the sentence: x, rewritten sentence: x” to the LM and pool the contextualized embeddings of the 2nd occurence of x

datasets / benchmarks

agentic search

Agentic search - agent actively plans, executes, and iterates on searches to answer a query, rather than just running a single keyword lookup and handing back results

  • FastContext: Training Efficient Repository Explorer for Coding Agents (zhang…fu, 2026) - exploration models from 4B-30B
  • Search-R1: Training LLMs to Reason and Leverage Search Engines with RL (jin…han, 2025)
  • Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction (li…zhang, 2026) - agent searches the raw corpus directly using grep, find, bash, shell pipelines rather than embedding models
  • Chroma Context-1: Training a Self-Editing Search Agent (bashir, hong, jiang, & shi, 2026)
  • MemReranker: Reasoning-Aware Reranking for Agent Memory Retrieval (li…li, 2026)
  • Are We Ready For An Agent-Native Memory System? (zhou…wu, 2026)
    • MemoryBank: Enhancing LLMs with Long-Term Memory (zhong…wang, 2023) - maintains experiences as timestamped memory streams and periodically summarizes them into higher-level reflections
  • Superintelligent Retrieval Agent: The Next Frontier of Information Retrieval (yang, ma, chen & shrivastava, 2026)
    • asks which terms are likely to separate the desired evidence from corpus-level confusers
    • doc side: LLM enriches each document offline with missing search vocabulary
    • query side: predicts evidence vocabulary omitted by the query & document-frequency statistics as a tool call to filter proposed terms that are absent, overly common, or unlikely to create retrieval margin
    • final retrieval step is weighted BM25 call combining the original query with the validated expansion
  • GrepSeek: Training Search Agents for Direct Corpus Interaction (salemi…zamani, 2026) - trains a compact search agent to find, filter, and compose evidence from large text corpora
    • Is Grep All You Need? How Agent Harnesses Reshape Agentic Search (sen…subbiah, 2026) - grep generally outperforms vector retrieval in agentic RAG workflows, but overall accuracy depends heavily on agent harness

  • MemEx: A Programmable Scratchpad for LLM Agents (databricks research team, 2026) - improves token efficiency with a wrapper layer over tools, that stores objects in python rather than text every time

  • https://jbarrow.ai/2026-06-12-searching-fast-and-slow/:

searching embeddings (approximate nearest neighbor)

  • locality-sensitive hashing (outdated) is a fuzzy hashing technique that hashes similar input items into the same “buckets” with high probability
    • idea is to project vector along random dimensions to create buckets, and then match those buckets before searching the rest
    • issue is that the number of dimensions needed to get good performance can quickly grow, obviating the benefit
    • hash collisions are maximized, rather than minimized as they are in dictionaries
    • finding embeddings via DNNs is a special case of this (e.g. might call it “semantic hashing”)

  • k-d (k-dimensional) tree
    • balanced binary tree over data with arbitrary dimensions, where each node splits one dim
    • at inference time, go to leaf and then search all neighbors of the leaf by going back up it
    • annoy optimized this to instead use a forest of random trees and to do data-driven splitting (take 2 random points and split on the direction between them) rather than the median of a single dim

  • inverted file (IVF) - k-means the embeddings into clusters, then at inference first search the cluster centers before searching within the top clusters
    • to improve performance, add neighbors into mutiple clusters

  • product quantization (jegou et al. 2011) - this speeds up the vector comparison time (different from everything else)
    • separate vector into subvectors (e.g. first half, second half), run k-means for each subvector, then replace the subvector with the index of its closest center mean
    • at inference time, split the query into the same subvectors, and pre-compute the squared distance to each of the center means for each subvector
      • then, for each vector, rather than doing the dot-product, just look up the pre-computed distances for each subvector (add them together to get the squared distance for the whole vector)
      • note: splitting vectors and clustering struggles if there are lots of correlations between different subvector parts, one fix for this is Optimized PQ (OPQ), which learns a rotation to apply before splitting so the variance is balanced and decorrelated across subspaces

  • HNSW (malkov & yashunin, 2016) - Hierarchical Navigable Small World graphs (most popular modern system including Lucene, pgvector, Qdrant, Weaviate, Milvus, Pinecone’s early stack for medium-large scale <10M vectors)
    • idea is to build a proximity graph over the vectors, and answer queries with greedy best-first traversal - start somewhere, repeatedly hop to the neighbor closest to the query, and search its neighbors
      • hierarchy: does search at a higher layer, then drops to neighbors in lower layer which are more refined
      • edge-selection at insert time
        • note: naive graph of neighbors doesn’t work because it’s very hard to find a new nieghbor if you search in the wrong place; instead NSW inserts points in random order and connects each to its M nearest at insertion time (this leaves long-range edges from the early added nodes)
    • supports incremental inserts (although build and insert are both expensive)
      • delete is a pain
      • requires a lot of RAM (rather than disk access)

  • moving more stuff to disk instead of RAM (default for very large scale, 100M+ vectors)

DiskANN
 SPANN

  • other considerations
    • modern libraries often deal carefully with quantization and GPU usability
    • building compressions/graphs that are filter-aware, e.g. user issues queries like “nearest neighbors WHERE tenant=X AND date>Y.” should not break the search
  • libraries:
Library What it is Methods it connects to
Faiss (Meta) The field’s reference library, CPU and GPU. Nearly every classical index type, composable (e.g. IVF coarse layer + PQ codes + rescoring). Brute force, IVF, PQ/OPQ, IVF-PQ (asymmetric distance computation), HNSW, scalar/binary quantization
hnswlib (Malkov) The canonical lightweight HNSW implementation by the paper’s author; embedded inside many engines. Sibling research library: nmslib (original NSW). HNSW exactly — M, efConstruction, efSearch, diversity pruning
ScaNN (Google) Partitioning plus anisotropic quantization — the PQ loss reshaped to penalize ranking-relevant error. IVF + PQ with a smarter estimator (“train the index for the metric”)
DiskANN (Microsoft) Vamana graphs served from SSD; Fresh (streaming) and Filtered variants. DiskANN — α-prune build, PQ-steered beam search, packed SSD sector reads
SPTAG (Microsoft) Tree-plus-graph library; home of the SPANN design used in Bing-scale search. SPANN — in-RAM centroid routing, closure assignment for boundary vectors
Annoy (Spotify) Random-projection tree forest; mmap-friendly, immutable, simple. Largely legacy now but still deployed. The tree lineage — the KD-tree’s randomized, ensembled descendant
FALCONN Research-grade LSH (cross-polytope and hyperplane families) with multi-probe. LSH, with the modern hash families and fewer tables
cuVS (NVIDIA) GPU-native ANN: CAGRA graph build and search, GPU IVF-PQ; build speedups that change reindexing economics. Graph traversal + IVF-PQ, rebuilt for massive parallelism
Engine What it is Methods it connects to
Lucene (→ Elasticsearch, OpenSearch, MongoDB Atlas) Segment-based search engine whose vector support underlies Elasticsearch, OpenSearch, and Atlas Vector Search; immutable segments sidestep graph deletes. HNSW per segment + int8/binary quantization with exact rescoring
pgvector Postgres extension; vectors as a column type with SQL filtering and joins. HNSW, IVF (IVFFlat), and exact brute-force scans
Milvus Distributed vector database; its Knowhere engine wraps multiple index families behind one API. Nearly everything: Faiss indexes, HNSW, DiskANN, GPU variants
Qdrant Rust vector database emphasizing payload filtering and compression. HNSW + scalar/binary quantization; filter-aware traversal
Weaviate Go vector database with hybrid (BM25 + vector) search built in. Custom HNSW with PQ/binary compression and rescoring

Notes: popularity and internals shift quickly — verify against ann-benchmarks.com before depending on any of these. The engines section contains no new algorithms: each is HNSW-or-IVF plus quantization plus the operational layer (filtering, replication, segments, hybrid search).

explainable embeddings

  • QA-Emb: Crafting Interpretable Embeddings by Asking LLMs Questions (benara…gao, 2024) - use yes/no questions to extract embeddings from text
    • A General Framework for Producing Interpretable Semantic Text Embeddings (sun…yu, 2024) - extend QA-Emb to systematically generate highly discriminative, low cognitive load yes/no questions
    • PromptReps: Prompting LLMs to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval (zhuang…zuccon, 2024)
    • InBedder: Answer is All You Need: Instruction-following Text Embedding via Answering the Question (peng…jingbo shang, 2024)
      • embeddings consist of answers to questions
      • answer models are finetuned on QA datasets
      • questions are given ahead of time
    • Meta-Task Prompting Elicits Embedding from LLMs (lei…yates, 2024) - ask a few pre-canned templates e.g. “Categorize into one of these categories ___” and look at logits for the outputs as an embedding
    • Learning Interpretable Style Embeddings via Prompting LLMs (patel, rao, kothary, mckeown, & callison-burch, 2023)
    • CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with LLMs (mcinerney…wallace, 2023)
      • extract interpretable feature (e.g. “Does this patient have a chronic illness?”) and use in a linear model (use Flan-T5)
      • example features: 10 ICD codes + (1) Does the patient have a chronic illness? (2) Is the condition life-threatening?
    • Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM (lam…bernstein, 2024)
    • Explain via Any Concept: Concept Bottleneck Model with Open Vocabulary Concepts (tan, zhou, & chen, 2024)
    • BC-LLM: Bayesian Concept Bottleneck Models with LLM Priors (feng…tan, 2024)
    • HypotheSAEs (movva…pierson, 2025)
    • Evaluating scientific theories as predictive models in language neuroscience (singh…huth, 2025)
      • Bridging Brains and Concepts: Interpretable Visual Decoding from fMRI with Semantic Bottlenecks (cammarota, ferrante & toschi, 2025)
      • Disentangling Superpositions: Interpretable Brain Encoding Model with Sparse Concept Atoms (zeng & gallant, 2025)
        • apply sparse coding to word embeddings (e.g. eng1000) before fitting linear encoding models to mitigate feature correlations
          • normalize word embeddings before fitting so that highly frequent words (which are closer to the origin) do not end up with the same dictionary codes simply because of their frequency
        • Interpreting Brain Responses to Language with Sparse Features from LMs (lepori, kay & tuckute, 2026)
  • Box Embeddings: An open-source library for representation learning using geometric structures (chheda…mccallum, 2021) - allow for learning non-symmetric relations (e.g. entailment)
    • Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations (huang…yu, 2023) - learn interpretable compositional operations, which helps with similarities for compositional tasks
  • Dense Retrievers Know More Than They Can Express (mixedbread ai blog post, 2026) - use SAE followed by BM25 on SAE embeddings
  • multimodal
    • SPLICE: Interpreting CLIP with Sparse Linear Concept Embeddings (bhalla…lakkaraju, 2024)
      • given CLIP, build an embedding concept dictionary by taking text embeddings of a bunch of individual semantic words
      • given a new image, get its image embedding and then decompose it into a sparse, nonnegative combination of the concept dictionary (this makes it interpretable)
    • Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning (qin…scherer, 2026) - post-train CLIP to use sparse features
  • Computer-vision focused
    • Axiomatic Explanations for Visual Search, Retrieval, and Similarity Learning (hamilton, lundberg…freeman, 2021) - add in “second-order” methods that look at similarities between different image features in the 2 images being compared
    • Why do These Match? Explaining the Behavior of Image Similarity Models (plummer…saenko, forsyth, 2020) - generate saliency map + with an attribute based on the salient region
    • Towards Visually Explaining Similarity Models (zheng…wu, 2020) - similarity of cnn embeddings
  • Interpretable entity representations through large-scale typing (onoe & durrett, 2020) - embedding is interpretable predictions for different entities
  • Explaining similarity with different outputs
    • Analogies and Feature Attributions for Model Agnostic Explanation of Similarity Learners (ramamurthy…tariq, 2022) - returned explanation is an analogy (pair from the training set) rather than a saliency map
    • Sim2Word: Explaining Similarity with Representative Attribute Words via Counterfactual Explanations (chen…cao, 2023) - give both saliency map + counterfactual explanation

retrieval augmented generation (RAG)

  • RAG perspective paper (asai, zhong, chen, koh, zettlemoyer, hajishirzi, & yih, 2024)

    •   Granularity Incorporation Frequency Training Data order
      DrQA Chunks Input One-time Independent $O\left(10^9\right)$
      REALM, RAG, ATLAS Chunks Input One-time Joint $O\left(10^9\right)$
      RALM, REPLUG Chunks Input Every $k$ tokens, One-time Independent* $O\left(10^9\right)$
      Active-Retriever, Self-RAG Chunks Input Adaptive Independent*, Sequential $O\left(10^9\right)$
      RETRO, InstructRetro Chunks Intermediate Every $k$ tokens Sequential $O\left(10^{12}\right)$
      kNN LM, TRIME Tokens Output Every token Independent*, Joint $O\left(10^9\right)$
      NPM, Copy Generator Phrases Output Every phrase Joint $O\left(10^9\right)$
      SPALM, Adaptive kNN Tokens Output Adaptive Independent*, Joint $O\left(10^9\right)$

    • RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems (hsia…neubig, 2024) - gives benchmark of multi-hop QA questions for evaluating RAG systems holistically

    • https://pageindex.ai/ - popular system that replaces vector-based rag with table-of-contents based search for docs that are already well organized

    • Seven Failure Points When Engineering a Retrieval Augmented Generation System (barnet…abdelrazek, 2024)

  • dynamic systems

    • Active RAG (jiang…neubig, 2023) - propose FLARE, which iteratively uses a prediction of the upcoming sentence to anticipate future content, which is then utilized as a query to retrieve relevant documents to regenerate the sentence if it contains low-confidence tokens
    • Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (asai…hajishirzi, 2023) - train an LM that adaptively retrieves passages on-demand, and generates and reflects on retrieved passages and its own generations using special tokens, called reflection token
    • Infer–Retrieve–Rank: In-Context Learning for Extreme Multi-Label Classification (D’Oosterlinck, …, potts, 2024)
      • Infer: an LM processes the input document and guesses a set of applicable terms
      • Retrieve: a retriever relates each predicted term to the actual label space
      • Rank: Finally, an LM is used to rerank retrieved labels
    • Adaptive-RAG: Learning to Adapt Retrieval-Augmented LLMs through Question Complexity (jeong…park, 2024) - dynamically selects the most suitable retrieval-augmented strategy based on the predicted complexity level of input query
    • From Local to Global: A Graph RAG Approach to Query-Focused Summarization (edge…larson, 2024) - build and summarize a graph of documents to be used at query time for summarizing docs
  • Original papers
  • retrieval-augmented in-context learning (put retrieved info into context, or something very similar)
    • RETRO (deepmind, 2022) - nearest neighbors to model’s input are retrieved, encoded, and conditioned on with chunked cross-attention
    • Decomposed prompting (khot et al., 2022) - decompose tasks via prompting which are delegated to a shared library of prompting-based LLMs dedicated to these sub-tasks
    • retrieval-in-context approach (shi, min et al. 2023; ram…shoham, 2023) - retrieve docs and preprend to the context
    • LLM-Augmenter (peng, galley…gao, 2023) - (1) consolidates evidence from external knowledge for the LLM to generate responses grounded in evidence, and (2) revising LLM’s (candidate) responses using automated feedback
    • DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval (Lin et al 2023)
    • Knowledgeable Prompt-tuning (Hu et al. 2021) - add knowledge-base info into the prompt search
    • Atlas: Few-shot Learning with Retrieval Augmented LMs (meta, 2022)
    • GRIT: Generative Representational Instruction Tuning (muennighoff et al 2024) - train same model for text generation and retrieval tasks
    • Fine-grained Hallucination Detection and Editing for LMs (mishra, …, hajishirzi, 2024) - train a retrieval-augmented LM to correct fine-grained hallucinations
    • RAG can even outperform LMs fine-tuned on the downstream domain data on QA (Ovadia et al., 2023; Gupta et al., 2024)
  • Different ideas
    • Transformer Memory as a Differentiable Search Index (Tay at al 2022) - Same model learns to encode documents and find closest search index (rather than retrieving with maximal inner product search)
      • Self-Retrieval: Building an Information Retrieval System with One LLM (tang…li, 2024) - LLM learns to generate retrieved document from query
    • xRAG: Extreme Context Compression for RAG with One Token (cheng…furu wei…zhao, 2024)

  • interpretable RAG

universal representations

embedding inversions

  • Generative Embedding Inversion Attack to Recover the Whole Sentence (li…song, 2023) - train projection to LM jointly to reconstruct input

    • Information Leakage from Embedding in LLMs (wan…wang, 2024)
      • base embed inversion - directly pass hidden states to the LM head for generation
      • hotmap embed inversion - find input which yields embedding with greatest cosine similarity
      • embed parrot - learn a linear mapping to embedding states that is then
  • vec2text (morris et al. 2023) - invert embeddings to text without using gradients
    • logit2prompt (morris, …, rush, 2024) - recover prompt from output logits
    • output2prompt (zhang, morris, & shmatikov, 2024) - recover prompt from long text outputs (by building a model of the sparse encodings of the outputs)
    • ZSInvert: universal zero-shot embedding inversion (zhang, morris, & shmatikov, 2025) - beam search but keep prefixes that have best similarity with given embedding & train text-to-text correction model that helps refine hypotheses
      • builds on adversarial decoding (zhang, zhang, & shmatikov, 2024) - use beam search with multiple scorers besides just perplexity (e.g. for defense evasion)
  • On the Theoretical Limitations of Embedding-Based Retrieval (weller, boratko, naim & lee, 2025)
    • Multi-Vector Embeddings are Provably More Expressive than Single Vector Embeddings (jayaram, 2026)

external memory enhancements for LLMs

  • The AI Hippocampus: How Far are We From Human Memory? (jia…zhu, 2026)
    • delineate implicit memory (neocortex, learned weights), explicit memory (hippocampus, episodic memory/RAG), agentic memory (PFC, state and scratchpad)

  • Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory (wang…wei, 2026)

  • Engram: Conditional Memory via Scalable Lookup: A New Axis of Sparsity for LLMs (cheng…liang; deepseek, 2026)
  • MeMo: Memory as a Model (quek…solar-lezama, 2026) - train a model to learn a corpus (replacing retrieval piece in RAG), and then use it along with a frozen LLM to retrieve knowledge
  • memorizing transformers (wu…szegedy, 2022) - knn-based learned indexing + retrieval at training time
    • at test time, you just need to index the entire context and the model will be able to use it
    • kNN Prompting: Learning Beyond the Context with Nearest Neighbor Inference (xu…zhang, 2023) - instead of verbalizer, use nearest-neighbor (nice results for dbpedia)
    • kNN-Prompt: Nearest Neighbor Zero-Shot Inference (shi…zettlemoyer, 2022)
  • Memory Networks (weston, chopra, & bordes, 2014) - external KB that can be read / written to (stores plain text)
  • SILO LMs: Isolating Legal Risk In a Nonparametric Datastore (min…smith, zettlemoyer, 2023)
    • Use a parametric LM on open data then one of 2 nonparametric datastores: kNN LM or retrieval in-context
  • A Soft and Fast Pattern Matcher for Billion-Scale Corpus Searches (deguchi…yokoi, 2025)
  • Hybrid computing using a neural network with dynamic external memory (graves…hassabis, 2016) [see blog post]
    • differentiable neural computer (DNC) - neural network that can read from and write to an external memory matrix
    • stores external memory matrix
      • reads from memory via multiplying by a vector (e.g. one-hot vector would yield single element)
    • extends Neural turing machines (graves, wayne, & danihelka, 2014)
  • Hopfield Networks is All You Need (ramsaeur…hochreiter, 2020)
    • keys: each input has a key vector which “represents info about this input” (e.g. this is a noun)
    • queries: each input has a query vector which “asks for other inputs that would be useful context” (e.g. what adjectives describe this word)
      • in self-attention these queries also come from the input whereas in just regular attention they come from somewhere else (e.g. the output of a translation task)
    • transformer finds similarity between each key with each query then takes softmax - this provides weights for each of the inputs, as context for the original input
      • in transformer, these weights are used to weight the values but in hopfield nets we would take a weighted sum of the keys and feed it back as the input
    • as we update becomes more skewed towards the things that match the most
  • pre-transformers
    • Improving Neural LMs with a Continuous Cache (grave…usunier, 2016) - cache previous embeddings as memory from a document to contextualize an LSTM