Fundamental Limitations of Single-Vector Embeddings
Reading
Paper audio
About this week
Why does something as simple as “finding people who like apples” break our best models? What do alternatives like multi-vector models or cross-encoders mean for real products?
This research establishes a fundamental mathematical constraint in dense retrieval: embedding models cannot represent all possible top-k combinations of relevant documents simultaneously. Weller et al. demonstrate this through both theory and a clever benchmark (LIMIT) where even advanced state-of-the-art models struggle with trivially simple queries, suggesting important implications for instruction-following retrieval systems.
Join us at CASI for discussion at 8 pm, and (optional) quiet reading from 7 pm.
Proof sketch
Mark Sun fleshed out his proof — Sphere packing for single vector query (PDF). This is a sketch with a few more proof steps outlined, for Theorem 1 in the paper this week.
Mark also brought the macarons — thank you, Mark!
Photos
