Recovering Temporal and Geographic Signals from Language Model Embeddings

A projection-based, black-box method operating directly on output embeddings recovers meaningful chronological and spatial structure from language-model representations, according to arXiv paper 2609.05721v1. The method defines an axis in embedding space from a small set of seed examples and ranks texts or entities by their projection onto that axis, requiring only embeddings and no model weights, internal activations, auxiliary probes, or additional training. The authors report the approach is model-agnostic and applicable to API-only embedding models, supporting interpretability and downstream temporal and geographic retrieval tasks such as temporal ordering, geographic ranking, and tagging.

arXiv:2609.05721v1 Announce Type: new Abstract: Understanding whether language-model embeddings encode structured real-world information is important for both representation analysis and information retrieval. We study this question for temporal and geographic signals using a simple projection-based method that operates directly on output embeddings. Given a small set of seed examples, the method defines an axis in embedding space and ranks texts or entities by their projection onto that axis. Our approach is fully black-box and model-agnostic: it requires only embeddings, without access to model weights, internal activations, auxiliary probes, or additional training. This makes it applicable to modern embedding models available only through APIs and provides a lightweight way to analyze whether temporal and spatial dimensions are present in their representation spaces. We apply the method to temporal and geographic datasets and find that embedding projections recover meaningful chronological and spatial structure. These results provide evidence that output embeddings encode signals relevant to time and space, while also offering a practical tool for interpretability and for downstream temporal and geographic information retrieval tasks, such as temporal ordering, geographic ranking, and tagging.