We asked six AI chat models what is at 41,526 points of San Francisco, from memory. The best three tie at 86% on the city map, against 61% for a trivial answer, and none of them can draw the Marina's streets.
In 2023, Gurnee and Tegmark showed that you can read the latitude and longitude of places straight out of a language model’s activations. So there is a map in there somewhere. We wanted to see it.
We gave six AI chat models, two each from Anthropic, OpenAI and Google, 41,526 points of San Francisco as latitude and longitude, and asked what is at each one - water, a park, a building, a street or something else. They answered from memory, with no maps and no internet, and we checked every answer against OpenStreetMap. We also asked two multiple-choice APIs, Jev by TypeSafe and OpenAI’s Decisions API, one point at a time.
There are three maps: the whole city at a point every 100 m, at low, medium and high reasoning; Crissy Field, the Palace of Fine Arts and the Marina at a point every 10 m, at medium; and one 500 x 500 m block of the Marina, at medium and high.
In geocoding terms, this is reverse geocoding from memory: a coordinate goes in, and the model says what is there. We did not test the other direction, from an address to a coordinate.
86% of the city map right for the best three models. "Not water, not park" everywhere gets 61%
$26 vs $135 for the same city score
0.32 best street score on one Marina block, against 0.23 by chance
23 of 25 requests Gemini 4 Argon left empty on that block at high reasoning
Three models tie at 86%, and one costs five times more #
At a point every 100 m, a point lands on a street or a building by chance, so we score the city map as water, park or everything else. At high reasoning, Gemini 4 Argon, GPT-6.1 Sol and GPT-6 Astra all got 86% of it right, against 61% for answering “not water, not park” everywhere. They are within a point of each other, so we call it a tie.
Astra’s run on the city map cost $135. Argon’s and Sol’s cost about $26 each. These are what our runs cost: Anthropic and OpenAI went through their Batch APIs at half price, and Gemini through the regular API.
The good maps have the coastline, Golden Gate Park, the Presidio and Lake Merced in the right place. Small parks are hit and miss - the best park score is 0.45. Every chat model drew its best map at high reasoning. GPT-6 Astra and Gemini 4 Argon gained about 6 points from low to high, and GPT-6.1 Sol got 83% at medium for $4.61.
Streets are the weak spot #
On the 10 m Marina map all five answers count. GPT-6 Astra, GPT-6.1 Sol, Claude Fable 5.1 and Claude Opus 5.5 put the Palace of Fine Arts lagoon in the right place. Fable, Astra and Sol score 0.19-0.21 on streets here, against 0.11-0.14 if their own answers were shuffled at random. That looks like knowing which parts of the map are dense grid and which are park.
Claude Opus 5.5 called 3% of the points a street, against 21% in reality, and painted the grid as one big building. That got it the most points right on the map (48.8%) and the lowest street score of any chat model (0.04).
OpenStreetMap
GPT-6 Astra
46%
GPT-6.1 Sol
46%
Claude Fable 5.1
44%
Claude Opus 5.5
49%
Gemini 4 Argon
44%
Gemini 3.8 Flash
36%
Jev
29%
OpenAI Decisions API
29%
To see if more thinking finds single streets, we took one 500 x 500 m block of the Marina grid, 46% buildings and 33% streets. That is 2,500 points in 25 requests of 100, and we asked every model at medium and at high reasoning.
OpenStreetMap
GPT-6 Astra medium
0.31
GPT-6 Astra high
0.32
GPT-6.1 Sol medium
0.27
GPT-6.1 Sol high
0.29
Claude Fable 5.1 high
0.25
Gemini 3.8 Flash high
0.27
Claude Opus 5.5 high
0.15
Gemini 4 Argon high
8% answered
The best street score was 0.32, from GPT-6 Astra at high, where 1 means every street in the right place. Its own answers shuffled around the block score 0.23. High reasoning added 0.01-0.03 to the street score and cost up to 3.6 times as much. Only Astra and GPT-6.1 Sol beat their shuffled answers, by 0.04-0.09, and that is a thin margin on 25 requests. The other four score about what they would by putting the same number of streets in random places.
So the best models know San Francisco like a tourist with a good memory. They get the coastline and the big parks, and they get the street grid wrong.
Two multiple-choice APIs #
Jev and OpenAI’s Decisions API, which runs GPT-6 Luna, both answer a multiple-choice question with a probability for each choice. That seemed a natural fit for “which of these five is it?”, so we asked them one point at a time. The chat models got 100 points per request.
On the city map Jev drew a box, with water around the edge and everything else inside, and got 66.7%, above the 61% for “not water, not park” everywhere. The Decisions API called 53% of the city a park and got 40.4%.
OpenStreetMap
Jev
67%
OpenAI Decisions API
40%
On the 10 m Marina map both got about 29% right. 84% of Jev’s answers there were water, and the Decisions API’s were split between water and park. Neither was sure: the answer each one picked got 37% (Jev) and 41% (Decisions API) on average, against 20% for a blind guess, and both were as sure of their wrong answers as of their right ones.
What broke #
Gemini 4 Argon at high reasoning thinks itself out of room. Its output limit is 32,768 tokens, thinking included. On the city map it thinks for about 15,000 tokens per request, and 4 of 159 requests came back empty. All 4 answered on a second try, and Argon reached the top only after that.
On the dense Marina block, 23 of 25 requests came back empty, and 24 retries in a row failed. There it thinks for about 30,000 tokens even when we send it only 10 points, so 10-point requests just fit. Redoing the block that way would have cost about $75, so we left the hole. One of the joint best models on the city map is the least reliable one.
OpenAI’s Batch API stalled on 5 of 8 batches for about an hour, and GPT-6.1 Sol at high can think for up to half an hour per request. We finished those through the regular API and OpenAI’s background mode. Apart from Argon high on the block, every run answered 100% of its points.
How precise is this? #
We ran GPT-6.1 Sol medium and Gemini 4 Argon low on the city map a second time. 12% and 11% of points changed, but the scores moved by less than half a point: 83.4% to 83.7%, and 80.1% to 79.7%.
Honest caveat - treat models within a point of each other on the city map as tied. On the small block the ranges are about ±3 points. OpenStreetMap is the answer key, so its mistakes count against the models. Different wording, points in order instead of shuffled, or fewer points per request would change the numbers. We did not tune the setup on the scores.
Run it yourself #
The code, every answer and the answer keys are open at github.com/voygr-tech/sf-from-memory. Three commands rebuild the scores, costs and figures without an API key. This started as a quick question and got out of hand.
If your AI agent needs real places #
If you need reverse geocoding, use a geocoder, not a model’s memory. At a point every 10 m, the best model got under half of the Marina right (Claude Opus 5.5, 48.8% at medium reasoning), and none of them drew its streets.
This test asked about the ground, not about businesses. We read the result as a picture of what a model’s memory is good for: the big shapes, not the detail. Whether the café on a given corner is still open, and when, is detail. We would have an agent look that up rather than remember it.
We are VOYGR. We build APIs that tell AI agents which places exist, whether they are open and what they offer:
- Validate a business Send a name and address, get back how strong the evidence is that the business exists at that location
- Enrich a place
open_closed, opening hours, contacts, menus, prices, and custom attributes you describe in plain English - Batch jobs Up to 10,000 places at a time, for checking a whole list
- PlaceCall Send a US phone number and a plain-language task, and an AI voice agent places the real call and returns a structured result
One request asks whether a place is open, when, and how to reach it:
{
"poi_name": "Example Bakery",
"address": "123 Example Street",
"country": "US",
"attributes": ["open_closed", "opening_hours", "phone"]
}
If you are an agent, start at api.voygr.tech/llms.txt or the OpenAPI spec. In Claude Code, run /plugin marketplace add voygr-tech/placecall, then /plugin install placecall@placecall, then /mcp, pick PlaceCall and sign in. The PlaceCall page has the other setups.
We have tested AI on real places before: our LLM local search benchmark ran 2,415 evaluations of local search answers, with every recommended place checked against Google Search and Maps, and our SF Tech Week study had an AI talk to 848 San Francisco businesses by phone.
FAQ #
Which AI model knows San Francisco best?
In our test, Gemini 4 Argon, GPT-6.1 Sol and GPT-6 Astra tie at 86% of points right on a map of San Francisco with a point every 100 m, scored as water, park or everything else, all at high reasoning. Answering 'not water, not park' everywhere gets 61%. Astra's run cost $135, the other two about $26 each.
Can AI models draw a city map from memory?
Roughly. The best models put the coastline, Golden Gate Park, the Presidio and Lake Merced in the right place, but none drew the Marina street grid. On a 500 x 500 m block the best street score was 0.32, from GPT-6 Astra, where its own answers shuffled at random score 0.23.
Does more reasoning help AI models with maps?
On the city map, yes: every chat model drew its best map at high reasoning, and GPT-6 Astra and Gemini 4 Argon gained about 6 points from low to high. On the Marina block, high reasoning added only 0.01-0.03 to the street score and cost up to 3.6 times as much.
Can an LLM do reverse geocoding from memory?
Only roughly. We gave six AI chat models San Francisco coordinates and asked what is at each one, from memory. At a point every 100 m the best three got 86% right on water, park or everything else, against 61% for a trivial answer. At a point every 10 m in the Marina, at medium reasoning, the best got 48.8% across water, park, building, street and other, and none drew the street grid. We did not test forward geocoding, from an address to a coordinate.
Should an AI agent answer questions about places from memory?
Our test covered what is at a point on the map, not businesses. There, the best models got the big picture right and the detail wrong. For facts about a specific place, such as whether a business is still open, we would have the agent look them up. VOYGR's APIs tell AI agents which places exist, whether they are open and what they offer; agents can start at https://api.voygr.tech/llms.txt.