Can AI draw San Francisco from memory?

Six AI chat models from Anthropic, OpenAI and Google were asked to identify what is at 41,526 San Francisco coordinates from memory, and Gemini 4 Argon, GPT-6.1 Sol and GPT-6 Astra tied at 86% accuracy on the city map at high reasoning, against 61% for answering "not water, not park" everywhere, according to the test checked against OpenStreetMap. The same city-map run cost $135 on GPT-6 Astra versus about $26 each on Argon and Sol, and on a 500 x 500 m Marina block the best street score was 0.32 (GPT-6 Astra at high reasoning) against 0.23 by chance, while Gemini 4 Argon left 23 of 25 requests empty at high reasoning. Claude Opus 5.5 labeled 3% of points as street against 21% in reality and scored the lowest street score of any chat model at 0.04, showing that more reasoning does not recover fine street detail.

Can AI draw San Francisco from memory? We asked six AI chat models what is at 41,526 points of San Francisco, from memory. The best three tie at 86% on the city map, against 61% for a trivial answer, and none of them can draw the Marina's streets. In 2023, Gurnee and Tegmark https://arxiv.org/abs/2310.02207 showed that you can read the latitude and longitude of places straight out of a language model’s activations. So there is a map in there somewhere. We wanted to see it. We gave six AI chat models, two each from Anthropic, OpenAI and Google, 41,526 points of San Francisco as latitude and longitude, and asked what is at each one - water, a park, a building, a street or something else. They answered from memory, with no maps and no internet, and we checked every answer against OpenStreetMap. We also asked two multiple-choice APIs, Jev by TypeSafe and OpenAI’s Decisions API, one point at a time. There are three maps: the whole city at a point every 100 m, at low, medium and high reasoning; Crissy Field, the Palace of Fine Arts and the Marina at a point every 10 m, at medium; and one 500 x 500 m block of the Marina, at medium and high. In geocoding terms, this is reverse geocoding from memory: a coordinate goes in, and the model says what is there. We did not test the other direction, from an address to a coordinate. 86% of the city map right for the best three models. "Not water, not park" everywhere gets 61% $26 vs $135 for the same city score 0.32 best street score on one Marina block, against 0.23 by chance 23 of 25 requests Gemini 4 Argon left empty on that block at high reasoning Three models tie at 86%, and one costs five times more At a point every 100 m, a point lands on a street or a building by chance, so we score the city map as water, park or everything else. At high reasoning, Gemini 4 Argon, GPT-6.1 Sol and GPT-6 Astra all got 86% of it right, against 61% for answering “not water, not park” everywhere. They are within a point of each other, so we call it a tie. Astra’s run on the city map cost $135. Argon’s and Sol’s cost about $26 each. These are what our runs cost: Anthropic and OpenAI went through their Batch APIs at half price, and Gemini through the regular API. The good maps have the coastline, Golden Gate Park, the Presidio and Lake Merced in the right place. Small parks are hit and miss - the best park score is 0.45. Every chat model drew its best map at high reasoning. GPT-6 Astra and Gemini 4 Argon gained about 6 points from low to high, and GPT-6.1 Sol got 83% at medium for $4.61. Streets are the weak spot On the 10 m Marina map all five answers count. GPT-6 Astra, GPT-6.1 Sol, Claude Fable 5.1 and Claude Opus 5.5 put the Palace of Fine Arts lagoon in the right place. Fable, Astra and Sol score 0.19-0.21 on streets here, against 0.11-0.14 if their own answers were shuffled at random. That looks like knowing which parts of the map are dense grid and which are park. Claude Opus 5.5 called 3% of the points a street, against 21% in reality, and painted the grid as one big building. That got it the most points right on the map 48.8% and the lowest street score of any chat model 0.04 . OpenStreetMap GPT-6 Astra 46% GPT-6.1 Sol 46% Claude Fable 5.1 44% Claude Opus 5.5 49% Gemini 4 Argon 44% Gemini 3.8 Flash 36% Jev 29% OpenAI Decisions API 29% To see if more thinking finds single streets, we took one 500 x 500 m block of the Marina grid, 46% buildings and 33% streets. That is 2,500 points in 25 requests of 100, and we asked every model at medium and at high reasoning. OpenStreetMap GPT-6 Astra medium 0.31 GPT-6 Astra high 0.32 GPT-6.1 Sol medium 0.27 GPT-6.1 Sol high 0.29 Claude Fable 5.1 high 0.25 Gemini 3.8 Flash high 0.27 Claude Opus 5.5 high 0.15 Gemini 4 Argon high 8% answered The best street score was 0.32, from GPT-6 Astra at high, where 1 means every street in the right place. Its own answers shuffled around the block score 0.23. High reasoning added 0.01-0.03 to the street score and cost up to 3.6 times as much. Only Astra and GPT-6.1 Sol beat their shuffled answers, by 0.04-0.09, and that is a thin margin on 25 requests. The other four score about what they would by putting the same number of streets in random places. So the best models know San Francisco like a tourist with a good memory. They get the coastline and the big parks, and they get the street grid wrong. Two multiple-choice APIs Jev and OpenAI’s Decisions API, which runs GPT-6 Luna, both answer a multiple-choice question with a probability for each choice. That seemed a natural fit for “which of these five is it?”, so we asked them one point at a time. The chat models got 100 points per request. On the city map Jev drew a box, with water around the edge and everything else inside, and got 66.7%, above the 61% for “not water, not park” everywhere. The Decisions API called 53% of the city a park and got 40.4%. OpenStreetMap Jev 67% OpenAI Decisions API 40% On the 10 m Marina map both got about 29% right. 84% of Jev’s answers there were water, and the Decisions API’s were split between water and park. Neither was sure: the answer each one picked got 37% Jev and 41% Decisions API on average, against 20% for a blind guess, and both were as sure of their wrong answers as of their right ones. What broke Gemini 4 Argon at high reasoning thinks itself out of room. Its output limit is 32,768 tokens, thinking included. On the city map it thinks for about 15,000 tokens per request, and 4 of 159 requests came back empty. All 4 answered on a second try, and Argon reached the top only after that. On the dense Marina block, 23 of 25 requests came back empty, and 24 retries in a row failed. There it thinks for about 30,000 tokens even when we send it only 10 points, so 10-point requests just fit. Redoing the block that way would have cost about $75, so we left the hole. One of the joint best models on the city map is the least reliable one. OpenAI’s Batch API stalled on 5 of 8 batches for about an hour, and GPT-6.1 Sol at high can think for up to half an hour per request. We finished those through the regular API and OpenAI’s background mode. Apart from Argon high on the block, every run answered 100% of its points. How precise is this? We ran GPT-6.1 Sol medium and Gemini 4 Argon low on the city map a second time. 12% and 11% of points changed, but the scores moved by less than half a point: 83.4% to 83.7%, and 80.1% to 79.7%. Honest caveat - treat models within a point of each other on the city map as tied. On the small block the ranges are about ±3 points. OpenStreetMap is the answer key, so its mistakes count against the models. Different wording, points in order instead of shuffled, or fewer points per request would change the numbers. We did not tune the setup on the scores. Run it yourself The code, every answer and the answer keys are open at github.com/voygr-tech/sf-from-memory https://github.com/voygr-tech/sf-from-memory . Three commands rebuild the scores, costs and figures without an API key. This started as a quick question and got out of hand. If your AI agent needs real places If you need reverse geocoding, use a geocoder, not a model’s memory. At a point every 10 m, the best model got under half of the Marina right Claude Opus 5.5, 48.8% at medium reasoning , and none of them drew its streets. This test asked about the ground, not about businesses. We read the result as a picture of what a model’s memory is good for: the big shapes, not the detail. Whether the café on a given corner is still open, and when, is detail. We would have an agent look that up rather than remember it. We are VOYGR. We build APIs that tell AI agents which places exist, whether they are open and what they offer: - Validate a business Send a name and address, get back how strong the evidence is that the business exists at that location - Enrich a place open closed , opening hours, contacts, menus, prices, and custom attributes you describe in plain English - Batch jobs Up to 10,000 places at a time, for checking a whole list - PlaceCall Send a US phone number and a plain-language task, and an AI voice agent places the real call and returns a structured result One request asks whether a place is open, when, and how to reach it: { "poi name": "Example Bakery", "address": "123 Example Street", "country": "US", "attributes": "open closed", "opening hours", "phone" } If you are an agent, start at api.voygr.tech/llms.txt https://api.voygr.tech/llms.txt or the OpenAPI spec https://api.voygr.tech/openapi.json . In Claude Code, run /plugin marketplace add voygr-tech/placecall , then /plugin install placecall@placecall , then /mcp , pick PlaceCall and sign in. The PlaceCall page https://voygr.tech/placecall/ has the other setups. We have tested AI on real places before: our LLM local search benchmark https://voygr.tech/report/ ran 2,415 evaluations of local search answers, with every recommended place checked against Google Search and Maps, and our SF Tech Week study https://voygr.tech/blog/does-sf-talk-to-ai/ had an AI talk to 848 San Francisco businesses by phone. FAQ Which AI model knows San Francisco best? In our test, Gemini 4 Argon, GPT-6.1 Sol and GPT-6 Astra tie at 86% of points right on a map of San Francisco with a point every 100 m, scored as water, park or everything else, all at high reasoning. Answering 'not water, not park' everywhere gets 61%. Astra's run cost $135, the other two about $26 each. Can AI models draw a city map from memory? Roughly. The best models put the coastline, Golden Gate Park, the Presidio and Lake Merced in the right place, but none drew the Marina street grid. On a 500 x 500 m block the best street score was 0.32, from GPT-6 Astra, where its own answers shuffled at random score 0.23. Does more reasoning help AI models with maps? On the city map, yes: every chat model drew its best map at high reasoning, and GPT-6 Astra and Gemini 4 Argon gained about 6 points from low to high. On the Marina block, high reasoning added only 0.01-0.03 to the street score and cost up to 3.6 times as much. Can an LLM do reverse geocoding from memory? Only roughly. We gave six AI chat models San Francisco coordinates and asked what is at each one, from memory. At a point every 100 m the best three got 86% right on water, park or everything else, against 61% for a trivial answer. At a point every 10 m in the Marina, at medium reasoning, the best got 48.8% across water, park, building, street and other, and none drew the street grid. We did not test forward geocoding, from an address to a coordinate. Should an AI agent answer questions about places from memory? Our test covered what is at a point on the map, not businesses. There, the best models got the big picture right and the detail wrong. For facts about a specific place, such as whether a business is still open, we would have the agent look them up. VOYGR's APIs tell AI agents which places exist, whether they are open and what they offer; agents can start at https://api.voygr.tech/llms.txt.