# Local vs. Frontier: Benchmarks for My Digital Nervous System > Source: > Published: 2026-08-03 02:24:36+00:00 # Local vs Frontier: Benchmarks for my Digital Nervous System * * LLM benchmarks are a contentious, if often unserious topic. There are many benchmarks. Some are made by companies that make LLMs. Some are hilariously pie-in-the-sky like AGI ones. But most of them are just a number, and most of us just look for "does number go up?" On top of that, you'll see all over the internet: "I use <~12B param local model> and it's just as good as <3T param frontier model>." But you rarely see, in these type of comments, any actual details about what they are trying to do with the models. As I've been building out my "Digital Nervous System", I decided to test out which local models would potentially be best, and what compromises I would be making. If you download any local model and ask "What do you know about To test the current state of local and frontier models in my digital nervous system context, I set up four benchmarks: - Figure out who my sister is. - Figure out who had a higher GPA in college - me, or my wife? - Figure out why there are 4 different dentist appointments on my calendar last Thursday. - Look at the MotionCor3 source code and figure out whether TIFF decompression is faster if done on the CPU or the GPU. What kind of benchmark questions are these? All a bit deeper than they look. Here's the explanations for each: I've developed an internal tool called K_ir. This is basically an annotated, embedded database with lexical and semantic search over every digital record I control or possess. This includes email, calendar, messaging apps, etc. This question is tricky because it needs to understand who I am, figure out who my sister might be by name search, but also incorporate that her name has changed post-marriage. This is the type of thinking that doesn't always show in a benchmark. Similar to the above, it must understand who I am, as well as who my wife is, and figure out her name change. Also, there are no official GPA documents sitting around anywhere, so it will need to be decent at searching and sorting. This test, as well as the above, is meant to represent how useful the models are with incomplete instructions that rely on context and inference to follow through - the exact type of use case a personal assistant will need heavily. Direct result of previous attempts at a personal assistant, and calendar timezone hijinks, but also with a gotcha built in. I somehow wound up with dentist appointments listed in my calendar as a result of timezone malfunctions, however, as a result of rendering malfunctions, there's also a fourth showing at a different time. A tricky question to deconvolute. OK this is at first a bog standard coding question. MotionCor3 is used in cryo-EM to align raw, non-motion corrected frames to a single motion corrected frame. These are often large (500 MB+) TIFF files that have been LZW compressed, and must be decompressed before the GPU can do the frame alignment. Obviously, GPUs tend to be much faster 1:1 than CPUs at decompression, plus you are sending a smaller file through the PCI-E bus, so you'd suspect that GPU would win. However, if you are spending GPU cycles on decompression, they are not available for the motion correction. Therefore, it's faster overall to do decompression on CPU. This isn't really ever explicitly stated in the codebase, but there is some forensic evidence that points to it. I tested the following models: **Qwen 3.6 35B-A3B** **Qwen 3.6 27B** **Bonsai 27B** **DeepSeek V4 Flash** (API) **Laguna 2.1S** (API) **Claude Sonnet 5** (Subscription) (Thinking: High) **Claude Opus 5** (Subscription) (Thinking: High) # Results (No Claude) ## 1) Who is my sister? All five non-Claude models got the right person, but they went about it differently, and some clearly did it worse. **Qwen 27B** and **V4 Flash** did it properly: first, they figured out who I was, looked at the entity graph in K_ir, found my asserted relations, noted that the maiden name is an alias on the same record, and produced the result. **V4 Flash** noticed that a plain lexical search for "sister" is dominated by a band called Sister John Angela (highly recommend btw), and determined that the entity graph was the right tool because the relation was asserted rather than inferred. **Laguna** started with lexical search but eventually got there via the assertion. **Qwen 35B-A3B** got the right name without ever querying K_ir. It searched the agent's own conversation memory instead (oops). **Bonsai** also found the record, but there is where the quantization limits begin to show their cracks. It read her entity profile and told me my sister is deeply interested in cryo-EM. She is not - it somehow conflated her interests with my interests because we are related in the graph network. **Sonnet 5** figured this out no problem, linking entities and names. **Opus 5** went a step further, including that my sister was older than me. While I didn't ask for that, this is not undesirable for a system that is supposed to infer important context. However, there was no reason to surface it, other than perhaps to show off? Which is a constant criticism I have of **Opus 5**. ## 2) Whose GPA was higher? My wife's, and while all the models that finished agreed, err, they didn't all finish. **V4 Flash** gave the best non-Claude answer here. It found her transcript, found my self-reported number in a conversation with a colleague, and then noticed something none of the local models did: the December 2015 email thread it was reading is us trading transcripts with each other - hers pasted into the body, mine attached as PDFs. K_ir doesn't include the attachments, just the body, and it correctly identified this as a limitation. It then went and pulled all the other mentioned instances of my GPA and constructed a rough, but incomplete timeline. This is as well as a model could have done in this context. **Qwen 27B** found the documents, noted that hers was higher (and final) but noted that I had not yet graduated at that point, and that my GPA may have gone up after this. This is what I consider the correct answer all things considered, although there isn't necessarily a hard and clear answer to this one. **Qwen 35B-A3B** compared her transcript against a number I once mentioned in a chat message. Not the worst outcome, but it was not thorough and did not search email records. **Laguna** searched, found nothing, and asked me to supply the numbers. **Bonsai** spent fourteen minutes and produced no answer at all. It delegated a personal question to a Kagi web search subagent, and that subagent's first query was the literal tool string k_ir__now(). **Sonnet 5** was sufficient here and noted that not all of my wife's entries with her maiden name were linked to her current name. **Opus 5** wound up about the same, with a few more details that I never asked it for. ## 3) The four dentist appointments. **Qwen 27B** did quite well: three calendar events, one appointment, and also correctly identified that the phantom fourth was the TZID="UTC-06:00" copy being indexed six hours early because that identifier wasn't valid, and NextCloud falls back to UTC. It also noticed mid-run that cross-referencing in K_ir would be beneficial here, and enabled the extension itself to do that. **V4 Flash** got the same structure and offered to clean it up. **Laguna** identified the timezone bug and pointed out a potential prevention technique. **Qwen 35B-A3B** produced a confident four-row table of four events. There are three. It manufactured one to match the incorrect premise in my question, which is the single worst failure mode in the whole benchmark for something that is supposed to allow you to outsource attention to fine details. You could perhaps argue that this was an unfair question, but seeing as how the other models all got it, I think this is just insufficiency on the part of **Qwen 35B-A3B**. **Bonsai**, to its credit, successfully found the timezone bug and then screwed up the conversions, calling a 22:00 UTC event "8:00 PM". A timezone conversion error inside a question about timezone conversion errors. **Sonnet 5** did surprisingly poorly here. It correctly identified the duplicate entries, but it did not figure out that it was due to the time zone issues. Instead it just hand-waved to "a duplicate bug". **Opus 5**, of course, over-delivered. It went through and found the timezone issues, but then also searched old chats and found the exact set of queries that led to its initial creation. ## 4) MotionCor3 TIFF decompression. **Qwen 27B**, **V4 Flash**, **Laguna** and **Sonnet 5** all discovered at least part of the nuance. They all found existing GPU code for EER decompression, and noted that the lack of it for TIFF seemed intentional. However, none of them reported the correct reason why. They all defaulted to "it's because you can't parallelize the TIFF file decompression due to the serial nature decoding stream" when in reality that is simply not true, at least with cryo-EM data, which has 5k strips per image, and each strip can be a separate thread on the GPU, giving you effective parallelization even with the serial decoding stream. **Qwen 35B-A3B** reached the same answer in 41 seconds but tried to open .cpp files with an image reader eight times before falling back to cat. **Bonsai** found the EER code and then declared the EER GPU decoder the clear winner, which totally missed the question being about TIFF. The hilarious one here was **Opus 5**. It was incredibly confident it couldn't be parallelized. It found a half-implementation in the source and used it as proof, cited the README, crowed about finding a latent bug, and in general pronounced that it couldn't be multi-threaded and so CPU was faster. It can be done, I've done it, the code runs, it engages up to 1000 threads per GPU and sees a near-linear increase in speed (up to a point). When I challenged it on this later while having it proof-read this post, it remained confidently incorrect and suggested a rewording that had me singing an entire ballad of its praises: Never change Anthropic... A quick reminder, the entire reason not to do GPU decode here is because it competes with GPU motion correction. While it's technically faster to decode on GPU vs CPU, motion correction itself on GPU is still slower than decode on CPU, so any decoding done on GPU is bottlenecking throughput. While the models correctly identified not to do it, they all failed to identify the proper reason. # Speed and Cost | Model | Weights on disk | Context | VRAM in use | tok/s | |---|---|---|---|---| Qwen 35B-A3B | 20.2 GB (Q4_K) + 0.8 mmproj | 256k | ~35 GB | 107 | Qwen 27B | 19.0 GB (Q5_K_XL) + 0.9 mmproj | 128k | ~32 GB | 33.5 | Bonsai | 6.7 GB (Q2_0 ternary) + 0.6 mmproj | 128k | ~22 GB | 42.5 | Laguna | — (API) | 1M | — | 52–78 | V4 Flash | — (API) | 1M | — | 20–31 | Tokens weren't recorded for the goose-driven runs (they ran sessionless), but tool calls were, and they're the better proxy for how hard something actually worked: | Model | sister | GPA | dentist | MotionCor3 | |---|---|---|---|---| Qwen 35B-A3B | 3 | 35 | 14 | 15 | Qwen 27B | 8 | 31 | 18 | 12 | Bonsai | 6 | 14 | 11 | 35 | Laguna | 9 | 7 | 15 | 21 | V4 Flash | 9 | 39 | 19 | 23 | Sonnet 5 | 5 | 31 | 11 | 9 | Opus 5 | 20 | 16 | 24 | 15 | For the Claude models I do have exact token counts, which map onto cost: | sister | GPA | dentist | MotionCor3 | total | | |---|---|---|---|---|---| Sonnet 5 | 50k | 162k | 65k | 74k | 351k | Opus 5 | 75k | 110k | 107k | 86k | 378k | Published rates for the four hosted models, per million tokens: | Model | Input | Output | How I accessed it | |---|---|---|---| V4 Flash | $0.09 | $0.18 | OpenRouter | Laguna | $0.09 | $0.18 | OpenRouter | Sonnet 5 | $3.00 | $15.00 | subscription, priced here at API rates | Opus 5 | $15.00 | $75.00 | subscription, priced here at API rates | # Conclusions OK, what did I learn? First of all, both **Bonsai** and **Qwen 35B-A3B** fall below the quality threshold for daily use in this context. Both had truly critical issues just generally conceptualizing what they were trying to do, tracking down inaccuracies, and understanding nuance. **Qwen 35B-A3B** did slightly better, but it also produced the original failure on question 2, so my priors didn't sufficiently update to bring it back into the mix. Obviously, **Opus 5** was the best. But even **Opus 5** was not perfect - for instance, as mentioned, in proof-reading this blog post, **Opus 5** told me that my explanation of the MotionCor3 programming problem is wrong because it concluded in its analysis that TIFF files can't be chunked which would bottleneck GPU decompression. The big surprise for me, though, was basically two things: For assistant usage, there is effectively no difference between the **Qwen 27B**-tier models that fit into ~20-30 GB, and the** Laguna**/** V4 Flash**models that want more like 100-150 GB. Those models probably far outweigh** Qwen 27B**at coding, and have much larger contexts which simply makes them more usable, but for interacting with my digital nervous system? No difference.**Sonnet 5** makes no sense. It was matched by virtually every other model in these tests, and is significantly more expensive. I do not understand the point or positioning of this model. Is it more efficient than Opus? Yes. Is it more efficient than Qwen, DeepSeek etc? No. This feels like a model for Anthropic to point to when people complain about rate limits, but if you are not fully in the Anthropic ecosystem only, there is no use for it. It is performance matched or beaten by many other models that are free or cheaper. A few more notes: I don't want to beat up on **Bonsai** here. It is an incredible model, taking like 6-7 GB for weights. Unfortunately, coherence is one of the first things that seems to go with non-native quantizations. Perhaps it'd perform better with native ternary training. I find it quite useful as is in narrow scopes like annotating database information for K_ir. And Opus. The elephant in the room. The most technically impressive for sure. Wildly expensive. And multiple instances of being overly confident when it really had no business being. Smartest-kid-in-the-room vibes. And paying those prices to have it argue back with you about things...maybe it's rose-tinted glasses but I don't remember Opus ~4.5 doing that. It's really annoying and I hope Anthropic gets a handle on it soon (or that DeepSeek v4 Pro final comes out soon).